Foray into Jenkins, Git, Docker, and Photon: Part 4

In previous Foray into Jenkins, Puppet, Docker, and Photon posts, I was able to clone a Photon OS VM (part 1), deploy a Docker container into the Photon OS VM (part 2), and do automated load testing (part 3). Now it is time to look at improving the security of my Git repository. Given the number of scripts out there to look through GitHub for API keys, usernames, and passwords in order to rack up serious bills on Amazon and other cloud services, it behooves us to be extra vigilant.

The simple fact of the matter is that we embed quite a few things into our source code files that we should not. Such items include but are not limited to:

  • API keys
  • PII (even for testing)
  • Passwords
  • Usernames
  • IP addresses.

The list is fairly long and growing daily. For the most common problems, however, we can easily scan the code, report any findings, and deny a commit or even push if they exist.

Why is this required? Because no code should be placed in a public location that contains sensitive information. As such, we need to be made aware of such findings, they should be reported, and they should not happen. Being able to add such tools to the commit process of any source code control mechanism allows us a great amount of power and flexibility.

However, this is not always enough. For auditing purposes, the information on commit problems should be gathered, sifted, and made part of any security governance. This way, issues can be brought up and studied in order to provide the proper awareness training to developers.

The simple case is to be aware that you may be placing into GitHub the keys to your kingdom. If that happens, you can be sure that your AWS account will be hacked and something created to rack up your expenses. Not only are alerts required within AWS, but you need to ensure that your repositories do not contain the data.

I wrote this simple Git hook to determine whether I am sending to my Git repositories anything that would be an issue. The script will follow, but first, how do I set this up?

Step 1: Create the script named pre-commit within /usr/share/git-core/templates (that script is at the end of this post).

Step 2: Ensure the script is executable: sudo chmod +x /usr/share/git-core/templates/pre-commit

Step 3: Clone a repository—any repository will do.

Step 4: Verify that the hook was copied to the proper location, which is .git/hooks/pre-commit within the repository you cloned.

Voila, you now have a pre-commit script that will scan  your code for API keys, PII, IP addresses, etc., before you commit it to Git. On finding an error, it will exit with warning messages and log those messages to syslog. Syslog logging allows other tools to track such issues as well. Since there is a way within Git to not call the pre-commit script, the logging allows others to monitor what is happening.

Refer to https://git-scm.com/book/en/v2/Customizing-Git-Git-Hooks for more information on how to use the hooks provided. There are even server side hooks that work on a push and will not allow the push to succeed if the scan shows that the files being uploaded contain unwanted items.

Also, check out https://securosis.com/blog/my-500-cloud-security-screwup for things to do on the Amazon side to help warn of API issues.

The script follows and has a TODO list that is growing:

  • Scan for IPV6 IP addresses
  • Perform static code analysis for common security issues, such as buffer overruns
  • Add more PII scans
  • Add more hash scans
  • Improve username scans for all forms of usernames, not just those domain users that are coded with user@domain.
pre-commit script
#!/bin/sh
#
# Copyright (c) 2015 AstroArch Consulting, Inc. All rights reserved
#
#
# A hook script to verify what is about to be committed.
#       -- Looks for IPV4 Addresses
#       -- Looks for Usernames
#       -- Looks for Domain Names
#       -- Looks for Passwords
#       -- Looks for Hashes/API Keys
# Called by "git commit" with no arguments.  The hook should
# exit with non-zero status after issuing an appropriate message if
# it wants to stop the commit.
#
# Reference: http://www.unix-ninja.com/p/A_cheat-sheet_for_password_crackers
#

# Redirect output to stderr.
exec 1>&2

# First Check for IP
regex4='([0-9]{1,3}[\.]){3}[0-9]{1,3}'

# Hashchecks including base64 and API Keys
hashchecks='[0-7][0-9a-f]\{7\}[0-7][0-9a-f]\{7\} \$2a\\$\08\\$\(.\)\{75\} ([0-9a-zA-Z]{32}):(\w{16,32}) ([0-9a-zA-Z]{32}):(\S{3,32}) \$H\$\S{31} \$P\$\S{31} \$S\$\S{52} \$1\$\w{8}\S{22} \$apr1\$\w{8}\S{22} \$6\$\w{8}\S{86} [0-9A-Za-z+/]{60,}= [0-9A-Za-z+/]{20,}'

# CC/SSN/Passport/etc
pii="'4[0-9]{3}[ -]?[0-9]{4}[ -]?[0-9]{4}[ -]?[0-9]{4}' '5[0-9]{3}[ -]?[0-9]{4}[ -]?[0-9]{4}[ -]?[0-9]{4}' '\b3[47][0-9]{13\b' '\b3(?:0[0-5]|[68][0-9])[0-9]{11}\' '6011[ -]?[0-9]{4}[ -]?[0-9]{4}[ -]?[0-9]{4}' '\(?:2131|1800|35\d{3})\d{11}\b' '3[47][0-9]{2}[ -]?[0-9]{6}[ -]?[0-9]{5}' '[0-9]{3}[ -]?[0-9]{2}[ -]?[0-9]{4}' '[0-9]{4}[ -]?[0-9]{2}[ -]?[0-9]{4}' 'C0[0-9]{7}' '[23][0-9]{8}'"

icount=0
ifile=""
dcount=0
dfile=""
hcount=0
hfile=""
pcount=0
pfile=""

for x in `/usr/bin/git diff-index --cached HEAD|/usr/bin/awk '{print $NF}'`
do
        # check for IP Addresses in files

        /bin/grep -Eo "$regex4" $x >& /dev/null
        if [ $? = 0 ]
        then
                if [ $icount = 0 ]
                then
                        ifile="$x"                else
                        ifile="$ifile $x"
                fi
                let icount=$icount+1
        fi

        # check for user@domain
        /bin/grep -Eo "[[:alnum:]]+\@[[:alnum:]]+" $x >& /dev/null
        if [ $? = 0 ]
        then
                if [ $dcount = 0 ]
                then
                        dfile="$x"
                else
                        dfile="$dfile $x"
                fi
                let dcount=$dcount+1
        fi

        # Hashes
        /bin/egrep -oE '(^|[^a-fA-F0-9])[a-fA-F0-9]{32}([^a-fA-F0-9]|$)' $x | /bin/egrep -o '[a-fA-F0-9]{32}' >& /dev/null
        if [ $? = 0 ]
        then
                if [ $hcount = 0 ]
                then
                        hfile="$x"
                else
                        hfile="$hfile $x"
                fi
                let hcount=$hcount+1
        fi

        for y in $hashchecks
        do
                /bin/egrep -o "$y" $x >& /dev/null
                if [ $? = 0 ]
                then
                        if [ $hcount = 0 ]
                        then
                                hfile="$x"
                        else
                                hfile="$hfile $x"
                        fi
                        let hcount=$hcount+1
                fi
        done

        for y in $pii
        do
                /bin/egrep -o "$y" $x >& /dev/null
                if [ $? = 0 ]
                then
                        if [ $pcount = 0 ]
                        then
                                pfile="$x"
                        else
                                pfile="$pfile $x"
                        fi
                        let pcount=$pcount+1
                fi
        done
done

doexit=0
logstr=""
if [ $icount -gt 0 ]
then
       echo ""
        echo "Cannot continue due to IP in file(s):"
        for x in $ifile
        do
                echo "  $x"
        done
        echo ""
        /bin/logger "$USER attempted to commit with IP included: $ifile"
        doexit=1
fi
if [ $dcount -gt 0 ]
then
        echo ""
        echo "Cannot continue due to Domain Users in file(s):"
        for x in $dfile
        do
                echo "  $x"
        done
        echo ""
        /bin/logger "$USER attempted to commit with Domain Users included: $dfile"
        doexit=1
fi
if [ $hcount -gt 0 ]
then
        echo ""
        echo "Cannot continue due to Password Hashes in file(s):"
        for x in $hfile
        do
                echo "  $x"
        done
        echo ""
        /bin/logger "$USER attempted to commit with Hashes included: $hfile"
        doexit=1
fi
if [ $pcount -gt 0 ]
then
        echo ""
        echo "Cannot continue due to PII in file(s):"
        for x in $pfile
        do
                echo "  $x"
        done
        echo ""
        /bin/logger "$USER attempted to commit with PII included: $pfile"
        doexit=1
fi
if [ $doexit -gt 0 ]
then
        exit 1
fi
Edward Haletky

Edward L. Haletky, aka Texiwill, is an author, analyst, developer, technologist, and business owner. Edward owns AstroArch Consulting, Inc., providing virtualization, security, network consulting and development and TVP Strategy where he is also an Analyst. Edward is the Moderator and Host of the Virtualization Security Podcast as well as a guru and moderator for the VMware Communities Forums, providing answers to security and configuration questions. Edward is working on new books on Virtualization.

Leave a Reply

Your email address will not be published. Required fields are marked *

three − 2 =