Using Git submodules for unified build scripts

I have recently worked on unifying the build scripts for many of our software components at Glasses Direct.

At each repository for a component, we typically have two files build.sh (for developer builds, relying on an existing virtualenv) and jenkins-build.sh (for Jenkins builds, where the Jenkins job re-creates the virtualenv if it does not exist or if the requirements files have changed since last build – we rely on the md5 checksums for that).

We already had sort-of-standardised scripts, but they were replicated in many repos. It defeated the point of having a single source for these. I decided to give git submodules a try, as a way to make code from one repo (our build scripts) easily accessible from others.

Adding the submodule

Our build scripts are (far than ideally) in a specific branch buildtools of a repo called libraries.

Here is how we add that submodule:

git submodule add git@github.com:glassesdirect/libraries.git

If we run git status we will see git has added a .gitmodules and a libraries directory.

Now we want to point to the desired branch:

cd libraries
git checkout buildtools
cd ..
git add libraries  # for the branch change

Once that is done, we can commit and add the changes.

The resulting scripts

Now the build scripts in each repo can look as simple as… build.sh:

#!/bin/bash
source libraries/buildtools/build-functions.sh
do_build lab_export_service

And jenkins-build.sh:

#!/bin/bash
source libraries/buildtools/build-functions.sh
do_jenkins_build $*

The bulk of our build scripts is now in libraries/buildtools/build-functions.sh. There we can have the basic building blocks as bash functions, such as:

function check_if_requirements_changed() {
# Verify whether any of the PIP requirements file has changed.
# This may save us the creation of a virtual environment if all is equal.

   MD5FILE=".requirements.md5"
   newMD5=`md5sum *requirements.txt | md5sum`
   oldMD5=`cat $MD5FILE`

   if [ "${oldMD5}" == "${newMD5}" ]
   then
      REQS_CHANGED=0
   else
      echo "Requirements files changed."
      REQS_CHANGED=1
   fi
}

etc.

• build
• git
• jenkins