Using Git submodules for unified build scripts

I have recently worked on unifying the build scripts for many of our software components at Glasses Direct.

At each repository for a component, we typically have two files (for developer builds, relying on an existing virtualenv) and (for Jenkins builds, where the Jenkins job re-creates the virtualenv if it does not exist or if the requirements files have changed since last build – we rely on the md5 checksums for that).

We already had sort-of-standardised scripts, but they were replicated in many repos. It defeated the point of having a single source for these. I decided to give git submodules a try, as a way to make code from one repo (our build scripts) easily accessible from others.

Adding the submodule

Our build scripts are (far than ideally) in a specific branch buildtools of a repo called libraries.

Here is how we add that submodule:

git submodule add

If we run git status we will see git has added a .gitmodules and a libraries directory.

Now we want to point to the desired branch:

cd libraries
git checkout buildtools
cd ..
git add libraries  # for the branch change

Once that is done, we can commit and add the changes.

The resulting scripts

Now the build scripts in each repo can look as simple as…

source libraries/buildtools/
do_build lab_export_service


source libraries/buildtools/
do_jenkins_build $*

The bulk of our build scripts is now in libraries/buildtools/ There we can have the basic building blocks as bash functions, such as:

function check_if_requirements_changed() {
# Verify whether any of the PIP requirements file has changed.
# This may save us the creation of a virtual environment if all is equal.

   newMD5=`md5sum *requirements.txt | md5sum`
   oldMD5=`cat $MD5FILE`

   if [ "${oldMD5}" == "${newMD5}" ]
      echo "Requirements files changed."