Git post-receive for multiple remote branches and work-trees
A while ago I set up my server to automatically deploy new content from my git repositories when changes where pushed to them. This automatic workflow is great and fairly simple to setup, but I recently needed to add a twist to it: what if I want to deploy separate remote branches to their own individual work-trees? Here’s how…
Below is a walkthrough of my findings along the way of debugging the initial implementation, but if you’re in a hurry, you can go straight to the solution
Initial, single remote branch setup
The initial setup has been explained here before, so you only have to follow these basic instructions to get yourself going on a single master branch auto-deployment.
More branches, more work-trees please!
Right, now let’s say you want not only the master branch of your code to be deployed to a live website, but you also want the development branch to be deployed to it’s own web root directory so you can test a development version of your site (and potentially f** it all up) while the live site still runs the last “gold” version of your code. You will undoubtedly have several local branches, several corresponding tracking remote branches. For this setup, you will also need several corresponding work-trees, all with their automated deployments upon pushing to the main repository.
Unsurprisingly, there is very little to change to our method explained in the previous setup, but the basic instructions I found did not initially work on my Ubuntu server, so you’ll have to be aware of a few things.
Initial findings
Dylan over dylanserver.com posted this handy bit of code earlier this year:
if ! [ -t 0 ]; then read -a ref fi IFS='/' read -ra REF <<< "${ref[2]}" branch="${REF[2]}" if [ "live" == "$branch" ]; then git --work-tree=/var/www/dylanstestserver.com/ --git-dir=.. checkout -f echo 'Changes pushed live.' fi
If you read carefully the post, you can notice that the checkout command is still there, but augmented with the –work-tree and –git-dir arguments.
Great! Thats a great starting point, firing up vi on my remote server, I edited my ‘post-receive’ hook and pasted the following code in it (after changing the branch name to “dev” instead of “live” to test only with my dev site). Note that I had also added the shebang at the top of the file…
#!bin/sh if ! [ -t 0 ]; then read -a ref fi IFS='/' read -ra REF <<< "${ref[2]}" branch="${REF[2]}" if [ "dev" == "$branch" ]; then git --work-tree=/path/under/root/dir/dev-site/ --git-dir=.. checkout -f echo 'Changes pushed dev.' fi
The next git push executed the hook but I got the following error in the git push output:
read: 4: Illegal option -a
The right shebang
The problem was that the shebang I added uses sh, which, according to some stackoverflow answers doesn’t use bash on ubuntu (my server). Incidentally, I found this out after fiddling around with the read command and removing the -a option, only to find out that an “unexpected redirect” has been found.
The solution was simple: change the shebang to:
#!bin/bash
Now, that hook was going to checkout my dev branch into my development site’s root directory… err… Not quite yet.
The right git directory
The next error I encountered was:
fatal: Not a git repository: '..'
That’s from the script itself, but I can’t understand why this was failing, as –git-dir should accept absolute or relative directories. However, the GIT_DIR environment variable should be set to the current directory, which during the push, is the git directory. So I elected to remove the option altogether:
#!bin/sh if ! [ -t 0 ]; then read -a ref fi IFS='/' read -ra REF <<< "${ref[2]}" branch="${REF[2]}" if [ "dev" == "$branch" ]; then git --work-tree=/path/under/root/dir/dev-site/ checkout -f echo 'Changes pushed dev.' fi
Now the push didn’t report any error, and “some” code to the correct development website root directory! Yay! BUT… That code was the MASTER branch’s code. Not the DEV branch.
The right solution
As all we are doing in the hook is performing a git checkout (with some options), we can simply pass the branch we want to checkout as a parameter and all works well! I’m yet to understand what environment variables are set and used by the hook, but for the moment, my final post-receive hook looks like this:
Edit - Thanks to Peter and his comments below, the script below now works while pushing multiple branches at once! Thanks to him!
#!/bin/bash while read oldrev newrev ref do branch=`echo $ref | cut -d/ -f3` if [ "master" == "$branch" ]; then git --work-tree=/path/under/root/dir/live-site/ checkout -f $branch echo 'Changes pushed live.' fi if [ "dev" == "$branch" ]; then git --work-tree=/path/under/root/dir/dev-site/ checkout -f $branch echo 'Changes pushed to dev.' fi done
You will notice I have added the master branch in there too, and you can expand the script with whatever branch you want to checkout, along with the correct work-tree.
comments
Matt Tue, 01 Nov 2011 - 23:46
It seems that this version of the hook doesn't fully work in scenarios where I'm pushing both master and dev (read multi-branches push) to the remote. As the script is executed once, I'm not sure if the references are correctly set. Maybe all this should go in a different hook? Suggestions?
Peter Fri, 02 Dec 2011 - 14:57
To solve Matt's problem I think this whole script should be wrapped in a while loop. I took the post-receive script from http://book.git-scm.com/5_git_hooks.html and combined it with yours to come up with a new version. I haven't tested it, but it should would work for multi-branch commits:
Matt Fri, 02 Dec 2011 - 15:03
Hi Peter, I'll give that one a go, that looks alright to me. I did not have time yet to investigate myself but that's what I had in mind as well. I'll try that asap and report. Thanks! (And I'll have to find a way to format code in comments...)
Peter Fri, 02 Dec 2011 - 18:38
The last one I posted won't work unless you change the shabang. Here's a better version that should be a little easier to read. I have tested this one successfully.
Peter Fri, 02 Dec 2011 - 18:56
And here is a version you can use if your live site is on another server. Keep in mind that you will have to setup a user who can ssh to the other server using keys instead of passwords to authenticate. I’m sure you can find those instructions online.
Matt Sun, 04 Dec 2011 - 18:33
Hi Peter, I've edited your comments so they look alright. They won't show indenting as I don't apply the pretty formatter to those. That's for another time! Regarding your code, I can confirm that the second one you posted (for a solution on the same server) does work properly and I've updated my original post as well to reflect those changes, which should help a few out there! Thanks a lot!
Marty Sun, 01 Jul 2012 - 14:01
Many thanks for your solution and explanation. I tested it on Windows machine and had to do a little edit. I changed this row:
to this:
And everything work fine. Thanks again.
Matt Tue, 03 Jul 2012 - 16:04
Happy to see these little tips still help!
Gustavo Mon, 16 Jul 2012 - 03:13
I've a little problem here... Here is step by step that i'm doing:
For now, "tutto bene" (is fine). I want to use the master branch as development branch, so, any push to master branch, deploy to dev/test server, and i want to add another branch for production enviroment, "production" branch. So I edit my config and post-receive files in server:
Now I change some code and push it to test/dev (master):
Fine, works for test/dev deploy, now test with production branch:
I push production branch to the origin because, that way, other developers can merge and push to production your works... now I merge master into production, for deploy a copy of test/dev into production enviroment: git merge master git push Fine!!!, deployed into production server, awesome!! OK, here come the bad think, now i change to master branch to fix some bug, push to dev/test, and.... nothing, don't deploy to test/dev server... Try again and nothing...
............. not deploy, como on mother f....
............. not deploy Let me try mergin master into production:
............. deploy into production Come back to master an push some change...
............ deployed into dev/test server, but no the commit "fixed bug 3", instead the second commit .... :( Any suggestion?
Matt Tue, 17 Jul 2012 - 09:12
Hey Gustavo! Other than some permission issues, I don't see what could be wrong here, as your method looks ok to me. To debug what is happening, instead of doing a checkout in the "master branch block", you could stick in there a bunch of logging lines of code that would print the git history being pushed and on which $ref you are etc.
Jake Wed, 10 Apr 2013 - 18:38
Sorry to post in an old page. I have been trying to set up a deployment similar to this, but I have been having an issue with speed. With both the script I wrote and the script here, it will sometimes take 15 minutes for a push to complete. It seems to me that the problem is when switching between the two working directories. When I push the same branch as I had previously, it only takes a few seconds to make a small change. However, if I had last pushed the master branch and now push the dev branch, it will take 15 minutes to complete, even just to make one change on a 2kB file. Additionally, it this case, windows shows all the files in the directory as having been modified, rather than just the 1 file that had been changed and committed.
Jake Wed, 10 Apr 2013 - 18:42
EDIT: After the 15 minutes, the checkout is correct.
Matt Thu, 11 Apr 2013 - 08:49
Hey Jake, I'm not sure what the problem could be here. Are you sure your machine has the correct permissions and that your filesystem is ok?
David Tue, 13 Aug 2013 - 16:45
Wonderful. Just a few questions... I'd like to incorporate this method without running or managing a git server like gitolite,... preferring to rely on GitHub and Bitbucket for things like collaboration, user management, rwx permissions, etc. So... a) Does a server running git and ssh qualify for using this, or is something like gitolite a must? b) Imagining a scenario in which you have some code that is part of a site that needs local => stage and => live branches, as well as a version that lives on Bitbucket or GH,...erhm...what might that look like? In other words, is there a sensible way of managing both sharing/collaboration AND the convenience of "push publishing", or is that asking too much of one repo?
Matt Tue, 13 Aug 2013 - 17:02
Hi David, I'll try to answer your questions below: a) Gitolite is only a permissions management system for repositories, not git itself. This is a git hook, so it should work on any git install providing that you modify the script to use the correct file system. You can also write the same script in pretty much any language the shell of said server can interpret. This is bash, but you could use ruby and others. You can find a more advanced script doign something else here: http://git-scm.com/book/en/Customizing-Git-An-Example-Git-Enforced-Policy b) There is nothing that would prevent you from building a more complex script, checking what branch you are pushing to and deploying the right code to the right staging, integration, live etc servers. Just bare in mind that this particular hook above is only intended to work on the same machine than the repository is. But because these hooks are essentially scripts, you can do whatever you want in them, even stuff completely unrelated to git itslef, like sending an email to all your team telling them you've pushed code etc. I don't have example of such scripts handy, but that's essentially how heroku or openshift work. The sharing/collaboration is the pure git branch part, the push-publish is in the hooks. I don't think the limitation will be in the repo here, but in how complex your hooks and management system will be.
David Thu, 15 Aug 2013 - 01:02
Thanks, Matt. Cleared up a lot for me. This solution is working great for me. I'd also like to recommend this write-up... http://toroid.org/ams/git-website-howto ...which covers some of the fundamentals you cover in part I. You can use the script Peter came up with in this solution as well. Very clever.
Vincent Thu, 22 Aug 2013 - 19:55
This worked perfectly for me. I've been looking for exactly this sort of solution for months. Not sure how I didn't come across this earlier. I created a Gist with my slightly modified version. http://blog.ekynoxe.com/2011/10/22/git-post-receive-for-multiple-remote-branches-and-work-trees/ Hopefully that'll make it easier to contribute to this block of code in the future. Fork and modify to your heart's content. Thanks again!
Vincent Thu, 22 Aug 2013 - 19:56
Oops--the link above is incorrect, obviously. It should be this: https://gist.github.com/bigsweater/6311240#file-post_receive
Norbert Sun, 29 Sep 2013 - 12:07
I encountered a problem with pushing my master branch to the live site. My workflow is as follows: - make changes locally on develop branch, test, commit, push to develop - test on server (develop url) - merge develop into master on local, push to master The problem is that when I push to master, files that I deleted locally don't get removed on the server. New files are added as expected. This only happens when I push to master, pushing to develop works just fine. Any ideas?
Matt Tue, 01 Oct 2013 - 08:25
Norbert, I am unsure why this happens, but possible issues could include file permissions that could be different for whatever reason on the directories your master branch checks out into, or you may have had some files in there prior to deploying with git. If these files weren't in your repository before, then it's normal they won't get deleted since git doesn't track them. That's only speculation though. Matt
Guillaume Lang Fri, 21 Mar 2014 - 15:38
Hello, i just wont to say big thanks to you for this tip, thank you very much, and sorry for my bad english, I'm french!
Jesse Mon, 15 Sep 2014 - 20:37
Great article. One problem I have, is I want all of my files to be backed up in the repo but I want to prevent certain files/directories from being checked out into the live directory. So gitignore doesn't seem to be the solution since it prevents certain directories from making it into the git backup. For instance, I have a '_db' directory where I store database backups. I want those to be registered in my git repo, but I don't want it to be accessible in my web root. Any ideas how to dot that?
Matt Mon, 15 Sep 2014 - 22:47
Hi Jesse, Thanks for taking the time to comment. You are right, .gitignore will only help in case you don't want to *commit* files to your repo. As far as I'm aware, there isn't a way to tell checkout to ignore some files, that would complicate things a lot! However, since the hook is a bash script, you're free to do whatever you want in it. One solution would be to use the --work-tree option to target a temporary folder, from which you would copy only what you really need to your web root: Instead of
You could have something like:
There are quite a few options to chose from. Rsync / cp / find with xargs / etc etc. Pick the one you prefer! And you're not bound to bash either, you can write your hooks in ruby perl etc as well. Whatever suits you, make those examples your own! /!\/!\ I haven't tried this code, so just make sure you know where you're going with it!