git notes

Various notes for doing things with git

Git subrepos

git subrepos are an additional git command that “clones” an external git repo into a subdirectory of your repo. Later on, upstream changes can be pulled in, and local changes can be pushed back. Simple.

To work with subrepos, you should clone this repository: github.com/ingydotnet/git-subrepo and then echo ‘source /path/to/git-subrepo/.rc’ >> ~/.bashrc

Be sure to read the README.pod in that repository to better understand how subrepos work.

Clone a repo

    git subrepo clone https://github.com/pixelated/leap_platform.git leap_platform

Update subrepo

Using pull

    git subrepo pull puppet/modules/shorewall --branch augeas_is_neeed --remote https://gitlab.com/varac/shorewall.git --update

Issues:

    git-subrepo: Can't find a common ancestor between refs/subrepo/mod_security/fetch and subrepo/mod_security. This might be the case if someone used push --force on the subrepo. You need to find out why and possibly reclone the subrepo.

Reclone with force

if pull doesn’t work:

    git subrepo clone -f -b master https://github.com/pixelated/leap_platform.git leap_platform

Push to a subrepo

Lets say you made some changes in a subrepo – you commit them and then push them back like follows:

git subrepo push code/modules/sympa -r git@0xacab.org:riseup-puppet-recipes/sympa.git

Remove a subrepo

    git subrepo clean --force modules/mysql
    rm -rf modules/mysql

Git’s submodules (old)

Git has something called submodule support. This allows a repository to contain, as a subdirectory, a checkout of some other external repository. Submodules maintain their own identity; the submodule support just stores the submodule repository location and commit ID.

The git submodule command is only available in git versions greater than 1.5.3.

Working with submodules

First lets look at how to track the existing submodules that we have setup, then we can go on to describe how to make changes to those, and then finally how to add new ones.

Tracking our submodules

If you have checked out the latest puppet repository, you will see under the modules directory some different modules. For the purposes of our example, we will talk about the puppet module called ‘sshd’.

If you look in the modules/sshd directory, you wont see anything at all. Lets look at the submodule status though:

$ cd modules/sshd
$ ls -a
.  ..
$ git submodule status
-d266b9873ad50488163457f025db7cdd9683d88b modules/sshd (heads/master)

NOTE: The commit object names shown above would be different for you.

In order to get the actual data, you need to pull down the submodule. First initialize the known submodules (this will add submodule repository URLs to .git/config):

Do this from the top-level git repository:

$ git submodule init

Now use git submodule update to clone the repositories and check out the commits specified in the superproject:

$ git submodule update
$ cd modules/sshd
$ ls -a

You will see the contents of the repository now.

Making changes to a submodule

To make a change in a submodule, you should make the change directly to the originating submodule, that is, make the change in that repository and then push that repository. Don’t try to make the change to the submodule within the repository itself.

first update the origin repository

To update a submodule, you first update the originating repository and push that. So, in the below example we first checkout the originating “module_sshd” repository, make some changes, and then push it. This repository is a totally separate repository, and is unconnected to our main repository. These changes will not be reflected in the main repository until the next step:

$ git clone gitolite@labs.riseup.net:module_sshd
$ cd module_sshd/
$ <do work>
$ git commit -a ; git push

now update the ‘superproject’

The superproject is the repository that holds the checkout of the submodule. In our case we want the puppet repository to have the new changes we made, so we go to that repository and then update the module_sshd submodule from its repository. This brings in the changes we made above, into the main repository, which we then can push to the main repository.

$ cd modules/sshd

Make sure you are not in a branch in the submodule. ‘git branch’ should show something like this, where ‘master’ is selected.

$ git branch
* master

If you haven’t done this to this particular module yet (master isn’t selected above or “You are not currently on a branch” error), you might need to do

$ git checkout master

Proceed…

$ git pull
$ cd ..
$ git add sshd

Lets stop here for a second, because there is a tricky problem that I kept running into that I want to note. If you did “git add sshd/” instead of “git add sshd”, you will be in a way different situation that will cause problems. Notice the only difference is the trailing slash. If you include the trailing slash, then you will be adding the actual files, rather than the commit ID of the updated subproject. If you were to do a ‘git status’ at this point, you would find out that you have added all this stuff you did not want to add. You only want to see a simple change like the following:

# On branch master
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#       modified:   modules/sshd
#

If you see all the files added as well, you’ve done it wrong. Do a ‘git reset’ then re-do the add, making sure you do not include the trailing slash.

Once you’ve made sure that you got the right thing added, then commit it:

$ git commit -m 'update to latest sshd submodule'
$ git push

Adding a new submodule

To add a new submodule, you need to know the git repository URI and where you want to put it. Once you have that information, you will have to add the submodule, from the top-level directory of the git repository, like follows:

pond (git)-[master]-% pwd
~/riseup/dev/puppet
pond (git)-[master]-% git submodule add git://labs.riseup.net/module_shorewall development/modules/shorewall   
Initialized empty Git repository in /home/micah/riseup/dev/puppet/development/modules/shorewall/.git/
remote: Counting objects: 243, done.
remote: Compressing objects: 100% (128/128), done.
Receiving objects: 100% (243/243), 31.21 KiB, done.
remote: Total 243 (delta 81), reused 223 (delta 72)
Resolving deltas: 100% (81/81), done.

Then we can look at the current status of the submodule state of the repository by using the ‘git submodule status’ command. This provides the SHA-1 of the currently checked out commit for each submodule. Each SHA-1 will be prefixed with – if the submodule is not initialized and + if the currently checked out submodule commit does not match the SHA-1 found in the index of the containing repository. This command is the default command for git-submodule.

pond (git)-[master]-% git submodule status                                                    
-7633322b490c02e2ba44faa1a0fd534577df6d82 development/modules/shorewall
 49eb18e510bb8dfedc762fd00bd3dff06e72d6de modules/apt (heads/master)
 fa4a86a8a6ef0d95475544c9c3bbc67af5bf9f45 modules/backupninja (remotes/origin/HEAD)
 26d018ba16969f942f8dcaa497bea3eac34437a6 modules/monit (remotes/origin/HEAD)
 ec14ad058ce969ddb1e8dc9619ad4785dfa51a51 modules/mysql (remotes/origin/HEAD)
 4d376ab62715f3ccfe4c1bc2e44a0df6527f014c modules/runlevel (heads/master)
 5161c4332a7f5b79c25c922a4450cec97b2ad276 modules/sshd (remotes/origin/HEAD)

As you can see the first SHA-1 is the newest submodule I just added, and its being put into our development/modules directory with the name ‘shorewall’. It has the – prefixed, so its not currently initialized, so we need to init it (we could have done this in one step during the add command above):

pond (git)-[master]-% git submodule init                                     
Submodule 'development/modules/shorewall' (gitolite@labs.riseup.net:module_shorewall) registered for path 'development/modules/shorewall'

Now a git status shows us that the .gitmodules file has been updated, and there is a new file (which is actually just the submodule) that needs to be committed:

pond (git)-[master]-% git status
# On branch master
# Changes to be committed:
#   (use "git reset HEAD <file>..." to unstage)
#
#	modified:   .gitmodules
#	new file:   development/modules/shorewall
#

Once you commit this and push it, then anyone who pulls from your repository will then have to follow the procedure outlined further up in this document for how to initialize and checkout submodules.

removing and readding a submodule

local:

git rm modules/foo
vi .git/config  # remove submodule
rm -rf .git/modules/modules/foo
git commit
git push

on puppetmaster:

su - puppet
cd /etc/puppet
git submodule sync
vi .git/config  # remove submodule
rm -rf .git/modules/modules/passenger
rm -rf modules/passenger

local:

git submodule add https://labs.riseup.net/git/module_foo.git modules/foo
git commit
git push

back to puppetmaster:

git pull
git submodule update --init