Mirroring bzr to github

pokorra

Hello,

I have started mirroring our bzr trunk to a github repository, mainly so that I can host private branches of my customers (including configuration etc) at Github (launchpad.net/+tour/join-launchpad#commercial).
I am using github.com/termie/git-bzr-ng for syncing the repositories.

The initial checkout for git is much faster than bzr for some reason.

But it also has some benefits for the OpenPetra team:
You can use the better history of files for example:
github.com/tpokorra/openpetragi ... Service.cs
Bazaar messes up the history of a file quite badly by mentioning any merge that is related to a file by some means, which makes it very confusing.

All the best,
Timo

Edit: here are the commands that I am using:

wget raw.github.com/termie/git-bzr-ng/master/git-bzr --output-document=/usr/bin/git-bzr
chmod a+x /usr/bin/git-bzr
git bzr clone lp:openpetraorg openpetraorg
cd openpetraorg
git branch -a

result ->

bzr/master

* master

git remote add origin [@github.com">git@[deleted].com](mailto:@github.com">git@[deleted].com):tpokorra/openpetragit.git

git bzr sync
git merge bzr/master
git push

joejoe2010

Hi Timo,

I was trying the link github.com/tpokorra/openpetragi ... Service.cs but it sent me to a 404-error page.

Greetings
Joachim

pokorra

Hello Joachim,

the link in my original post works again. The link in your post has a space.

I had to reupload the branch, because the initial bridge from bzr to git did still contain some deleted files.

Now I am using this script:
felipec.wordpress.com/2012/11/13 ... -hg-bzr-2/
github.com/felipec/git/blob/fc/ ... remote-bzr

I submitted a patch so that it works with filenames containing Umlauts as well, and it is working fine now.

Here are the commands that I am using (on CentOS):

wget https://raw.github.com/felipec/git/fc/master/git-remote-bzr.py
chmod a+x git-remote-bzr.py
sudo mv git-remote-bzr.py /usr/bin/git-remote-bzr

install latest bzr:

sudo yum -y install gcc python-devel
wget https://launchpad.net/bzr/2.5/2.5.1/+download/bzr-2.5.1.tar.gz
tar xzf bzr-2.5.1.tar.gz
cd bzr-2.5.1
sudo python setup.py install

install latest git:

sudo yum -y install curl-devel expat-devel gettext-devel openssl-devel zlib-devel perl-ExtUtils-MakeMaker
wget https://git-core.googlecode.com/files/git-1.8.2.tar.gz
tar xzf git-1.8.2.tar.gz
cd git-1.8.2
make prefix=/usr/local all
sudo make prefix=/usr/local install

Initial clone:

git clone "bzr::lp:openpetraorg"

bzr checkout lp:openpetraorg
git clone "bzr::/home/timotheusp/bzr/openpetraorg"

Push to github:

git remote add origin2 git@[deleted].com:tpokorra/openpetragit.git
git push origin2 master

new changes from bzr to git:

cd ~/bzr/openpetraorg; bzr update
cd ~/openpetragit/openpetraorg; git pull origin
git push origin2 master
git push origin2 master --tags

pokorra

I just tried to see how the error messages are nowadays when trying to push a branch to the github repository, that is not compatible with the remote branch, eg. a branch that is not a descendant of the remote branch. This used to be a problem years ago when we moved from git to bzr.

Here is the output of the error that comes up now:

git.exe push --progress "origin" master
To [@github.com">git@[deleted].com](mailto:@github.com">git@[deleted].com):tpokorra/testpush.git
! [rejected] master -> master (non-fast-forward)
error: failed to push some refs to ['@github.com">git@[deleted].com](mailto:'@github.com">git@[deleted].com):tpokorra/testpush.git'
hint: Updates were rejected because the tip of your current branch is behind
hint: its remote counterpart. Merge the remote changes (e.g. 'git pull')
hint: before pushing again.
hint: See the 'Note about fast-forwards' in 'git push --help' for details.

This tells me that it is much harder to overwrite trunk by accident, or similar mistakes!

pokorra

I have now installed a cronjob, which will regularly (several times per hour) check for updates on lp:openpetraorg and push the changes to github.com/tpokorra/openpetragit.

We are using this now to build the job ci-win.openpetra.org:8080/job/Op ... WinSQLite/, which succeeds now!

I have also reported a bug with the bzr plugin of Jenkins: issues.jenkins-ci.org/browse/JENKINS-17971
which is causing trouble in the latest version, with trying to run bzr switch on a http checkout, which is not possible.
EDIT: bzr 2.4.x does not do the switch on http checkout, but bzr 2.5.1 works fine...

Here are my scripts for the bazaar to git bridge (assuming you have installed bazaar, git and the remote-bzr script (github.com/felipec/git/blob/fc/ ... ote-bzr.py), like described in my post from 12 Apr 2013 05:16 above.

initial clone from existing openpetragit:

cd ~/bzr
bzr checkout lp:openpetraorg
cd ~/git
git clone git://github.com/tpokorra/openpetragit.git openpetragit
cd openpetragit
git remote add originBzr "bzr::/home/timotheusp/bzr/openpetraorg"
git fetch originBzr
git merge remotes/originBzr/master -m "merge from lp:openpetraorg"
git push origin master --tags

cron/updateOpenPetraGit.sh runs eg. every 15 minutes:

eval `ssh-agent`
ssh-add ~/.ssh/id_rsa_cronjob
export PATH=/usr/local/bin:$PATH
cd ~/bzr/openpetraorg
bzr update
cd ~/git/openpetragit
git fetch originBzr
git merge remotes/originBzr/master -m "merge from lp:openpetraorg"
git push origin master --tags
kill $SSH_AGENT_PID

pokorra

I just had to fix a problem because the script ran fine when executed manually on the console, but failed when run in a cronjob.

error message is about a file in our repo with Umlaut in the name:

Tree is up to date at revision 2365 of branch http://bazaar.launchpad.net/~openpetracore/openpetraorg/trunkhosted
Traceback (most recent call last):
  File "/usr/bin/git-remote-bzr", line 939, in <module>
    sys.exit(main(sys.argv))
  File "/usr/bin/git-remote-bzr", line 911, in main
    repo = get_repo(url, alias)
  File "/usr/bin/git-remote-bzr", line 856, in get_repo
    for name, url in branch_list:
  File "/usr/bin/git-remote-bzr", line 784, in find_branches
    for fn in transport.iter_files_recursive():
  File "/usr/lib64/python2.6/site-packages/bzrlib/transport/local.py", line 303, in iter_files_recursive
    st = self.stat(relpath)
  File "/usr/lib64/python2.6/site-packages/bzrlib/transport/local.py", line 492, in stat
    return os.lstat(path)
UnicodeEncodeError: 'ascii' codec can't encode character u'\xfc' in position 77: ordinal not in range(128)

solution: fix line 492:
bazaar.launchpad.net/_bzr-pqm/bz ... al.py#L492
to be:

return os.lstat(path.encode('utf-8'))

now the synchronisation between lp:openpetraorg and github.com/tpokorra/openpetragit/ should work fine again!

christiankatict

Thanks, Timo, for finding the issue and for fixing it!

Just a question:
Would it be possible to reduce the period in which the script checks whether it needs to do a mirror to git to e.g. 5 minutes so (down from 15 mintues) - or would that put too high a load too regularly on the server where you run the script?

It would be neat if that could be done as then I would consider running all jobs on Windows Jenkins off the git mirror rather than off the Bazaar repository as the checkouts from git are way faster than from Bazaar on Windows Jenkins...

Thanks,

ChristianK

pokorra

Hello Christian,
I have changed it to a 5 minute interval.
Let me know if find any issues.

Timotheus

pokorra

I have now reviewed the instructions and posted an updated version of this thread on my blog:
pokorra.de/2013/10/mirror-a- ... to-github/
Hopefully this will make it into the Google Index, because the Forum does not get indexed somehow.

I also created an organisation account at github, and we can use now the URL github.com/tpokorra/openpetragit/).

I also changed the mirroring so that we don't have a separate "merge" commit in the history.

I will still post changes to the mirroring here, but will update the instructions in my blog post.