This document is intended mainly for kernel developers, occasional or full-time, but sysadmins and power users may find parts of it useful as well. This in *not* intended to replace the Subversion documentation. Always run 'svn --help' or browse the manual at http://svnbook.red-bean.com for reference documentation. But I thought the kernel used BitKeeper not Subversion ? -------------------------------------------------------- Indeed, some kernel developers, including Linus, use BitKeeper to manage the kernel sources. The Linus BitKeeper repository (hosted on http://linux.bkbits.net) is the reference repository for the latest kernel code. However, BitKeeper is not free software. It is provided either free of charge under a restrictive license or under a classical commercial license. For more details, read the BitKeeper license. This document is intended for those who can't or don't want to use BitKeeper to follow the kernel development, but still want to use a SCM tool to: * keep up to date to Linus tree * easily consult change logs * do kernel related development and merge them with the latest Linus changes * etc... In this document, the SCM tool used is Subversion. How does it work ? ------------------ Thanks to Larry McVoy of BitMover, the kernel metadata (change logs, authors etc) are not only available in the BitKeeper tree, but are also made available through a special BK to CVS bridge. In the past, this bridge was accessible using a CVS pserver running on cvs.kernel.org, but due to several reasons this was changed and the only way to access the CVS repository today is by using rsync from: rsync://rsync.kernel.org/pub/scm/linux/kernel/bkcvs/ Ben Collins wrote a small tool which converts the kernel CVS repository into a SVN repository. The same script can be used in an incremental fashion afterwards to keep the two repositories synchronized. How can I setup a SVN repository ? ---------------------------------- Get the CVS repository: rsync -avz --delete \ rsync://rsync.kernel.org/pub/scm/linux/kernel/bkcvs/linux-2.5/ \ /repos/kernel/linux-2.6-bkcvs Create the SVN repository: svnadmin create /repos/kernel/linux-2.6-svn Convert the CVS repository to SVN: bkcvs2svn -s \ /repos/kernel/linux-2.6-svn \ /repos/kernel/linux-2.6-bkcvs The previous step will take a LONG time. On my AMD64 2000+ it took about 8 hours to complete. But you'll have to pay this price only once, subsequent re-syncs will be much quicker. You also need about 500 MB of disk space for the CVS repository and 900 MB for the SVN one (example for a kernel 2.6.11). In addition, each working copy of this repository will take 650 MB. If you're using Subversion 1.1, you can chose to use the 'fsfs' repository type instead of the default 'bdb' one, and you will gain a factor of 4 in performance for the repository creation. To do this, just create the repository with: svnadmin create --fs-type fsfs /repos/kernel/linux-2.6-svn What amount of kernel history will I have in my SVN repository ? ---------------------------------------------------------------- 'bkcvs2svn' will create one SVN revision per CVS 'ChangeSet,v' file revision. This file maps the BitKeeper changesets done on the trunk (done by Linus directly and not imported by him from some other developer BitKeeper tree). BitKeeper (and the exported CVS tree) can have both changeset revisions (changesets) *and* per file revisions (deltas). SVN does have only changesets, so all the per file revisions information (delta itself and the change comment) are lost when importing the tree. For the changes which are imported by Linus directly this is not a big problem, because there is (most of the time) a 1:1 mapping between the changeset revisions and the file revisions, and the per delta log messages are the same as the changeset log messages. A bigger problem are changes which happen on some other BitKeeper tree and which get merged by Linus. The CVS tree will record only the merge changeset, and all the branch changes will be placed inside this changeset. This means that you will have one SVN revision, which will contain several logical changes, and no simple way to differentiate them. I've modified bkcvs2svn to no longer ignore the file deltas but to use them in order to try splitting a changeset in several logical changes (based on the assumption that a logical change has the same log comment in all the deltas involved). This is obviously useful especially for the merge changeset. It won't help when the delta comments are not the same or when a file is modified several times on a branch. This is not a limitation of the tool but a limitation in the amount of (meta)data exported in CVS by BitMover. Since the conversion uses the file deltas log message, the changeset log message could be ignored. However, it could still contain some useful information, and bkcvs2svn+deltas generates an extra SVN revision containing no data change, only this log message. That means that for each BitKeeper changeset (as exported by bkcvs) you will have one or more SVN revisions followed by an extra SVN revision containing the log of the BitKeeper changeset. Each SVN revision log contains the text 'Logical revision 1.xxx' you can use if you want to regroup several revisions into what was originaly exported. The modified version is called bkcvs2svn+deltas and you can use it instead of the original to generate your SVN repository. Note that the augmented repository will take the double of disk space (1.4 GB) and time (5 hours - using fsfs storage) to generate. How do I access this repository ? --------------------------------- You can use the SVN repository just to track the Linus latest commits. The SVN repository is not easily readable, you will want to get a 'working copy' of it, by doing a 'svn checkout': svn checkout file://repos/kernel/linux-2.6-svn/trunk \ /home/user/kernel/linux-2.6-linus You will then have a full kernel source tree in '/home/user/kernel/linux-2.6-linus'. In this document the repository and the working copy are on the same machine, but this is not required. You can easily have the repository on a remote machine and access it using directly SVN (svnserve) or SSH (svn+ssh). Please refer to the SVN documentation for more details. How can I update the repository to the latest version ? ------------------------------------------------------- The repository can be updated on real-time to show Linus latest commits in his BitKeeper tree. The CVS tree exported by BitMover is updated nightly (could be more often in the future) and you can rerun 'bkcvs2svn' in an incremental fashion, to get only the new changes insert them into the SVN repository. You will then just redo the 'rsync' and 'bkcvs2svn' steps and you will obtain a full updated SVN repository. (the script 'bk2svn' does just that, and you can run it from 'cron' to periodically update your repository). Back into your working copy, you can get the updated code by issuing a 'svn update'. Where do I make my development ? -------------------------------- The Linus repository gets converted to SVN using the main SVN branch (called 'trunk'). You cannot do your own modification directly on the 'trunk' because Linus changes will overwrite your owns ('bkcvs2svn' does overwrite not merge the modifications it imports into the repository). So, if you are doing kernel related work, you will have to use a separate SVN branch. All your development will occur on this branch, you will regularly merge the trunk into the branch (to adapt to the latest Linus tree). Branch creation: svn mkdir file://repos/kernel/linux-2.6-svn/branches/ svn copy file://repos/kernel/linux-2.6-svn/trunk \ file://repos/kernel/linux-2.6-svn/branches/user ... note the revision you copy, referenced later by ... Branch checkout: svn checkout file://repos/kernel/linux-2.6-svn/branches/user \ /home/user/kernel/linux-2.6-user Do the development on the branch: ... hack in /home/user/kernel/linux-2.6-user ... svn update svn commit ... and repeat for each logical change ... Merge the updated changes: cd /home/user/kernel/linux-2.6-linus svn update ... note the revision here, referenced later by ... cd /home/user/kernel/linux-2.6-user svn merge -r : file://repos/kernel/linux-2.6-svn/trunk ... if there are conflicts, resolve them ... svn commit ... give the value of ... Note on merges: you will need to remember the revision numbers you used for a merge because on subsequent merges you'll need to start from the last revision merged ( in the example). A common way to do this is to record the revision range in the SVN commit log. Patch submission day. How do I do it ? -------------------------------------- When submitting patches to Linus (or to some other kernel maintainer), you will have to generate patches against the 'trunk'. Before starting, remember to always update the 'trunk' to the latest revision, in order to submit patches against the most recent code. You can find all the differences between the trunk and your branch using: svn diff file://repos/kernel/linux-2.6-svn/trunk \ file://repos/kernel/linux-2.6-svn/branches/user A cleaner way to submit the differences (which enables you to properly split the changes into small patches, document them etc), is to create a temporary branch just for the submission: svn copy file://repos/kernel/linux-2.6-svn/trunk \ file://repos/kernel/linux-2.6-svn/branches/to-linus Then, find the differences between your branch and the temporary one: svn diff file://repos/kernel/linux-2.6-svn/branches/to-linus \ file://repos/kernel/linux-2.6-svn/branches/user And for each logical change, copy it by hand from the 'user' branch to the 'to-linus' branch, and commit the change. Once you've finished, all you have to do is send each revision (beginning with the creation of the branch) in the 'to-linus' branch as a separate patch (the script 'svnsend' can help you presenting the patch is a clean format for submission to lkml. You can also use 'svnsendall' which calls 'svnsend' for each revision in a revision range). The temporary branch can be removed when it is no longer needed via a simple: svn remove file://repos/kernel/linux-2.6-svn/branches/to-linus How do I ignore temporary build files ? --------------------------------------- It is useful to ignore temporary build kernel files so they are not shown in 'svn diff' etc. The lines below show how to do this on a SVN kernel tree: cd /tmp wget http://www.moses.uklinux.net/patches/dontdiff cd /home/user/kernel/linux-2.6-user svn propset svn:ignore -R -F /tmp/dontdiff . svn commit -m "Added svn:ignore properties." How do I generate 'proper' diffs ? ---------------------------------- Subversion by default does generate patches to be applied with 'patch -p0', which is not recommended for the kernel. The patch format also doesn't use '-p' (tell the name of the function being modified) which is also recommended. In order to generate proper kernel patches you must use 'svn-diff' as a diff wrapper (get it into this directory) and instruct svn to use 'svn-diff' instead of its internal diff mechanism: svn diff --diff-cmd=/usr/local/bin/svn-diff ... If you want to use the 'svn-diff' wrapper automatically, you will have to edit your '$HOME/.subversion/config' file and put the following lines into it: [helpers] diff-cmd = /usr/local/bin/svn-diff Beware, this setting is in place for all your SVN repositories, not only the kernel one. What are the possible problems with using SVN for kernel development ? ---------------------------------------------------------------------- There are several possible problems you should be aware of when using Subversion for kernel related development. 1. Unfortunately, the BK->CVS conversion is not perfect and the granularity of changes is bigger than the original. The explanation is that BK branches and consequent merges into the mainline are not reproduced on the CVS side, so all changes done in a parallel trees will be shown as a single commit on the CVS side. This is why you'll sometimes find a single SVN revision for many BitKeeper changesets. 2. Subversion is based on a centralized, server based model, and this imposes limitations on the operations you can do while being disconnected from the server (no commits, no history accesses etc). Those interested in more disconnected operations can take a look at SVK (http://svk.elixus.org/) a distributed SCM written in Perl and based on SVN (but quite different in usage). SubMaster is another possible alternative (http://www.rocklinux.org/submaster.html). 3. Subversion can be rather slow on some operations. This is due to the fact that the kernel tree is rather large, and also to the fact that the operations are done on the server rather than doing them on the client. 4. The SVN repository could need complete regeneration, and this would require hand recreation of the development branches. The CVS repository is generated by a tool written by Larry McVoy of BitMover, and it happened in the past (and could happen again in the future) that bugs are discovered in this tool which requires regenerating the whole CVS repository, and the revision numbers could become incompatible with the old repository. This will require regenerating the SVN repository as well, and all the branches contents will be lost. There is unfortunately no simple way to moving the data you had in the old branches to the new repository, you'll have to do it manually (using 'svnsendall' for example). 5. Backuping the data is not simple. You cannot backup only the development branches, you will have to backup the whole SVN repository (using 'svnadmin dump' for example).