Kernel Traffic #335 For 27 Nov 2005

By Zack Brown

It's always worked for me, but hey, maybe I'm just lucky.

--Linus Torvalds

Table Of Contents

Mailing List Stats For This Week

We looked at 2243 posts in 14MB. See the Full Statistics.

There were 662 different contributors. 246 posted more than once. The average length of each message was 106 lines.

The top posters of the week were: The top subjects of the week were:
88 posts in 386KB by [email protected]
74 posts in 515KB by russell king
73 posts in 560KB by adrian bunk
68 posts in 384KB by [email protected]
51 posts in 492KB by greg kh
78 posts in 364KB for "new (now current development process)"
41 posts in 172KB for "3d video card recommendations"
38 posts in 178KB for "best way to handle leds"
34 posts in 160KB for "[patch]: clean up of __alloc_pages"
33 posts in 159KB for "ntp broken with 2.6.14"

These stats generated by mboxstats version 2.8

1. Describing And Debating The Development Process

29 Oct 2005 - 7 Nov 2005 (78 posts) Archive Link: "New (now current development process)"

Topics: Bug Tracking, Disks: SCSI, PCI

People: Paolo CiarrocchiJesper JuhlTony LuckRussell KingLinus TorvaldsChristopher FriesenWilly TarreauAndi KleenAlexander ViroAndrew MortonMartin J. BlighZwane MwaikamboRoland DreierTim BirdRoman ZippelEric SandallTheodore Y. Ts'oKrzysztof Halasa

Paolo Ciarrocchi said:

I would like to write a short article about the new development process discussed during last Linux Kernel Developers' Summit for my local LUG.

Since I'm not able to find an accurate report of what has been discussed during that meeting I try to summariza what is my understanding of the current process:

The are two kind of releases, 2.6.x kerneles and 2.6.x.y

2.6.x.y are maintained by the "stable" team (stable at kernel dot org), are released amost every week.

Here is the latest ( I hope ) announce I found in the archive including an explanation of how stable development works:

Rules on what kind of patches are accepted, and what ones are not, into the "-stable" tree:

Procedure for submitting patches to the -stable tree:

Review cycle:

Review committe:

2.6.x kernel are maintained by Linus Torvalds and Andrew Morton (both are from OSDL) and the development is as follow:

I'm sure I'm missing lot of details so I would really appreciate if you could support me in writing this article that hopefully will be usefull outside my LUG as well :-)

Jesper Juhl replied:

I recently wrote a document on the different kernel trees and how to apply patches for them. In that document I also give a description of the various trees.

Perhaps that document will be useful to you. You can find it in a recent kernel source tree as Documentation/applying-patches.txt or you can read it online via lrx at this URL: http://sosdg.org/~coywolf/lxr/source/Documentation/applying-patches.txt

Good luck on your article. Once you are done with it I'd love to read it and I guess a lot of other people would too, so please supply an URL to it at that time :-)

Elsewhere, Tony Luck replied to Paolo's initial post. Regarging the description of the -rc1 timeline, Tony said, "Initially Linus said that he would only accept e-mail patches after rc1 ... but common sense prevailed and he later clarified to say that git merges could still be used, but only to include bug fixes." He added, "Also note that a whole new driver (or filesystem) might be accepted after -rc1 because there is no risk of causing regressions with such a change." Tony also pointed out that the only release candidate with an official schedule was -rc1. Subsequent candidates happened when they happened. Tony also addressed Paolo's description of the three month schedule between official kernel releases. He said, "the goal was to make a release in around 8 weeks (which would be closer to six per year). But you have the important part, which is that Linus doesn't make the release until it is "ready". 2.6.13 was released on August 28th, and 2.6.14 on October 27th ... so we appear to have hit the goal this time through." Russell King replied:

However, three months is _far_ too long. Look at what is happening post-2.6.14? There's loads of really huge changes going in which were backed up over the last two months.

Continuing like this will just push the release of each kernel further and further out as there's more stuff merged during the mayhem than can possibly be properly reviewed and tested.

Shorter release cycles means that there's less mayhem, which in turn means more time to review.

And Linus Torvalds said:

Yes. Three months would be too much. I considered even 2.6.14 to be delayed by at least one week. So 8 weeks is certainly better than it used to be, but I think 6 weeks should be the goal-post.

But to hit that, we'd might need to shrink the "merge window" from two weeks to just one, otherwise there's not enough time to calm down. With most of the real development happening during the previous calm-down stage, that might not be impossible: we'd only have one week for merges, but that isn't the week when development actually happens, so who cares?

Everything said, I think 2.6.13->14 worked well enough, even if it's hard to say how well a process works after one release. Considering that 2.6.13 had the painful PCI changes (you may not have noticed too much, since they were x86 only) and there were some potentially painful SCSI changes in the .14 early merges, so it's not like 13->14 was an "easy" release - so the process certainly _seems_ to be workable.

So I'm planning on continuing with it unchanged for now. Two-week merge window until -rc1, and then another -rc kernel roughly every week until release. With the goal being 6 weeks, and 8 weeks being ok.

I don't think anybody has been really unhappy with this approach? Hmm?

There were a couple of subthreads emanating from this one. In one of them, several folks began discussing the relative merits of having an official kernel release be unchanged from the most recent -rc release. Eric Sandall argued that the whole point of an -rc release was to create a "release candidate", i.e. something that would become the official release if no further problems arose. Krzysztof Halasa argued that it was okay for the official release to be different from the most recent -rc release, so long as no new bugs were introduced. Christopher Friesen pointed out that it was difficult to know that no new bugs had been introduced. Christopher said, "The safe bet is to simply rename the final -rc with no further changes." But Linus replied:

That's not actually a very safe bet at all.

Why? Because only a small percentage of people actually test -rc kernels in the first place.

So if you release the last -rc as the standard kernel, you (a) get no real coverage advantage anyway and (b) you just wasted a week in order to try to get some coverage that you aren't getting.

The current kernel development model is to merge stuff early, which hopefully motivates the people who _do_ test -rc kernels to actually test -rc1, since they know that that's the one that has _all_ the really relevant goodies.

And most of those that do test -rc1 will never see any problems at all. Those that do, are now more likely to test the rest of the -rcs, hopefully until their problem is resolved. And those that don't test -rc releases (because they simply don't upgrade very often) will _never_ test an -rc release, whether it's the first one or the last one.

So what we have to fight this problem is the stable kernel series. We release the final 2.6.x kernel with as much testing as we can, but it's just an undeniable fact that a lot of people will try that kernel only after the release, and often it might be weeks after the release. Doing -rc kernels didn't do anything for those cases.

But when they find a problem (or somebody who _did_ test an -rc kernel, but didn't notice a problem until much later), we try to have a process to get those issues fixed.

So repeat after me: "Most people never test -rc kernels".

He continued in reply to himself:

Btw, the ones that _do_ test -rc kernels usually don't test all of them. The current model is set up in a way where there is _one_ special -rc kernel that we should try to get people to test: the first one.

That hopefully encourages people to try an -rc kernel who might otherwise decide that there's too many -rc kernels to bother with. If they know that all of the real development happened before -rc1, they also are thus aware that it doesn't really matter which -rc kernel they test, so just testing _one_ is very good indeed.

The first -rc kernel is also special in another way: it's the one we "wait" for. It's the one that happens after two weeks, and has a deadline. The others happen more frequently, and are really objectively less important than the first one.

(In contrast, some other projects try to make the _last_ -rc be the important one. That's totally the wrong way around, because if there are more people testing the last one, the testing happens at _exactly_ the wrong point in time from a "let's fix the problems" standpoint)

So the call to people who can be bothered to test at all: if you only test one -rc kernel, please test the first one. That way we get a heads-up on problems earlier.

(And if you like testing -rc kernels, please test all of them. Or even the nightly snapshots. Or track the git tree several times a day. The more, the merrier, but if you only want to boot one kernel a month, make it be the -rc1 kernel).

Willy Tarreau pointed out that at least for some developers, the situation was different from what Linus described. He said:

I *try* to test *ALL* versions which are announced as the most likely future -final. When Marcelo tells us that -rc2 will be -final if nobody complains, I really test it. I build it on several archs and try to catch stupid bugs because I hate stupid bugs in final releases, they pollute bug reports (worst ones being build errors).

However, when he announces -preX (X>1) or when you annonce any -rc which is not likely to become -final, I just check the changelog, and build it if I both see something which applies to my setup, and I have nothing else to do.

Elsewhere, Andi Kleen also replied to Linus's request for feedback on the current development approach. He said, "The long freeze periods were nothing much happens are painful. It would be better to have some more overlap of merging and stabilizing (stable does that already kind of, but not enough)" . Theodore Y. Ts'o replied that Andrew Morton accepted patches into the -mm tree during the freeze periods, and so it couldn't really be called a period where "nothing much happens". But Andi replied, "The problem is that -mm* contains typically so many more or less broken changes that any extensive work on there is futile because you never know whose bugs you're debugging (and if the patch that is broken will even make it anywhere). In short mainline is frozen too long and -mm* is too unstable." And Alexander Viro added, "Besides, -mm is changing so fscking fast that it doesn't build on a lot of configs most of the time. And trying to keep track of it and at least deal with build breakage at real time is, IME, hopeless." He added later that he routinely tried building -mm releases for a wide range of configurations, and found large numbers of problems. Andrew Morton pointed out that "Every -mm release is built with allmodconfig on x86 and on x86_64. It's also cross-compiled on fat configs for alpha, ppc32, ppc64, sparc64, arm and ia64. It's booted on x86, x86_64, ppc64 and ia64." He added, "I usually do allnoconfig and allyesconfig, too." Alexander said he tested with a much broader array of configurations. He listed some off, and said, "_IF_ somebody wants to do that for -mm, yell and you are more than welcome to all infrastructure, except for the cycles on build box I'm using. Incidentally, it is a box at work - my energy bill is high enough as it is, without adding an 8-way 3GHz iamd64 to it..." Martin J. Bligh replied, "I'll do that if you want. I have a big lab full of largish boxes with serious aircon, and IBM can afford the power bill. I'm assuming this is a farm of cross-compilers that'll run on x86 (or x86_64)?" But the thread ended or went private at that point.

Elsewhere, also in reply to Andi's initial criticism of the long freeze periods, Russell King agreed strongly, saying, "I find the long freeze periods painful and very very very boring, to the point of looking for other stuff to do (such as cleaning up bits of the kernel and queuing mega-patches for the next round of merging.)" Andrew replied:

The freezes are for fixing bugs, especially recent regressions. There's no shortage of them, you know.

I you can think of a better way to get kernel developers off their butts and actually fixing bugs, I'm all ears.

He added later:

a) you're sitting around feeling very very very bored while

b) the kernel is in long freeze due to the lack of kernel developer attention to known bugs

The solution seems fairly obvious to me?

Russell said, "That's fine if you have the hardware to be able to debug these issues." And Andrew replied:

Most driver bugs cannot be reproduced by the developer. Almost by definition: if the patch had caused problems on the developer's machine, he wouldn't have shipped it.

This is why we have this wonderful group of long-suffering people who download and test development kernels for us, and who take the time to report defects.

Yes, it's painful to get into a long-range debugging session, sending debug patches, twelve-hour turnaround, etc. But what alternative have we?

Close by, Andi said:

The problem is that you usually cannot do proper bug fixing because the release might be just around the corner, so you typically chose the ugly workaround or revert, or just reject changes for bugs that a are too risky or the impact too low because there is not enough time to properly test anymore.

It might work better if we were told when the releases would actually happen and you don't need to fear that this not quite tested everywhere bugfix you're about to submit might make it into the gold kernel, breaking the world for some subset of users.

Andrew replied:

Nobody knows when a kernel will be released, because it's released according to perceived bug status, not according to a preconceived timeline.

I just don't buy what you're saying. Unless the kernel is at -rc3 or -rc4, we *know* the release is weeks away - it's always been thus.

He added, "There is absolutely nothing preventing people from working on bugs at any time! It's not as if a bugfix can ever come too early."

Close by, Russell backed Andi up, saying:

Indeed - a prime example is the bootmem initialisation problem. Had I known on 1st October that the final release wouldn't have been for a long time, I'd have applied that patch and you wouldn't have had that problem with ARM relying on the bootmem initialisation order clashing with your ia64 (or x86_64) swiotlb fix.

But stupid rmk - again I hasten to add - was suckered into this "-rc is frozen" idea again and decided it wasn't appropriate to submit it. In hind-sight, there would have been plenty of time to sort out any issues.

Hell, we held up 2.6.14 for swiotlb _anyway_.

Andrew said:

I'd say that was an *exception*, not a "prime example".

I've filed away 322 unresolved bug reports here. The great majority are busted drivers on random hardware dating back as far as 2.6.11. Many of them are regressions.

There is nothing stopping anyone from working with the originators to get these things fixed up at any time.

Why is it necessary for me to chase maintainers to get their bugs fixed?

Why are maintainers working on new features when they have unresolved bugs?

Why is it so often me who has to do the followup for un-responded-to bug reports against subsystems which I don't know anything about?

(Those are rhetorical questions, btw).

Andi said:

Because zero bugs is just unrealistic and they would never get anything done if that was the requirement?

(especially considering that a lot of the bugs at least on x86-64 are hardware/firmware bugs of some sort, so often it is not really a linux bug but just a missing ha^w^wwork^w^w^w^wfix for something else)

I agree regressions are a problem and need to be addressed, but handling all non regressions on a non trivial platforms is just impossible IMHO...

Perhaps it would be nice to have better bug classification: e.g. regression/new hardware/reported by more than one person etc. I think with some prioritization like that it would be much easier to keep the bugs under control.

Sometimes bugs are less important than others. e.g. on x86-64 the bootmem issue was obscure at best, affecting only a very small part of the user base. Sure the few people hit by it will be annoyed, but trying to make everyone happy is impossible so one has to try to just make the majority of users happy. So it was imho ok to just revert the patch to fix ARM again and not breaking IA64 (I cannot assess how many users were on ARM/IA64 affected) for the next release and try to sort it out later.

Regressions are important, everything else has to be prioritized based on the impact on the user base (and this doesn't necessarily mean the most noisy part of the userbase)

Perhaps some people could volunteer to set some flags in bugzilla for obvious things, like regression or new hardware or missing basic information or for really old kernel and no report for a new one and that could be used to filter the queries better. Should be an relatively easy task.

Zwane Mwaikambo remarked, "I don't think we need any special flags, we just need more people paying attention to it. It doesn't take that much time to go through pending bugs and trying to identify real ones, but the problem is that we need knowledgeable people to do this." And Andrew added, "I don't believe that what we're seeing is some prioritisation process. The problem is that some bugs are *harder* than others, and people are just ducking things which cannot be locally reproduced."

At this point, the discussion veered sharply off, when Roman Zippel pointed out that the compiled kernels were rapidly growing in size. He listed a bunch of kernels, starting with 2.2.18 at 495K, and going up to 2.6.14 at over three times that size.

Andrew asked, "Are you sure these kernels are feature-equivalent?" And Linus replied:

They may not be feature-equivalent in reality, but it's hard to generate something that has the features (or lack there-of) of old kernels these days. Which is problematic.

But some of it is likely also compilers. gcc does insane padding in many cases these days.

And a lot of it is us just being bloated. Argh.

A few posts later, he added:

I really don't see the point of shaving less than a kB with ugly calling convention magic, when switching to -Os can save us much more, and when the networking code is several hundred kB.

If we start doing size optimizations, we need to think big.

Roland Dreier said, "it would be great to find ways to make big improvements. But I think the most realistic way to shrink the kernel is the same way it grows in the first place -- one small piece at a time." Linus replied:

No, I think that's a lost cause.

It doesn't grow by 700 bytes once in a while. It grows by much more, and much more often. And we can't fight it that way, that's just not going to work. Maybe have something that tracks individual object file sizes and shames people into not growing them..

Tim Bird replied:

As mentioned at the kernel summit, we're working on it at the CE Linux Forum. These things take time to set up, but we have code already for something that tests the size impact of individual kernel configs, and we're working on a test to track individual function sizes for each new kernel (using Andi Kleen's bloat-o-meter).

We're still in process of setting up the test lab, but we have a number of hardware boards already, and we're hoping to be publishing size regression data for the kernel on a regular basis, starting in about April of next year.

Andi asked, "Any particular reason it's taking so long? Would it perhaps be possible to do a subset (e.g. only a single architecture and less configs) sooner? If the problem is writing the necessary scripts, I'm sure some people (e.g. M. Bligh or Al Viro) have already written some that do at least the automated building and would be happy to share them." And Tim said:

The main reason it's taking so long is that it's a background task for everyone working on it. (Basically, me and one guy at NEC. ;-) We've gotten sidetracked with lab setup issues (signing a lease for office space, processing paperwork for board donations, etc. etc.) But I can run some tests now on a few different platforms just on my desk.

I had hoped to have at least some preliminary results by the end of November, with a bigger call for help inside the forum to finish up something big and professional-looking in April. However, maybe I can cobble together some things, and let people (both in and out of the forum) look at the preliminary results sooner.

I'll see what I can do by the end of November.

2. Linux 2.4.32-rc2 Released

31 Oct 2005 - 4 Nov 2005 (15 posts) Archive Link: "Linux 2.4.32-rc2"

Topics: Networking, SMP, USB

People: Marcelo TosattiDan AloniPete ZaitcevJulian AnastasovRalf BaechleNick PigginAlexey KuznetsovAndrew MortonWilly Tarreau

Marcelo Tosatti announced Linux 2.4.32-rc2, saying:

Here goes the second release candidate for v2.4.32.

The most significant changes are v2.6 backports of IPv4/IPv6 bugfixes, and a USB OHCI regression introduced during v2.4.28 which could lead to crashes on SMP kernels.

He posted the changelog:

Aleksey Gorelov:
asus vt8235 router buggy bios workaround

Alexey Kuznetsov:
[TCP]: Don't over-clamp window in tcp_clamp_window()

Andrew Morton:
loadkeys requires root priviledges

Dan Aloni:
fix memory leak in sd_mod.o

Denis Lukianov:
[MCAST]: Fix MCAST_EXCLUDE line dupes

Herbert Xu:
Clear stale pred_flags when snd_wnd change

Horms:
[IPVS]: Add netdev and me as maintainer contacts
Fix infinite loop in udp_v6_get_port()

Julian Anastasov:
[IPVS]: ip_vs_ftp breaks connections using persistence
[IPVS]: really invalidate persistent templates

Marcelo Tosatti:
Change VERSION to 2.4.32-rc2

Marcus Sundberg:
[NETFILTER]: this patch fixes a compilation issue with gcc 3.4.3.

Nick Piggin:
possible memory ordering bug in page reclaim

Pete Zaitcev:
usb: regression in usb-ohci

Ralf Baechle:
AX.25: signed char bug

Willy Tarreau:
Fix jiffies overflow in delay.h

Later, he added:

There is nothing else pending for v2.4.32 on my part.

Will wait a couple of days and tag it as v2.4.32.

3. Sharp Zaurus-5500 Testers Wanted

2 Nov 2005 - 4 Nov 2005 (7 posts) Archive Link: "sharp zaurus-5500: looking for testers"

Topics: Touchscreen

People: Pavel MachekCarlos Martin

Pavel Machek said:

Is there someone out there, with sharp zaurus sl-5500, willing to test kernels? There's a linux-z tree on kernel.org, which I try to more or less keep in sync with mainline, that is slowly starting to get usable. It could use some testing.

Main drawback is that battery charging is not yet done; touchscreen is there but I did not have chance to test it with proper userspace filtering.

Keenan Pepper leapt through his monitor to hug Pavel, and said he'd begin testing as soon as he could. Carlos Martin also grabbed Pavel's patches, but reported, "your git tree (repository?) in kernel.org is a bit broken. The git web interface gives me 403 error when I try to see a diff in your zaurus.git tree, and there's stuff that appears to be missing (history and commits)." Pavel worked on fixing this, and a couple days later reported that the repository should be working again. Carlos confirmed this, and said he'd report on his tests as soon as they were finished.

4. New eCryptFS Filesystem

2 Nov 2005 - 7 Nov 2005 (42 posts) Archive Link: "[PATCH 0/12: eCryptfs] eCryptfs version 0.1"

Topics: Ottawa Linux Symposium, Version Control

People: Phillip HellewellDavid HowellsErez Zadok

Phillip Hellewell said:

This set of patches constitutes eCryptfs version 0.1. We are presenting it to be reviewed and considered for inclusion into the kernel.

eCryptfs is a stackable filesystem that is based off of the Cryptfs that is generated by the FiST stackable filesystem framework written by Erez Zadok:

http://filesystems.org/

eCryptfs stores cryptographic metadata in the headers of each file; the headers contain OpenPGP-like packets (see RFC 2440). This allows the encrypted underlying files to be copied between hosts, and all of the information necessary to decrypt the files stays with the files themselves. eCryptfs aims to make the encryption and the decryption of each individual file completely transparent to userspace applications, so long as the recipient has the requisite key or passphrase to access the file available.

Michael Halcrow presented eCryptfs at the 2004 and the 2005 Ottawa Linux Symposiums; the high-level overview from this year's symposium starts on page 209 of the first half of the symposium proceedings:

http://www.linuxsymposium.org/2005/linuxsymposium_procv1.pdf

Note that this set of patches contains a considerably trimmed-down version of eCryptfs than what was sent to the LKML earlier this year. Release 0.1 includes mount-wide passphrase support only; this will make eCryptfs easier to analyze and debug before the more advanced policy and public key features are merged in.

eCryptfs performs well under a variety of tests, including FSX and Connectathon (Basic and General functional). There is a bug that crops up on a kernel compile. We would appreciate any insight that the VFS guru's could give us in tracking down and fixing any extant bugs.

eCryptfs utilizes David Howells' keyring; at mount, eCryptfs version 0.1 expects an existing authentication token in the user's session keyring. The tarball containing the code to do this is available from the eCryptfs SourceForge site (ecryptfs-v0_1.tar.bz2):

http://sourceforge.net/projects/ecryptfs/

Future releases will have policy support, which will entail per-file passphrase and per-file public key support. Those who are interested in looking at that code are welcome to obtain it from the eCryptfs CVS repository on SourceForge:

cvs -d:pserver:[email protected]:/cvsroot/ecryptfs login
cvs -z3 -d:pserver:[email protected]:/cvsroot/ecryptfs co -P ecryptfs

5. git Binary Safe

3 Nov 2005 - 5 Nov 2005 (9 posts) Archive Link: "binary safe?"

Topics: Version Control

People: Randal L. SchwartzLinus TorvaldsNick Hengeveld

On the git mailing list, Randal L. Schwartz said, "I'm currently about to abandon CVS for my website management, replacing it with git." He asked if git was binary-safe, in the sense that it could be used to house non-text data. Linus Torvalds replied:

Yes. I don't think it's been heavily tested, but the very architecture of git should mean that there just shouldn't be any issues with binary files outside of the obvious ones.

The only binary file the kernel ever uses is the logo.gif thing, so it's been "tested" in the sense that binary files exist, but there's never been any changes to that file, so..

Nick Hengeveld also remarked, "We're now using git in production to distribute content, of which well over 90% is binary files. Works great."

6. git 0.99.9c Released; Querying The git Network

3 Nov 2005 - 5 Nov 2005 (5 posts) Archive Link: "GIT 0.99.9c"

People: Junio C. HamanoJeff GarzikJunio C HamanoAlex Riesen

Junio C. Hamano announced git 0.99.9c, saying:

GIT 0.99.9c is found at http://kernel.org/pub/software/scm/git/ as usual. It includes the following fixes and documentation updates since 0.99.9b:

Alex Riesen:
remove CR/LF from .gitignore

Jon Loeliger
Illustration: "Fundamental Git Index Operations"
Illustration: "Git Diff Types"
Illustration: "Commit DAG Revision Naming"

Junio C Hamano:
Do not put automatic merge message after signed-off-by line.
git-clone: do not forget to create origin branch.
Make test-date buildable again.
Do not fail on hierarchical branch names.
Ignore '\r' at the end of line in $GIT_DIR/config
Be careful when dereferencing tags (credits Pasky).
Document --since and --until options to rev-parse.
Add --no-commit to git-merge/git-pull.
Add 'ours' merge strategy.
git-merge-ours: make sure our index matches HEAD
GIT 0.99.9c

Peter Eriksen:
Clean up the SunOS Makefile rule

The slow and steady march toward 1.0 continues.

I plan to do another full sweep in the documentation directory on my next GIT day.

On the proposed updates front, I am hoping to include Nick's http-push using DAV in the "master" branch soon. And I would appreciate somebody who actually uses svnimport to Ack on Yaacov's svnimport fix.

Jeff Garzik replied:

Here's an experiment I've been dying to try.

The current "tracker-less" BitTorrent (http://www.bittorrent.com/trackerless.html) employs a distributed hash table (http://www.etse.urv.es/~cpairot/dhts.html) called Kademlia, where the total content is spread across a bunch of computers on the network. I kinda prefer TANGLE (http://www.nicemice.net/amc/research/tangle/) to Kademlia.

Anyway, I was thinking that it would be a neat experiment to add simple TANGLE-like peer-to-peer code, to enable git to query "the git network hash table" for content.

Comments, or any pre-code-creation objections? How easy is it to add a new storage backend to git?

To restrict unlimited uploading, I'm thinking that I'll want the system to fall back to {www,git,rsync}.kernel.org as the original source of content. [though the code will obviously be generic, and not hardcode *.kernel.org policy]

Junio addressed the issue of adding a new storage back-end, saying:

Almost everything is contained within sha1_file.c.

Object creation side is simple -- everybody who creates an object (e.g update-index registering blobs, write-tree writing the toplevel and intermediate level trees, commit-tree building a commit object, unpack-objects exploding a pack) goes through write_sha1_file(), which checks if the object is already available using has_sha1_file() and creates a new object in the local .git/objects/?? directory. I am assuming that you are not planning to create objects in a remote peer from within the git code path, and instead to have background process that replicate them over the network to peer repositories, so you probably do not have to touch this side.

Extending inspection and reading from existing objects for your networked storage may be somewhat messy, but starting points are:

The above three are the primary read interfaces, but there are some places that cheat by assuming that packs and individual objects are the only two kinds of sources for the object data, so you need to be careful. For example, write_sha1_to_fd(), which is used only by ssh-upload, first tries to call map_sha1_file_internal(), which is only valid for individual objects, to grab the object data, and when it fails, calls read_packed_sha1(), which is only valid for objects in packs, without even checking if read_packed_sha1() succeeded. This doesn't crash only because the caller of write_sha1_to_fd() checks if the object is available by calling has_sha1_file() itself before calling this function, but you would need to change it to fall back on your networked storage if you do not want to crash when used by ssh-upload.

HTH, and have fun.

Jeff thanked him, adding, "It looks pretty straightforward to add a new storage backend, grepping on the symbols you listed."

7. Fixing Incorrect Authors And Committers In git Repositories

4 Nov 2005 (3 posts) Archive Link: "Fixing Commit And Author"

Topics: Compression

People: Darrin ThompsonLinus Torvalds

On the git mailing list, Darrin Thompson said:

I've got a small project in git where I made a dumb error. All my commits have author/committer information like this:

Author: Darrin Thompson <[email protected](none)> 2005-10-20 16:50:38
Committer: Darrin Thompson <[email protected](none)> 2005-10-20c16:50:38
Tags: svn-5099

I'd like to replace the commits (yes, I know that means all of them) with new ones with corrected email addresses and also manage to migrate my tags. A push in the right direction would be appreciated.

Next I'd like to do the same with the kernel sources... :-)

Linus Torvalds replied:

There's a program in the git sources called "git-convert-objects.c".

It basically knows how to walk the git object chains, and rewrite each object according to a few rules.

The rules currently do _not_ include changing the author/committer info, but it does know how to parse the really old-style dates, for example, which are on those same lines, so adding some code there to also re-write the author/committer name and email wouldn't be impossible.

The code isn't necessarily all that easy to understand, and usage-wise you also have to convert each head separately (you tell it which branch head you want to convert, it trawls every reachable object from that head, and will create the new objects and return the new head value).

What I'm trying to say is that it might not be _pleasant_, but it's certainly something you can automate and do in a timely manner (ie a small project will take just a few seconds - or minutes - to convert).

As far as doing this for the Linux kernel, Linus added:

The same program will work, but it will take some time.

Actually, as long as you only rewrite commits, it should even be reasonably efficient. It's when you start rewriting every single object (like I did when I switched the compression scheme around) that it gets _really_ expensive, and a project like the kernel would take a long long time.

Hint to the wise: don't do the conversion on the only copy of the repository you have. It's always worked for me, but hey, maybe I'm just lucky and never write buggy conversion software.

8. Ubuntu Kernel Maintained In git

5 Nov 2005 - 9 Nov 2005 (18 posts) Archive Link: "[ANNOUNCE] Ubuntu kernel tree"

People: Ben CollinsPavel MachekGreg KH

Ben Collins said:

Some people may have noticed the new git tree located at:

rsync.kernel.org:/pub/scm/linux/kernel/git/bcollins/ubuntu-2.6.git

This tree will directly reflect the Ubuntu Linux Kernel that is available in our distribution (along with build system). First use of this kernel tree is slated for Dapper Drake (Ubuntu 6.04), and will stay synced with the just released 2.6.14(.y).

There are several reasons for making this repo available on kernel.org. Primary reasons include a more open development model, better visibility with the kernel developer community, and to make the kernel available to other distro's who may want to base their kernel off of ours.

Primary goals include:

Pavel Machek remarked, " We were thinking about using git for internal suse kernel trees, but we thought it would not work, as we need more something like quilt. Do you really use git internally, or do you just export to it? If git is usable for distro develompent... that would be good news." Ben replied, "Prior to this we were using baz (arch derivative), but all the patches were still applied at build time." He went on, "We're giving git a go just to see. It's all being done right there, I push directly to master.kernel.org (and also mirror it to a local machine where ubuntu devs can make better use of it)." Greg KH remarked, "good luck with this, it will be interesting to see how long you can handle maintaining the kernel in this manner."

9. git 0.99.9e Released

6 Nov 2005 - 7 Nov 2005 (5 posts) Archive Link: "GIT 0.99.9e"

People: Junio C. HamanoJunio C HamanoJohannesJosef Weidendorfer

Junio C. Hamano announced git 0.99.9e, saying:

GIT 0.99.9e maintenance release is found at the usual places:

RPM, tarballs, and deb:

http://www.kernel.org/pub/software/scm/git/

With git, fetch maint branch from

git://git.kernel.org/pub/scm/git/git.git/

It contains everything from the master branch. Since we seem to be shelving the separate git binary directory idea indefinitely, what we have here is pretty much what will be in 1.0, from the source code POV.

He posted the changelog:

Changes since 0.99.9d are:

Johannes Schindelin:
Allow GIT_DIR to be an absolute path
http-fetch: do not use curl_message after releasing it

Jon Loeliger:
Refactor merge strategies into separate includable file.

Junio C Hamano:
test: t4102-apply-rename fails with strict umask (Peter Baumann).
git-format-patch: silly typo fix.
Documentation: pull/clone ref mapping clarification (Josef Weidendorfer).
git-fetch: fail if specified refspec does not match remote.
Simplify CFLAGS/DEFINES in Makefile
Package split: Debian.
Install asciidoc sources as well.
Further Debian split fixes.
Debian: test build.
Merge in http-push first stage.
Document expat dependency when using http-push.
ls-files: --others should not say unmerged paths are unknown.
git-status: do not mark unmerged paths as committable.
Set up remotes/origin to track all remote branches.

Nick Hengeveld:
Add support for pushing to a remote repository using HTTP/DAV
Verify remote packs, speed up pending request queue
Support remote references with slashes in their names
Improve lock handling
Refresh the remote lock if it is about to expire

Paul Collins:
http-push.c: include with angle bracket, not dq.

Randal L. Schwartz:
Use fink/darwinport paths for OSX

10. New Trivial Patch Monkey Maintainer

7 Nov 2005 (3 posts) Archive Link: "[2.6 patch] GIT trivial tree"

Topics: Version Control

People: Adrian BunkLinus Torvalds

Adrian Bunk said:

Linus, please do an update from:

http://www.kernel.org/pub/scm/linux/kernel/git/bunk/trivial.git

I've taken over the trivial patch monkey from Rusty, and I'll send the really trivial patches like spelling corrections through this tree.

Is this tree OK or are there any problems with it?

Linus Torvalds replied:

Please don't try to make me update over http. Either point to master (which is not accessible by all), or point to git://git.kernel.org/. Or do both..

And if you do the latter (or, in fact, rsync or http for others), please make sure that you delay your "please pull" sufficiently that the contents have actually mirrored out, because otherwise, if the mail comes in while I'm in merging mode (like right now), and I try to pull, I may not have anything to pull at all just because it hasn't mirrored out yet.

A side comment (this was true with BK too): I prefer not to see unnecessary two-way merges, since that just makes the history much messier.

11. Another Attempt To Remove DevFS

7 Nov 2005 (1 post) Archive Link: "[GIT PATCH] Remove devfs from 2.6.14 - try 2"

Topics: FS: devfs

People: Greg KH

Greg KH said:

Here are the same "delete devfs" patches that I submitted for 2.6.12 and 2.6.13. It rips out all of devfs from the kernel and ends up saving a lot of space. Since 2.6.13 came out, I have seen no complaints about the fact that devfs was not able to be enabled anymore, and in fact, a lot of different subsystems have already been deleting devfs support for a while now, with apparently no complaints (due to the lack of users.)

Please pull from:
rsync://rsync.kernel.org/pub/scm/linux/kernel/git/gregkh/devfs-2.6.git/
or if master.kernel.org hasn't synced up yet:
master.kernel.org:/pub/scm/linux/kernel/git/gregkh/devfs-2.6.git/ (http://www.master.kernel.org:/pub/scm/linux/kernel/git/gregkh/devfs-2.6.git/)

I've posted all of these patches before, but if people really want to look at them, they can be found
http://www.kernel.org/pub/linux/kernel/people/gregkh/gregkh-2.6/gregkh-05-devfs/

Also, if people _really_ are in love with the idea of an in-kernel devfs, I have posted a patch that does this in about 300 lines of code, called ndevfs. It is available in the archives if anyone wants to use that instead (it is quite easy to maintain that patch outside of the kernel tree, due to it only needing 3 hooks into the main kernel tree.)

12. Secure Digital Host Controller Support

8 Nov 2005 (1 post) Archive Link: "[ANNOUNCE] Driver project for Secure Digital Host Controller Interface"

People: Pierre Ossman

Pierre Ossman said, "I've started working on a driver for the Secure Digital Host Controller specification. This seems to be used by the majority of the controllers found in todays PCs, so this driver would add support for a lot of chips. Information is scarce so I do not know if this driver will get production ready, but for those of you willing to live dangerously: http://mmc.drzeus.cx/wiki/Linux/Drivers/sdhci"

13. Linux 2.6.14.1 Released

4 Nov 2005 - 9 Nov 2005 (19 posts) Archive Link: "Linux 2.6.14.1"

People: Greg KHAlexander Viro

Greg KH announced Linux 2.6.14.1, saying:

We (the -stable team) are announcing the release of the 2.6.14.1 kernel.

The diffstat and short summary of the fixes are below.

I'll also be replying to this message with a copy of the patch between 2.6.14 and 2.6.14.1, as it is small enough to do so.

The updated 2.6.14.y git tree can be found at:
rsync://rsync.kernel.org/pub/scm/linux/kernel/git/gregkh/linux-2.6.14.y.git
and can be browsed at the normal kernel.org git web browser:
www.kernel.org/git/ (http://www.kernel.org/git/)

He posted the changelog, which had only a single entry, a fix from Alexander Viro for a "CVE-2005-2709 sysctl unregistration oops" .

14. Linux 2.4.32-rc3 Released

9 Nov 2005 - 10 Nov 2005 (6 posts) Archive Link: "Linux 2.4.32-rc3"

Topics: Ioctls

People: Marcelo Tosatti

Marcelo Tosatti announced Linux 2.4.32-rc3, saying:

Two small issues showed up,

So here goes -rc3.

Attaching the full changelog since this is probably the last -rc release.

 

 

 

 

 

 

Sharon And Joy
 

Kernel Traffic is grateful to be developed on a computer donated by Professor Greg Benson and Professor Allan Cruse in the Department of Computer Science at the University of San Francisco. This is the same department that invented FlashMob Computing. Kernel Traffic is hosted by the generous folks at kernel.org. All pages on this site are copyright their original authors, and distributed under the terms of the GNU General Public License version 2.0.