If anyone knows Linux kernel driver development, it’s Greg Kroah-Hartman, who’s been working deep in Linux for over a decade. In this interview, Greg talks about how the Linux project has accommodated the accelerating rate of change for the kernel, and offers some insight on where Linux is headed.
Scott Swigart: Can you please introduce yourself and your association with Linux?
Greg Kroah-Hartman: Sure. I’m a Novell fellow, and I’m working on the Linux kernel with the SUSE Labs division at Novell.
I’ve been doing Linux kernel work for more than 10 years now. I’m a maintainer of several different subsystems, including USB, the driver core, and some other minor things. I’m also responsible for releasing the stable Linux kernels. I used to be a maintainer of a lot of other driver subsystems and such over the years, but I’ve thankfully handed them off to other people.
Scott: Let me start with a really big-picture question, since you’ve been involved in Linux for a long time. Obviously, there’s still lots of stuff happening on the mailing list, but fundamentally, the way code makes it into the kernel hasn’t really changed, and yet the scale of development is much larger than it was, say, five years ago.
How have you seen Linux kernel development change to handle the scale of the effort that’s going on today?
Greg: I’ve been tracking this for over five years now, and we’ve gotten a lot better. There are a lot more people doing the work.
To give an example, for the 2.5 to 2.6 kernel development series, which took about two years, the top 30 people did 80 percent of the work. Now, the top 30 people do 30 percent of the work. The sheer number of developers has also increased. We were running a couple hundred developers, and now we’re running a couple thousand.
Scott: How does that get managed? Are there more layers in the hierarchy?
Greg: We’ve invented some tools to help us out a lot. It used to be that Linus was the only one who could commit anything, or who would accept anything. It was all done through email.
And then BitKeeper was developed, and a number of us started using that. It was very nice, because we had a much faster feedback loop with Linus. I could have him pull 50 patches, see that he pulled them, and then I could hit him with another 50 patches, if I wanted to, within a day. Before, we had a lag of a couple of weeks.
Then, when we couldn’t use BitKeeper anymore, Linus wrote Git, and then that just increased it even more, because everybody can use Git. Our development model increased its capacity, and part of that was that we started trusting more people. Now we have subsystem maintainers and such.
As I said, I maintain the subsystems such as USB, and I have people who I trust enough that if they send me a patch, I’ll take it, no questions asked. Because the most important thing is I know that they will still be around in case there’s a problem with it. [laughs]
And then I send stuff off to Linus. So, Linus trusts 10 to 15 people, and I trust 10 to 15 people. And I’m one of the subsystem maintainers. So, it’s a big, giant web of trust helping this go on.
We’ve developed some procedures that help guide how and when we do merges and releases. And we’ve a very regular release schedule: every three months, we do a release. We have a two-week merge window, and all those patches in that merge window have to have been tested. We’ve gotten the development process down really well over the past four years.
We’re also increasing the rate of change in our development. The same amount of work one of the top 10 developers did last year wouldn’t have even made it into the top 20 this year. Our individual developers have got the work flow down, so we can actually contribute more, to an extent that’s amazing.
Scott: I understand that one of the things you’re involved with is the Linux Driver Project. It seems like there was a certain tension between Linux and certain hardware vendors, in the sense that, for a long time, hardware vendors had been really used to not shipping driver code.
They were used to shipping for Windows, and they didn’t have to be involved with drivers. They just provided an installer. And then, as Linux started to become a lot more popular in the server space and make some inroads in the desktop space, they wanted access to that market.
There was kind of a culture shift in a lot of companies to realize how to provide things like drivers in a way that matched with what the Linux community expected. Take the example of some hypothetical Acme hardware vendor that wants to submit a driver but has never done it before. What does the process look like?
Greg: First, people need to realize that our driver model is different than other operating systems, because all our drivers ship with the kernel. The license requires our drivers to be open, so everything is in the main kernel tree.
Because of our huge rate of change, they pretty much have to be in the kernel tree. Otherwise, keeping a driver outside the kernel is technically a very difficult thing to do, because our internal kernel APIs change very, very rapidly.
In other operating systems, APIs change more slowly, since their development process is comparatively very slow. So, it makes sense, from a technical standpoint and just to save money, to get your code into the kernel. It is maintained that way, and when changes happen within the kernel, your driver code will be fixed for you. If I have to change an API, I change it everywhere.
That approach also lets us see commonality. If see that a driver’s doing the same thing as another driver, I might merge them together in the shared core. That way, everybody benefits. The new driver gets smaller and easier to maintain.
That’s happened a lot. We’ve merged a lot of drivers from a lot of different companies, and everybody’s happy about that. The companies are happy, the users are happy with the smaller code base, and it’s easier to maintain.
The Linux Driver Project started out because of the perception that Linux doesn’t support many devices. It turns out that Linux supports all devices out there. There’s really nothing manufactured today that Linux doesn’t support, in a major consumer market. There are some one-offs, like some small video-capture devices I know of that we don’t have support for, but people are actually working on those.
The initial goal of the Linux Driver Project was to remove all barriers that could possibly be there, which were mostly managerial. To attain that goal, we said that we will write and maintain any driver for free for any company.
It turned out that not very many companies really needed that. A few niche markets needed it, and we’re writing some drivers for them, but most companies had existing drivers floating around inside their company. Their challenge was typically that they needed to get it into the kernel tree, and they didn’t know how.
So, the Linux Driver Project, over the past couple years, has just been a big educational thing. I work with the companies and show them how to get the code in. I maintain the code, massage it and get it cleaned up, and then merge it into the kernel tree. That’s been the majority of the work we’ve been doing over the past two years.
We’ve also now codified a few ways that we can play in the kernel. We can accept code into the kernel that doesn’t meet our normal standards into a staging area. Everybody can clean it up there, and then the code graduates into the real stuff. That’s worked out really well.
Scott: That makes good sense, because someone coming in from the outside isn’t necessarily going to know everything from coding standards to the best practices for doing things the right way.
And so what you’re saying is that you’ve got a place to get that code in, where people who know the right way to do things can look for things that can be factored out of drivers, drivers that can be merged, and that kind of stuff.
It doesn’t force every hardware vendor or everybody who needs to get a driver into the kernel to be a Linux driver development expert. Do I have that right?
Greg: Yeah, although we do have lots of documentation now. We have free books on how to write Linux drivers, and we have documentation of things like our coding style and how to submit code. If people don’t know where it is, we’ll point them in the right direction. In case anyone’s curious, it all starts in a file called HOWTO in the kernel, so they can start there.
Even though we do have those education tools, though, we will also work with companies that want it. We have lots of people who have done this work before and who will do it for you.
It’s a great place for people wanting to get involved in the kernel to start out, because they can just run a script to find some errors and go fix them. We have people that have started off just wanting to get involved who have ended up maintaining whole drivers and subsystems over time.
Still, in the end, my goal is actually to work with engineers from those companies and have them maintain and own it, so they become full-fledged members of the kernel community. That’s the only way we’re going to grow, and it’s working. The number of companies involved in the kernel has grown year over year.
Companies want to get the most value out of Linux, so I counsel them that they should drive the development of their driver and of Linux as a whole in the direction that they think makes the most sense. If they rely on Linux and feel that Linux is going to be part of their business, I think they should become involved so they can help change it for the better.
Scott: That makes sense. In a different area, I saw somewhere that around half of the Linux code that’s being contributed nowadays is driver code. Do I have that right?
Greg: That’s the percentage. Like I said, everything’s in the kernel itself. We’re at something like six or seven million lines of code, and over 50 percent of those lines of code are drivers.
I think 30 percent is architecture-specific stuff, for things like processors and networking. The core kernel is like five percent of the overall code. Those numbers have stayed pretty much the same over the past four or five years.
We change something like 5,000 lines a day, which is just scary. Fifty percent of that change will be in the drivers, and five percent will be in the core kernel. In other words, the kernel is being modified everywhere at that rate of change.
Scott: Do I understand correctly that the core kernel is memory management, process scheduling, and those kinds of fundamental things that an OS has to do?
Greg: Exactly. Basic system calls, memory management, scheduling, and inter-process communications.
And the only reason we’re changing is because people want that change. It’s not like we’re sitting there and