Patching Kernel

[JonTrainer]

I've never patched the Debian kernel before, and I remember that Knightfoo had mentioned that we had custom kernels on these servers.

Mine seems to be running 2.4.21. I attempted the usual apt-get upgrade method of installing patches, but I don't seem to pickup this new kernel patch to fix the latest kernel security issues that are now fixed in 2.4.23.

What am I doing wrong? Do I need to patch this kernel differently since it isn't the normal "stable" kernel?

[BlackNet]

Hello,

Yes you can custom patch the debian kernel, or any linux kernel for that matter.

I would classify any of the 'stable' kernel releases ok for roll out, my general rule is 6 months in the field before i move to it, unless there's something really nasty. If you do roll your own kernel then use the config file in /boot as a base, this should help prevent any disaster. A rule that I use, i test everything on my local 'play' machine before it goes out the door.

I am urging everyone to not panic as this is a local only exploit. In due time the 'apt-get install kernel-image-2.4.23' will be available, I suspect sooner rather than later. In fact if it's not out by the end of the week I will be suprised. A few things to keep in mind a) this is a local only exploit so the user MUST have access on the box, i.e. ssh. b) if you have questionable users then suspend them untill you get a patch, c) make sure there's no automated tasks running that may hinder your security. and lasty d) read the foo-Hardening, where foo = the service(s) you use. By all means apt-get update && apt-get upgrade, be sure you have the security.debian.org in your sources.list file

Just remember, panic is what leads to chaos, good calm methodical order is what's needed, esp now.

Ed

[JonTrainer]

Thanks BlackNet.

I wasn't so much in a panic as I was just wondering why the apt-get upgrade packages weren't available.

The debian security mail list had instructions on installing them. Guess there is a longer delay between announcement and availablility than I thought.

[BlackNet]

Hello,

Ok so they have a new kernel image for it.

http://security.debian.org/pool/updates/main/k/kernel-image-2.4.18-1-i386/kernel-image-2.4.18-1-386_2.4.18-12_i386.deb

I'm not sure how the stock kernel will work, I suspect it will work with no problems but I would hate to shoot myself in the foot. I would recomend contacting knightfoo and ask him. Who knows he may have a .deb of 2.4.23 already, I think I would hold out for 2.4.23 rather than downgrade.

Ed

[knightfoo]

The kernel-image-2.4.18-1 package has been available for a while .. it kind of snuck in there with the -1 revision, I didn't even notice it till I went to look for it. Installing the stock Debian kernel package won't cause any problems. The only difference between stock Debian kernels and the kernel ServerBeach installs is the initrd (well, lack of an initrd). The necessary filesystems for booting are built into the kernel so an initrd isn't necessary.

-knightfoo

[fcgreg]

Originally posted by knightfoo
The kernel-image-2.4.18-1 package has been available for a while ... Installing the stock Debian kernel package won't cause any problems.
Does this mean you recommend the downgrade to this kernel package? I just did an "apt-get update" and checked the cache and I still don't see a stable 2.4.23 kernel image. 2.4.22 was the highest I saw, and of course the backported kernel images of 2.4.18xxx.

I am fairly new to Debian (but very experienced with other distros), and my question is this:

What is the recommended way to patch your Debian kernel on our SB machines? I'm looking for actual apt-get/similar commands that I can run to confirm a patched kernel.

Finding information on this in the Debian world (debian.org, mailing lists, etc.) has been EXTREMELY difficult for me (and probably other Debian newbies).

Thanks for any help you all can offer me.

[JonTrainer]

I'd love to have a step by step on this as well -- or maybe a link to a HOWTO on this.

[BlackNet]

Hello,

I would urge you to take a 'test box' one that is local to you and play with the kernel untill you feel confident enough to install one remotely.

That said there's several good articles on the web:
http://myrddin.org/howto/debian-kernel-recompile.html

http://newbiedoc.sourceforge.net/system/kernel-pkg.html <=- lengthy and you do not need half the stuff they say.

http://anarka.org/linux/debian_kernel.html <=- very good

Ed

[JonTrainer]

Ed,

Beautiful! Thank you very much! These links are very helpful.

[fcgreg]

Thanks Ed. Definitely good links. My only concern is this:

Even with a test box locally, I can't adequately account for hardware differences between that and my Serverbeach machine. Therefore, short of a totally vanilla/stock kernel image, I'm a little uncomfortable trying to patch and recompile manually, even with those instructions.

Therefore a couple more questions for you folks:

1) Is the .config file I see in my /usr/src/kernel-headers-{kernel-version} the one that was used for my current build?

2) If so, I should be able to use it again for a fairly safe and compatible rebuild, yes?

3) Do any of you have any "best practices" for this process, particularly that may relate specifically to SB machines, etc.?

TIA.

[Blubb]

Yes some _reliable_ info from serverbeach about this matter would be nice.
Local root exploits are a serious security hole, even if you don't have local users (and actually many server have, for webmasters etc). So I want to patch my kernel, but I cannot afford any downtime because something doesn't works with the downgraded kernel and in the worst case I have to wait for the support to get my server back (since SB doesn't offers any rescue-interface afaik).

Besides, if everything works fine with the standart kernel, why does SB use a custom one at all? Just makes patching security bugs more time consuming.

[knightfoo]

The customer kernel is used simply to make the installation process easier for us. You can always install the official Debian kernel images for security updates. We don't use any off-the-wall hardware, so a stock kernel from kernel.org will also work without any problems. The /boot/config-<kernel_version> file will contain the kernel configuration which you can use to configure a stock kernel.

-knightfoo

[Blubb]

Well I tried it. used apt-get to install the a 2.4.18 kernel image and then removed the installed one. Apperently I didn't worked out, for the server didn't come back from its reboot.
Now I must admit a linux pro (tho not a complete newb too), so probably I forgot something. But I find it somewhat annoying that I have this trouble because it was easier for SB not to use the standartkernel. Anyway don't want to complain, I hope the support can fix it.

[knightfoo]

Originally posted by Blubb
But I find it somewhat annoying that I have this trouble because it was easier for SB not to use the standartkernel. Anyway don't want to complain, I hope the support can fix it.

I highly doubt that you had any issues due to the fact that we do not use a standard Debian kernel on new installations. Having a non-standard kernel installed does not change the kernel upgrade procedure, especially if you are upgrading to a standard Debian kernel. I just performed a kernel upgrade (2.4.18-1-k7) on a freshly installed Debian system and it worked fine

-knightfoo

[Blubb]

Well i have other servers other hosters too and there wasn't any problem here, with updating the (standard) kernel.

But anyway, no need to discuss this.

[knightfoo]

Let us know what the solution was. If there is a problem that is a result of our initial OS configuration then we will definitely want to fix it.

-knightfoo

[Blubb]

I think I rather forgot to update grub, because when updating the other server apt-get configured lilo for me. But I'll see.

[Blubb]

Now the support fixed it and the server is up and running again, which is good :)

However they reinstalled the custom kernel based on 2.4.23, which (in case Sb didn't modified it sources and applied a patch) has still a known security hole. Thats not so good.

On a sidenote, I guess nearly noone tries to update his kernel, meaning that most SB-Debian server are probably vulnerable to known exploits.

[knightfoo]

Well, since you deleted the old kernel (which worked), the DC guys had to install something to get the server booting. Most Debian users at SB have upgraded their kernel successfully (including myself). If you want someone from ServerBeach to do it for you, you can always submit a ticket.

-knightfoo

[Blubb]

Hmm if the support is doing such things that's might be a good idea. Even if it lets me look like a complete linux newb, its better than risking anohter downtime and worth the tickets :)

[Capo]

Originally posted by Blubb
However they reinstalled the custom kernel based on 2.4.23, which (in case Sb didn't modified it sources and applied a patch) has still a known security hole.

Did I miss something? I thought that 2.4.23 fixed the local root exploit?

[Blubb]

yes the security hole from dec, but there is a new one: http://www.debian.org/security/2004/dsa-413

[Capo]

Yup just found it....Yeeee Haaa :clue:

[Blubb]

Just to keep this updated:

I submitted two (for two servers) tickets for updateing the kernel, in order to avoid any downtimes.
The first one was closed right after created and got the status "resolved", it also couldnt be reopned. I guess it was some kind of error in the ticketsystem. But anyway:
The second ticket was waiting for a response since nearly 4 days.
So I figured out before waiting another week, I would try to update the kernel myself again, since other didn't seem to have problems to. The support didn'T told me what was wrong last time, but I figured I forgot to update grub.
So I made it again and tried to do everything right this time (also checked the mini howto in this forum). Now the same happend as last time and the server is unreachable - and I start to get a bit annoyed. I need to get a server which cannot be exploited by local users, so an update is really nesseary for me.
So now i'm going to open another ticket, to get my server back. However if the support will again only make the server reboot from a flawed kernel, not telling me what the reason was and therefor making it for me impossible to update the kernel myself, I can't keep my servers here.
I love serverbeach and it worked everything fine yet. I wouldn't even need ever support if there was a rescue system (like booting from network and allowing you to login with your standard PW). But it isn't and I need a secure server, so in this case I need support in some cases even if it costs more money.

[Blubb]

One last update: The next day the support bought the server back on Kernel 2.4.23 fo r the first. I finished fixing the error I made when installing 2.4.24 and asked the support to reboot the server (so in case new problems would raise, there would be no further downtime). Worked well and I did the same for my second server which the support also rebooted. None of them caused problems this time (of course, only happens when I do the reboot ;) ).
So both server I have here are now updated :)