Kernal Problems

[btwatts]

I have been having a problem with my server since I had it installed. Every couple days the server stops responding to everything except ping. The logs say the Kernal is having problems.

I'm not certain how to decipher the log for this one. I'm using an Athlon - with Athlon kernal.

My suspicion is that this is a MEMORY (a.k.a. hardware) problem.

What next? This really REALLY needs to be fixed (soon).


Sep 27 05:07:51 server1 kernel: Unable to handle kernel paging request at virtual address 84f82170
Sep 27 05:07:51 server1 kernel: printing eip:
Sep 27 05:07:51 server1 kernel: c012ec9d
Sep 27 05:07:51 server1 kernel: *pde = 00000000
Sep 27 05:07:51 server1 kernel: Oops: 0000
Sep 27 05:07:51 server1 kernel: 8139too mii ipt_limit iptable_filter ip_tables ext3 jbd
Sep 27 05:07:51 server1 kernel: CPU: 0
Sep 27 05:07:51 server1 kernel: EIP: 0010:[<c012ec9d>] Not tainted
Sep 27 05:07:51 server1 kernel: EFLAGS: 00010807
Sep 27 05:07:51 server1 kernel:
Sep 27 05:07:51 server1 kernel: EIP is at kmem_cache_alloc [kernel] 0x7d (2.4.20-20.7)
Sep 27 05:07:51 server1 kernel: eax: 6b750006 ebx: c369212c ecx: d7242140 edx: 00008641
Sep 27 05:07:51 server1 kernel: esi: 00000246 edi: ea000c00 ebp: 000001f0 esp: ee30bca8
Sep 27 05:07:51 server1 kernel: ds: 0018 es: 0018 ss: 0018
Sep 27 05:07:51 server1 kernel: Process updatedb (pid: 21674, stackpage=ee30b000)
Sep 27 05:07:51 server1 kernel: Stack: 00003294 eb0b78a8 c473f700 00000004 00000000 00000000 f7f43208 c3604000
Sep 27 05:07:51 server1 kernel: c014e060 c369212c 000001f0 00000000 f7f43208 f7f43208 c3604000 c014f185
Sep 27 05:07:51 server1 kernel: c3604000 00000001 000001f0 00000000 03024cd0 00000000 f7f43208 000c0228
Sep 27 05:07:51 server1 kernel: Call Trace: [<c014e060>] alloc_inode [kernel] 0x30 (0xee30bcc8))
Sep 27 05:07:51 server1 kernel: [<c014f185>] get_new_inode [kernel] 0x15 (0xee30bce4))
Sep 27 05:07:51 server1 kernel: [<c014f450>] iget4 [kernel] 0xc0 (0xee30bd0c))
Sep 27 05:07:51 server1 kernel: [<f881dc37>] ext3_getblk [ext3] 0xc7 (0xee30bd30))
Sep 27 05:07:51 server1 kernel: [<f8820ae7>] ext3_lookup [ext3] 0x57 (0xee30bd44))
Sep 27 05:07:51 server1 kernel: [<c014503d>] real_lookup [kernel] 0x4d (0xee30bd84))
Sep 27 05:07:51 server1 kernel: [<c0145828>] link_path_walk [kernel] 0x678 (0xee30bda0))
Sep 27 05:07:51 server1 kernel: [<c0134a05>] __alloc_pages [kernel] 0x75 (0xee30bdf0))
Sep 27 05:07:51 server1 kernel: [<c01390c6>] __pte_chain_free [kernel] 0x16 (0xee30be0c))
Sep 27 05:07:51 server1 kernel: [<c0126d22>] do_anonymous_page [kernel] 0x1f2 (0xee30be18))
Sep 27 05:07:51 server1 kernel: [<c014968a>] filldir64 [kernel] 0xfa (0xee30be30))
Sep 27 05:07:51 server1 kernel: [<c0126d64>] do_no_page [kernel] 0x34 (0xee30be50))
Sep 27 05:07:51 server1 kernel: [<f881b9da>] ext3_readdir [ext3] 0x30a (0xee30be78))
Sep 27 05:07:51 server1 kernel: [<f881baa9>] ext3_readdir [ext3] 0x3d9 (0xee30be90))
Sep 27 05:07:51 server1 kernel: [<c012703a>] handle_mm_fault [kernel] 0xca (0xee30beac))
Sep 27 05:07:51 server1 kernel: [<c0146256>] open_namei [kernel] 0x2e6 (0xee30bed8))
Sep 27 05:07:51 server1 kernel: [<c0128601>] __insert_vm_struct [kernel] 0x51 (0xee30bef4))
Sep 27 05:07:51 server1 kernel: [<c0144d8d>] getname [kernel] 0x5d (0xee30bf0c))
Sep 27 05:07:51 server1 kernel: [<c0145c2b>] path_lookup [kernel] 0x1b (0xee30bf20))
Sep 27 05:07:51 server1 kernel: [<c0145e74>] __user_walk [kernel] 0x24 (0xee30bf30))
Sep 27 05:07:51 server1 kernel: [<c0142477>] vfs_lstat [kernel] 0x17 (0xee30bf44))
Sep 27 05:07:51 server1 kernel: [<c0142a10>] sys_lstat64 [kernel] 0x10 (0xee30bf70))
Sep 27 05:07:51 server1 kernel: [<c01140c0>] do_page_fault [kernel] 0x0 (0xee30bfb0))
Sep 27 05:07:51 server1 kernel: [<c0108904>] error_code [kernel] 0x34 (0xee30bfb8))
Sep 27 05:07:51 server1 kernel: [<c0108813>] system_call [kernel] 0x33 (0xee30bfc0))
Sep 27 05:07:51 server1 kernel:
Sep 27 05:07:51 server1 kernel:
Sep 27 05:07:51 server1 kernel: Code: 8b 44 81 18 89 41 14 03 79 0c 40 75 23 8b 41 04 8b 11 89 42

.
.
.
There are several more of these, but limits prevent the complete post here.
.
.
.
Sep 27 05:22:14 server1 kernel: <1>Unable to handle kernel NULL pointer dereference at virtual address 00000013
Sep 27 05:22:14 server1 kernel: printing eip:
Sep 27 05:22:14 server1 kernel: c014d7b4
Sep 27 05:22:14 server1 kernel: *pde = 00000000
Sep 27 05:22:14 server1 kernel: Oops: 0002
Sep 27 05:22:14 server1 kernel: 8139too mii ipt_limit iptable_filter ip_tables ext3 jbd
Sep 27 05:22:14 server1 kernel: CPU: 0
Sep 27 05:22:14 server1 kernel: EIP: 0010:[<c014d7b4>] Not tainted
Sep 27 05:22:14 server1 kernel: EFLAGS: 00010282
Sep 27 05:22:14 server1 kernel:
Sep 27 05:22:14 server1 kernel: EIP is at d_instantiate [kernel] 0x24 (2.4.20-20.7)
Sep 27 05:22:14 server1 kernel: eax: 0000000f ebx: c1242d80 ecx: ec0ff5f0 edx: c1242d90
Sep 27 05:22:14 server1 kernel: esi: ec0ff5c0 edi: e67afec0 ebp: c1242e94 esp: e67afe7c
Sep 27 05:22:14 server1 kernel: ds: 0018 es: 0018 ss: 0018
Sep 27 05:22:14 server1 kernel: Process proftpd (pid: 5420, stackpage=e67af000)
Sep 27 05:22:14 server1 kernel: Stack: ec0ff5c0 f778e140 c01cb795 ec0ff5c0 c1242d80 ffffffff c012ec9d 00000246
Sep 27 05:22:14 server1 kernel: f6e98040 3139345b 36343936 c01e005d c36927ec f53aa4c0 00000802 f790ab94
Sep 27 05:22:14 server1 kernel: bfffe798 e67afea0 00000009 004b06d2 c36437dc 00000001 f790ab94 e67afefc
Sep 27 05:22:14 server1 kernel: Call Trace: [<c01cb795>] sock_map_file [kernel] 0x95 (0xe67afe84))
Sep 27 05:22:14 server1 kernel: [<c012ec9d>] kmem_cache_alloc [kernel] 0x7d (0xe67afe94))
Sep 27 05:22:14 server1 kernel: [<c01e005d>] netlink_read_proc [kernel] 0x7d (0xe67afea8))
Sep 27 05:22:14 server1 kernel: [<c01cb815>] sock_map_fd [kernel] 0x15 (0xe67afee0))
Sep 27 05:22:14 server1 kernel: [<c01cc523>] sys_accept [kernel] 0xb3 (0xe67afeec))
Sep 27 05:22:14 server1 kernel: [<c0149814>] poll_freewait [kernel] 0x44 (0xe67aff20))
Sep 27 05:22:14 server1 kernel: [<c0149ba9>] do_select [kernel] 0x219 (0xe67aff30))
Sep 27 05:22:14 server1 kernel: [<c014a058>] sys_select [kernel] 0x468 (0xe67aff70))
Sep 27 05:22:14 server1 kernel: [<c01ccecc>] sys_socketcall [kernel] 0xac (0xe67aff8c))
Sep 27 05:22:14 server1 kernel: [<c014892b>] sys_fcntl64 [kernel] 0x7b (0xe67affac))
Sep 27 05:22:14 server1 kernel: [<c0108813>] system_call [kernel] 0x33 (0xe67affc0))
Sep 27 05:22:14 server1 kernel:
Sep 27 05:22:14 server1 kernel:
Sep 27 05:22:14 server1 kernel: Code: 89 48 04 89 46 30 89 51 04 89 4b 10 89 5e 08 5b 5e c3 8d 76
Sep 27 05:22:17 server1 kernel: <1>Unable to handle kernel NULL pointer dereference at virtual address 00000070
Sep 27 05:22:17 server1 kernel: printing eip:
Sep 27 05:22:17 server1 kernel: c01391f5
Sep 27 05:22:17 server1 kernel: *pde = 00000000
Sep 27 05:22:17 server1 kernel: Oops: 0000
Sep 27 05:22:17 server1 kernel: 8139too mii ipt_limit iptable_filter ip_tables ext3 jbd
Sep 27 05:22:17 server1 kernel: CPU: 0
Sep 27 05:22:17 server1 kernel: EIP: 0010:[<c01391f5>] Not tainted
Sep 27 05:22:17 server1 kernel: EFLAGS: 00010246
Sep 27 05:22:17 server1 kernel:
Sep 27 05:22:17 server1 kernel: EIP is at page_referenced [kernel] 0x125 (2.4.20-20.7)
Sep 27 05:22:17 server1 kernel: eax: c1000030 ebx: 00000000 ecx: 00000000 edx: 00000000
Sep 27 05:22:17 server1 kernel: esi: c1242da0 edi: 00000000 ebp: c1242da0 esp: c1dfff7c
Sep 27 05:22:17 server1 kernel: ds: 0018 es: 0018 ss: 0018
Sep 27 05:22:17 server1 kernel: Process kscand (pid: 6, stackpage=c1dff000)
Sep 27 05:22:17 server1 kernel: Stack: 00000000 00000000 00000000 c1dfffac c1242dbc c1242da0 00000000 000001f4
Sep 27 05:22:17 server1 kernel: c01323ad c1dfffac 000001f4 c02dadbc 00000001 c1dfe000 c02dac48 00000000
Sep 27 05:22:17 server1 kernel: 000001f4 c0133f79 c02dac48 00000000 00000000 c1dfe000 00000001 00000000
Sep 27 05:22:17 server1 kernel: Call Trace: [<c01323ad>] scan_active_list [kernel] 0x5d (0xc1dfff9c))
Sep 27 05:22:17 server1 kernel: [<c0133f79>] kscand [kernel] 0x109 (0xc1dfffc0))
Sep 27 05:22:17 server1 kernel: [<c0105000>] stext [kernel] 0x0 (0xc1dfffe8))
Sep 27 05:22:17 server1 kernel: [<c01070d6>] arch_kernel_thread [kernel] 0x26 (0xc1dffff0))
Sep 27 05:22:17 server1 kernel: [<c0133e70>] kscand [kernel] 0x0 (0xc1dffff8))
Sep 27 05:22:17 server1 kernel:
Sep 27 05:22:17 server1 kernel:
Sep 27 05:22:17 server1 kernel: Code: 8b 41 70 39 41 5c 0f 83 63 01 00 00 ff 44 24 04 e9 5a 01 00

[knightfoo]

This probably is a memory issue. You should submit a ticket to have the memory tested .. we can either do a hardware memory test which is 100% thorough or an online memory test which is only about 50% thorough. The hardware memory test means at least 15 minutes of complete downtime while the online test just makes your server sluggish for about an hour. If you use the category ServerBeach Account -> Hardware Issue it will not cost you a troubleshooting ticket.

-knightfoo

[elderban]

I've been getting the same thing on my server...

I'm submitting a ticket as well.