2.5.33-mm3 dbench hang and 2.5.33 page allocation failures

2.5.33-mm3 dbench hang and 2.5.33 page allocation failures

[Steven Cole]

I booted 2.5.33-mm3 and ran dbench with increasing
numbers of clients: 1,2,3,4,6,8,10,12,16,etc. while
running vmstat -n 1 600 from another terminal.

After about 3 minutes, the output from vmstat stopped,
and the dbench 16 output stopped.  The machine would
respond to pings, but not to anything else. I had to 
hard-reset the box. Nothing interesting was saved in 
/var/log/messages. I have the output from vmstat if needed.

The test box is dual p3, 1GB, scsi, ext3 fs.
Kernels are SMP,_HIGHMEM4G, no PREEMPT, no HIGHPTE. 

Earlier this morning, I ran 2.5.33 and the dbench test and got many
page allocation failure messages before I terminated the test.

Steven

Sep  5 07:20:01 spc5 kernel: dbench: page allocation failure. order:0, mode:0x50
Sep  5 07:28:32 spc5 kernel: dbench: page allocation failure. order:0, mode:0x50
Sep  5 07:37:46 spc5 last message repeated 2 times
Sep  5 07:37:47 spc5 last message repeated 9 times
Sep  5 07:37:47 spc5 kernel: klogd: page allocation failure. order:0, mode:0x50
Sep  5 07:37:56 spc5 last message repeated 23 times
Sep  5 07:37:56 spc5 kernel: dbench: page allocation failure. order:0, mode:0x50
Sep  5 07:38:00 spc5 last message repeated 17 times
Sep  5 07:38:00 spc5 kernel: dbench: page allocation failure. order:0, mode:0x20
Sep  5 07:38:00 spc5 kernel: dbench: page allocation failure. order:0, mode:0x50
Sep  5 07:38:06 spc5 last message repeated 22 times
Sep  5 07:41:01 spc5 kernel: kjournald: page allocation failure. order:0, mode:0x0
Sep  5 07:43:23 spc5 kernel: kjournald: page allocation failure. order:0, mode:0x0
Sep  5 07:44:36 spc5 kernel: dbench: page allocation failure. order:0, mode:0x50
Sep  5 07:44:44 spc5 last message repeated 37 times
Sep  5 07:44:44 spc5 kernel: dbench: page allocation failure. order:0, mode:0x0
Sep  5 07:44:44 spc5 kernel: dbench: page allocation failure. order:0, mode:0x50
Sep  5 07:44:54 spc5 last message repeated 48 times
Sep  5 07:45:44 spc5 kernel: pdflush: page allocation failure. order:0, mode:0x0
Sep  5 07:49:13 spc5 kernel: dbench: page allocation failure. order:0, mode:0x50
Sep  5 07:50:03 spc5 last message repeated 2 times
Sep  5 07:50:12 spc5 last message repeated 39 times
Sep  5 07:50:12 spc5 kernel: kswapd: page allocation failure. order:0, mode:0x50
Sep  5 07:50:49 spc5 kernel: dbench: page allocation failure. order:0, mode:0x50
Sep  5 07:50:55 spc5 last message repeated 62 times
Sep  5 07:51:45 spc5 kernel: kjournald: page allocation failure. order:0, mode:0x0
Sep  5 07:54:40 spc5 kernel: dbench: page allocation failure. order:0, mode:0x50
Sep  5 07:54:48 spc5 last message repeated 35 times
Sep  5 07:54:48 spc5 kernel: age allocation failure. order:0, mode:0x50
Sep  5 07:54:48 spc5 kernel: dbench: page allocation failure. order:0, mode:0x50
Sep  5 07:55:18 spc5 last message repeated 627 times




--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: 2.5.33-mm3 dbench hang and 2.5.33 page allocation failures

[Andrew Morton]

Steven Cole wrote:
> 
> I booted 2.5.33-mm3 and ran dbench with increasing
> numbers of clients: 1,2,3,4,6,8,10,12,16,etc. while
> running vmstat -n 1 600 from another terminal.
> 
> After about 3 minutes, the output from vmstat stopped,
> and the dbench 16 output stopped.  The machine would
> respond to pings, but not to anything else. I had to
> hard-reset the box. Nothing interesting was saved in
> /var/log/messages. I have the output from vmstat if needed.

That sounds like a race-leading-to-deadlock.  Feeding the SYSRQ-T
output into ksymoops is about the only way you have of diagnosing that
I'm afraid.

> The test box is dual p3, 1GB, scsi, ext3 fs.
> Kernels are SMP,_HIGHMEM4G, no PREEMPT, no HIGHPTE.
> 
> Earlier this morning, I ran 2.5.33 and the dbench test and got many
> page allocation failure messages before I terminated the test.
> 
> Steven
> 
> Sep  5 07:20:01 spc5 kernel: dbench: page allocation failure. order:0, mode:0x50
> Sep  5 07:28:32 spc5 kernel: dbench: page allocation failure. order:0, mode:0x50

Presumably, this was when running a lot more than 16 clients?

It's just a warning, btw.  Allocation failures are expected for GFP_NOIO
allocations.  Increasingly so lately, actually.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: 2.5.33-mm3 dbench hang and 2.5.33 page allocation failures

[Steven Cole]

On Thu, 2002-09-05 at 12:03, Andrew Morton wrote:
> Steven Cole wrote:
> > 
> > I booted 2.5.33-mm3 and ran dbench with increasing
> > numbers of clients: 1,2,3,4,6,8,10,12,16,etc. while
> > running vmstat -n 1 600 from another terminal.
> > 
> > After about 3 minutes, the output from vmstat stopped,
> > and the dbench 16 output stopped.  The machine would
> > respond to pings, but not to anything else. I had to
> > hard-reset the box. Nothing interesting was saved in
> > /var/log/messages. I have the output from vmstat if needed.
> 
> That sounds like a race-leading-to-deadlock.  Feeding the SYSRQ-T
> output into ksymoops is about the only way you have of diagnosing that
> I'm afraid.

I have CONFIG_MAGIC_SYSRQ=y for 2.5.33-mm3, so I'll reboot and try to
get some useful information.

> 
> > The test box is dual p3, 1GB, scsi, ext3 fs.
> > Kernels are SMP,_HIGHMEM4G, no PREEMPT, no HIGHPTE.
> > 
> > Earlier this morning, I ran 2.5.33 and the dbench test and got many
> > page allocation failure messages before I terminated the test.
> > 
> > Steven
> > 
> > Sep  5 07:20:01 spc5 kernel: dbench: page allocation failure. order:0, mode:0x50
> > Sep  5 07:28:32 spc5 kernel: dbench: page allocation failure. order:0, mode:0x50
> 
> Presumably, this was when running a lot more than 16 clients?

Yep, about 52 clients when those messages started coming in hot and
heavy, but they started with fewer clients.  I can rerun the test if
that threshold is of interest.

> 
> It's just a warning, btw.  Allocation failures are expected for GFP_NOIO
> allocations.  Increasingly so lately, actually.

Ok, thanks.  But the total hang is something else. I'll try alt-sysrq-t
with 2.5.33-mm3 and see what can be determined.

Steven

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: 2.5.33-mm3 dbench hang and 2.5.33 page allocation failures

[Steven Cole]

On Thu, 2002-09-05 at 12:22, Steven Cole wrote:
> On Thu, 2002-09-05 at 12:03, Andrew Morton wrote:

> > That sounds like a race-leading-to-deadlock.  Feeding the SYSRQ-T
> > output into ksymoops is about the only way you have of diagnosing that
> > I'm afraid.
> 
> I have CONFIG_MAGIC_SYSRQ=y for 2.5.33-mm3, so I'll reboot and try to
> get some useful information.

It looks like I'll have to set up a serial console to capture anything
useful from sysrq-t.  I got 2.5.33-mm1 to hang at dbench 16.  The box
responds to sysrq commands, but nothing really happens.  I tested
sysrq-s before I stared dbench, and that worked OK.  But now even with
sysrq-e and sysrq-i, the system still reports dbench and pdflush with
sysrq-t.  And sysrq-s doesn't finish.  I managed to save the output of
sysrq-p by typing that into a file manually.  I'll have to wait until
the test box recovers from sysrq-b to feed that into ksymoops since I'm
using by test box as the build box now.

BTW, the note in Documentation/sysrq.txt about not needing to enable
/proc/sys/kernel/sysrq anymore appears to be incorrect.  I had to set
this to 1 as it was set to 0 on boot.

Steven


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: 2.5.33-mm3 dbench hang and 2.5.33 page allocation failures

[Andrew Morton]

Steven Cole wrote:
> 
> On Thu, 2002-09-05 at 12:22, Steven Cole wrote:
> > On Thu, 2002-09-05 at 12:03, Andrew Morton wrote:
> 
> > > That sounds like a race-leading-to-deadlock.  Feeding the SYSRQ-T
> > > output into ksymoops is about the only way you have of diagnosing that
> > > I'm afraid.
> >
> > I have CONFIG_MAGIC_SYSRQ=y for 2.5.33-mm3, so I'll reboot and try to
> > get some useful information.
> 
> It looks like I'll have to set up a serial console to capture anything
> useful from sysrq-t.  I got 2.5.33-mm1 to hang at dbench 16.  The box
> responds to sysrq commands, but nothing really happens.  I tested
> sysrq-s before I stared dbench, and that worked OK.  But now even with
> sysrq-e and sysrq-i, the system still reports dbench and pdflush with
> sysrq-t.  And sysrq-s doesn't finish.  I managed to save the output of
> sysrq-p by typing that into a file manually.  I'll have to wait until
> the test box recovers from sysrq-b to feed that into ksymoops since I'm
> using by test box as the build box now.

grr.  I run dbench all night, so any insight you can get into this
would be appreciated.  (I've had a few hangs, but they're due
to bust disk drivers, aic7xxx not handling IO errors correctly, etc)

> BTW, the note in Documentation/sysrq.txt about not needing to enable
> /proc/sys/kernel/sysrq anymore appears to be incorrect.  I had to set
> this to 1 as it was set to 0 on boot.

grep your initscripts.  Some distros turn it off by hand.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: 2.5.33-mm3 dbench hang and 2.5.33 page allocation failures

[Steven Cole]

On Thu, 2002-09-05 at 13:37, Andrew Morton wrote:

> 
> grr.  I run dbench all night, so any insight you can get into this
> would be appreciated.  (I've had a few hangs, but they're due
> to bust disk drivers, aic7xxx not handling IO errors correctly, etc)

Double grr.  I just ran 2.5.33-mm4 and got the hang at dbench 8.  And in
my haste, I forgot to enable sysrq after boot.
> 
> > BTW, the note in Documentation/sysrq.txt about not needing to enable
> > /proc/sys/kernel/sysrq anymore appears to be incorrect.  I had to set
> > this to 1 as it was set to 0 on boot.
> 
> grep your initscripts.  Some distros turn it off by hand.

[root@spc5 steven]# find /etc -name "*" | xargs grep sysrq
/etc/sysctl.conf:kernel.sysrq = 0

This is from RH 7.3.  Fixed.  Thanks.

For what it's worth, I ran the output from sysrq-p (2.5.33-mm3
yesterday) through ksymoops, and here is the result. I typed those
numbers in manually. I'll try to get time to set up a serial console
today.

Steven

[steven@spc5 linux-2.5.33-mm3]$ ksymoops -K -L -O -v vmlinux -m System.map <regdump.txt
ksymoops 2.4.4 on i686 2.4.18-3smp.  Options used
     -v vmlinux (specified)
     -K (specified)
     -L (specified)
     -O (specified)
     -m System.map (specified)

Pid: 1219, comm:        pdflush
EIP: 0060:[<c015a388>] CPU:1 EFLASHS: 00000202  Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EAX: eaf09f88 EBX: 00000000 ECX: 00000020 edx: 00000400
ESI: eaf09f88 EDI: 000065c2 EBP: eaf09fd0 DS: 0068 es: 0068
CR0: 8005003b CR2: 40262000 CR3: 1e5b5000 CR4: 00000690
Call Trace: [<c013bb1a>] [<c013b73b>] [<c013b7e0>] [<c013b7eb>] [<c013baa0>]
[<c01072284>] [<c0107289>]
Warning (Oops_read): Code line not seen, dumping what data is available

>>EIP; c015a388 <.text.lock.fs_writeback+47/cf>   <=====
Trace; c013bb1a <background_writeout+7a/c0>
Trace; c013b73b <__pdflush+12b/1d0>
Trace; c013b7e0 <pdflush+0/10>
Trace; c013b7eb <pdflush+b/10>
Trace; c013baa0 <background_writeout+0/c0>
Trace; 0000000c01072284 <END_OF_CODE+b40ce7870/????>
Trace; c0107289 <kernel_thread_helper+5/c>


1 warning issued.  Results may not be reliable.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: 2.5.33-mm3 dbench hang and 2.5.33 page allocation failures

[Daniel Phillips]

On Thursday 05 September 2002 19:23, Steven Cole wrote:
> I booted 2.5.33-mm3 and ran dbench with increasing
> numbers of clients: 1,2,3,4,6,8,10,12,16,etc. while
> running vmstat -n 1 600 from another terminal.
> 
> After about 3 minutes, the output from vmstat stopped,
> and the dbench 16 output stopped.  The machine would
> respond to pings, but not to anything else. I had to 
> hard-reset the box. Nothing interesting was saved in 
> /var/log/messages. I have the output from vmstat if needed.

That happened to me yesterday while hacking 2.4 and the reason was
failed oom detection.  Memory leak?

-- 
Daniel
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: 2.5.33-mm3 dbench hang and 2.5.33 page allocation failures

[Martin Josefsson]

[-- Attachment #1: Type: text/plain, Size: 1053 bytes --]

On Thu, 2002-09-05 at 21:15, Daniel Phillips wrote:
> On Thursday 05 September 2002 19:23, Steven Cole wrote:
> > I booted 2.5.33-mm3 and ran dbench with increasing
> > numbers of clients: 1,2,3,4,6,8,10,12,16,etc. while
> > running vmstat -n 1 600 from another terminal.
> > 
> > After about 3 minutes, the output from vmstat stopped,
> > and the dbench 16 output stopped.  The machine would
> > respond to pings, but not to anything else. I had to 
> > hard-reset the box. Nothing interesting was saved in 
> > /var/log/messages. I have the output from vmstat if needed.
> 
> That happened to me yesterday while hacking 2.4 and the reason was
> failed oom detection.  Memory leak?

I've seen this on 6 diffrent machines (master.kernel.org is one of
them). I have a fileserver here that hits this all the time, sometimes
as much as a few times a day.

If anyone has any ideas or patches for 2.4 I'll happily test them.

-- 
/Martin

Never argue with an idiot. They drag you down to their level, then beat
you with experience.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: 2.5.33-mm3 dbench hang and 2.5.33 page allocation failures

[Andrew Morton]

Martin Josefsson wrote:
> 
> On Thu, 2002-09-05 at 21:15, Daniel Phillips wrote:
> > On Thursday 05 September 2002 19:23, Steven Cole wrote:
> > > I booted 2.5.33-mm3 and ran dbench with increasing
> > > numbers of clients: 1,2,3,4,6,8,10,12,16,etc. while
> > > running vmstat -n 1 600 from another terminal.
> > >
> > > After about 3 minutes, the output from vmstat stopped,
> > > and the dbench 16 output stopped.  The machine would
> > > respond to pings, but not to anything else. I had to
> > > hard-reset the box. Nothing interesting was saved in
> > > /var/log/messages. I have the output from vmstat if needed.
> >
> > That happened to me yesterday while hacking 2.4 and the reason was
> > failed oom detection.  Memory leak?
> 
> I've seen this on 6 diffrent machines (master.kernel.org is one of
> them). I have a fileserver here that hits this all the time, sometimes
> as much as a few times a day.
> 

What have you seen?  I doubt if it's a memory leak - they tend to
be preceded by a very obvious swapstorm.

It seems that you have boxes which lock up, and we have no more info
than that.

If the machine remains pingable then yes, it may be a VM deadlock/livelock.
We'd need to know the kernel version, system description, and a SYSRQ-T
trace passed through ksymoops would be helpful.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: 2.5.33-mm3 dbench hang and 2.5.33 page allocation failures

[Martin Josefsson]

[-- Attachment #1: Type: text/plain, Size: 1555 bytes --]

On Fri, 2002-09-06 at 01:48, Andrew Morton wrote:

> > I've seen this on 6 diffrent machines (master.kernel.org is one of
> > them). I have a fileserver here that hits this all the time, sometimes
> > as much as a few times a day.
> > 
> 
> What have you seen?  I doubt if it's a memory leak - they tend to
> be preceded by a very obvious swapstorm.
> 
> It seems that you have boxes which lock up, and we have no more info
> than that.
> 
> If the machine remains pingable then yes, it may be a VM deadlock/livelock.
> We'd need to know the kernel version, system description, and a SYSRQ-T
> trace passed through ksymoops would be helpful.

They lock up, they respond to ping and accept connections. But nothing
responds in userspace. You can switch consoles and sysrq works but
otherwise they are unresponsive to keyboard input.
No kernelmessages or anything and no heavy swapout storms before lockup.

The fileserver is running 2.4.19-pre8-xfs (I've seen it with vanilla
2.4.19 aswell on other machines), Athlon 1600+, 256MB ram, 1+ TB disk
(IDE). I can't seem to find my traces right now but I know they included
references to empty_zero_page, I'll see if I can get a new trace next
time. Upgraded to 2.4.19-xfs with kdb yesterday, maybe that will give
some more info.

I've seen the same thing on my workstation (2.4.19 with ext3) a few
times, the last time it happened was when I switched tabs in galeon.

--
/Martin

Never argue with an idiot. They drag you down to their level, then beat
you with experience.

[-- Attachment #2: This is a digitally signed message part --]
[-- Type: application/pgp-signature, Size: 189 bytes --]

Re: 2.5.33-mm3 dbench hang and 2.5.33 page allocation failures

[Daniel Phillips]

On Friday 06 September 2002 01:38, Martin Josefsson wrote:
> On Thu, 2002-09-05 at 21:15, Daniel Phillips wrote:
> > On Thursday 05 September 2002 19:23, Steven Cole wrote:
> > > I booted 2.5.33-mm3 and ran dbench with increasing
> > > numbers of clients: 1,2,3,4,6,8,10,12,16,etc. while
> > > running vmstat -n 1 600 from another terminal.
> > > 
> > > After about 3 minutes, the output from vmstat stopped,
> > > and the dbench 16 output stopped.  The machine would
> > > respond to pings, but not to anything else. I had to 
> > > hard-reset the box. Nothing interesting was saved in 
> > > /var/log/messages. I have the output from vmstat if needed.
> > 
> > That happened to me yesterday while hacking 2.4 and the reason was
> > failed oom detection.  Memory leak?
> 
> I've seen this on 6 diffrent machines (master.kernel.org is one of
> them). I have a fileserver here that hits this all the time, sometimes
> as much as a few times a day.
> 
> If anyone has any ideas or patches for 2.4 I'll happily test them.

I'd like to be able to say "just enable kdb under kernel debugging
and we'll see where it's stuck next time it's stuck".  Unfortunately,
I'm going to have to replace that with "go get the patch from sgi
if there is a current one and figure out how to install it".

Or maybe, "let's mess around with Alt-SysRq and write down funny
numbers on a piece of paper, type them into a file and run ksymoops
on them."  Nope, sorry, life is too short for that.

-- 
Daniel
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/