2.5.33-mm4 filemap_copy_from_user: Unexpected page fault

2.5.33-mm4 filemap_copy_from_user: Unexpected page fault

[Steven Cole]

With 2.5.33-mm4, I tried running dbench on an ext2 partition and was
able to run up to dbench 80 successfully.  However, at dbench 96, I got
four messages like this:

filemap_copy_from_user: Unexpected page fault

Shortly after this, the box hung again, responsive to pings but little
else.  I did sysrq-p and typed in the results, which are slightly
different than before using an ext3 partition. Sysrq-e and sysrq-i had
no effect, so I had to sysrq-b. The following fsck on the ext2 disk was
not fun. It may be worth noting that this hang occurred at 96 clients
with dbench on ext2 and at 8 clients on ext3 (data=ordered).

Here is the output of ksymoops on the sysrq-p result:

Steven

[steven@spc5 linux-2.5.33-mm4]$ ksymoops -K -L -O -v vmlinux -m System.map <mm4-sysrq-p.txt
ksymoops 2.4.4 on i686 2.5.33.  Options used
     -v vmlinux (specified)
     -K (specified)
     -L (specified)
     -O (specified)
     -m System.map (specified)

Pid: 1945, comm:        pdflush
EIP: 068:[<c0159bf4>] CPU: 1 EFLAGS: 00000202   Not tainted
Using defaults from ksymoops -t elf32-i386 -a i386
EAX: 00000001 EBX: c90f3730 ECX: c90f3738 EDX: eb32bf88
ESI: c90f3730 EDI: 000065c2 EBP: 00000001 DS: 0068 ES: 0068
CR0: 8005003b CR2: 4212db0c CR3: 2a378000 CR4: 00000690
Call Trace: [<c0159e1e>] [<c013b8aa>] [<c013b4cb>] [<c013b570>] [<c013b57b>]
   [<c013b830>] [<c0107284>] [<c0107289>]
Warning (Oops_read): Code line not seen, dumping what data is available

>>EIP; c0159bf4 <sync_sb_inodes+84/260>   <=====
Trace; c0159e1e <writeback_inodes+4e/80>
Trace; c013b8aa <background_writeout+7a/c0>
Trace; c013b4cb <__pdflush+12b/1d0>
Trace; c013b570 <pdflush+0/10>
Trace; c013b57b <pdflush+b/10>
Trace; c013b830 <background_writeout+0/c0>
Trace; c0107284 <kernel_thread_helper+0/c>
Trace; c0107289 <kernel_thread_helper+5/c>


1 warning issued.  Results may not be reliable.


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: 2.5.33-mm4 filemap_copy_from_user: Unexpected page fault

[Andrew Morton]

Steven Cole wrote:
> 
> With 2.5.33-mm4, I tried running dbench on an ext2 partition and was
> able to run up to dbench 80 successfully.  However, at dbench 96, I got
> four messages like this:
> 
> filemap_copy_from_user: Unexpected page fault

Yep.  This means that the page we faulted in by-hand in generic_file_write()
wasn't resident during the subsequent copy_from_user().

That fault-in by-hand is there to prevent a deadlock.  That printk
meand that it isnt working all the time.   We have (always had) a
problem.

> Shortly after this, the box hung again,
> ...
> >>EIP; c0159bf4 <sync_sb_inodes+84/260>   <=====
> Trace; c0159e1e <writeback_inodes+4e/80>
> Trace; c013b8aa <background_writeout+7a/c0>
> Trace; c013b4cb <__pdflush+12b/1d0>

Hum.  Thanks for that.

I've been dbenching and compiling all night.  And yet, it
seems that the dirty inode search in sync_sb_inodes() can trivially
lock up.

Does this fix?

 fs-writeback.c |    4 ++--
 1 files changed, 2 insertions(+), 2 deletions(-)

--- 2.5.33/fs/fs-writeback.c~scole	Fri Sep  6 09:50:25 2002
+++ 2.5.33-akpm/fs/fs-writeback.c	Fri Sep  6 09:50:47 2002
@@ -242,16 +242,16 @@ sync_sb_inodes(struct super_block *sb, s
 
 		if (wbc->nonblocking && bdi_write_congested(bdi)) {
 			wbc->encountered_congestion = 1;
+			list_move(&inode->i_list, &sb->s_dirty);
 			if (sb != blockdev_superblock)
 				break;		/* Skip the entire fs */
-			list_move(&inode->i_list, &sb->s_dirty);
 			continue;
 		}
 
 		if (wbc->bdi && bdi != wbc->bdi) {
+			list_move(&inode->i_list, &sb->s_dirty);
 			if (sb != blockdev_superblock)
 				break;		/* inappropriate superblock */
-			list_move(&inode->i_list, &sb->s_dirty);
 			continue;		/* not this blockdev */
 		}
 

.
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: 2.5.33-mm4 filemap_copy_from_user: Unexpected page fault

[Steven Cole]

On Fri, 2002-09-06 at 10:51, Andrew Morton wrote:
> Steven Cole wrote:
> > 
> > With 2.5.33-mm4, I tried running dbench on an ext2 partition and was
> > able to run up to dbench 80 successfully.  However, at dbench 96, I got
> > four messages like this:
> > 
> > filemap_copy_from_user: Unexpected page fault
> 
> Yep.  This means that the page we faulted in by-hand in generic_file_write()
> wasn't resident during the subsequent copy_from_user().
> 
> That fault-in by-hand is there to prevent a deadlock.  That printk
> meand that it isnt working all the time.   We have (always had) a
> problem.
> 
> > Shortly after this, the box hung again,
> > ...
> > >>EIP; c0159bf4 <sync_sb_inodes+84/260>   <=====
> > Trace; c0159e1e <writeback_inodes+4e/80>
> > Trace; c013b8aa <background_writeout+7a/c0>
> > Trace; c013b4cb <__pdflush+12b/1d0>
> 
> Hum.  Thanks for that.
> 
> I've been dbenching and compiling all night.  And yet, it
> seems that the dirty inode search in sync_sb_inodes() can trivially
> lock up.
> 
> Does this fix?
> 
>  fs-writeback.c |    4 ++--
>  1 files changed, 2 insertions(+), 2 deletions(-)
[patch snipped]

Unfortunately no.  With that patch, it got up to dbench 10 on ext3 and
then it hung again.

I haven't reboot the box yet in case the output from sysrq-p would be
useful.  It is a pain to type in though.

I had vmstat -n 1 1200 running in another terminal and here are the last
several lines of that output before the freeze:

 0  0  0      0 636688  25356  19192   0   0     0     0 1077  2983  19  44  37
10  0  0      0 589504  25856  65452   0   0    24     0 1022    41   2  37  61
 0 10  4      0 531940  26644 122060   0   0    12 22684 1194   431   6  51  42
 0 10  3      0 527316  26736 126592   0   0     4 12944 1189   246   2   7  91
 0 10  3      0 487488  27372 165676   0   0    12 15652 1238   569   7  41  52
 0 10  1      0 487476  27372 165676   0   0     0  4868 1167    84   0   3  97
 3  7  0      0 466636  27692 185728   0   0    20 12340 1188   202   6  24  70
 0 10  2      0 462580  27772 189280   0   0    20 13240 1302   282   1   7  91
 0 10  3      0 457832  27848 191244   0   0     8  9680 1350   334   2   5  93
 0 10  3      0 457448  27864 191564   0   0     4 10064 1406   277   0   2  99
 0 10  3      0 450812  27940 197804   0   0    12 12552 1382   351   2   6  92
 4  6  3      0 442804  28132 205204   0   0    12 21928 1231   721   7  17  75
 0 10  3      0 441248  28432 206376   0   0    12 18552 1195  5658  17  34  49
 0 10  2      0 440500  28468 206712   0   0     4  2288 1153   394   1   3  96
 9  3  5      0 438728  28580 207632   0   0    24 15100 1293   319   4  13  83

Steven

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: 2.5.33-mm4 filemap_copy_from_user: Unexpected page fault

[Andrew Morton]

Steven Cole wrote:
> 
> ...
> > Does this fix?
> ...
> Unfortunately no.

Well, isn't this fun?  umm.  You're _sure_ you ran the right kernel
and such?

Could you send your /proc/mounts, and tell me which of those partitions
you're running the test on?
--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/

Re: 2.5.33-mm4 filemap_copy_from_user: Unexpected page fault

[Steven Cole]

On Fri, 2002-09-06 at 11:36, Andrew Morton wrote:
> Steven Cole wrote:
> > 
> > ...
> > > Does this fix?
> > ...
> > Unfortunately no.
> 
> Well, isn't this fun?  umm.  You're _sure_ you ran the right kernel
> and such?

Oops, forgot to include linux-mm on the earlier reply.
Yes I am sure I ran the right kernel.

> 
> Could you send your /proc/mounts, and tell me which of those partitions
> you're running the test on?
> 
Here that is again, for the linux-mm list:

[steven@spc5 linux-2.5.33-mm4]$ cat /proc/mounts
rootfs / rootfs rw 0 0
/dev/root / ext3 rw 0 0
/proc /proc proc rw 0 0
none /dev/pts devpts rw 0 0
/dev/sda5 /home ext3 rw 0 0
none /dev/shm tmpfs rw 0 0
/dev/sdb2 /share ext2 rw 0 0
/dev/sda3 /usr ext3 rw 0 0


Test were run on /home (ext3) with single exception of one earlier test
on /share (ext2).

Steven

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/