On Thu, Apr 19 2007, Vladimir V. Saveliev wrote:
> Hello
> 
> On Thursday 19 April 2007 12:34, Jens Axboe wrote:
> > On Thu, Apr 19 2007, Andrew Morton wrote:
> > > On Thu, 19 Apr 2007 10:01:57 +0200 Jens Axboe <jens.axboe@oracle.com> wrote:
> > > 
> > > > On Thu, Apr 19 2007, Andrew Morton wrote:
> > > > > On Thu, 19 Apr 2007 09:38:30 +0200 Jens Axboe <jens.axboe@oracle.com> wrote:
> > > > > 
> > > > > > Hi,
> > > > > > 
> > > > > > Doing some testing on CFQ, I ran into this 100% reproducible report:
> > > > > > 
> > > > > > =======================================================
> > > > > > [ INFO: possible circular locking dependency detected ]
> > > > > > 2.6.21-rc7 #5
> > > > > > -------------------------------------------------------
> > > > > > fio/9741 is trying to acquire lock:
> > > > > >  (&mm->mmap_sem){----}, at: [<b018cb34>] dio_get_page+0x54/0x161
> > > > > > 
> > > > > > but task is already holding lock:
> > > > > >  (&inode->i_mutex){--..}, at: [<b038c6e5>] mutex_lock+0x1c/0x1f
> > > > > > 
> > > > > > which lock already depends on the new lock.
> > > > > > 
> > > > > 
> > > > > This is the correct ranking: i_mutex outside mmap_sem.
> > > > > 
> > > > > > 
> > > > > > the existing dependency chain (in reverse order) is:
> > > > > > 
> > > > > > -> #1 (&inode->i_mutex){--..}:
> > > > > >        [<b013e3fb>] __lock_acquire+0xdee/0xf9c
> > > > > >        [<b013e600>] lock_acquire+0x57/0x70
> > > > > >        [<b038c4a5>] __mutex_lock_slowpath+0x73/0x297
> > > > > >        [<b038c6e5>] mutex_lock+0x1c/0x1f
> > > > > >        [<b01b17e9>] reiserfs_file_release+0x54/0x447
> > > > > >        [<b016afe7>] __fput+0x53/0x101
> > > > > >        [<b016b0ee>] fput+0x19/0x1c
> > > > > >        [<b015bcd5>] remove_vma+0x3b/0x4d
> > > > > >        [<b015c659>] do_munmap+0x17f/0x1cf
> > > > > >        [<b015c6db>] sys_munmap+0x32/0x42
> > > > > >        [<b0103f04>] sysenter_past_esp+0x5d/0x99
> > > > > >        [<ffffffff>] 0xffffffff
> > > > > > 
> > > > > > -> #0 (&mm->mmap_sem){----}:
> > > > > >        [<b013e259>] __lock_acquire+0xc4c/0xf9c
> > > > > >        [<b013e600>] lock_acquire+0x57/0x70
> > > > > >        [<b0137b92>] down_read+0x3a/0x4c
> > > > > >        [<b018cb34>] dio_get_page+0x54/0x161
> > > > > >        [<b018d7a9>] __blockdev_direct_IO+0x514/0xe2a
> > > > > >        [<b01cf449>] ext3_direct_IO+0x98/0x1e5
> > > > > >        [<b014e8df>] generic_file_direct_IO+0x63/0x133
> > > > > >        [<b01500e9>] generic_file_aio_read+0x16b/0x222
> > > > > >        [<b017f8b6>] aio_rw_vect_retry+0x5a/0x116
> > > > > >        [<b0180147>] aio_run_iocb+0x69/0x129
> > > > > >        [<b0180a78>] io_submit_one+0x194/0x2eb
> > > > > >        [<b0181331>] sys_io_submit+0x92/0xe7
> > > > > >        [<b0103f90>] syscall_call+0x7/0xb
> > > > > >        [<ffffffff>] 0xffffffff
> > > > > 
> > > > > But here reiserfs is taking i_mutex in its file_operations.release(),
> > > > > which can be called under mmap_sem.
> > > > > 
> > > > > Vladimir's recent de14569f94513279e3d44d9571a421e9da1759ae.
> > > > > "resierfs: avoid tail packing if an inode was ever mmapped" comes real
> > > > > close to this code, but afaict it did not cause this bug.
> > > > > 
> > > > > I can't think of anything which we've done in the 2.6.21 cycle which
> > > > > would have caused this to start happening.  Odd.
> > > > 
> > > > The bug may be holder, let me know if you want me to check 2.6.20 or
> > > > earlier.
> > > 
> > > Would be great if you could test 2.6.20.  I have a feeling that I missed
> > > something, but what?  We didn't change the refcounting of lifetime of
> > > vma.vm_file...
> > 
> > 2.6.20.7 tested, same lockdep triggers. Attached for reference.
> > 
> 
> Did you have CFQ patches mentioned below applied?

Nope, stock 2.6.20.7. The CFQ patches should not make a difference,
unless I royally screwed something up :-)

> Would you please send your .config?

Attached. It's the 2.6.21-rc7 config, for 2.6.20.7 I just did a make
oldconfig, the options that showed up should not impact anything.

> I tried fio (1.15) with this job file and did not get the possible
> circular locking dependency detected

Perhaps some of the preempt settings? The box is an emc centera, it's a
lowly p4/ht.

-- 
Jens Axboe