Re: ext4 hang and per-memcg dirty throttling

[Liu Bo <bo.liu@linux.alibaba.com>]

On Wed, Sep 12, 2018 at 02:11:30PM +0200, Jan Kara wrote:
> Hi!
> 
> On Tue 11-09-18 17:10:55, Liu Bo wrote:
> > With ext4's data=ordered mode and the underlying blk throttle setting, we
> > can easily run to hang,
> > 
> > 1.
> > mount /dev/sdc /mnt -odata=ordered
> > 2.
> > mkdir /sys/fs/cgroup/unified/cg
> > 3.
> > echo "+io" > /sys/fs/cgroup/unified/cgroup.subtree_control
> > 4.
> > echo "`cat /sys/block/sdc/dev` wbps=$((1 << 20))" > /sys/fs/cgroup/unified/cg/io.max
> > 5.
> > echo $$ >  /sys/fs/cgroup/unified/cg/cgroup.procs
> > 6.
> > // background dirtier
> > xfs_io -f -c "pwrite 0 1G" $M/dummy &
> > 7.
> > echo $$ > /sys/fs/cgroup/unified/cgroup.procs
> > 8.
> > // issue synchronous IO
> > for i in `seq 1 100`;
> > do
> >     xfs_io -f -s -c "pwrite 0 4k" $M/foo > /dev/null
> > done
> > 
> > 
> > And the hang is like
> > 
> >       [jbd2-sdc]
> > jbd2_journal_commit_transaction                              
> >   journal_submit_data_buffers
> >     # file 'dummy' has been written by writeback kthread
> >   journal_finish_inode_data_buffers
> >     # wait on page's writeback
> 
> Yes, I guess you're speaking about the one Chris Mason mentioned [1].

Exactly.

> Essentially it's a priority inversion where jbd2 thread gets blocked behind
> writeback done on behalf of a heavily restricted process. It actually is
> not related to dirty throttling or anything like that. And the solution for
> this priority inversion is to use unwritten extents for writeback
> unconditionally as I wrote in that thread. The core of this is implemented
> and hidden behind dioread_nolock mount option but it needs some serious
> polishing work and testing...

Thank you so much for the details, so setting extent to unwritten and
then converting it in endio does work and keeps the data=ordered
semantic but I have to say the name, "dioread_nolock", is really
confusing...

thanks,
-liubo
> 
> [1] https://marc.info/?l=linux-fsdevel&m=151688776319077
> 
> 								Honza
> -- 
> Jan Kara <jack@suse.com>
> SUSE Labs, CR