linux-mm.kvack.org archive mirror
 help / color / mirror / Atom feed
From: Tejun Heo <tj@kernel.org>
To: Miklos Szeredi <miklos@szeredi.hu>
Cc: Michael Stapelberg <michael+lkml@stapelberg.ch>,
	Jack Smith <smith.jack.sidman@gmail.com>,
	fuse-devel <fuse-devel@lists.sourceforge.net>,
	linux-fsdevel@vger.kernel.org, linux-mm@kvack.org
Subject: Re: [fuse-devel] Writing to FUSE via mmap extremely slow (sometimes) on some machines?
Date: Tue, 3 Mar 2020 08:04:21 -0500	[thread overview]
Message-ID: <20200303130421.GA5186@mtj.thefacebook.com> (raw)
In-Reply-To: <CAJfpegtkEU9=3cvy8VNr4SnojErYFOTaCzUZLYvMuQMi050bPQ@mail.gmail.com>

Hello,

Sorry about the delay.

On Wed, Feb 26, 2020 at 08:59:55PM +0100, Miklos Szeredi wrote:
> - apparently memcpy is copying downwards (from largest address to
> smallest address).  Not sure why, when I run the reproducer, it copies
> upwards.
> - there's a slow batch of reads of the first ~4MB of data, then a
> quick writeback
> - there's a quick read of the rest (~95MB) of data, then a quick
> writeback of the same
> 
> Plots of the whole and closeups of slow and quick segments attached.
> X axis is time, Y axis is offset.
> 
> Tejun, could this behavior be attributed to dirty throttling?  What
> would be the best way to trace this?

Yeah, seems likely. Can you please try offcputime (or just sample
/proc/PID/stack) and see whether it's in balance dirty pages?

  https://github.com/iovisor/bcc/blob/master/tools/offcputime.py

If it's dirty throttling, the next step would be watching the bdp
tracepoints to find out what kind of numbers it's getting.

Thanks.

-- 
tejun


  parent reply	other threads:[~2020-03-03 13:06 UTC|newest]

Thread overview: 12+ messages / expand[flat|nested]  mbox.gz  Atom feed  top
     [not found] <CANnVG6kZzN1Ja0EmxG3pVTdMx8Kf8fezGWBtCYUzk888VaFThg@mail.gmail.com>
     [not found] ` <CACQJH27s4HKzPgUkVT+FXWLGqJAAMYEkeKe7cidcesaYdE2Vog@mail.gmail.com>
     [not found]   ` <CANnVG6=Ghu5r44mTkr0uXx_ZrrWo2N5C_UEfM59110Zx+HApzw@mail.gmail.com>
     [not found]     ` <CAJfpegvzhfO7hg1sb_ttQF=dmBeg80WVkV8srF3VVYHw9ybV0w@mail.gmail.com>
     [not found]       ` <CANnVG6kSJJw-+jtjh-ate7CC3CsB2=ugnQpA9ACGFdMex8sftg@mail.gmail.com>
2020-02-26 19:59         ` Miklos Szeredi
2020-03-03 10:34           ` Michael Stapelberg
2020-03-03 13:04           ` Tejun Heo [this message]
2020-03-03 14:03             ` Michael Stapelberg
2020-03-03 14:13               ` Tejun Heo
2020-03-03 14:21                 ` Michael Stapelberg
2020-03-03 14:25                   ` Tejun Heo
2020-03-05 14:45                     ` Michael Stapelberg
2020-03-09 14:32                       ` Michael Stapelberg
2020-03-09 14:36                         ` Miklos Szeredi
2020-03-09 15:11                           ` Michael Stapelberg
2020-03-12 15:45                             ` Michael Stapelberg

Reply instructions:

You may reply publicly to this message via plain-text email
using any one of the following methods:

* Save the following mbox file, import it into your mail client,
  and reply-to-all from there: mbox

  Avoid top-posting and favor interleaved quoting:
  https://en.wikipedia.org/wiki/Posting_style#Interleaved_style

* Reply using the --to, --cc, and --in-reply-to
  switches of git-send-email(1):

  git send-email \
    --in-reply-to=20200303130421.GA5186@mtj.thefacebook.com \
    --to=tj@kernel.org \
    --cc=fuse-devel@lists.sourceforge.net \
    --cc=linux-fsdevel@vger.kernel.org \
    --cc=linux-mm@kvack.org \
    --cc=michael+lkml@stapelberg.ch \
    --cc=miklos@szeredi.hu \
    --cc=smith.jack.sidman@gmail.com \
    /path/to/YOUR_REPLY

  https://kernel.org/pub/software/scm/git/docs/git-send-email.html

* If your mail client supports setting the In-Reply-To header
  via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line before the message body. 
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox