From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, MAILING_LIST_MULTI,MENTIONS_GIT_HOSTING,SPF_HELO_NONE,SPF_PASS autolearn=unavailable autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 75C9AC3F2CD for ; Tue, 3 Mar 2020 13:06:38 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 390B32086A for ; Tue, 3 Mar 2020 13:06:38 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=gmail.com header.i=@gmail.com header.b="e1czJgec" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 390B32086A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id C98B46B0005; Tue, 3 Mar 2020 08:06:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C234B6B0006; Tue, 3 Mar 2020 08:06:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B11D56B0007; Tue, 3 Mar 2020 08:06:37 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0229.hostedemail.com [216.40.44.229]) by kanga.kvack.org (Postfix) with ESMTP id 9525C6B0005 for ; Tue, 3 Mar 2020 08:06:37 -0500 (EST) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay02.hostedemail.com (Postfix) with ESMTP id 5F4E72A499 for ; Tue, 3 Mar 2020 13:06:37 +0000 (UTC) X-FDA: 76554075234.22.rain36_6082b48ff523c Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin22.hostedemail.com (Postfix) with ESMTP id E05DC1807D7DC for ; Tue, 3 Mar 2020 13:04:23 +0000 (UTC) X-HE-Tag: rain36_6082b48ff523c X-Filterd-Recvd-Size: 4486 Received: from mail-qt1-f179.google.com (mail-qt1-f179.google.com [209.85.160.179]) by imf46.hostedemail.com (Postfix) with ESMTP for ; Tue, 3 Mar 2020 13:04:23 +0000 (UTC) Received: by mail-qt1-f179.google.com with SMTP id j34so2693298qtk.4 for ; Tue, 03 Mar 2020 05:04:23 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=sender:date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=jV9MOqj3WfpAvtB6pN8ntyBJlB6ixy+o1z9uv0T/wJ8=; b=e1czJgeczWvVajGwOP+z+FkY1TGYcksRitzO/boEV+27Up3xOd+ML1WMd36/dvfmtX FSHAl7TeJT7R2g+9BGewO7uEFH3wcoTDWPy/AJP/4dkarpzlvPjbnzepEQLzP0Qv10uk jJKJIS+e53attz6FAuvIfhQRiIX4guOAblBMFsskoFxJTYfKU3bdLh0lGG7az7iETbjw ePF+dYPzFd+NQ5zBwaZb2pBDL9KnYdc1DmkqM9exgDQ+7VCTmYo2lfWs503C9M2SSYe5 eZMHz86oOZZWfvLy5uHqJg0TY7xRY6EK3iK9cTm/duhnAugML/wJNbIF65Tlp2ULES3h 9Etw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:sender:date:from:to:cc:subject:message-id :references:mime-version:content-disposition:in-reply-to; bh=jV9MOqj3WfpAvtB6pN8ntyBJlB6ixy+o1z9uv0T/wJ8=; b=M5HdnFZvL33JBahFwvc5g5xJsJFU+wmr9dMcWzndfgxrNFMbCv2rqzD9eGa9Ap4u5W mKL1Yn3g1igJ1DCEQbEAVnw95WI9zTrIwDFf/p3CHm1m18iOE2PVw49nwPQZafAAzMRr y9/GHz+e1MtTVbr6BLuIItJaAUyiwxJyQOadETGZptUISlmWzwVO97JH39EZDeiS33s1 Lc4e5D5VPoObsE3O5oteRbzEy3h4sqyRugmQB0rSlm/ljR5k3BRb3Jj8uEHq7b6Uqr+w u4rDDf6mBinDgzr8g330rw/oAwDNp1/hCqZEWGJA71xF7j92ulyD3EbHWuXvKA92zr19 DBFw== X-Gm-Message-State: ANhLgQ0FPUFK6zImjaQer+FSLOgI0VmK5rEHbajJm4ITBwDMkIIgBC2B zlIX297QRjI9WNrQBrAmmEQ= X-Google-Smtp-Source: ADFU+vstoC3xFL6LK08SXwDvPUGTjYp3QYPGY5MkFBS7kLk9J+plna+Y2j509AStPcVDIZFAVjEGPA== X-Received: by 2002:ac8:5283:: with SMTP id s3mr4236292qtn.47.1583240662300; Tue, 03 Mar 2020 05:04:22 -0800 (PST) Received: from localhost ([2620:10d:c091:500::7f70]) by smtp.gmail.com with ESMTPSA id t13sm11860006qkm.60.2020.03.03.05.04.21 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 03 Mar 2020 05:04:21 -0800 (PST) Date: Tue, 3 Mar 2020 08:04:21 -0500 From: Tejun Heo To: Miklos Szeredi Cc: Michael Stapelberg , Jack Smith , fuse-devel , linux-fsdevel@vger.kernel.org, linux-mm@kvack.org Subject: Re: [fuse-devel] Writing to FUSE via mmap extremely slow (sometimes) on some machines? Message-ID: <20200303130421.GA5186@mtj.thefacebook.com> References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hello, Sorry about the delay. On Wed, Feb 26, 2020 at 08:59:55PM +0100, Miklos Szeredi wrote: > - apparently memcpy is copying downwards (from largest address to > smallest address). Not sure why, when I run the reproducer, it copies > upwards. > - there's a slow batch of reads of the first ~4MB of data, then a > quick writeback > - there's a quick read of the rest (~95MB) of data, then a quick > writeback of the same > > Plots of the whole and closeups of slow and quick segments attached. > X axis is time, Y axis is offset. > > Tejun, could this behavior be attributed to dirty throttling? What > would be the best way to trace this? Yeah, seems likely. Can you please try offcputime (or just sample /proc/PID/stack) and see whether it's in balance dirty pages? https://github.com/iovisor/bcc/blob/master/tools/offcputime.py If it's dirty throttling, the next step would be watching the bdp tracepoints to find out what kind of numbers it's getting. Thanks. -- tejun