From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB4B0C433EF for ; Fri, 25 Mar 2022 12:31:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E97BB6B0071; Fri, 25 Mar 2022 08:31:32 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E47E46B0073; Fri, 25 Mar 2022 08:31:32 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CC2026B0074; Fri, 25 Mar 2022 08:31:32 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0021.hostedemail.com [216.40.44.21]) by kanga.kvack.org (Postfix) with ESMTP id B70D86B0071 for ; Fri, 25 Mar 2022 08:31:32 -0400 (EDT) Received: from smtpin30.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 77B709F301 for ; Fri, 25 Mar 2022 12:31:32 +0000 (UTC) X-FDA: 79282844424.30.2C74976 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf15.hostedemail.com (Postfix) with ESMTP id F185BA0038 for ; Fri, 25 Mar 2022 12:31:31 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1648211490; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=yPjA/sNpnOcC3OZqw/MwHjQE8KDwlLZ97r+qgQx76dw=; b=gQXz9mi20ggg2DFjufFXrPECRrIdm1oFopb3y8CBzYa1RGLWeA5v54OGIZ3YzrXRPVZjlZ JCFqhPwr4o0vdk6hrJKK1XpgR8DCGSeWsH9//v6yfNc9GkSdW5qmc59/QXCCJLlrtaxUIX 55deRwbDGSAiAb+HonoXbp/CJ5J5A4U= Received: from mail-yw1-f200.google.com (mail-yw1-f200.google.com [209.85.128.200]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id us-mta-518-VAE-Y-BEPiSRpZcYqEQpbA-1; Fri, 25 Mar 2022 08:31:29 -0400 X-MC-Unique: VAE-Y-BEPiSRpZcYqEQpbA-1 Received: by mail-yw1-f200.google.com with SMTP id 00721157ae682-2e689dfe112so59166237b3.20 for ; Fri, 25 Mar 2022 05:31:28 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=yPjA/sNpnOcC3OZqw/MwHjQE8KDwlLZ97r+qgQx76dw=; b=TSKsQxGC13AhH7rPxW6XJ5AypV3k0FmO31d6GdMxCW8wzLtxriGzhVW4TrpZU2It4i PZ733Emg9BuIdebN0jFDnaHlxLovBLNgGmeNnguyz5wcKf58NneaNkv0grEFxvTVfQQA j0lGguzSau3mRNy5SIZc/jXcl+OXICTaFfF54uK7/uUpzfMJnqCegW8D8GK4MxkxpkPU FIQmwLmzhk9YjpRSbaIECEeD3XndngPmGje9ERFdBGVcqZOJoqvdIKrtTMyHzhMXWfcf 3qslxute0Gb98I5ZuE3BvvP9ui8JmBazfKlu9SzSlKFhxTHrLI7/RpvkB8TCoK1EQlfm V24Q== X-Gm-Message-State: AOAM531OYipD7zkfB7SZOGg6b5i6I6SQvvjsdNQLUauELrC3+P0QJCdd csAerEHi6tgJQJETP94G5NV9rDlYDw9KtioL/ulcySt4eSAVe3o0/js2vsyD7+SJkSJw3qDIlac wZv3Ghme/x8Bh8EikWbZ+YLN+Eyg= X-Received: by 2002:a25:22d6:0:b0:633:90f5:a3d with SMTP id i205-20020a2522d6000000b0063390f50a3dmr9732496ybi.402.1648211488034; Fri, 25 Mar 2022 05:31:28 -0700 (PDT) X-Google-Smtp-Source: ABdhPJwLbkMySWcVby1qkRC7ubGRx41+P5/WfhnoVCsR7ijV5nDvgjZqE3XiUpuJVnnCKjaklKxotiIsqs4q8DkvuU8= X-Received: by 2002:a25:22d6:0:b0:633:90f5:a3d with SMTP id i205-20020a2522d6000000b0063390f50a3dmr9732460ybi.402.1648211487785; Fri, 25 Mar 2022 05:31:27 -0700 (PDT) MIME-Version: 1.0 References: <20220323201841.4166549-1-tbecker@redhat.com> In-Reply-To: From: Thiago Becker Date: Fri, 25 Mar 2022 09:31:16 -0300 Message-ID: Subject: Re: [PATCH RFC v3 0/6] Intruduce nfsrahead To: Matthew Wilcox Cc: linux-nfs@vger.kernel.org, Steved , trond.myklebust@hammerspace.com, anna.schumaker@netapp.com, Olga Kornievskaia , linux-fsdevel@vger.kernel.org, linux-mm@kvack.org X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain; charset="UTF-8" X-Stat-Signature: af9twtm6cq5qp4ss97zffnmddhow5as8 Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gQXz9mi2; spf=none (imf15.hostedemail.com: domain of tbecker@redhat.com has no SPF policy when checking 170.10.133.124) smtp.mailfrom=tbecker@redhat.com; dmarc=pass (policy=none) header.from=redhat.com X-Rspam-User: X-Rspamd-Server: rspam08 X-Rspamd-Queue-Id: F185BA0038 X-HE-Tag: 1648211491-219110 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000014, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Hello, On Wed, Mar 23, 2022 at 6:32 PM Matthew Wilcox wrote: > Which recent changes? Something in NFS or something in the VFS/MM? > Did you even think about asking a wider audience than the NFS mailing > list? I only happened to notice this while I was looking for something > else, otherwise I would never have seen it. The responses from other > people to your patches were right; you're trying to do this all wrong. > > Let's start out with a bug report instead of a solution. What changed > and when? > As Trond stated, c128e575514c ("NFS: Optimise the default readahead size") changed the way readahead is calculated for NFS mounts. This caused some read workloads to underperform, compared to the performance from previous revisions. To recall, the current policy is to adopt the system default readahead of 128kiB, and mounts with sec=krb5p take a performance hit of 50-75% when readahead is 128. I haven't performed an exhaustive search for other workloads that might also be affected, but I noticed the meaningful drop in performance in sec=sys mounts, notes at the end. The previous policy was to calculate the readahead as a multiple of rsize, so we prescribed increasing the value to the complaining part, and this fixed the issue. We are now trying to find a solution that we can incorporate into the system. thiago. ----- Tests ===== RAWHIDE (35% performance hit) ===== # uname -r 5.16.0-0.rc0.20211112git5833291ab6de.12.fc36.x86_64 # grep nfs /proc/self/mountinfo 601 60 0:55 / /mnt rw,relatime shared:332 - nfs4 192.168.122.225:/exports rw,vers=4.2,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.122.83,local_lock=none,addr=192.168.122.225 # cat /sys/class/bdi/0\:55/read_ahead_kb 128 # for i in {0..3} ; do dd if=/mnt/testfile.bin of=/dev/null bs=1M 2>&1 | grep copied ; echo 3 > /proc/sys/vm/drop_caches ; done 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 16.5025 s, 260 MB/s 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 16.4474 s, 261 MB/s 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 18.0181 s, 238 MB/s 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 18.2323 s, 236 MB/s # echo 15360 > /sys/class/bdi/0\:55/read_ahead_kb # for i in {0..3} ; do dd if=/mnt/testfile.bin of=/dev/null bs=1M 2>&1 | grep copied ; echo 3 > /proc/sys/vm/drop_caches ; done 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 11.2601 s, 381 MB/s 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 11.1885 s, 384 MB/s 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 11.5877 s, 371 MB/s 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 10.9475 s, 392 MB/s ===== UPSTREAM (30% performance hit) ===== # uname -r 5.17.0+ # grep nfs /proc/self/mountinfo 583 60 0:55 / /mnt rw,relatime shared:302 - nfs4 192.168.122.225:/exports rw,vers=4.2,rsize=262144,wsize=262144,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,clientaddr=192.168.122.83,local_lock=none,addr=192.168.122.225 # cat /sys/class/bdi/0\:55/read_ahead_kb 128 # for i in {0..3} ; do dd if=/mnt/testfile.bin of=/dev/null bs=1M 2>&1 | grep copied ; echo 3 > /proc/sys/vm/drop_caches ; done 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 17.056 s, 252 MB/s 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 17.1258 s, 251 MB/s 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 16.5981 s, 259 MB/s 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 16.5487 s, 260 MB/s # echo 15360 > /sys/class/bdi/0\:55/read_ahead_kb # for i in {0..3} ; do dd if=/mnt/testfile.bin of=/dev/null bs=1M 2>&1 | grep copied ; echo 3 > /proc/sys/vm/drop_caches ; done 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 12.3855 s, 347 MB/s 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 11.2528 s, 382 MB/s 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 11.9849 s, 358 MB/s 4294967296 bytes (4.3 GB, 4.0 GiB) copied, 11.2953 s, 380 MB/s