From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1B013EB64D7 for ; Tue, 20 Jun 2023 21:43:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 52F008D0002; Tue, 20 Jun 2023 17:43:45 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4DDF78D0001; Tue, 20 Jun 2023 17:43:45 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3A6558D0002; Tue, 20 Jun 2023 17:43:45 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 2A8458D0001 for ; Tue, 20 Jun 2023 17:43:45 -0400 (EDT) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id B44111605CA for ; Tue, 20 Jun 2023 21:43:44 +0000 (UTC) X-FDA: 80924453568.08.3AABB5D Received: from mail-wr1-f41.google.com (mail-wr1-f41.google.com [209.85.221.41]) by imf27.hostedemail.com (Postfix) with ESMTP id D115D40008 for ; Tue, 20 Jun 2023 21:43:42 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=OW3C6O+z; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf27.hostedemail.com: domain of lstoakes@gmail.com designates 209.85.221.41 as permitted sender) smtp.mailfrom=lstoakes@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1687297422; a=rsa-sha256; cv=none; b=KR6KUGS3s2aEVrh4AGhIm1slS7+BoY7rm1SrW62Rgki8IuxozPSJPkhZyZK1cef0CvKYmk ChLpchNqR86KGwl8regTQ27Kw0oe8wWYBp3iaORJyEsftKlZn6W5EGEkQgx+r2WFr0E801 E0wSyAYardGe6HOPhE6qXL28m9a6Tvc= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=gmail.com header.s=20221208 header.b=OW3C6O+z; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf27.hostedemail.com: domain of lstoakes@gmail.com designates 209.85.221.41 as permitted sender) smtp.mailfrom=lstoakes@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1687297422; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=OFOIV5n7LYzSxODkxerJSJAEZ596/FRwMkn7FgiJCPM=; b=O5P9qicDJr2r+dqu7FY3PIp3+V5hcqIqd2i30Y7ZSXdwBg3vGPknW1X61XilgCJp95bueI N98T8edurje2xS9PU1L8XSXLeU/tTkaReII92QHlr93ALExhDfEcPxeERF9RKQv98skAIK t73yVq31GQ2skPr6H7D9mWsuaAxFGDk= Received: by mail-wr1-f41.google.com with SMTP id ffacd0b85a97d-30fbf253dc7so4139302f8f.0 for ; Tue, 20 Jun 2023 14:43:42 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20221208; t=1687297421; x=1689889421; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=OFOIV5n7LYzSxODkxerJSJAEZ596/FRwMkn7FgiJCPM=; b=OW3C6O+zZOenJS3jTySNr7Xrwf1eS4GHjg0AiTKXTVvRFNnvkZupDNuh8ErLLKUKX+ NdxOc25lYONtyKd40AZGJBWVqC4313oQ10voKXdNV/lkJJmit2E+7RKf5ZlZ8+4ydL3E 69WSh5Y83A+OhCkl2Q6CIRwBLOjzob8TdlEzK2XwyX03H5fAjNHnC0WpWBe0+uBLCTjH XyfqIFG6X7uLBHyA/+v8ZjFFqi6cH8rIdZqB9qUJW7OlrBiTdVEBjIqX5fT/ggfY+8Bn QAaMBF3acmbInhir+dzE/DQAaHKKmR73+E0Zl/e09CCidEa+gQN0xva+mlHSOMDpH+DB Izmw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1687297421; x=1689889421; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=OFOIV5n7LYzSxODkxerJSJAEZ596/FRwMkn7FgiJCPM=; b=GWQanwt7/ut0a7/+/rsoV2MRoaY3BUl9/j2oEMQKdJIbVKPAWUVqrezs1jQg4zBgl5 zZp8uKf4lib1a81TFmH038G+amTJ/v2YPrBawkYA+ovZFvicFPUi/8eqLUW7GZGwVCI0 fNflbNN9Zr7aADBDYPdorLhE1Is5FQISjOR2Gaq9ENPXgy+jQn+K2McyWITTt8Qr556J LU/ptYycKyRQkI4lqb7AcP7GCGsVq9ymo53+bFcJ0vfqHtVa0keh2H59WZ7j0LMnBAge BRdwkWNph21zyhLOJenqoMqZ71A2mMbl4tXTQehm7fn3eLxa6hmFgCcE5jVgl6/+dgHR IANw== X-Gm-Message-State: AC+VfDziPzcaAC0RtaOjRTgA1fJWh5kH9cpZrfMsXRu7RSOzMEKEoICc FqrZl9IYh8r60ci8bMt834E= X-Google-Smtp-Source: ACHHUZ45fYpIIZajjBz0hE2VIpo921OqmDAvp90wUO1IreQfyV3apYvKNRUnEVb4DlVT1q98uSd7lw== X-Received: by 2002:a5d:45c3:0:b0:311:1944:ad33 with SMTP id b3-20020a5d45c3000000b003111944ad33mr10814813wrs.12.1687297421070; Tue, 20 Jun 2023 14:43:41 -0700 (PDT) Received: from localhost ([2a00:23c5:dc8c:8701:1663:9a35:5a7b:1d76]) by smtp.gmail.com with ESMTPSA id t16-20020a5d49d0000000b0030aed4223e0sm2784675wrs.105.2023.06.20.14.43.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 20 Jun 2023 14:43:40 -0700 (PDT) Date: Tue, 20 Jun 2023 22:43:39 +0100 From: Lorenzo Stoakes To: Peter Xu Cc: linux-mm@kvack.org, linux-kernel@vger.kernel.org, Andrea Arcangeli , Mike Rapoport , David Hildenbrand , Matthew Wilcox , Vlastimil Babka , John Hubbard , "Kirill A . Shutemov" , James Houghton , Andrew Morton , Hugh Dickins , Mike Kravetz , Jason Gunthorpe Subject: Re: [PATCH v2 5/8] mm/gup: Accelerate thp gup even for "pages != NULL" Message-ID: <956f7c72-4c7d-43a5-8786-5fdaa9010f7b@lucifer.local> References: <20230619231044.112894-1-peterx@redhat.com> <20230619231044.112894-6-peterx@redhat.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20230619231044.112894-6-peterx@redhat.com> X-Rspam-User: X-Rspamd-Server: rspam06 X-Rspamd-Queue-Id: D115D40008 X-Stat-Signature: 7m3q8z1b3ukfao5fiij7tc3nzs1jb6fx X-HE-Tag: 1687297422-188634 X-HE-Meta: U2FsdGVkX1+/MNh4AqMjBBhorGBxQyfIGaN/PdPgBxJtoyk4pFTkkV+TukjVllc/DZLW3PvgbqgRgN3zFAx8roPZaCLNHZksP2KQj/UicaucVOHb0B0fhuM/hZMyi/tHAMRDMJmauo/mIG3Q0YIrx4qpwh7Ds1xFMWBo1yogM+MA4kYBidjngnUs6AVqtyZNJjxqYiVCkRYFCsUNf3NcvChOQuLv45fS1woPBqeqAmu9og/9dayJHmHxLbEdYSMWgVrzSstppF1gsV0b26AC2YVps7UYKES/9g4e/DgZSXJ99quzdhs3bvzKFCG0aeVrGZIEjRM81xnB5QEMEQkhP2E5Bt1OE/ePqsM6RVRLVtUEzF92vsh2FIrQAPO/ybrnGAHW8vNCfIR/JXDfeDkYJKj+WbSyPAZdl0aJtnCKQDz+qa65qmTEuTMur1XQs/UPWUahrSaQ6c389EgI8v9bKgYBks/on7+VjC4eJTXQLNsekZ177sBdGaqFrsD/E9Ze1bF6scZznC5TO7d3XgbWkVc41hQmYB08n+U4gbOdqFi9MRqz+Gf+MyrYdZu1N9XBzCN5FRBwiAcrXO8RmnJGSEgOVqg0mtER24s5Jxw38l+yKmqz9yXHqVszQeTCd6QkXfbNka5Tn2o0c/vQuG62OrxFWP0xPu651fA/OzrqomxGGfFnkCUfiqTH3AF9R+ruZh6tpbWRjndyST4/JvLmZ4ITIdwTSZg3dH3qHErIFqUc6y3hy+YB7Dj1Y5NlfaeGQcEVwo20VrdI+QG8ElHCf5KktBU4hnX8wUp2dMj1hnf2r3CvR8xnpaWb6PhrAh6gP42NywYMIjaHyRDoFdu0fMttr4nlkyNt5WYfSl3kq8Pj6KR81/Np1wHYV9nmgTEBZKPH7TsMByIxHsq3K9Ep+LrPU8UkXzKXiPnU+p3RMq/6QUYjXvTdNkekrm5NGkyWOHZFkZIqxw013KxJGWu XxFWhKT4 jb1ppsl9qWoTK9FoyIbkw6POOiDCiFXvyCUw6Ihot930Xv+anc/m6AlhO/OyvpTgTE5IuzZgQ5ken1y4XrGyVkoDIri2WQwchHmSSgTYKcj2omQ9u+ePYZPBbbDNYUim0zWsMbodjErSdH03jOBbrKwabu4TZcXpBJZQMzgMkHMFwthve5yDdHD3Vj6ZgVylMLUuMXdl0637QcsisQ7VyqxPA+KoE+odBxDH0/0bc+ZNWrjnzO8LSvykP1P0nvN0kdBvH1W+fl/OpGwLX3g0l8/Bhsg9n97q0Z1V5SV2Z2sMqFmB5+NrkYllv2Qkn/TwFmg62L4f9m7tYasl6zcfsG5Ka/UswSyE+KFxq6rsusjHKF6NfI29jq9GghLyTms6C79MtYaZvs9jBcEcPLiWj4wUlkeqzCcL0R9BNaT32gmR0IhiEsywbanEV9eIVhap32GqfX8nMugS4/1T/ZEc6HwEqyGjuZmIwA07ifEpy4v9wYes= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Jun 19, 2023 at 07:10:41PM -0400, Peter Xu wrote: > The acceleration of THP was done with ctx.page_mask, however it'll be > ignored if **pages is non-NULL. > > The old optimization was introduced in 2013 in 240aadeedc4a ("mm: > accelerate mm_populate() treatment of THP pages"). It didn't explain why > we can't optimize the **pages non-NULL case. It's possible that at that > time the major goal was for mm_populate() which should be enough back then. > > Optimize thp for all cases, by properly looping over each subpage, doing > cache flushes, and boost refcounts / pincounts where needed in one go. > > This can be verified using gup_test below: > > # chrt -f 1 ./gup_test -m 512 -t -L -n 1024 -r 10 > > Before: 13992.50 ( +-8.75%) > After: 378.50 (+-69.62%) > > Signed-off-by: Peter Xu > --- > mm/gup.c | 51 ++++++++++++++++++++++++++++++++++++++++++++------- > 1 file changed, 44 insertions(+), 7 deletions(-) > > diff --git a/mm/gup.c b/mm/gup.c > index 4a00d609033e..b50272012e49 100644 > --- a/mm/gup.c > +++ b/mm/gup.c > @@ -1199,16 +1199,53 @@ static long __get_user_pages(struct mm_struct *mm, > goto out; > } > next_page: > - if (pages) { > - pages[i] = page; > - flush_anon_page(vma, page, start); > - flush_dcache_page(page); > - ctx.page_mask = 0; > - } > - > page_increm = 1 + (~(start >> PAGE_SHIFT) & ctx.page_mask); > if (page_increm > nr_pages) > page_increm = nr_pages; > + > + if (pages) { > + struct page *subpage; > + unsigned int j; > + > + /* > + * This must be a large folio (and doesn't need to > + * be the whole folio; it can be part of it), do > + * the refcount work for all the subpages too. > + * > + * NOTE: here the page may not be the head page > + * e.g. when start addr is not thp-size aligned. > + * try_grab_folio() should have taken care of tail > + * pages. > + */ > + if (page_increm > 1) { > + struct folio *folio; > + > + /* > + * Since we already hold refcount on the > + * large folio, this should never fail. > + */ > + folio = try_grab_folio(page, page_increm - 1, > + foll_flags); > + if (WARN_ON_ONCE(!folio)) { > + /* > + * Release the 1st page ref if the > + * folio is problematic, fail hard. > + */ > + gup_put_folio(page_folio(page), 1, > + foll_flags); > + ret = -EFAULT; > + goto out; > + } Thanks this looks good to me, I agree it'd be quite surprising for us not to retrieve folio here and probably something has gone wrong if so, so not actually too unreasonable to warn, as long as we error out. > + } > + > + for (j = 0; j < page_increm; j++) { > + subpage = nth_page(page, j); > + pages[i+j] = subpage; > + flush_anon_page(vma, subpage, start + j * PAGE_SIZE); > + flush_dcache_page(subpage); > + } > + } > + > i += page_increm; > start += page_increm * PAGE_SIZE; > nr_pages -= page_increm; > -- > 2.40.1 > Looks good to me overall, Reviewed-by: Lorenzo Stoakes