From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.3 required=3.0 tests=BAYES_00,DKIMWL_WL_HIGH, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_CR_TRAILER,INCLUDES_PATCH,MAILING_LIST_MULTI,NICE_REPLY_A, SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED,USER_AGENT_SANE_1 autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9E7B2C4361B for ; Wed, 9 Dec 2020 20:57:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3491B23C91 for ; Wed, 9 Dec 2020 20:57:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3491B23C91 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=nvidia.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6243A6B00FE; Wed, 9 Dec 2020 15:57:43 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 5D2E08D0031; Wed, 9 Dec 2020 15:57:43 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4C20D6B0100; Wed, 9 Dec 2020 15:57:43 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0053.hostedemail.com [216.40.44.53]) by kanga.kvack.org (Postfix) with ESMTP id 33B1F6B00FE for ; Wed, 9 Dec 2020 15:57:43 -0500 (EST) Received: from smtpin04.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id F20CA1EE6 for ; Wed, 9 Dec 2020 20:57:42 +0000 (UTC) X-FDA: 77574955164.04.crook78_240a155273f2 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin04.hostedemail.com (Postfix) with ESMTP id CE4F9800CE1B for ; Wed, 9 Dec 2020 20:57:42 +0000 (UTC) X-HE-Tag: crook78_240a155273f2 X-Filterd-Recvd-Size: 8574 Received: from hqnvemgate26.nvidia.com (hqnvemgate26.nvidia.com [216.228.121.65]) by imf11.hostedemail.com (Postfix) with ESMTP for ; Wed, 9 Dec 2020 20:57:41 +0000 (UTC) Received: from hqmail.nvidia.com (Not Verified[216.228.121.13]) by hqnvemgate26.nvidia.com (using TLS: TLSv1.2, AES256-SHA) id ; Wed, 09 Dec 2020 12:57:40 -0800 Received: from [10.2.60.96] (172.20.13.39) by HQMAIL107.nvidia.com (172.20.187.13) with Microsoft SMTP Server (TLS) id 15.0.1473.3; Wed, 9 Dec 2020 20:57:38 +0000 Subject: Re: [PATCH] mm/up: combine put_compound_head() and unpin_user_page() To: Jason Gunthorpe , Andrew Morton , Joao Martins , linux-mm CC: Jonathan Corbet , Dan Williams , Dave Chinner , Christoph Hellwig , Ira Weiny , Jan Kara , Jane Chu , "Kirill A. Shutemov" , , Michal Hocko , Mike Kravetz , Shuah Khan , Muchun Song , Vlastimil Babka , Matthew Wilcox References: <0-v1-6730d4ee0d32+40e6-gup_combine_put_jgg@nvidia.com> From: John Hubbard Message-ID: <16128311-9874-dcd4-c641-c68b34c9d634@nvidia.com> Date: Wed, 9 Dec 2020 12:57:38 -0800 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:84.0) Gecko/20100101 Thunderbird/84.0 MIME-Version: 1.0 In-Reply-To: <0-v1-6730d4ee0d32+40e6-gup_combine_put_jgg@nvidia.com> Content-Type: text/plain; charset="UTF-8"; format=flowed Content-Language: en-US Content-Transfer-Encoding: 7bit X-Originating-IP: [172.20.13.39] X-ClientProxiedBy: HQMAIL107.nvidia.com (172.20.187.13) To HQMAIL107.nvidia.com (172.20.187.13) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=nvidia.com; s=n1; t=1607547460; bh=kAU3uFvSz/nEO0cxIKc87yDDQtmd0SQyKa8HfojZbg0=; h=Subject:To:CC:References:From:Message-ID:Date:User-Agent: MIME-Version:In-Reply-To:Content-Type:Content-Language: Content-Transfer-Encoding:X-Originating-IP:X-ClientProxiedBy; b=X7Nrap8Mp/7CdaFw/Aw3PzAwAwXk5388RfTOo8JHkZcHbs4G56iafaLRBITb1BLsz IsBOXBS9nAA4MYPztcgcO/reON7W57vwwAic+A6ytUXmjq8ueysqymnvQUHrjg9pCW 6L21ZmSokfzeJSRnwDswMwA+L4P6Qag1QGydkb9sdzxPCV57D5RcuwZYJ/ipCtM4Ws i7X+y0QFtPsDkgPCz1LXDaY2Dfhku3Kprsx9zepL9/+BKxXWqhD0W5AS4T3y26RoCP K/LGCuReegzbrxY4i1IuLTz96KalVUzOM6g3I47F3NxS6Js1RINU1EQXDzLT9wag4F p6YEk8CkrwFaw== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 12/9/20 11:13 AM, Jason Gunthorpe wrote: > These functions accomplish the same thing but have different > implementations. > > unpin_user_page() has a bug where it calls mod_node_page_state() after > calling put_page() which creates a risk that the page could have been > hot-uplugged from the system. > > Fix this by using put_compound_head() as the only implementation. > > __unpin_devmap_managed_user_page() and related can be deleted as well in > favour of the simpler, but slower, version in put_compound_head() that has > an extra atomic page_ref_sub, but always calls put_page() which internally > contains the special devmap code. > > Move put_compound_head() to be directly after try_grab_compound_head() so > people can find it in future. > > Fixes: 1970dc6f5226 ("mm/gup: /proc/vmstat: pin_user_pages (FOLL_PIN) reporting") > Signed-off-by: Jason Gunthorpe > --- > mm/gup.c | 103 +++++++++++++------------------------------------------ > 1 file changed, 23 insertions(+), 80 deletions(-) > Reviewed-by: John Hubbard With a couple of minor notes below: > With Matt's folio idea I'd next to go to make a > put_folio(folio, refs) > > Which would cleanly eliminate that extra atomic here without duplicating the > devmap special case. > > This should also be called 'ungrab_compound_head' as we seem to be using the > word 'grab' to mean 'pin or get' depending on GUP flags. > > diff --git a/mm/gup.c b/mm/gup.c > index 98eb8e6d2609c3..7b33b7d4b324d7 100644 > --- a/mm/gup.c > +++ b/mm/gup.c > @@ -123,6 +123,28 @@ static __maybe_unused struct page *try_grab_compound_head(struct page *page, > return NULL; > } > > +static void put_compound_head(struct page *page, int refs, unsigned int flags) > +{ It might be nice to rename "page" to "head", here. While reading this I toyed with the idea of having this at the top: VM_BUG_ON_PAGE(compound_head(page) != page, page); ...but it's overkill in a static function with pretty clear call sites. So I think it's just right as-is. > + if (flags & FOLL_PIN) { > + mod_node_page_state(page_pgdat(page), NR_FOLL_PIN_RELEASED, > + refs); > + > + if (hpage_pincount_available(page)) > + hpage_pincount_sub(page, refs); > + else > + refs *= GUP_PIN_COUNTING_BIAS; > + } > + > + VM_BUG_ON_PAGE(page_ref_count(page) < refs, page); > + /* > + * Calling put_page() for each ref is unnecessarily slow. Only the last > + * ref needs a put_page(). > + */ > + if (refs > 1) > + page_ref_sub(page, refs - 1); > + put_page(page); > +} > + > /** > * try_grab_page() - elevate a page's refcount by a flag-dependent amount > * > @@ -177,41 +199,6 @@ bool __must_check try_grab_page(struct page *page, unsigned int flags) > return true; > } > > -#ifdef CONFIG_DEV_PAGEMAP_OPS > -static bool __unpin_devmap_managed_user_page(struct page *page) > -{ > - int count, refs = 1; > - > - if (!page_is_devmap_managed(page)) > - return false; > - > - if (hpage_pincount_available(page)) > - hpage_pincount_sub(page, 1); > - else > - refs = GUP_PIN_COUNTING_BIAS; > - > - count = page_ref_sub_return(page, refs); > - > - mod_node_page_state(page_pgdat(page), NR_FOLL_PIN_RELEASED, 1); > - /* > - * devmap page refcounts are 1-based, rather than 0-based: if > - * refcount is 1, then the page is free and the refcount is > - * stable because nobody holds a reference on the page. > - */ > - if (count == 1) > - free_devmap_managed_page(page); > - else if (!count) > - __put_page(page); > - > - return true; > -} > -#else > -static bool __unpin_devmap_managed_user_page(struct page *page) > -{ > - return false; > -} > -#endif /* CONFIG_DEV_PAGEMAP_OPS */ > - Wow, getting rid of that duplication is beautiful! thanks, -- John Hubbard NVIDIA > /** > * unpin_user_page() - release a dma-pinned page > * @page: pointer to page to be released > @@ -223,28 +210,7 @@ static bool __unpin_devmap_managed_user_page(struct page *page) > */ > void unpin_user_page(struct page *page) > { > - int refs = 1; > - > - page = compound_head(page); > - > - /* > - * For devmap managed pages we need to catch refcount transition from > - * GUP_PIN_COUNTING_BIAS to 1, when refcount reach one it means the > - * page is free and we need to inform the device driver through > - * callback. See include/linux/memremap.h and HMM for details. > - */ > - if (__unpin_devmap_managed_user_page(page)) > - return; > - > - if (hpage_pincount_available(page)) > - hpage_pincount_sub(page, 1); > - else > - refs = GUP_PIN_COUNTING_BIAS; > - > - if (page_ref_sub_and_test(page, refs)) > - __put_page(page); > - > - mod_node_page_state(page_pgdat(page), NR_FOLL_PIN_RELEASED, 1); > + put_compound_head(compound_head(page), 1, FOLL_PIN); > } > EXPORT_SYMBOL(unpin_user_page); > > @@ -2062,29 +2028,6 @@ EXPORT_SYMBOL(get_user_pages_unlocked); > * This code is based heavily on the PowerPC implementation by Nick Piggin. > */ > #ifdef CONFIG_HAVE_FAST_GUP > - > -static void put_compound_head(struct page *page, int refs, unsigned int flags) > -{ > - if (flags & FOLL_PIN) { > - mod_node_page_state(page_pgdat(page), NR_FOLL_PIN_RELEASED, > - refs); > - > - if (hpage_pincount_available(page)) > - hpage_pincount_sub(page, refs); > - else > - refs *= GUP_PIN_COUNTING_BIAS; > - } > - > - VM_BUG_ON_PAGE(page_ref_count(page) < refs, page); > - /* > - * Calling put_page() for each ref is unnecessarily slow. Only the last > - * ref needs a put_page(). > - */ > - if (refs > 1) > - page_ref_sub(page, refs - 1); > - put_page(page); > -} > - > #ifdef CONFIG_GUP_GET_PTE_LOW_HIGH > > /* >