From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B252C54EE9 for ; Tue, 27 Sep 2022 16:28:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 99D6B8E00DF; Tue, 27 Sep 2022 12:28:05 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 94BE18E00C1; Tue, 27 Sep 2022 12:28:05 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7C5288E00DF; Tue, 27 Sep 2022 12:28:05 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 6A5AB8E00C1 for ; Tue, 27 Sep 2022 12:28:05 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 1C82B140230 for ; Tue, 27 Sep 2022 16:28:05 +0000 (UTC) X-FDA: 79958397330.14.3FD5519 Received: from mail-pf1-f181.google.com (mail-pf1-f181.google.com [209.85.210.181]) by imf24.hostedemail.com (Postfix) with ESMTP id A775F18001C for ; Tue, 27 Sep 2022 16:28:04 +0000 (UTC) Received: by mail-pf1-f181.google.com with SMTP id a80so10150768pfa.4 for ; Tue, 27 Sep 2022 09:28:04 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=XuRjx7DiMyFErohU6l5P8iRkgXDYC8Q+F8gmrtJ+ilo=; b=BUZqiRpZc+y9cuKG56oBy+Cfa11WNSqYjMXnosAmPhlk/lSKqYHn4hBSpj/jIf58w5 I9qqOV/xm+s71mwq9zjCCSYffCPx9a50VMkrfSsqXCK+mN/b4e6hEOlKOolaJUTLxIqC VwRC3b9iS7HmqV77y7pGxVVKSrzAOhQgEcl/7dch2nR+pKj1CxC+vlFOZChPdDepcxnv UFDQMokXHNYITZHeIoOXNkO4NIDhzmsx66zSvT78H/d0eIndhhGEjIwAVklKSYQZiD/V 6H7xjsp1lbsiNpMH3hkT8bqlX2SYf2GrGg3f8YRsP/r9wEive/ykWyXiTUmWSniup8Hj W8cQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=XuRjx7DiMyFErohU6l5P8iRkgXDYC8Q+F8gmrtJ+ilo=; b=E0lbqMc3GFNT/VO4mbGSSP//v/sQDftH6PmP3+s+jUYPpfUowebBi+RjoJT2juraUE l8ult9SDrdmitBC8uby0XMxwQC2RP2F25+LbnuYn3HB+nUSeyvcc01Pj2ruMDM2PRDG8 SESvEqeNP2S5vm8k3ZrUoyIIv0BFBo9FSwRZuVcYVvbBWlrmxfy2JWeLPd1OKWHiAT6K F7TL7XDHukAav5jlqlvmNGnHQuDw/bDofIwONhd75wJ6upWrpWaMCZev0SXpN9uSum0A BfCBDJJdW9TR+BgIVHiC5yqWc9chgHMQZBB0KTwYS265gnHMt8GcvRjLQ3CaYYEXxIru MUvQ== X-Gm-Message-State: ACrzQf3kELywK17x9utT6bqP3lu2wWzUD91tBKVilXWGrijB0dVWYzf3 QSGIXaVdc3px2h21v4gprzs= X-Google-Smtp-Source: AMsMyM4GbFT0PuBeQyFV0l8Zleho1UdvgIIpjeUqz6KjGWEv03YsI5DcOCows8rB9mxqR/VT7zNmkg== X-Received: by 2002:a05:6a00:1253:b0:546:3d50:3284 with SMTP id u19-20020a056a00125300b005463d503284mr29961879pfi.72.1664296083652; Tue, 27 Sep 2022 09:28:03 -0700 (PDT) Received: from archlinux.localdomain ([140.121.198.213]) by smtp.googlemail.com with ESMTPSA id 9-20020a17090a0f0900b001f333fab3d6sm8602360pjy.18.2022.09.27.09.27.58 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Sep 2022 09:28:02 -0700 (PDT) From: Chih-En Lin To: Andrew Morton , Qi Zheng , David Hildenbrand , Matthew Wilcox , Christophe Leroy Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Luis Chamberlain , Kees Cook , Iurii Zaikin , Vlastimil Babka , William Kucharski , "Kirill A . Shutemov" , Peter Xu , Suren Baghdasaryan , Arnd Bergmann , Tong Tiangen , Pasha Tatashin , Li kunyu , Nadav Amit , Anshuman Khandual , Minchan Kim , Yang Shi , Song Liu , Miaohe Lin , Thomas Gleixner , Sebastian Andrzej Siewior , Andy Lutomirski , Fenghua Yu , Dinglan Peng , Pedro Fonseca , Jim Huang , Huichun Feng , Chih-En Lin Subject: [RFC PATCH v2 5/9] mm, pgtable: Add a refcount to PTE table Date: Wed, 28 Sep 2022 00:29:53 +0800 Message-Id: <20220927162957.270460-6-shiyn.lin@gmail.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220927162957.270460-1-shiyn.lin@gmail.com> References: <20220927162957.270460-1-shiyn.lin@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit ARC-Authentication-Results: i=1; imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=BUZqiRpZ; spf=pass (imf24.hostedemail.com: domain of shiyn.lin@gmail.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=shiyn.lin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1664296084; a=rsa-sha256; cv=none; b=RPfy1vMpNZB9pHye9KRZfXOYSRbxFgRawZ9p0Rsfpt4yy69YmK7oTVNkozX3CrOPfsJrBA UVG+7VHRIRsssNDdyjwCYas2lAUHvimEiTDkW+KnBTERrLa3G1yU6ECXIUC7XSIwFsu0HZ Wv/OUl6ppkUIfDI+zd79C7N34u7JoEA= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1664296084; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=XuRjx7DiMyFErohU6l5P8iRkgXDYC8Q+F8gmrtJ+ilo=; b=loKKTN920lb6UErk/G6q69ofusuMckl2rNY4A7UcDBYeTdNrVqKZTnKGU2YMtXQOzQdY+W lGXu4Pc/u9IjmKcHj73XBbGN7AtgwLHhy7SxnmdYiI++GHfl+JiLuLJZjTN8KltjgMKuzQ p6LapEKr3iXfkxoWJtQDN0vzbT5xCnw= X-Rspamd-Server: rspam11 X-Rspamd-Queue-Id: A775F18001C X-Rspam-User: Authentication-Results: imf24.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=BUZqiRpZ; spf=pass (imf24.hostedemail.com: domain of shiyn.lin@gmail.com designates 209.85.210.181 as permitted sender) smtp.mailfrom=shiyn.lin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Stat-Signature: jzpeq3wkghgoe5a3ybyffp7mfzq37fx8 X-HE-Tag: 1664296084-788014 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Reuse the _refcount in struct page for the page table to maintain the number of process references to COWed PTE table. Before decreasing the refcount, it will check whether refcount is one or not for reusing shared PTE table. Signed-off-by: Chih-En Lin --- include/linux/mm.h | 1 + include/linux/pgtable.h | 28 ++++++++++++++++++++++++++++ mm/memory.c | 1 + 3 files changed, 30 insertions(+) diff --git a/include/linux/mm.h b/include/linux/mm.h index 965523dcca3b8..bfe6a8c7ab9ed 100644 --- a/include/linux/mm.h +++ b/include/linux/mm.h @@ -2290,6 +2290,7 @@ static inline bool pgtable_pte_page_ctor(struct page *page) __SetPageTable(page); inc_lruvec_page_state(page, NR_PAGETABLE); page->cow_pte_owner = NULL; + set_page_count(page, 1); return true; } diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 25c1e5c42fdf3..8b497d7d800ed 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -9,6 +9,7 @@ #ifdef CONFIG_MMU #include +#include #include #include #include @@ -628,6 +629,33 @@ static inline bool cow_pte_owner_is_same(pmd_t *pmd, pmd_t *owner) return smp_load_acquire(&pmd_page(*pmd)->cow_pte_owner) == owner; } +static inline int pmd_get_pte(pmd_t *pmd) +{ + return page_ref_inc_return(pmd_page(*pmd)); +} + +/* + * If the COW PTE refcount is 1, instead of decreasing the counter, + * clear write protection of the corresponding PMD entry and reset + * the COW PTE owner to reuse the table. + * But if the reuse parameter is false, do not thing. This help us + * to handle the situation that PTE table we already handled. + */ +static inline int pmd_put_pte(struct vm_area_struct *vma, pmd_t *pmd, + unsigned long addr, bool reuse) +{ + if (!page_ref_add_unless(pmd_page(*pmd), -1, 1) && reuse) { + cow_pte_fallback(vma, pmd, addr); + return 1; + } + return 0; +} + +static inline int cow_pte_count(pmd_t *pmd) +{ + return page_count(pmd_page(*pmd)); +} + #ifndef pte_access_permitted #define pte_access_permitted(pte, write) \ (pte_present(pte) && (!(write) || pte_write(pte))) diff --git a/mm/memory.c b/mm/memory.c index d29f84801f3cd..3e66e229f4169 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -2875,6 +2875,7 @@ void cow_pte_fallback(struct vm_area_struct *vma, pmd_t *pmd, pmd_t new; VM_WARN_ON(pmd_write(*pmd)); + VM_WARN_ON(cow_pte_count(pmd) != 1); start = addr & PMD_MASK; end = (addr + PMD_SIZE) & PMD_MASK; -- 2.37.3