From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 7E4A3C07E9D for ; Tue, 27 Sep 2022 16:28:00 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 15F888E00DE; Tue, 27 Sep 2022 12:28:00 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 10FB18E00C1; Tue, 27 Sep 2022 12:28:00 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EC9E38E00DE; Tue, 27 Sep 2022 12:27:59 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id D88808E00C1 for ; Tue, 27 Sep 2022 12:27:59 -0400 (EDT) Received: from smtpin18.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 84D9FAADA9 for ; Tue, 27 Sep 2022 16:27:59 +0000 (UTC) X-FDA: 79958397078.18.3B822C5 Received: from mail-pf1-f182.google.com (mail-pf1-f182.google.com [209.85.210.182]) by imf22.hostedemail.com (Postfix) with ESMTP id 11246C001F for ; Tue, 27 Sep 2022 16:27:58 +0000 (UTC) Received: by mail-pf1-f182.google.com with SMTP id c198so10111475pfc.13 for ; Tue, 27 Sep 2022 09:27:58 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date; bh=qtDTgqvL6bCy2no1plZzpxkuhtLBIBb+n3/hplN/Z/Y=; b=FzFtQXjj92j3O/H1zVoIJZp8ly+GIWZRl3xTccpG09pEOnFdvbMOlsIcUIBeNI/v3B UD1S8mWZ2yXBrb3n6jgJSt0E5RF+gXqgGtuh86h2BpC3F3FRMrrWsXamK9xElcwQqols jPrnfjL1eCfd1DqzvEYirEyTIVoeW5jBow7P94Ini/tlROY4ZMG0DYedWqT3RcRe+iui +It/6Wdgz5o7ZszruRcP1toM0eek1btHojwSLVbC4aUpskYK4HHxV8x1NH+R9xk+kX6Q EaYJWFpB9iayD9ALgD3Mh1qmX+HqtmsGlOhWtft/GRaOkkauRHPGjUXzF7Uy1ap1K/AG nPhQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date; bh=qtDTgqvL6bCy2no1plZzpxkuhtLBIBb+n3/hplN/Z/Y=; b=kSWDm1dF8lu3qvEbLiCRZ/b0fSLykFMK+E9HHZsc6Bu9S2CZXyvkwaQ4f1R5cMsuux p4Zn7EwrnjmQdA5b/gUHs09EY/JhwyUhlAnIV8nePT4uaX0iLvrairrbjBOk+XArGqHo SFN7OVn1Kp6hlDMpeW/fCSmrX6Frfs+9/MENT6d5LrgZLjqphSXXVQBHGzQQNI595+ww zpzsctuxDTIUBDo7CIHJ1Vtu+rED6DIZMkFrrBrIEBTrN+aFnJzhTGfYe4OYVoRFf6O/ BUhyV4uQiTCg8oCUCA3N+S6rOJ9w6TYEjS3/aiIkQkIoyJb5YtLHdWWmiU3MaaJeW5ZF JT0Q== X-Gm-Message-State: ACrzQf1B32MlzGWqnEf7eADtTqcAkE4XlBdX/tlT/2G6E2vrK+SWJHg/ cyvW7a/9EINqvQeF90uw6rQ= X-Google-Smtp-Source: AMsMyM76qYcRgHJgPAFLzh39+ye4Tgr8AF4QcPQDirN61vSKKf/FpRs4uUf3Owokdc+QLh8yo/Czvg== X-Received: by 2002:a63:500d:0:b0:439:dcdd:87af with SMTP id e13-20020a63500d000000b00439dcdd87afmr25634968pgb.231.1664296078073; Tue, 27 Sep 2022 09:27:58 -0700 (PDT) Received: from archlinux.localdomain ([140.121.198.213]) by smtp.googlemail.com with ESMTPSA id 9-20020a17090a0f0900b001f333fab3d6sm8602360pjy.18.2022.09.27.09.27.52 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 27 Sep 2022 09:27:57 -0700 (PDT) From: Chih-En Lin To: Andrew Morton , Qi Zheng , David Hildenbrand , Matthew Wilcox , Christophe Leroy Cc: linux-kernel@vger.kernel.org, linux-mm@kvack.org, Luis Chamberlain , Kees Cook , Iurii Zaikin , Vlastimil Babka , William Kucharski , "Kirill A . Shutemov" , Peter Xu , Suren Baghdasaryan , Arnd Bergmann , Tong Tiangen , Pasha Tatashin , Li kunyu , Nadav Amit , Anshuman Khandual , Minchan Kim , Yang Shi , Song Liu , Miaohe Lin , Thomas Gleixner , Sebastian Andrzej Siewior , Andy Lutomirski , Fenghua Yu , Dinglan Peng , Pedro Fonseca , Jim Huang , Huichun Feng , Chih-En Lin Subject: [RFC PATCH v2 4/9] mm: Add COW PTE fallback functions Date: Wed, 28 Sep 2022 00:29:52 +0800 Message-Id: <20220927162957.270460-5-shiyn.lin@gmail.com> X-Mailer: git-send-email 2.37.3 In-Reply-To: <20220927162957.270460-1-shiyn.lin@gmail.com> References: <20220927162957.270460-1-shiyn.lin@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit ARC-Authentication-Results: i=1; imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=FzFtQXjj; spf=pass (imf22.hostedemail.com: domain of shiyn.lin@gmail.com designates 209.85.210.182 as permitted sender) smtp.mailfrom=shiyn.lin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1664296079; a=rsa-sha256; cv=none; b=Ch5hnT84ON8tOUlpFluJ3LArFYgxEa2NelSLoVWe7nZymu+CRu/xt1E8FB8vobSaOiwSZH hLWHnQ1TxXNL8qGlYVDRKMsL4oAK2AG2zXK/xcpHjD4QJvYE4Fxoas3OosB1es2xOiyGAw Xmwe83ttv3LTgvVn2cKd6XzlHpEJ1os= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1664296079; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=qtDTgqvL6bCy2no1plZzpxkuhtLBIBb+n3/hplN/Z/Y=; b=VkQixkqEVY+TOhr4Nj7x0sSp1OklfnnZXJxbMLYVam1zR7OEQIchUeMDg2KnMj/r5XpFP1 XtwXXsTfnvAndwxSEcni9zSHbc+6hoz0KxvahBDKaaPe2Y9hAIcza90xYc/HMQnu+iwQGi 6r3iROyY0Uwd2eZkzBz3nVehltKk0ks= X-Stat-Signature: dbp663qchwje3zzqexa3we76uqrwp4zy X-Rspamd-Queue-Id: 11246C001F Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=FzFtQXjj; spf=pass (imf22.hostedemail.com: domain of shiyn.lin@gmail.com designates 209.85.210.182 as permitted sender) smtp.mailfrom=shiyn.lin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspam-User: X-Rspamd-Server: rspam07 X-HE-Tag: 1664296078-138394 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The lifetime of COWed PTE table will handle by a reference count. When the process wants to write the COWed PTE table, which refcount is 1, it will reuse the shared PTE. Since only the owner will update their page table state. the fallback function also needs to handle the situation of non-owner COWed PTE table fallback to normal PTE. This commit prepares for the following implementation of the reference count for COW PTE. Signed-off-by: Chih-En Lin --- include/linux/pgtable.h | 3 ++ mm/memory.c | 93 +++++++++++++++++++++++++++++++++++++++++ 2 files changed, 96 insertions(+) diff --git a/include/linux/pgtable.h b/include/linux/pgtable.h index 9dca787a3f4dd..25c1e5c42fdf3 100644 --- a/include/linux/pgtable.h +++ b/include/linux/pgtable.h @@ -615,6 +615,9 @@ static inline int pte_unused(pte_t pte) } #endif +void cow_pte_fallback(struct vm_area_struct *vma, pmd_t *pmd, + unsigned long addr); + static inline void set_cow_pte_owner(pmd_t *pmd, pmd_t *owner) { smp_store_release(&pmd_page(*pmd)->cow_pte_owner, owner); diff --git a/mm/memory.c b/mm/memory.c index 4ba73f5aa8bb7..d29f84801f3cd 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -509,6 +509,37 @@ static inline void add_mm_rss_vec(struct mm_struct *mm, int *rss) add_mm_counter(mm, i, rss[i]); } +static void cow_pte_rss(struct mm_struct *mm, struct vm_area_struct *vma, + pmd_t *pmdp, unsigned long addr, + unsigned long end, bool inc_dec) +{ + int rss[NR_MM_COUNTERS]; + spinlock_t *ptl; + pte_t *orig_ptep, *ptep; + struct page *page; + + init_rss_vec(rss); + + ptep = pte_offset_map_lock(mm, pmdp, addr, &ptl); + orig_ptep = ptep; + arch_enter_lazy_mmu_mode(); + do { + if (pte_none(*ptep)) + continue; + + page = vm_normal_page(vma, addr, *ptep); + if (page) { + if (inc_dec) + rss[mm_counter(page)]++; + else + rss[mm_counter(page)]--; + } + } while (ptep++, addr += PAGE_SIZE, addr != end); + arch_leave_lazy_mmu_mode(); + pte_unmap_unlock(orig_ptep, ptl); + add_mm_rss_vec(mm, rss); +} + /* * This function is called to print an error when a bad pte * is found. For example, we might have a PFN-mapped pte in @@ -2817,6 +2848,68 @@ int apply_to_existing_page_range(struct mm_struct *mm, unsigned long addr, } EXPORT_SYMBOL_GPL(apply_to_existing_page_range); +/** + * cow_pte_fallback - reuse the shared PTE table + * @vma: vma that coever the shared PTE table + * @pmd: pmd index maps to the shared PTE table + * @addr: the address trigger the break COW, + * + * Reuse the shared (COW) PTE table when the refcount is equal to one. + * @addr needs to be in the range of the shared PTE table that @vma and + * @pmd mapped to it. + * + * COW PTE fallback to normal PTE: + * - two state here + * - After break child : [parent, rss=1, ref=1, write=NO , owner=parent] + * to [parent, rss=1, ref=1, write=YES, owner=NULL ] + * - After break parent: [child , rss=0, ref=1, write=NO , owner=NULL ] + * to [child , rss=1, ref=1, write=YES, owner=NULL ] + */ +void cow_pte_fallback(struct vm_area_struct *vma, pmd_t *pmd, + unsigned long addr) +{ + struct mm_struct *mm = vma->vm_mm; + struct vm_area_struct *prev = vma->vm_prev; + struct vm_area_struct *next = vma->vm_next; + unsigned long start, end; + pmd_t new; + + VM_WARN_ON(pmd_write(*pmd)); + + start = addr & PMD_MASK; + end = (addr + PMD_SIZE) & PMD_MASK; + + /* + * If pmd is not owner, it needs to increase the rss. + * Since only the owner has the RSS state for the COW PTE. + */ + if (!cow_pte_owner_is_same(pmd, pmd)) { + /* The part of address range is covered by previous. */ + if (start < vma->vm_start && prev && start < prev->vm_end) { + cow_pte_rss(mm, prev, pmd, + start, prev->vm_end, true /* inc */); + start = vma->vm_start; + } + /* The part of address range is covered by next. */ + if (end > vma->vm_end && next && end > next->vm_start) { + cow_pte_rss(mm, next, pmd, + next->vm_start, end, true /* inc */); + end = vma->vm_end; + } + cow_pte_rss(mm, vma, pmd, start, end, true /* inc */); + + mm_inc_nr_ptes(mm); + /* Memory barrier here is the same as pmd_install(). */ + smp_wmb(); + pmd_populate(mm, pmd, pmd_page(*pmd)); + } + + /* Reuse the pte page */ + set_cow_pte_owner(pmd, NULL); + new = pmd_mkwrite(*pmd); + set_pmd_at(mm, addr, pmd, new); +} + /* * handle_pte_fault chooses page fault handler according to an entry which was * read non-atomically. Before making any commitment, on those architectures -- 2.37.3