From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB016C433F5 for ; Thu, 19 May 2022 18:30:33 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 518236B0078; Thu, 19 May 2022 14:30:33 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4A0A76B007B; Thu, 19 May 2022 14:30:33 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 31B6D6B007D; Thu, 19 May 2022 14:30:33 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 245C46B0078 for ; Thu, 19 May 2022 14:30:33 -0400 (EDT) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id DF79B335FB for ; Thu, 19 May 2022 18:30:32 +0000 (UTC) X-FDA: 79483333104.07.FD50D72 Received: from mail-pf1-f174.google.com (mail-pf1-f174.google.com [209.85.210.174]) by imf22.hostedemail.com (Postfix) with ESMTP id A3780C00F0 for ; Thu, 19 May 2022 18:30:28 +0000 (UTC) Received: by mail-pf1-f174.google.com with SMTP id j6so5813573pfe.13 for ; Thu, 19 May 2022 11:30:30 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=from:to:cc:subject:date:message-id:in-reply-to:references :mime-version:content-transfer-encoding; bh=0GM+Jpl2PB5bL1v4oYzWVj7sToezUiR+DzB0/ofcA2U=; b=edalsKs+cr1bHh8k68c1cKtEAXiGwPDe+9mwnH4RI6wby9g93BgMBhEo2FvH1BiNE/ Rf+mHA2Avzet8LIPH8/gei2VwN5rRzg5NjmV0iOB/d33JAiwPchFJ9a5YyeQop5KvfBJ xFGjMQxMBs6fyOWtImOPcl8yz553+6++yGTZA8+IjrC73jMjrs8/l1KiPB0fSA/9IloT AxTnr8x+VXzunejPPM1Z5lNAGMyCaRpIAe5ngyNUIXVqKjwip9YS5xFhkzU/n5rYFmBI uf17WAO9nDvdwAl18G8ppoKLB/3TsZv110JHXS4noK0X0hhxJcxYWuQuROcuw2ohusip LgKw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:from:to:cc:subject:date:message-id:in-reply-to :references:mime-version:content-transfer-encoding; bh=0GM+Jpl2PB5bL1v4oYzWVj7sToezUiR+DzB0/ofcA2U=; b=NX2wV+Os1MQznNy5A7avipQqX5YSNqhbvubEUwsxQr8pFNMN7Xuk93q7aL2PbW4U/Z NBXKm+fb6jG25dK5/tEz/iAggm2W0gOmBn7Z2pCc3d+KsY6O8p78UtjNM0rGbHxEScH8 RTqAPB3jfEFqjfn8IupC7HyvGTK0TntAWO5391ue4C1IIK6AtS8l0ONJIvKhww8a+tbP O8AN3jtnulevXtxvRO1Tot8+M8gPcgbB/9834hAPEiphIOlc12fN4USgRsQdaqbngOQy IXniFrjcfM9OHDSJPzrmEa6j0EPVzxDRFUpz0eDK1W2NugcoGO3JpMGpliTKzf1FjsvK fKMg== X-Gm-Message-State: AOAM5309Y+47bmR6LrQvKTjDC7bkDj7Qedfdwkrjfhpc1qQNAevJbaRr rb9kRxX2/1rQ+/ggwuU9bBA= X-Google-Smtp-Source: ABdhPJwd4x1e0UIjlmbwBKk07N79TOTNwuw1JTWaurKAs+KJuC4ETrijeUFGJjgHf84Ek3MdEHyQLQ== X-Received: by 2002:a65:6d08:0:b0:3c6:8a08:3b9f with SMTP id bf8-20020a656d08000000b003c68a083b9fmr5013534pgb.147.1652985030008; Thu, 19 May 2022 11:30:30 -0700 (PDT) Received: from archlinux.localdomain ([140.121.198.213]) by smtp.googlemail.com with ESMTPSA id z5-20020a63e105000000b003c14af505f6sm3884674pgh.14.2022.05.19.11.30.24 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 19 May 2022 11:30:29 -0700 (PDT) From: Chih-En Lin To: Andrew Morton , linux-mm@kvack.org Cc: Ingo Molnar , Peter Zijlstra , Juri Lelli , Vincent Guittot , Dietmar Eggemann , Steven Rostedt , Ben Segall , Mel Gorman , Daniel Bristot de Oliveira , Christian Brauner , "Matthew Wilcox (Oracle)" , Vlastimil Babka , William Kucharski , John Hubbard , Yunsheng Lin , Arnd Bergmann , Suren Baghdasaryan , Chih-En Lin , Colin Cross , Feng Tang , "Eric W. Biederman" , Mike Rapoport , Geert Uytterhoeven , Anshuman Khandual , "Aneesh Kumar K.V" , Daniel Axtens , Jonathan Marek , Christophe Leroy , Pasha Tatashin , Peter Xu , Andrea Arcangeli , Thomas Gleixner , Andy Lutomirski , Sebastian Andrzej Siewior , Fenghua Yu , David Hildenbrand , linux-kernel@vger.kernel.org, Kaiyang Zhao , Huichun Feng , Jim Huang Subject: [RFC PATCH 4/6] mm: Add COW PTE fallback function Date: Fri, 20 May 2022 02:31:25 +0800 Message-Id: <20220519183127.3909598-5-shiyn.lin@gmail.com> X-Mailer: git-send-email 2.36.1 In-Reply-To: <20220519183127.3909598-1-shiyn.lin@gmail.com> References: <20220519183127.3909598-1-shiyn.lin@gmail.com> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Stat-Signature: wz3z7ite6y9fkc17746cju6y1r1mu53y Authentication-Results: imf22.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=edalsKs+; spf=pass (imf22.hostedemail.com: domain of shiyn.lin@gmail.com designates 209.85.210.174 as permitted sender) smtp.mailfrom=shiyn.lin@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspam-User: X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: A3780C00F0 X-HE-Tag: 1652985028-288925 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The lifetime of COW PTE will handle by ownership and a reference count. When the process wants to write the COW PTE, which reference count is 1, it will reuse the COW PTE instead of copying then free. Only the owner will update its RSS state and the record of page table bytes allocation. So we need to handle when the non-owner process gets the fallback COW PTE. This commit prepares for the following implementation of the reference count for COW PTE. Signed-off-by: Chih-En Lin --- mm/memory.c | 66 +++++++++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 66 insertions(+) diff --git a/mm/memory.c b/mm/memory.c index 76e3af9639d9..dcb678cbb051 100644 --- a/mm/memory.c +++ b/mm/memory.c @@ -1000,6 +1000,34 @@ page_copy_prealloc(struct mm_struct *src_mm, struct vm_area_struct *vma, return new_page; } +static inline void cow_pte_rss(struct mm_struct *mm, struct vm_area_struct *vma, + pmd_t *pmdp, unsigned long addr, unsigned long end, bool inc_dec) +{ + int rss[NR_MM_COUNTERS]; + pte_t *orig_ptep, *ptep; + struct page *page; + + init_rss_vec(rss); + + ptep = pte_offset_map(pmdp, addr); + orig_ptep = ptep; + arch_enter_lazy_mmu_mode(); + do { + if (pte_none(*ptep) || pte_special(*ptep)) + continue; + + page = vm_normal_page(vma, addr, *ptep); + if (page) { + if (inc_dec) + rss[mm_counter(page)]++; + else + rss[mm_counter(page)]--; + } + } while (ptep++, addr += PAGE_SIZE, addr != end); + arch_leave_lazy_mmu_mode(); + add_mm_rss_vec(mm, rss); +} + static int copy_pte_range(struct vm_area_struct *dst_vma, struct vm_area_struct *src_vma, pmd_t *dst_pmd, pmd_t *src_pmd, unsigned long addr, @@ -4554,6 +4582,44 @@ static vm_fault_t wp_huge_pud(struct vm_fault *vmf, pud_t orig_pud) return VM_FAULT_FALLBACK; } +/* COW PTE fallback to normal PTE: + * - two state here + * - After break child : [parent, rss=1, ref=1, write=NO , owner=parent] + * to [parent, rss=1, ref=1, write=YES, owner=NULL ] + * - After break parent: [child , rss=0, ref=1, write=NO , owner=NULL ] + * to [child , rss=1, ref=1, write=YES, owner=NULL ] + */ +void cow_pte_fallback(struct vm_area_struct *vma, pmd_t *pmd, + unsigned long addr) +{ + struct mm_struct *mm = vma->vm_mm; + unsigned long start, end; + pmd_t new; + + BUG_ON(pmd_write(*pmd)); + + start = addr & PMD_MASK; + end = (addr + PMD_SIZE) & PMD_MASK; + + /* If pmd is not owner, it needs to increase the rss. + * Since only the owner has the RSS state for the COW PTE. + */ + if (!cow_pte_owner_is_same(pmd, pmd)) { + cow_pte_rss(mm, vma, pmd, start, end, true /* inc */); + mm_inc_nr_ptes(mm); + smp_wmb(); + pmd_populate(mm, pmd, pmd_page(*pmd)); + } + + /* Reuse the pte page */ + set_cow_pte_owner(pmd, NULL); + new = pmd_mkwrite(*pmd); + set_pmd_at(mm, addr, pmd, new); + + BUG_ON(!pmd_write(*pmd)); + BUG_ON(pmd_page(*pmd)->cow_pte_owner); +} + /* * These routines also need to handle stuff like marking pages dirty * and/or accessed for architectures that don't do it in hardware (most -- 2.36.1