From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 92DA9C433EF for ; Tue, 26 Jul 2022 12:43:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EA27A8E0002; Tue, 26 Jul 2022 08:43:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id E51E58E0001; Tue, 26 Jul 2022 08:43:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D41098E0002; Tue, 26 Jul 2022 08:43:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id C46D88E0001 for ; Tue, 26 Jul 2022 08:43:34 -0400 (EDT) Received: from smtpin01.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id 93548AB191 for ; Tue, 26 Jul 2022 12:43:34 +0000 (UTC) X-FDA: 79729217148.01.DAF5912 Received: from szxga03-in.huawei.com (szxga03-in.huawei.com [45.249.212.189]) by imf06.hostedemail.com (Postfix) with ESMTP id AF65B180093 for ; Tue, 26 Jul 2022 12:43:32 +0000 (UTC) Received: from dggemv704-chm.china.huawei.com (unknown [172.30.72.55]) by szxga03-in.huawei.com (SkyGuard) with ESMTP id 4Lsc4R72Ybz9svw; Tue, 26 Jul 2022 20:42:15 +0800 (CST) Received: from kwepemm600010.china.huawei.com (7.193.23.86) by dggemv704-chm.china.huawei.com (10.3.19.47) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Tue, 26 Jul 2022 20:43:16 +0800 Received: from huawei.com (10.174.179.164) by kwepemm600010.china.huawei.com (7.193.23.86) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.2375.24; Tue, 26 Jul 2022 20:43:16 +0800 From: Liu Zixian To: , , CC: , Subject: [PATCH] shmem: support huge_fault to avoid pmd split Date: Tue, 26 Jul 2022 20:43:15 +0800 Message-ID: <20220726124315.1606-1-liuzixian4@huawei.com> X-Mailer: git-send-email 2.29.2.windows.3 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit Content-Type: text/plain X-Originating-IP: [10.174.179.164] X-ClientProxiedBy: dggems706-chm.china.huawei.com (10.3.19.183) To kwepemm600010.china.huawei.com (7.193.23.86) X-CFilter-Loop: Reflected ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1658839413; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding:in-reply-to: references; bh=yapSewnizxZleG5cixP/KKNavx+NePsx1qu7tV6uRLk=; b=TmLSuvfrCUvbqCat2bp2Kzx/FXjHCVSn2UbntOXjEJGAr5C7IC94QvD/7OzfmoeJjQfHP8 eHwqqIi3kkP0+Ox/tWq+mb+02uZvp2FIX70Cx1Kn/hnoDc2axrinRMsVs5/1l5MrkV3k2i KROL4oQzDx6T9sZPvtCtE5j4qe0yPsc= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1658839413; a=rsa-sha256; cv=none; b=CC4jIjrt0Bntzjv7yJBRY+dGit2EnLlGyc5tSuihtTJN2GK0gOQOfGp4Hx5heqwyeZe2/z jvka7VMxg8/tSyaKn20Y5YFfkl1EVeNxQPIvzLQG8SGdgiHET3rcooydMcPHZ3EIB82TK+ fBPYpJ9X3L1nmDSXFs7NeqBdDnrBSzo= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=none; spf=pass (imf06.hostedemail.com: domain of liuzixian4@huawei.com designates 45.249.212.189 as permitted sender) smtp.mailfrom=liuzixian4@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: AF65B180093 Authentication-Results: imf06.hostedemail.com; dkim=none; spf=pass (imf06.hostedemail.com: domain of liuzixian4@huawei.com designates 45.249.212.189 as permitted sender) smtp.mailfrom=liuzixian4@huawei.com; dmarc=pass (policy=quarantine) header.from=huawei.com X-Rspam-User: X-Stat-Signature: np8qytx4q9nksweh4p9oxximrmiqs95m X-HE-Tag: 1658839412-239202 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Transparent hugepage of tmpfs is useful to improve TLB miss, but it will be split during cow memory fault. This will happen if we mprotect and rewrite code segment (which is private file map) to hotpatch a running process. We can avoid the splitting by adding a huge_fault function. Signed-off-by: Liu Zixian --- mm/shmem.c | 46 ++++++++++++++++++++++++++++++++++++++++++++++ 1 file changed, 46 insertions(+) diff --git a/mm/shmem.c b/mm/shmem.c index a6f565308..12b2b5140 100644 --- a/mm/shmem.c +++ b/mm/shmem.c @@ -2120,6 +2120,51 @@ static vm_fault_t shmem_fault(struct vm_fault *vmf) return ret; } +static vm_fault_t shmem_huge_fault(struct vm_fault *vmf, enum page_entry_size pe_size) +{ + vm_fault_t ret = VM_FAULT_FALLBACK; + unsigned long haddr = vmf->address & HPAGE_PMD_MASK; + struct page *old_page, *new_page; + int gfp_flags = GFP_HIGHUSER_MOVABLE | __GFP_COMP; + + /* read or shared fault will not split huge pmd */ + if (!(vmf->flags & FAULT_FLAG_WRITE) + || (vmf->vma->vm_flags & VM_SHARED)) + return VM_FAULT_FALLBACK; + if (pe_size != PE_SIZE_PMD) + return VM_FAULT_FALLBACK; + + if (pmd_none(*vmf->pmd)) { + if (shmem_fault(vmf) & VM_FAULT_ERROR) + goto out; + if (!PageTransHuge(vmf->page)) + goto out; + old_page = vmf->page; + } else { + old_page = pmd_page(*vmf->pmd); + page_remove_rmap(old_page, vmf->vma, true); + pmdp_huge_clear_flush(vmf->vma, haddr, vmf->pmd); + add_mm_counter(vmf->vma->vm_mm, MM_SHMEMPAGES, -HPAGE_PMD_NR); + } + + new_page = &vma_alloc_folio(gfp_flags, HPAGE_PMD_ORDER, + vmf->vma, haddr, true)->page; + if (!new_page) + goto out; + prep_transhuge_page(new_page); + copy_user_huge_page(new_page, old_page, haddr, vmf->vma, HPAGE_PMD_NR); + __SetPageUptodate(new_page); + + ret = do_set_pmd(vmf, new_page); + +out: + if (vmf->page) { + unlock_page(vmf->page); + put_page(vmf->page); + } + return ret; +} + unsigned long shmem_get_unmapped_area(struct file *file, unsigned long uaddr, unsigned long len, unsigned long pgoff, unsigned long flags) @@ -3884,6 +3929,7 @@ static const struct super_operations shmem_ops = { static const struct vm_operations_struct shmem_vm_ops = { .fault = shmem_fault, + .huge_fault = shmem_huge_fault, .map_pages = filemap_map_pages, #ifdef CONFIG_NUMA .set_policy = shmem_set_policy, -- 2.33.0