From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A540FC54798 for ; Tue, 27 Feb 2024 17:43:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B03996B00E4; Tue, 27 Feb 2024 12:43:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id A8A1B6B00E5; Tue, 27 Feb 2024 12:43:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4C6E06B00E6; Tue, 27 Feb 2024 12:43:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id ED8EB6B00E7 for ; Tue, 27 Feb 2024 12:43:03 -0500 (EST) Received: from smtpin06.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id CB9114061F for ; Tue, 27 Feb 2024 17:43:03 +0000 (UTC) X-FDA: 81838304646.06.04AE17F Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf29.hostedemail.com (Postfix) with ESMTP id 4F516120004 for ; Tue, 27 Feb 2024 17:43:01 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=DWWvRhmX; dmarc=none; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709055782; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=o3omPoxMLsk94BWO3XT1KRKpFrR6WgQB/H8HRcd+FzA=; b=w0xzkliL5KQ1bUZTDgtz7o+ZRPiwep1xkfFqhV1UZQcFRUM8zuVQfSvl+tcjmRs+/aD6P8 fKU8uW/0FH3Efx13kAGfTgndv/FgnmESl5i/n9trqOCGz0UJmztYZeOfT9Drg5wX68KFlF bTKD18ZtfwYxx39vj70EI8peRU30TFg= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=DWWvRhmX; dmarc=none; spf=none (imf29.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709055782; a=rsa-sha256; cv=none; b=yks94MIFf2CU1mHnqLuHUd+3zUXqM+uaDELYt8ipJ1T6chW0qcrw4+RIs8XOBOcqXZJV6L OsbeUebNjd64crU9TSVwfIEcYOrPC1jathyU2So6uCc8EzITGc4BTlic7W0qBk41qmh1LP 7DvOpdhI2zrdEWZCw3aMATRIgx+LMbA= DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-ID:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=o3omPoxMLsk94BWO3XT1KRKpFrR6WgQB/H8HRcd+FzA=; b=DWWvRhmXp5wd0D5DDCBoTMJ3G/ 0kebLqw3WdBr3SFIllqZw1ijZlx8KUItP0v4iuaDs5hJ6yRAfooueQN11p0kg//Vlxy/J4Nc4jw9S acIiBrRKuxosiqP7nlCJKPpYKoxa0/6cQBi6Kfb0bSDdPIxdARG6wFFCzBFKOH8vtpJ7RucUKdiZE LZdzS0EhBvIf9tMyT3ou8Q9qPpYNR/dfpTDJ042Qgam1zlJfi48Zl0EJ13zAIpOY+08Yn4gRs4KWE n+9r1XYOjuoSQIeVyS5LhpFRQRSZffZ15xK8gpgBhYepIIJmwNSVhTRDEbXrt8IRTHmjARNr/cfF2 H8xZOSYA==; Received: from willy by casper.infradead.org with local (Exim 4.97.1 #2 (Red Hat Linux)) id 1rf1Tn-00000002ysM-3NW2; Tue, 27 Feb 2024 17:42:59 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH v3 17/18] mm: Use a folio in __collapse_huge_page_copy_succeeded() Date: Tue, 27 Feb 2024 17:42:51 +0000 Message-ID: <20240227174254.710559-18-willy@infradead.org> X-Mailer: git-send-email 2.43.0 In-Reply-To: <20240227174254.710559-1-willy@infradead.org> References: <20240227174254.710559-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: 4F516120004 X-Rspam-User: X-Rspamd-Server: rspam02 X-Stat-Signature: q73ytyoqxywrxk9j93jizkfu3pj5pn3u X-HE-Tag: 1709055781-289822 X-HE-Meta: U2FsdGVkX1/5/BLrKdohfFw1Wrtf+1Jl9B28+t1wcdZMXHQ+6WlSguI+iisqtJxHKDoOqwRO8HYOQXgLL2I/5DKqroAlDSvEN5baJ+qQexrIKYUhDOmcFqLWQ/plQkE4m3I0Sp8wEuKHmI3sw04dsN5HTVz2wRqu9JO9ZYQxg2HGpWuT3qFhBmPN7iDLmNpksQeT7RCTrHvByUNHkYAv88zqfOYhUMAGn6Dm4SN8l7sUCyUDnIiBdcVB4vvpAMz6Vk4QpFz2E11wEV4TZ33Tjad6A85FwyXJKQTBFVWeXFK9qixbugm8j8XAwW+zKFCHYDSLTg1iAvKPtUF0pb7xRrgd/IZ3wPkC1jC3wPYL/Y1LwsphMTxq9UccSICRbEN7Xe4dQ4zT6lB+zMfUhY464aADleF6Owy4zA46GnEHtpXv6CvnblMN02AelOHg4cY50XNsUBoZa1ITG1oYtNs8QJMD5W9wWamOp/keXQ1KTJA4Ji9ISlnJwdlUhF6QN6VXndiysS4rwFt88DWTUaPon+hIKay5U9+mZkecb6AeLxf/TFiPm0TA3v6YUsyCtNCbkHcYxl1Jkm66IHRVLGIAVonz675UqnloqJ3FHXZN+mxZZnUc7HzC4FIvsu1RizGMan5xvGW+sxh6UKxpyX1wV1WGZyNssz+lA+3xB+hfmiDjfFfrN27j3jfqNEvt/BDkUiA3QQZjvRoGreUFbqPT0NTxPD6VJLr/obhtRGElcerWGIX5MlF3RzAwaoeKSpwkEgunM2zzpdwzbUt6pfN/UIKTkB/pIDf+FjR5kTuSf+0MugXhRWr+ekoxOFdtzw5LA+AxE+kTjkRafye8JVp2EsrsexieM0GFnGH2XYN9kv2qRUub15aN2cSJNmAvZK7SmRfXgl9lh5H4xhUR7iv+7H3a7jEDbV6+ X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: These pages are all chained together through the lru list, so we know they're folios. Use the folio APIs to save three hidden calls to compound_head(). Signed-off-by: Matthew Wilcox (Oracle) --- mm/khugepaged.c | 30 ++++++++++++++---------------- 1 file changed, 14 insertions(+), 16 deletions(-) diff --git a/mm/khugepaged.c b/mm/khugepaged.c index 2771fc043b3b..5cc39c3f3847 100644 --- a/mm/khugepaged.c +++ b/mm/khugepaged.c @@ -689,9 +689,7 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, spinlock_t *ptl, struct list_head *compound_pagelist) { - struct folio *src_folio; - struct page *src_page; - struct page *tmp; + struct folio *src, *tmp; pte_t *_pte; pte_t pteval; @@ -710,10 +708,11 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, ksm_might_unmap_zero_page(vma->vm_mm, pteval); } } else { - src_page = pte_page(pteval); - src_folio = page_folio(src_page); - if (!folio_test_large(src_folio)) - release_pte_folio(src_folio); + struct page *src_page = pte_page(pteval); + + src = page_folio(src_page); + if (!folio_test_large(src)) + release_pte_folio(src); /* * ptl mostly unnecessary, but preempt has to * be disabled to update the per-cpu stats @@ -721,20 +720,19 @@ static void __collapse_huge_page_copy_succeeded(pte_t *pte, */ spin_lock(ptl); ptep_clear(vma->vm_mm, address, _pte); - folio_remove_rmap_pte(src_folio, src_page, vma); + folio_remove_rmap_pte(src, src_page, vma); spin_unlock(ptl); free_page_and_swap_cache(src_page); } } - list_for_each_entry_safe(src_page, tmp, compound_pagelist, lru) { - list_del(&src_page->lru); - mod_node_page_state(page_pgdat(src_page), - NR_ISOLATED_ANON + page_is_file_lru(src_page), - -compound_nr(src_page)); - unlock_page(src_page); - free_swap_cache(src_page); - putback_lru_page(src_page); + list_for_each_entry_safe(src, tmp, compound_pagelist, lru) { + list_del(&src->lru); + node_stat_sub_folio(src, NR_ISOLATED_ANON + + folio_is_file_lru(src)); + folio_unlock(src); + free_swap_cache(&src->page); + folio_putback_lru(src); } } -- 2.43.0