From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-9.6 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI,SIGNED_OFF_BY, SPF_HELO_NONE,SPF_PASS,USER_AGENT_GIT autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id CA91AC433E0 for ; Wed, 10 Jun 2020 20:39:57 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 830EC206A4 for ; Wed, 10 Jun 2020 20:39:57 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=infradead.org header.i=@infradead.org header.b="I/97wCaN" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 830EC206A4 Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=infradead.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 66FBE8D002D; Wed, 10 Jun 2020 16:39:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 45AE48D0035; Wed, 10 Jun 2020 16:39:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 237428D0033; Wed, 10 Jun 2020 16:39:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0252.hostedemail.com [216.40.44.252]) by kanga.kvack.org (Postfix) with ESMTP id E6EF48D002D for ; Wed, 10 Jun 2020 16:39:33 -0400 (EDT) Received: from smtpin27.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id A1EDE3EEFA for ; Wed, 10 Jun 2020 20:39:33 +0000 (UTC) X-FDA: 76914467826.27.waste84_07147b826dce Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin27.hostedemail.com (Postfix) with ESMTP id 1EC4D3DF40 for ; Wed, 10 Jun 2020 20:39:33 +0000 (UTC) X-HE-Tag: waste84_07147b826dce X-Filterd-Recvd-Size: 4580 Received: from bombadil.infradead.org (bombadil.infradead.org [198.137.202.133]) by imf24.hostedemail.com (Postfix) with ESMTP for ; Wed, 10 Jun 2020 20:39:32 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=bombadil.20170209; h=Content-Transfer-Encoding: MIME-Version:References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender :Reply-To:Content-Type:Content-ID:Content-Description; bh=82Nf07arRh2ddQXM6dUk5PZd/nwdd9dRkn6QAYZMbpg=; b=I/97wCaN/5C2K/kQyPuyQujzpa YRWA5GcqLOs2vkpFgnLJWaJQB6T7QxNfsMd3eeQR3Feu818PYzH92vaasIMMZwbhHysfiLYZX132s LcfFTuSOfhquq7DCwyyFfOMqyJcjGntmxql5tDNPn9tz6JkI9mprEHH7knvvaGIDehpNUbXE9kKS6 vZlNmBOmKt57BgcUPR9dvCnEf5wCxJ1gj+jR7vnPA1C2HpBHFR5J2tP8FD4UafK3HRBsGWjeyrn7n i0Mxe2FQN7xAApt7xHyYyhGKmGeFDkWZMNErILFRbTrxvgwBJwPPfyqu9OztjzizUtMCEyL9g5e5b KZwtQDiQ==; Received: from willy by bombadil.infradead.org with local (Exim 4.92.3 #3 (Red Hat Linux)) id 1jj76a-0003Wg-MM; Wed, 10 Jun 2020 20:13:48 +0000 From: Matthew Wilcox To: linux-fsdevel@vger.kernel.org Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org, linux-kernel@vger.kernel.org Subject: [PATCH v6 35/51] mm: Allow THPs to be added to the page cache Date: Wed, 10 Jun 2020 13:13:29 -0700 Message-Id: <20200610201345.13273-36-willy@infradead.org> X-Mailer: git-send-email 2.21.1 In-Reply-To: <20200610201345.13273-1-willy@infradead.org> References: <20200610201345.13273-1-willy@infradead.org> MIME-Version: 1.0 X-Rspamd-Queue-Id: 1EC4D3DF40 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam01 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: From: "Matthew Wilcox (Oracle)" We return -EEXIST if there are any non-shadow entries in the page cache in the range covered by the THP. If there are multiple shadow entries in the range, we set *shadowp to one of them (currently the one at the highest index). If that turns out to be the wrong answer, we can implement something more complex. This is mostly modelled after the equivalent function in the shmem code. Signed-off-by: Matthew Wilcox (Oracle) --- mm/filemap.c | 52 +++++++++++++++++++++++++++++++++++----------------- 1 file changed, 35 insertions(+), 17 deletions(-) diff --git a/mm/filemap.c b/mm/filemap.c index 3a9579a1ffa7..ab9746aff766 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -834,41 +834,58 @@ static int __add_to_page_cache_locked(struct page *= page, XA_STATE(xas, &mapping->i_pages, offset); int huge =3D PageHuge(page); int error; + unsigned int nr =3D 1; void *old; =20 VM_BUG_ON_PAGE(!PageLocked(page), page); VM_BUG_ON_PAGE(PageSwapBacked(page), page); mapping_set_update(&xas, mapping); =20 - get_page(page); - page->mapping =3D mapping; - page->index =3D offset; - if (!huge) { error =3D mem_cgroup_charge(page, current->mm, gfp_mask); if (error) - goto error; + return error; + xas_set_order(&xas, offset, thp_order(page)); + nr =3D thp_nr_pages(page); } =20 + page_ref_add(page, nr); + page->mapping =3D mapping; + page->index =3D offset; + do { + unsigned long exceptional =3D 0; + unsigned int i =3D 0; + xas_lock_irq(&xas); - old =3D xas_load(&xas); - if (old && !xa_is_value(old)) - xas_set_err(&xas, -EEXIST); - xas_store(&xas, page); + xas_for_each_conflict(&xas, old) { + if (!xa_is_value(old)) { + xas_set_err(&xas, -EEXIST); + break; + } + exceptional++; + if (shadowp) + *shadowp =3D old; + } + xas_create_range(&xas); if (xas_error(&xas)) goto unlock; =20 - if (xa_is_value(old)) { - mapping->nrexceptional--; - if (shadowp) - *shadowp =3D old; +next: + xas_store(&xas, page); + if (++i < nr) { + xas_next(&xas); + goto next; } - mapping->nrpages++; + mapping->nrexceptional -=3D exceptional; + mapping->nrpages +=3D nr; =20 /* hugetlb pages do not participate in page cache accounting */ - if (!huge) - __inc_lruvec_page_state(page, NR_FILE_PAGES); + if (!huge) { + __mod_lruvec_page_state(page, NR_FILE_PAGES, nr); + if (nr > 1) + __inc_lruvec_page_state(page, NR_FILE_THPS); + } unlock: xas_unlock_irq(&xas); } while (xas_nomem(&xas, gfp_mask & GFP_RECLAIM_MASK)); @@ -883,7 +900,8 @@ static int __add_to_page_cache_locked(struct page *pa= ge, error: page->mapping =3D NULL; /* Leave page->index set: truncation relies upon it */ - put_page(page); + page_ref_sub(page, nr); + VM_BUG_ON_PAGE(page_count(page) <=3D 0, page); return error; } ALLOW_ERROR_INJECTION(__add_to_page_cache_locked, ERRNO); --=20 2.26.2