From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B9450C27C53 for ; Wed, 12 Jun 2024 06:23:31 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1F7C46B0155; Wed, 12 Jun 2024 02:23:31 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 1A8396B0156; Wed, 12 Jun 2024 02:23:31 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 06F0C6B0157; Wed, 12 Jun 2024 02:23:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id DD52F6B0155 for ; Wed, 12 Jun 2024 02:23:30 -0400 (EDT) Received: from smtpin28.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 860ED1C14BC for ; Wed, 12 Jun 2024 06:23:30 +0000 (UTC) X-FDA: 82221244980.28.1DF6EE4 Received: from out30-131.freemail.mail.aliyun.com (out30-131.freemail.mail.aliyun.com [115.124.30.131]) by imf17.hostedemail.com (Postfix) with ESMTP id E010640014 for ; Wed, 12 Jun 2024 06:23:26 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=xZun7ONV; spf=pass (imf17.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.131 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1718173408; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=VkbxeyTbzZHZmW5pIOy78DIOXXvghz5goTacRzjkGcY=; b=xVJrnt81YZ2K9K91pnzYqp/NM1JO7eewc3os9XJixVbVFFQr3gTFIy1OXVtAQXub07wxMQ 8XfIQtu+iJG07KAdQqZFgS9oNu9Tc5lN/r44lOH625iIOeRE8k5Y+hZ+TKRyiDdRAo3I3p mdu2nacD1ZEOO48lghwLiQDtlrp9Pcw= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1718173408; a=rsa-sha256; cv=none; b=Wy8C39rCjrwQ4ZNgSc9fhP+xULarp7wDW6R/2ci9GRDhsTUIMlrASL/20IIxKGQ85ncco4 SugnaJk0M11Tcep5x/Z8GLihlSooHeftsz/dzQW976InpO98/mAPxKOlmEVt6GHw/lfeAV sStaHCkgf7E/nrAprv/yXWF0kIYLA1c= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=linux.alibaba.com header.s=default header.b=xZun7ONV; spf=pass (imf17.hostedemail.com: domain of baolin.wang@linux.alibaba.com designates 115.124.30.131 as permitted sender) smtp.mailfrom=baolin.wang@linux.alibaba.com; dmarc=pass (policy=none) header.from=linux.alibaba.com DKIM-Signature:v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.alibaba.com; s=default; t=1718173398; h=Message-ID:Date:MIME-Version:Subject:To:From:Content-Type; bh=VkbxeyTbzZHZmW5pIOy78DIOXXvghz5goTacRzjkGcY=; b=xZun7ONVSp59YHlaF88XlpNhvWvx/mKEtjYh0DtNHEnG8liZK5fDiONRvL6q5kop37pXvLyRsjefAGf5CYStfFI/p7O4D+773IYlMrvwVK5QjfdgY+6d+dA89bxHCZ061BFx4Z0a7O8maFVRYtQ1ttftealY+/zXVBj435jTUlI= X-Alimail-AntiSpam:AC=PASS;BC=-1|-1;BR=01201311R161e4;CH=green;DM=||false|;DS=||;FP=0|-1|-1|-1|0|-1|-1|-1;HT=maildocker-contentspam033037067112;MF=baolin.wang@linux.alibaba.com;NM=1;PH=DS;RN=16;SR=0;TI=SMTPD_---0W8J9Tpa_1718173396; Received: from 30.97.56.60(mailfrom:baolin.wang@linux.alibaba.com fp:SMTPD_---0W8J9Tpa_1718173396) by smtp.aliyun-inc.com; Wed, 12 Jun 2024 14:23:17 +0800 Message-ID: <3d8087e4-ff84-48cc-823a-a6ce2a3c76b4@linux.alibaba.com> Date: Wed, 12 Jun 2024 14:23:13 +0800 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH 0/7] support large folio swap-out and swap-in for shmem To: Hugh Dickins Cc: akpm@linux-foundation.org, willy@infradead.org, david@redhat.com, wangkefeng.wang@huawei.com, chrisl@kernel.org, ying.huang@intel.com, 21cnbao@gmail.com, ryan.roberts@arm.com, shy828301@gmail.com, ziy@nvidia.com, ioworker0@gmail.com, da.gomez@samsung.com, p.raghav@samsung.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org References: <7477de0e-e5bb-529d-e2c0-cabd0d39fb5c@google.com> From: Baolin Wang In-Reply-To: <7477de0e-e5bb-529d-e2c0-cabd0d39fb5c@google.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Server: rspam03 X-Stat-Signature: dy17prdssz1kiyz6jw51ecdghn75rk5x X-Rspamd-Queue-Id: E010640014 X-Rspam-User: X-HE-Tag: 1718173406-777424 X-HE-Meta: U2FsdGVkX1/UsrQJms7cR8B61rGx1ACEuj3UHY4lvxnxIAwtUmpOhlMl91xzFxFaeVM5yGiBUpKrrNWa4ek9vWPC0/zEL99/O0xB5g6TF/XNHGZITVrMnZdBpjGgeYe+GLO3BYjkiJGbrs/G+n/dHacxltgNfUSip7mFfY6vbkfm6JWFqwYkGwOYlYr+MJrTIHGuQIfEvL7y1XV0RAlHMnrAJLTnKBGMeqlaM1sVZ+IAR4053VykydN6dtZ5XSrZffTC734UvcTQ4k3Siu+Vzh3am+DqkDK82UofYJN6tYfubyQ8fuZYCk15gPmfSsjyzLCfp8fLIC4Ux4SesUmQKafsKbnD5ESsejIdJ0sdN3qSsrKOnT0auFvyp/Ol7tKzQC5gMaKy14KIEuM+FAaUMfF9VUh5LDsMs7jPmacwkJVRaKu3wdw1zdd1UedRvmW1o4E9n6jHgFE3njJGHqaTVflbBSLmYElRUNvikuVP33XgFamMxWfh7kxjJPHsik88+GARuqcy8CUeCi6uhG2sRswMpWpsEMXSUy2TgzgiJBuU1WTE3K0P6K58Tp5xt5Yku0D/0JOHbYW25CmRY153IiNYjshqziOc4UDfq3/I/upVdt99QwVz+w0jSptaOl6gS6tBtuf96fhcOcIKdy60l/ffO7m4pwkR+RsoB6mecx72gL0VjE7tf36XuJ8/r+7N3+DCpxiWiCb6bT+YaPOArp+uUtHi6APoc2l/tyEHgaKn9wYUtBzy7/Gh1OwWdCnpLI8xPplKJsMYTDkTFQBFPj3FfmUkFKxxYkZZ7z1oTScAnhgdGOfuhFkK/7pxu/Y89yTXsMz+N8ePOCcjGk9vI81el0A9vlzIW3gr5nT3qb+UMy2qVA57D5jsyHYKWAgTy2vN7tSI2C4u2LjDvfGNbu7i2q9vWkqAGr1JnpCk3LBJCMFva/JF+AnUva3IVXd2XOAouIH264Ev32uY7I5 gn1Jesxy dbRcrkuX9orhLLBnf+5wnjTRFbrHSv2opZoVOtfBaCFQarDCePJjg1WFQ5U1eLkJvO0JBIhhRCN4s68HRYWk35G0p9IaSKkTY7uc+AHWFyhHfaYe6RgzDmHAdkMkevyymRMnbN5eiXkFpeQsTtOU+S3qTH+3sIZUFoc57TQLenXfWW1kNZQZ0VDHKYHSdK9SllPblEr2AQ3orXARskGyZLwEHFl+upY4d+zYolN45D33ORGYvcSHPvvc6mOHxe2gLMw4PMIu2zIpiyWlcANc17E8m7XiF5nVjdnaUVmST6HWPEr5UUsYWNOD2nyKbGDepbp1jSYc5JeTp7/VewNYpCodNTOmlORn/YFxe8EMJvf/H29goUdEckSlTEz2Ux129uzAxl+KpA94JjO2KAK7hyPzXWCCOI4v3rPnx4H10jOM6nt+EqOoMZot64A== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi Hugh, On 2024/6/12 13:46, Hugh Dickins wrote: > On Thu, 6 Jun 2024, Baolin Wang wrote: > >> Shmem will support large folio allocation [1] [2] to get a better performance, >> however, the memory reclaim still splits the precious large folios when trying >> to swap-out shmem, which may lead to the memory fragmentation issue and can not >> take advantage of the large folio for shmeme. >> >> Moreover, the swap code already supports for swapping out large folio without >> split, and large folio swap-in[3] series is queued into mm-unstable branch. >> Hence this patch set also supports the large folio swap-out and swap-in for >> shmem. >> >> [1] https://lore.kernel.org/all/cover.1717495894.git.baolin.wang@linux.alibaba.com/ >> [2] https://lore.kernel.org/all/20240515055719.32577-1-da.gomez@samsung.com/ >> [3] https://lore.kernel.org/all/20240508224040.190469-6-21cnbao@gmail.com/T/ >> >> Changes from RFC: >> - Rebased to the latest mm-unstable. >> - Drop the counter name fixing patch, which was queued into mm-hotfixes-stable >> branch. >> >> Baolin Wang (7): >> mm: vmscan: add validation before spliting shmem large folio >> mm: swap: extend swap_shmem_alloc() to support batch SWAP_MAP_SHMEM >> flag setting >> mm: shmem: support large folio allocation for shmem_replace_folio() >> mm: shmem: extend shmem_partial_swap_usage() to support large folio >> swap >> mm: add new 'orders' parameter for find_get_entries() and >> find_lock_entries() >> mm: shmem: use swap_free_nr() to free shmem swap entries >> mm: shmem: support large folio swap out >> >> drivers/gpu/drm/i915/gem/i915_gem_shmem.c | 1 + >> include/linux/swap.h | 4 +- >> include/linux/writeback.h | 1 + >> mm/filemap.c | 27 ++++++- >> mm/internal.h | 4 +- >> mm/shmem.c | 58 ++++++++------ >> mm/swapfile.c | 98 ++++++++++++----------- >> mm/truncate.c | 8 +- >> mm/vmscan.c | 22 ++++- >> 9 files changed, 140 insertions(+), 83 deletions(-) > > I wanted to have some tests running, while looking through these > and your shmem mTHP patches; but I wasted too much time on that by > applying these on top and hitting crash, OOMs and dreadful thrashing - > testing did not get very far at all. Thanks for testing. I am sorry I haven't found the issues with my testing. > Perhaps all easily fixed, but I don't have more time to spend on it, > and think this series cannot expect to go into 6.11: I'll have another > try with it next cycle. > > I really must turn my attention to your shmem mTHP series: no doubt > I'll have minor adjustments to ask there - but several other people > are also waiting for me to respond (or given up on me completely). Sure. Thanks. > > The little crash fix needed in this series appears to be: > > --- a/mm/shmem.c > +++ b/mm/shmem.c > @@ -2053,7 +2053,8 @@ static int shmem_swapin_folio(struct ino > goto failed; > } > > - error = shmem_add_to_page_cache(folio, mapping, index, > + error = shmem_add_to_page_cache(folio, mapping, > + round_down(index, nr_pages), > swp_to_radix_entry(swap), gfp); > if (error) > goto failed; Good catch. I missed this. > Then the OOMs and dreadful thrashing are due to refcount confusion: > I did not even glance at these patches to work out what's wanted, > but a printk in __remove_mapping() showed that folio->_refcount was > 1024 where 513 was expected, so reclaim was freeing none of them. I will look at this issue and continue to do more tesing before sending out new version. Thanks.