From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0F252C54798 for ; Tue, 5 Mar 2024 07:42:05 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 972776B0095; Tue, 5 Mar 2024 02:42:04 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 923706B0098; Tue, 5 Mar 2024 02:42:04 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7EA2B6B0099; Tue, 5 Mar 2024 02:42:04 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 6F0BD6B0095 for ; Tue, 5 Mar 2024 02:42:04 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4340E8094E for ; Tue, 5 Mar 2024 07:42:04 +0000 (UTC) X-FDA: 81862191768.26.F1A15EB Received: from foss.arm.com (foss.arm.com [217.140.110.172]) by imf18.hostedemail.com (Postfix) with ESMTP id AA1771C0005 for ; Tue, 5 Mar 2024 07:42:01 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=none; spf=pass (imf18.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1709624522; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=daHd4/rk4YR8JgTHCAN7Y317bGNMDE6rXZNNjeGvtCI=; b=caivso2afgObTAAJALs3Ut7gYsX5SY41X8BxqDjF57oZB3lER7eq4cUL64/ktF6tj1OkwW +KnJ5sLe81vv1CHr9UZAVpV8oLU+ixgZ6FYLd6ij5gyVS9+x9HUAt66WDvsejEBOpqm74s pD4ft74W5EfgTcLBI0XGwlKTye/7fAE= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1709624522; a=rsa-sha256; cv=none; b=SyNNfRAAU3pxf9Rr5q4X+VfvdXvbdsjSYn3xGV6S2z+sWkv3pO6rdEBLwk2yEs0/guB6Wg 1fEdarH8RcHpVo64MFmRBZj1VceJloDLzgFfon5Y1feVxY2vjt9d4pF4b8Wdaez9VdmR8B rPt4rgOE6vnw98pJJQarDsWJWGLS8jk= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=none; spf=pass (imf18.hostedemail.com: domain of ryan.roberts@arm.com designates 217.140.110.172 as permitted sender) smtp.mailfrom=ryan.roberts@arm.com; dmarc=pass (policy=none) header.from=arm.com Received: from usa-sjc-imap-foss1.foss.arm.com (unknown [10.121.207.14]) by usa-sjc-mx-foss1.foss.arm.com (Postfix) with ESMTP id 413682F4; Mon, 4 Mar 2024 23:42:37 -0800 (PST) Received: from [10.57.68.162] (unknown [10.57.68.162]) by usa-sjc-imap-foss1.foss.arm.com (Postfix) with ESMTPSA id DB8703F73F; Mon, 4 Mar 2024 23:41:58 -0800 (PST) Message-ID: <62d740f2-f8df-40df-b624-36e099ec1671@arm.com> Date: Tue, 5 Mar 2024 07:41:57 +0000 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [PATCH v3 1/4] mm: swap: Remove CLUSTER_FLAG_HUGE from swap_cluster_info:flags Content-Language: en-GB To: Barry Song <21cnbao@gmail.com> Cc: Matthew Wilcox , David Hildenbrand , Andrew Morton , Huang Ying , Gao Xiang , Yu Zhao , Yang Shi , Michal Hocko , Kefeng Wang , linux-kernel@vger.kernel.org, linux-mm@kvack.org References: <6541e29b-f25a-48b8-a553-fd8febe85e5a@redhat.com> <2934125a-f2e2-417c-a9f9-3cb1e074a44f@redhat.com> <049818ca-e656-44e4-b336-934992c16028@arm.com> <4a73b16e-9317-477a-ac23-8033004b0637@arm.com> <1195531c-d985-47e2-b7a2-8895fbb49129@redhat.com> <5ebac77a-5c61-481f-8ac1-03bc4f4e2b1d@arm.com> From: Ryan Roberts In-Reply-To: Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit X-Rspamd-Queue-Id: AA1771C0005 X-Rspam-User: X-Stat-Signature: jpf4wirxs9x1pzrb6g44uthkb6otyudu X-Rspamd-Server: rspam03 X-HE-Tag: 1709624521-708496 X-HE-Meta: U2FsdGVkX1/JguOfjdeYsr5pnreF/Ka1/HfFJO6yMj9qfsjw6H6T7vbzTcVOeFkFV9kKyU+1EKwPB5vCwkYHe07P8Qy7ZRapT938lWccIqATkkgP7fEXq0yh8YaT/lOm/+KV/W4RMz3frfZJPrSBNt9kI9LE6Q0PIRwlOUTOWRjO3AGUjxJyC2u91Z1OKf9p/Vqy4JXQ11n0KdcF5Hg3N36nbLy5uQwics781cxdxNCucDUX83C1KbsvAikX/GZX4uu8OOdqTBDVRDeN3SgwsE0YZdotixty/5UA59vlme5LRi6BIInJgL0ENC627Cm3mITu2D1lhMYPNWT1GVl5mzHeweywGeDyIKadHiwp/uETTGdeeJaw10cHqcD6WoSEzOhfdN4MuSVbunCdy/ibdtYWwprftasVA0uJ7jWGHYiBrWB85uysFSPwKb//YcWMQ0jYwmh/lQ1xtfHIqjNAhOpwGS4/bnTC4WcIkanejCUWeOUhWnCWZgsoStxcOVNABzSmQ/rQdCHIcnJGAunEtezMtB21KubsvuCoXZ4kZnvxRi3CVz2ZTrdID50DtR8hAJ+24JluruACY40gNh4FgxPiNIzJrWNP5JxO/e9tVPZ/+3OvMbUUlDolx1bRYyQpvkl45BE8bT29pLPGP/hld3IrZq2V9iVxOiSrDpUjCtr24DsnKR0Zy+L2rJSrVF/UCf8jOqlMACafrLREpBcy527iXpD3/YCeszty25drDGCuEXL+h9ef6BHTfl/99lYe56B97If3Wz1YI0TWxxh0kM6z153ZAW2EtbIMFMuNvFAsP47OpA6XNHzzAg7QCZGwrJbXrDAH4k/SmtsI146WViZb4Y+ymjBGg0FwkP8BKMYToqojJ6TsUKs14MpFn13IqD2xTwgC1dA= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 04/03/2024 05:42, Barry Song wrote: > On Mon, Mar 4, 2024 at 5:52 PM Barry Song <21cnbao@gmail.com> wrote: >> >> On Sat, Mar 2, 2024 at 6:08 AM Ryan Roberts wrote: >>> >>> On 01/03/2024 16:44, Ryan Roberts wrote: >>>> On 01/03/2024 16:31, Matthew Wilcox wrote: >>>>> On Fri, Mar 01, 2024 at 04:27:32PM +0000, Ryan Roberts wrote: >>>>>> I've implemented the batching as David suggested, and I'm pretty confident it's >>>>>> correct. The only problem is that during testing I can't provoke the code to >>>>>> take the path. I've been pouring through the code but struggling to figure out >>>>>> under what situation you would expect the swap entry passed to >>>>>> free_swap_and_cache() to still have a cached folio? Does anyone have any idea? >>>>>> >>>>>> This is the original (unbatched) function, after my change, which caused David's >>>>>> concern that we would end up calling __try_to_reclaim_swap() far too much: >>>>>> >>>>>> int free_swap_and_cache(swp_entry_t entry) >>>>>> { >>>>>> struct swap_info_struct *p; >>>>>> unsigned char count; >>>>>> >>>>>> if (non_swap_entry(entry)) >>>>>> return 1; >>>>>> >>>>>> p = _swap_info_get(entry); >>>>>> if (p) { >>>>>> count = __swap_entry_free(p, entry); >>>>>> if (count == SWAP_HAS_CACHE) >>>>>> __try_to_reclaim_swap(p, swp_offset(entry), >>>>>> TTRS_UNMAPPED | TTRS_FULL); >>>>>> } >>>>>> return p != NULL; >>>>>> } >>>>>> >>>>>> The trouble is, whenever its called, count is always 0, so >>>>>> __try_to_reclaim_swap() never gets called. >>>>>> >>>>>> My test case is allocating 1G anon memory, then doing madvise(MADV_PAGEOUT) over >>>>>> it. Then doing either a munmap() or madvise(MADV_FREE), both of which cause this >>>>>> function to be called for every PTE, but count is always 0 after >>>>>> __swap_entry_free() so __try_to_reclaim_swap() is never called. I've tried for >>>>>> order-0 as well as PTE- and PMD-mapped 2M THP. >>>>> >>>>> I think you have to page it back in again, then it will have an entry in >>>>> the swap cache. Maybe. I know little about anon memory ;-) >>>> >>>> Ahh, I was under the impression that the original folio is put into the swap >>>> cache at swap out, then (I guess) its removed once the IO is complete? I'm sure >>>> I'm miles out... what exactly is the lifecycle of a folio going through swap out? >>>> >>>> I guess I can try forking after swap out, then fault it back in in the child and >>>> exit. Then do the munmap in the parent. I guess that could force it? Thanks for >>>> the tip - I'll have a play. >>> >>> That has sort of solved it, the only problem now is that all the folios in the >>> swap cache are small (because I don't have Barry's large swap-in series). So >>> really I need to figure out how to avoid removing the folio from the cache in >>> the first place... >> >> I am quite sure we have a chance to hit a large swapcache even using zRAM - >> a sync swapfile and even during swap-out. >> >> I have a test case as below, >> 1. two threads to run MADV_PAGEOUT >> 2. two threads to read data being swapped-out >> >> in do_swap_page, from time to time, I can get a large swapcache. >> >> We have a short time window after add_to_swap() and before >> __removing_mapping() of >> vmscan, a large folio is still in swapcache. >> >> So Ryan, I guess you can trigger this by adding one more thread of >> MADV_DONTNEED to do zap_pte_range? > > Ryan, I have modified my test case to have 4 threads: > 1. MADV_PAGEOUT > 2. MADV_DONTNEED > 3. write data > 4. read data > > and git push the code here so that you can get it, > https://github.com/BarrySong666/swaptest/blob/main/swptest.c Thanks for this, Barry! > > I can reproduce the issue in zap_pte_range() in just a couple of minutes. > >> >> >>> >>>> >>>>> >>>>> If that doesn't work, perhaps use tmpfs, and use some memory pressure to >>>>> force that to swap? >>>>> >>>>>> I'm guessing the swapcache was already reclaimed as part of MADV_PAGEOUT? I'm >>>>>> using a block ram device as my backing store - I think this does synchronous IO >>>>>> so perhaps if I have a real block device with async IO I might have more luck? >>>>>> Just a guess... >>>>>> >>>>>> Or perhaps this code path is a corner case? In which case, perhaps its not worth >>>>>> adding the batching optimization after all? >>>>>> >>>>>> Thanks, >>>>>> Ryan >>>>>> >>>> > > Thanks > Barry