From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 5BC47C433F5 for ; Thu, 19 May 2022 03:59:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id D30636B0072; Wed, 18 May 2022 23:59:13 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id CB91D6B0073; Wed, 18 May 2022 23:59:13 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B59296B0074; Wed, 18 May 2022 23:59:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id A00956B0072 for ; Wed, 18 May 2022 23:59:13 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay11.hostedemail.com (Postfix) with ESMTP id 6F594807CB for ; Thu, 19 May 2022 03:59:13 +0000 (UTC) X-FDA: 79481137386.19.1887C0D Received: from mail-pf1-f180.google.com (mail-pf1-f180.google.com [209.85.210.180]) by imf11.hostedemail.com (Postfix) with ESMTP id 2DBD3400D4 for ; Thu, 19 May 2022 03:59:06 +0000 (UTC) Received: by mail-pf1-f180.google.com with SMTP id w200so4002436pfc.10 for ; Wed, 18 May 2022 20:59:11 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance-com.20210112.gappssmtp.com; s=20210112; h=message-id:date:mime-version:user-agent:subject:content-language:to :cc:references:from:in-reply-to:content-transfer-encoding; bh=jgzwIp+OLqmV4OrwP8NZjS9UzJNPt/KLVim9ZoWuO2s=; b=Q3IF358UtjRo01Rw7Uxhizqgu6PT+HfYRMWgSHoDIvhjDIA5Daxjy/coNvC5gB/6jT U4MHSFWPzoPZfcryNbtRDxPaxpoyEghT9ii39CTQaJvdrsG7wXuOnRCY8kIsi5rQhIBT hT67XUXbWkOUYSgCCbCS4hraUKwfHC+Z3mvkJAqrd9OP+T/a1emdrp3Ry5oVgUnCs8/C N4rbGtvX+0sywF4OYA0ubMvof58c17dIYr/nTD0FnnF3Decb0wnNfSGbQC2APzk6q1W4 IEyM1yfuYFZv214PdaHPm0tlXlwy108wBaLbnq7tIHbf1I0e5pOLuqL5wzEEQ10TL0as g9JA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:message-id:date:mime-version:user-agent:subject :content-language:to:cc:references:from:in-reply-to :content-transfer-encoding; bh=jgzwIp+OLqmV4OrwP8NZjS9UzJNPt/KLVim9ZoWuO2s=; b=uYAYXcYBcZ/XMe1O/xq+zAvwrMLyEClKFFBFfYLwCvKdfkEupCdxOpu8JnirwIHtvM kqtJMAQK0zcgPItLpUEO8ywrPAj/HjvnIk80WGqB1gzNvnhJ3NVWyKXzIkp6xb2RnPPT HYedcliGtCZe9KfNTdpB4CvGVd7O7KcGs4GezxpHisDInmHm62dmQhBcsCqLa2rYdklQ +OYoaXC9bG29lHyLrFDJ7p1TNMC6bQUdGcGfnC7jkTNK4wNwIHv0nGrg0O+37qTi/+0V VjoOEgVgauWW1Zi836EipToItU/9eB4ZZaYygrNLkMY8pjl0occZoTvZ9XkrhE9svwhs 29ag== X-Gm-Message-State: AOAM531EtczGji8clq7okZ0k+0MVMnb7Eyph6sm16imzbB0l692dLpDs JrzT645rRrjZJdHJrtRWp3D0Mw== X-Google-Smtp-Source: ABdhPJz/KxbhH/zBrf3OLSkZTNGNkIGNa+MVOM+9QLitznVlr6UaoJP3iAuiiwNVlQ1MPXbw7yt4EQ== X-Received: by 2002:a63:6d8c:0:b0:3f6:3ff:234d with SMTP id i134-20020a636d8c000000b003f603ff234dmr2354200pgc.320.1652932751187; Wed, 18 May 2022 20:59:11 -0700 (PDT) Received: from [10.70.252.135] ([139.177.225.229]) by smtp.gmail.com with ESMTPSA id x11-20020a170902a38b00b0015e9d4a5d27sm2483691pla.23.2022.05.18.20.58.58 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 18 May 2022 20:59:10 -0700 (PDT) Message-ID: <4cd048bb-49b4-bd84-1d89-95dbb3bfa0fc@bytedance.com> Date: Thu, 19 May 2022 11:58:52 +0800 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (Macintosh; Intel Mac OS X 10.15; rv:91.0) Gecko/20100101 Thunderbird/91.9.0 Subject: Re: [RFC PATCH 00/18] Try to free user PTE page table pages Content-Language: en-US To: David Hildenbrand Cc: linux-doc@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, songmuchun@bytedance.com, zhouchengming@bytedance.com, akpm@linux-foundation.org, tglx@linutronix.de, kirill.shutemov@linux.intel.com, jgg@nvidia.com, tj@kernel.org, dennis@kernel.org, ming.lei@redhat.com References: <20220429133552.33768-1-zhengqi.arch@bytedance.com> <8c51d9ae-5a8e-74a9-ddc2-70b5fcd38427@bytedance.com> <37055be1-05af-f7ef-c33e-27f90fa0f9ca@redhat.com> From: Qi Zheng In-Reply-To: <37055be1-05af-f7ef-c33e-27f90fa0f9ca@redhat.com> Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 2DBD3400D4 X-Stat-Signature: fep5ydcoqitz9nurqxnhwubg6tuzxfsd X-Rspam-User: Authentication-Results: imf11.hostedemail.com; dkim=pass header.d=bytedance-com.20210112.gappssmtp.com header.s=20210112 header.b=Q3IF358U; spf=pass (imf11.hostedemail.com: domain of zhengqi.arch@bytedance.com designates 209.85.210.180 as permitted sender) smtp.mailfrom=zhengqi.arch@bytedance.com; dmarc=pass (policy=none) header.from=bytedance.com X-HE-Tag: 1652932746-410298 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 2022/5/18 10:51 PM, David Hildenbrand wrote: > On 17.05.22 10:30, Qi Zheng wrote: >> >> >> On 2022/4/29 9:35 PM, Qi Zheng wrote: >>> Hi, >>> >>> Different from the idea of another patchset of mine before[1], the pte_ref >>> becomes a struct percpu_ref type, and we switch it to atomic mode only in cases >>> such as MADV_DONTNEED and MADV_FREE that may clear the user PTE page table >>> entryies, and then release the user PTE page table page when checking that >>> pte_ref is 0. The advantage of this is that there is basically no performance >>> overhead in percpu mode, but it can also free the empty PTEs. In addition, the >>> code implementation of this patchset is much simpler and more portable than the >>> another patchset[1]. >> >> Hi David, >> >> I learned from the LWN article[1] that you led a session at the LSFMM on >> the problems posed by the lack of page-table reclaim (And thank you very >> much for mentioning some of my work in this direction). So I want to >> know, what are the further plans of the community for this problem? > > Hi, > > yes, I talked about the involved challenges, especially, how malicious > user space can trigger allocation of almost elusively page tables and > essentially consume a lot of unmovable+unswappable memory and even store > secrets in the page table structure. It is indeed difficult to deal with malicious user space programs, because as long as there is an entry in PTE page table page that maps the physical page, the entire PTE page cannot be freed. So maybe we should first solve the problems encountered in engineering practice. We encountered the problems I mentioned in the cover letter several times on our server: VIRT: 55t RES: 590g VmPTE: 110g They are not malicious programs, they just use jemalloc/tcmalloc normally (currently jemalloc/tcmalloc often uses mmap+madvise instead of mmap+munmap to improve performance). And we checked and found taht most of these VmPTEs are empty. Of course, normal operations may also lead to the consequences of similar malicious programs, but we have not found such examples on our servers. > > Empty PTE tables is one such case we care about, but there is more. Even > with your approach, we can still end up with many page tables that are > allocated on higher levels (e.g., PMD tables) or page tables that are Yes, currently my patch does not consider PMD tables. The reason is that its maximum memory consumption is only 1G on 64-bits system, the impact is smaller that 512G of PTE tables. > not empty (especially, filled with the shared zeropage). This case is indeed a problem, and more difficult. :( > > Ideally, we'd have some mechanism that can reclaim also other > reclaimable page tables (e.g., filled with shared zeropage). One idea > was to add reclaimable page tables to the LRU list and to then > scan+reclaim them on demand. There are multiple challenges involved, > obviously. One is how to synchronize against concurrent page table Agree, the current situation is that holding the read lock of mmap_lock can ensure that the PTE tables is stable. If the refcount method is not considered or the logic of the lock that protects the PTE tables is not changed, then the write lock of mmap_lock should be held to ensure synchronization (this has a huge impact on performance). > walkers, another one is how to invalidate MMU notifiers from reclaim > context. It would most probably involve storing required information in > the memmap to be able to lock+synchronize. This may also be a way to explore. > > Having that said, adding infrastructure that might not be easy to extend > to the more general case of reclaiming other reclaimable page tables on > multiple levels (esp PMD tables) might not be what we want. OTOH, it > gets the job done for once case we care about. > > It's really hard to tell what to do because reclaiming page tables and > eventually handling malicious user space correctly is far from trivial :) Yeah, agree :( > > I'll be on vacation until end of May, I'll come back to this mail once > I'm back. > OK, thanks, and have a nice holiday. -- Thanks, Qi