From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id C2791C54E68 for ; Thu, 21 Mar 2024 10:20:38 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 454046B0083; Thu, 21 Mar 2024 06:20:38 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4044F6B0085; Thu, 21 Mar 2024 06:20:38 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2A4A96B0087; Thu, 21 Mar 2024 06:20:38 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id 1658D6B0083 for ; Thu, 21 Mar 2024 06:20:38 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id D7C6CA1D95 for ; Thu, 21 Mar 2024 10:20:37 +0000 (UTC) X-FDA: 81920652114.11.EDC9384 Received: from mail-vk1-f169.google.com (mail-vk1-f169.google.com [209.85.221.169]) by imf17.hostedemail.com (Postfix) with ESMTP id EE20B4000E for ; Thu, 21 Mar 2024 10:20:35 +0000 (UTC) Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=bjrfBtr2; spf=pass (imf17.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.221.169 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1711016436; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=fGZfHShf8cu5x0RkynH4JraFdCNV05guyu1vnYqsvdg=; b=4/skiJByfD8/4jdWDVUF19MkmLaUA5LVr0O031SfBO0E0c1L61pzu0vFKK1/rxbPE6SznE /TGMq8Lu8joT8/u/gv6AGW1tWNcOpb2be2rh+IPF4+MFozINYuUbyz9XG7W8GHTSJdzim/ FrF2izlOrHNkqgKDRdJMEAQpSheMK50= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1711016436; a=rsa-sha256; cv=none; b=eAJ9Uqvhcynhp9SqGKCY48ImvxM431MggQGBz61vDsXb9hX39zCa1auKmnA/wYaTbzR3mH ClMEn0/cV/5MBLAWzZ/+AiwT7lTRV+vNlARbEE8xteF7ZMbAzjyhX21wF7zxGqtLzZOuVT XxbBTGS9y7FburqkTTEYI2uHeyaAU3k= ARC-Authentication-Results: i=1; imf17.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=bjrfBtr2; spf=pass (imf17.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.221.169 as permitted sender) smtp.mailfrom=21cnbao@gmail.com; dmarc=pass (policy=none) header.from=gmail.com Received: by mail-vk1-f169.google.com with SMTP id 71dfb90a1353d-4affeacaff9so256266e0c.3 for ; Thu, 21 Mar 2024 03:20:35 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1711016435; x=1711621235; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=fGZfHShf8cu5x0RkynH4JraFdCNV05guyu1vnYqsvdg=; b=bjrfBtr2cr5JupFNUPrNGgRIWGnKnbtuRKLuAkA+F1kx7t+4zELJhywSS5tU6dk1ID O4gA+3o6SPBkXH8FEOKZXWB8vZ4qfbNEFMXZa5fkxQ3OOilLbh6nXvKDYA6TZwlu1l7Y b6tlFjSsZ+MF39gk0N92ihegSYvhHLF8Wq9smU2GtNUE7twd76YAA+K6pe2F3wumaDWu vCGn8Lg+MoCwN1MSs9fBnewiGSQdmlJTizS+Ftu64039FO0NWyxNEt8CImAkTeDWtNLN SsL8SvFbTpb2m76vbvWu0KncQfdgqygc3ObdeM+25xcFuy4hU1n5bNmIKqyLisNXE9IS NC8Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1711016435; x=1711621235; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=fGZfHShf8cu5x0RkynH4JraFdCNV05guyu1vnYqsvdg=; b=svK7oDFevzCg+X4FU8lNRl1rrbV8dHfPJYNYo+XfSb14SxWFqfLyuzfGiZD407Rsmg aKApgYL4hocuOSE20Keo+R5K9LWAtMf2A0FS5tUl8/PXKz8Btw3UXVMM/fIzUknKudYc +Mamr+zUy1qQq8YhoGvRR4TjbKTPGt8knVXk3LIoZxcgtCgvrb+aw/yCkmUU3rHm7BoY hhL39+fOsw7u0PhuQpLt12YXexmm5tYWrCfjvXz3rtZL3DFR3tTixOWpn3t3crqc/Tqi S83o2fjQTJRrcpyv+FHCuECLE7apZhwE8100b7mQCPsyI0q2FrBBOkUjI1ACTWYFhx6L pzlw== X-Forwarded-Encrypted: i=1; AJvYcCVJsXO294CRbfMmt797cD7dc6SQ4Y7A5Od10GnOGBs6W8Ke8qJwDnaPoA6u3MkLICIXN7WBOrevMbswGmosXoG2ZZE= X-Gm-Message-State: AOJu0YydBvjmWwjisG3Qn8uUCyMyF/CkLEvah3cpB3ihye9Ph8UmO/dX 6VpgkcKWdkyCZdqjc2dy1riV5+ne+Qc8zhCm49RgLmhvfqiNPUaPHL6lCLadNudxjiV1IAjA38w 2+qtlcB6TVcUGN/9J4YnUZrI1nu0= X-Google-Smtp-Source: AGHT+IGRfCz7kzvP0Ejqk7dWG4ee/kOzQMXlvoxiZ0s0QmlwQF2ws14PTJlNaVzZp30vTe4Jvy2QOMjKlTFqCvncEo8= X-Received: by 2002:a1f:f80a:0:b0:4d4:1b6a:7924 with SMTP id w10-20020a1ff80a000000b004d41b6a7924mr1160950vkh.8.1711016434870; Thu, 21 Mar 2024 03:20:34 -0700 (PDT) MIME-Version: 1.0 References: <20240304081348.197341-1-21cnbao@gmail.com> <20240304081348.197341-6-21cnbao@gmail.com> <87wmq3yji6.fsf@yhuang6-desk2.ccr.corp.intel.com> <87sf0rx3d6.fsf@yhuang6-desk2.ccr.corp.intel.com> <87jzm0wblq.fsf@yhuang6-desk2.ccr.corp.intel.com> <9ec62266-26f1-46b6-8bb7-9917d04ed04e@arm.com> <87jzlyvar3.fsf@yhuang6-desk2.ccr.corp.intel.com> In-Reply-To: From: Barry Song <21cnbao@gmail.com> Date: Thu, 21 Mar 2024 23:20:23 +1300 Message-ID: Subject: Re: [RFC PATCH v3 5/5] mm: support large folios swapin as a whole To: Ryan Roberts Cc: "Huang, Ying" , Matthew Wilcox , akpm@linux-foundation.org, linux-mm@kvack.org, chengming.zhou@linux.dev, chrisl@kernel.org, david@redhat.com, hannes@cmpxchg.org, kasong@tencent.com, linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, mhocko@suse.com, nphamcs@gmail.com, shy828301@gmail.com, steven.price@arm.com, surenb@google.com, wangkefeng.wang@huawei.com, xiang@kernel.org, yosryahmed@google.com, yuzhao@google.com, Chuanhua Han , Barry Song Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: EE20B4000E X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: 9yff9xjtgky1x85t3gkcachjbsouwfaj X-HE-Tag: 1711016435-893379 X-HE-Meta: U2FsdGVkX1/ryS+GN6nCqS7hOvQxZiSXfC8mtNAf5/S/RloiJ2mL0F5jMmd5XRDMfLf6TKtz8QvNBh8TOAcLhe13nRq/ZgqTLqk7lt1LENCdS2V7l9xCknbNW7gWP1GJVQOGKgr6iPu3aPylkjZfuwDYE/l3jdzojiO0vjy6Ebu223tPppb/n217bCNqzHxXY7/jzheoq+4VbbzmeNHBsSmvvR1qXDcYdrMVIMmLQT7+fhI3HosiYuy9WM/dTRVKpIYH13IiVsYFD8EOMjzIg0t7nsD2o3M6baziK3ju89OxPhpzAqP8zOMx/XkRVOpUbwyL7fp0LnesnvcPDM8zFtOcY1ij/uLSayu2WmxX0OZkbzLyNNCzPKNnVwsdSHabkHJwFnbEPTnBo0J97V7lL10mhKLQXpBLmbSCCm5tiYq3U/zBfPcQ4fhPrxArcKClWmwk93csPY6cuvqWe/iCtcWh0/kJ7hrnn2Qk6RfiZMhZMN2tvecIhFoPWMxrupgRIiAiO2ASEeOTHev2EUU5qyWpl6Ol+UIFdcSec180s2IBJUd1OMTJr7rOt/b8OVYLd6PSvEz+s5lYqz06sq24kIDZKn2TXh4X1LFSjsKfHCewI1FPRzM8e7au1mzaQLGufR73vP7fVD3fhFJO4Z+n6cpM7bJmpoSLcX14zuMr0HfVfixYoQ/9AFVtMq2DycMco59tEud8Q1DLliBqnaFWDYfd1/ExCf2T4OI/5ajrthNbR4tHUbDWUEujUdvfGufR7yVPp+TeLZxfCP/22Rq2c0poSLVYPiG/RnZa/jgxwaebf/Nf4y44mfB9TKqTbB5sqMe0yAo6EW8oWk44FYJBbc2HprxLkOUwCXYxuC3Jd7/Wknpo9RgjdcUPwZA16Lr1oKTkALow6FjAbxaZ/+pkYfsiLaHVQEC2XNSWlAjfYuXGzw6ZtnykZnMwL117w+QfWcMXRm6asHk40yutFTR keiubbHT eeukfq6oN93kmB9gwM8i0PEq3vN4wxcVfgab6cxwUCVXajgyO0T/sVhsM0xIguQchiy5y0MkD7UUT1rB0uxqTDxz71d5nS/RhICQ7Vf1bddVamIfm+DuM2J7QRGnz4uHjpG06sxTVtfIo5+uZjVRkPgpqbO47uPUINTUEyyPdcBLwkpjcqBvqO1BXxdGAbEVUpn3fkqskI2IJcmX/OWvpzZ6CdZV/6PVeN/R+rvm14aRlUfygs8CM+zX5WVxXVqzdwncaY3/f2KzNGbWNKCDOjiZ2a09nsj9rG3f72tFg/UTF1DgRBRaQtXHIBsxpu56cF2fZZenCNbTcFzMXe59CmhM1B4+gxSm8Ba+T+Ja7fPlDkES7nSvOnSUF368UOvBdXQtb6rl0y/bFATQdt7SGZmHWb03BjcWndtPN X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Mar 20, 2024 at 1:19=E2=80=AFAM Ryan Roberts = wrote: > > On 19/03/2024 09:20, Huang, Ying wrote: > > Ryan Roberts writes: > > > >>>>> I agree phones are not the only platform. But Rome wasn't built in = a > >>>>> day. I can only get > >>>>> started on a hardware which I can easily reach and have enough hard= ware/test > >>>>> resources on it. So we may take the first step which can be applied= on > >>>>> a real product > >>>>> and improve its performance, and step by step, we broaden it and ma= ke it > >>>>> widely useful to various areas in which I can't reach :-) > >>>> > >>>> We must guarantee the normal swap path runs correctly and has no > >>>> performance regression when developing SWP_SYNCHRONOUS_IO optimizati= on. > >>>> So we have to put some effort on the normal path test anyway. > >>>> > >>>>> so probably we can have a sysfs "enable" entry with default "n" or > >>>>> have a maximum > >>>>> swap-in order as Ryan's suggestion [1] at the beginning, > >>>>> > >>>>> " > >>>>> So in the common case, swap-in will pull in the same size of folio = as was > >>>>> swapped-out. Is that definitely the right policy for all folio size= s? Certainly > >>>>> it makes sense for "small" large folios (e.g. up to 64K IMHO). But = I'm not sure > >>>>> it makes sense for 2M THP; As the size increases the chances of act= ually needing > >>>>> all of the folio reduces so chances are we are wasting IO. There ar= e similar > >>>>> arguments for CoW, where we currently copy 1 page per fault - it pr= obably makes > >>>>> sense to copy the whole folio up to a certain size. > >>>>> " > >> > >> I thought about this a bit more. No clear conclusions, but hoped this = might help > >> the discussion around policy: > >> > >> The decision about the size of the THP is made at first fault, with so= me help > >> from user space and in future we might make decisions to split based o= n > >> munmap/mremap/etc hints. In an ideal world, the fact that we have had = to swap > >> the THP out at some point in its lifetime should not impact on its siz= e. It's > >> just being moved around in the system and the reason for our original = decision > >> should still hold. > >> > >> So from that PoV, it would be good to swap-in to the same size that wa= s > >> swapped-out. > > > > Sorry, I don't agree with this. It's better to swap-in and swap-out in > > smallest size if the page is only accessed seldom to avoid to waste > > memory. > > If we want to optimize only for memory consumption, I'm sure there are ma= ny > things we would do differently. We need to find a balance between memory = and > performance. The benefits of folios are well documented and the kernel is > heading in the direction of managing memory in variable-sized blocks. So = I don't > think it's as simple as saying we should always swap-in the smallest poss= ible > amount of memory. Absolutely agreed. With 64KiB large folios implemented, there may have been a slight uptick in memory usage due to fragmentation. Nevertheless, through= the optimization of zRAM and zsmalloc to compress entire large folios, we found= that the compressed data could be up to 1GiB smaller compared to compressing the= m in 4KiB increments on a typical phone with 12~16GiB memory. Consequently, w= e not only reclaimed our memory loss entirely but also gained the benefits of CONT-PTE , reduced TLB misses etc. > > You also said we should swap *out* in smallest size possible. Have I > misunderstood you? I thought the case for swapping-out a whole folio with= out > splitting was well established and non-controversial? > > > > >> But we only kind-of keep that information around, via the swap > >> entry contiguity and alignment. With that scheme it is possible that m= ultiple > >> virtually adjacent but not physically contiguous folios get swapped-ou= t to > >> adjacent swap slot ranges and then they would be swapped-in to a singl= e, larger > >> folio. This is not ideal, and I think it would be valuable to try to m= aintain > >> the original folio size information with the swap slot. One way to do = this would > >> be to store the original order for which the cluster was allocated in = the > >> cluster. Then we at least know that a given swap slot is either for a = folio of > >> that order or an order-0 folio (due to cluster exhaustion/scanning). C= an we > >> steal a bit from swap_map to determine which case it is? Or are there = better > >> approaches? > > > > [snip] > > > > -- > > Best Regards, > > Huang, Ying Thanks Barry