From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A8503D3C92A for ; Sun, 20 Oct 2024 20:09:03 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 299EB6B007B; Sun, 20 Oct 2024 16:09:03 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2499D6B0082; Sun, 20 Oct 2024 16:09:03 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 138116B0083; Sun, 20 Oct 2024 16:09:03 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id EBA426B007B for ; Sun, 20 Oct 2024 16:09:02 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id DEF55C08B1 for ; Sun, 20 Oct 2024 20:08:46 +0000 (UTC) X-FDA: 82695068610.09.D5288ED Received: from out-178.mta0.migadu.com (out-178.mta0.migadu.com [91.218.175.178]) by imf16.hostedemail.com (Postfix) with ESMTP id BEA3618000D for ; Sun, 20 Oct 2024 20:08:46 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=YsiqSh4p; spf=pass (imf16.hostedemail.com: domain of kent.overstreet@linux.dev designates 91.218.175.178 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1729454791; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YLBMovQKhEK4NKL1ZpIk8uqIHLznemyS3ZLRNZ/M+X8=; b=HBx+Bp/iS2KNHta2pq8NSBje0oxvIRZZbSqfp6CtgjDclhrz3JkGVp3nn3nnYb+ePg0qrk 4Ux0kZhVIONuFd3T4N0yTqa4e0QkB0tCUmoYU00Ab2CfTxidSqUww2A+E5oOZoAwJucdhk pHr5X3ubTz7xF43/MAznBX3GuYCOYkg= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1729454791; a=rsa-sha256; cv=none; b=pCZP/AbQDKNL7pYbNJuJmxKTQfrnIh8f7zUWngPl40V7UvPsPyfztmDC4GSQgAmgijj2tp TDQMvixL4Kk5+EwyfxG0cay0DkY3nvhDK4zLGrrnbKLxmqujPgmNsPotyWLpbw6JMINrst O3tYGie9tF5lE7P0sgbjZ+380gT4V2A= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=YsiqSh4p; spf=pass (imf16.hostedemail.com: domain of kent.overstreet@linux.dev designates 91.218.175.178 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev; dmarc=pass (policy=none) header.from=linux.dev Date: Sun, 20 Oct 2024 16:08:54 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1729454938; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=YLBMovQKhEK4NKL1ZpIk8uqIHLznemyS3ZLRNZ/M+X8=; b=YsiqSh4p2hTOFar5pwm4uGHpLfWG2XdU/iTus+KWBZl2xvR4taFP9rInNOYDtxlOTkQz+a 8faDi2FPrE0uSKhuj9THPgeEKPH69XWcotmI1pxyWO+UqLhPfZ2bnJzIw8oPJRBPtFv5oI hY4m/ou0X61wGQ3vkIzRguNjEPgJiAU= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Kent Overstreet To: Vlastimil Babka Cc: Linus Torvalds , Lorenzo Stoakes , linux-bcachefs@vger.kernel.org, linux-mm@kvack.org, Andrew Morton , Uladzislau Rezki , Christoph Hellwig Subject: Re: [PATCH] mm: Drop INT_MAX limit from kvmalloc() Message-ID: References: <20241019210037.146825-1-kent.overstreet@linux.dev> <90bc0794-4cab-415f-a442-4af85a32eed8@lucifer.local> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: BEA3618000D X-Stat-Signature: tua9orf3tp3tr5zk6t6ufxhn37w9wcpd X-HE-Tag: 1729454926-768486 X-HE-Meta: U2FsdGVkX1+lhfA1OAHYE4u2add05iKc1DcAI+8mA10zx6euqfWUF1bXpUFI3f65jVjODAHKQ43c3soO0tdu2dFEu0NrnhirfEzHBIL2YVkenhTS4YfrWWDtJG8FZ3dVaMvbyoOn9qKxb931D3GMdjOMheZLsfPe0zALj796FN2tqxcD6KgRA3+d+shyfET2jIQQUOJ+0muWPlOhnb5/3a71yEgBCWdiP4Cm5wAIZLP6uF/NcmZfLQUIbIwR++k2n93st1sK7fWZ/4Oyx9ykAI7oEwwwO/Ib6UdlXcoEZ8K3vtLPUemL4vO2hfzlKzWykKbFCMRYtzpkYa1Y2HsyI+Kjz2xXoWMysj5ihtmpp7Sl4MWfw4gdiXTmTzEBMqizxCT2TBGDC5nULIb5vtCy0Q3MN4zArh6xX0f2eoo0C8UgqsDXVJD84DDqi3laTAU0flIUHMthXQcVbJvTflxSwNIWg+sCeCPceK7aCXi82YTvL4NUKQ4l75svR+r1tRmN5dtg2actbyOalnvL1M0YcDov8XISOwvsm10NJHCvvxY9Wnfw92F+qPRRevUrJr2jsYruNuc/zJ81Yz4QIravXyqmMcskx041QqFxeP74wNhYY+6+HE5VRBFLLzPlCkF+ahjaFLzIFSS0iYLREOibRcsXGn0eoW9+RHRQssH1OEx1jt4T6JdPVTv98uVJ4pLT5lWCXWG7KNSc+B0KRCObeihb0jWuXDfKnor80LOwDxlBGbtWlpfb57rC7DgNV7iRS+IRMUCg00TRYhIz5HeyHsvT67IhDFbz60gokYMxYMa/OKaqiawyYwaT3iYpYm9oTT1EIIhU0scRUA857EHVWMOVjOikyZmGFVX6ZuT/iGEZvZ/kNLpERrv7WBq7rmsyJF19WyPBb4asoULGjNFJvSaE/mCk3zklrzItXFrU1i/IQtPfbzoCWKXR9M2b4bYW7z5RXWrmeOt+mHNKwI2 5LTiInIL bgqWutljaMvMQCryTjYZ38/c8QN2M1y09+Me8WzBLkQfL4LKi2kBgSu1ID3zJS7w1vGdEbL9EOgNKlRjQA55NUt/lmxYy1Rj6Lxjc0dMMvf2UV2/MEQBgfy54IaGdhbzfh4AOu+0SnPapfeixaE2ZZyoPfFYRLvn+VFc+yWsXdIyxBvj6ZHMwSd1yznYE6yIOUUph X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Sun, Oct 20, 2024 at 09:53:06PM +0200, Vlastimil Babka wrote: > On 10/20/24 20:53, Kent Overstreet wrote: > > On Sun, Oct 20, 2024 at 11:46:11AM -0700, Linus Torvalds wrote: > >> On Sun, 20 Oct 2024 at 10:04, Kent Overstreet wrote: > >> > > >> > But given that vmalloc() already supports > INT_MAX requests, and memory > >> > sizes keep growing so 2GB is getting pretty small - I think it's time, > >> > this is going to come up in other places sooner or later. > >> > >> No. > >> > >> If you need 2GB+ memory for filesystem operations, you fix your code. > > > > This is for journal replay, where we've got a big array of keys and we > > need to sort them. > > > > The keys have to fit in memory (and had to fit in memory previously, for > > them to be dirty in the journal); > > What if the disk is moved to a smaller system, should the fs still mount > there? (I don't mean such a small system that it can't vmalloc() 2GB > specifically, but in principle...) You'll have to do journal replay on the bigger system. Once you've done that, it'll work just fine on the smaller system. (Now, trying to work with a 75TB filesystem on a small machine is going to be really painful if you ever need to fsck. That's just an inherently hard problem, but we've got fsck scalability/performance improvements in the works). But journal replay does inherently require the whole contents of the journal to fit in memory - we have to do the sort + dedup so that we can overlay the contents of the journal over the btree until journal replay is finished so that we can get a consistent view of the filesystem, which we need so that we can run the allocator, and go read-write, which we need in order to do journal replay. Fun bootstrap problems.