From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AFA55D4977B for ; Sun, 1 Dec 2024 18:45:37 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1E3456B0082; Sun, 1 Dec 2024 13:45:37 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 16CCD6B0083; Sun, 1 Dec 2024 13:45:37 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 00D026B0085; Sun, 1 Dec 2024 13:45:36 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id D03E96B0082 for ; Sun, 1 Dec 2024 13:45:36 -0500 (EST) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 53576160412 for ; Sun, 1 Dec 2024 18:45:36 +0000 (UTC) X-FDA: 82847268000.09.5DB3A79 Received: from mail-ed1-f54.google.com (mail-ed1-f54.google.com [209.85.208.54]) by imf06.hostedemail.com (Postfix) with ESMTP id 70D6C1811FF for ; Sun, 1 Dec 2024 18:45:26 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="BUbXrXn/"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf06.hostedemail.com: domain of 9erthalion6@gmail.com designates 209.85.208.54 as permitted sender) smtp.mailfrom=9erthalion6@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733078729; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=mW4xmEjchyEQzTB7gE+r/IBTnpsHdr6kuqGgY4jEsZg=; b=lJUmF4gmEB/bj6Uj3TJ3sAGWE+EM8DYVN180MvTOkzUVgyqNvJBb1df4ahztSgEIP7LVph J9guuwtbDCX8ydBZkNozJ43WXIV45qlNmqqsOZTuLMSJUic8LOgcmLfIQ0JO8P/kYHsLEl +tmU79glK0b1m5h2A2oSR0m00FdQ5tk= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733078729; a=rsa-sha256; cv=none; b=ei6sTjj7G3dV0IjdA46K0AFh0OngtHHfPPESnN05IDeyn//rBuQpyp8VBE0BaMCFWyYvQV ycwtD4Yq/3GSXsNAdeDCG17oUTbs2qUsFb4/TiIKPUN+ReQEqgq7UY1srnOt+v2N+dn7fj m5UkmfuhQ1bLLsfTyBoTd0BPD+ZdSXs= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b="BUbXrXn/"; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf06.hostedemail.com: domain of 9erthalion6@gmail.com designates 209.85.208.54 as permitted sender) smtp.mailfrom=9erthalion6@gmail.com Received: by mail-ed1-f54.google.com with SMTP id 4fb4d7f45d1cf-5d0c92e335bso1956562a12.2 for ; Sun, 01 Dec 2024 10:45:33 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1733078732; x=1733683532; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=mW4xmEjchyEQzTB7gE+r/IBTnpsHdr6kuqGgY4jEsZg=; b=BUbXrXn/mOnpt52aOUtAZ5Sg75peH/iHLNW6BVjWIOkRw+LRibUXLd/yY5FD+t66TB SRpXNSAGeh1WPRL8ZI5yUZLNLRWHems9+lHHIN9nV6M/pPtxDQi5r/C5OX2Qg1q1DO+i SYCdUpkM1GhdaTHlA+Q/eRxS6NJKhrtFgGHjBRY/V281eUeE+Zp44QBj9Kd0W8sgMwvf 4HLcZUJwDC5mjMijPo3Vi0telSGVvwCY+LXL2N5o+7Hagna6I4l4CPL+zgX/afdF83wt 0xV2JMUruZQUrKqY2FXDFhuGqFj0ySx3Q5PZwBd3iyR02y7/LkhII87Fqoy1QwclrCtJ j0yA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733078732; x=1733683532; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=mW4xmEjchyEQzTB7gE+r/IBTnpsHdr6kuqGgY4jEsZg=; b=l7u9SpcQcmXc584l28/YUd1JhDDVOH3Bo6zNzf70FMYAzjGwfqVrxo3C/OTwtbPVwn sW4QmDQEgVO9UcUJu6AXiyJqFryB853bxupF8mQh/bhWoZfU2qfqVGjtyx1fDNMqLmWk MQ3wbkdT96ugY30jCvG7rAKjjAwV0cANdepWEwQmkhBr2hY5DCaLqmPEopb54QWN2UQK SIWBo2/sLeDT1aNgQqK+otJUpt2L9DLsPP6EveLbtPQZ2bZITpxBLwpHC7hGoACQ3rIR gcnkwJjJhpCaBHIg3EAAKAZU6LQBFctJwcMb4/UH2HdbTioadJf4tuILVdD9qHbrJ7bR 2SSg== X-Gm-Message-State: AOJu0Yy+W0wxR6XGyFCPQNtenujatwBsVPoyDm6W3awBMTaynsUw4sZm NiYlWDuEiJ5Ghc0wHGizjFSnw9VAKgVCLjDYlsUKxJ8z5ZuwhUMO X-Gm-Gg: ASbGnct9b3VMsPXaYY7Q7hTjy7X6dBeXlp+2ln+WWihSHr/8kSaLmol9raXbLPvgL7z c4JmysFFwYrd3EfwmzLPVz+yBIthNxQiCS5ow34i+a1kkkPmuF71r1siefh/cdSHm5GSAndjpL+ NDH9jc3wnTOkaaMTaYS17Sd440TsX4zPvskKZSxNO5uYm2Snu6Qwl8tWr50+Yy0+BadEWhRaeBi E3kCixJ/dyfwEX2CbPa5bsftMKHygyfc0jUcWbNzLkLfDDtbnS3qTiuneueAwnbI4ECrjkRRuMH 0rlaSoZjI4YbH3meSdA5FwyGRn2e6UwpuHKNJg== X-Google-Smtp-Source: AGHT+IED02Yw28nqFVppQGzRkU8omejIZVyPy8dMlSyPs6zyg//2rPrkTeR6nyTi3S4Obhc/NGmA1Q== X-Received: by 2002:a05:6402:4342:b0:5d0:d5d1:359a with SMTP id 4fb4d7f45d1cf-5d0d5d13710mr5530969a12.28.1733078732091; Sun, 01 Dec 2024 10:45:32 -0800 (PST) Received: from erthalion.local (dslb-178-005-232-220.178.005.pools.vodafone-ip.de. [178.5.232.220]) by smtp.gmail.com with ESMTPSA id a640c23a62f3a-aa5998e6d96sm426446466b.126.2024.12.01.10.45.31 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Sun, 01 Dec 2024 10:45:31 -0800 (PST) Date: Sun, 1 Dec 2024 19:44:10 +0100 From: Dmitry Dolgov <9erthalion6@gmail.com> To: Matthew Wilcox Cc: linux-mm@kvack.org Subject: Re: [QUESTION] Resizing shared mapping without clashing with others Message-ID: <20241201184410.gl2huwqkbdwm6jvj@erthalion.local> References: <3kpxpd3dbjgg6epasi2554c4qyils4t3cm2pjnyzer7gkyoaxl@khhdxjiggyhp> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Stat-Signature: e38oamkdjtm15z1fenkbhondqcp3cgxp X-Rspamd-Queue-Id: 70D6C1811FF X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1733078726-769113 X-HE-Meta: U2FsdGVkX19XUxi9KJw8R04q+qZw9s54R1mQS2tteVTlhB+AEHiS9UshEgVfmMexnubCvyLZlmo1IXFOSm4FLfZfoNX99VNQUR1pDwR052pB2HKKwW7pg++dALA7ZX8W2UL1OiUkRwuZvGNyENdg1vwgBAhN33WIPAeXbUL1UP0DaeEsziuK4AWTr6wQYLv4n8JgRiE/hDW2AogutSrhrySQAUgZ/hVZbSCS8uWVTZD2M7AiyucmGYrsTwMRbcXgIuhUZ0deNT/WPFJXsRl6NcXK14D3kOxLXbZhNnfd1CgHxZ1Z7KzO2pTk73ON0KVcyBmAsK9I3n96x9iwiBGyQdpmekr7Nv3BpAMY5J/5dbzXMtD6vPE8Lah+PgTQ0hKrUUl10NnT894pKYXPv1EoUFQ6dm07TRIEbNNLwFFwHbn1dOjTNZe4X2Kb0+FTUEd3ydjnxCY9+J/YybeKuGhZupZ/WkoqxOvA/jMHEiCbe3ptKvL69sohnQF4x5xyJepCv5LY5UMU0DrJOUDZQFKkK7wTzYpP3nSvI4EssaqObX+WNfXkb7mKwN/VdhVsfRIcaoeZTyfQZ8+FF1OItH43SXKZfjQEi1RyWsNLq8yBH2x3qnC5lOgin/BfNaVOyEKZVhPc0ZhYPHXuPB+gScvmJdLlWfcUrL0bNKtCKUsXYyxbYARizpruo2MJjYB7W+KabHNtS17pa8Q7CyXUf6naRYknLJrthbC7YaI6ic7tackvRveyFlVyEg9w1ONejzux78KkqJUtpl7rrZ5C5GqOYfaDoBglJwcx0VeamhZCk/NN0UlYQk3qfiX/90zHGa45TKNZcE1Im+pN9JEvBPL/uMAz++gnPU59wv/Euidl3Di+PxgUGeDJLLPu/aDtxBD6GcpsCnY75JLWmH3BWzPEG0rBvoAxUQs7eagW3AzaaWtDgY5ysEJyjVcrqZnCqQHzglu/sKRL2OevSxG2jAn FxVz8bUN Y4RZAq6k9J6nuWVCNgqfrmhuptihGiyCBuyz8FKdnMDMSoD6zbM4IjFaM8PxKpVdCN+1JfsW/kOO7yeeG4cA9hcBnHnWET8MTJDL8LJ/yi0wyEOO7txnuUrqrcWJGuhgcT5ldrmUe35DAZ5ZLLbC0WfWJtQdzLmioiKKWMeN48kxdrGifLRx00eBklrnSJ+p5jHfud8n1+bk65CpvuI5nAgSQHXK0/Gi/iXpTe/pun/DqAa9vcVSy5lpkLohfONq/k6ifX2DZw1gYSTpHbf+0S7fbqn2mKdS9nTu9pvUxeDv+qR9eTVfcSYn9NTSxqUbC4jU5oOSgN/x4ctJmngbRXR+F2d2KLWMzYrye X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: > On Sun, Dec 01, 2024 at 11:55:37AM +0000, Matthew Wilcox wrote: > On Sat, Nov 30, 2024 at 05:24:13PM +0100, Dmitry Dolgov wrote: > > Hi, > > > > While working on PostgreSQL [1] we've stumbled upon a question regarding > > resizing of shared mappings without conflicting with any other possible > > mappings. Before making any wrong conclusions, I would love to get some > > consultation from kernel folks on that topic. > > > > To put it into a context, PostgreSQL uses anonymous shared memory > > mapping as a buffer cache for data. The mapping size is configured at > > the start, and could not be changed without a restart. Now, we would > > like to make it more flexible and allow to change it at runtime, ideally > > without changing already used addresses and copying stuff back and > > forth. > > > > The idea is to place the shared mapping at a specified address (with > > MAP_FIXED if needed) with a gap, then use mremap to resize it into the > > gap. This approach has an open question -- how to make sure there will > > be no other mapping created withing the same address space, where we > > want to expand the shared mapping? E.g. the shared mapping was created, > > then large memory allocation caused another mapping to be created close > > to it, so that expanding is not possible. > > I think there's a very straightforward answer, which is to mmap() it to > the larger size to begin with. If, say, you create a file of 1GB, you > can mmap() the first 100GB of that file. If you access the last 99GB of > the mapping, you'll get SIGBUS, but you can truncate() the file larger > and gain access to the new memory that way. Does that work for you? > > Or if you're doing MAP_ANON | MAP_SHARED, just don't access the last > 99GB until your configuration changes. Memory is allocated on demand, > so you won't be charged for it until you use it. Right, mapping with the larger size than needed is one option we're considering. But there are few arguments against that: * Folks are wary of unnecessary large shared mappings, since in the past there were issues with OOM killer making unfavorable to postgres decisions because of that. It might have changed over time, but to confirm that will require some investigation. * It can cause memory accounting problems. E.g. if we use hugetlb inside a cgroup with reservation limits set (something like hugetlb.2MB.rsvd.limit_in_bytes), then such mmap() will be counted against the limit, even though the memory wasn't allocated -- meaning that we claim some resource without using it.