From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9D878D7830D for ; Mon, 2 Dec 2024 11:07:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 033896B0082; Mon, 2 Dec 2024 06:07:12 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F24D66B0083; Mon, 2 Dec 2024 06:07:11 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id DECF06B0085; Mon, 2 Dec 2024 06:07:11 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id C203B6B0082 for ; Mon, 2 Dec 2024 06:07:11 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 41B6B1A1CF0 for ; Mon, 2 Dec 2024 11:07:11 +0000 (UTC) X-FDA: 82849741716.08.26D4263 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf18.hostedemail.com (Postfix) with ESMTP id 68BD91C0018 for ; Mon, 2 Dec 2024 11:07:03 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=H6ZU0osI; spf=pass (imf18.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1733137616; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=S90wULdp3Bn2kRAnFlq39BZxpx0zGo10GM4yz+pYdWg=; b=QjnDwqQU9J6bRd4WmCoRW2K9+O4vJrWgNyetLbT0r9RqqyG1pvzL742qsOx+IfVYvxOT/S ZjA5XFOS2Mhc4mICA9KgSFCjMpskToLcN9N0hRsqbqng2m9fXWL3onma7jgZgSYPSxdNYy naDE/tnmplek6BKs+8Lr4E/MkWgPAk8= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1733137616; a=rsa-sha256; cv=none; b=Hx6EK15s5/HlJFZ/uAptVRwnvqYLzgvKETGPKnFq9TURFKIaDoA4R1VZEUAzg+teh94tJq 1LUZdf7pYec61zMYzdT5LK+5ooABZqvzh5uahsSZwnq/yozNWnML8U1AX31yiLyWEY98/o PwRbLNyDdZZy1rCgmvgvwHRdeQPAHGI= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=H6ZU0osI; spf=pass (imf18.hostedemail.com: domain of david@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=david@redhat.com; dmarc=pass (policy=none) header.from=redhat.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1733137628; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:autocrypt:autocrypt; bh=S90wULdp3Bn2kRAnFlq39BZxpx0zGo10GM4yz+pYdWg=; b=H6ZU0osIfpxC5sI9eXBJP7X/Xdy5aqKJ9yiOQz3Klscn2WP0uc4SFYMbIFrUdMIDe2fIvj okGoP5xTEQSJnKdpRGybu+9VkjOQUqnt/HB8XUApK48ek594xqGDXWHm50KVPTNPknIcuv h58zsEFy84GKmeD/ajYklylYbsQiPUg= Received: from mail-lf1-f72.google.com (mail-lf1-f72.google.com [209.85.167.72]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-595-_lXAFdDpMYSua1sND7jvHQ-1; Mon, 02 Dec 2024 06:07:06 -0500 X-MC-Unique: _lXAFdDpMYSua1sND7jvHQ-1 X-Mimecast-MFC-AGG-ID: _lXAFdDpMYSua1sND7jvHQ Received: by mail-lf1-f72.google.com with SMTP id 2adb3069b0e04-53ddbee66c8so2521158e87.0 for ; Mon, 02 Dec 2024 03:07:06 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1733137625; x=1733742425; h=content-transfer-encoding:in-reply-to:organization:autocrypt:from :content-language:references:to:subject:user-agent:mime-version:date :message-id:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=S90wULdp3Bn2kRAnFlq39BZxpx0zGo10GM4yz+pYdWg=; b=DwlH+nSjFqIh8cf1X/53Mg9HpP+vwJkM//BcIHqvR9gS92ZQoVCW2E+40y3aSIqzus KDkrA5a35XH55Qa5RIfOqNDgtaVHi6Q7dAUbezhjdUMwDKBFvhOQEvosXOxAYitpfFVo jPzDBNmugXdwP88mZJi8ocb+DA/VCqeZmgaqVivCVGgW+8AGTIneLdaCWy0SKOcVEU5V zx3YKbN4keDtPNMrw8WfMxFGiOcT8v1WyU9bvUWmfYNLkz7RLNDMEqo0o9k/6a2W4lDL hkyKO4KLmZ5ax9QNr3hPFQUHtqZOK8SMaUVydaI/7ixWQZOckPVyv2HaWsduHInC8f2d TjOQ== X-Forwarded-Encrypted: i=1; AJvYcCV/kTLAVxwFxYhhjeGcgvjZ9VaGyonzZo13aAOwwZOsmpy9lr6SjHt/R6TeXQms+5ig7B0PqkfW5w==@kvack.org X-Gm-Message-State: AOJu0YydNbV9S+qWZWevoaFESYXDgcxrfStpdhsdruOsO//i3qcLOE9f 2zpJgghX3KcEqV6XHydtDbplS/a6ogwFR3n2zkxzUizE1LjguwA/vW4toM5tDIOc79WwfWvcczI oLQrWqoTrWLmabNx+5TKxcDU/SJ1kxgY2e8bHNJJNqndpw4kd X-Gm-Gg: ASbGncsIFq3o7fuvvAHXfWzTmbACkO+Vik41nHVHCsao5tA/r5LxLDQeIfXj3JB0K1h TEyayAqDU2yTrUNE3Xg1wWrTZqFV5wfv2CwlXq+ZQiO/qw8ZDrhanwz4EP6Gl4dLqu4FJ1Q9X8v gEe2RWx/Ky7ZoaXCG83OKccXvyAaYH7eVkA7FY3n3KkhfhbHDbP6TRBq9Xi8yIMNZQ7qj7Q8Rjx VLcy3j+BYA+UhtdosRM/puE5f2z7rXo8mvU/t0OOIaJqep/i10xioB7pf1ahnidbZQyJUbk16d4 oZkugd8q X-Received: by 2002:a05:6512:3085:b0:539:e9f8:d45d with SMTP id 2adb3069b0e04-53df01125d5mr11071328e87.52.1733137625318; Mon, 02 Dec 2024 03:07:05 -0800 (PST) X-Google-Smtp-Source: AGHT+IHkDQ2pVDPa3VgDkBJv5Gxa+L2ULkkgtQ41AlJd0SHznIlu3MXYC4fUcXLva4ytZiaiBbjdUA== X-Received: by 2002:a05:6512:3085:b0:539:e9f8:d45d with SMTP id 2adb3069b0e04-53df01125d5mr11071299e87.52.1733137624842; Mon, 02 Dec 2024 03:07:04 -0800 (PST) Received: from ?IPV6:2a09:80c0:192:0:5dac:bf3d:c41:c3e7? ([2a09:80c0:192:0:5dac:bf3d:c41:c3e7]) by smtp.gmail.com with ESMTPSA id 5b1f17b1804b1-434aa76a981sm178599385e9.16.2024.12.02.03.07.02 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Mon, 02 Dec 2024 03:07:04 -0800 (PST) Message-ID: Date: Mon, 2 Dec 2024 12:07:01 +0100 MIME-Version: 1.0 User-Agent: Mozilla Thunderbird Subject: Re: [QUESTION] Resizing shared mapping without clashing with others To: Dmitry Dolgov <9erthalion6@gmail.com>, linux-mm@kvack.org References: <3kpxpd3dbjgg6epasi2554c4qyils4t3cm2pjnyzer7gkyoaxl@khhdxjiggyhp> From: David Hildenbrand Autocrypt: addr=david@redhat.com; keydata= xsFNBFXLn5EBEAC+zYvAFJxCBY9Tr1xZgcESmxVNI/0ffzE/ZQOiHJl6mGkmA1R7/uUpiCjJ dBrn+lhhOYjjNefFQou6478faXE6o2AhmebqT4KiQoUQFV4R7y1KMEKoSyy8hQaK1umALTdL QZLQMzNE74ap+GDK0wnacPQFpcG1AE9RMq3aeErY5tujekBS32jfC/7AnH7I0v1v1TbbK3Gp XNeiN4QroO+5qaSr0ID2sz5jtBLRb15RMre27E1ImpaIv2Jw8NJgW0k/D1RyKCwaTsgRdwuK Kx/Y91XuSBdz0uOyU/S8kM1+ag0wvsGlpBVxRR/xw/E8M7TEwuCZQArqqTCmkG6HGcXFT0V9 PXFNNgV5jXMQRwU0O/ztJIQqsE5LsUomE//bLwzj9IVsaQpKDqW6TAPjcdBDPLHvriq7kGjt WhVhdl0qEYB8lkBEU7V2Yb+SYhmhpDrti9Fq1EsmhiHSkxJcGREoMK/63r9WLZYI3+4W2rAc UucZa4OT27U5ZISjNg3Ev0rxU5UH2/pT4wJCfxwocmqaRr6UYmrtZmND89X0KigoFD/XSeVv jwBRNjPAubK9/k5NoRrYqztM9W6sJqrH8+UWZ1Idd/DdmogJh0gNC0+N42Za9yBRURfIdKSb B3JfpUqcWwE7vUaYrHG1nw54pLUoPG6sAA7Mehl3nd4pZUALHwARAQABzSREYXZpZCBIaWxk ZW5icmFuZCA8ZGF2aWRAcmVkaGF0LmNvbT7CwZgEEwEIAEICGwMGCwkIBwMCBhUIAgkKCwQW AgMBAh4BAheAAhkBFiEEG9nKrXNcTDpGDfzKTd4Q9wD/g1oFAl8Ox4kFCRKpKXgACgkQTd4Q 9wD/g1oHcA//a6Tj7SBNjFNM1iNhWUo1lxAja0lpSodSnB2g4FCZ4R61SBR4l/psBL73xktp rDHrx4aSpwkRP6Epu6mLvhlfjmkRG4OynJ5HG1gfv7RJJfnUdUM1z5kdS8JBrOhMJS2c/gPf wv1TGRq2XdMPnfY2o0CxRqpcLkx4vBODvJGl2mQyJF/gPepdDfcT8/PY9BJ7FL6Hrq1gnAo4 3Iv9qV0JiT2wmZciNyYQhmA1V6dyTRiQ4YAc31zOo2IM+xisPzeSHgw3ONY/XhYvfZ9r7W1l pNQdc2G+o4Di9NPFHQQhDw3YTRR1opJaTlRDzxYxzU6ZnUUBghxt9cwUWTpfCktkMZiPSDGd KgQBjnweV2jw9UOTxjb4LXqDjmSNkjDdQUOU69jGMUXgihvo4zhYcMX8F5gWdRtMR7DzW/YE BgVcyxNkMIXoY1aYj6npHYiNQesQlqjU6azjbH70/SXKM5tNRplgW8TNprMDuntdvV9wNkFs 9TyM02V5aWxFfI42+aivc4KEw69SE9KXwC7FSf5wXzuTot97N9Phj/Z3+jx443jo2NR34XgF 89cct7wJMjOF7bBefo0fPPZQuIma0Zym71cP61OP/i11ahNye6HGKfxGCOcs5wW9kRQEk8P9 M/k2wt3mt/fCQnuP/mWutNPt95w9wSsUyATLmtNrwccz63XOwU0EVcufkQEQAOfX3n0g0fZz Bgm/S2zF/kxQKCEKP8ID+Vz8sy2GpDvveBq4H2Y34XWsT1zLJdvqPI4af4ZSMxuerWjXbVWb T6d4odQIG0fKx4F8NccDqbgHeZRNajXeeJ3R7gAzvWvQNLz4piHrO/B4tf8svmRBL0ZB5P5A 2uhdwLU3NZuK22zpNn4is87BPWF8HhY0L5fafgDMOqnf4guJVJPYNPhUFzXUbPqOKOkL8ojk CXxkOFHAbjstSK5Ca3fKquY3rdX3DNo+EL7FvAiw1mUtS+5GeYE+RMnDCsVFm/C7kY8c2d0G NWkB9pJM5+mnIoFNxy7YBcldYATVeOHoY4LyaUWNnAvFYWp08dHWfZo9WCiJMuTfgtH9tc75 7QanMVdPt6fDK8UUXIBLQ2TWr/sQKE9xtFuEmoQGlE1l6bGaDnnMLcYu+Asp3kDT0w4zYGsx 5r6XQVRH4+5N6eHZiaeYtFOujp5n+pjBaQK7wUUjDilPQ5QMzIuCL4YjVoylWiBNknvQWBXS lQCWmavOT9sttGQXdPCC5ynI+1ymZC1ORZKANLnRAb0NH/UCzcsstw2TAkFnMEbo9Zu9w7Kv AxBQXWeXhJI9XQssfrf4Gusdqx8nPEpfOqCtbbwJMATbHyqLt7/oz/5deGuwxgb65pWIzufa N7eop7uh+6bezi+rugUI+w6DABEBAAHCwXwEGAEIACYCGwwWIQQb2cqtc1xMOkYN/MpN3hD3 AP+DWgUCXw7HsgUJEqkpoQAKCRBN3hD3AP+DWrrpD/4qS3dyVRxDcDHIlmguXjC1Q5tZTwNB boaBTPHSy/Nksu0eY7x6HfQJ3xajVH32Ms6t1trDQmPx2iP5+7iDsb7OKAb5eOS8h+BEBDeq 3ecsQDv0fFJOA9ag5O3LLNk+3x3q7e0uo06XMaY7UHS341ozXUUI7wC7iKfoUTv03iO9El5f XpNMx/YrIMduZ2+nd9Di7o5+KIwlb2mAB9sTNHdMrXesX8eBL6T9b+MZJk+mZuPxKNVfEQMQ a5SxUEADIPQTPNvBewdeI80yeOCrN+Zzwy/Mrx9EPeu59Y5vSJOx/z6OUImD/GhX7Xvkt3kq Er5KTrJz3++B6SH9pum9PuoE/k+nntJkNMmQpR4MCBaV/J9gIOPGodDKnjdng+mXliF3Ptu6 3oxc2RCyGzTlxyMwuc2U5Q7KtUNTdDe8T0uE+9b8BLMVQDDfJjqY0VVqSUwImzTDLX9S4g/8 kC4HRcclk8hpyhY2jKGluZO0awwTIMgVEzmTyBphDg/Gx7dZU1Xf8HFuE+UZ5UDHDTnwgv7E th6RC9+WrhDNspZ9fJjKWRbveQgUFCpe1sa77LAw+XFrKmBHXp9ZVIe90RMe2tRL06BGiRZr jPrnvUsUUsjRoRNJjKKA/REq+sAnhkNPPZ/NNMjaZ5b8Tovi8C0tmxiCHaQYqj7G2rgnT0kt WNyWQQ== Organization: Red Hat In-Reply-To: <3kpxpd3dbjgg6epasi2554c4qyils4t3cm2pjnyzer7gkyoaxl@khhdxjiggyhp> X-Mimecast-Spam-Score: 0 X-Mimecast-MFC-PROC-ID: ewXbaaS32t_b3sRVJL6XUeV8YUV8Kozir-ijim42Akc_1733137625 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 68BD91C0018 X-Rspam-User: X-Rspamd-Server: rspam07 X-Stat-Signature: fjws8ubt1mp5mubc5qcgmgzcnkmimp43 X-HE-Tag: 1733137623-558898 X-HE-Meta: U2FsdGVkX184wdF68/Z2xV4olPTLdZuCdsSHA6QgwiSKdDnrI2Lki4BfkBubxFHtHhoQ2gq55URFS94C6iykACPy1C9pwc6E5ijh5PRv4D+cFxH+LHH6bSkm6iENGZltpfNJn9i4orNF4BlY84WU4yWbPjjGdycr0II97nvCMCGDSTL67GXVtIQQzPcwTTO3bPbXOx/An0BG1InQvlbsoE9UccVJcsfFemCsvjFcpN5lFth3hX3Up+oS9a8I+mdJkexeOvaIzuI1mFCaATaHxdRDv1pSfwosFoZf/DsQri8qufHywbnZYsvJP5JtcFJeDoTyaIGvlCZxKvLfs2fd7ymUBu09NVUTgXlm19EWPTh5k4teVqylIaEwiaTGNY/uWPW0iAA45+mABnhIl/dDdkvkf1qjd94+97Ip2jjCiKXA4Ztergz70DxVKHtBeY1TMd472ocyyZjG70loW8twM+51HleKbH66GHdFnApDeESZBT3j2lxD7PU+vcmm1vBu9m5LVWuFnUjVyM0id8zd0JDtWV1juLhJ/L3/YvMasEY8yvrT4bY2EYleFKe5pyytLaPd803TKynYag8yXm3IXX9ZMDRiIzNkcN16iCF6W/nOPnT2KCiJ0gc4bSFHcu8swTWkpfO5ZG2zKcjgjCq5nWaKeqOJ+MPq+MT8IQ7JpIRlKE3oYlxdarKjxUfK3qYaNxuuKj/BUhMJ9eAWzDQQWg9FUk4ilSg/Jujti8GuKyYI2xEjHGBHLdey91Up9xbMKgpQPYxz1hYI4dtqXF/XbfCX8Lm53Al2wqGIb4TnsIYXKpdIU0TRPFl891iVptoSE9eTH8W9LC6sMqBr9IZzQGIqU2glB5oZTH7K6+ahXxHIIvu6sLz0gmtcwoLX84zLdVcVhUYsNVSGFyzBFG8MTif33I9xmzraejSz7WkjtcRVC0wfeC4cVXi1zg46TnAvFqpZI05WseAZlgcTgmy kKXktI/A NkTkfjQd5m04IsvXhD8Nn4ZpdX2DbH+yxPfkzjG36Hvpc+yLPepNyCZWpdvPK4AlsSjoMcmS4cz2+P4WWQ7YwvfjA+MiCq7BM60AmELWXgmx/LabKCGhIwMJPxN2EAjtmvjPTHKEmPbATrwOHjX/OKil8rmwsMYxQC4/uuMKVZF/CLFRnQ6JPTxpRxd5uCLQ/yYPFxMYIQOgQPhjVJYtyV5bgeum2wrHpETDbI+IcsvexcSiv5+eUOycjmqjdoTUpkMejOIsIN+ZI9qh4OCn2CcwU0jYYr5zGaiHGbi1vDx9zYTHxvbgM6UHiwJrnEd6ejT2xO4LTyEOmSkn537RHo1it0x8yAhX1czY1SFQYd8baPs3cJT8/v+JsTtCLQjsUGctLDUyiIX6LaSWFWFaXYkLTuEOeo/Id6JDy9uWd8Cv5prymuhBMExyTOfAJBZS6SfoDVJ7ByNrRAmQ= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On 30.11.24 17:24, Dmitry Dolgov wrote: > Hi, Hi, > > While working on PostgreSQL [1] we've stumbled upon a question regarding > resizing of shared mappings without conflicting with any other possible > mappings. Before making any wrong conclusions, I would love to get some > consultation from kernel folks on that topic. > > To put it into a context, PostgreSQL uses anonymous shared memory > mapping as a buffer cache for data. The mapping size is configured at > the start, and could not be changed without a restart. Now, we would > like to make it more flexible and allow to change it at runtime, ideally > without changing already used addresses and copying stuff back and > forth. > > The idea is to place the shared mapping at a specified address (with > MAP_FIXED if needed) with a gap, then use mremap to resize it into the > gap. This approach has an open question -- how to make sure there will > be no other mapping created withing the same address space, where we > want to expand the shared mapping? E.g. the shared mapping was created, > then large memory allocation caused another mapping to be created close > to it, so that expanding is not possible. Likely you want to reserve the not-used-yet portion using a MAP_PRIVATE|MA_ANON, PROT_NONE mapping. > > One way to solve it would be to rely on the fact that the kernel, if > addr is NULL, always picks up a lowest address that allows for enough > space (AFAICT in vma_iter_area_lowest). This makes it possible to leave > the gap to lowest address large enough to accumulate any possible > allocations, as well as leaving space for resizing, something like: > > 01339000-0139e000 [heap] > 0139e000-014aa000 [heap] > 7f2dd72f6000-7f2dfbc9c000 /memfd:strategy (deleted) > 7f2e0209c000-7f2e269b0000 /memfd:checkpoint (deleted) > 7f2e2cdb0000-7f2e516b4000 /memfd:iocv (deleted) > 7f2e57ab4000-7f2e7c478000 /memfd:descriptors (deleted) > 7f2ebc478000-7f2ee8d3c000 /memfd:buffers (deleted) > ^ note the distance between two mappings, > which is intended for resize > 7f3168d3c000-7f318d600000 /memfd:main (deleted) > ^ here is where the gap starts > 7f4194c00000-7f4194e7d000 > ^ this one is an anonymous maping created due to large > memory allocation after shared mappings were created > 7f4195000000-7f419527d000 > 7f41952dc000-7f4195416000 > 7f4195416000-7f4195600000 /dev/shm/PostgreSQL.2529797530 > 7f4195600000-7f41a311d000 /usr/lib/locale/locale-archive > 7f41a317f000-7f41a3200000 > 7f41a3200000-7f41a3201000 /usr/lib64/libicudata.so.74.2 > [...] > > What I'm not sure about is how safe it is to rely on such behavior in > long term? It seems like the way how mapping address will be chosen if > addr is NULL is not standartized, and considered to be implementation > specific. What are the chances this behavior becomes different over > time? > > Another question is whether there are better ways or mechanisms to > achieve resizable shared mapping? The topic sounds natural enough, it > feels like there should be other use cases that require this as well. Likely it's best to look into reserving a large VMA space using mmap(MAP_PRIVATE|MA_ANON, PROT_NONE); it's reserved but inaccessible, so it cannot get reallocated for different purposes. Then converting pieces of that into actually usable shared anonymous memory (e.g., MAP_FIXED). -- Cheers, David / dhildenb