[QUESTION] Resizing shared mapping without clashing with others

[QUESTION] Resizing shared mapping without clashing with others

[Dmitry Dolgov]

Hi,

While working on PostgreSQL [1] we've stumbled upon a question regarding
resizing of shared mappings without conflicting with any other possible
mappings. Before making any wrong conclusions, I would love to get some
consultation from kernel folks on that topic.

To put it into a context, PostgreSQL uses anonymous shared memory
mapping as a buffer cache for data. The mapping size is configured at
the start, and could not be changed without a restart. Now, we would
like to make it more flexible and allow to change it at runtime, ideally
without changing already used addresses and copying stuff back and
forth.

The idea is to place the shared mapping at a specified address (with
MAP_FIXED if needed) with a gap, then use mremap to resize it into the
gap. This approach has an open question -- how to make sure there will
be no other mapping created withing the same address space, where we
want to expand the shared mapping? E.g. the shared mapping was created,
then large memory allocation caused another mapping to be created close
to it, so that expanding is not possible.

One way to solve it would be to rely on the fact that the kernel, if
addr is NULL, always picks up a lowest address that allows for enough
space (AFAICT in vma_iter_area_lowest). This makes it possible to leave
the gap to lowest address large enough to accumulate any possible
allocations, as well as leaving space for resizing, something like:

	01339000-0139e000 [heap]
	0139e000-014aa000 [heap]
	7f2dd72f6000-7f2dfbc9c000 /memfd:strategy (deleted)
	7f2e0209c000-7f2e269b0000 /memfd:checkpoint (deleted)
	7f2e2cdb0000-7f2e516b4000 /memfd:iocv (deleted)
	7f2e57ab4000-7f2e7c478000 /memfd:descriptors (deleted)
	7f2ebc478000-7f2ee8d3c000 /memfd:buffers (deleted)
	^ note the distance between two mappings,
	  which is intended for resize
	7f3168d3c000-7f318d600000 /memfd:main (deleted)
	^ here is where the gap starts
	7f4194c00000-7f4194e7d000
	^ this one is an anonymous maping created due to large
	  memory allocation after shared mappings were created
	7f4195000000-7f419527d000
	7f41952dc000-7f4195416000
	7f4195416000-7f4195600000 /dev/shm/PostgreSQL.2529797530
	7f4195600000-7f41a311d000 /usr/lib/locale/locale-archive
	7f41a317f000-7f41a3200000
	7f41a3200000-7f41a3201000 /usr/lib64/libicudata.so.74.2
    [...]

What I'm not sure about is how safe it is to rely on such behavior in
long term? It seems like the way how mapping address will be chosen if
addr is NULL is not standartized, and considered to be implementation
specific. What are the chances this behavior becomes different over
time?

Another question is whether there are better ways or mechanisms to
achieve resizable shared mapping? The topic sounds natural enough, it
feels like there should be other use cases that require this as well.

[1]: https://www.postgresql.org/message-id/scor5gscd42d4nwszuwvtwss6e22fg3dnvxmqwrcsdkpyyigny%40efjlkj6ccv7u

Re: [QUESTION] Resizing shared mapping without clashing with others

[Matthew Wilcox]

On Sat, Nov 30, 2024 at 05:24:13PM +0100, Dmitry Dolgov wrote:
> Hi,
> 
> While working on PostgreSQL [1] we've stumbled upon a question regarding
> resizing of shared mappings without conflicting with any other possible
> mappings. Before making any wrong conclusions, I would love to get some
> consultation from kernel folks on that topic.
> 
> To put it into a context, PostgreSQL uses anonymous shared memory
> mapping as a buffer cache for data. The mapping size is configured at
> the start, and could not be changed without a restart. Now, we would
> like to make it more flexible and allow to change it at runtime, ideally
> without changing already used addresses and copying stuff back and
> forth.
> 
> The idea is to place the shared mapping at a specified address (with
> MAP_FIXED if needed) with a gap, then use mremap to resize it into the
> gap. This approach has an open question -- how to make sure there will
> be no other mapping created withing the same address space, where we
> want to expand the shared mapping? E.g. the shared mapping was created,
> then large memory allocation caused another mapping to be created close
> to it, so that expanding is not possible.

I think there's a very straightforward answer, which is to mmap() it to
the larger size to begin with.  If, say, you create a file of 1GB, you
can mmap() the first 100GB of that file.  If you access the last 99GB of
the mapping, you'll get SIGBUS, but you can truncate() the file larger
and gain access to the new memory that way.  Does that work for you?

Or if you're doing MAP_ANON | MAP_SHARED, just don't access the last
99GB until your configuration changes.  Memory is allocated on demand,
so you won't be charged for it until you use it.

Re: [QUESTION] Resizing shared mapping without clashing with others

[Dmitry Dolgov]

> On Sun, Dec 01, 2024 at 11:55:37AM +0000, Matthew Wilcox wrote:
> On Sat, Nov 30, 2024 at 05:24:13PM +0100, Dmitry Dolgov wrote:
> > Hi,
> >
> > While working on PostgreSQL [1] we've stumbled upon a question regarding
> > resizing of shared mappings without conflicting with any other possible
> > mappings. Before making any wrong conclusions, I would love to get some
> > consultation from kernel folks on that topic.
> >
> > To put it into a context, PostgreSQL uses anonymous shared memory
> > mapping as a buffer cache for data. The mapping size is configured at
> > the start, and could not be changed without a restart. Now, we would
> > like to make it more flexible and allow to change it at runtime, ideally
> > without changing already used addresses and copying stuff back and
> > forth.
> >
> > The idea is to place the shared mapping at a specified address (with
> > MAP_FIXED if needed) with a gap, then use mremap to resize it into the
> > gap. This approach has an open question -- how to make sure there will
> > be no other mapping created withing the same address space, where we
> > want to expand the shared mapping? E.g. the shared mapping was created,
> > then large memory allocation caused another mapping to be created close
> > to it, so that expanding is not possible.
>
> I think there's a very straightforward answer, which is to mmap() it to
> the larger size to begin with.  If, say, you create a file of 1GB, you
> can mmap() the first 100GB of that file.  If you access the last 99GB of
> the mapping, you'll get SIGBUS, but you can truncate() the file larger
> and gain access to the new memory that way.  Does that work for you?
>
> Or if you're doing MAP_ANON | MAP_SHARED, just don't access the last
> 99GB until your configuration changes.  Memory is allocated on demand,
> so you won't be charged for it until you use it.

Right, mapping with the larger size than needed is one option we're
considering. But there are few arguments against that:

* Folks are wary of unnecessary large shared mappings, since in the past
  there were issues with OOM killer making unfavorable to postgres
  decisions because of that. It might have changed over time, but to
  confirm that will require some investigation.

* It can cause memory accounting problems. E.g. if we use hugetlb inside
  a cgroup with reservation limits set (something like
  hugetlb.2MB.rsvd.limit_in_bytes), then such mmap() will be counted
  against the limit, even though the memory wasn't allocated -- meaning
  that we claim some resource without using it.

Re: [QUESTION] Resizing shared mapping without clashing with others

[Matthew Wilcox]

On Sun, Dec 01, 2024 at 07:44:10PM +0100, Dmitry Dolgov wrote:
> > On Sun, Dec 01, 2024 at 11:55:37AM +0000, Matthew Wilcox wrote:
> > On Sat, Nov 30, 2024 at 05:24:13PM +0100, Dmitry Dolgov wrote:
> > > Hi,
> > >
> > > While working on PostgreSQL [1] we've stumbled upon a question regarding
> > > resizing of shared mappings without conflicting with any other possible
> > > mappings. Before making any wrong conclusions, I would love to get some
> > > consultation from kernel folks on that topic.
> > >
> > > To put it into a context, PostgreSQL uses anonymous shared memory
> > > mapping as a buffer cache for data. The mapping size is configured at
> > > the start, and could not be changed without a restart. Now, we would
> > > like to make it more flexible and allow to change it at runtime, ideally
> > > without changing already used addresses and copying stuff back and
> > > forth.
> > >
> > > The idea is to place the shared mapping at a specified address (with
> > > MAP_FIXED if needed) with a gap, then use mremap to resize it into the
> > > gap. This approach has an open question -- how to make sure there will
> > > be no other mapping created withing the same address space, where we
> > > want to expand the shared mapping? E.g. the shared mapping was created,
> > > then large memory allocation caused another mapping to be created close
> > > to it, so that expanding is not possible.
> >
> > I think there's a very straightforward answer, which is to mmap() it to
> > the larger size to begin with.  If, say, you create a file of 1GB, you
> > can mmap() the first 100GB of that file.  If you access the last 99GB of
> > the mapping, you'll get SIGBUS, but you can truncate() the file larger
> > and gain access to the new memory that way.  Does that work for you?
> >
> > Or if you're doing MAP_ANON | MAP_SHARED, just don't access the last
> > 99GB until your configuration changes.  Memory is allocated on demand,
> > so you won't be charged for it until you use it.
> 
> Right, mapping with the larger size than needed is one option we're
> considering. But there are few arguments against that:
> 
> * Folks are wary of unnecessary large shared mappings, since in the past
>   there were issues with OOM killer making unfavorable to postgres
>   decisions because of that. It might have changed over time, but to
>   confirm that will require some investigation.
> 
> * It can cause memory accounting problems. E.g. if we use hugetlb inside
>   a cgroup with reservation limits set (something like
>   hugetlb.2MB.rsvd.limit_in_bytes), then such mmap() will be counted
>   against the limit, even though the memory wasn't allocated -- meaning
>   that we claim some resource without using it.

If it does turn out to be a problem, you can use a similar trick to how
ld.so maps binaries:

mmap(NULL, 2055640, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f221a758000
mmap(0x7f221a780000, 1462272, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x28000) = 0x7f221a780000
mmap(0x7f221a8e5000, 352256, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18d000) = 0x7f221a8e5000
mmap(0x7f221a93b000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e2000) = 0x7f221a93b000
mmap(0x7f221a941000, 52696, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f221a941000

Although you wouldn't want to do consecutive mmaps, you'd want to use
mremap() with MREMAP_FIXED -- not to change new_address, but to expand
length over the initial reserving-space mapping.

Re: [QUESTION] Resizing shared mapping without clashing with others

[David Hildenbrand]

On 30.11.24 17:24, Dmitry Dolgov wrote:
> Hi,

Hi,

> 
> While working on PostgreSQL [1] we've stumbled upon a question regarding
> resizing of shared mappings without conflicting with any other possible
> mappings. Before making any wrong conclusions, I would love to get some
> consultation from kernel folks on that topic.
> 
> To put it into a context, PostgreSQL uses anonymous shared memory
> mapping as a buffer cache for data. The mapping size is configured at
> the start, and could not be changed without a restart. Now, we would
> like to make it more flexible and allow to change it at runtime, ideally
> without changing already used addresses and copying stuff back and
> forth.
> 
> The idea is to place the shared mapping at a specified address (with
> MAP_FIXED if needed) with a gap, then use mremap to resize it into the
> gap. This approach has an open question -- how to make sure there will
> be no other mapping created withing the same address space, where we
> want to expand the shared mapping? E.g. the shared mapping was created,
> then large memory allocation caused another mapping to be created close
> to it, so that expanding is not possible.

Likely you want to reserve the not-used-yet portion using a 
MAP_PRIVATE|MA_ANON, PROT_NONE mapping.

> 
> One way to solve it would be to rely on the fact that the kernel, if
> addr is NULL, always picks up a lowest address that allows for enough
> space (AFAICT in vma_iter_area_lowest). This makes it possible to leave
> the gap to lowest address large enough to accumulate any possible
> allocations, as well as leaving space for resizing, something like:
> 
> 	01339000-0139e000 [heap]
> 	0139e000-014aa000 [heap]
> 	7f2dd72f6000-7f2dfbc9c000 /memfd:strategy (deleted)
> 	7f2e0209c000-7f2e269b0000 /memfd:checkpoint (deleted)
> 	7f2e2cdb0000-7f2e516b4000 /memfd:iocv (deleted)
> 	7f2e57ab4000-7f2e7c478000 /memfd:descriptors (deleted)
> 	7f2ebc478000-7f2ee8d3c000 /memfd:buffers (deleted)
> 	^ note the distance between two mappings,
> 	  which is intended for resize
> 	7f3168d3c000-7f318d600000 /memfd:main (deleted)
> 	^ here is where the gap starts
> 	7f4194c00000-7f4194e7d000
> 	^ this one is an anonymous maping created due to large
> 	  memory allocation after shared mappings were created
> 	7f4195000000-7f419527d000
> 	7f41952dc000-7f4195416000
> 	7f4195416000-7f4195600000 /dev/shm/PostgreSQL.2529797530
> 	7f4195600000-7f41a311d000 /usr/lib/locale/locale-archive
> 	7f41a317f000-7f41a3200000
> 	7f41a3200000-7f41a3201000 /usr/lib64/libicudata.so.74.2
>      [...]
> 
> What I'm not sure about is how safe it is to rely on such behavior in
> long term? It seems like the way how mapping address will be chosen if
> addr is NULL is not standartized, and considered to be implementation
> specific. What are the chances this behavior becomes different over
> time?
> 
> Another question is whether there are better ways or mechanisms to
> achieve resizable shared mapping? The topic sounds natural enough, it
> feels like there should be other use cases that require this as well.

Likely it's best to look into reserving a large VMA space using 
mmap(MAP_PRIVATE|MA_ANON, PROT_NONE); it's reserved but inaccessible, so 
it cannot get reallocated for different purposes. Then converting pieces 
of that into actually usable shared anonymous memory (e.g., MAP_FIXED).

-- 
Cheers,

David / dhildenb

Re: [QUESTION] Resizing shared mapping without clashing with others

[Dmitry Dolgov]

> On Sun, Dec 01, 2024 at 08:57:07PM GMT, Matthew Wilcox wrote:
> > Right, mapping with the larger size than needed is one option we're
> > considering. But there are few arguments against that:
> >
> > * Folks are wary of unnecessary large shared mappings, since in the past
> >   there were issues with OOM killer making unfavorable to postgres
> >   decisions because of that. It might have changed over time, but to
> >   confirm that will require some investigation.
> >
> > * It can cause memory accounting problems. E.g. if we use hugetlb inside
> >   a cgroup with reservation limits set (something like
> >   hugetlb.2MB.rsvd.limit_in_bytes), then such mmap() will be counted
> >   against the limit, even though the memory wasn't allocated -- meaning
> >   that we claim some resource without using it.
>
> If it does turn out to be a problem, you can use a similar trick to how
> ld.so maps binaries:
>
> mmap(NULL, 2055640, PROT_READ, MAP_PRIVATE|MAP_DENYWRITE, 3, 0) = 0x7f221a758000
> mmap(0x7f221a780000, 1462272, PROT_READ|PROT_EXEC, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x28000) = 0x7f221a780000
> mmap(0x7f221a8e5000, 352256, PROT_READ, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x18d000) = 0x7f221a8e5000
> mmap(0x7f221a93b000, 24576, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_DENYWRITE, 3, 0x1e2000) = 0x7f221a93b000
> mmap(0x7f221a941000, 52696, PROT_READ|PROT_WRITE, MAP_PRIVATE|MAP_FIXED|MAP_ANONYMOUS, -1, 0) = 0x7f221a941000
>
> Although you wouldn't want to do consecutive mmaps, you'd want to use
> mremap() with MREMAP_FIXED -- not to change new_address, but to expand
> length over the initial reserving-space mapping.

Hm, I don't follow how would that help? From what I understand the
suggestion is to have an initial mapping to "reserve" the space, right?
But this initial mapping would also be a subject of reservation limits,
mentioned above. I was originally experimenting with that, "reserving"
some mapping space with PROT_NONE, then slicing off chunks of it for
real usage -- but in case of hugetlb and a cgroup it was accounting
against the reservation limits for huge pages.

Re: [QUESTION] Resizing shared mapping without clashing with others

[Dmitry Dolgov]

> On Mon, Dec 02, 2024 at 12:07:01PM GMT, David Hildenbrand wrote:
>
> Likely it's best to look into reserving a large VMA space using
> mmap(MAP_PRIVATE|MA_ANON, PROT_NONE); it's reserved but inaccessible, so it
> cannot get reallocated for different purposes. Then converting pieces of
> that into actually usable shared anonymous memory (e.g., MAP_FIXED).

Yes, we've considered this option, preparing an initial "reserving"
mapping, then take pieces out of it for usage. As I've mentioned to
Matthew, there seems to be arguments against this approach. In a few
words:

* The impact on OOM decisions is not clear (at least to me).
* In certain cases this will be counted against a reservation limits,
  although we don't use the memory yet (e.g. with hugetlb insige a
  cgroup with hugetlb.2MB.rsvd.limit_in_bytes set).

Re: [QUESTION] Resizing shared mapping without clashing with others

[David Hildenbrand]

On 02.12.24 16:04, Dmitry Dolgov wrote:
>> On Mon, Dec 02, 2024 at 12:07:01PM GMT, David Hildenbrand wrote:
>>
>> Likely it's best to look into reserving a large VMA space using
>> mmap(MAP_PRIVATE|MA_ANON, PROT_NONE); it's reserved but inaccessible, so it
>> cannot get reallocated for different purposes. Then converting pieces of
>> that into actually usable shared anonymous memory (e.g., MAP_FIXED).
> 
> Yes, we've considered this option, preparing an initial "reserving"
> mapping, then take pieces out of it for usage. As I've mentioned to
> Matthew, there seems to be arguments against this approach. In a few
> words:

Note that this approach is extremely common ;)

> 
> * The impact on OOM decisions is not clear (at least to me).

There are none. mmap(MAP_PRIVATE|MAP_ANON, PROT_NONE) will neither 
allocate memory/page tables nor commit memory.

> * In certain cases this will be counted against a reservation limits,
>    although we don't use the memory yet (e.g. with hugetlb insige a
>    cgroup with hugetlb.2MB.rsvd.limit_in_bytes set).


With  mmap(MAP_PRIVATE|MAP_ANON, PROT_NONE)? Unlikely. In any case, you 
might be able to tame reservations using MAP_NORESERVE.


-- 
Cheers,

David / dhildenb

Re: [QUESTION] Resizing shared mapping without clashing with others

[Dmitry Dolgov]

> On Mon, Dec 02, 2024 at 04:40:30PM GMT, David Hildenbrand wrote:
> On 02.12.24 16:04, Dmitry Dolgov wrote:
> > > On Mon, Dec 02, 2024 at 12:07:01PM GMT, David Hildenbrand wrote:
> > >
> > > Likely it's best to look into reserving a large VMA space using
> > > mmap(MAP_PRIVATE|MA_ANON, PROT_NONE); it's reserved but inaccessible, so it
> > > cannot get reallocated for different purposes. Then converting pieces of
> > > that into actually usable shared anonymous memory (e.g., MAP_FIXED).
> >
> > Yes, we've considered this option, preparing an initial "reserving"
> > mapping, then take pieces out of it for usage. As I've mentioned to
> > Matthew, there seems to be arguments against this approach. In a few
> > words:
>
> Note that this approach is extremely common ;)

That's why I'm asking your folks, trying to get some wisdom :) Although
I have to translated this knowledge it into the database world, is it
common for access patterns like we see in databases (large amount of
memory used as a buffer cache) as well?

> > * In certain cases this will be counted against a reservation limits,
> >    although we don't use the memory yet (e.g. with hugetlb insige a
> >    cgroup with hugetlb.2MB.rsvd.limit_in_bytes set).
>
>
> With  mmap(MAP_PRIVATE|MAP_ANON, PROT_NONE)? Unlikely. In any case, you
> might be able to tame reservations using MAP_NORESERVE.

Oh, interesting, I didn't realize this flag controls that as well. This
might just work, although there are still annoying parts -- postgres
actually needs reservation limits checking at the start to figure out of
any huge pages are available at all (there were too many complains about
getting SIGBUS, when limitations applied at page fault time). But maybe
I can work something out. Thanks!