Re: [RFC PATCH ] mm/slub: Reducing slub memory wastage

[Jay Patel <jaypatel@linux.ibm.com>]

On Mon, 2023-06-12 at 19:17 +0900, Hyeonggon Yoo wrote:
> On Mon, Jun 12, 2023 at 02:25:35PM +0530, Jay Patel wrote:
> > 3) If the minimum order is less than the slub_max_order, iterate
> > through
> > a loop from minimum order to slub_max_order and check if the
> > condition
> > (rem <= slab_size / fract_leftover) holds true. Here, slab_size is
> > calculated as (PAGE_SIZE << order), rem is (slab_size %
> > object_size),
> > and fract_leftover can have values of 16, 8, or 4. If the condition
> > is
> > true, select that order for the slab.
> > 
> > However, in point 3, when calculating the fraction left over, it
> > can
> > result in a large range of values (like 1 Kb to 256 bytes on 4K
> > page
> > size & 4 Kb to 16 Kb on 64K page size with order 0 and goes on
> > increasing with higher order) when compared to the remainder (rem).
> > This
> > can lead to the selection of an order that results in more memory
> > wastage. To mitigate such wastage, we have modified point 3 as
> > follows:
> > instead of selecting the first order that satisfies the condition
> > (rem
> > <= slab_size / fract_leftover), we iterate through the loop from
> > min_order to slub_max_order and choose the order that minimizes
> > memory
> > wastage for the slab.
> 
> Hi Jay,
> 
> If understand correctly, slub currently chooses an order if it
> does not waste too much memory, but the order could be sub-optimal
> because there can be an order that wastes less memory. right?
> 
> Hmm, the new code might choose larger order than before, as SLUB
> previously
> wasted more memory instead of increasing order.
> 
> BUT the maximum slub order is still bound by slub_max_order,
> so that looks fine to me. If using high order for less fragmentation
> becomes a problem, slub_max_order should be changed.
> 
> 
Hi Hyeonggon

Based on my understanding, the slub_max_order parameter is derived from
the PAGE_ALLOC_COSTLY_ORDER configuration option. Any changes made to
can have an impact on system performance.

If you reduce the value of slub_max_order, it means that smaller
contiguous memory will be used for certain slab caches. However, this
can lead to a situation where the minimum number of objects required
(min_objects) cannot fit within those smaller memory. As a result,
performance issues may arise.
> <...snip...> 
> 
> > I conducted tests on systems with 160 CPUs and 16 CPUs, using  4K
> > and
> > 64K page sizes. Through these tests, it was observed that the patch
> > successfully reduces the wastage of slab memory without any
> > noticeable
> > performance degradation in the hackbench test report. However, it
> > should
> > be noted that the patch also increases the total number of objects,
> > leading to an overall increase in total slab memory usage.
> 
> <...snip...>
> 
> Then my question is that, why is this a useful change if total memory
> usage is increased?
> 
This patch aimed in reducing memory wastage can potentially lead to an
increase in the slab order for a slab cache. Consequently, this
increase in page order can result in a higher number of objects per
slab, reducing wastage and leading to a more efficient utilization of
memory. This enhancement is advantageous since the presence of unused
objects can be leveraged in the future, depending on varying
workloads. 
> 
> > Test results are as follows:
> > 3) On 16 CPUs with 4K Page size
> > 
> > +-----------------+----------------+------------------+
> > >          Total wastage in slub memory               |
> > +-----------------+----------------+------------------+
> > >                 | After Boot     | After Hackbench  |
> > > Normal          | 666 Kb         | 902 Kb           |
> > > With Patch      | 533 Kb         | 694 Kb           |
> > > Wastage reduce  | ~20%           | ~23%             |
> > +-----------------+----------------+------------------+
> > 
> > +-----------------+----------------+----------------+
> > >            Total slub memory                      |
> > +-----------------+----------------+----------------+
> > >                 | After Boot     | After Hackbench|
> > > Normal          | 82360          | 122532         |
> > > With Patch      | 87372          | 129180         |
> > > Memory increase | ~6%            | ~5%            |
> > +-----------------+----------------+----------------+
> > 
> 
> How should we understand this data?
> reducing amount of memory wastage by increasing slab order
> might not reduce total SLUB memory usage?
> 

Indeed, the total slub memory is increase with this patch. However, the
memory utilization is improved. The slub memory comprises both active
objects and unused objects, along with memory wastage. With this patch,
the memory wastage is reduced, leading to a higher number of unused
objects (the numbers of active objects mostly remains the same). The
presence of these unused objects provides the opportunity for their
utilization, which can vary depending on different workloads.

> > hackbench-process-sockets
> > +-------+----+---------+---------+-----------+
> > > Amean | 1  | 1.4983  | 1.4867  | ( 0.78%)  |
> > > Amean | 4  | 5.6613  | 5.6793  | ( -0.32%) |
> > > Amean | 7  | 9.9813  | 9.9873  | ( -0.06%) |
> > > Amean | 12 | 17.6963 | 17.8527 | ( -0.88%) |
> > > Amean | 21 | 31.2017 | 31.2060 | ( -0.01%) |
> > > Amean | 30 | 44.0297 | 44.1750 | ( -0.33%) |
> > > Amean | 48 | 70.2073 | 69.6210 | ( 0.84%)  |
> > > Amean | 64 | 92.3257 | 93.7410 | ( -1.53%) |
> > +-------+----+---------+---------+-----------+

--
Jay Patel