How to controll Buffers to be dilligently reclaimed?

How to controll Buffers to be dilligently reclaimed?

[Cheng Rk]

Hi,

I have a system that application is doing a loop on top of block device,
(which I think is stupid,)
as more and more memory goes into Buffers, then applications started
to get -ENOMEM or be oom-killed later (depends on vm.overcommit_memory setting)


In this case, if I do a manual reclaim (echo 3 > /proc/sys/vm/drop_caches)
I see 90+% of the Buffers is reclaimable, but why it's not reclaimed
to fullfill applications' memory allocation request?



-bash-4.2$ sudo losetup -a
/dev/loop0: [0005]:16512 (/dev/dm-2)
-bash-4.2$ free -m
                     total          used         free      shared       buffers     cached
Mem:             48094        46081         2012              40           40324   2085
-/+ buffers/cache:             3671        44422
Swap:             8191                5         8186


I've tried sysctl mm.vfs_cache_pressure=10000 but that seems working to Cached
memory, I wonder is there another sysctl for reclaming Buffers?


Thanks,

- Derek

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: How to controll Buffers to be dilligently reclaimed?

[Konstantin Khlebnikov]

On Fri, Feb 13, 2015 at 3:52 AM, Cheng Rk <crquan@ymail.com> wrote:
>
>
> Hi,
>
> I have a system that application is doing a loop on top of block device,
> (which I think is stupid,)
> as more and more memory goes into Buffers, then applications started
> to get -ENOMEM or be oom-killed later (depends on vm.overcommit_memory setting)
>
>
> In this case, if I do a manual reclaim (echo 3 > /proc/sys/vm/drop_caches)
> I see 90+% of the Buffers is reclaimable, but why it's not reclaimed
> to fullfill applications' memory allocation request?
>
>
>
> -bash-4.2$ sudo losetup -a
> /dev/loop0: [0005]:16512 (/dev/dm-2)
> -bash-4.2$ free -m
>                      total          used         free      shared       buffers     cached
> Mem:             48094        46081         2012              40           40324   2085
> -/+ buffers/cache:             3671        44422
> Swap:             8191                5         8186
>
>
> I've tried sysctl mm.vfs_cache_pressure=10000 but that seems working to Cached
> memory, I wonder is there another sysctl for reclaming Buffers?

AFAIK "Buffers" is just a page-cache of block devices.
>From reclaimer's point of view they have no difference from file page-cache.

Could you post oom-killer log, there should be a lot of numbers
describing memory state.

>
>
> Thanks,
>
> - Derek
>
> --
> To unsubscribe, send a message with 'unsubscribe linux-mm' in
> the body to majordomo@kvack.org.  For more info on Linux MM,
> see: http://www.linux-mm.org/ .
> Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: How to controll Buffers to be dilligently reclaimed?

[Cheng Rk]

On Thursday, February 12, 2015 11:34 PM, Konstantin Khlebnikov <koct9i@gmail.com> wrote:

>>
>> -bash-4.2$ sudo losetup -a
>> /dev/loop0: [0005]:16512 (/dev/dm-2)
>> -bash-4.2$ free -m
>>                 total          used         free      shared       buffers     cached
>> Mem:             48094         46081         2012          40         40324       2085
>> -/+ buffers/cache:              3671       44422
>> Swap:             8191             5         8186
>>
>>
>> I've tried sysctl mm.vfs_cache_pressure=10000 but that seems working to Cached
>> memory, I wonder is there another sysctl for reclaming Buffers?

> AFAIK "Buffers" is just a page-cache of block devices.
> From reclaimer's point of view they have no difference from file page-cache.

> Could you post oom-killer log, there should be a lot of numbers
> describing memory state.


in this case, 40GB memory got stuck in Buffers, and 90+% of them are reclaimable (can be verified by vm.drop_caches manual reclaim)
if Buffers are treated same as Cached, why mm.vfs_cache_pressure=10000 (or even I tried up to 1,000,000) can't get Buffers reclaimed early?


I have some oom-killer msgs but were with older kernels, after set vm.overcommit_memory=2, it simply returns -ENOMEM, unable to spawn any new container, why doesn't it even try to reclaim some memory from those 40GB Buffers,



Thanks,

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: How to controll Buffers to be dilligently reclaimed?

[Cheng Rk]

On Thursday, February 12, 2015 11:34 PM, Konstantin Khlebnikov <koct9i@gmail.com> wrote:

>>
>> -bash-4.2$ sudo losetup -a
>> /dev/loop0: [0005]:16512 (/dev/dm-2)
>> -bash-4.2$ free -m
>>                 total          used         free      shared       buffers     cached
>> Mem:             48094         46081         2012          40         40324       2085
>> -/+ buffers/cache:              3671       44422
>> Swap:             8191             5         8186
>>
>> I've tried sysctl mm.vfs_cache_pressure=10000 but that seems working to Cached
>> memory, I wonder is there another sysctl for reclaming Buffers?

> AFAIK "Buffers" is just a page-cache of block devices.
> From reclaimer's point of view they have no difference from file page-cache.

> Could you post oom-killer log, there should be a lot of numbers
> describing memory state.


in this case, 40GB memory got stuck in Buffers, and 90+% of them are reclaimable (can be verified by vm.drop_caches manual reclaim)
if Buffers are treated same as Cached, why mm.vfs_cache_pressure=10000 (or even I tried up to 1,000,000) can't get Buffers reclaimed early?

I have some oom-killer msgs but were with older kernels, after set vm.overcommit_memory=2, it simply returns -ENOMEM, unable to spawn any new container, why doesn't it even try to reclaim some memory from those 40GB Buffers,


The Buffers in use is 44GB, from total memory of 48GB, it's the Inactive(file) 41GB consumed the most, why this much memory is reclaimable to vm/drop_caches but not to application requesting memory?


Is there a sysctl can make Buffers / Inactive(file) be reclaimed early and often ?

(since to this system it's mounting /dev/loop0 and have a lot of small temporary files created there, keeping them in Buffers for longer time is useless, how can I make it reclaimed earlier than later when applications need memory? )



-bash-4.2$ cat /proc/meminfo 
MemTotal:       49286656 kB
MemFree:         2040944 kB
MemAvailable:   47809824 kB
Buffers:        44258776 kB
Cached:           456868 kB
SwapCached:            0 kB
Active:          3783592 kB
Inactive:       41535112 kB
Active(anon):     402776 kB
Inactive(anon):   282308 kB
Active(file):    3380816 kB
Inactive(file): 41252804 kB


Thanks,

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: How to controll Buffers to be dilligently reclaimed?

[Michal Hocko]

On Fri 13-02-15 09:52:16, Cheng Rk wrote:
> 
> 
> On Thursday, February 12, 2015 11:34 PM, Konstantin Khlebnikov <koct9i@gmail.com> wrote:
> 
> >>
> >> -bash-4.2$ sudo losetup -a
> >> /dev/loop0: [0005]:16512 (/dev/dm-2)
> >> -bash-4.2$ free -m
> >>                 total          used         free      shared       buffers     cached
> >> Mem:             48094         46081         2012          40         40324       2085
> >> -/+ buffers/cache:              3671       44422
> >> Swap:             8191             5         8186
> >>
> >>
> >> I've tried sysctl mm.vfs_cache_pressure=10000 but that seems working to Cached
> >> memory, I wonder is there another sysctl for reclaming Buffers?
> 
> > AFAIK "Buffers" is just a page-cache of block devices.
> > From reclaimer's point of view they have no difference from file page-cache.
> 
> > Could you post oom-killer log, there should be a lot of numbers
> > describing memory state.
> 
> 
> in this case, 40GB memory got stuck in Buffers, and 90+% of them are
> reclaimable (can be verified by vm.drop_caches manual reclaim) if
> Buffers are treated same as Cached, why mm.vfs_cache_pressure=10000
> (or even I tried up to 1,000,000) can't get Buffers reclaimed early?

As per Documentation/sysctl/vm.txt the knob doesn't affect the page
cache reclaim but rather inode vs. dentry reclaim.

> I have some oom-killer msgs but were with older kernels, after set
> vm.overcommit_memory=2, it simply returns -ENOMEM, unable to spawn any
> new container, why doesn't it even try to reclaim some memory from
> those 40GB Buffers,

overcommit_memory controls behavior of the _virtual_ memory
reservations. OVERCOMMIT_NEVER (2) means that even virtual memory cannot
be overcommit outside of the configured value (RAM + swap basically -
see Documentation/vm/overcommit-accounting for more information). So
your application most probably consumes a lot of virtual memory (mmaps
etc.) and that is why it gets ENOMEM.

OOM report would tell us what was the memory state at the time when you
were short of memory and why the cache (buffers in your case) were not
reclaimed properly.
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: How to controll Buffers to be dilligently reclaimed?

[Cheng Rk]

On Wednesday, February 18, 2015 6:23 AM, Michal Hocko <mhocko@suse.cz> wrote:
On Fri 13-02-15 09:52:16, Cheng Rk wrote:

> As per Documentation/sysctl/vm.txt the knob doesn't affect the page
> cache reclaim but rather inode vs. dentry reclaim.


So do you think is it worth to work on something to give pressure similar
to vm.vfs_cache_pressure to vfs inode & dentry cache?

I am looking for an effect to let the kernel more aggressively reclaim
memory from Buffers,


By reading fs/super.c:prune_super I've also realized taht, which is the
only place referening sysctl_vfs_cache_pressure,
that block_devices' inode are in "bdev" mount, its super_block just
have nr_cached_objects as NULL,
s_nr_dentry_unused and s_nr_inodes_unused both 0, get total_objects to be
reclaimed is 0;

So is why sysctl_vfs_cache_pressure doesn't give pressure to Buffers,


         if (sb->s_op && sb->s_op->nr_cached_objects)
                   fs_objects = sb->s_op->nr_cached_objects(sb);

  total_objects = sb->s_nr_dentry_unused +
                                         sb->s_nr_inodes_unused + fs_objects;

  total_objects = (total_objects / 100) * sysctl_vfs_cache_pressure;
  drop_super(sb);


In crash, I got to know this block_device (253:2, or /dev/dm-2)has 10536805 pages mapped, that is the 40GB memory in Buffers, I wonder is there a sysctl can controll this to be reclaimed earlier?


crash> block_device.bd_dev,bd_inode -x ffff880619c78000
bd_dev = 0xfd00002
bd_inode = 0xffff880619c780f0
crash> inode.i_mapping 0xffff880619c780f0
i_mapping = 0xffff880619c78240
crash> address_space.nrpages 0xffff880619c78240
nrpages = 10536805


>> I have some oom-killer msgs but were with older kernels, after set>> vm.overcommit_memory=2, it simply returns -ENOMEM, unable to spawn any
>> new container, why doesn't it even try to reclaim some memory from

>> those 40GB Buffers,> overcommit_memory controls behavior of the _virtual_ memory
> reservations. OVERCOMMIT_NEVER (2) means that even virtual memory cannot
> be overcommit outside of the configured value (RAM + swap basically -
> see Documentation/vm/overcommit-accounting for more information). So
> your application most probably consumes a lot of virtual memory (mmaps
> etc.) and that is why it gets ENOMEM.


I have read that Doc as well, will post again when I get a more concrete example

> OOM report would tell us what was the memory state at the time when you
> were short of memory and why the cache (buffers in your case) were not
> reclaimed properly.


Thanks,

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: How to controll Buffers to be dilligently reclaimed?

[Michal Hocko]

On Wed 18-02-15 19:44:06, Cheng Rk wrote:
> 
> 
> On Wednesday, February 18, 2015 6:23 AM, Michal Hocko <mhocko@suse.cz> wrote:
> On Fri 13-02-15 09:52:16, Cheng Rk wrote:
> 
> > As per Documentation/sysctl/vm.txt the knob doesn't affect the page
> > cache reclaim but rather inode vs. dentry reclaim.
> 
> 
> So do you think is it worth to work on something to give pressure similar
> to vm.vfs_cache_pressure to vfs inode & dentry cache?
> 
> I am looking for an effect to let the kernel more aggressively reclaim
> memory from Buffers,

more aggressively than what? Anonymous memory, other types of caches?
To be honest I do not see why we should treat buffers any different from
any other cache. So far it is not clear what might be the issue you are
seeing but I would suspect that too many buffers is not the primary one.
It is hard to say anything more without any specific numbers, though.
 
> By reading fs/super.c:prune_super I've also realized taht, which is the
> only place referening sysctl_vfs_cache_pressure,
> that block_devices' inode are in "bdev" mount, its super_block just
> have nr_cached_objects as NULL,
> s_nr_dentry_unused and s_nr_inodes_unused both 0, get total_objects to be
> reclaimed is 0;
> 
> So is why sysctl_vfs_cache_pressure doesn't give pressure to Buffers,
> 
> 
>          if (sb->s_op && sb->s_op->nr_cached_objects)
>                    fs_objects = sb->s_op->nr_cached_objects(sb);
> 
>   total_objects = sb->s_nr_dentry_unused +
>                                          sb->s_nr_inodes_unused + fs_objects;
> 
>   total_objects = (total_objects / 100) * sysctl_vfs_cache_pressure;
>   drop_super(sb);
> 
> 
> In crash, I got to know this block_device (253:2, or /dev/dm-2)has
> 10536805 pages mapped, that is the 40GB memory in Buffers,

I still do not see why is that a problem. They should get reclaimed on
demand.

> I wonder is there a sysctl can controll this to be reclaimed earlier?

I do not know about any.

[...]
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: How to controll Buffers to be dilligently reclaimed?

[Cheng Rk]

On Thursday, February 19, 2015 2:03 AM, Michal Hocko <mhocko@suse.cz> wrote:
>> So do you think is it worth to work on something to give pressure similar
>> to vm.vfs_cache_pressure to vfs inode & dentry cache?
>> 
>> I am looking for an effect to let the kernel more aggressively reclaim
>> memory from Buffers,

> more aggressively than what? Anonymous memory, other types of caches?
> To be honest I do not see why we should treat buffers any different from
> any other cache. So far it is not clear what might be the issue you are
> seeing but I would suspect that too many buffers is not the primary one.
> It is hard to say anything more without any specific numbers, though.


to get Buffers more aggresively, or ealier reclaimed than lazy on demand.

Suppose if someone can do a similar sysctl (say vm.vfs_buffers_pressure, or reuse vm.vfs_cache_presssure), do you think is that worth to do and useful?


I think what makes sense to vm.vfs_cache_presssure would also make sense
to controll Buffers, right?


So far I see people adjust vm.vfs_cache_pressure for various purposes;
from default 100 they set it to 50 for more conservatively reclaim
memory from cache, or set to a larger value (like 10000) to reclaim
the Cached memory earlier, or more aggresively, for Build farms,
or in any scenarios that files are all temporary and accessed only in a short time;

If those temporary files are finally removed, the Cached memory can be reclaimed,
but some cases they may be never removed,

For file backup applications, they can do madvise MADV_DONTNEED, but for
other applications unaware of MADV_DONTNEED, the kernel may never that
Cached memory is not used anymore, keep them for long time, and reclaimed on demand;


To reclaim early also has a benefit to save time of reclaim on demand; when in future application really need memory; I'm not sure if any applications are sensitive on time of allocation,,

> By reading fs/super.c:prune_super I've also realized taht, which is the
> only place referening sysctl_vfs_cache_pressure,
> that block_devices' inode are in "bdev" mount, its super_block just
> have nr_cached_objects as NULL,
> s_nr_dentry_unused and s_nr_inodes_unused both 0, get total_objects to be
> reclaimed is 0;
> 
> So is why sysctl_vfs_cache_pressure doesn't give pressure to Buffers,
> 
> 
>          if (sb->s_op && sb->s_op->nr_cached_objects)
>                    fs_objects = sb->s_op->nr_cached_objects(sb);
> 
>   total_objects = sb->s_nr_dentry_unused +
>                                          sb->s_nr_inodes_unused + fs_objects;
> 
>   total_objects = (total_objects / 100) * sysctl_vfs_cache_pressure;
>   drop_super(sb);
> 
> 
> In crash, I got to know this block_device (253:2, or /dev/dm-2)has
> 10536805 pages mapped, that is the 40GB memory in Buffers,

I still do not see why is that a problem. They should get reclaimed on
demand.

> I wonder is there a sysctl can controll this to be reclaimed earlier?

I do not know about any.


[...]
-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>