Silly question about dethrottling

Silly question about dethrottling

[Raymond Jennings]

[-- Attachment #1: Type: text/plain, Size: 456 bytes --]

I have an application that is generating HUGE amounts of dirty data.
Multiple GiB worth, and I'd like to allow it to fill at least half of my
RAM.

I already have /proc/sys/vm/dirty_ratio pegged at 80 and the background one
pegged at 50.  RAM is 32GiB.

it appears to be butting heads with clean memory.  How do I tell my system
to prefer using RAM to soak up writes instead of caching?

Atm I'm at the stage where I'm prepared to patch the kernel itself.

[-- Attachment #2: Type: text/html, Size: 566 bytes --]

Re: Silly question about dethrottling

[Michal Hocko]

On Sun 04-12-16 13:56:54, Raymond Jennings wrote:
> I have an application that is generating HUGE amounts of dirty data.
> Multiple GiB worth, and I'd like to allow it to fill at least half of my
> RAM.

Could you be more specific why and what kind of problem you are trying
to solve?

> I already have /proc/sys/vm/dirty_ratio pegged at 80 and the background one
> pegged at 50.  RAM is 32GiB.

There is also dirty_bytes alternative which is an absolute numer.

> it appears to be butting heads with clean memory.  How do I tell my system
> to prefer using RAM to soak up writes instead of caching?

I am not sure I understand. Could you be more specific about what is the
actual problem? Is it possible that your dirty data is already being
flushed and that is wy you see a clean cache?

-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: Silly question about dethrottling

[Raymond Jennings]

[-- Attachment #1: Type: text/plain, Size: 1146 bytes --]

On Sun, Dec 4, 2016 at 11:05 PM, Michal Hocko <mhocko@kernel.org> wrote:

> On Sun 04-12-16 13:56:54, Raymond Jennings wrote:
> > I have an application that is generating HUGE amounts of dirty data.
> > Multiple GiB worth, and I'd like to allow it to fill at least half of my
> > RAM.
>
> Could you be more specific why and what kind of problem you are trying
> to solve?
>
> > I already have /proc/sys/vm/dirty_ratio pegged at 80 and the background
> one
> > pegged at 50.  RAM is 32GiB.
>
> There is also dirty_bytes alternative which is an absolute numer.
>

How does this compare to setting dirty_ratio to a high percentage?

>
> > it appears to be butting heads with clean memory.  How do I tell my
> system
> > to prefer using RAM to soak up writes instead of caching?
>
> I am not sure I understand. Could you be more specific about what is the
> actual problem? Is it possible that your dirty data is already being
> flushed and that is wy you see a clean cache?
>

What I'm wanting is for my writing process not to get throttled, even when
the dirty memory it starts creating starts hogging memory the system would
rather use for cache.

[-- Attachment #2: Type: text/html, Size: 1760 bytes --]

Re: Silly question about dethrottling

[Michal Hocko]

On Mon 05-12-16 01:15:39, Raymond Jennings wrote:
> On Sun, Dec 4, 2016 at 11:05 PM, Michal Hocko <mhocko@kernel.org> wrote:
> 
> > On Sun 04-12-16 13:56:54, Raymond Jennings wrote:
> > > I have an application that is generating HUGE amounts of dirty data.
> > > Multiple GiB worth, and I'd like to allow it to fill at least half of my
> > > RAM.
> >
> > Could you be more specific why and what kind of problem you are trying
> > to solve?
> >
> > > I already have /proc/sys/vm/dirty_ratio pegged at 80 and the background
> > one
> > > pegged at 50.  RAM is 32GiB.
> >
> > There is also dirty_bytes alternative which is an absolute numer.
> >
> 
> How does this compare to setting dirty_ratio to a high percentage?

Well, dirty_bytes is an absolute number when to start to throttle while
ratio is relative to node_dirtyable_memory

> > > it appears to be butting heads with clean memory.  How do I tell my
> > system
> > > to prefer using RAM to soak up writes instead of caching?
> >
> > I am not sure I understand. Could you be more specific about what is the
> > actual problem? Is it possible that your dirty data is already being
> > flushed and that is wy you see a clean cache?
> >
> 
> What I'm wanting is for my writing process not to get throttled, even when
> the dirty memory it starts creating starts hogging memory the system would
> rather use for cache.

Then you can configure dirty_background_{bytes,ratio} to start flushing
dirty data sooner. Having a lot of dirty data in the system just asks
for troubles elsewhere as it would take a lot of time to sync that to
the backing store. That means that many unrelated processes might get
stuck on sync etc. for an unconfortably large amount of time.

-- 
Michal Hocko
SUSE Labs

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>