High system load and 3TB of memory.

High system load and 3TB of memory.

[jesper]

Hi.

I have a 3.13 (ubuntu LTS) server with 3TB of memory and under certain load
conditions it can spiral off to 80+% system load. Per recommendation on IRC
yesterday I have captured 2 perf reports (I'm new to perf, so I'm not
sure they tell precisely whats needed.

Bad situation (high sysload 80%+)

Samples: 381K of event 'cycles', Event count (approx.): 1228296411165
+  27.84%         postgres  [kernel.kallsyms]     [k] isolate_freepages_block
+  21.08%             psql  [kernel.kallsyms]     [k] isolate_freepages_block
+  20.72%       pg_restore  [kernel.kallsyms]     [k] isolate_freepages_block
+   3.94%         postgres  postgres              [.] pglz_compress
+   2.86%         postgres  [kernel.kallsyms]     [k]
set_pageblock_flags_mask
+   2.35%        bacula-fd  [kernel.kallsyms]     [k] isolate_freepages_block
+   2.07%       pg_restore  [kernel.kallsyms]     [k]
set_pageblock_flags_mask
+   2.06%             psql  [kernel.kallsyms]     [k]
set_pageblock_flags_mask
+   1.56%         postgres  libc-2.15.so          [.] 0x000000000003c95f
+   0.93%       irqbalance  [kernel.kallsyms]     [k] isolate_freepages_block
+   0.88%       pg_restore  [kernel.kallsyms]     [k] isolate_freepages
+   0.87%             psql  [kernel.kallsyms]     [k] isolate_freepages
+   0.86%         postgres  [kernel.kallsyms]     [k] isolate_freepages
+   0.81%         postgres  postgres              [.] 0x000000000027ff5b
+   0.60%         postgres  [kernel.kallsyms]     [k]
get_pageblock_flags_mask
+   0.44%         proc_pri  [kernel.kallsyms]     [k] isolate_freepages_block

Good situation .. sysload < 5%

Samples: 509K of event 'cycles', Event count (approx.): 1635259826919
+  21.14%         postgres  postgres                  [.] pglz_compress
+  14.46%         postgres  postgres                  [.] 0x000000000016b643
+  10.11%         postgres  libc-2.15.so              [.] 0x0000000000092f69
+   5.74%         postgres  postgres                  [.] s_lock
+   2.86%         postgres  postgres                  [.] LWLockAcquire
+   2.51%       pg_restore  [kernel.kallsyms]         [k]
isolate_freepages_block
+   2.33%         postgres  postgres                  [.]
NextCopyFromRawFields
+   2.15%         postgres  postgres                  [.] LWLockRelease
+   2.10%         postgres  postgres                  [.] _start
+   1.93%         postgres  [kernel.kallsyms]         [k]
copy_user_enhanced_fast_string
+   1.70%         postgres  [kernel.kallsyms]         [k] change_pte_range
+   1.61%         postgres  postgres                  [.] pg_verify_mbstr_len
+   1.31%         postgres  postgres                  [.]
hash_search_with_hash_value
+   1.21%         postgres  libc-2.15.so              [.] __strcoll_l
+   0.86%          kswapd0  [kernel.kallsyms]         [k]
__mem_cgroup_uncharge_common
+   0.72%         postgres  postgres                  [.] heap_fill_tuple
+   0.68%        bacula-fd  [kernel.kallsyms]         [k]
isolate_freepages_block
+   0.66%         postgres  [kernel.kallsyms]         [k] clear_page_c_e
+   0.63%       pg_restore  [kernel.kallsyms]         [k]
copy_user_enhanced_fast_string


Hugepages are disabled. All suggestions for configuration changes, etc are
welcome?

IO subsystem is not particulary busy in any of the situations. A sar
output can be seen here:
http://thread.gmane.org/gmane.linux.kernel/1908263

Jesper



--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: High system load and 3TB of memory.

[Andrey Korolyov]

On Sat, Mar 14, 2015 at 8:05 PM,  <jesper@krogh.cc> wrote:
> Hi.
>
> I have a 3.13 (ubuntu LTS) server with 3TB of memory and under certain load
> conditions it can spiral off to 80+% system load. Per recommendation on IRC
> yesterday I have captured 2 perf reports (I'm new to perf, so I'm not
> sure they tell precisely whats needed.
>
> Bad situation (high sysload 80%+)
>
> Samples: 381K of event 'cycles', Event count (approx.): 1228296411165
> +  27.84%         postgres  [kernel.kallsyms]     [k] isolate_freepages_block
> +  21.08%             psql  [kernel.kallsyms]     [k] isolate_freepages_block
> +  20.72%       pg_restore  [kernel.kallsyms]     [k] isolate_freepages_block
> +   3.94%         postgres  postgres              [.] pglz_compress
> +   2.86%         postgres  [kernel.kallsyms]     [k]
> set_pageblock_flags_mask
> +   2.35%        bacula-fd  [kernel.kallsyms]     [k] isolate_freepages_block
> +   2.07%       pg_restore  [kernel.kallsyms]     [k]
> set_pageblock_flags_mask
> +   2.06%             psql  [kernel.kallsyms]     [k]
> set_pageblock_flags_mask
> +   1.56%         postgres  libc-2.15.so          [.] 0x000000000003c95f
> +   0.93%       irqbalance  [kernel.kallsyms]     [k] isolate_freepages_block
> +   0.88%       pg_restore  [kernel.kallsyms]     [k] isolate_freepages
> +   0.87%             psql  [kernel.kallsyms]     [k] isolate_freepages
> +   0.86%         postgres  [kernel.kallsyms]     [k] isolate_freepages
> +   0.81%         postgres  postgres              [.] 0x000000000027ff5b
> +   0.60%         postgres  [kernel.kallsyms]     [k]
> get_pageblock_flags_mask
> +   0.44%         proc_pri  [kernel.kallsyms]     [k] isolate_freepages_block
>
> Good situation .. sysload < 5%
>
> Samples: 509K of event 'cycles', Event count (approx.): 1635259826919
> +  21.14%         postgres  postgres                  [.] pglz_compress
> +  14.46%         postgres  postgres                  [.] 0x000000000016b643
> +  10.11%         postgres  libc-2.15.so              [.] 0x0000000000092f69
> +   5.74%         postgres  postgres                  [.] s_lock
> +   2.86%         postgres  postgres                  [.] LWLockAcquire
> +   2.51%       pg_restore  [kernel.kallsyms]         [k]
> isolate_freepages_block
> +   2.33%         postgres  postgres                  [.]
> NextCopyFromRawFields
> +   2.15%         postgres  postgres                  [.] LWLockRelease
> +   2.10%         postgres  postgres                  [.] _start
> +   1.93%         postgres  [kernel.kallsyms]         [k]
> copy_user_enhanced_fast_string
> +   1.70%         postgres  [kernel.kallsyms]         [k] change_pte_range
> +   1.61%         postgres  postgres                  [.] pg_verify_mbstr_len
> +   1.31%         postgres  postgres                  [.]
> hash_search_with_hash_value
> +   1.21%         postgres  libc-2.15.so              [.] __strcoll_l
> +   0.86%          kswapd0  [kernel.kallsyms]         [k]
> __mem_cgroup_uncharge_common
> +   0.72%         postgres  postgres                  [.] heap_fill_tuple
> +   0.68%        bacula-fd  [kernel.kallsyms]         [k]
> isolate_freepages_block
> +   0.66%         postgres  [kernel.kallsyms]         [k] clear_page_c_e
> +   0.63%       pg_restore  [kernel.kallsyms]         [k]
> copy_user_enhanced_fast_string
>
>
> Hugepages are disabled. All suggestions for configuration changes, etc are
> welcome?
>
> IO subsystem is not particulary busy in any of the situations. A sar
> output can be seen here:
> http://thread.gmane.org/gmane.linux.kernel/1908263
>
> Jesper

Hi Jesper, please take a look on
http://marc.info/?l=linux-mm&m=141605213522925&w=2, there is a long
and unfinished discussion as it seems very problematic to make a
deterministic reproduction of the bug in our environments. If you can
observe same lockups with more ease, it`ll help a lot in the issue
pinning and fixing.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: High system load and 3TB of memory.

[jesper]

> On Sat, Mar 14, 2015 at 8:05 PM,  <jesper@krogh.cc> wrote:
>> Hi
>> I have a 3.13 (ubuntu LTS) server with 3TB of memory and under certain
>> load
>> conditions it can spiral off to 80+% system load. Per recommendation on
>> IRC
>> yesterday I have captured 2 perf reports (I'm new to perf, so I'm not
>> sure they tell precisely whats needed.
>>
>> Bad situation (high sysload 80%+)


> Hi Jesper, please take a look on
> http://marc.info/?l=linux-mm&m=141605213522925&w=2, there is a long
> and unfinished discussion as it seems very problematic to make a
> deterministic reproduction of the bug in our environments. If you can
> observe same lockups with more ease, it`ll help a lot in the issue
> pinning and fixing.


Hi Andrey.

Yes it looks indeed familiar. I can do a fair amount of testing and our
normal production load triggers the problem 6-10 times per day and I'm
willing to garther data to help move forward. What do you suggest is next?

Jesper


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: High system load and 3TB of memory.

[Andrey Korolyov]

On Sat, Mar 14, 2015 at 8:25 PM,  <jesper@krogh.cc> wrote:
>> On Sat, Mar 14, 2015 at 8:05 PM,  <jesper@krogh.cc> wrote:
>>> Hi
>>> I have a 3.13 (ubuntu LTS) server with 3TB of memory and under certain
>>> load
>>> conditions it can spiral off to 80+% system load. Per recommendation on
>>> IRC
>>> yesterday I have captured 2 perf reports (I'm new to perf, so I'm not
>>> sure they tell precisely whats needed.
>>>
>>> Bad situation (high sysload 80%+)
>
>
>> Hi Jesper, please take a look on
>> http://marc.info/?l=linux-mm&m=141605213522925&w=2, there is a long
>> and unfinished discussion as it seems very problematic to make a
>> deterministic reproduction of the bug in our environments. If you can
>> observe same lockups with more ease, it`ll help a lot in the issue
>> pinning and fixing.
>
>
> Hi Andrey.
>
> Yes it looks indeed familiar. I can do a fair amount of testing and our
> normal production load triggers the problem 6-10 times per day and I'm
> willing to garther data to help move forward. What do you suggest is next?
>
> Jesper
>
>

There is a couple of patches suggested by Vlastimil and others through
discussion, not me neither Christian was able to test them properly
due to kind of environment where bug primarily live (production envs
for both of us). The bare test-env reproducer is a big step forward
indeed. Since then bug was reported a couple of times and workarounded
(by setting ridiculously large amount of memory for vm.min_free), the
larger memory room is (given intensive disk i/o which is able to fill
all memory with certain ratio of active/inactive pages I suppose), the
easier it is to catch the issue.

--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: High system load and 3TB of memory.

[Vlastimil Babka]

On 03/14/2015 06:33 PM, Andrey Korolyov wrote:
> On Sat, Mar 14, 2015 at 8:25 PM,  <jesper@krogh.cc> wrote:
>>> On Sat, Mar 14, 2015 at 8:05 PM,  <jesper@krogh.cc> wrote:
>>>> Hi
>>>> I have a 3.13 (ubuntu LTS) server with 3TB of memory and under certain
>>>> load
>>>> conditions it can spiral off to 80+% system load. Per recommendation on
>>>> IRC
>>>> yesterday I have captured 2 perf reports (I'm new to perf, so I'm not
>>>> sure they tell precisely whats needed.
>>>>
>>>> Bad situation (high sysload 80%+)
>>
>>
>>> Hi Jesper, please take a look on
>>> http://marc.info/?l=linux-mm&m=141605213522925&w=2, there is a long
>>> and unfinished discussion as it seems very problematic to make a
>>> deterministic reproduction of the bug in our environments. If you can
>>> observe same lockups with more ease, it`ll help a lot in the issue
>>> pinning and fixing.
>>
>>
>> Hi Andrey.
>>
>> Yes it looks indeed familiar. I can do a fair amount of testing and our
>> normal production load triggers the problem 6-10 times per day and I'm
>> willing to garther data to help move forward. What do you suggest is next?
>>
>> Jesper
>>
>>
>
> There is a couple of patches suggested by Vlastimil and others through
> discussion, not me neither Christian was able to test them properly
> due to kind of environment where bug primarily live (production envs
> for both of us). The bare test-env reproducer is a big step forward
> indeed. Since then bug was reported a couple of times and workarounded
> (by setting ridiculously large amount of memory for vm.min_free), the
> larger memory room is (given intensive disk i/o which is able to fill
> all memory with certain ratio of active/inactive pages I suppose), the
> easier it is to catch the issue.

Right, it would be great if you could try it with 3.18+ kernel and 
possibly Joonsoo's patch from
http://marc.info/?l=linux-mm&m=141774145601066


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: High system load and 3TB of memory.

[Jesper Krogh]

> On 18/03/2015, at 15.15, Vlastimil Babka <vbabka@suse.cz>
> Right, it would be great if you could try it with 3.18+ kernel and possibly Joonsoo's patch from
> http://marc.info/?l=linux-mm&m=141774145601066
> 

Thanks, we will do that.

We actually upgraded to 3.18.9 monday (together whith moving the database from postgresql 9.2 to 9.3) and we havent seen the problem since.

Sysload is sitting around 8-10%

But we will test

Jesper


--
To unsubscribe, send a message with 'unsubscribe linux-mm' in
the body to majordomo@kvack.org.  For more info on Linux MM,
see: http://www.linux-mm.org/ .
Don't email: <a href=mailto:"dont@kvack.org"> email@kvack.org </a>

Re: High system load and 3TB of memory.

[Joonsoo Kim]

2015-03-19 0:14 GMT+09:00 Jesper Krogh <jesper@krogh.cc>:
>
>> On 18/03/2015, at 15.15, Vlastimil Babka <vbabka@suse.cz>
>> Right, it would be great if you could try it with 3.18+ kernel and possibly Joonsoo's patch from
>> http://marc.info/?l=linux-mm&m=141774145601066
>>
>
> Thanks, we will do that.
>
> We actually upgraded to 3.18.9 monday (together whith moving the database from postgresql 9.2 to 9.3) and we havent seen the problem since.
>
> Sysload is sitting around 8-10%
>
> But we will test

Hello,

It would be really nice if you could test my patch.
If possible, please test below patch rather than old one.
It solves some issues commented by Vlastimil and back ported to v3.18.

Thanks.

--------->8------------