From: Luis Henriques <lhenriques@suse.com>
To: Dennis Zhou <dennisszhou@gmail.com>
Cc: Tejun Heo <tj@kernel.org>, Christoph Lameter <cl@linux.com>,
linux-mm@kvack.org, linux-kernel@vger.kernel.org
Subject: Re: percpu allocation failures
Date: Wed, 27 Sep 2017 10:05:44 +0100 [thread overview]
Message-ID: <87lgl0o6jb.fsf@hermes> (raw)
In-Reply-To: <20170927015323.GA19100@Big-Sky.local> (Dennis Zhou's message of "Tue, 26 Sep 2017 20:54:00 -0500")
[-- Attachment #1: Type: text/plain, Size: 1038 bytes --]
Dennis Zhou <dennisszhou@gmail.com> writes:
> Hi Luis,
>
> This seems to be an issue with the reserved chunk being unable to
> allocate memory when loading kernel modules. Unfortunately, I have not
> been successful in reproducing this with the reserved chunk allocation
> path exposed or by inserting the nft_meta module.
>
> Could you please send me the output when ran with the following patch
> and the output of the percpu memory statistics file before and after
> inserting the module (PERCPU_STATS)? The stats are in
> /sys/kernel/debug/percpu_stats.
Please find attached all the info you requested. Hope it helps.
Also, here's the nft script I'm using to trigger the issue locally:
#!/usr/sbin/nft -f
flush ruleset
table inet filter {
chain input {
type filter hook input priority 0;
iif lo accept
ct state established,related accept
tcp dport { 22 } ct state new accept
ip6 nexthdr icmpv6 icmpv6 type { nd-neighbor-solicit, nd-router-advert, nd-neighbor-advert } accept
counter drop
}
}
Cheers,
--
Luis
[-- Attachment #2: percpu_stats.before --]
[-- Type: application/octet-stream, Size: 2031 bytes --]
Percpu Memory Statistics
Allocation Info:
----------------------------------------
unit_size : 262144
static_size : 110488
reserved_size : 8192
dyn_size : 28776
atom_size : 2097152
alloc_size : 2097152
Global Stats:
----------------------------------------
nr_alloc : 2204
nr_dealloc : 764
nr_cur_alloc : 1440
nr_max_alloc : 1638
nr_chunks : 2
nr_max_chunks : 2
min_alloc_size : 4
max_alloc_size : 4104
empty_pop_pages : 2
Per Chunk Stats:
----------------------------------------
Chunk: <- Reserved Chunk
nr_alloc : 4
max_alloc_size : 4104
empty_pop_pages : 0
first_bit : 999
free_bytes : 4004
contig_bytes : 8
sum_frag : 0
max_frag : 0
cur_min_alloc : 0
cur_med_alloc : 0
cur_max_alloc : 0
Chunk: <- First Chunk
nr_alloc : 113
max_alloc_size : 976
empty_pop_pages : 0
first_bit : 8192
free_bytes : 0
contig_bytes : 0
sum_frag : 0
max_frag : 0
cur_min_alloc : 4
cur_med_alloc : 976
cur_max_alloc : 976
Chunk:
nr_alloc : 1323
max_alloc_size : 4096
empty_pop_pages : 2
first_bit : 11005
free_bytes : 162604
contig_bytes : 155648
sum_frag : 6956
max_frag : 2808
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 4096
[-- Attachment #3: percpu_stats.after --]
[-- Type: application/octet-stream, Size: 2031 bytes --]
Percpu Memory Statistics
Allocation Info:
----------------------------------------
unit_size : 262144
static_size : 110488
reserved_size : 8192
dyn_size : 28776
atom_size : 2097152
alloc_size : 2097152
Global Stats:
----------------------------------------
nr_alloc : 2215
nr_dealloc : 765
nr_cur_alloc : 1450
nr_max_alloc : 1638
nr_chunks : 2
nr_max_chunks : 2
min_alloc_size : 4
max_alloc_size : 4104
empty_pop_pages : 2
Per Chunk Stats:
----------------------------------------
Chunk: <- Reserved Chunk
nr_alloc : 7
max_alloc_size : 4104
empty_pop_pages : 0
first_bit : 2052
free_bytes : 3976
contig_bytes : 3976
sum_frag : 0
max_frag : 0
cur_min_alloc : 0
cur_med_alloc : 0
cur_max_alloc : 0
Chunk: <- First Chunk
nr_alloc : 113
max_alloc_size : 976
empty_pop_pages : 0
first_bit : 8192
free_bytes : 0
contig_bytes : 0
sum_frag : 0
max_frag : 0
cur_min_alloc : 4
cur_med_alloc : 976
cur_max_alloc : 976
Chunk:
nr_alloc : 1330
max_alloc_size : 4096
empty_pop_pages : 2
first_bit : 11005
free_bytes : 160492
contig_bytes : 155648
sum_frag : 4844
max_frag : 2808
cur_min_alloc : 4
cur_med_alloc : 8
cur_max_alloc : 4096
[-- Attachment #4: dmesg.debug --]
[-- Type: application/octet-stream, Size: 5911 bytes --]
[ 87.709692] percpu: reserved chunk: 4004, 2, 2030, 999, 3992, 104, 3
[ 87.709694] percpu: rchunk md blocks
[ 87.709697] percpu: 1, 999, 0, 0, 999
[ 87.709699] percpu: 2, 1006, 0, 0, 1006
[ 87.709701] percpu: 998, 0, 998, 0, 0
[ 87.709704] percpu: pcpu_find_block_fit: -1, 4, 4
[ 87.709707] percpu: allocation failed, size=16 align=16 atomic=0, alloc from reserved chunk failed to find fit
[ 87.709717] CPU: 0 PID: 1328 Comm: modprobe Tainted: G E 4.14.0-rc2+ #123
[ 87.709721] Hardware name: Dell Inc. Precision 5510/0N8J4R, BIOS 1.2.25 05/07/2017
[ 87.709723] Call Trace:
[ 87.709738] dump_stack+0x63/0x89
[ 87.709745] pcpu_alloc+0x6c8/0x720
[ 87.709753] __alloc_reserved_percpu+0x18/0x20
[ 87.709760] load_module+0x733/0x2c00
[ 87.709769] ? kernel_read_file+0x1a3/0x1d0
[ 87.709776] SYSC_finit_module+0xfc/0x120
[ 87.709781] ? SYSC_finit_module+0xfc/0x120
[ 87.709789] SyS_finit_module+0xe/0x10
[ 87.709795] entry_SYSCALL_64_fastpath+0x1e/0xa9
[ 87.709800] RIP: 0033:0x7f211163f219
[ 87.709804] RSP: 002b:00007ffe68de7318 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[ 87.709809] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f211163f219
[ 87.709812] RDX: 0000000000000000 RSI: 000055dc7a86a8cc RDI: 0000000000000000
[ 87.709815] RBP: 00007ffe68de6310 R08: 0000000000000000 R09: 0000000000000001
[ 87.709818] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000005
[ 87.709821] R13: 000055dc7c54c600 R14: 00007ffe68de62f0 R15: 0000000000000005
[ 87.709827] nft_meta: Could not allocate 16 bytes percpu data
[ 87.749131] nf_conntrack version 0.5.0 (65536 buckets, 262144 max)
[ 87.749791] percpu: reserved chunk: 4004, 2, 2030, 999, 3992, 104, 3
[ 87.749793] percpu: rchunk md blocks
[ 87.749796] percpu: 1, 999, 0, 0, 999
[ 87.749798] percpu: 2, 1006, 0, 0, 1006
[ 87.749800] percpu: 998, 0, 998, 0, 0
[ 87.749802] percpu: pcpu_find_block_fit: 2030, 2, 2
[ 87.749806] percpu: pcpu_alloc_area: 8120, 2, 2
[ 87.753575] percpu: reserved chunk: 3996, 1, 999, 999, 3992, 104, 3
[ 87.753578] percpu: rchunk md blocks
[ 87.753580] percpu: 1, 999, 0, 0, 999
[ 87.753582] percpu: 0, 1006, 0, 0, 1024
[ 87.753584] percpu: 998, 0, 998, 0, 0
[ 87.753587] percpu: pcpu_find_block_fit: -1, 4, 4
[ 87.753590] percpu: allocation failed, size=16 align=16 atomic=0, alloc from reserved chunk failed to find fit
[ 87.753599] CPU: 1 PID: 1334 Comm: modprobe Tainted: G E 4.14.0-rc2+ #123
[ 87.753603] Hardware name: Dell Inc. Precision 5510/0N8J4R, BIOS 1.2.25 05/07/2017
[ 87.753605] Call Trace:
[ 87.753619] dump_stack+0x63/0x89
[ 87.753626] pcpu_alloc+0x6c8/0x720
[ 87.753635] __alloc_reserved_percpu+0x18/0x20
[ 87.753642] load_module+0x733/0x2c00
[ 87.753650] ? kernel_read_file+0x1a3/0x1d0
[ 87.753657] SYSC_finit_module+0xfc/0x120
[ 87.753663] ? SYSC_finit_module+0xfc/0x120
[ 87.753671] SyS_finit_module+0xe/0x10
[ 87.753677] entry_SYSCALL_64_fastpath+0x1e/0xa9
[ 87.753681] RIP: 0033:0x7fb1b2421219
[ 87.753685] RSP: 002b:00007ffe287a6d88 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[ 87.753690] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007fb1b2421219
[ 87.753693] RDX: 0000000000000000 RSI: 00005622387988cc RDI: 0000000000000000
[ 87.753696] RBP: 00007ffe287a5d80 R08: 0000000000000000 R09: 0000000000000001
[ 87.753699] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000005
[ 87.753702] R13: 0000562239046600 R14: 00007ffe287a5d60 R15: 0000000000000005
[ 87.753708] nft_meta: Could not allocate 16 bytes percpu data
[ 87.782057] percpu: reserved chunk: 3996, 1, 999, 999, 3992, 104, 3
[ 87.782060] percpu: rchunk md blocks
[ 87.782062] percpu: 1, 999, 0, 0, 999
[ 87.782065] percpu: 0, 1006, 0, 0, 1024
[ 87.782067] percpu: 998, 0, 998, 0, 0
[ 87.782069] percpu: pcpu_find_block_fit: -1, 4, 4
[ 87.782072] percpu: allocation failed, size=16 align=16 atomic=0, alloc from reserved chunk failed to find fit
[ 87.782081] CPU: 0 PID: 1336 Comm: modprobe Tainted: G E 4.14.0-rc2+ #123
[ 87.782085] Hardware name: Dell Inc. Precision 5510/0N8J4R, BIOS 1.2.25 05/07/2017
[ 87.782087] Call Trace:
[ 87.782099] dump_stack+0x63/0x89
[ 87.782107] pcpu_alloc+0x6c8/0x720
[ 87.782115] __alloc_reserved_percpu+0x18/0x20
[ 87.782121] load_module+0x733/0x2c00
[ 87.782129] ? kernel_read_file+0x1a3/0x1d0
[ 87.782136] SYSC_finit_module+0xfc/0x120
[ 87.782141] ? SYSC_finit_module+0xfc/0x120
[ 87.782149] SyS_finit_module+0xe/0x10
[ 87.782155] entry_SYSCALL_64_fastpath+0x1e/0xa9
[ 87.782159] RIP: 0033:0x7f4dd4073219
[ 87.782163] RSP: 002b:00007ffefd82e658 EFLAGS: 00000246 ORIG_RAX: 0000000000000139
[ 87.782169] RAX: ffffffffffffffda RBX: 0000000000000000 RCX: 00007f4dd4073219
[ 87.782172] RDX: 0000000000000000 RSI: 000055d5cd3fd8cc RDI: 0000000000000000
[ 87.782175] RBP: 00007ffefd82d650 R08: 0000000000000000 R09: 0000000000000001
[ 87.782178] R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000005
[ 87.782181] R13: 000055d5ce138600 R14: 00007ffefd82d630 R15: 0000000000000005
[ 87.782186] nft_meta: Could not allocate 16 bytes percpu data
[ 87.822555] percpu: reserved chunk: 3996, 1, 999, 999, 3992, 104, 3
[ 87.822557] percpu: rchunk md blocks
[ 87.822560] percpu: 1, 999, 0, 0, 999
[ 87.822562] percpu: 0, 1006, 0, 0, 1024
[ 87.822564] percpu: 998, 0, 998, 0, 0
[ 87.822567] percpu: pcpu_find_block_fit: 999, 1, 1
[ 87.822571] percpu: pcpu_alloc_area: 3996, 1, 1
[ 87.897805] percpu: reserved chunk: 3992, 998, 2048, 2048, 3992, 104, 3
[ 87.897807] percpu: rchunk md blocks
[ 87.897809] percpu: 0, 999, 0, 0, 1024
[ 87.897812] percpu: 0, 1006, 0, 0, 1024
[ 87.897814] percpu: 998, 0, 998, 0, 0
[ 87.897816] percpu: pcpu_find_block_fit: 2048, 4, 4
prev parent reply other threads:[~2017-09-27 9:05 UTC|newest]
Thread overview: 3+ messages / expand[flat|nested] mbox.gz Atom feed top
2017-09-26 15:37 Luis Henriques
2017-09-27 1:54 ` Dennis Zhou
2017-09-27 9:05 ` Luis Henriques [this message]
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=87lgl0o6jb.fsf@hermes \
--to=lhenriques@suse.com \
--cc=cl@linux.com \
--cc=dennisszhou@gmail.com \
--cc=linux-kernel@vger.kernel.org \
--cc=linux-mm@kvack.org \
--cc=tj@kernel.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox