* Ask help about this patch b951aaff5035 " mm: enable page allocation tagging"
@ 2025-03-20 11:07 Carlos Song
2025-03-20 11:24 ` Kent Overstreet
0 siblings, 1 reply; 8+ messages in thread
From: Carlos Song @ 2025-03-20 11:07 UTC (permalink / raw)
To: surenb, kent.overstreet, akpm, willy
Cc: linux-mm, linux-kernel, linux-fsdevel
[-- Attachment #1: Type: text/plain, Size: 19580 bytes --]
Hi, all
I found a 300ms~600ms IRQ off when writing 1Gb data to mmc device at I.MX7d SDB board at Linux-kernel-v6.14.
But I test the same case at Linux-kernel-v6.7, this longest IRQ off time is only 1ms~2ms. So the issue is introduced from v6.7~v6.14.
Run this cmd to test:
dd if=/dev/zero of=/dev/mmcblk2p4 bs=4096 seek=12500 count=256000 conv=fsync
This issue looks from blkdev_buffered_write() function. Because when I run this cmd with "oflag=direct" to use
blkdev_direct_write(), I can not see any long time IRQ off.
Then I use Ftrace irqoff tracer to trace the longest IRQ off event, I found some differences between v6.7 and v6.14:
In iomap_file_buffered_write(), __folio_alloc (in v6.7) is replaced by _folio_alloc_noprof (in v6.14) by this patch.
The spinlock disabled IRQ ~300ms+. It looks there are some fixes for this patch, but I still can see IRQ off 300ms+ at 6.14.0-rc7-next-20250319.
Do I trigger one bug? I know little about mem so I have to report it and hope I can get some help or guide.
I put my ftrace log at the mail tail to help trace and explain.
commit b951aaff503502a7fe066eeed2744ba8a6413c89
Author: Suren Baghdasaryan surenb@google.com<mailto:surenb@google.com>
Date: Thu Mar 21 09:36:40 2024 -0700
mm: enable page allocation tagging
Redefine page allocators to record allocation tags upon their invocation.
Instrument post_alloc_hook and free_pages_prepare to modify current
allocation tag.
[surenb@google.com: undo _noprof additions in the documentation]
Link: https://lkml.kernel.org/r/20240326231453.1206227-3-surenb@google.com
Link: https://lkml.kernel.org/r/20240321163705.3067592-19-surenb@google.com
Signed-off-by: Suren Baghdasaryan surenb@google.com<mailto:surenb@google.com>
Co-developed-by: Kent Overstreet kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev>
Signed-off-by: Kent Overstreet kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev>
Reviewed-by: Kees Cook keescook@chromium.org<mailto:keescook@chromium.org>
Tested-by: Kees Cook keescook@chromium.org<mailto:keescook@chromium.org>
Cc: Alexander Viro viro@zeniv.linux.org.uk<mailto:viro@zeniv.linux.org.uk>
Cc: Alex Gaynor alex.gaynor@gmail.com<mailto:alex.gaynor@gmail.com>
Cc: Alice Ryhl aliceryhl@google.com<mailto:aliceryhl@google.com>
Cc: Andreas Hindborg a.hindborg@samsung.com<mailto:a.hindborg@samsung.com>
Cc: Benno Lossin benno.lossin@proton.me<mailto:benno.lossin@proton.me>
Cc: "Björn Roy Baron" bjorn3_gh@protonmail.com<mailto:bjorn3_gh@protonmail.com>
Cc: Boqun Feng boqun.feng@gmail.com<mailto:boqun.feng@gmail.com>
Cc: Christoph Lameter cl@linux.com<mailto:cl@linux.com>
Cc: Dennis Zhou dennis@kernel.org<mailto:dennis@kernel.org>
Cc: Gary Guo gary@garyguo.net<mailto:gary@garyguo.net>
Cc: Miguel Ojeda ojeda@kernel.org<mailto:ojeda@kernel.org>
Cc: Pasha Tatashin pasha.tatashin@soleen.com<mailto:pasha.tatashin@soleen.com>
Cc: Peter Zijlstra peterz@infradead.org<mailto:peterz@infradead.org>
Cc: Tejun Heo tj@kernel.org<mailto:tj@kernel.org>
Cc: Vlastimil Babka vbabka@suse.cz<mailto:vbabka@suse.cz>
Cc: Wedson Almeida Filho wedsonaf@gmail.com<mailto:wedsonaf@gmail.com>
Signed-off-by: Andrew Morton akpm@linux-foundation.org<mailto:akpm@linux-foundation.org>
Ftrace irqoff tracer shows detail:
At v6.14:
# tracer: irqsoff
#
# irqsoff latency trace v1.1.5 on 6.14.0-rc7-next-20250319
# --------------------------------------------------------------------
# latency: 279663 us, #21352/21352, CPU#0 | (M:NONE VP:0, KP:0, SP:0 HP:0 #P:2)
# -----------------
# | task: dd-805 (uid:0 nice:0 policy:0 rt_prio:0)
# -----------------
# => started at: __rmqueue_pcplist
# => ended at: _raw_spin_unlock_irqrestore
#
#
# _------=> CPU#
# / _-----=> irqs-off/BH-disabled
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / _-=> migrate-disable
# ||||| / delay
# cmd pid |||||| time | caller
# \ / |||||| \ | /
dd-805 0d.... 1us : __rmqueue_pcplist
dd-805 0d.... 3us : _raw_spin_trylock <-__rmqueue_pcplist
dd-805 0d.... 7us : __mod_zone_page_state <-__rmqueue_pcplist
dd-805 0d.... 10us : __mod_zone_page_state <-__rmqueue_pcplist
dd-805 0d.... 12us : __mod_zone_page_state <-__rmqueue_pcplist
dd-805 0d.... 15us : __mod_zone_page_state <-__rmqueue_pcplist
dd-805 0d.... 17us : __mod_zone_page_state <-__rmqueue_pcplist
dd-805 0d.... 19us : __mod_zone_page_state <-__rmqueue_pcplist
...
dd-805 0d.... 1535us : find_suitable_fallback <-__rmqueue_pcplist
dd-805 0d.... 1538us : find_suitable_fallback <-__rmqueue_pcplist
dd-805 0d.... 1539us : find_suitable_fallback <-__rmqueue_pcplist
dd-805 0d.... 1542us+: try_to_claim_block <-__rmqueue_pcplist
dd-805 0d.... 1597us : find_suitable_fallback <-__rmqueue_pcplist
dd-805 0d.... 1599us+: try_to_claim_block <-__rmqueue_pcplist
dd-805 0d.... 1674us : find_suitable_fallback <-__rmqueue_pcplist
dd-805 0d.... 1676us+: try_to_claim_block <-__rmqueue_pcplist
dd-805 0d.... 1716us : find_suitable_fallback <-__rmqueue_pcplist
dd-805 0d.... 1718us+: try_to_claim_block <-__rmqueue_pcplist
dd-805 0d.... 1801us : find_suitable_fallback <-__rmqueue_pcplist
dd-805 0d.... 1803us+: try_to_claim_block <-__rmqueue_pcplist
...
dd-805 0d.... 279555us : find_suitable_fallback <-__rmqueue_pcplist
dd-805 0d.... 279556us : find_suitable_fallback <-__rmqueue_pcplist
dd-805 0d.... 279558us : find_suitable_fallback <-__rmqueue_pcplist
dd-805 0d.... 279560us+: try_to_claim_block <-__rmqueue_pcplist
dd-805 0d.... 279616us : find_suitable_fallback <-__rmqueue_pcplist
dd-805 0d.... 279618us : __mod_zone_page_state <-__rmqueue_pcplist
dd-805 0d.... 279620us : find_suitable_fallback <-__rmqueue_pcplist
...
dd-805 0d.... 279658us : find_suitable_fallback <-__rmqueue_pcplist
dd-805 0d.... 279660us : _raw_spin_unlock_irqrestore <-__rmqueue_pcplist
dd-805 0d.... 279662us : _raw_spin_unlock_irqrestore
dd-805 0d.... 279666us+: trace_hardirqs_on <-_raw_spin_unlock_irqrestore
dd-805 0d.... 279712us : <stack trace>
=> get_page_from_freelist
=> __alloc_frozen_pages_noprof
=> __folio_alloc_noprof
=> __filemap_get_folio
=> iomap_write_begin
=> iomap_file_buffered_write
=> blkdev_write_iter
=> vfs_write
=> ksys_write
=> ret_fast_syscall
At v6.7:
# tracer: irqsoff
#
# irqsoff latency trace v1.1.5 on 6.7.0
# --------------------------------------------------------------------
# latency: 2477 us, #146/146, CPU#0 | (M:server VP:0, KP:0, SP:0 HP:0 #P:2)
# -----------------
# | task: dd-808 (uid:0 nice:0 policy:0 rt_prio:0)
# -----------------
# => started at: _raw_spin_lock_irqsave
# => ended at: _raw_spin_unlock_irqrestore
#
#
# _------=> CPU#
# / _-----=> irqs-off/BH-disabled
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / _-=> migrate-disable
# ||||| / delay
# cmd pid |||||| time | caller
# \ / |||||| \ | /
dd-808 0d.... 1us!: _raw_spin_lock_irqsave
dd-808 0d.... 186us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 189us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 191us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 192us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 194us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 196us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 199us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 203us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 330us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 332us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 334us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 336us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 338us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 339us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 341us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 343us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 479us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 481us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 483us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 485us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 486us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 488us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 490us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 492us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 630us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 632us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 634us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 636us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 638us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 640us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 642us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 644us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 771us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 773us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 775us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 777us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 778us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 780us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 782us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 784us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 911us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 913us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 915us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 916us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 918us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 920us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 922us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 924us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 1055us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1058us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1059us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1061us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1063us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1065us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1066us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1068us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 1194us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1196us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1198us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1200us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1202us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1203us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1205us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1208us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 1333us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1335us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1337us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1339us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1341us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1342us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1344us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1346us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 1480us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1482us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1484us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1486us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1488us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1490us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1492us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1494us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 1621us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1623us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1625us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1627us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1629us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1630us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1632us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1634us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 1761us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1763us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1765us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1766us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1768us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1770us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1772us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1774us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 1900us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1902us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1903us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1905us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1907us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1909us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1911us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1913us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 2038us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2040us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2042us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2044us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2046us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2047us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2049us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2051us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2053us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2055us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 2175us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2176us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2178us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2180us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2182us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2183us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2185us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2187us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2189us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2191us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2192us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2194us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2196us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 2323us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2325us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2327us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2328us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2330us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2332us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2334us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2335us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2337us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2339us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2341us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2343us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2345us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 2470us : __mod_zone_page_state <-__rmqueue_pcplist
dd-808 0d.... 2473us : _raw_spin_unlock_irqrestore <-__rmqueue_pcplist
dd-808 0d.... 2476us : _raw_spin_unlock_irqrestore
dd-808 0d.... 2479us+: tracer_hardirqs_on <-_raw_spin_unlock_irqrestore
dd-808 0d.... 2520us : <stack trace>
=> get_page_from_freelist
=> __alloc_pages
=> __folio_alloc
=> __filemap_get_folio
=> iomap_write_begin
=> iomap_file_buffered_write
=> blkdev_write_iter
=> vfs_write
=> ksys_write
=> ret_fast_syscall
Best Regard
Carlos Song
[-- Attachment #2: Type: text/html, Size: 54419 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Ask help about this patch b951aaff5035 " mm: enable page allocation tagging"
2025-03-20 11:07 Ask help about this patch b951aaff5035 " mm: enable page allocation tagging" Carlos Song
@ 2025-03-20 11:24 ` Kent Overstreet
2025-03-20 15:50 ` Suren Baghdasaryan
2025-03-27 8:25 ` Carlos Song
0 siblings, 2 replies; 8+ messages in thread
From: Kent Overstreet @ 2025-03-20 11:24 UTC (permalink / raw)
To: Carlos Song; +Cc: surenb, akpm, willy, linux-mm, linux-kernel, linux-fsdevel
On Thu, Mar 20, 2025 at 11:07:41AM +0000, Carlos Song wrote:
> Hi, all
>
> I found a 300ms~600ms IRQ off when writing 1Gb data to mmc device at I.MX7d SDB board at Linux-kernel-v6.14.
> But I test the same case at Linux-kernel-v6.7, this longest IRQ off time is only 1ms~2ms. So the issue is introduced from v6.7~v6.14.
>
> Run this cmd to test:
> dd if=/dev/zero of=/dev/mmcblk2p4 bs=4096 seek=12500 count=256000 conv=fsync
>
> This issue looks from blkdev_buffered_write() function. Because when I run this cmd with "oflag=direct" to use
> blkdev_direct_write(), I can not see any long time IRQ off.
>
> Then I use Ftrace irqoff tracer to trace the longest IRQ off event, I found some differences between v6.7 and v6.14:
> In iomap_file_buffered_write(), __folio_alloc (in v6.7) is replaced by _folio_alloc_noprof (in v6.14) by this patch.
> The spinlock disabled IRQ ~300ms+. It looks there are some fixes for this patch, but I still can see IRQ off 300ms+ at 6.14.0-rc7-next-20250319.
>
> Do I trigger one bug? I know little about mem so I have to report it and hope I can get some help or guide.
> I put my ftrace log at the mail tail to help trace and explain.
Did you track down which spinlock?
>
> commit b951aaff503502a7fe066eeed2744ba8a6413c89
> Author: Suren Baghdasaryan surenb@google.com<mailto:surenb@google.com>
> Date: Thu Mar 21 09:36:40 2024 -0700
>
> mm: enable page allocation tagging
>
> Redefine page allocators to record allocation tags upon their invocation.
> Instrument post_alloc_hook and free_pages_prepare to modify current
> allocation tag.
>
> [surenb@google.com: undo _noprof additions in the documentation]
> Link: https://lkml.kernel.org/r/20240326231453.1206227-3-surenb@google.com
> Link: https://lkml.kernel.org/r/20240321163705.3067592-19-surenb@google.com
> Signed-off-by: Suren Baghdasaryan surenb@google.com<mailto:surenb@google.com>
> Co-developed-by: Kent Overstreet kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev>
> Signed-off-by: Kent Overstreet kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev>
> Reviewed-by: Kees Cook keescook@chromium.org<mailto:keescook@chromium.org>
> Tested-by: Kees Cook keescook@chromium.org<mailto:keescook@chromium.org>
> Cc: Alexander Viro viro@zeniv.linux.org.uk<mailto:viro@zeniv.linux.org.uk>
> Cc: Alex Gaynor alex.gaynor@gmail.com<mailto:alex.gaynor@gmail.com>
> Cc: Alice Ryhl aliceryhl@google.com<mailto:aliceryhl@google.com>
> Cc: Andreas Hindborg a.hindborg@samsung.com<mailto:a.hindborg@samsung.com>
> Cc: Benno Lossin benno.lossin@proton.me<mailto:benno.lossin@proton.me>
> Cc: "Björn Roy Baron" bjorn3_gh@protonmail.com<mailto:bjorn3_gh@protonmail.com>
> Cc: Boqun Feng boqun.feng@gmail.com<mailto:boqun.feng@gmail.com>
> Cc: Christoph Lameter cl@linux.com<mailto:cl@linux.com>
> Cc: Dennis Zhou dennis@kernel.org<mailto:dennis@kernel.org>
> Cc: Gary Guo gary@garyguo.net<mailto:gary@garyguo.net>
> Cc: Miguel Ojeda ojeda@kernel.org<mailto:ojeda@kernel.org>
> Cc: Pasha Tatashin pasha.tatashin@soleen.com<mailto:pasha.tatashin@soleen.com>
> Cc: Peter Zijlstra peterz@infradead.org<mailto:peterz@infradead.org>
> Cc: Tejun Heo tj@kernel.org<mailto:tj@kernel.org>
> Cc: Vlastimil Babka vbabka@suse.cz<mailto:vbabka@suse.cz>
> Cc: Wedson Almeida Filho wedsonaf@gmail.com<mailto:wedsonaf@gmail.com>
> Signed-off-by: Andrew Morton akpm@linux-foundation.org<mailto:akpm@linux-foundation.org>
>
>
> Ftrace irqoff tracer shows detail:
> At v6.14:
> # tracer: irqsoff
> #
> # irqsoff latency trace v1.1.5 on 6.14.0-rc7-next-20250319
> # --------------------------------------------------------------------
> # latency: 279663 us, #21352/21352, CPU#0 | (M:NONE VP:0, KP:0, SP:0 HP:0 #P:2)
> # -----------------
> # | task: dd-805 (uid:0 nice:0 policy:0 rt_prio:0)
> # -----------------
> # => started at: __rmqueue_pcplist
> # => ended at: _raw_spin_unlock_irqrestore
> #
> #
> # _------=> CPU#
> # / _-----=> irqs-off/BH-disabled
> # | / _----=> need-resched
> # || / _---=> hardirq/softirq
> # ||| / _--=> preempt-depth
> # |||| / _-=> migrate-disable
> # ||||| / delay
> # cmd pid |||||| time | caller
> # \ / |||||| \ | /
> dd-805 0d.... 1us : __rmqueue_pcplist
> dd-805 0d.... 3us : _raw_spin_trylock <-__rmqueue_pcplist
> dd-805 0d.... 7us : __mod_zone_page_state <-__rmqueue_pcplist
> dd-805 0d.... 10us : __mod_zone_page_state <-__rmqueue_pcplist
> dd-805 0d.... 12us : __mod_zone_page_state <-__rmqueue_pcplist
> dd-805 0d.... 15us : __mod_zone_page_state <-__rmqueue_pcplist
> dd-805 0d.... 17us : __mod_zone_page_state <-__rmqueue_pcplist
> dd-805 0d.... 19us : __mod_zone_page_state <-__rmqueue_pcplist
> ...
> dd-805 0d.... 1535us : find_suitable_fallback <-__rmqueue_pcplist
> dd-805 0d.... 1538us : find_suitable_fallback <-__rmqueue_pcplist
> dd-805 0d.... 1539us : find_suitable_fallback <-__rmqueue_pcplist
> dd-805 0d.... 1542us+: try_to_claim_block <-__rmqueue_pcplist
> dd-805 0d.... 1597us : find_suitable_fallback <-__rmqueue_pcplist
> dd-805 0d.... 1599us+: try_to_claim_block <-__rmqueue_pcplist
> dd-805 0d.... 1674us : find_suitable_fallback <-__rmqueue_pcplist
> dd-805 0d.... 1676us+: try_to_claim_block <-__rmqueue_pcplist
> dd-805 0d.... 1716us : find_suitable_fallback <-__rmqueue_pcplist
> dd-805 0d.... 1718us+: try_to_claim_block <-__rmqueue_pcplist
> dd-805 0d.... 1801us : find_suitable_fallback <-__rmqueue_pcplist
> dd-805 0d.... 1803us+: try_to_claim_block <-__rmqueue_pcplist
> ...
> dd-805 0d.... 279555us : find_suitable_fallback <-__rmqueue_pcplist
> dd-805 0d.... 279556us : find_suitable_fallback <-__rmqueue_pcplist
> dd-805 0d.... 279558us : find_suitable_fallback <-__rmqueue_pcplist
> dd-805 0d.... 279560us+: try_to_claim_block <-__rmqueue_pcplist
> dd-805 0d.... 279616us : find_suitable_fallback <-__rmqueue_pcplist
> dd-805 0d.... 279618us : __mod_zone_page_state <-__rmqueue_pcplist
> dd-805 0d.... 279620us : find_suitable_fallback <-__rmqueue_pcplist
> ...
> dd-805 0d.... 279658us : find_suitable_fallback <-__rmqueue_pcplist
> dd-805 0d.... 279660us : _raw_spin_unlock_irqrestore <-__rmqueue_pcplist
> dd-805 0d.... 279662us : _raw_spin_unlock_irqrestore
> dd-805 0d.... 279666us+: trace_hardirqs_on <-_raw_spin_unlock_irqrestore
> dd-805 0d.... 279712us : <stack trace>
> => get_page_from_freelist
> => __alloc_frozen_pages_noprof
> => __folio_alloc_noprof
> => __filemap_get_folio
> => iomap_write_begin
> => iomap_file_buffered_write
> => blkdev_write_iter
> => vfs_write
> => ksys_write
> => ret_fast_syscall
>
> At v6.7:
> # tracer: irqsoff
> #
> # irqsoff latency trace v1.1.5 on 6.7.0
> # --------------------------------------------------------------------
> # latency: 2477 us, #146/146, CPU#0 | (M:server VP:0, KP:0, SP:0 HP:0 #P:2)
> # -----------------
> # | task: dd-808 (uid:0 nice:0 policy:0 rt_prio:0)
> # -----------------
> # => started at: _raw_spin_lock_irqsave
> # => ended at: _raw_spin_unlock_irqrestore
> #
> #
> # _------=> CPU#
> # / _-----=> irqs-off/BH-disabled
> # | / _----=> need-resched
> # || / _---=> hardirq/softirq
> # ||| / _--=> preempt-depth
> # |||| / _-=> migrate-disable
> # ||||| / delay
> # cmd pid |||||| time | caller
> # \ / |||||| \ | /
> dd-808 0d.... 1us!: _raw_spin_lock_irqsave
> dd-808 0d.... 186us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 189us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 191us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 192us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 194us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 196us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 199us : steal_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 203us!: move_freepages_block <-steal_suitable_fallback
> dd-808 0d.... 330us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 332us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 334us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 336us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 338us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 339us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 341us : steal_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 343us!: move_freepages_block <-steal_suitable_fallback
> dd-808 0d.... 479us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 481us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 483us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 485us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 486us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 488us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 490us : steal_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 492us!: move_freepages_block <-steal_suitable_fallback
> dd-808 0d.... 630us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 632us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 634us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 636us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 638us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 640us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 642us : steal_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 644us!: move_freepages_block <-steal_suitable_fallback
> dd-808 0d.... 771us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 773us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 775us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 777us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 778us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 780us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 782us : steal_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 784us!: move_freepages_block <-steal_suitable_fallback
> dd-808 0d.... 911us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 913us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 915us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 916us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 918us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 920us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 922us : steal_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 924us!: move_freepages_block <-steal_suitable_fallback
> dd-808 0d.... 1055us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1058us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1059us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1061us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1063us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1065us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1066us : steal_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1068us!: move_freepages_block <-steal_suitable_fallback
> dd-808 0d.... 1194us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1196us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1198us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1200us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1202us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1203us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1205us : steal_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1208us!: move_freepages_block <-steal_suitable_fallback
> dd-808 0d.... 1333us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1335us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1337us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1339us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1341us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1342us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1344us : steal_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1346us!: move_freepages_block <-steal_suitable_fallback
> dd-808 0d.... 1480us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1482us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1484us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1486us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1488us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1490us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1492us : steal_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1494us!: move_freepages_block <-steal_suitable_fallback
> dd-808 0d.... 1621us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1623us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1625us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1627us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1629us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1630us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1632us : steal_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1634us!: move_freepages_block <-steal_suitable_fallback
> dd-808 0d.... 1761us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1763us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1765us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1766us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1768us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1770us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1772us : steal_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1774us!: move_freepages_block <-steal_suitable_fallback
> dd-808 0d.... 1900us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1902us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1903us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1905us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1907us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1909us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1911us : steal_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 1913us!: move_freepages_block <-steal_suitable_fallback
> dd-808 0d.... 2038us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2040us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2042us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2044us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2046us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2047us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2049us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2051us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2053us : steal_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2055us!: move_freepages_block <-steal_suitable_fallback
> dd-808 0d.... 2175us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2176us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2178us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2180us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2182us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2183us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2185us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2187us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2189us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2191us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2192us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2194us : steal_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2196us!: move_freepages_block <-steal_suitable_fallback
> dd-808 0d.... 2323us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2325us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2327us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2328us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2330us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2332us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2334us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2335us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2337us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2339us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2341us : find_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2343us : steal_suitable_fallback <-__rmqueue_pcplist
> dd-808 0d.... 2345us!: move_freepages_block <-steal_suitable_fallback
> dd-808 0d.... 2470us : __mod_zone_page_state <-__rmqueue_pcplist
> dd-808 0d.... 2473us : _raw_spin_unlock_irqrestore <-__rmqueue_pcplist
> dd-808 0d.... 2476us : _raw_spin_unlock_irqrestore
> dd-808 0d.... 2479us+: tracer_hardirqs_on <-_raw_spin_unlock_irqrestore
> dd-808 0d.... 2520us : <stack trace>
> => get_page_from_freelist
> => __alloc_pages
> => __folio_alloc
> => __filemap_get_folio
> => iomap_write_begin
> => iomap_file_buffered_write
> => blkdev_write_iter
> => vfs_write
> => ksys_write
> => ret_fast_syscall
>
> Best Regard
> Carlos Song
>
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Ask help about this patch b951aaff5035 " mm: enable page allocation tagging"
2025-03-20 11:24 ` Kent Overstreet
@ 2025-03-20 15:50 ` Suren Baghdasaryan
2025-03-27 8:20 ` [EXT] " Carlos Song
2025-03-27 8:25 ` Carlos Song
1 sibling, 1 reply; 8+ messages in thread
From: Suren Baghdasaryan @ 2025-03-20 15:50 UTC (permalink / raw)
To: Kent Overstreet
Cc: Carlos Song, akpm, willy, linux-mm, linux-kernel, linux-fsdevel
On Thu, Mar 20, 2025 at 4:24 AM Kent Overstreet
<kent.overstreet@linux.dev> wrote:
>
> On Thu, Mar 20, 2025 at 11:07:41AM +0000, Carlos Song wrote:
> > Hi, all
> >
> > I found a 300ms~600ms IRQ off when writing 1Gb data to mmc device at I.MX7d SDB board at Linux-kernel-v6.14.
> > But I test the same case at Linux-kernel-v6.7, this longest IRQ off time is only 1ms~2ms. So the issue is introduced from v6.7~v6.14.
> >
> > Run this cmd to test:
> > dd if=/dev/zero of=/dev/mmcblk2p4 bs=4096 seek=12500 count=256000 conv=fsync
> >
> > This issue looks from blkdev_buffered_write() function. Because when I run this cmd with "oflag=direct" to use
> > blkdev_direct_write(), I can not see any long time IRQ off.
> >
> > Then I use Ftrace irqoff tracer to trace the longest IRQ off event, I found some differences between v6.7 and v6.14:
> > In iomap_file_buffered_write(), __folio_alloc (in v6.7) is replaced by _folio_alloc_noprof (in v6.14) by this patch.
> > The spinlock disabled IRQ ~300ms+. It looks there are some fixes for this patch, but I still can see IRQ off 300ms+ at 6.14.0-rc7-next-20250319.
Do you have CONFIG_MEM_ALLOC_PROFILING enabled and if so, does the
issue disappear if you disable CONFIG_MEM_ALLOC_PROFILING?
> >
> > Do I trigger one bug? I know little about mem so I have to report it and hope I can get some help or guide.
> > I put my ftrace log at the mail tail to help trace and explain.
>
> Did you track down which spinlock?
>
> >
> > commit b951aaff503502a7fe066eeed2744ba8a6413c89
> > Author: Suren Baghdasaryan surenb@google.com<mailto:surenb@google.com>
> > Date: Thu Mar 21 09:36:40 2024 -0700
> >
> > mm: enable page allocation tagging
> >
> > Redefine page allocators to record allocation tags upon their invocation.
> > Instrument post_alloc_hook and free_pages_prepare to modify current
> > allocation tag.
> >
> > [surenb@google.com: undo _noprof additions in the documentation]
> > Link: https://lkml.kernel.org/r/20240326231453.1206227-3-surenb@google.com
> > Link: https://lkml.kernel.org/r/20240321163705.3067592-19-surenb@google.com
> > Signed-off-by: Suren Baghdasaryan surenb@google.com<mailto:surenb@google.com>
> > Co-developed-by: Kent Overstreet kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev>
> > Signed-off-by: Kent Overstreet kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev>
> > Reviewed-by: Kees Cook keescook@chromium.org<mailto:keescook@chromium.org>
> > Tested-by: Kees Cook keescook@chromium.org<mailto:keescook@chromium.org>
> > Cc: Alexander Viro viro@zeniv.linux.org.uk<mailto:viro@zeniv.linux.org.uk>
> > Cc: Alex Gaynor alex.gaynor@gmail.com<mailto:alex.gaynor@gmail.com>
> > Cc: Alice Ryhl aliceryhl@google.com<mailto:aliceryhl@google.com>
> > Cc: Andreas Hindborg a.hindborg@samsung.com<mailto:a.hindborg@samsung.com>
> > Cc: Benno Lossin benno.lossin@proton.me<mailto:benno.lossin@proton.me>
> > Cc: "Björn Roy Baron" bjorn3_gh@protonmail.com<mailto:bjorn3_gh@protonmail.com>
> > Cc: Boqun Feng boqun.feng@gmail.com<mailto:boqun.feng@gmail.com>
> > Cc: Christoph Lameter cl@linux.com<mailto:cl@linux.com>
> > Cc: Dennis Zhou dennis@kernel.org<mailto:dennis@kernel.org>
> > Cc: Gary Guo gary@garyguo.net<mailto:gary@garyguo.net>
> > Cc: Miguel Ojeda ojeda@kernel.org<mailto:ojeda@kernel.org>
> > Cc: Pasha Tatashin pasha.tatashin@soleen.com<mailto:pasha.tatashin@soleen.com>
> > Cc: Peter Zijlstra peterz@infradead.org<mailto:peterz@infradead.org>
> > Cc: Tejun Heo tj@kernel.org<mailto:tj@kernel.org>
> > Cc: Vlastimil Babka vbabka@suse.cz<mailto:vbabka@suse.cz>
> > Cc: Wedson Almeida Filho wedsonaf@gmail.com<mailto:wedsonaf@gmail.com>
> > Signed-off-by: Andrew Morton akpm@linux-foundation.org<mailto:akpm@linux-foundation.org>
> >
> >
> > Ftrace irqoff tracer shows detail:
> > At v6.14:
> > # tracer: irqsoff
> > #
> > # irqsoff latency trace v1.1.5 on 6.14.0-rc7-next-20250319
> > # --------------------------------------------------------------------
> > # latency: 279663 us, #21352/21352, CPU#0 | (M:NONE VP:0, KP:0, SP:0 HP:0 #P:2)
> > # -----------------
> > # | task: dd-805 (uid:0 nice:0 policy:0 rt_prio:0)
> > # -----------------
> > # => started at: __rmqueue_pcplist
> > # => ended at: _raw_spin_unlock_irqrestore
> > #
> > #
> > # _------=> CPU#
> > # / _-----=> irqs-off/BH-disabled
> > # | / _----=> need-resched
> > # || / _---=> hardirq/softirq
> > # ||| / _--=> preempt-depth
> > # |||| / _-=> migrate-disable
> > # ||||| / delay
> > # cmd pid |||||| time | caller
> > # \ / |||||| \ | /
> > dd-805 0d.... 1us : __rmqueue_pcplist
> > dd-805 0d.... 3us : _raw_spin_trylock <-__rmqueue_pcplist
> > dd-805 0d.... 7us : __mod_zone_page_state <-__rmqueue_pcplist
> > dd-805 0d.... 10us : __mod_zone_page_state <-__rmqueue_pcplist
> > dd-805 0d.... 12us : __mod_zone_page_state <-__rmqueue_pcplist
> > dd-805 0d.... 15us : __mod_zone_page_state <-__rmqueue_pcplist
> > dd-805 0d.... 17us : __mod_zone_page_state <-__rmqueue_pcplist
> > dd-805 0d.... 19us : __mod_zone_page_state <-__rmqueue_pcplist
> > ...
> > dd-805 0d.... 1535us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-805 0d.... 1538us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-805 0d.... 1539us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-805 0d.... 1542us+: try_to_claim_block <-__rmqueue_pcplist
> > dd-805 0d.... 1597us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-805 0d.... 1599us+: try_to_claim_block <-__rmqueue_pcplist
> > dd-805 0d.... 1674us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-805 0d.... 1676us+: try_to_claim_block <-__rmqueue_pcplist
> > dd-805 0d.... 1716us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-805 0d.... 1718us+: try_to_claim_block <-__rmqueue_pcplist
> > dd-805 0d.... 1801us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-805 0d.... 1803us+: try_to_claim_block <-__rmqueue_pcplist
> > ...
> > dd-805 0d.... 279555us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-805 0d.... 279556us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-805 0d.... 279558us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-805 0d.... 279560us+: try_to_claim_block <-__rmqueue_pcplist
> > dd-805 0d.... 279616us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-805 0d.... 279618us : __mod_zone_page_state <-__rmqueue_pcplist
> > dd-805 0d.... 279620us : find_suitable_fallback <-__rmqueue_pcplist
> > ...
> > dd-805 0d.... 279658us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-805 0d.... 279660us : _raw_spin_unlock_irqrestore <-__rmqueue_pcplist
> > dd-805 0d.... 279662us : _raw_spin_unlock_irqrestore
> > dd-805 0d.... 279666us+: trace_hardirqs_on <-_raw_spin_unlock_irqrestore
> > dd-805 0d.... 279712us : <stack trace>
> > => get_page_from_freelist
> > => __alloc_frozen_pages_noprof
> > => __folio_alloc_noprof
> > => __filemap_get_folio
> > => iomap_write_begin
> > => iomap_file_buffered_write
> > => blkdev_write_iter
> > => vfs_write
> > => ksys_write
> > => ret_fast_syscall
> >
> > At v6.7:
> > # tracer: irqsoff
> > #
> > # irqsoff latency trace v1.1.5 on 6.7.0
> > # --------------------------------------------------------------------
> > # latency: 2477 us, #146/146, CPU#0 | (M:server VP:0, KP:0, SP:0 HP:0 #P:2)
> > # -----------------
> > # | task: dd-808 (uid:0 nice:0 policy:0 rt_prio:0)
> > # -----------------
> > # => started at: _raw_spin_lock_irqsave
> > # => ended at: _raw_spin_unlock_irqrestore
> > #
> > #
> > # _------=> CPU#
> > # / _-----=> irqs-off/BH-disabled
> > # | / _----=> need-resched
> > # || / _---=> hardirq/softirq
> > # ||| / _--=> preempt-depth
> > # |||| / _-=> migrate-disable
> > # ||||| / delay
> > # cmd pid |||||| time | caller
> > # \ / |||||| \ | /
> > dd-808 0d.... 1us!: _raw_spin_lock_irqsave
> > dd-808 0d.... 186us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 189us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 191us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 192us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 194us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 196us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 199us : steal_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 203us!: move_freepages_block <-steal_suitable_fallback
> > dd-808 0d.... 330us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 332us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 334us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 336us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 338us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 339us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 341us : steal_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 343us!: move_freepages_block <-steal_suitable_fallback
> > dd-808 0d.... 479us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 481us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 483us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 485us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 486us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 488us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 490us : steal_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 492us!: move_freepages_block <-steal_suitable_fallback
> > dd-808 0d.... 630us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 632us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 634us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 636us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 638us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 640us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 642us : steal_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 644us!: move_freepages_block <-steal_suitable_fallback
> > dd-808 0d.... 771us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 773us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 775us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 777us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 778us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 780us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 782us : steal_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 784us!: move_freepages_block <-steal_suitable_fallback
> > dd-808 0d.... 911us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 913us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 915us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 916us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 918us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 920us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 922us : steal_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 924us!: move_freepages_block <-steal_suitable_fallback
> > dd-808 0d.... 1055us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1058us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1059us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1061us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1063us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1065us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1066us : steal_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1068us!: move_freepages_block <-steal_suitable_fallback
> > dd-808 0d.... 1194us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1196us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1198us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1200us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1202us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1203us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1205us : steal_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1208us!: move_freepages_block <-steal_suitable_fallback
> > dd-808 0d.... 1333us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1335us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1337us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1339us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1341us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1342us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1344us : steal_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1346us!: move_freepages_block <-steal_suitable_fallback
> > dd-808 0d.... 1480us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1482us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1484us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1486us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1488us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1490us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1492us : steal_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1494us!: move_freepages_block <-steal_suitable_fallback
> > dd-808 0d.... 1621us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1623us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1625us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1627us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1629us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1630us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1632us : steal_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1634us!: move_freepages_block <-steal_suitable_fallback
> > dd-808 0d.... 1761us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1763us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1765us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1766us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1768us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1770us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1772us : steal_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1774us!: move_freepages_block <-steal_suitable_fallback
> > dd-808 0d.... 1900us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1902us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1903us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1905us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1907us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1909us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1911us : steal_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 1913us!: move_freepages_block <-steal_suitable_fallback
> > dd-808 0d.... 2038us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2040us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2042us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2044us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2046us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2047us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2049us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2051us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2053us : steal_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2055us!: move_freepages_block <-steal_suitable_fallback
> > dd-808 0d.... 2175us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2176us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2178us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2180us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2182us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2183us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2185us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2187us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2189us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2191us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2192us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2194us : steal_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2196us!: move_freepages_block <-steal_suitable_fallback
> > dd-808 0d.... 2323us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2325us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2327us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2328us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2330us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2332us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2334us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2335us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2337us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2339us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2341us : find_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2343us : steal_suitable_fallback <-__rmqueue_pcplist
> > dd-808 0d.... 2345us!: move_freepages_block <-steal_suitable_fallback
> > dd-808 0d.... 2470us : __mod_zone_page_state <-__rmqueue_pcplist
> > dd-808 0d.... 2473us : _raw_spin_unlock_irqrestore <-__rmqueue_pcplist
> > dd-808 0d.... 2476us : _raw_spin_unlock_irqrestore
> > dd-808 0d.... 2479us+: tracer_hardirqs_on <-_raw_spin_unlock_irqrestore
> > dd-808 0d.... 2520us : <stack trace>
> > => get_page_from_freelist
> > => __alloc_pages
> > => __folio_alloc
> > => __filemap_get_folio
> > => iomap_write_begin
> > => iomap_file_buffered_write
> > => blkdev_write_iter
> > => vfs_write
> > => ksys_write
> > => ret_fast_syscall
> >
> > Best Regard
> > Carlos Song
> >
^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: [EXT] Re: Ask help about this patch b951aaff5035 " mm: enable page allocation tagging"
2025-03-20 15:50 ` Suren Baghdasaryan
@ 2025-03-27 8:20 ` Carlos Song
0 siblings, 0 replies; 8+ messages in thread
From: Carlos Song @ 2025-03-27 8:20 UTC (permalink / raw)
To: Suren Baghdasaryan, Kent Overstreet
Cc: akpm, willy, linux-mm, linux-kernel, linux-fsdevel
> -----Original Message-----
> From: Suren Baghdasaryan <surenb@google.com>
> Sent: Thursday, March 20, 2025 11:50 PM
> To: Kent Overstreet <kent.overstreet@linux.dev>
> Cc: Carlos Song <carlos.song@nxp.com>; akpm@linux-foundation.org;
> willy@infradead.org; linux-mm@kvack.org; linux-kernel@vger.kernel.org;
> linux-fsdevel@vger.kernel.org
> Subject: [EXT] Re: Ask help about this patch b951aaff5035 " mm: enable page
> allocation tagging"
>
> Caution: This is an external email. Please take care when clicking links or
> opening attachments. When in doubt, report the message using the 'Report this
> email' button
>
>
> On Thu, Mar 20, 2025 at 4:24 AM Kent Overstreet <kent.overstreet@linux.dev>
> wrote:
> >
> > On Thu, Mar 20, 2025 at 11:07:41AM +0000, Carlos Song wrote:
> > > Hi, all
> > >
> > > I found a 300ms~600ms IRQ off when writing 1Gb data to mmc device at
> I.MX7d SDB board at Linux-kernel-v6.14.
> > > But I test the same case at Linux-kernel-v6.7, this longest IRQ off time is only
> 1ms~2ms. So the issue is introduced from v6.7~v6.14.
> > >
> > > Run this cmd to test:
> > > dd if=/dev/zero of=/dev/mmcblk2p4 bs=4096 seek=12500 count=256000
> > > conv=fsync
> > >
> > > This issue looks from blkdev_buffered_write() function. Because when
> > > I run this cmd with "oflag=direct" to use blkdev_direct_write(), I can not see
> any long time IRQ off.
> > >
> > > Then I use Ftrace irqoff tracer to trace the longest IRQ off event, I found
> some differences between v6.7 and v6.14:
> > > In iomap_file_buffered_write(), __folio_alloc (in v6.7) is replaced by
> _folio_alloc_noprof (in v6.14) by this patch.
> > > The spinlock disabled IRQ ~300ms+. It looks there are some fixes for this
> patch, but I still can see IRQ off 300ms+ at 6.14.0-rc7-next-20250319.
>
> Do you have CONFIG_MEM_ALLOC_PROFILING enabled and if so, does the issue
> disappear if you disable CONFIG_MEM_ALLOC_PROFILING?
>
Sorry for my late reply and thank you for your quick reply!
We don't enable CONFIG_MEM_ALLOC_PROFILING in kernel.
> > >
> > > Do I trigger one bug? I know little about mem so I have to report it and hope
> I can get some help or guide.
> > > I put my ftrace log at the mail tail to help trace and explain.
> >
> > Did you track down which spinlock?
> >
> > >
> > > commit b951aaff503502a7fe066eeed2744ba8a6413c89
> > > Author: Suren Baghdasaryan
> surenb@google.com<mailto:surenb@google.com>
> > > Date: Thu Mar 21 09:36:40 2024 -0700
> > >
> > > mm: enable page allocation tagging
> > >
> > > Redefine page allocators to record allocation tags upon their
> invocation.
> > > Instrument post_alloc_hook and free_pages_prepare to modify
> current
> > > allocation tag.
> > >
> > > [surenb@google.com: undo _noprof additions in the documentation]
> > > Link:
> https://lkml.kern/
> el.org%2Fr%2F20240326231453.1206227-3-surenb%40google.com&data=05%
> 7C02%7Ccarlos.song%40nxp.com%7Cd28a491e16da40a24a6508dd67c6e95b%
> 7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C638780826228162008
> %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMD
> AwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7
> C&sdata=jCoaXCuYC2cLtjSdHRJn73p0wrrtVc2I36Z16R1dsZI%3D&reserved=0
> > > Link:
> https://lkml.kern/
> el.org%2Fr%2F20240321163705.3067592-19-surenb%40google.com&data=05
> %7C02%7Ccarlos.song%40nxp.com%7Cd28a491e16da40a24a6508dd67c6e95b
> %7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C63878082622817871
> 6%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuM
> DAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%
> 7C&sdata=6ey4%2BMmBB%2FEN%2FIwSrUaWhZFql%2BNYyuuqkW6mYe9weN
> k%3D&reserved=0
> > > Signed-off-by: Suren Baghdasaryan
> surenb@google.com<mailto:surenb@google.com>
> > > Co-developed-by: Kent Overstreet
> kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev>
> > > Signed-off-by: Kent Overstreet
> kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev>
> > > Reviewed-by: Kees Cook
> keescook@chromium.org<mailto:keescook@chromium.org>
> > > Tested-by: Kees Cook
> keescook@chromium.org<mailto:keescook@chromium.org>
> > > Cc: Alexander Viro
> viro@zeniv.linux.org.uk<mailto:viro@zeniv.linux.org.uk>
> > > Cc: Alex Gaynor
> alex.gaynor@gmail.com<mailto:alex.gaynor@gmail.com>
> > > Cc: Alice Ryhl aliceryhl@google.com<mailto:aliceryhl@google.com>
> > > Cc: Andreas Hindborg
> a.hindborg@samsung.com<mailto:a.hindborg@samsung.com>
> > > Cc: Benno Lossin
> benno.lossin@proton.me<mailto:benno.lossin@proton.me>
> > > Cc: "Björn Roy Baron"
> bjorn3_gh@protonmail.com<mailto:bjorn3_gh@protonmail.com>
> > > Cc: Boqun Feng
> boqun.feng@gmail.com<mailto:boqun.feng@gmail.com>
> > > Cc: Christoph Lameter cl@linux.com<mailto:cl@linux.com>
> > > Cc: Dennis Zhou dennis@kernel.org<mailto:dennis@kernel.org>
> > > Cc: Gary Guo gary@garyguo.net<mailto:gary@garyguo.net>
> > > Cc: Miguel Ojeda ojeda@kernel.org<mailto:ojeda@kernel.org>
> > > Cc: Pasha Tatashin
> pasha.tatashin@soleen.com<mailto:pasha.tatashin@soleen.com>
> > > Cc: Peter Zijlstra
> peterz@infradead.org<mailto:peterz@infradead.org>
> > > Cc: Tejun Heo tj@kernel.org<mailto:tj@kernel.org>
> > > Cc: Vlastimil Babka vbabka@suse.cz<mailto:vbabka@suse.cz>
> > > Cc: Wedson Almeida Filho
> wedsonaf@gmail.com<mailto:wedsonaf@gmail.com>
> > > Signed-off-by: Andrew Morton
> > > akpm@linux-foundation.org<mailto:akpm@linux-foundation.org>
> > >
> > >
> > > Ftrace irqoff tracer shows detail:
> > > At v6.14:
> > > # tracer: irqsoff
> > > #
> > > # irqsoff latency trace v1.1.5 on 6.14.0-rc7-next-20250319 #
> > > --------------------------------------------------------------------
> > > # latency: 279663 us, #21352/21352, CPU#0 | (M:NONE VP:0, KP:0, SP:0
> HP:0 #P:2)
> > > # -----------------
> > > # | task: dd-805 (uid:0 nice:0 policy:0 rt_prio:0)
> > > # -----------------
> > > # => started at: __rmqueue_pcplist
> > > # => ended at: _raw_spin_unlock_irqrestore
> > > #
> > > #
> > > # _------=> CPU#
> > > # / _-----=> irqs-off/BH-disabled
> > > # | / _----=> need-resched
> > > # || / _---=> hardirq/softirq
> > > # ||| / _--=> preempt-depth
> > > # |||| / _-=> migrate-disable
> > > # ||||| / delay
> > > # cmd pid |||||| time | caller
> > > # \ / |||||| \ | /
> > > dd-805 0d.... 1us : __rmqueue_pcplist
> > > dd-805 0d.... 3us : _raw_spin_trylock
> <-__rmqueue_pcplist
> > > dd-805 0d.... 7us : __mod_zone_page_state
> <-__rmqueue_pcplist
> > > dd-805 0d.... 10us : __mod_zone_page_state
> <-__rmqueue_pcplist
> > > dd-805 0d.... 12us : __mod_zone_page_state
> <-__rmqueue_pcplist
> > > dd-805 0d.... 15us : __mod_zone_page_state
> <-__rmqueue_pcplist
> > > dd-805 0d.... 17us : __mod_zone_page_state
> <-__rmqueue_pcplist
> > > dd-805 0d.... 19us : __mod_zone_page_state
> <-__rmqueue_pcplist
> > > ...
> > > dd-805 0d.... 1535us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-805 0d.... 1538us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-805 0d.... 1539us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-805 0d.... 1542us+: try_to_claim_block
> <-__rmqueue_pcplist
> > > dd-805 0d.... 1597us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-805 0d.... 1599us+: try_to_claim_block
> <-__rmqueue_pcplist
> > > dd-805 0d.... 1674us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-805 0d.... 1676us+: try_to_claim_block
> <-__rmqueue_pcplist
> > > dd-805 0d.... 1716us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-805 0d.... 1718us+: try_to_claim_block
> <-__rmqueue_pcplist
> > > dd-805 0d.... 1801us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-805 0d.... 1803us+: try_to_claim_block
> <-__rmqueue_pcplist
> > > ...
> > > dd-805 0d.... 279555us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-805 0d.... 279556us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-805 0d.... 279558us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-805 0d.... 279560us+: try_to_claim_block
> <-__rmqueue_pcplist
> > > dd-805 0d.... 279616us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-805 0d.... 279618us : __mod_zone_page_state
> <-__rmqueue_pcplist
> > > dd-805 0d.... 279620us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > ...
> > > dd-805 0d.... 279658us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-805 0d.... 279660us : _raw_spin_unlock_irqrestore
> <-__rmqueue_pcplist
> > > dd-805 0d.... 279662us : _raw_spin_unlock_irqrestore
> > > dd-805 0d.... 279666us+: trace_hardirqs_on
> <-_raw_spin_unlock_irqrestore
> > > dd-805 0d.... 279712us : <stack trace>
> > > => get_page_from_freelist
> > > => __alloc_frozen_pages_noprof
> > > => __folio_alloc_noprof
> > > => __filemap_get_folio
> > > => iomap_write_begin
> > > => iomap_file_buffered_write
> > > => blkdev_write_iter
> > > => vfs_write
> > > => ksys_write
> > > => ret_fast_syscall
> > >
> > > At v6.7:
> > > # tracer: irqsoff
> > > #
> > > # irqsoff latency trace v1.1.5 on 6.7.0 #
> > > --------------------------------------------------------------------
> > > # latency: 2477 us, #146/146, CPU#0 | (M:server VP:0, KP:0, SP:0 HP:0 #P:2)
> > > # -----------------
> > > # | task: dd-808 (uid:0 nice:0 policy:0 rt_prio:0)
> > > # -----------------
> > > # => started at: _raw_spin_lock_irqsave
> > > # => ended at: _raw_spin_unlock_irqrestore
> > > #
> > > #
> > > # _------=> CPU#
> > > # / _-----=> irqs-off/BH-disabled
> > > # | / _----=> need-resched
> > > # || / _---=> hardirq/softirq
> > > # ||| / _--=> preempt-depth
> > > # |||| / _-=> migrate-disable
> > > # ||||| / delay
> > > # cmd pid |||||| time | caller
> > > # \ / |||||| \ | /
> > > dd-808 0d.... 1us!: _raw_spin_lock_irqsave
> > > dd-808 0d.... 186us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 189us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 191us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 192us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 194us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 196us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 199us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 203us!: move_freepages_block
> <-steal_suitable_fallback
> > > dd-808 0d.... 330us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 332us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 334us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 336us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 338us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 339us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 341us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 343us!: move_freepages_block
> <-steal_suitable_fallback
> > > dd-808 0d.... 479us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 481us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 483us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 485us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 486us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 488us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 490us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 492us!: move_freepages_block
> <-steal_suitable_fallback
> > > dd-808 0d.... 630us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 632us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 634us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 636us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 638us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 640us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 642us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 644us!: move_freepages_block
> <-steal_suitable_fallback
> > > dd-808 0d.... 771us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 773us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 775us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 777us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 778us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 780us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 782us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 784us!: move_freepages_block
> <-steal_suitable_fallback
> > > dd-808 0d.... 911us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 913us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 915us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 916us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 918us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 920us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 922us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 924us!: move_freepages_block
> <-steal_suitable_fallback
> > > dd-808 0d.... 1055us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1058us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1059us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1061us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1063us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1065us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1066us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1068us!: move_freepages_block
> <-steal_suitable_fallback
> > > dd-808 0d.... 1194us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1196us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1198us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1200us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1202us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1203us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1205us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1208us!: move_freepages_block
> <-steal_suitable_fallback
> > > dd-808 0d.... 1333us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1335us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1337us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1339us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1341us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1342us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1344us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1346us!: move_freepages_block
> <-steal_suitable_fallback
> > > dd-808 0d.... 1480us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1482us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1484us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1486us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1488us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1490us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1492us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1494us!: move_freepages_block
> <-steal_suitable_fallback
> > > dd-808 0d.... 1621us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1623us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1625us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1627us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1629us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1630us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1632us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1634us!: move_freepages_block
> <-steal_suitable_fallback
> > > dd-808 0d.... 1761us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1763us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1765us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1766us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1768us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1770us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1772us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1774us!: move_freepages_block
> <-steal_suitable_fallback
> > > dd-808 0d.... 1900us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1902us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1903us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1905us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1907us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1909us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1911us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 1913us!: move_freepages_block
> <-steal_suitable_fallback
> > > dd-808 0d.... 2038us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2040us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2042us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2044us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2046us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2047us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2049us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2051us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2053us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2055us!: move_freepages_block
> <-steal_suitable_fallback
> > > dd-808 0d.... 2175us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2176us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2178us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2180us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2182us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2183us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2185us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2187us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2189us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2191us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2192us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2194us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2196us!: move_freepages_block
> <-steal_suitable_fallback
> > > dd-808 0d.... 2323us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2325us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2327us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2328us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2330us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2332us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2334us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2335us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2337us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2339us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2341us : find_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2343us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2345us!: move_freepages_block
> <-steal_suitable_fallback
> > > dd-808 0d.... 2470us : __mod_zone_page_state
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2473us : _raw_spin_unlock_irqrestore
> <-__rmqueue_pcplist
> > > dd-808 0d.... 2476us : _raw_spin_unlock_irqrestore
> > > dd-808 0d.... 2479us+: tracer_hardirqs_on
> <-_raw_spin_unlock_irqrestore
> > > dd-808 0d.... 2520us : <stack trace>
> > > => get_page_from_freelist
> > > => __alloc_pages
> > > => __folio_alloc
> > > => __filemap_get_folio
> > > => iomap_write_begin
> > > => iomap_file_buffered_write
> > > => blkdev_write_iter
> > > => vfs_write
> > > => ksys_write
> > > => ret_fast_syscall
> > >
> > > Best Regard
> > > Carlos Song
> > >
^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: [EXT] Re: Ask help about this patch b951aaff5035 " mm: enable page allocation tagging"
2025-03-20 11:24 ` Kent Overstreet
2025-03-20 15:50 ` Suren Baghdasaryan
@ 2025-03-27 8:25 ` Carlos Song
2025-03-27 11:02 ` Carlos Song
1 sibling, 1 reply; 8+ messages in thread
From: Carlos Song @ 2025-03-27 8:25 UTC (permalink / raw)
To: Kent Overstreet
Cc: surenb, akpm, willy, linux-mm, linux-kernel, linux-fsdevel
> -----Original Message-----
> From: Kent Overstreet <kent.overstreet@linux.dev>
> Sent: Thursday, March 20, 2025 7:25 PM
> To: Carlos Song <carlos.song@nxp.com>
> Cc: surenb@google.com; akpm@linux-foundation.org; willy@infradead.org;
> linux-mm@kvack.org; linux-kernel@vger.kernel.org;
> linux-fsdevel@vger.kernel.org
> Subject: [EXT] Re: Ask help about this patch b951aaff5035 " mm: enable page
> allocation tagging"
>
> Caution: This is an external email. Please take care when clicking links or
> opening attachments. When in doubt, report the message using the 'Report this
> email' button
>
>
> On Thu, Mar 20, 2025 at 11:07:41AM +0000, Carlos Song wrote:
> > Hi, all
> >
> > I found a 300ms~600ms IRQ off when writing 1Gb data to mmc device at
> I.MX7d SDB board at Linux-kernel-v6.14.
> > But I test the same case at Linux-kernel-v6.7, this longest IRQ off time is only
> 1ms~2ms. So the issue is introduced from v6.7~v6.14.
> >
> > Run this cmd to test:
> > dd if=/dev/zero of=/dev/mmcblk2p4 bs=4096 seek=12500 count=256000
> > conv=fsync
> >
> > This issue looks from blkdev_buffered_write() function. Because when I
> > run this cmd with "oflag=direct" to use blkdev_direct_write(), I can not see
> any long time IRQ off.
> >
> > Then I use Ftrace irqoff tracer to trace the longest IRQ off event, I found some
> differences between v6.7 and v6.14:
> > In iomap_file_buffered_write(), __folio_alloc (in v6.7) is replaced by
> _folio_alloc_noprof (in v6.14) by this patch.
> > The spinlock disabled IRQ ~300ms+. It looks there are some fixes for this patch,
> but I still can see IRQ off 300ms+ at 6.14.0-rc7-next-20250319.
> >
> > Do I trigger one bug? I know little about mem so I have to report it and hope I
> can get some help or guide.
> > I put my ftrace log at the mail tail to help trace and explain.
>
> Did you track down which spinlock?
>
> >
Hi,
Sorry for my late reply and thank you for your quick reply!
From the trace log, I think the spinlock is from here like this:
__alloc_frozen_pages_noprof ==> get_page_from_freelist==>spinclok
Do you need other log not only this?
> > => get_page_from_freelist"
> > => __alloc_frozen_pages_noprof
> > => __folio_alloc_noprof
> > => __filemap_get_folio
> > commit b951aaff503502a7fe066eeed2744ba8a6413c89
> > Author: Suren Baghdasaryan
> surenb@google.com<mailto:surenb@google.com>
> > Date: Thu Mar 21 09:36:40 2024 -0700
> >
> > mm: enable page allocation tagging
> >
> > Redefine page allocators to record allocation tags upon their invocation.
> > Instrument post_alloc_hook and free_pages_prepare to modify current
> > allocation tag.
> >
> > [surenb@google.com: undo _noprof additions in the documentation]
> > Link:
> https://lkml.kern/
> el.org%2Fr%2F20240326231453.1206227-3-surenb%40google.com&data=05%
> 7C02%7Ccarlos.song%40nxp.com%7Cdf027bbb97074fc2cde808dd67a1d6c5%7
> C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C638780666983046738%
> 7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDA
> wMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C
> &sdata=Aby13KyQjF5pcbYW%2BkfsEJLiaPmS2ZiJLUHJ%2BCr2JXM%3D&reserve
> d=0
> > Link:
> https://lkml.kern/
> el.org%2Fr%2F20240321163705.3067592-19-surenb%40google.com&data=05
> %7C02%7Ccarlos.song%40nxp.com%7Cdf027bbb97074fc2cde808dd67a1d6c5%
> 7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C638780666983065790
> %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMD
> AwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7
> C&sdata=n0hjs2fhjYs%2BcnbrxHy4vFK6D4GFVL4%2Fu72anOOLiEI%3D&reserve
> d=0
> > Signed-off-by: Suren Baghdasaryan
> surenb@google.com<mailto:surenb@google.com>
> > Co-developed-by: Kent Overstreet
> kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev>
> > Signed-off-by: Kent Overstreet
> kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev>
> > Reviewed-by: Kees Cook
> keescook@chromium.org<mailto:keescook@chromium.org>
> > Tested-by: Kees Cook
> keescook@chromium.org<mailto:keescook@chromium.org>
> > Cc: Alexander Viro
> viro@zeniv.linux.org.uk<mailto:viro@zeniv.linux.org.uk>
> > Cc: Alex Gaynor
> alex.gaynor@gmail.com<mailto:alex.gaynor@gmail.com>
> > Cc: Alice Ryhl aliceryhl@google.com<mailto:aliceryhl@google.com>
> > Cc: Andreas Hindborg
> a.hindborg@samsung.com<mailto:a.hindborg@samsung.com>
> > Cc: Benno Lossin
> benno.lossin@proton.me<mailto:benno.lossin@proton.me>
> > Cc: "Björn Roy Baron"
> bjorn3_gh@protonmail.com<mailto:bjorn3_gh@protonmail.com>
> > Cc: Boqun Feng
> boqun.feng@gmail.com<mailto:boqun.feng@gmail.com>
> > Cc: Christoph Lameter cl@linux.com<mailto:cl@linux.com>
> > Cc: Dennis Zhou dennis@kernel.org<mailto:dennis@kernel.org>
> > Cc: Gary Guo gary@garyguo.net<mailto:gary@garyguo.net>
> > Cc: Miguel Ojeda ojeda@kernel.org<mailto:ojeda@kernel.org>
> > Cc: Pasha Tatashin
> pasha.tatashin@soleen.com<mailto:pasha.tatashin@soleen.com>
> > Cc: Peter Zijlstra peterz@infradead.org<mailto:peterz@infradead.org>
> > Cc: Tejun Heo tj@kernel.org<mailto:tj@kernel.org>
> > Cc: Vlastimil Babka vbabka@suse.cz<mailto:vbabka@suse.cz>
> > Cc: Wedson Almeida Filho
> wedsonaf@gmail.com<mailto:wedsonaf@gmail.com>
> > Signed-off-by: Andrew Morton
> > akpm@linux-foundation.org<mailto:akpm@linux-foundation.org>
> >
> >
> > Ftrace irqoff tracer shows detail:
> > At v6.14:
> > # tracer: irqsoff
> > #
> > # irqsoff latency trace v1.1.5 on 6.14.0-rc7-next-20250319 #
> > --------------------------------------------------------------------
> > # latency: 279663 us, #21352/21352, CPU#0 | (M:NONE VP:0, KP:0, SP:0 HP:0
> #P:2)
> > # -----------------
> > # | task: dd-805 (uid:0 nice:0 policy:0 rt_prio:0)
> > # -----------------
> > # => started at: __rmqueue_pcplist
> > # => ended at: _raw_spin_unlock_irqrestore
> > #
> > #
> > # _------=> CPU#
> > # / _-----=> irqs-off/BH-disabled
> > # | / _----=> need-resched
> > # || / _---=> hardirq/softirq
> > # ||| / _--=> preempt-depth
> > # |||| / _-=> migrate-disable
> > # ||||| / delay
> > # cmd pid |||||| time | caller
> > # \ / |||||| \ | /
> > dd-805 0d.... 1us : __rmqueue_pcplist
> > dd-805 0d.... 3us : _raw_spin_trylock
> <-__rmqueue_pcplist
> > dd-805 0d.... 7us : __mod_zone_page_state
> <-__rmqueue_pcplist
> > dd-805 0d.... 10us : __mod_zone_page_state
> <-__rmqueue_pcplist
> > dd-805 0d.... 12us : __mod_zone_page_state
> <-__rmqueue_pcplist
> > dd-805 0d.... 15us : __mod_zone_page_state
> <-__rmqueue_pcplist
> > dd-805 0d.... 17us : __mod_zone_page_state
> <-__rmqueue_pcplist
> > dd-805 0d.... 19us : __mod_zone_page_state
> <-__rmqueue_pcplist
> > ...
> > dd-805 0d.... 1535us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-805 0d.... 1538us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-805 0d.... 1539us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-805 0d.... 1542us+: try_to_claim_block
> <-__rmqueue_pcplist
> > dd-805 0d.... 1597us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-805 0d.... 1599us+: try_to_claim_block
> <-__rmqueue_pcplist
> > dd-805 0d.... 1674us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-805 0d.... 1676us+: try_to_claim_block
> <-__rmqueue_pcplist
> > dd-805 0d.... 1716us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-805 0d.... 1718us+: try_to_claim_block
> <-__rmqueue_pcplist
> > dd-805 0d.... 1801us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-805 0d.... 1803us+: try_to_claim_block
> <-__rmqueue_pcplist
> > ...
> > dd-805 0d.... 279555us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-805 0d.... 279556us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-805 0d.... 279558us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-805 0d.... 279560us+: try_to_claim_block
> <-__rmqueue_pcplist
> > dd-805 0d.... 279616us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-805 0d.... 279618us : __mod_zone_page_state
> <-__rmqueue_pcplist
> > dd-805 0d.... 279620us : find_suitable_fallback
> <-__rmqueue_pcplist
> > ...
> > dd-805 0d.... 279658us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-805 0d.... 279660us : _raw_spin_unlock_irqrestore
> <-__rmqueue_pcplist
> > dd-805 0d.... 279662us : _raw_spin_unlock_irqrestore
> > dd-805 0d.... 279666us+: trace_hardirqs_on
> <-_raw_spin_unlock_irqrestore
> > dd-805 0d.... 279712us : <stack trace>
> > => get_page_from_freelist
> > => __alloc_frozen_pages_noprof
> > => __folio_alloc_noprof
> > => __filemap_get_folio
> > => iomap_write_begin
> > => iomap_file_buffered_write
> > => blkdev_write_iter
> > => vfs_write
> > => ksys_write
> > => ret_fast_syscall
> >
> > At v6.7:
> > # tracer: irqsoff
> > #
> > # irqsoff latency trace v1.1.5 on 6.7.0 #
> > --------------------------------------------------------------------
> > # latency: 2477 us, #146/146, CPU#0 | (M:server VP:0, KP:0, SP:0 HP:0 #P:2)
> > # -----------------
> > # | task: dd-808 (uid:0 nice:0 policy:0 rt_prio:0)
> > # -----------------
> > # => started at: _raw_spin_lock_irqsave
> > # => ended at: _raw_spin_unlock_irqrestore
> > #
> > #
> > # _------=> CPU#
> > # / _-----=> irqs-off/BH-disabled
> > # | / _----=> need-resched
> > # || / _---=> hardirq/softirq
> > # ||| / _--=> preempt-depth
> > # |||| / _-=> migrate-disable
> > # ||||| / delay
> > # cmd pid |||||| time | caller
> > # \ / |||||| \ | /
> > dd-808 0d.... 1us!: _raw_spin_lock_irqsave
> > dd-808 0d.... 186us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 189us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 191us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 192us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 194us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 196us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 199us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 203us!: move_freepages_block
> <-steal_suitable_fallback
> > dd-808 0d.... 330us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 332us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 334us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 336us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 338us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 339us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 341us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 343us!: move_freepages_block
> <-steal_suitable_fallback
> > dd-808 0d.... 479us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 481us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 483us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 485us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 486us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 488us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 490us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 492us!: move_freepages_block
> <-steal_suitable_fallback
> > dd-808 0d.... 630us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 632us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 634us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 636us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 638us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 640us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 642us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 644us!: move_freepages_block
> <-steal_suitable_fallback
> > dd-808 0d.... 771us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 773us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 775us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 777us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 778us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 780us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 782us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 784us!: move_freepages_block
> <-steal_suitable_fallback
> > dd-808 0d.... 911us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 913us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 915us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 916us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 918us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 920us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 922us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 924us!: move_freepages_block
> <-steal_suitable_fallback
> > dd-808 0d.... 1055us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1058us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1059us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1061us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1063us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1065us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1066us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1068us!: move_freepages_block
> <-steal_suitable_fallback
> > dd-808 0d.... 1194us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1196us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1198us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1200us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1202us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1203us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1205us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1208us!: move_freepages_block
> <-steal_suitable_fallback
> > dd-808 0d.... 1333us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1335us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1337us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1339us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1341us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1342us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1344us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1346us!: move_freepages_block
> <-steal_suitable_fallback
> > dd-808 0d.... 1480us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1482us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1484us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1486us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1488us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1490us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1492us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1494us!: move_freepages_block
> <-steal_suitable_fallback
> > dd-808 0d.... 1621us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1623us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1625us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1627us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1629us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1630us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1632us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1634us!: move_freepages_block
> <-steal_suitable_fallback
> > dd-808 0d.... 1761us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1763us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1765us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1766us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1768us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1770us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1772us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1774us!: move_freepages_block
> <-steal_suitable_fallback
> > dd-808 0d.... 1900us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1902us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1903us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1905us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1907us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1909us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1911us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 1913us!: move_freepages_block
> <-steal_suitable_fallback
> > dd-808 0d.... 2038us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2040us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2042us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2044us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2046us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2047us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2049us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2051us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2053us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2055us!: move_freepages_block
> <-steal_suitable_fallback
> > dd-808 0d.... 2175us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2176us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2178us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2180us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2182us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2183us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2185us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2187us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2189us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2191us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2192us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2194us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2196us!: move_freepages_block
> <-steal_suitable_fallback
> > dd-808 0d.... 2323us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2325us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2327us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2328us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2330us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2332us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2334us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2335us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2337us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2339us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2341us : find_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2343us : steal_suitable_fallback
> <-__rmqueue_pcplist
> > dd-808 0d.... 2345us!: move_freepages_block
> <-steal_suitable_fallback
> > dd-808 0d.... 2470us : __mod_zone_page_state
> <-__rmqueue_pcplist
> > dd-808 0d.... 2473us : _raw_spin_unlock_irqrestore
> <-__rmqueue_pcplist
> > dd-808 0d.... 2476us : _raw_spin_unlock_irqrestore
> > dd-808 0d.... 2479us+: tracer_hardirqs_on
> <-_raw_spin_unlock_irqrestore
> > dd-808 0d.... 2520us : <stack trace>
> > => get_page_from_freelist
> > => __alloc_pages
> > => __folio_alloc
> > => __filemap_get_folio
> > => iomap_write_begin
> > => iomap_file_buffered_write
> > => blkdev_write_iter
> > => vfs_write
> > => ksys_write
> > => ret_fast_syscall
> >
> > Best Regard
> > Carlos Song
> >
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Ask help about this patch b951aaff5035 " mm: enable page allocation tagging"
2025-03-27 8:25 ` Carlos Song
@ 2025-03-27 11:02 ` Carlos Song
2025-03-31 20:01 ` Suren Baghdasaryan
0 siblings, 1 reply; 8+ messages in thread
From: Carlos Song @ 2025-03-27 11:02 UTC (permalink / raw)
To: Kent Overstreet
Cc: surenb, akpm, willy, linux-mm, linux-kernel, linux-fsdevel
Hi,
I get the function calling step log, I hope it can help debug little.
The spinlock_irqsave from 136.725021 to 136.966962_raw_spin_unlock_irqrestore.
Around 260ms? It is Just one case, sometimes it will spend more time.
dd-828 [000] ..... 136.724980: fdget_pos <-ksys_write
dd-828 [000] ..... 136.724982: vfs_write <-ksys_write
dd-828 [000] ..... 136.724983: blkdev_write_iter <-vfs_write
dd-828 [000] ..... 136.724984: I_BDEV <-blkdev_write_iter
dd-828 [000] ..... 136.724985: file_update_time <-blkdev_write_iter
dd-828 [000] ..... 136.724986: inode_needs_update_time <-file_update_time
dd-828 [000] ..... 136.724987: ktime_get_coarse_real_ts64 <-inode_needs_update_time
dd-828 [000] ..... 136.724989: timestamp_truncate <-inode_needs_update_time
dd-828 [000] ..... 136.724990: iomap_file_buffered_write <-blkdev_write_iter
dd-828 [000] ..... 136.724992: iomap_iter <-iomap_file_buffered_write
dd-828 [000] ..... 136.724993: blkdev_iomap_begin <-iomap_iter
dd-828 [000] ..... 136.724994: I_BDEV <-blkdev_iomap_begin
dd-828 [000] ..... 136.724995: balance_dirty_pages_ratelimited_flags <-iomap_file_buffered_write
dd-828 [000] ..... 136.724996: inode_to_bdi <-balance_dirty_pages_ratelimited_flags
dd-828 [000] ..... 136.724997: I_BDEV <-inode_to_bdi
dd-828 [000] ..... 136.724998: preempt_count_add <-balance_dirty_pages_ratelimited_flags
dd-828 [000] ...1. 136.725000: preempt_count_sub <-balance_dirty_pages_ratelimited_flags
dd-828 [000] ..... 136.725001: fault_in_readable <-fault_in_iov_iter_readable
dd-828 [000] ..... 136.725003: iomap_write_begin <-iomap_file_buffered_write
dd-828 [000] ..... 136.725004: iomap_get_folio <-iomap_write_begin
dd-828 [000] ..... 136.725005: __filemap_get_folio <-iomap_write_begin
dd-828 [000] ..... 136.725006: filemap_get_entry <-__filemap_get_folio
dd-828 [000] ..... 136.725007: __rcu_read_lock <-filemap_get_entry
dd-828 [000] ..... 136.725008: __rcu_read_unlock <-filemap_get_entry
dd-828 [000] ..... 136.725009: inode_to_bdi <-__filemap_get_folio
dd-828 [000] ..... 136.725010: I_BDEV <-inode_to_bdi
dd-828 [000] ..... 136.725012: __folio_alloc_noprof <-__filemap_get_folio
dd-828 [000] ..... 136.725013: __alloc_pages_noprof <-__folio_alloc_noprof
dd-828 [000] ..... 136.725014: get_page_from_freelist <-__alloc_pages_noprof
dd-828 [000] ..... 136.725015: node_dirty_ok <-get_page_from_freelist
dd-828 [000] ..... 136.725016: preempt_count_add <-get_page_from_freelist
dd-828 [000] ...1. 136.725018: _raw_spin_trylock <-get_page_from_freelist
dd-828 [000] ...1. 136.725019: preempt_count_add <-_raw_spin_trylock
dd-828 [000] ...2. 136.725021: _raw_spin_lock_irqsave <-__rmqueue_pcplist
dd-828 [000] d..2. 136.725022: preempt_count_add <-_raw_spin_lock_irqsave
dd-828 [000] d..3. 136.725025: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725028: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725029: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725031: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725032: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725034: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725036: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725037: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725038: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725039: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725040: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725042: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725043: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725044: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725046: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725047: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725048: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725050: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725051: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725052: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725053: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725055: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725056: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725057: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725058: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725060: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725061: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725062: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725063: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725065: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725066: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725067: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725068: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725070: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725071: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725072: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725073: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725075: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725076: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725077: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725079: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725080: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725081: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725082: __mod_zone_page_state <-__rmqueue_pcplist
dd-828 [000] d..3. 136.725083: __mod_zone_page_state <-__rmqueue_pcplist
...
dd-828 [000] d..3. 136.820279: find_suitable_fallback <-__rmqueue_pcplist
dd-828 [000] d..3. 136.820280: find_suitable_fallback <-__rmqueue_pcplist
dd-828 [000] d..3. 136.820281: find_suitable_fallback <-__rmqueue_pcplist
dd-828 [000] d..3. 136.820282: find_suitable_fallback <-__rmqueue_pcplist
dd-828 [000] d..3. 136.820283: find_suitable_fallback <-__rmqueue_pcplist
dd-828 [000] d..3. 136.820284: find_suitable_fallback <-__rmqueue_pcplist
dd-828 [000] d..3. 136.820285: find_suitable_fallback <-__rmqueue_pcplist
dd-828 [000] d..3. 136.820286: find_suitable_fallback <-__rmqueue_pcplist
dd-828 [000] d..3. 136.820287: find_suitable_fallback <-__rmqueue_pcplist
dd-828 [000] d..3. 136.820289: find_suitable_fallback <-__rmqueue_pcplist
dd-828 [000] d..3. 136.820290: steal_suitable_fallback <-__rmqueue_pcplist
dd-828 [000] d..3. 136.820525: __mod_zone_page_state <-steal_suitable_fallback
...
dd-828 [000] d..3. 136.966945: __mod_zone_page_state <-steal_suitable_fallback
dd-828 [000] d..3. 136.966946: find_suitable_fallback <-__rmqueue_pcplist
dd-828 [000] d..3. 136.966947: find_suitable_fallback <-__rmqueue_pcplist
dd-828 [000] d..3. 136.966948: find_suitable_fallback <-__rmqueue_pcplist
dd-828 [000] d..3. 136.966950: find_suitable_fallback <-__rmqueue_pcplist
dd-828 [000] d..3. 136.966951: find_suitable_fallback <-__rmqueue_pcplist
dd-828 [000] d..3. 136.966952: find_suitable_fallback <-__rmqueue_pcplist
dd-828 [000] d..3. 136.966953: find_suitable_fallback <-__rmqueue_pcplist
dd-828 [000] d..3. 136.966954: find_suitable_fallback <-__rmqueue_pcplist
dd-828 [000] d..3. 136.966955: find_suitable_fallback <-__rmqueue_pcplist
dd-828 [000] d..3. 136.966956: find_suitable_fallback <-__rmqueue_pcplist
dd-828 [000] d..3. 136.966957: find_suitable_fallback <-__rmqueue_pcplist
dd-828 [000] d..3. 136.966958: find_suitable_fallback <-__rmqueue_pcplist
dd-828 [000] d..3. 136.966959: find_suitable_fallback <-__rmqueue_pcplist
dd-828 [000] d..3. 136.966961: find_suitable_fallback <-__rmqueue_pcplist
dd-828 [000] d..3. 136.966962: _raw_spin_unlock_irqrestore <-__rmqueue_pcplist
dd-828 [000] d..3. 136.966965: irq_enter <-generic_handle_arch_irq
dd-828 [000] d..3. 136.966966: irq_enter_rcu <-generic_handle_arch_irq
dd-828 [000] d..3. 136.966967: preempt_count_add <-irq_enter_rcu
dd-828 [000] d.h3. 136.966969: gic_handle_irq <-generic_handle_arch_irq
dd-828 [000] d.h3. 136.966970: generic_handle_domain_irq <-gic_handle_irq
dd-828 [000] d.h3. 136.966971: __irq_resolve_mapping <-generic_handle_domain_irq
dd-828 [000] d.h3. 136.966972: __rcu_read_lock <-__irq_resolve_mapping
dd-828 [000] d.h3. 136.966974: __rcu_read_unlock <-__irq_resolve_mapping
dd-828 [000] d.h3. 136.966975: handle_irq_desc <-gic_handle_irq
dd-828 [000] d.h3. 136.966976: handle_percpu_devid_irq <-handle_irq_desc
dd-828 [000] d.h3. 136.966978: ipi_handler <-handle_percpu_devid_irq
dd-828 [000] d.h3. 136.966979: do_handle_IPI <-ipi_handler
dd-828 [000] d.h3. 136.966981: generic_smp_call_function_single_interrupt <-do_handle_IPI
dd-828 [000] d.h3. 136.966982: __flush_smp_call_function_queue <-do_handle_IPI
dd-828 [000] d.h3. 136.966986: sched_ttwu_pending <-__flush_smp_call_function_queue
dd-828 [000] d.h3. 136.966987: preempt_count_add <-sched_ttwu_pending
dd-828 [000] d.h4. 136.966990: _raw_spin_lock <-sched_ttwu_pending
dd-828 [000] d.h4. 136.966991: preempt_count_add <-_raw_spin_lock
dd-828 [000] d.h5. 136.966993: preempt_count_sub <-sched_ttwu_pending
dd-828 [000] d.h4. 136.966994: update_rq_clock.part.0 <-sched_ttwu_pending
dd-828 [000] d.h4. 136.966996: ttwu_do_activate <-sched_ttwu_pending
dd-828 [000] d.h4. 136.966997: activate_task <-ttwu_do_activate
dd-828 [000] d.h4. 136.966998: enqueue_task_fair <-activate_task
dd-828 [000] d.h4. 136.967000: update_curr <-enqueue_task_fair
dd-828 [000] d.h4. 136.967001: update_min_vruntime <-update_curr
dd-828 [000] d.h4. 136.967003: __cgroup_account_cputime <-update_curr
BR
Carlos Song
> > On Thu, Mar 20, 2025 at 11:07:41AM +0000, Carlos Song wrote:
> > > Hi, all
> > >
> > > I found a 300ms~600ms IRQ off when writing 1Gb data to mmc device at
> > I.MX7d SDB board at Linux-kernel-v6.14.
> > > But I test the same case at Linux-kernel-v6.7, this longest IRQ off
> > > time is only
> > 1ms~2ms. So the issue is introduced from v6.7~v6.14.
> > >
> > > Run this cmd to test:
> > > dd if=/dev/zero of=/dev/mmcblk2p4 bs=4096 seek=12500 count=256000
> > > conv=fsync
> > >
> > > This issue looks from blkdev_buffered_write() function. Because when
> > > I run this cmd with "oflag=direct" to use blkdev_direct_write(), I
> > > can not see
> > any long time IRQ off.
> > >
> > > Then I use Ftrace irqoff tracer to trace the longest IRQ off event,
> > > I found some
> > differences between v6.7 and v6.14:
> > > In iomap_file_buffered_write(), __folio_alloc (in v6.7) is replaced
> > > by
> > _folio_alloc_noprof (in v6.14) by this patch.
> > > The spinlock disabled IRQ ~300ms+. It looks there are some fixes for
> > > this patch,
> > but I still can see IRQ off 300ms+ at 6.14.0-rc7-next-20250319.
> > >
> > > Do I trigger one bug? I know little about mem so I have to report it
> > > and hope I
> > can get some help or guide.
> > > I put my ftrace log at the mail tail to help trace and explain.
> >
> > Did you track down which spinlock?
> >
> > >
>
> Hi,
>
> Sorry for my late reply and thank you for your quick reply!
> From the trace log, I think the spinlock is from here like this:
>
> __alloc_frozen_pages_noprof ==> get_page_from_freelist==>spinclok
>
> Do you need other log not only this?
>
> > > => get_page_from_freelist"
> > > => __alloc_frozen_pages_noprof
> > > => __folio_alloc_noprof
> > > => __filemap_get_folio
>
>
> > > commit b951aaff503502a7fe066eeed2744ba8a6413c89
> > > Author: Suren Baghdasaryan
> > surenb@google.com<mailto:surenb@google.com>
> > > Date: Thu Mar 21 09:36:40 2024 -0700
> > >
> > > mm: enable page allocation tagging
> > >
> > > Redefine page allocators to record allocation tags upon their
> invocation.
> > > Instrument post_alloc_hook and free_pages_prepare to modify
> current
> > > allocation tag.
> > >
> > > [surenb@google.com: undo _noprof additions in the documentation]
> > > Link:
> > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flkml
> > .kern
> >
> el.org%2Fr%2F20240326231453.1206227-3-surenb%40google.com&data=05%
> >
> 7C02%7Ccarlos.song%40nxp.com%7Cdf027bbb97074fc2cde808dd67a1d6c5%7
> >
> C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C638780666983046738%
> >
> 7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDA
> >
> wMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C
> >
> &sdata=Aby13KyQjF5pcbYW%2BkfsEJLiaPmS2ZiJLUHJ%2BCr2JXM%3D&reserve
> > d=0
> > > Link:
> > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flkml
> > .kern
> >
> el.org%2Fr%2F20240321163705.3067592-19-surenb%40google.com&data=05
> > %7C02%7Ccarlos.song%40nxp.com%7Cdf027bbb97074fc2cde808dd67a1d6c5
> %
> >
> 7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C638780666983065790
> > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuM
> D
> >
> AwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7
> >
> C&sdata=n0hjs2fhjYs%2BcnbrxHy4vFK6D4GFVL4%2Fu72anOOLiEI%3D&reserve
> > d=0
> > > Signed-off-by: Suren Baghdasaryan
> > surenb@google.com<mailto:surenb@google.com>
> > > Co-developed-by: Kent Overstreet
> > kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev>
> > > Signed-off-by: Kent Overstreet
> > kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev>
> > > Reviewed-by: Kees Cook
> > keescook@chromium.org<mailto:keescook@chromium.org>
> > > Tested-by: Kees Cook
> > keescook@chromium.org<mailto:keescook@chromium.org>
> > > Cc: Alexander Viro
> > viro@zeniv.linux.org.uk<mailto:viro@zeniv.linux.org.uk>
> > > Cc: Alex Gaynor
> > alex.gaynor@gmail.com<mailto:alex.gaynor@gmail.com>
> > > Cc: Alice Ryhl aliceryhl@google.com<mailto:aliceryhl@google.com>
> > > Cc: Andreas Hindborg
> > a.hindborg@samsung.com<mailto:a.hindborg@samsung.com>
> > > Cc: Benno Lossin
> > benno.lossin@proton.me<mailto:benno.lossin@proton.me>
> > > Cc: "Björn Roy Baron"
> > bjorn3_gh@protonmail.com<mailto:bjorn3_gh@protonmail.com>
> > > Cc: Boqun Feng
> > boqun.feng@gmail.com<mailto:boqun.feng@gmail.com>
> > > Cc: Christoph Lameter cl@linux.com<mailto:cl@linux.com>
> > > Cc: Dennis Zhou dennis@kernel.org<mailto:dennis@kernel.org>
> > > Cc: Gary Guo gary@garyguo.net<mailto:gary@garyguo.net>
> > > Cc: Miguel Ojeda ojeda@kernel.org<mailto:ojeda@kernel.org>
> > > Cc: Pasha Tatashin
> > pasha.tatashin@soleen.com<mailto:pasha.tatashin@soleen.com>
> > > Cc: Peter Zijlstra
> peterz@infradead.org<mailto:peterz@infradead.org>
> > > Cc: Tejun Heo tj@kernel.org<mailto:tj@kernel.org>
> > > Cc: Vlastimil Babka vbabka@suse.cz<mailto:vbabka@suse.cz>
> > > Cc: Wedson Almeida Filho
> > wedsonaf@gmail.com<mailto:wedsonaf@gmail.com>
> > > Signed-off-by: Andrew Morton
> > > akpm@linux-foundation.org<mailto:akpm@linux-foundation.org>
> > >
> > >
> > > Ftrace irqoff tracer shows detail:
> > > At v6.14:
> > > # tracer: irqsoff
> > > #
> > > # irqsoff latency trace v1.1.5 on 6.14.0-rc7-next-20250319 #
> > > --------------------------------------------------------------------
> > > # latency: 279663 us, #21352/21352, CPU#0 | (M:NONE VP:0, KP:0, SP:0
> > > HP:0
> > #P:2)
> > > # -----------------
> > > # | task: dd-805 (uid:0 nice:0 policy:0 rt_prio:0)
> > > # -----------------
> > > # => started at: __rmqueue_pcplist
> > > # => ended at: _raw_spin_unlock_irqrestore
> > > #
> > > #
> > > # _------=> CPU#
> > > # / _-----=> irqs-off/BH-disabled
> > > # | / _----=> need-resched
> > > # || / _---=> hardirq/softirq
> > > # ||| / _--=> preempt-depth
> > > # |||| / _-=> migrate-disable
> > > # ||||| / delay
> > > # cmd pid |||||| time | caller
> > > # \ / |||||| \ | /
> > > dd-805 0d.... 1us : __rmqueue_pcplist
> > > dd-805 0d.... 3us : _raw_spin_trylock
> > <-__rmqueue_pcplist
> > > dd-805 0d.... 7us : __mod_zone_page_state
> > <-__rmqueue_pcplist
> > > dd-805 0d.... 10us : __mod_zone_page_state
> > <-__rmqueue_pcplist
> > > dd-805 0d.... 12us : __mod_zone_page_state
> > <-__rmqueue_pcplist
> > > dd-805 0d.... 15us : __mod_zone_page_state
> > <-__rmqueue_pcplist
> > > dd-805 0d.... 17us : __mod_zone_page_state
> > <-__rmqueue_pcplist
> > > dd-805 0d.... 19us : __mod_zone_page_state
> > <-__rmqueue_pcplist
> > > ...
> > > dd-805 0d.... 1535us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-805 0d.... 1538us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-805 0d.... 1539us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-805 0d.... 1542us+: try_to_claim_block
> > <-__rmqueue_pcplist
> > > dd-805 0d.... 1597us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-805 0d.... 1599us+: try_to_claim_block
> > <-__rmqueue_pcplist
> > > dd-805 0d.... 1674us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-805 0d.... 1676us+: try_to_claim_block
> > <-__rmqueue_pcplist
> > > dd-805 0d.... 1716us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-805 0d.... 1718us+: try_to_claim_block
> > <-__rmqueue_pcplist
> > > dd-805 0d.... 1801us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-805 0d.... 1803us+: try_to_claim_block
> > <-__rmqueue_pcplist
> > > ...
> > > dd-805 0d.... 279555us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-805 0d.... 279556us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-805 0d.... 279558us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-805 0d.... 279560us+: try_to_claim_block
> > <-__rmqueue_pcplist
> > > dd-805 0d.... 279616us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-805 0d.... 279618us : __mod_zone_page_state
> > <-__rmqueue_pcplist
> > > dd-805 0d.... 279620us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > ...
> > > dd-805 0d.... 279658us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-805 0d.... 279660us : _raw_spin_unlock_irqrestore
> > <-__rmqueue_pcplist
> > > dd-805 0d.... 279662us : _raw_spin_unlock_irqrestore
> > > dd-805 0d.... 279666us+: trace_hardirqs_on
> > <-_raw_spin_unlock_irqrestore
> > > dd-805 0d.... 279712us : <stack trace>
> > > => get_page_from_freelist
> > > => __alloc_frozen_pages_noprof
> > > => __folio_alloc_noprof
> > > => __filemap_get_folio
> > > => iomap_write_begin
> > > => iomap_file_buffered_write
> > > => blkdev_write_iter
> > > => vfs_write
> > > => ksys_write
> > > => ret_fast_syscall
> > >
> > > At v6.7:
> > > # tracer: irqsoff
> > > #
> > > # irqsoff latency trace v1.1.5 on 6.7.0 #
> > > --------------------------------------------------------------------
> > > # latency: 2477 us, #146/146, CPU#0 | (M:server VP:0, KP:0, SP:0 HP:0 #P:2)
> > > # -----------------
> > > # | task: dd-808 (uid:0 nice:0 policy:0 rt_prio:0)
> > > # -----------------
> > > # => started at: _raw_spin_lock_irqsave
> > > # => ended at: _raw_spin_unlock_irqrestore
> > > #
> > > #
> > > # _------=> CPU#
> > > # / _-----=> irqs-off/BH-disabled
> > > # | / _----=> need-resched
> > > # || / _---=> hardirq/softirq
> > > # ||| / _--=> preempt-depth
> > > # |||| / _-=> migrate-disable
> > > # ||||| / delay
> > > # cmd pid |||||| time | caller
> > > # \ / |||||| \ | /
> > > dd-808 0d.... 1us!: _raw_spin_lock_irqsave
> > > dd-808 0d.... 186us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 189us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 191us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 192us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 194us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 196us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 199us : steal_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 203us!: move_freepages_block
> > <-steal_suitable_fallback
> > > dd-808 0d.... 330us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 332us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 334us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 336us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 338us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 339us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 341us : steal_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 343us!: move_freepages_block
> > <-steal_suitable_fallback
> > > dd-808 0d.... 479us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 481us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 483us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 485us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 486us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 488us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 490us : steal_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 492us!: move_freepages_block
> > <-steal_suitable_fallback
> > > dd-808 0d.... 630us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 632us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 634us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 636us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 638us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 640us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 642us : steal_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 644us!: move_freepages_block
> > <-steal_suitable_fallback
> > > dd-808 0d.... 771us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 773us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 775us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 777us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 778us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 780us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 782us : steal_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 784us!: move_freepages_block
> > <-steal_suitable_fallback
> > > dd-808 0d.... 911us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 913us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 915us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 916us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 918us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 920us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 922us : steal_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 924us!: move_freepages_block
> > <-steal_suitable_fallback
> > > dd-808 0d.... 1055us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1058us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1059us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1061us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1063us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1065us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1066us : steal_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1068us!: move_freepages_block
> > <-steal_suitable_fallback
> > > dd-808 0d.... 1194us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1196us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1198us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1200us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1202us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1203us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1205us : steal_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1208us!: move_freepages_block
> > <-steal_suitable_fallback
> > > dd-808 0d.... 1333us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1335us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1337us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1339us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1341us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1342us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1344us : steal_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1346us!: move_freepages_block
> > <-steal_suitable_fallback
> > > dd-808 0d.... 1480us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1482us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1484us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1486us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1488us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1490us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1492us : steal_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1494us!: move_freepages_block
> > <-steal_suitable_fallback
> > > dd-808 0d.... 1621us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1623us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1625us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1627us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1629us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1630us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1632us : steal_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1634us!: move_freepages_block
> > <-steal_suitable_fallback
> > > dd-808 0d.... 1761us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1763us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1765us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1766us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1768us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1770us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1772us : steal_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1774us!: move_freepages_block
> > <-steal_suitable_fallback
> > > dd-808 0d.... 1900us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1902us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1903us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1905us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1907us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1909us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1911us : steal_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 1913us!: move_freepages_block
> > <-steal_suitable_fallback
> > > dd-808 0d.... 2038us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2040us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2042us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2044us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2046us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2047us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2049us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2051us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2053us : steal_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2055us!: move_freepages_block
> > <-steal_suitable_fallback
> > > dd-808 0d.... 2175us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2176us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2178us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2180us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2182us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2183us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2185us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2187us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2189us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2191us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2192us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2194us : steal_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2196us!: move_freepages_block
> > <-steal_suitable_fallback
> > > dd-808 0d.... 2323us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2325us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2327us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2328us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2330us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2332us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2334us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2335us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2337us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2339us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2341us : find_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2343us : steal_suitable_fallback
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2345us!: move_freepages_block
> > <-steal_suitable_fallback
> > > dd-808 0d.... 2470us : __mod_zone_page_state
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2473us : _raw_spin_unlock_irqrestore
> > <-__rmqueue_pcplist
> > > dd-808 0d.... 2476us : _raw_spin_unlock_irqrestore
> > > dd-808 0d.... 2479us+: tracer_hardirqs_on
> > <-_raw_spin_unlock_irqrestore
> > > dd-808 0d.... 2520us : <stack trace>
> > > => get_page_from_freelist
> > > => __alloc_pages
> > > => __folio_alloc
> > > => __filemap_get_folio
> > > => iomap_write_begin
> > > => iomap_file_buffered_write
> > > => blkdev_write_iter
> > > => vfs_write
> > > => ksys_write
> > > => ret_fast_syscall
> > >
> > > Best Regard
> > > Carlos Song
> > >
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Ask help about this patch b951aaff5035 " mm: enable page allocation tagging"
2025-03-27 11:02 ` Carlos Song
@ 2025-03-31 20:01 ` Suren Baghdasaryan
2025-04-02 2:30 ` Carlos Song
0 siblings, 1 reply; 8+ messages in thread
From: Suren Baghdasaryan @ 2025-03-31 20:01 UTC (permalink / raw)
To: Carlos Song
Cc: Kent Overstreet, akpm, willy, linux-mm, linux-kernel, linux-fsdevel
On Thu, Mar 27, 2025 at 4:02 AM Carlos Song <carlos.song@nxp.com> wrote:
>
> Hi,
>
> I get the function calling step log, I hope it can help debug little.
Hi Carlos,
Since you do not enable CONFIG_MEM_ALLOC_PROFILING, "mm: enable page
allocation tagging" patch would most definitely not cause this issue.
From the traces you posted I think the spinlock involved here is
zone->lock inside rmqueue_bulk():
https://elixir.bootlin.com/linux/v6.14-rc6/source/mm/page_alloc.c#L2310
I quickly scanned the code within that loop comparing 6.7 and 6.14 and
while I did not find anything obvious, steal_suitable_fallback()
function seems quite different. I would suggest profiling this path to
see which part might have regressed and go from there.
BTW, are you using pure upstream code or it has some out-of-tree modifications?
Thanks,
Suren.
>
> The spinlock_irqsave from 136.725021 to 136.966962_raw_spin_unlock_irqrestore.
> Around 260ms? It is Just one case, sometimes it will spend more time.
>
> dd-828 [000] ..... 136.724980: fdget_pos <-ksys_write
> dd-828 [000] ..... 136.724982: vfs_write <-ksys_write
> dd-828 [000] ..... 136.724983: blkdev_write_iter <-vfs_write
> dd-828 [000] ..... 136.724984: I_BDEV <-blkdev_write_iter
> dd-828 [000] ..... 136.724985: file_update_time <-blkdev_write_iter
> dd-828 [000] ..... 136.724986: inode_needs_update_time <-file_update_time
> dd-828 [000] ..... 136.724987: ktime_get_coarse_real_ts64 <-inode_needs_update_time
> dd-828 [000] ..... 136.724989: timestamp_truncate <-inode_needs_update_time
> dd-828 [000] ..... 136.724990: iomap_file_buffered_write <-blkdev_write_iter
> dd-828 [000] ..... 136.724992: iomap_iter <-iomap_file_buffered_write
> dd-828 [000] ..... 136.724993: blkdev_iomap_begin <-iomap_iter
> dd-828 [000] ..... 136.724994: I_BDEV <-blkdev_iomap_begin
> dd-828 [000] ..... 136.724995: balance_dirty_pages_ratelimited_flags <-iomap_file_buffered_write
> dd-828 [000] ..... 136.724996: inode_to_bdi <-balance_dirty_pages_ratelimited_flags
> dd-828 [000] ..... 136.724997: I_BDEV <-inode_to_bdi
> dd-828 [000] ..... 136.724998: preempt_count_add <-balance_dirty_pages_ratelimited_flags
> dd-828 [000] ...1. 136.725000: preempt_count_sub <-balance_dirty_pages_ratelimited_flags
> dd-828 [000] ..... 136.725001: fault_in_readable <-fault_in_iov_iter_readable
> dd-828 [000] ..... 136.725003: iomap_write_begin <-iomap_file_buffered_write
> dd-828 [000] ..... 136.725004: iomap_get_folio <-iomap_write_begin
> dd-828 [000] ..... 136.725005: __filemap_get_folio <-iomap_write_begin
> dd-828 [000] ..... 136.725006: filemap_get_entry <-__filemap_get_folio
> dd-828 [000] ..... 136.725007: __rcu_read_lock <-filemap_get_entry
> dd-828 [000] ..... 136.725008: __rcu_read_unlock <-filemap_get_entry
> dd-828 [000] ..... 136.725009: inode_to_bdi <-__filemap_get_folio
> dd-828 [000] ..... 136.725010: I_BDEV <-inode_to_bdi
> dd-828 [000] ..... 136.725012: __folio_alloc_noprof <-__filemap_get_folio
> dd-828 [000] ..... 136.725013: __alloc_pages_noprof <-__folio_alloc_noprof
> dd-828 [000] ..... 136.725014: get_page_from_freelist <-__alloc_pages_noprof
> dd-828 [000] ..... 136.725015: node_dirty_ok <-get_page_from_freelist
> dd-828 [000] ..... 136.725016: preempt_count_add <-get_page_from_freelist
> dd-828 [000] ...1. 136.725018: _raw_spin_trylock <-get_page_from_freelist
> dd-828 [000] ...1. 136.725019: preempt_count_add <-_raw_spin_trylock
> dd-828 [000] ...2. 136.725021: _raw_spin_lock_irqsave <-__rmqueue_pcplist
> dd-828 [000] d..2. 136.725022: preempt_count_add <-_raw_spin_lock_irqsave
> dd-828 [000] d..3. 136.725025: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725028: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725029: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725031: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725032: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725034: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725036: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725037: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725038: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725039: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725040: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725042: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725043: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725044: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725046: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725047: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725048: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725050: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725051: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725052: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725053: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725055: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725056: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725057: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725058: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725060: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725061: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725062: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725063: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725065: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725066: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725067: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725068: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725070: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725071: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725072: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725073: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725075: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725076: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725077: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725079: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725080: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725081: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725082: __mod_zone_page_state <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.725083: __mod_zone_page_state <-__rmqueue_pcplist
> ...
> dd-828 [000] d..3. 136.820279: find_suitable_fallback <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.820280: find_suitable_fallback <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.820281: find_suitable_fallback <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.820282: find_suitable_fallback <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.820283: find_suitable_fallback <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.820284: find_suitable_fallback <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.820285: find_suitable_fallback <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.820286: find_suitable_fallback <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.820287: find_suitable_fallback <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.820289: find_suitable_fallback <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.820290: steal_suitable_fallback <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.820525: __mod_zone_page_state <-steal_suitable_fallback
> ...
> dd-828 [000] d..3. 136.966945: __mod_zone_page_state <-steal_suitable_fallback
> dd-828 [000] d..3. 136.966946: find_suitable_fallback <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.966947: find_suitable_fallback <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.966948: find_suitable_fallback <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.966950: find_suitable_fallback <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.966951: find_suitable_fallback <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.966952: find_suitable_fallback <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.966953: find_suitable_fallback <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.966954: find_suitable_fallback <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.966955: find_suitable_fallback <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.966956: find_suitable_fallback <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.966957: find_suitable_fallback <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.966958: find_suitable_fallback <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.966959: find_suitable_fallback <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.966961: find_suitable_fallback <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.966962: _raw_spin_unlock_irqrestore <-__rmqueue_pcplist
> dd-828 [000] d..3. 136.966965: irq_enter <-generic_handle_arch_irq
> dd-828 [000] d..3. 136.966966: irq_enter_rcu <-generic_handle_arch_irq
> dd-828 [000] d..3. 136.966967: preempt_count_add <-irq_enter_rcu
> dd-828 [000] d.h3. 136.966969: gic_handle_irq <-generic_handle_arch_irq
> dd-828 [000] d.h3. 136.966970: generic_handle_domain_irq <-gic_handle_irq
> dd-828 [000] d.h3. 136.966971: __irq_resolve_mapping <-generic_handle_domain_irq
> dd-828 [000] d.h3. 136.966972: __rcu_read_lock <-__irq_resolve_mapping
> dd-828 [000] d.h3. 136.966974: __rcu_read_unlock <-__irq_resolve_mapping
> dd-828 [000] d.h3. 136.966975: handle_irq_desc <-gic_handle_irq
> dd-828 [000] d.h3. 136.966976: handle_percpu_devid_irq <-handle_irq_desc
> dd-828 [000] d.h3. 136.966978: ipi_handler <-handle_percpu_devid_irq
> dd-828 [000] d.h3. 136.966979: do_handle_IPI <-ipi_handler
> dd-828 [000] d.h3. 136.966981: generic_smp_call_function_single_interrupt <-do_handle_IPI
> dd-828 [000] d.h3. 136.966982: __flush_smp_call_function_queue <-do_handle_IPI
> dd-828 [000] d.h3. 136.966986: sched_ttwu_pending <-__flush_smp_call_function_queue
> dd-828 [000] d.h3. 136.966987: preempt_count_add <-sched_ttwu_pending
> dd-828 [000] d.h4. 136.966990: _raw_spin_lock <-sched_ttwu_pending
> dd-828 [000] d.h4. 136.966991: preempt_count_add <-_raw_spin_lock
> dd-828 [000] d.h5. 136.966993: preempt_count_sub <-sched_ttwu_pending
> dd-828 [000] d.h4. 136.966994: update_rq_clock.part.0 <-sched_ttwu_pending
> dd-828 [000] d.h4. 136.966996: ttwu_do_activate <-sched_ttwu_pending
> dd-828 [000] d.h4. 136.966997: activate_task <-ttwu_do_activate
> dd-828 [000] d.h4. 136.966998: enqueue_task_fair <-activate_task
> dd-828 [000] d.h4. 136.967000: update_curr <-enqueue_task_fair
> dd-828 [000] d.h4. 136.967001: update_min_vruntime <-update_curr
> dd-828 [000] d.h4. 136.967003: __cgroup_account_cputime <-update_curr
>
> BR
> Carlos Song
>
> > > On Thu, Mar 20, 2025 at 11:07:41AM +0000, Carlos Song wrote:
> > > > Hi, all
> > > >
> > > > I found a 300ms~600ms IRQ off when writing 1Gb data to mmc device at
> > > I.MX7d SDB board at Linux-kernel-v6.14.
> > > > But I test the same case at Linux-kernel-v6.7, this longest IRQ off
> > > > time is only
> > > 1ms~2ms. So the issue is introduced from v6.7~v6.14.
> > > >
> > > > Run this cmd to test:
> > > > dd if=/dev/zero of=/dev/mmcblk2p4 bs=4096 seek=12500 count=256000
> > > > conv=fsync
> > > >
> > > > This issue looks from blkdev_buffered_write() function. Because when
> > > > I run this cmd with "oflag=direct" to use blkdev_direct_write(), I
> > > > can not see
> > > any long time IRQ off.
> > > >
> > > > Then I use Ftrace irqoff tracer to trace the longest IRQ off event,
> > > > I found some
> > > differences between v6.7 and v6.14:
> > > > In iomap_file_buffered_write(), __folio_alloc (in v6.7) is replaced
> > > > by
> > > _folio_alloc_noprof (in v6.14) by this patch.
> > > > The spinlock disabled IRQ ~300ms+. It looks there are some fixes for
> > > > this patch,
> > > but I still can see IRQ off 300ms+ at 6.14.0-rc7-next-20250319.
> > > >
> > > > Do I trigger one bug? I know little about mem so I have to report it
> > > > and hope I
> > > can get some help or guide.
> > > > I put my ftrace log at the mail tail to help trace and explain.
> > >
> > > Did you track down which spinlock?
> > >
> > > >
> >
> > Hi,
> >
> > Sorry for my late reply and thank you for your quick reply!
> > From the trace log, I think the spinlock is from here like this:
> >
> > __alloc_frozen_pages_noprof ==> get_page_from_freelist==>spinclok
> >
> > Do you need other log not only this?
> >
> > > > => get_page_from_freelist"
> > > > => __alloc_frozen_pages_noprof
> > > > => __folio_alloc_noprof
> > > > => __filemap_get_folio
> >
> >
> > > > commit b951aaff503502a7fe066eeed2744ba8a6413c89
> > > > Author: Suren Baghdasaryan
> > > surenb@google.com<mailto:surenb@google.com>
> > > > Date: Thu Mar 21 09:36:40 2024 -0700
> > > >
> > > > mm: enable page allocation tagging
> > > >
> > > > Redefine page allocators to record allocation tags upon their
> > invocation.
> > > > Instrument post_alloc_hook and free_pages_prepare to modify
> > current
> > > > allocation tag.
> > > >
> > > > [surenb@google.com: undo _noprof additions in the documentation]
> > > > Link:
> > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flkml
> > > .kern
> > >
> > el.org%2Fr%2F20240326231453.1206227-3-surenb%40google.com&data=05%
> > >
> > 7C02%7Ccarlos.song%40nxp.com%7Cdf027bbb97074fc2cde808dd67a1d6c5%7
> > >
> > C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C638780666983046738%
> > >
> > 7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDA
> > >
> > wMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C
> > >
> > &sdata=Aby13KyQjF5pcbYW%2BkfsEJLiaPmS2ZiJLUHJ%2BCr2JXM%3D&reserve
> > > d=0
> > > > Link:
> > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flkml
> > > .kern
> > >
> > el.org%2Fr%2F20240321163705.3067592-19-surenb%40google.com&data=05
> > > %7C02%7Ccarlos.song%40nxp.com%7Cdf027bbb97074fc2cde808dd67a1d6c5
> > %
> > >
> > 7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C638780666983065790
> > > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuM
> > D
> > >
> > AwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7
> > >
> > C&sdata=n0hjs2fhjYs%2BcnbrxHy4vFK6D4GFVL4%2Fu72anOOLiEI%3D&reserve
> > > d=0
> > > > Signed-off-by: Suren Baghdasaryan
> > > surenb@google.com<mailto:surenb@google.com>
> > > > Co-developed-by: Kent Overstreet
> > > kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev>
> > > > Signed-off-by: Kent Overstreet
> > > kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev>
> > > > Reviewed-by: Kees Cook
> > > keescook@chromium.org<mailto:keescook@chromium.org>
> > > > Tested-by: Kees Cook
> > > keescook@chromium.org<mailto:keescook@chromium.org>
> > > > Cc: Alexander Viro
> > > viro@zeniv.linux.org.uk<mailto:viro@zeniv.linux.org.uk>
> > > > Cc: Alex Gaynor
> > > alex.gaynor@gmail.com<mailto:alex.gaynor@gmail.com>
> > > > Cc: Alice Ryhl aliceryhl@google.com<mailto:aliceryhl@google.com>
> > > > Cc: Andreas Hindborg
> > > a.hindborg@samsung.com<mailto:a.hindborg@samsung.com>
> > > > Cc: Benno Lossin
> > > benno.lossin@proton.me<mailto:benno.lossin@proton.me>
> > > > Cc: "Björn Roy Baron"
> > > bjorn3_gh@protonmail.com<mailto:bjorn3_gh@protonmail.com>
> > > > Cc: Boqun Feng
> > > boqun.feng@gmail.com<mailto:boqun.feng@gmail.com>
> > > > Cc: Christoph Lameter cl@linux.com<mailto:cl@linux.com>
> > > > Cc: Dennis Zhou dennis@kernel.org<mailto:dennis@kernel.org>
> > > > Cc: Gary Guo gary@garyguo.net<mailto:gary@garyguo.net>
> > > > Cc: Miguel Ojeda ojeda@kernel.org<mailto:ojeda@kernel.org>
> > > > Cc: Pasha Tatashin
> > > pasha.tatashin@soleen.com<mailto:pasha.tatashin@soleen.com>
> > > > Cc: Peter Zijlstra
> > peterz@infradead.org<mailto:peterz@infradead.org>
> > > > Cc: Tejun Heo tj@kernel.org<mailto:tj@kernel.org>
> > > > Cc: Vlastimil Babka vbabka@suse.cz<mailto:vbabka@suse.cz>
> > > > Cc: Wedson Almeida Filho
> > > wedsonaf@gmail.com<mailto:wedsonaf@gmail.com>
> > > > Signed-off-by: Andrew Morton
> > > > akpm@linux-foundation.org<mailto:akpm@linux-foundation.org>
> > > >
> > > >
> > > > Ftrace irqoff tracer shows detail:
> > > > At v6.14:
> > > > # tracer: irqsoff
> > > > #
> > > > # irqsoff latency trace v1.1.5 on 6.14.0-rc7-next-20250319 #
> > > > --------------------------------------------------------------------
> > > > # latency: 279663 us, #21352/21352, CPU#0 | (M:NONE VP:0, KP:0, SP:0
> > > > HP:0
> > > #P:2)
> > > > # -----------------
> > > > # | task: dd-805 (uid:0 nice:0 policy:0 rt_prio:0)
> > > > # -----------------
> > > > # => started at: __rmqueue_pcplist
> > > > # => ended at: _raw_spin_unlock_irqrestore
> > > > #
> > > > #
> > > > # _------=> CPU#
> > > > # / _-----=> irqs-off/BH-disabled
> > > > # | / _----=> need-resched
> > > > # || / _---=> hardirq/softirq
> > > > # ||| / _--=> preempt-depth
> > > > # |||| / _-=> migrate-disable
> > > > # ||||| / delay
> > > > # cmd pid |||||| time | caller
> > > > # \ / |||||| \ | /
> > > > dd-805 0d.... 1us : __rmqueue_pcplist
> > > > dd-805 0d.... 3us : _raw_spin_trylock
> > > <-__rmqueue_pcplist
> > > > dd-805 0d.... 7us : __mod_zone_page_state
> > > <-__rmqueue_pcplist
> > > > dd-805 0d.... 10us : __mod_zone_page_state
> > > <-__rmqueue_pcplist
> > > > dd-805 0d.... 12us : __mod_zone_page_state
> > > <-__rmqueue_pcplist
> > > > dd-805 0d.... 15us : __mod_zone_page_state
> > > <-__rmqueue_pcplist
> > > > dd-805 0d.... 17us : __mod_zone_page_state
> > > <-__rmqueue_pcplist
> > > > dd-805 0d.... 19us : __mod_zone_page_state
> > > <-__rmqueue_pcplist
> > > > ...
> > > > dd-805 0d.... 1535us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-805 0d.... 1538us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-805 0d.... 1539us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-805 0d.... 1542us+: try_to_claim_block
> > > <-__rmqueue_pcplist
> > > > dd-805 0d.... 1597us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-805 0d.... 1599us+: try_to_claim_block
> > > <-__rmqueue_pcplist
> > > > dd-805 0d.... 1674us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-805 0d.... 1676us+: try_to_claim_block
> > > <-__rmqueue_pcplist
> > > > dd-805 0d.... 1716us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-805 0d.... 1718us+: try_to_claim_block
> > > <-__rmqueue_pcplist
> > > > dd-805 0d.... 1801us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-805 0d.... 1803us+: try_to_claim_block
> > > <-__rmqueue_pcplist
> > > > ...
> > > > dd-805 0d.... 279555us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-805 0d.... 279556us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-805 0d.... 279558us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-805 0d.... 279560us+: try_to_claim_block
> > > <-__rmqueue_pcplist
> > > > dd-805 0d.... 279616us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-805 0d.... 279618us : __mod_zone_page_state
> > > <-__rmqueue_pcplist
> > > > dd-805 0d.... 279620us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > ...
> > > > dd-805 0d.... 279658us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-805 0d.... 279660us : _raw_spin_unlock_irqrestore
> > > <-__rmqueue_pcplist
> > > > dd-805 0d.... 279662us : _raw_spin_unlock_irqrestore
> > > > dd-805 0d.... 279666us+: trace_hardirqs_on
> > > <-_raw_spin_unlock_irqrestore
> > > > dd-805 0d.... 279712us : <stack trace>
> > > > => get_page_from_freelist
> > > > => __alloc_frozen_pages_noprof
> > > > => __folio_alloc_noprof
> > > > => __filemap_get_folio
> > > > => iomap_write_begin
> > > > => iomap_file_buffered_write
> > > > => blkdev_write_iter
> > > > => vfs_write
> > > > => ksys_write
> > > > => ret_fast_syscall
> > > >
> > > > At v6.7:
> > > > # tracer: irqsoff
> > > > #
> > > > # irqsoff latency trace v1.1.5 on 6.7.0 #
> > > > --------------------------------------------------------------------
> > > > # latency: 2477 us, #146/146, CPU#0 | (M:server VP:0, KP:0, SP:0 HP:0 #P:2)
> > > > # -----------------
> > > > # | task: dd-808 (uid:0 nice:0 policy:0 rt_prio:0)
> > > > # -----------------
> > > > # => started at: _raw_spin_lock_irqsave
> > > > # => ended at: _raw_spin_unlock_irqrestore
> > > > #
> > > > #
> > > > # _------=> CPU#
> > > > # / _-----=> irqs-off/BH-disabled
> > > > # | / _----=> need-resched
> > > > # || / _---=> hardirq/softirq
> > > > # ||| / _--=> preempt-depth
> > > > # |||| / _-=> migrate-disable
> > > > # ||||| / delay
> > > > # cmd pid |||||| time | caller
> > > > # \ / |||||| \ | /
> > > > dd-808 0d.... 1us!: _raw_spin_lock_irqsave
> > > > dd-808 0d.... 186us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 189us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 191us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 192us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 194us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 196us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 199us : steal_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 203us!: move_freepages_block
> > > <-steal_suitable_fallback
> > > > dd-808 0d.... 330us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 332us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 334us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 336us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 338us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 339us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 341us : steal_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 343us!: move_freepages_block
> > > <-steal_suitable_fallback
> > > > dd-808 0d.... 479us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 481us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 483us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 485us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 486us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 488us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 490us : steal_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 492us!: move_freepages_block
> > > <-steal_suitable_fallback
> > > > dd-808 0d.... 630us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 632us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 634us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 636us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 638us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 640us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 642us : steal_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 644us!: move_freepages_block
> > > <-steal_suitable_fallback
> > > > dd-808 0d.... 771us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 773us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 775us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 777us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 778us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 780us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 782us : steal_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 784us!: move_freepages_block
> > > <-steal_suitable_fallback
> > > > dd-808 0d.... 911us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 913us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 915us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 916us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 918us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 920us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 922us : steal_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 924us!: move_freepages_block
> > > <-steal_suitable_fallback
> > > > dd-808 0d.... 1055us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1058us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1059us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1061us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1063us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1065us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1066us : steal_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1068us!: move_freepages_block
> > > <-steal_suitable_fallback
> > > > dd-808 0d.... 1194us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1196us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1198us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1200us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1202us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1203us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1205us : steal_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1208us!: move_freepages_block
> > > <-steal_suitable_fallback
> > > > dd-808 0d.... 1333us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1335us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1337us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1339us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1341us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1342us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1344us : steal_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1346us!: move_freepages_block
> > > <-steal_suitable_fallback
> > > > dd-808 0d.... 1480us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1482us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1484us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1486us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1488us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1490us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1492us : steal_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1494us!: move_freepages_block
> > > <-steal_suitable_fallback
> > > > dd-808 0d.... 1621us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1623us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1625us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1627us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1629us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1630us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1632us : steal_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1634us!: move_freepages_block
> > > <-steal_suitable_fallback
> > > > dd-808 0d.... 1761us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1763us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1765us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1766us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1768us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1770us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1772us : steal_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1774us!: move_freepages_block
> > > <-steal_suitable_fallback
> > > > dd-808 0d.... 1900us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1902us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1903us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1905us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1907us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1909us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1911us : steal_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 1913us!: move_freepages_block
> > > <-steal_suitable_fallback
> > > > dd-808 0d.... 2038us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2040us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2042us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2044us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2046us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2047us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2049us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2051us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2053us : steal_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2055us!: move_freepages_block
> > > <-steal_suitable_fallback
> > > > dd-808 0d.... 2175us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2176us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2178us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2180us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2182us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2183us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2185us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2187us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2189us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2191us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2192us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2194us : steal_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2196us!: move_freepages_block
> > > <-steal_suitable_fallback
> > > > dd-808 0d.... 2323us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2325us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2327us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2328us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2330us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2332us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2334us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2335us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2337us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2339us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2341us : find_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2343us : steal_suitable_fallback
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2345us!: move_freepages_block
> > > <-steal_suitable_fallback
> > > > dd-808 0d.... 2470us : __mod_zone_page_state
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2473us : _raw_spin_unlock_irqrestore
> > > <-__rmqueue_pcplist
> > > > dd-808 0d.... 2476us : _raw_spin_unlock_irqrestore
> > > > dd-808 0d.... 2479us+: tracer_hardirqs_on
> > > <-_raw_spin_unlock_irqrestore
> > > > dd-808 0d.... 2520us : <stack trace>
> > > > => get_page_from_freelist
> > > > => __alloc_pages
> > > > => __folio_alloc
> > > > => __filemap_get_folio
> > > > => iomap_write_begin
> > > > => iomap_file_buffered_write
> > > > => blkdev_write_iter
> > > > => vfs_write
> > > > => ksys_write
> > > > => ret_fast_syscall
> > > >
> > > > Best Regard
> > > > Carlos Song
> > > >
^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Ask help about this patch b951aaff5035 " mm: enable page allocation tagging"
2025-03-31 20:01 ` Suren Baghdasaryan
@ 2025-04-02 2:30 ` Carlos Song
0 siblings, 0 replies; 8+ messages in thread
From: Carlos Song @ 2025-04-02 2:30 UTC (permalink / raw)
To: Suren Baghdasaryan
Cc: Kent Overstreet, akpm, willy, linux-mm, linux-kernel, linux-fsdevel
> -----Original Message-----
> From: Suren Baghdasaryan <surenb@google.com>
> Sent: Tuesday, April 1, 2025 4:02 AM
> To: Carlos Song <carlos.song@nxp.com>
> Cc: Kent Overstreet <kent.overstreet@linux.dev>; akpm@linux-foundation.org;
> willy@infradead.org; linux-mm@kvack.org; linux-kernel@vger.kernel.org;
> linux-fsdevel@vger.kernel.org
> Subject: [EXT] Re: Ask help about this patch b951aaff5035 " mm: enable page
> allocation tagging"
>
> Caution: This is an external email. Please take care when clicking links or
> opening attachments. When in doubt, report the message using the 'Report
> this email' button
>
>
> On Thu, Mar 27, 2025 at 4:02 AM Carlos Song <carlos.song@nxp.com> wrote:
> >
> > Hi,
> >
> > I get the function calling step log, I hope it can help debug little.
>
> Hi Carlos,
> Since you do not enable CONFIG_MEM_ALLOC_PROFILING, "mm: enable page
> allocation tagging" patch would most definitely not cause this issue.
>
> From the traces you posted I think the spinlock involved here is
> zone->lock inside rmqueue_bulk():
> https://elixir.bo/
> otlin.com%2Flinux%2Fv6.14-rc6%2Fsource%2Fmm%2Fpage_alloc.c%23L2310
> &data=05%7C02%7Ccarlos.song%40nxp.com%7Cb83b5b83b6f94339e2f108d
> d708edead%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C638790
> 481110586829%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydW
> UsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3
> D%7C0%7C%7C%7C&sdata=gXVB6wbfdbT6kH1QBc7tcfyeUnGpChGDc5IpVTcuz
> MM%3D&reserved=0
> I quickly scanned the code within that loop comparing 6.7 and 6.14 and while I
> did not find anything obvious, steal_suitable_fallback() function seems quite
> different. I would suggest profiling this path to see which part might have
> regressed and go from there.
> BTW, are you using pure upstream code or it has some out-of-tree
> modifications?
> Thanks,
> Suren.
>
>
Hi, Suren
Thank you for your help, I found the regression is really from steal_suitable_fallback() function.
When add this patch, the spinlock time will become 100ms from 1ms, I will write a new
email to discuss this.
commit c0cd6f557b9090525d288806cccbc73440ac235a
Author: Johannes Weiner <hannes@cmpxchg.org>
Date: Wed Mar 20 14:02:11 2024 -0400
mm: page_alloc: fix freelist movement during block conversion
Currently, page block type conversion during fallbacks, atomic
reservations and isolation can strand various amounts of free pages on
incorrect freelists.
For example, fallback stealing moves free pages in the block to the new
type's freelists, but then may not actually claim the block for that type
if there aren't enough compatible pages already allocated.
In all cases, free page moving might fail if the block straddles more than
one zone, in which case no free pages are moved at all, but the block type
is changed anyway.
This is detrimental to type hygiene on the freelists. It encourages
incompatible page mixing down the line (ask for one type, get another) and
thus contributes to long-term fragmentation.
Split the process into a proper transaction: check first if conversion
will happen, then try to move the free pages, and only if that was
successful convert the block to the new type.
[baolin.wang@linux.alibaba.com: fix allocation failures with CONFIG_CMA]
Link: https://lkml.kernel.org/r/a97697e0-45b0-4f71-b087-fdc7a1d43c0e@linux.alibaba.com
Link: https://lkml.kernel.org/r/20240320180429.678181-7-hannes@cmpxchg.org
Signed-off-by: Johannes Weiner <hannes@cmpxchg.org>
Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Tested-by: "Huang, Ying" <ying.huang@intel.com>
Reviewed-by: Vlastimil Babka <vbabka@suse.cz>
Tested-by: Baolin Wang <baolin.wang@linux.alibaba.com>
Cc: David Hildenbrand <david@redhat.com>
Cc: Mel Gorman <mgorman@techsingularity.net>
Cc: Zi Yan <ziy@nvidia.com>
Signed-off-by: Andrew Morton <akpm@linux-foundation.org>
> >
> > The spinlock_irqsave from 136.725021 to
> 136.966962_raw_spin_unlock_irqrestore.
> > Around 260ms? It is Just one case, sometimes it will spend more time.
> >
> > dd-828 [000] ..... 136.724980:
> fdget_pos <-ksys_write
> > dd-828 [000] ..... 136.724982: vfs_write
> <-ksys_write
> > dd-828 [000] ..... 136.724983: blkdev_write_iter
> <-vfs_write
> > dd-828 [000] ..... 136.724984: I_BDEV
> <-blkdev_write_iter
> > dd-828 [000] ..... 136.724985: file_update_time
> <-blkdev_write_iter
> > dd-828 [000] ..... 136.724986:
> inode_needs_update_time <-file_update_time
> > dd-828 [000] ..... 136.724987:
> ktime_get_coarse_real_ts64 <-inode_needs_update_time
> > dd-828 [000] ..... 136.724989: timestamp_truncate
> <-inode_needs_update_time
> > dd-828 [000] ..... 136.724990:
> iomap_file_buffered_write <-blkdev_write_iter
> > dd-828 [000] ..... 136.724992: iomap_iter
> <-iomap_file_buffered_write
> > dd-828 [000] ..... 136.724993: blkdev_iomap_begin
> <-iomap_iter
> > dd-828 [000] ..... 136.724994: I_BDEV
> <-blkdev_iomap_begin
> > dd-828 [000] ..... 136.724995:
> balance_dirty_pages_ratelimited_flags <-iomap_file_buffered_write
> > dd-828 [000] ..... 136.724996: inode_to_bdi
> <-balance_dirty_pages_ratelimited_flags
> > dd-828 [000] ..... 136.724997: I_BDEV
> <-inode_to_bdi
> > dd-828 [000] ..... 136.724998: preempt_count_add
> <-balance_dirty_pages_ratelimited_flags
> > dd-828 [000] ...1. 136.725000: preempt_count_sub
> <-balance_dirty_pages_ratelimited_flags
> > dd-828 [000] ..... 136.725001: fault_in_readable
> <-fault_in_iov_iter_readable
> > dd-828 [000] ..... 136.725003: iomap_write_begin
> <-iomap_file_buffered_write
> > dd-828 [000] ..... 136.725004: iomap_get_folio
> <-iomap_write_begin
> > dd-828 [000] ..... 136.725005: __filemap_get_folio
> <-iomap_write_begin
> > dd-828 [000] ..... 136.725006: filemap_get_entry
> <-__filemap_get_folio
> > dd-828 [000] ..... 136.725007: __rcu_read_lock
> <-filemap_get_entry
> > dd-828 [000] ..... 136.725008: __rcu_read_unlock
> <-filemap_get_entry
> > dd-828 [000] ..... 136.725009: inode_to_bdi
> <-__filemap_get_folio
> > dd-828 [000] ..... 136.725010: I_BDEV
> <-inode_to_bdi
> > dd-828 [000] ..... 136.725012: __folio_alloc_noprof
> <-__filemap_get_folio
> > dd-828 [000] ..... 136.725013:
> __alloc_pages_noprof <-__folio_alloc_noprof
> > dd-828 [000] ..... 136.725014:
> get_page_from_freelist <-__alloc_pages_noprof
> > dd-828 [000] ..... 136.725015: node_dirty_ok
> <-get_page_from_freelist
> > dd-828 [000] ..... 136.725016: preempt_count_add
> <-get_page_from_freelist
> > dd-828 [000] ...1. 136.725018: _raw_spin_trylock
> <-get_page_from_freelist
> > dd-828 [000] ...1. 136.725019: preempt_count_add
> <-_raw_spin_trylock
> > dd-828 [000] ...2. 136.725021:
> _raw_spin_lock_irqsave <-__rmqueue_pcplist
> > dd-828 [000] d..2. 136.725022:
> preempt_count_add <-_raw_spin_lock_irqsave
> > dd-828 [000] d..3. 136.725025:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725028:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725029:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725031:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725032:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725034:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725036:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725037:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725038:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725039:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725040:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725042:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725043:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725044:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725046:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725047:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725048:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725050:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725051:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725052:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725053:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725055:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725056:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725057:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725058:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725060:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725061:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725062:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725063:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725065:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725066:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725067:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725068:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725070:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725071:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725072:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725073:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725075:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725076:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725077:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725079:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725080:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725081:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725082:
> __mod_zone_page_state <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.725083:
> __mod_zone_page_state <-__rmqueue_pcplist
> > ...
> > dd-828 [000] d..3. 136.820279:
> find_suitable_fallback <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.820280:
> find_suitable_fallback <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.820281:
> find_suitable_fallback <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.820282:
> find_suitable_fallback <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.820283:
> find_suitable_fallback <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.820284:
> find_suitable_fallback <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.820285:
> find_suitable_fallback <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.820286:
> find_suitable_fallback <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.820287:
> find_suitable_fallback <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.820289:
> find_suitable_fallback <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.820290:
> steal_suitable_fallback <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.820525:
> __mod_zone_page_state <-steal_suitable_fallback
> > ...
> > dd-828 [000] d..3. 136.966945:
> __mod_zone_page_state <-steal_suitable_fallback
> > dd-828 [000] d..3. 136.966946:
> find_suitable_fallback <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.966947:
> find_suitable_fallback <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.966948:
> find_suitable_fallback <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.966950:
> find_suitable_fallback <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.966951:
> find_suitable_fallback <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.966952:
> find_suitable_fallback <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.966953:
> find_suitable_fallback <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.966954:
> find_suitable_fallback <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.966955:
> find_suitable_fallback <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.966956:
> find_suitable_fallback <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.966957:
> find_suitable_fallback <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.966958:
> find_suitable_fallback <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.966959:
> find_suitable_fallback <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.966961:
> find_suitable_fallback <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.966962:
> _raw_spin_unlock_irqrestore <-__rmqueue_pcplist
> > dd-828 [000] d..3. 136.966965: irq_enter
> <-generic_handle_arch_irq
> > dd-828 [000] d..3. 136.966966: irq_enter_rcu
> <-generic_handle_arch_irq
> > dd-828 [000] d..3. 136.966967:
> preempt_count_add <-irq_enter_rcu
> > dd-828 [000] d.h3. 136.966969: gic_handle_irq
> <-generic_handle_arch_irq
> > dd-828 [000] d.h3. 136.966970:
> generic_handle_domain_irq <-gic_handle_irq
> > dd-828 [000] d.h3. 136.966971:
> __irq_resolve_mapping <-generic_handle_domain_irq
> > dd-828 [000] d.h3. 136.966972: __rcu_read_lock
> <-__irq_resolve_mapping
> > dd-828 [000] d.h3. 136.966974: __rcu_read_unlock
> <-__irq_resolve_mapping
> > dd-828 [000] d.h3. 136.966975: handle_irq_desc
> <-gic_handle_irq
> > dd-828 [000] d.h3. 136.966976:
> handle_percpu_devid_irq <-handle_irq_desc
> > dd-828 [000] d.h3. 136.966978: ipi_handler
> <-handle_percpu_devid_irq
> > dd-828 [000] d.h3. 136.966979: do_handle_IPI
> <-ipi_handler
> > dd-828 [000] d.h3. 136.966981:
> generic_smp_call_function_single_interrupt <-do_handle_IPI
> > dd-828 [000] d.h3. 136.966982:
> __flush_smp_call_function_queue <-do_handle_IPI
> > dd-828 [000] d.h3. 136.966986:
> sched_ttwu_pending <-__flush_smp_call_function_queue
> > dd-828 [000] d.h3. 136.966987:
> preempt_count_add <-sched_ttwu_pending
> > dd-828 [000] d.h4. 136.966990: _raw_spin_lock
> <-sched_ttwu_pending
> > dd-828 [000] d.h4. 136.966991:
> preempt_count_add <-_raw_spin_lock
> > dd-828 [000] d.h5. 136.966993:
> preempt_count_sub <-sched_ttwu_pending
> > dd-828 [000] d.h4. 136.966994:
> update_rq_clock.part.0 <-sched_ttwu_pending
> > dd-828 [000] d.h4. 136.966996: ttwu_do_activate
> <-sched_ttwu_pending
> > dd-828 [000] d.h4. 136.966997: activate_task
> <-ttwu_do_activate
> > dd-828 [000] d.h4. 136.966998: enqueue_task_fair
> <-activate_task
> > dd-828 [000] d.h4. 136.967000: update_curr
> <-enqueue_task_fair
> > dd-828 [000] d.h4. 136.967001:
> update_min_vruntime <-update_curr
> > dd-828 [000] d.h4. 136.967003:
> __cgroup_account_cputime <-update_curr
> >
> > BR
> > Carlos Song
> >
> > > > On Thu, Mar 20, 2025 at 11:07:41AM +0000, Carlos Song wrote:
> > > > > Hi, all
> > > > >
> > > > > I found a 300ms~600ms IRQ off when writing 1Gb data to mmc
> > > > > device at
> > > > I.MX7d SDB board at Linux-kernel-v6.14.
> > > > > But I test the same case at Linux-kernel-v6.7, this longest IRQ
> > > > > off time is only
> > > > 1ms~2ms. So the issue is introduced from v6.7~v6.14.
> > > > >
> > > > > Run this cmd to test:
> > > > > dd if=/dev/zero of=/dev/mmcblk2p4 bs=4096 seek=12500
> > > > > count=256000 conv=fsync
> > > > >
> > > > > This issue looks from blkdev_buffered_write() function. Because
> > > > > when I run this cmd with "oflag=direct" to use
> > > > > blkdev_direct_write(), I can not see
> > > > any long time IRQ off.
> > > > >
> > > > > Then I use Ftrace irqoff tracer to trace the longest IRQ off
> > > > > event, I found some
> > > > differences between v6.7 and v6.14:
> > > > > In iomap_file_buffered_write(), __folio_alloc (in v6.7) is
> > > > > replaced by
> > > > _folio_alloc_noprof (in v6.14) by this patch.
> > > > > The spinlock disabled IRQ ~300ms+. It looks there are some fixes
> > > > > for this patch,
> > > > but I still can see IRQ off 300ms+ at 6.14.0-rc7-next-20250319.
> > > > >
> > > > > Do I trigger one bug? I know little about mem so I have to
> > > > > report it and hope I
> > > > can get some help or guide.
> > > > > I put my ftrace log at the mail tail to help trace and explain.
> > > >
> > > > Did you track down which spinlock?
> > > >
> > > > >
> > >
> > > Hi,
> > >
> > > Sorry for my late reply and thank you for your quick reply!
> > > From the trace log, I think the spinlock is from here like this:
> > >
> > > __alloc_frozen_pages_noprof ==> get_page_from_freelist==>spinclok
> > >
> > > Do you need other log not only this?
> > >
> > > > > => get_page_from_freelist"
> > > > > => __alloc_frozen_pages_noprof
> > > > > => __folio_alloc_noprof
> > > > > => __filemap_get_folio
> > >
> > >
> > > > > commit b951aaff503502a7fe066eeed2744ba8a6413c89
> > > > > Author: Suren Baghdasaryan
> > > > surenb@google.com<mailto:surenb@google.com>
> > > > > Date: Thu Mar 21 09:36:40 2024 -0700
> > > > >
> > > > > mm: enable page allocation tagging
> > > > >
> > > > > Redefine page allocators to record allocation tags upon
> > > > > their
> > > invocation.
> > > > > Instrument post_alloc_hook and free_pages_prepare to modify
> > > current
> > > > > allocation tag.
> > > > >
> > > > > [surenb@google.com: undo _noprof additions in the
> documentation]
> > > > > Link:
> > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2F
> > > >
> lkml%2F&data=05%7C02%7Ccarlos.song%40nxp.com%7Cb83b5b83b6f94339
> e2f
> > > >
> 108dd708edead%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C63
> 87904
> > > >
> 81110616161%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUs
> IlYiOi
> > > >
> IwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0
> > > > %7C%7C%7C&sdata=Ebsi8btyFmX6xHr%2BvVEUHIFbsPKqETE%2Brf5Hs41
> %2Ff8o%
> > > > 3D&reserved=0
> > > > .kern
> > > >
> > >
> el.org%2Fr%2F20240326231453.1206227-3-surenb%40google.com&data=05
> %
> > > >
> > >
> 7C02%7Ccarlos.song%40nxp.com%7Cdf027bbb97074fc2cde808dd67a1d6c5%
> 7
> > > >
> > >
> C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C63878066698304673
> 8%
> > > >
> > >
> 7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMD
> A
> > > >
> > >
> wMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7
> C
> > > >
> > >
> &sdata=Aby13KyQjF5pcbYW%2BkfsEJLiaPmS2ZiJLUHJ%2BCr2JXM%3D&reserv
> e
> > > > d=0
> > > > > Link:
> > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2F
> > > >
> lkml%2F&data=05%7C02%7Ccarlos.song%40nxp.com%7Cb83b5b83b6f94339
> e2f
> > > >
> 108dd708edead%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C63
> 87904
> > > >
> 81110632328%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUs
> IlYiOi
> > > >
> IwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0
> > > > %7C%7C%7C&sdata=f5xbAEifpWVbOVuxJ5LV5FS1TmaTD5T4Q%2BhtX2Ju
> lsw%3D&r
> > > > eserved=0
> > > > .kern
> > > >
> > >
> el.org%2Fr%2F20240321163705.3067592-19-surenb%40google.com&data=0
> 5
> > > > %7C02%7Ccarlos.song%40nxp.com%7Cdf027bbb97074fc2cde808dd67a
> 1d6c5
> > > %
> > > >
> > >
> 7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6387806669830657
> 90
> > > > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwL
> jAuM
> > > D
> > > >
> > >
> AwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%
> 7
> > > >
> > >
> C&sdata=n0hjs2fhjYs%2BcnbrxHy4vFK6D4GFVL4%2Fu72anOOLiEI%3D&reserv
> e
> > > > d=0
> > > > > Signed-off-by: Suren Baghdasaryan
> > > > surenb@google.com<mailto:surenb@google.com>
> > > > > Co-developed-by: Kent Overstreet
> > > > kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev>
> > > > > Signed-off-by: Kent Overstreet
> > > > kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev>
> > > > > Reviewed-by: Kees Cook
> > > > keescook@chromium.org<mailto:keescook@chromium.org>
> > > > > Tested-by: Kees Cook
> > > > keescook@chromium.org<mailto:keescook@chromium.org>
> > > > > Cc: Alexander Viro
> > > > viro@zeniv.linux.org.uk<mailto:viro@zeniv.linux.org.uk>
> > > > > Cc: Alex Gaynor
> > > > alex.gaynor@gmail.com<mailto:alex.gaynor@gmail.com>
> > > > > Cc: Alice Ryhl
> aliceryhl@google.com<mailto:aliceryhl@google.com>
> > > > > Cc: Andreas Hindborg
> > > > a.hindborg@samsung.com<mailto:a.hindborg@samsung.com>
> > > > > Cc: Benno Lossin
> > > > benno.lossin@proton.me<mailto:benno.lossin@proton.me>
> > > > > Cc: "Björn Roy Baron"
> > > > bjorn3_gh@protonmail.com<mailto:bjorn3_gh@protonmail.com>
> > > > > Cc: Boqun Feng
> > > > boqun.feng@gmail.com<mailto:boqun.feng@gmail.com>
> > > > > Cc: Christoph Lameter cl@linux.com<mailto:cl@linux.com>
> > > > > Cc: Dennis Zhou dennis@kernel.org<mailto:dennis@kernel.org>
> > > > > Cc: Gary Guo gary@garyguo.net<mailto:gary@garyguo.net>
> > > > > Cc: Miguel Ojeda ojeda@kernel.org<mailto:ojeda@kernel.org>
> > > > > Cc: Pasha Tatashin
> > > > pasha.tatashin@soleen.com<mailto:pasha.tatashin@soleen.com>
> > > > > Cc: Peter Zijlstra
> > > peterz@infradead.org<mailto:peterz@infradead.org>
> > > > > Cc: Tejun Heo tj@kernel.org<mailto:tj@kernel.org>
> > > > > Cc: Vlastimil Babka vbabka@suse.cz<mailto:vbabka@suse.cz>
> > > > > Cc: Wedson Almeida Filho
> > > > wedsonaf@gmail.com<mailto:wedsonaf@gmail.com>
> > > > > Signed-off-by: Andrew Morton
> > > > > akpm@linux-foundation.org<mailto:akpm@linux-foundation.org>
> > > > >
> > > > >
> > > > > Ftrace irqoff tracer shows detail:
> > > > > At v6.14:
> > > > > # tracer: irqsoff
> > > > > #
> > > > > # irqsoff latency trace v1.1.5 on 6.14.0-rc7-next-20250319 #
> > > > > ----------------------------------------------------------------
> > > > > ---- # latency: 279663 us, #21352/21352, CPU#0 | (M:NONE VP:0,
> > > > > KP:0, SP:0
> > > > > HP:0
> > > > #P:2)
> > > > > # -----------------
> > > > > # | task: dd-805 (uid:0 nice:0 policy:0 rt_prio:0)
> > > > > # -----------------
> > > > > # => started at: __rmqueue_pcplist
> > > > > # => ended at: _raw_spin_unlock_irqrestore
> > > > > #
> > > > > #
> > > > > # _------=> CPU#
> > > > > # / _-----=> irqs-off/BH-disabled
> > > > > # | / _----=> need-resched
> > > > > # || / _---=> hardirq/softirq
> > > > > # ||| / _--=> preempt-depth
> > > > > # |||| / _-=> migrate-disable
> > > > > # ||||| / delay
> > > > > # cmd pid |||||| time | caller
> > > > > # \ / |||||| \ | /
> > > > > dd-805 0d.... 1us : __rmqueue_pcplist
> > > > > dd-805 0d.... 3us : _raw_spin_trylock
> > > > <-__rmqueue_pcplist
> > > > > dd-805 0d.... 7us : __mod_zone_page_state
> > > > <-__rmqueue_pcplist
> > > > > dd-805 0d.... 10us : __mod_zone_page_state
> > > > <-__rmqueue_pcplist
> > > > > dd-805 0d.... 12us : __mod_zone_page_state
> > > > <-__rmqueue_pcplist
> > > > > dd-805 0d.... 15us : __mod_zone_page_state
> > > > <-__rmqueue_pcplist
> > > > > dd-805 0d.... 17us : __mod_zone_page_state
> > > > <-__rmqueue_pcplist
> > > > > dd-805 0d.... 19us : __mod_zone_page_state
> > > > <-__rmqueue_pcplist
> > > > > ...
> > > > > dd-805 0d.... 1535us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-805 0d.... 1538us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-805 0d.... 1539us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-805 0d.... 1542us+: try_to_claim_block
> > > > <-__rmqueue_pcplist
> > > > > dd-805 0d.... 1597us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-805 0d.... 1599us+: try_to_claim_block
> > > > <-__rmqueue_pcplist
> > > > > dd-805 0d.... 1674us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-805 0d.... 1676us+: try_to_claim_block
> > > > <-__rmqueue_pcplist
> > > > > dd-805 0d.... 1716us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-805 0d.... 1718us+: try_to_claim_block
> > > > <-__rmqueue_pcplist
> > > > > dd-805 0d.... 1801us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-805 0d.... 1803us+: try_to_claim_block
> > > > <-__rmqueue_pcplist
> > > > > ...
> > > > > dd-805 0d.... 279555us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-805 0d.... 279556us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-805 0d.... 279558us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-805 0d.... 279560us+: try_to_claim_block
> > > > <-__rmqueue_pcplist
> > > > > dd-805 0d.... 279616us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-805 0d.... 279618us : __mod_zone_page_state
> > > > <-__rmqueue_pcplist
> > > > > dd-805 0d.... 279620us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > ...
> > > > > dd-805 0d.... 279658us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-805 0d.... 279660us : _raw_spin_unlock_irqrestore
> > > > <-__rmqueue_pcplist
> > > > > dd-805 0d.... 279662us : _raw_spin_unlock_irqrestore
> > > > > dd-805 0d.... 279666us+: trace_hardirqs_on
> > > > <-_raw_spin_unlock_irqrestore
> > > > > dd-805 0d.... 279712us : <stack trace>
> > > > > => get_page_from_freelist
> > > > > => __alloc_frozen_pages_noprof
> > > > > => __folio_alloc_noprof
> > > > > => __filemap_get_folio
> > > > > => iomap_write_begin
> > > > > => iomap_file_buffered_write
> > > > > => blkdev_write_iter
> > > > > => vfs_write
> > > > > => ksys_write
> > > > > => ret_fast_syscall
> > > > >
> > > > > At v6.7:
> > > > > # tracer: irqsoff
> > > > > #
> > > > > # irqsoff latency trace v1.1.5 on 6.7.0 #
> > > > > ----------------------------------------------------------------
> > > > > ---- # latency: 2477 us, #146/146, CPU#0 | (M:server VP:0, KP:0,
> > > > > SP:0 HP:0 #P:2)
> > > > > # -----------------
> > > > > # | task: dd-808 (uid:0 nice:0 policy:0 rt_prio:0)
> > > > > # -----------------
> > > > > # => started at: _raw_spin_lock_irqsave
> > > > > # => ended at: _raw_spin_unlock_irqrestore
> > > > > #
> > > > > #
> > > > > # _------=> CPU#
> > > > > # / _-----=> irqs-off/BH-disabled
> > > > > # | / _----=> need-resched
> > > > > # || / _---=> hardirq/softirq
> > > > > # ||| / _--=> preempt-depth
> > > > > # |||| / _-=> migrate-disable
> > > > > # ||||| / delay
> > > > > # cmd pid |||||| time | caller
> > > > > # \ / |||||| \ | /
> > > > > dd-808 0d.... 1us!: _raw_spin_lock_irqsave
> > > > > dd-808 0d.... 186us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 189us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 191us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 192us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 194us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 196us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 199us : steal_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 203us!: move_freepages_block
> > > > <-steal_suitable_fallback
> > > > > dd-808 0d.... 330us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 332us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 334us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 336us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 338us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 339us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 341us : steal_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 343us!: move_freepages_block
> > > > <-steal_suitable_fallback
> > > > > dd-808 0d.... 479us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 481us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 483us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 485us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 486us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 488us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 490us : steal_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 492us!: move_freepages_block
> > > > <-steal_suitable_fallback
> > > > > dd-808 0d.... 630us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 632us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 634us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 636us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 638us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 640us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 642us : steal_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 644us!: move_freepages_block
> > > > <-steal_suitable_fallback
> > > > > dd-808 0d.... 771us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 773us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 775us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 777us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 778us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 780us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 782us : steal_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 784us!: move_freepages_block
> > > > <-steal_suitable_fallback
> > > > > dd-808 0d.... 911us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 913us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 915us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 916us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 918us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 920us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 922us : steal_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 924us!: move_freepages_block
> > > > <-steal_suitable_fallback
> > > > > dd-808 0d.... 1055us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1058us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1059us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1061us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1063us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1065us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1066us : steal_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1068us!: move_freepages_block
> > > > <-steal_suitable_fallback
> > > > > dd-808 0d.... 1194us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1196us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1198us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1200us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1202us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1203us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1205us : steal_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1208us!: move_freepages_block
> > > > <-steal_suitable_fallback
> > > > > dd-808 0d.... 1333us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1335us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1337us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1339us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1341us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1342us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1344us : steal_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1346us!: move_freepages_block
> > > > <-steal_suitable_fallback
> > > > > dd-808 0d.... 1480us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1482us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1484us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1486us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1488us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1490us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1492us : steal_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1494us!: move_freepages_block
> > > > <-steal_suitable_fallback
> > > > > dd-808 0d.... 1621us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1623us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1625us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1627us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1629us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1630us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1632us : steal_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1634us!: move_freepages_block
> > > > <-steal_suitable_fallback
> > > > > dd-808 0d.... 1761us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1763us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1765us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1766us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1768us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1770us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1772us : steal_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1774us!: move_freepages_block
> > > > <-steal_suitable_fallback
> > > > > dd-808 0d.... 1900us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1902us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1903us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1905us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1907us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1909us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1911us : steal_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 1913us!: move_freepages_block
> > > > <-steal_suitable_fallback
> > > > > dd-808 0d.... 2038us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2040us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2042us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2044us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2046us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2047us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2049us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2051us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2053us : steal_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2055us!: move_freepages_block
> > > > <-steal_suitable_fallback
> > > > > dd-808 0d.... 2175us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2176us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2178us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2180us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2182us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2183us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2185us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2187us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2189us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2191us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2192us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2194us : steal_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2196us!: move_freepages_block
> > > > <-steal_suitable_fallback
> > > > > dd-808 0d.... 2323us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2325us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2327us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2328us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2330us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2332us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2334us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2335us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2337us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2339us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2341us : find_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2343us : steal_suitable_fallback
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2345us!: move_freepages_block
> > > > <-steal_suitable_fallback
> > > > > dd-808 0d.... 2470us : __mod_zone_page_state
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2473us : _raw_spin_unlock_irqrestore
> > > > <-__rmqueue_pcplist
> > > > > dd-808 0d.... 2476us : _raw_spin_unlock_irqrestore
> > > > > dd-808 0d.... 2479us+: tracer_hardirqs_on
> > > > <-_raw_spin_unlock_irqrestore
> > > > > dd-808 0d.... 2520us : <stack trace>
> > > > > => get_page_from_freelist
> > > > > => __alloc_pages
> > > > > => __folio_alloc
> > > > > => __filemap_get_folio
> > > > > => iomap_write_begin
> > > > > => iomap_file_buffered_write
> > > > > => blkdev_write_iter
> > > > > => vfs_write
> > > > > => ksys_write
> > > > > => ret_fast_syscall
> > > > >
> > > > > Best Regard
> > > > > Carlos Song
> > > > >
^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2025-04-02 2:31 UTC | newest]
Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2025-03-20 11:07 Ask help about this patch b951aaff5035 " mm: enable page allocation tagging" Carlos Song
2025-03-20 11:24 ` Kent Overstreet
2025-03-20 15:50 ` Suren Baghdasaryan
2025-03-27 8:20 ` [EXT] " Carlos Song
2025-03-27 8:25 ` Carlos Song
2025-03-27 11:02 ` Carlos Song
2025-03-31 20:01 ` Suren Baghdasaryan
2025-04-02 2:30 ` Carlos Song
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox