* Ask help about this patch b951aaff5035 " mm: enable page allocation tagging"
@ 2025-03-20 11:07 Carlos Song
2025-03-20 11:24 ` Kent Overstreet
0 siblings, 1 reply; 8+ messages in thread
From: Carlos Song @ 2025-03-20 11:07 UTC (permalink / raw)
To: surenb, kent.overstreet, akpm, willy
Cc: linux-mm, linux-kernel, linux-fsdevel
[-- Attachment #1: Type: text/plain, Size: 19580 bytes --]
Hi, all
I found a 300ms~600ms IRQ off when writing 1Gb data to mmc device at I.MX7d SDB board at Linux-kernel-v6.14.
But I test the same case at Linux-kernel-v6.7, this longest IRQ off time is only 1ms~2ms. So the issue is introduced from v6.7~v6.14.
Run this cmd to test:
dd if=/dev/zero of=/dev/mmcblk2p4 bs=4096 seek=12500 count=256000 conv=fsync
This issue looks from blkdev_buffered_write() function. Because when I run this cmd with "oflag=direct" to use
blkdev_direct_write(), I can not see any long time IRQ off.
Then I use Ftrace irqoff tracer to trace the longest IRQ off event, I found some differences between v6.7 and v6.14:
In iomap_file_buffered_write(), __folio_alloc (in v6.7) is replaced by _folio_alloc_noprof (in v6.14) by this patch.
The spinlock disabled IRQ ~300ms+. It looks there are some fixes for this patch, but I still can see IRQ off 300ms+ at 6.14.0-rc7-next-20250319.
Do I trigger one bug? I know little about mem so I have to report it and hope I can get some help or guide.
I put my ftrace log at the mail tail to help trace and explain.
commit b951aaff503502a7fe066eeed2744ba8a6413c89
Author: Suren Baghdasaryan surenb@google.com<mailto:surenb@google.com>
Date: Thu Mar 21 09:36:40 2024 -0700
mm: enable page allocation tagging
Redefine page allocators to record allocation tags upon their invocation.
Instrument post_alloc_hook and free_pages_prepare to modify current
allocation tag.
[surenb@google.com: undo _noprof additions in the documentation]
Link: https://lkml.kernel.org/r/20240326231453.1206227-3-surenb@google.com
Link: https://lkml.kernel.org/r/20240321163705.3067592-19-surenb@google.com
Signed-off-by: Suren Baghdasaryan surenb@google.com<mailto:surenb@google.com>
Co-developed-by: Kent Overstreet kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev>
Signed-off-by: Kent Overstreet kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev>
Reviewed-by: Kees Cook keescook@chromium.org<mailto:keescook@chromium.org>
Tested-by: Kees Cook keescook@chromium.org<mailto:keescook@chromium.org>
Cc: Alexander Viro viro@zeniv.linux.org.uk<mailto:viro@zeniv.linux.org.uk>
Cc: Alex Gaynor alex.gaynor@gmail.com<mailto:alex.gaynor@gmail.com>
Cc: Alice Ryhl aliceryhl@google.com<mailto:aliceryhl@google.com>
Cc: Andreas Hindborg a.hindborg@samsung.com<mailto:a.hindborg@samsung.com>
Cc: Benno Lossin benno.lossin@proton.me<mailto:benno.lossin@proton.me>
Cc: "Björn Roy Baron" bjorn3_gh@protonmail.com<mailto:bjorn3_gh@protonmail.com>
Cc: Boqun Feng boqun.feng@gmail.com<mailto:boqun.feng@gmail.com>
Cc: Christoph Lameter cl@linux.com<mailto:cl@linux.com>
Cc: Dennis Zhou dennis@kernel.org<mailto:dennis@kernel.org>
Cc: Gary Guo gary@garyguo.net<mailto:gary@garyguo.net>
Cc: Miguel Ojeda ojeda@kernel.org<mailto:ojeda@kernel.org>
Cc: Pasha Tatashin pasha.tatashin@soleen.com<mailto:pasha.tatashin@soleen.com>
Cc: Peter Zijlstra peterz@infradead.org<mailto:peterz@infradead.org>
Cc: Tejun Heo tj@kernel.org<mailto:tj@kernel.org>
Cc: Vlastimil Babka vbabka@suse.cz<mailto:vbabka@suse.cz>
Cc: Wedson Almeida Filho wedsonaf@gmail.com<mailto:wedsonaf@gmail.com>
Signed-off-by: Andrew Morton akpm@linux-foundation.org<mailto:akpm@linux-foundation.org>
Ftrace irqoff tracer shows detail:
At v6.14:
# tracer: irqsoff
#
# irqsoff latency trace v1.1.5 on 6.14.0-rc7-next-20250319
# --------------------------------------------------------------------
# latency: 279663 us, #21352/21352, CPU#0 | (M:NONE VP:0, KP:0, SP:0 HP:0 #P:2)
# -----------------
# | task: dd-805 (uid:0 nice:0 policy:0 rt_prio:0)
# -----------------
# => started at: __rmqueue_pcplist
# => ended at: _raw_spin_unlock_irqrestore
#
#
# _------=> CPU#
# / _-----=> irqs-off/BH-disabled
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / _-=> migrate-disable
# ||||| / delay
# cmd pid |||||| time | caller
# \ / |||||| \ | /
dd-805 0d.... 1us : __rmqueue_pcplist
dd-805 0d.... 3us : _raw_spin_trylock <-__rmqueue_pcplist
dd-805 0d.... 7us : __mod_zone_page_state <-__rmqueue_pcplist
dd-805 0d.... 10us : __mod_zone_page_state <-__rmqueue_pcplist
dd-805 0d.... 12us : __mod_zone_page_state <-__rmqueue_pcplist
dd-805 0d.... 15us : __mod_zone_page_state <-__rmqueue_pcplist
dd-805 0d.... 17us : __mod_zone_page_state <-__rmqueue_pcplist
dd-805 0d.... 19us : __mod_zone_page_state <-__rmqueue_pcplist
...
dd-805 0d.... 1535us : find_suitable_fallback <-__rmqueue_pcplist
dd-805 0d.... 1538us : find_suitable_fallback <-__rmqueue_pcplist
dd-805 0d.... 1539us : find_suitable_fallback <-__rmqueue_pcplist
dd-805 0d.... 1542us+: try_to_claim_block <-__rmqueue_pcplist
dd-805 0d.... 1597us : find_suitable_fallback <-__rmqueue_pcplist
dd-805 0d.... 1599us+: try_to_claim_block <-__rmqueue_pcplist
dd-805 0d.... 1674us : find_suitable_fallback <-__rmqueue_pcplist
dd-805 0d.... 1676us+: try_to_claim_block <-__rmqueue_pcplist
dd-805 0d.... 1716us : find_suitable_fallback <-__rmqueue_pcplist
dd-805 0d.... 1718us+: try_to_claim_block <-__rmqueue_pcplist
dd-805 0d.... 1801us : find_suitable_fallback <-__rmqueue_pcplist
dd-805 0d.... 1803us+: try_to_claim_block <-__rmqueue_pcplist
...
dd-805 0d.... 279555us : find_suitable_fallback <-__rmqueue_pcplist
dd-805 0d.... 279556us : find_suitable_fallback <-__rmqueue_pcplist
dd-805 0d.... 279558us : find_suitable_fallback <-__rmqueue_pcplist
dd-805 0d.... 279560us+: try_to_claim_block <-__rmqueue_pcplist
dd-805 0d.... 279616us : find_suitable_fallback <-__rmqueue_pcplist
dd-805 0d.... 279618us : __mod_zone_page_state <-__rmqueue_pcplist
dd-805 0d.... 279620us : find_suitable_fallback <-__rmqueue_pcplist
...
dd-805 0d.... 279658us : find_suitable_fallback <-__rmqueue_pcplist
dd-805 0d.... 279660us : _raw_spin_unlock_irqrestore <-__rmqueue_pcplist
dd-805 0d.... 279662us : _raw_spin_unlock_irqrestore
dd-805 0d.... 279666us+: trace_hardirqs_on <-_raw_spin_unlock_irqrestore
dd-805 0d.... 279712us : <stack trace>
=> get_page_from_freelist
=> __alloc_frozen_pages_noprof
=> __folio_alloc_noprof
=> __filemap_get_folio
=> iomap_write_begin
=> iomap_file_buffered_write
=> blkdev_write_iter
=> vfs_write
=> ksys_write
=> ret_fast_syscall
At v6.7:
# tracer: irqsoff
#
# irqsoff latency trace v1.1.5 on 6.7.0
# --------------------------------------------------------------------
# latency: 2477 us, #146/146, CPU#0 | (M:server VP:0, KP:0, SP:0 HP:0 #P:2)
# -----------------
# | task: dd-808 (uid:0 nice:0 policy:0 rt_prio:0)
# -----------------
# => started at: _raw_spin_lock_irqsave
# => ended at: _raw_spin_unlock_irqrestore
#
#
# _------=> CPU#
# / _-----=> irqs-off/BH-disabled
# | / _----=> need-resched
# || / _---=> hardirq/softirq
# ||| / _--=> preempt-depth
# |||| / _-=> migrate-disable
# ||||| / delay
# cmd pid |||||| time | caller
# \ / |||||| \ | /
dd-808 0d.... 1us!: _raw_spin_lock_irqsave
dd-808 0d.... 186us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 189us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 191us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 192us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 194us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 196us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 199us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 203us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 330us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 332us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 334us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 336us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 338us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 339us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 341us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 343us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 479us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 481us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 483us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 485us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 486us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 488us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 490us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 492us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 630us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 632us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 634us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 636us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 638us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 640us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 642us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 644us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 771us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 773us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 775us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 777us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 778us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 780us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 782us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 784us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 911us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 913us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 915us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 916us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 918us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 920us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 922us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 924us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 1055us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1058us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1059us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1061us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1063us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1065us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1066us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1068us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 1194us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1196us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1198us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1200us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1202us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1203us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1205us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1208us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 1333us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1335us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1337us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1339us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1341us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1342us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1344us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1346us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 1480us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1482us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1484us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1486us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1488us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1490us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1492us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1494us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 1621us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1623us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1625us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1627us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1629us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1630us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1632us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1634us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 1761us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1763us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1765us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1766us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1768us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1770us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1772us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1774us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 1900us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1902us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1903us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1905us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1907us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1909us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1911us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 1913us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 2038us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2040us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2042us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2044us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2046us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2047us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2049us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2051us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2053us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2055us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 2175us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2176us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2178us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2180us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2182us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2183us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2185us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2187us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2189us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2191us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2192us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2194us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2196us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 2323us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2325us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2327us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2328us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2330us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2332us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2334us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2335us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2337us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2339us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2341us : find_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2343us : steal_suitable_fallback <-__rmqueue_pcplist
dd-808 0d.... 2345us!: move_freepages_block <-steal_suitable_fallback
dd-808 0d.... 2470us : __mod_zone_page_state <-__rmqueue_pcplist
dd-808 0d.... 2473us : _raw_spin_unlock_irqrestore <-__rmqueue_pcplist
dd-808 0d.... 2476us : _raw_spin_unlock_irqrestore
dd-808 0d.... 2479us+: tracer_hardirqs_on <-_raw_spin_unlock_irqrestore
dd-808 0d.... 2520us : <stack trace>
=> get_page_from_freelist
=> __alloc_pages
=> __folio_alloc
=> __filemap_get_folio
=> iomap_write_begin
=> iomap_file_buffered_write
=> blkdev_write_iter
=> vfs_write
=> ksys_write
=> ret_fast_syscall
Best Regard
Carlos Song
[-- Attachment #2: Type: text/html, Size: 54419 bytes --]
^ permalink raw reply [flat|nested] 8+ messages in thread* Re: Ask help about this patch b951aaff5035 " mm: enable page allocation tagging" 2025-03-20 11:07 Ask help about this patch b951aaff5035 " mm: enable page allocation tagging" Carlos Song @ 2025-03-20 11:24 ` Kent Overstreet 2025-03-20 15:50 ` Suren Baghdasaryan 2025-03-27 8:25 ` Carlos Song 0 siblings, 2 replies; 8+ messages in thread From: Kent Overstreet @ 2025-03-20 11:24 UTC (permalink / raw) To: Carlos Song; +Cc: surenb, akpm, willy, linux-mm, linux-kernel, linux-fsdevel On Thu, Mar 20, 2025 at 11:07:41AM +0000, Carlos Song wrote: > Hi, all > > I found a 300ms~600ms IRQ off when writing 1Gb data to mmc device at I.MX7d SDB board at Linux-kernel-v6.14. > But I test the same case at Linux-kernel-v6.7, this longest IRQ off time is only 1ms~2ms. So the issue is introduced from v6.7~v6.14. > > Run this cmd to test: > dd if=/dev/zero of=/dev/mmcblk2p4 bs=4096 seek=12500 count=256000 conv=fsync > > This issue looks from blkdev_buffered_write() function. Because when I run this cmd with "oflag=direct" to use > blkdev_direct_write(), I can not see any long time IRQ off. > > Then I use Ftrace irqoff tracer to trace the longest IRQ off event, I found some differences between v6.7 and v6.14: > In iomap_file_buffered_write(), __folio_alloc (in v6.7) is replaced by _folio_alloc_noprof (in v6.14) by this patch. > The spinlock disabled IRQ ~300ms+. It looks there are some fixes for this patch, but I still can see IRQ off 300ms+ at 6.14.0-rc7-next-20250319. > > Do I trigger one bug? I know little about mem so I have to report it and hope I can get some help or guide. > I put my ftrace log at the mail tail to help trace and explain. Did you track down which spinlock? > > commit b951aaff503502a7fe066eeed2744ba8a6413c89 > Author: Suren Baghdasaryan surenb@google.com<mailto:surenb@google.com> > Date: Thu Mar 21 09:36:40 2024 -0700 > > mm: enable page allocation tagging > > Redefine page allocators to record allocation tags upon their invocation. > Instrument post_alloc_hook and free_pages_prepare to modify current > allocation tag. > > [surenb@google.com: undo _noprof additions in the documentation] > Link: https://lkml.kernel.org/r/20240326231453.1206227-3-surenb@google.com > Link: https://lkml.kernel.org/r/20240321163705.3067592-19-surenb@google.com > Signed-off-by: Suren Baghdasaryan surenb@google.com<mailto:surenb@google.com> > Co-developed-by: Kent Overstreet kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev> > Signed-off-by: Kent Overstreet kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev> > Reviewed-by: Kees Cook keescook@chromium.org<mailto:keescook@chromium.org> > Tested-by: Kees Cook keescook@chromium.org<mailto:keescook@chromium.org> > Cc: Alexander Viro viro@zeniv.linux.org.uk<mailto:viro@zeniv.linux.org.uk> > Cc: Alex Gaynor alex.gaynor@gmail.com<mailto:alex.gaynor@gmail.com> > Cc: Alice Ryhl aliceryhl@google.com<mailto:aliceryhl@google.com> > Cc: Andreas Hindborg a.hindborg@samsung.com<mailto:a.hindborg@samsung.com> > Cc: Benno Lossin benno.lossin@proton.me<mailto:benno.lossin@proton.me> > Cc: "Björn Roy Baron" bjorn3_gh@protonmail.com<mailto:bjorn3_gh@protonmail.com> > Cc: Boqun Feng boqun.feng@gmail.com<mailto:boqun.feng@gmail.com> > Cc: Christoph Lameter cl@linux.com<mailto:cl@linux.com> > Cc: Dennis Zhou dennis@kernel.org<mailto:dennis@kernel.org> > Cc: Gary Guo gary@garyguo.net<mailto:gary@garyguo.net> > Cc: Miguel Ojeda ojeda@kernel.org<mailto:ojeda@kernel.org> > Cc: Pasha Tatashin pasha.tatashin@soleen.com<mailto:pasha.tatashin@soleen.com> > Cc: Peter Zijlstra peterz@infradead.org<mailto:peterz@infradead.org> > Cc: Tejun Heo tj@kernel.org<mailto:tj@kernel.org> > Cc: Vlastimil Babka vbabka@suse.cz<mailto:vbabka@suse.cz> > Cc: Wedson Almeida Filho wedsonaf@gmail.com<mailto:wedsonaf@gmail.com> > Signed-off-by: Andrew Morton akpm@linux-foundation.org<mailto:akpm@linux-foundation.org> > > > Ftrace irqoff tracer shows detail: > At v6.14: > # tracer: irqsoff > # > # irqsoff latency trace v1.1.5 on 6.14.0-rc7-next-20250319 > # -------------------------------------------------------------------- > # latency: 279663 us, #21352/21352, CPU#0 | (M:NONE VP:0, KP:0, SP:0 HP:0 #P:2) > # ----------------- > # | task: dd-805 (uid:0 nice:0 policy:0 rt_prio:0) > # ----------------- > # => started at: __rmqueue_pcplist > # => ended at: _raw_spin_unlock_irqrestore > # > # > # _------=> CPU# > # / _-----=> irqs-off/BH-disabled > # | / _----=> need-resched > # || / _---=> hardirq/softirq > # ||| / _--=> preempt-depth > # |||| / _-=> migrate-disable > # ||||| / delay > # cmd pid |||||| time | caller > # \ / |||||| \ | / > dd-805 0d.... 1us : __rmqueue_pcplist > dd-805 0d.... 3us : _raw_spin_trylock <-__rmqueue_pcplist > dd-805 0d.... 7us : __mod_zone_page_state <-__rmqueue_pcplist > dd-805 0d.... 10us : __mod_zone_page_state <-__rmqueue_pcplist > dd-805 0d.... 12us : __mod_zone_page_state <-__rmqueue_pcplist > dd-805 0d.... 15us : __mod_zone_page_state <-__rmqueue_pcplist > dd-805 0d.... 17us : __mod_zone_page_state <-__rmqueue_pcplist > dd-805 0d.... 19us : __mod_zone_page_state <-__rmqueue_pcplist > ... > dd-805 0d.... 1535us : find_suitable_fallback <-__rmqueue_pcplist > dd-805 0d.... 1538us : find_suitable_fallback <-__rmqueue_pcplist > dd-805 0d.... 1539us : find_suitable_fallback <-__rmqueue_pcplist > dd-805 0d.... 1542us+: try_to_claim_block <-__rmqueue_pcplist > dd-805 0d.... 1597us : find_suitable_fallback <-__rmqueue_pcplist > dd-805 0d.... 1599us+: try_to_claim_block <-__rmqueue_pcplist > dd-805 0d.... 1674us : find_suitable_fallback <-__rmqueue_pcplist > dd-805 0d.... 1676us+: try_to_claim_block <-__rmqueue_pcplist > dd-805 0d.... 1716us : find_suitable_fallback <-__rmqueue_pcplist > dd-805 0d.... 1718us+: try_to_claim_block <-__rmqueue_pcplist > dd-805 0d.... 1801us : find_suitable_fallback <-__rmqueue_pcplist > dd-805 0d.... 1803us+: try_to_claim_block <-__rmqueue_pcplist > ... > dd-805 0d.... 279555us : find_suitable_fallback <-__rmqueue_pcplist > dd-805 0d.... 279556us : find_suitable_fallback <-__rmqueue_pcplist > dd-805 0d.... 279558us : find_suitable_fallback <-__rmqueue_pcplist > dd-805 0d.... 279560us+: try_to_claim_block <-__rmqueue_pcplist > dd-805 0d.... 279616us : find_suitable_fallback <-__rmqueue_pcplist > dd-805 0d.... 279618us : __mod_zone_page_state <-__rmqueue_pcplist > dd-805 0d.... 279620us : find_suitable_fallback <-__rmqueue_pcplist > ... > dd-805 0d.... 279658us : find_suitable_fallback <-__rmqueue_pcplist > dd-805 0d.... 279660us : _raw_spin_unlock_irqrestore <-__rmqueue_pcplist > dd-805 0d.... 279662us : _raw_spin_unlock_irqrestore > dd-805 0d.... 279666us+: trace_hardirqs_on <-_raw_spin_unlock_irqrestore > dd-805 0d.... 279712us : <stack trace> > => get_page_from_freelist > => __alloc_frozen_pages_noprof > => __folio_alloc_noprof > => __filemap_get_folio > => iomap_write_begin > => iomap_file_buffered_write > => blkdev_write_iter > => vfs_write > => ksys_write > => ret_fast_syscall > > At v6.7: > # tracer: irqsoff > # > # irqsoff latency trace v1.1.5 on 6.7.0 > # -------------------------------------------------------------------- > # latency: 2477 us, #146/146, CPU#0 | (M:server VP:0, KP:0, SP:0 HP:0 #P:2) > # ----------------- > # | task: dd-808 (uid:0 nice:0 policy:0 rt_prio:0) > # ----------------- > # => started at: _raw_spin_lock_irqsave > # => ended at: _raw_spin_unlock_irqrestore > # > # > # _------=> CPU# > # / _-----=> irqs-off/BH-disabled > # | / _----=> need-resched > # || / _---=> hardirq/softirq > # ||| / _--=> preempt-depth > # |||| / _-=> migrate-disable > # ||||| / delay > # cmd pid |||||| time | caller > # \ / |||||| \ | / > dd-808 0d.... 1us!: _raw_spin_lock_irqsave > dd-808 0d.... 186us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 189us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 191us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 192us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 194us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 196us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 199us : steal_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 203us!: move_freepages_block <-steal_suitable_fallback > dd-808 0d.... 330us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 332us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 334us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 336us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 338us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 339us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 341us : steal_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 343us!: move_freepages_block <-steal_suitable_fallback > dd-808 0d.... 479us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 481us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 483us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 485us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 486us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 488us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 490us : steal_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 492us!: move_freepages_block <-steal_suitable_fallback > dd-808 0d.... 630us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 632us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 634us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 636us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 638us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 640us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 642us : steal_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 644us!: move_freepages_block <-steal_suitable_fallback > dd-808 0d.... 771us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 773us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 775us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 777us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 778us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 780us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 782us : steal_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 784us!: move_freepages_block <-steal_suitable_fallback > dd-808 0d.... 911us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 913us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 915us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 916us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 918us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 920us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 922us : steal_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 924us!: move_freepages_block <-steal_suitable_fallback > dd-808 0d.... 1055us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1058us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1059us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1061us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1063us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1065us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1066us : steal_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1068us!: move_freepages_block <-steal_suitable_fallback > dd-808 0d.... 1194us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1196us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1198us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1200us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1202us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1203us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1205us : steal_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1208us!: move_freepages_block <-steal_suitable_fallback > dd-808 0d.... 1333us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1335us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1337us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1339us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1341us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1342us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1344us : steal_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1346us!: move_freepages_block <-steal_suitable_fallback > dd-808 0d.... 1480us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1482us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1484us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1486us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1488us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1490us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1492us : steal_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1494us!: move_freepages_block <-steal_suitable_fallback > dd-808 0d.... 1621us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1623us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1625us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1627us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1629us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1630us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1632us : steal_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1634us!: move_freepages_block <-steal_suitable_fallback > dd-808 0d.... 1761us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1763us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1765us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1766us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1768us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1770us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1772us : steal_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1774us!: move_freepages_block <-steal_suitable_fallback > dd-808 0d.... 1900us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1902us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1903us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1905us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1907us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1909us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1911us : steal_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 1913us!: move_freepages_block <-steal_suitable_fallback > dd-808 0d.... 2038us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2040us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2042us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2044us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2046us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2047us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2049us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2051us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2053us : steal_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2055us!: move_freepages_block <-steal_suitable_fallback > dd-808 0d.... 2175us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2176us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2178us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2180us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2182us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2183us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2185us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2187us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2189us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2191us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2192us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2194us : steal_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2196us!: move_freepages_block <-steal_suitable_fallback > dd-808 0d.... 2323us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2325us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2327us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2328us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2330us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2332us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2334us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2335us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2337us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2339us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2341us : find_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2343us : steal_suitable_fallback <-__rmqueue_pcplist > dd-808 0d.... 2345us!: move_freepages_block <-steal_suitable_fallback > dd-808 0d.... 2470us : __mod_zone_page_state <-__rmqueue_pcplist > dd-808 0d.... 2473us : _raw_spin_unlock_irqrestore <-__rmqueue_pcplist > dd-808 0d.... 2476us : _raw_spin_unlock_irqrestore > dd-808 0d.... 2479us+: tracer_hardirqs_on <-_raw_spin_unlock_irqrestore > dd-808 0d.... 2520us : <stack trace> > => get_page_from_freelist > => __alloc_pages > => __folio_alloc > => __filemap_get_folio > => iomap_write_begin > => iomap_file_buffered_write > => blkdev_write_iter > => vfs_write > => ksys_write > => ret_fast_syscall > > Best Regard > Carlos Song > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Ask help about this patch b951aaff5035 " mm: enable page allocation tagging" 2025-03-20 11:24 ` Kent Overstreet @ 2025-03-20 15:50 ` Suren Baghdasaryan 2025-03-27 8:20 ` [EXT] " Carlos Song 2025-03-27 8:25 ` Carlos Song 1 sibling, 1 reply; 8+ messages in thread From: Suren Baghdasaryan @ 2025-03-20 15:50 UTC (permalink / raw) To: Kent Overstreet Cc: Carlos Song, akpm, willy, linux-mm, linux-kernel, linux-fsdevel On Thu, Mar 20, 2025 at 4:24 AM Kent Overstreet <kent.overstreet@linux.dev> wrote: > > On Thu, Mar 20, 2025 at 11:07:41AM +0000, Carlos Song wrote: > > Hi, all > > > > I found a 300ms~600ms IRQ off when writing 1Gb data to mmc device at I.MX7d SDB board at Linux-kernel-v6.14. > > But I test the same case at Linux-kernel-v6.7, this longest IRQ off time is only 1ms~2ms. So the issue is introduced from v6.7~v6.14. > > > > Run this cmd to test: > > dd if=/dev/zero of=/dev/mmcblk2p4 bs=4096 seek=12500 count=256000 conv=fsync > > > > This issue looks from blkdev_buffered_write() function. Because when I run this cmd with "oflag=direct" to use > > blkdev_direct_write(), I can not see any long time IRQ off. > > > > Then I use Ftrace irqoff tracer to trace the longest IRQ off event, I found some differences between v6.7 and v6.14: > > In iomap_file_buffered_write(), __folio_alloc (in v6.7) is replaced by _folio_alloc_noprof (in v6.14) by this patch. > > The spinlock disabled IRQ ~300ms+. It looks there are some fixes for this patch, but I still can see IRQ off 300ms+ at 6.14.0-rc7-next-20250319. Do you have CONFIG_MEM_ALLOC_PROFILING enabled and if so, does the issue disappear if you disable CONFIG_MEM_ALLOC_PROFILING? > > > > Do I trigger one bug? I know little about mem so I have to report it and hope I can get some help or guide. > > I put my ftrace log at the mail tail to help trace and explain. > > Did you track down which spinlock? > > > > > commit b951aaff503502a7fe066eeed2744ba8a6413c89 > > Author: Suren Baghdasaryan surenb@google.com<mailto:surenb@google.com> > > Date: Thu Mar 21 09:36:40 2024 -0700 > > > > mm: enable page allocation tagging > > > > Redefine page allocators to record allocation tags upon their invocation. > > Instrument post_alloc_hook and free_pages_prepare to modify current > > allocation tag. > > > > [surenb@google.com: undo _noprof additions in the documentation] > > Link: https://lkml.kernel.org/r/20240326231453.1206227-3-surenb@google.com > > Link: https://lkml.kernel.org/r/20240321163705.3067592-19-surenb@google.com > > Signed-off-by: Suren Baghdasaryan surenb@google.com<mailto:surenb@google.com> > > Co-developed-by: Kent Overstreet kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev> > > Signed-off-by: Kent Overstreet kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev> > > Reviewed-by: Kees Cook keescook@chromium.org<mailto:keescook@chromium.org> > > Tested-by: Kees Cook keescook@chromium.org<mailto:keescook@chromium.org> > > Cc: Alexander Viro viro@zeniv.linux.org.uk<mailto:viro@zeniv.linux.org.uk> > > Cc: Alex Gaynor alex.gaynor@gmail.com<mailto:alex.gaynor@gmail.com> > > Cc: Alice Ryhl aliceryhl@google.com<mailto:aliceryhl@google.com> > > Cc: Andreas Hindborg a.hindborg@samsung.com<mailto:a.hindborg@samsung.com> > > Cc: Benno Lossin benno.lossin@proton.me<mailto:benno.lossin@proton.me> > > Cc: "Björn Roy Baron" bjorn3_gh@protonmail.com<mailto:bjorn3_gh@protonmail.com> > > Cc: Boqun Feng boqun.feng@gmail.com<mailto:boqun.feng@gmail.com> > > Cc: Christoph Lameter cl@linux.com<mailto:cl@linux.com> > > Cc: Dennis Zhou dennis@kernel.org<mailto:dennis@kernel.org> > > Cc: Gary Guo gary@garyguo.net<mailto:gary@garyguo.net> > > Cc: Miguel Ojeda ojeda@kernel.org<mailto:ojeda@kernel.org> > > Cc: Pasha Tatashin pasha.tatashin@soleen.com<mailto:pasha.tatashin@soleen.com> > > Cc: Peter Zijlstra peterz@infradead.org<mailto:peterz@infradead.org> > > Cc: Tejun Heo tj@kernel.org<mailto:tj@kernel.org> > > Cc: Vlastimil Babka vbabka@suse.cz<mailto:vbabka@suse.cz> > > Cc: Wedson Almeida Filho wedsonaf@gmail.com<mailto:wedsonaf@gmail.com> > > Signed-off-by: Andrew Morton akpm@linux-foundation.org<mailto:akpm@linux-foundation.org> > > > > > > Ftrace irqoff tracer shows detail: > > At v6.14: > > # tracer: irqsoff > > # > > # irqsoff latency trace v1.1.5 on 6.14.0-rc7-next-20250319 > > # -------------------------------------------------------------------- > > # latency: 279663 us, #21352/21352, CPU#0 | (M:NONE VP:0, KP:0, SP:0 HP:0 #P:2) > > # ----------------- > > # | task: dd-805 (uid:0 nice:0 policy:0 rt_prio:0) > > # ----------------- > > # => started at: __rmqueue_pcplist > > # => ended at: _raw_spin_unlock_irqrestore > > # > > # > > # _------=> CPU# > > # / _-----=> irqs-off/BH-disabled > > # | / _----=> need-resched > > # || / _---=> hardirq/softirq > > # ||| / _--=> preempt-depth > > # |||| / _-=> migrate-disable > > # ||||| / delay > > # cmd pid |||||| time | caller > > # \ / |||||| \ | / > > dd-805 0d.... 1us : __rmqueue_pcplist > > dd-805 0d.... 3us : _raw_spin_trylock <-__rmqueue_pcplist > > dd-805 0d.... 7us : __mod_zone_page_state <-__rmqueue_pcplist > > dd-805 0d.... 10us : __mod_zone_page_state <-__rmqueue_pcplist > > dd-805 0d.... 12us : __mod_zone_page_state <-__rmqueue_pcplist > > dd-805 0d.... 15us : __mod_zone_page_state <-__rmqueue_pcplist > > dd-805 0d.... 17us : __mod_zone_page_state <-__rmqueue_pcplist > > dd-805 0d.... 19us : __mod_zone_page_state <-__rmqueue_pcplist > > ... > > dd-805 0d.... 1535us : find_suitable_fallback <-__rmqueue_pcplist > > dd-805 0d.... 1538us : find_suitable_fallback <-__rmqueue_pcplist > > dd-805 0d.... 1539us : find_suitable_fallback <-__rmqueue_pcplist > > dd-805 0d.... 1542us+: try_to_claim_block <-__rmqueue_pcplist > > dd-805 0d.... 1597us : find_suitable_fallback <-__rmqueue_pcplist > > dd-805 0d.... 1599us+: try_to_claim_block <-__rmqueue_pcplist > > dd-805 0d.... 1674us : find_suitable_fallback <-__rmqueue_pcplist > > dd-805 0d.... 1676us+: try_to_claim_block <-__rmqueue_pcplist > > dd-805 0d.... 1716us : find_suitable_fallback <-__rmqueue_pcplist > > dd-805 0d.... 1718us+: try_to_claim_block <-__rmqueue_pcplist > > dd-805 0d.... 1801us : find_suitable_fallback <-__rmqueue_pcplist > > dd-805 0d.... 1803us+: try_to_claim_block <-__rmqueue_pcplist > > ... > > dd-805 0d.... 279555us : find_suitable_fallback <-__rmqueue_pcplist > > dd-805 0d.... 279556us : find_suitable_fallback <-__rmqueue_pcplist > > dd-805 0d.... 279558us : find_suitable_fallback <-__rmqueue_pcplist > > dd-805 0d.... 279560us+: try_to_claim_block <-__rmqueue_pcplist > > dd-805 0d.... 279616us : find_suitable_fallback <-__rmqueue_pcplist > > dd-805 0d.... 279618us : __mod_zone_page_state <-__rmqueue_pcplist > > dd-805 0d.... 279620us : find_suitable_fallback <-__rmqueue_pcplist > > ... > > dd-805 0d.... 279658us : find_suitable_fallback <-__rmqueue_pcplist > > dd-805 0d.... 279660us : _raw_spin_unlock_irqrestore <-__rmqueue_pcplist > > dd-805 0d.... 279662us : _raw_spin_unlock_irqrestore > > dd-805 0d.... 279666us+: trace_hardirqs_on <-_raw_spin_unlock_irqrestore > > dd-805 0d.... 279712us : <stack trace> > > => get_page_from_freelist > > => __alloc_frozen_pages_noprof > > => __folio_alloc_noprof > > => __filemap_get_folio > > => iomap_write_begin > > => iomap_file_buffered_write > > => blkdev_write_iter > > => vfs_write > > => ksys_write > > => ret_fast_syscall > > > > At v6.7: > > # tracer: irqsoff > > # > > # irqsoff latency trace v1.1.5 on 6.7.0 > > # -------------------------------------------------------------------- > > # latency: 2477 us, #146/146, CPU#0 | (M:server VP:0, KP:0, SP:0 HP:0 #P:2) > > # ----------------- > > # | task: dd-808 (uid:0 nice:0 policy:0 rt_prio:0) > > # ----------------- > > # => started at: _raw_spin_lock_irqsave > > # => ended at: _raw_spin_unlock_irqrestore > > # > > # > > # _------=> CPU# > > # / _-----=> irqs-off/BH-disabled > > # | / _----=> need-resched > > # || / _---=> hardirq/softirq > > # ||| / _--=> preempt-depth > > # |||| / _-=> migrate-disable > > # ||||| / delay > > # cmd pid |||||| time | caller > > # \ / |||||| \ | / > > dd-808 0d.... 1us!: _raw_spin_lock_irqsave > > dd-808 0d.... 186us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 189us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 191us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 192us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 194us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 196us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 199us : steal_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 203us!: move_freepages_block <-steal_suitable_fallback > > dd-808 0d.... 330us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 332us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 334us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 336us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 338us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 339us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 341us : steal_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 343us!: move_freepages_block <-steal_suitable_fallback > > dd-808 0d.... 479us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 481us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 483us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 485us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 486us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 488us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 490us : steal_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 492us!: move_freepages_block <-steal_suitable_fallback > > dd-808 0d.... 630us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 632us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 634us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 636us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 638us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 640us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 642us : steal_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 644us!: move_freepages_block <-steal_suitable_fallback > > dd-808 0d.... 771us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 773us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 775us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 777us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 778us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 780us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 782us : steal_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 784us!: move_freepages_block <-steal_suitable_fallback > > dd-808 0d.... 911us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 913us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 915us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 916us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 918us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 920us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 922us : steal_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 924us!: move_freepages_block <-steal_suitable_fallback > > dd-808 0d.... 1055us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1058us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1059us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1061us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1063us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1065us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1066us : steal_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1068us!: move_freepages_block <-steal_suitable_fallback > > dd-808 0d.... 1194us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1196us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1198us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1200us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1202us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1203us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1205us : steal_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1208us!: move_freepages_block <-steal_suitable_fallback > > dd-808 0d.... 1333us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1335us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1337us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1339us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1341us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1342us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1344us : steal_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1346us!: move_freepages_block <-steal_suitable_fallback > > dd-808 0d.... 1480us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1482us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1484us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1486us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1488us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1490us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1492us : steal_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1494us!: move_freepages_block <-steal_suitable_fallback > > dd-808 0d.... 1621us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1623us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1625us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1627us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1629us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1630us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1632us : steal_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1634us!: move_freepages_block <-steal_suitable_fallback > > dd-808 0d.... 1761us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1763us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1765us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1766us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1768us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1770us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1772us : steal_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1774us!: move_freepages_block <-steal_suitable_fallback > > dd-808 0d.... 1900us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1902us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1903us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1905us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1907us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1909us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1911us : steal_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 1913us!: move_freepages_block <-steal_suitable_fallback > > dd-808 0d.... 2038us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2040us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2042us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2044us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2046us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2047us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2049us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2051us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2053us : steal_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2055us!: move_freepages_block <-steal_suitable_fallback > > dd-808 0d.... 2175us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2176us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2178us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2180us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2182us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2183us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2185us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2187us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2189us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2191us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2192us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2194us : steal_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2196us!: move_freepages_block <-steal_suitable_fallback > > dd-808 0d.... 2323us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2325us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2327us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2328us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2330us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2332us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2334us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2335us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2337us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2339us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2341us : find_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2343us : steal_suitable_fallback <-__rmqueue_pcplist > > dd-808 0d.... 2345us!: move_freepages_block <-steal_suitable_fallback > > dd-808 0d.... 2470us : __mod_zone_page_state <-__rmqueue_pcplist > > dd-808 0d.... 2473us : _raw_spin_unlock_irqrestore <-__rmqueue_pcplist > > dd-808 0d.... 2476us : _raw_spin_unlock_irqrestore > > dd-808 0d.... 2479us+: tracer_hardirqs_on <-_raw_spin_unlock_irqrestore > > dd-808 0d.... 2520us : <stack trace> > > => get_page_from_freelist > > => __alloc_pages > > => __folio_alloc > > => __filemap_get_folio > > => iomap_write_begin > > => iomap_file_buffered_write > > => blkdev_write_iter > > => vfs_write > > => ksys_write > > => ret_fast_syscall > > > > Best Regard > > Carlos Song > > ^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: [EXT] Re: Ask help about this patch b951aaff5035 " mm: enable page allocation tagging" 2025-03-20 15:50 ` Suren Baghdasaryan @ 2025-03-27 8:20 ` Carlos Song 0 siblings, 0 replies; 8+ messages in thread From: Carlos Song @ 2025-03-27 8:20 UTC (permalink / raw) To: Suren Baghdasaryan, Kent Overstreet Cc: akpm, willy, linux-mm, linux-kernel, linux-fsdevel > -----Original Message----- > From: Suren Baghdasaryan <surenb@google.com> > Sent: Thursday, March 20, 2025 11:50 PM > To: Kent Overstreet <kent.overstreet@linux.dev> > Cc: Carlos Song <carlos.song@nxp.com>; akpm@linux-foundation.org; > willy@infradead.org; linux-mm@kvack.org; linux-kernel@vger.kernel.org; > linux-fsdevel@vger.kernel.org > Subject: [EXT] Re: Ask help about this patch b951aaff5035 " mm: enable page > allocation tagging" > > Caution: This is an external email. Please take care when clicking links or > opening attachments. When in doubt, report the message using the 'Report this > email' button > > > On Thu, Mar 20, 2025 at 4:24 AM Kent Overstreet <kent.overstreet@linux.dev> > wrote: > > > > On Thu, Mar 20, 2025 at 11:07:41AM +0000, Carlos Song wrote: > > > Hi, all > > > > > > I found a 300ms~600ms IRQ off when writing 1Gb data to mmc device at > I.MX7d SDB board at Linux-kernel-v6.14. > > > But I test the same case at Linux-kernel-v6.7, this longest IRQ off time is only > 1ms~2ms. So the issue is introduced from v6.7~v6.14. > > > > > > Run this cmd to test: > > > dd if=/dev/zero of=/dev/mmcblk2p4 bs=4096 seek=12500 count=256000 > > > conv=fsync > > > > > > This issue looks from blkdev_buffered_write() function. Because when > > > I run this cmd with "oflag=direct" to use blkdev_direct_write(), I can not see > any long time IRQ off. > > > > > > Then I use Ftrace irqoff tracer to trace the longest IRQ off event, I found > some differences between v6.7 and v6.14: > > > In iomap_file_buffered_write(), __folio_alloc (in v6.7) is replaced by > _folio_alloc_noprof (in v6.14) by this patch. > > > The spinlock disabled IRQ ~300ms+. It looks there are some fixes for this > patch, but I still can see IRQ off 300ms+ at 6.14.0-rc7-next-20250319. > > Do you have CONFIG_MEM_ALLOC_PROFILING enabled and if so, does the issue > disappear if you disable CONFIG_MEM_ALLOC_PROFILING? > Sorry for my late reply and thank you for your quick reply! We don't enable CONFIG_MEM_ALLOC_PROFILING in kernel. > > > > > > Do I trigger one bug? I know little about mem so I have to report it and hope > I can get some help or guide. > > > I put my ftrace log at the mail tail to help trace and explain. > > > > Did you track down which spinlock? > > > > > > > > commit b951aaff503502a7fe066eeed2744ba8a6413c89 > > > Author: Suren Baghdasaryan > surenb@google.com<mailto:surenb@google.com> > > > Date: Thu Mar 21 09:36:40 2024 -0700 > > > > > > mm: enable page allocation tagging > > > > > > Redefine page allocators to record allocation tags upon their > invocation. > > > Instrument post_alloc_hook and free_pages_prepare to modify > current > > > allocation tag. > > > > > > [surenb@google.com: undo _noprof additions in the documentation] > > > Link: > https://lkml.kern/ > el.org%2Fr%2F20240326231453.1206227-3-surenb%40google.com&data=05% > 7C02%7Ccarlos.song%40nxp.com%7Cd28a491e16da40a24a6508dd67c6e95b% > 7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C638780826228162008 > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMD > AwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7 > C&sdata=jCoaXCuYC2cLtjSdHRJn73p0wrrtVc2I36Z16R1dsZI%3D&reserved=0 > > > Link: > https://lkml.kern/ > el.org%2Fr%2F20240321163705.3067592-19-surenb%40google.com&data=05 > %7C02%7Ccarlos.song%40nxp.com%7Cd28a491e16da40a24a6508dd67c6e95b > %7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C63878082622817871 > 6%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuM > DAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C% > 7C&sdata=6ey4%2BMmBB%2FEN%2FIwSrUaWhZFql%2BNYyuuqkW6mYe9weN > k%3D&reserved=0 > > > Signed-off-by: Suren Baghdasaryan > surenb@google.com<mailto:surenb@google.com> > > > Co-developed-by: Kent Overstreet > kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev> > > > Signed-off-by: Kent Overstreet > kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev> > > > Reviewed-by: Kees Cook > keescook@chromium.org<mailto:keescook@chromium.org> > > > Tested-by: Kees Cook > keescook@chromium.org<mailto:keescook@chromium.org> > > > Cc: Alexander Viro > viro@zeniv.linux.org.uk<mailto:viro@zeniv.linux.org.uk> > > > Cc: Alex Gaynor > alex.gaynor@gmail.com<mailto:alex.gaynor@gmail.com> > > > Cc: Alice Ryhl aliceryhl@google.com<mailto:aliceryhl@google.com> > > > Cc: Andreas Hindborg > a.hindborg@samsung.com<mailto:a.hindborg@samsung.com> > > > Cc: Benno Lossin > benno.lossin@proton.me<mailto:benno.lossin@proton.me> > > > Cc: "Björn Roy Baron" > bjorn3_gh@protonmail.com<mailto:bjorn3_gh@protonmail.com> > > > Cc: Boqun Feng > boqun.feng@gmail.com<mailto:boqun.feng@gmail.com> > > > Cc: Christoph Lameter cl@linux.com<mailto:cl@linux.com> > > > Cc: Dennis Zhou dennis@kernel.org<mailto:dennis@kernel.org> > > > Cc: Gary Guo gary@garyguo.net<mailto:gary@garyguo.net> > > > Cc: Miguel Ojeda ojeda@kernel.org<mailto:ojeda@kernel.org> > > > Cc: Pasha Tatashin > pasha.tatashin@soleen.com<mailto:pasha.tatashin@soleen.com> > > > Cc: Peter Zijlstra > peterz@infradead.org<mailto:peterz@infradead.org> > > > Cc: Tejun Heo tj@kernel.org<mailto:tj@kernel.org> > > > Cc: Vlastimil Babka vbabka@suse.cz<mailto:vbabka@suse.cz> > > > Cc: Wedson Almeida Filho > wedsonaf@gmail.com<mailto:wedsonaf@gmail.com> > > > Signed-off-by: Andrew Morton > > > akpm@linux-foundation.org<mailto:akpm@linux-foundation.org> > > > > > > > > > Ftrace irqoff tracer shows detail: > > > At v6.14: > > > # tracer: irqsoff > > > # > > > # irqsoff latency trace v1.1.5 on 6.14.0-rc7-next-20250319 # > > > -------------------------------------------------------------------- > > > # latency: 279663 us, #21352/21352, CPU#0 | (M:NONE VP:0, KP:0, SP:0 > HP:0 #P:2) > > > # ----------------- > > > # | task: dd-805 (uid:0 nice:0 policy:0 rt_prio:0) > > > # ----------------- > > > # => started at: __rmqueue_pcplist > > > # => ended at: _raw_spin_unlock_irqrestore > > > # > > > # > > > # _------=> CPU# > > > # / _-----=> irqs-off/BH-disabled > > > # | / _----=> need-resched > > > # || / _---=> hardirq/softirq > > > # ||| / _--=> preempt-depth > > > # |||| / _-=> migrate-disable > > > # ||||| / delay > > > # cmd pid |||||| time | caller > > > # \ / |||||| \ | / > > > dd-805 0d.... 1us : __rmqueue_pcplist > > > dd-805 0d.... 3us : _raw_spin_trylock > <-__rmqueue_pcplist > > > dd-805 0d.... 7us : __mod_zone_page_state > <-__rmqueue_pcplist > > > dd-805 0d.... 10us : __mod_zone_page_state > <-__rmqueue_pcplist > > > dd-805 0d.... 12us : __mod_zone_page_state > <-__rmqueue_pcplist > > > dd-805 0d.... 15us : __mod_zone_page_state > <-__rmqueue_pcplist > > > dd-805 0d.... 17us : __mod_zone_page_state > <-__rmqueue_pcplist > > > dd-805 0d.... 19us : __mod_zone_page_state > <-__rmqueue_pcplist > > > ... > > > dd-805 0d.... 1535us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-805 0d.... 1538us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-805 0d.... 1539us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-805 0d.... 1542us+: try_to_claim_block > <-__rmqueue_pcplist > > > dd-805 0d.... 1597us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-805 0d.... 1599us+: try_to_claim_block > <-__rmqueue_pcplist > > > dd-805 0d.... 1674us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-805 0d.... 1676us+: try_to_claim_block > <-__rmqueue_pcplist > > > dd-805 0d.... 1716us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-805 0d.... 1718us+: try_to_claim_block > <-__rmqueue_pcplist > > > dd-805 0d.... 1801us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-805 0d.... 1803us+: try_to_claim_block > <-__rmqueue_pcplist > > > ... > > > dd-805 0d.... 279555us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-805 0d.... 279556us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-805 0d.... 279558us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-805 0d.... 279560us+: try_to_claim_block > <-__rmqueue_pcplist > > > dd-805 0d.... 279616us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-805 0d.... 279618us : __mod_zone_page_state > <-__rmqueue_pcplist > > > dd-805 0d.... 279620us : find_suitable_fallback > <-__rmqueue_pcplist > > > ... > > > dd-805 0d.... 279658us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-805 0d.... 279660us : _raw_spin_unlock_irqrestore > <-__rmqueue_pcplist > > > dd-805 0d.... 279662us : _raw_spin_unlock_irqrestore > > > dd-805 0d.... 279666us+: trace_hardirqs_on > <-_raw_spin_unlock_irqrestore > > > dd-805 0d.... 279712us : <stack trace> > > > => get_page_from_freelist > > > => __alloc_frozen_pages_noprof > > > => __folio_alloc_noprof > > > => __filemap_get_folio > > > => iomap_write_begin > > > => iomap_file_buffered_write > > > => blkdev_write_iter > > > => vfs_write > > > => ksys_write > > > => ret_fast_syscall > > > > > > At v6.7: > > > # tracer: irqsoff > > > # > > > # irqsoff latency trace v1.1.5 on 6.7.0 # > > > -------------------------------------------------------------------- > > > # latency: 2477 us, #146/146, CPU#0 | (M:server VP:0, KP:0, SP:0 HP:0 #P:2) > > > # ----------------- > > > # | task: dd-808 (uid:0 nice:0 policy:0 rt_prio:0) > > > # ----------------- > > > # => started at: _raw_spin_lock_irqsave > > > # => ended at: _raw_spin_unlock_irqrestore > > > # > > > # > > > # _------=> CPU# > > > # / _-----=> irqs-off/BH-disabled > > > # | / _----=> need-resched > > > # || / _---=> hardirq/softirq > > > # ||| / _--=> preempt-depth > > > # |||| / _-=> migrate-disable > > > # ||||| / delay > > > # cmd pid |||||| time | caller > > > # \ / |||||| \ | / > > > dd-808 0d.... 1us!: _raw_spin_lock_irqsave > > > dd-808 0d.... 186us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 189us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 191us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 192us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 194us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 196us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 199us : steal_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 203us!: move_freepages_block > <-steal_suitable_fallback > > > dd-808 0d.... 330us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 332us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 334us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 336us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 338us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 339us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 341us : steal_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 343us!: move_freepages_block > <-steal_suitable_fallback > > > dd-808 0d.... 479us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 481us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 483us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 485us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 486us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 488us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 490us : steal_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 492us!: move_freepages_block > <-steal_suitable_fallback > > > dd-808 0d.... 630us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 632us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 634us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 636us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 638us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 640us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 642us : steal_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 644us!: move_freepages_block > <-steal_suitable_fallback > > > dd-808 0d.... 771us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 773us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 775us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 777us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 778us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 780us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 782us : steal_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 784us!: move_freepages_block > <-steal_suitable_fallback > > > dd-808 0d.... 911us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 913us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 915us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 916us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 918us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 920us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 922us : steal_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 924us!: move_freepages_block > <-steal_suitable_fallback > > > dd-808 0d.... 1055us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1058us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1059us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1061us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1063us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1065us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1066us : steal_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1068us!: move_freepages_block > <-steal_suitable_fallback > > > dd-808 0d.... 1194us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1196us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1198us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1200us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1202us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1203us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1205us : steal_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1208us!: move_freepages_block > <-steal_suitable_fallback > > > dd-808 0d.... 1333us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1335us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1337us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1339us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1341us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1342us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1344us : steal_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1346us!: move_freepages_block > <-steal_suitable_fallback > > > dd-808 0d.... 1480us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1482us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1484us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1486us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1488us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1490us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1492us : steal_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1494us!: move_freepages_block > <-steal_suitable_fallback > > > dd-808 0d.... 1621us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1623us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1625us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1627us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1629us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1630us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1632us : steal_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1634us!: move_freepages_block > <-steal_suitable_fallback > > > dd-808 0d.... 1761us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1763us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1765us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1766us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1768us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1770us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1772us : steal_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1774us!: move_freepages_block > <-steal_suitable_fallback > > > dd-808 0d.... 1900us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1902us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1903us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1905us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1907us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1909us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1911us : steal_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 1913us!: move_freepages_block > <-steal_suitable_fallback > > > dd-808 0d.... 2038us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2040us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2042us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2044us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2046us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2047us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2049us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2051us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2053us : steal_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2055us!: move_freepages_block > <-steal_suitable_fallback > > > dd-808 0d.... 2175us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2176us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2178us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2180us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2182us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2183us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2185us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2187us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2189us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2191us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2192us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2194us : steal_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2196us!: move_freepages_block > <-steal_suitable_fallback > > > dd-808 0d.... 2323us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2325us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2327us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2328us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2330us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2332us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2334us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2335us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2337us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2339us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2341us : find_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2343us : steal_suitable_fallback > <-__rmqueue_pcplist > > > dd-808 0d.... 2345us!: move_freepages_block > <-steal_suitable_fallback > > > dd-808 0d.... 2470us : __mod_zone_page_state > <-__rmqueue_pcplist > > > dd-808 0d.... 2473us : _raw_spin_unlock_irqrestore > <-__rmqueue_pcplist > > > dd-808 0d.... 2476us : _raw_spin_unlock_irqrestore > > > dd-808 0d.... 2479us+: tracer_hardirqs_on > <-_raw_spin_unlock_irqrestore > > > dd-808 0d.... 2520us : <stack trace> > > > => get_page_from_freelist > > > => __alloc_pages > > > => __folio_alloc > > > => __filemap_get_folio > > > => iomap_write_begin > > > => iomap_file_buffered_write > > > => blkdev_write_iter > > > => vfs_write > > > => ksys_write > > > => ret_fast_syscall > > > > > > Best Regard > > > Carlos Song > > > ^ permalink raw reply [flat|nested] 8+ messages in thread
* RE: [EXT] Re: Ask help about this patch b951aaff5035 " mm: enable page allocation tagging" 2025-03-20 11:24 ` Kent Overstreet 2025-03-20 15:50 ` Suren Baghdasaryan @ 2025-03-27 8:25 ` Carlos Song 2025-03-27 11:02 ` Carlos Song 1 sibling, 1 reply; 8+ messages in thread From: Carlos Song @ 2025-03-27 8:25 UTC (permalink / raw) To: Kent Overstreet Cc: surenb, akpm, willy, linux-mm, linux-kernel, linux-fsdevel > -----Original Message----- > From: Kent Overstreet <kent.overstreet@linux.dev> > Sent: Thursday, March 20, 2025 7:25 PM > To: Carlos Song <carlos.song@nxp.com> > Cc: surenb@google.com; akpm@linux-foundation.org; willy@infradead.org; > linux-mm@kvack.org; linux-kernel@vger.kernel.org; > linux-fsdevel@vger.kernel.org > Subject: [EXT] Re: Ask help about this patch b951aaff5035 " mm: enable page > allocation tagging" > > Caution: This is an external email. Please take care when clicking links or > opening attachments. When in doubt, report the message using the 'Report this > email' button > > > On Thu, Mar 20, 2025 at 11:07:41AM +0000, Carlos Song wrote: > > Hi, all > > > > I found a 300ms~600ms IRQ off when writing 1Gb data to mmc device at > I.MX7d SDB board at Linux-kernel-v6.14. > > But I test the same case at Linux-kernel-v6.7, this longest IRQ off time is only > 1ms~2ms. So the issue is introduced from v6.7~v6.14. > > > > Run this cmd to test: > > dd if=/dev/zero of=/dev/mmcblk2p4 bs=4096 seek=12500 count=256000 > > conv=fsync > > > > This issue looks from blkdev_buffered_write() function. Because when I > > run this cmd with "oflag=direct" to use blkdev_direct_write(), I can not see > any long time IRQ off. > > > > Then I use Ftrace irqoff tracer to trace the longest IRQ off event, I found some > differences between v6.7 and v6.14: > > In iomap_file_buffered_write(), __folio_alloc (in v6.7) is replaced by > _folio_alloc_noprof (in v6.14) by this patch. > > The spinlock disabled IRQ ~300ms+. It looks there are some fixes for this patch, > but I still can see IRQ off 300ms+ at 6.14.0-rc7-next-20250319. > > > > Do I trigger one bug? I know little about mem so I have to report it and hope I > can get some help or guide. > > I put my ftrace log at the mail tail to help trace and explain. > > Did you track down which spinlock? > > > Hi, Sorry for my late reply and thank you for your quick reply! From the trace log, I think the spinlock is from here like this: __alloc_frozen_pages_noprof ==> get_page_from_freelist==>spinclok Do you need other log not only this? > > => get_page_from_freelist" > > => __alloc_frozen_pages_noprof > > => __folio_alloc_noprof > > => __filemap_get_folio > > commit b951aaff503502a7fe066eeed2744ba8a6413c89 > > Author: Suren Baghdasaryan > surenb@google.com<mailto:surenb@google.com> > > Date: Thu Mar 21 09:36:40 2024 -0700 > > > > mm: enable page allocation tagging > > > > Redefine page allocators to record allocation tags upon their invocation. > > Instrument post_alloc_hook and free_pages_prepare to modify current > > allocation tag. > > > > [surenb@google.com: undo _noprof additions in the documentation] > > Link: > https://lkml.kern/ > el.org%2Fr%2F20240326231453.1206227-3-surenb%40google.com&data=05% > 7C02%7Ccarlos.song%40nxp.com%7Cdf027bbb97074fc2cde808dd67a1d6c5%7 > C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C638780666983046738% > 7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDA > wMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C > &sdata=Aby13KyQjF5pcbYW%2BkfsEJLiaPmS2ZiJLUHJ%2BCr2JXM%3D&reserve > d=0 > > Link: > https://lkml.kern/ > el.org%2Fr%2F20240321163705.3067592-19-surenb%40google.com&data=05 > %7C02%7Ccarlos.song%40nxp.com%7Cdf027bbb97074fc2cde808dd67a1d6c5% > 7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C638780666983065790 > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMD > AwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7 > C&sdata=n0hjs2fhjYs%2BcnbrxHy4vFK6D4GFVL4%2Fu72anOOLiEI%3D&reserve > d=0 > > Signed-off-by: Suren Baghdasaryan > surenb@google.com<mailto:surenb@google.com> > > Co-developed-by: Kent Overstreet > kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev> > > Signed-off-by: Kent Overstreet > kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev> > > Reviewed-by: Kees Cook > keescook@chromium.org<mailto:keescook@chromium.org> > > Tested-by: Kees Cook > keescook@chromium.org<mailto:keescook@chromium.org> > > Cc: Alexander Viro > viro@zeniv.linux.org.uk<mailto:viro@zeniv.linux.org.uk> > > Cc: Alex Gaynor > alex.gaynor@gmail.com<mailto:alex.gaynor@gmail.com> > > Cc: Alice Ryhl aliceryhl@google.com<mailto:aliceryhl@google.com> > > Cc: Andreas Hindborg > a.hindborg@samsung.com<mailto:a.hindborg@samsung.com> > > Cc: Benno Lossin > benno.lossin@proton.me<mailto:benno.lossin@proton.me> > > Cc: "Björn Roy Baron" > bjorn3_gh@protonmail.com<mailto:bjorn3_gh@protonmail.com> > > Cc: Boqun Feng > boqun.feng@gmail.com<mailto:boqun.feng@gmail.com> > > Cc: Christoph Lameter cl@linux.com<mailto:cl@linux.com> > > Cc: Dennis Zhou dennis@kernel.org<mailto:dennis@kernel.org> > > Cc: Gary Guo gary@garyguo.net<mailto:gary@garyguo.net> > > Cc: Miguel Ojeda ojeda@kernel.org<mailto:ojeda@kernel.org> > > Cc: Pasha Tatashin > pasha.tatashin@soleen.com<mailto:pasha.tatashin@soleen.com> > > Cc: Peter Zijlstra peterz@infradead.org<mailto:peterz@infradead.org> > > Cc: Tejun Heo tj@kernel.org<mailto:tj@kernel.org> > > Cc: Vlastimil Babka vbabka@suse.cz<mailto:vbabka@suse.cz> > > Cc: Wedson Almeida Filho > wedsonaf@gmail.com<mailto:wedsonaf@gmail.com> > > Signed-off-by: Andrew Morton > > akpm@linux-foundation.org<mailto:akpm@linux-foundation.org> > > > > > > Ftrace irqoff tracer shows detail: > > At v6.14: > > # tracer: irqsoff > > # > > # irqsoff latency trace v1.1.5 on 6.14.0-rc7-next-20250319 # > > -------------------------------------------------------------------- > > # latency: 279663 us, #21352/21352, CPU#0 | (M:NONE VP:0, KP:0, SP:0 HP:0 > #P:2) > > # ----------------- > > # | task: dd-805 (uid:0 nice:0 policy:0 rt_prio:0) > > # ----------------- > > # => started at: __rmqueue_pcplist > > # => ended at: _raw_spin_unlock_irqrestore > > # > > # > > # _------=> CPU# > > # / _-----=> irqs-off/BH-disabled > > # | / _----=> need-resched > > # || / _---=> hardirq/softirq > > # ||| / _--=> preempt-depth > > # |||| / _-=> migrate-disable > > # ||||| / delay > > # cmd pid |||||| time | caller > > # \ / |||||| \ | / > > dd-805 0d.... 1us : __rmqueue_pcplist > > dd-805 0d.... 3us : _raw_spin_trylock > <-__rmqueue_pcplist > > dd-805 0d.... 7us : __mod_zone_page_state > <-__rmqueue_pcplist > > dd-805 0d.... 10us : __mod_zone_page_state > <-__rmqueue_pcplist > > dd-805 0d.... 12us : __mod_zone_page_state > <-__rmqueue_pcplist > > dd-805 0d.... 15us : __mod_zone_page_state > <-__rmqueue_pcplist > > dd-805 0d.... 17us : __mod_zone_page_state > <-__rmqueue_pcplist > > dd-805 0d.... 19us : __mod_zone_page_state > <-__rmqueue_pcplist > > ... > > dd-805 0d.... 1535us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-805 0d.... 1538us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-805 0d.... 1539us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-805 0d.... 1542us+: try_to_claim_block > <-__rmqueue_pcplist > > dd-805 0d.... 1597us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-805 0d.... 1599us+: try_to_claim_block > <-__rmqueue_pcplist > > dd-805 0d.... 1674us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-805 0d.... 1676us+: try_to_claim_block > <-__rmqueue_pcplist > > dd-805 0d.... 1716us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-805 0d.... 1718us+: try_to_claim_block > <-__rmqueue_pcplist > > dd-805 0d.... 1801us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-805 0d.... 1803us+: try_to_claim_block > <-__rmqueue_pcplist > > ... > > dd-805 0d.... 279555us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-805 0d.... 279556us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-805 0d.... 279558us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-805 0d.... 279560us+: try_to_claim_block > <-__rmqueue_pcplist > > dd-805 0d.... 279616us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-805 0d.... 279618us : __mod_zone_page_state > <-__rmqueue_pcplist > > dd-805 0d.... 279620us : find_suitable_fallback > <-__rmqueue_pcplist > > ... > > dd-805 0d.... 279658us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-805 0d.... 279660us : _raw_spin_unlock_irqrestore > <-__rmqueue_pcplist > > dd-805 0d.... 279662us : _raw_spin_unlock_irqrestore > > dd-805 0d.... 279666us+: trace_hardirqs_on > <-_raw_spin_unlock_irqrestore > > dd-805 0d.... 279712us : <stack trace> > > => get_page_from_freelist > > => __alloc_frozen_pages_noprof > > => __folio_alloc_noprof > > => __filemap_get_folio > > => iomap_write_begin > > => iomap_file_buffered_write > > => blkdev_write_iter > > => vfs_write > > => ksys_write > > => ret_fast_syscall > > > > At v6.7: > > # tracer: irqsoff > > # > > # irqsoff latency trace v1.1.5 on 6.7.0 # > > -------------------------------------------------------------------- > > # latency: 2477 us, #146/146, CPU#0 | (M:server VP:0, KP:0, SP:0 HP:0 #P:2) > > # ----------------- > > # | task: dd-808 (uid:0 nice:0 policy:0 rt_prio:0) > > # ----------------- > > # => started at: _raw_spin_lock_irqsave > > # => ended at: _raw_spin_unlock_irqrestore > > # > > # > > # _------=> CPU# > > # / _-----=> irqs-off/BH-disabled > > # | / _----=> need-resched > > # || / _---=> hardirq/softirq > > # ||| / _--=> preempt-depth > > # |||| / _-=> migrate-disable > > # ||||| / delay > > # cmd pid |||||| time | caller > > # \ / |||||| \ | / > > dd-808 0d.... 1us!: _raw_spin_lock_irqsave > > dd-808 0d.... 186us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 189us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 191us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 192us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 194us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 196us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 199us : steal_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 203us!: move_freepages_block > <-steal_suitable_fallback > > dd-808 0d.... 330us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 332us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 334us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 336us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 338us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 339us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 341us : steal_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 343us!: move_freepages_block > <-steal_suitable_fallback > > dd-808 0d.... 479us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 481us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 483us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 485us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 486us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 488us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 490us : steal_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 492us!: move_freepages_block > <-steal_suitable_fallback > > dd-808 0d.... 630us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 632us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 634us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 636us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 638us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 640us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 642us : steal_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 644us!: move_freepages_block > <-steal_suitable_fallback > > dd-808 0d.... 771us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 773us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 775us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 777us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 778us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 780us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 782us : steal_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 784us!: move_freepages_block > <-steal_suitable_fallback > > dd-808 0d.... 911us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 913us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 915us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 916us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 918us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 920us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 922us : steal_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 924us!: move_freepages_block > <-steal_suitable_fallback > > dd-808 0d.... 1055us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1058us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1059us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1061us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1063us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1065us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1066us : steal_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1068us!: move_freepages_block > <-steal_suitable_fallback > > dd-808 0d.... 1194us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1196us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1198us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1200us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1202us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1203us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1205us : steal_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1208us!: move_freepages_block > <-steal_suitable_fallback > > dd-808 0d.... 1333us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1335us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1337us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1339us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1341us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1342us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1344us : steal_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1346us!: move_freepages_block > <-steal_suitable_fallback > > dd-808 0d.... 1480us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1482us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1484us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1486us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1488us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1490us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1492us : steal_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1494us!: move_freepages_block > <-steal_suitable_fallback > > dd-808 0d.... 1621us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1623us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1625us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1627us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1629us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1630us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1632us : steal_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1634us!: move_freepages_block > <-steal_suitable_fallback > > dd-808 0d.... 1761us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1763us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1765us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1766us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1768us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1770us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1772us : steal_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1774us!: move_freepages_block > <-steal_suitable_fallback > > dd-808 0d.... 1900us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1902us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1903us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1905us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1907us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1909us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1911us : steal_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 1913us!: move_freepages_block > <-steal_suitable_fallback > > dd-808 0d.... 2038us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2040us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2042us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2044us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2046us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2047us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2049us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2051us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2053us : steal_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2055us!: move_freepages_block > <-steal_suitable_fallback > > dd-808 0d.... 2175us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2176us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2178us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2180us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2182us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2183us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2185us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2187us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2189us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2191us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2192us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2194us : steal_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2196us!: move_freepages_block > <-steal_suitable_fallback > > dd-808 0d.... 2323us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2325us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2327us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2328us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2330us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2332us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2334us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2335us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2337us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2339us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2341us : find_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2343us : steal_suitable_fallback > <-__rmqueue_pcplist > > dd-808 0d.... 2345us!: move_freepages_block > <-steal_suitable_fallback > > dd-808 0d.... 2470us : __mod_zone_page_state > <-__rmqueue_pcplist > > dd-808 0d.... 2473us : _raw_spin_unlock_irqrestore > <-__rmqueue_pcplist > > dd-808 0d.... 2476us : _raw_spin_unlock_irqrestore > > dd-808 0d.... 2479us+: tracer_hardirqs_on > <-_raw_spin_unlock_irqrestore > > dd-808 0d.... 2520us : <stack trace> > > => get_page_from_freelist > > => __alloc_pages > > => __folio_alloc > > => __filemap_get_folio > > => iomap_write_begin > > => iomap_file_buffered_write > > => blkdev_write_iter > > => vfs_write > > => ksys_write > > => ret_fast_syscall > > > > Best Regard > > Carlos Song > > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Ask help about this patch b951aaff5035 " mm: enable page allocation tagging" 2025-03-27 8:25 ` Carlos Song @ 2025-03-27 11:02 ` Carlos Song 2025-03-31 20:01 ` Suren Baghdasaryan 0 siblings, 1 reply; 8+ messages in thread From: Carlos Song @ 2025-03-27 11:02 UTC (permalink / raw) To: Kent Overstreet Cc: surenb, akpm, willy, linux-mm, linux-kernel, linux-fsdevel Hi, I get the function calling step log, I hope it can help debug little. The spinlock_irqsave from 136.725021 to 136.966962_raw_spin_unlock_irqrestore. Around 260ms? It is Just one case, sometimes it will spend more time. dd-828 [000] ..... 136.724980: fdget_pos <-ksys_write dd-828 [000] ..... 136.724982: vfs_write <-ksys_write dd-828 [000] ..... 136.724983: blkdev_write_iter <-vfs_write dd-828 [000] ..... 136.724984: I_BDEV <-blkdev_write_iter dd-828 [000] ..... 136.724985: file_update_time <-blkdev_write_iter dd-828 [000] ..... 136.724986: inode_needs_update_time <-file_update_time dd-828 [000] ..... 136.724987: ktime_get_coarse_real_ts64 <-inode_needs_update_time dd-828 [000] ..... 136.724989: timestamp_truncate <-inode_needs_update_time dd-828 [000] ..... 136.724990: iomap_file_buffered_write <-blkdev_write_iter dd-828 [000] ..... 136.724992: iomap_iter <-iomap_file_buffered_write dd-828 [000] ..... 136.724993: blkdev_iomap_begin <-iomap_iter dd-828 [000] ..... 136.724994: I_BDEV <-blkdev_iomap_begin dd-828 [000] ..... 136.724995: balance_dirty_pages_ratelimited_flags <-iomap_file_buffered_write dd-828 [000] ..... 136.724996: inode_to_bdi <-balance_dirty_pages_ratelimited_flags dd-828 [000] ..... 136.724997: I_BDEV <-inode_to_bdi dd-828 [000] ..... 136.724998: preempt_count_add <-balance_dirty_pages_ratelimited_flags dd-828 [000] ...1. 136.725000: preempt_count_sub <-balance_dirty_pages_ratelimited_flags dd-828 [000] ..... 136.725001: fault_in_readable <-fault_in_iov_iter_readable dd-828 [000] ..... 136.725003: iomap_write_begin <-iomap_file_buffered_write dd-828 [000] ..... 136.725004: iomap_get_folio <-iomap_write_begin dd-828 [000] ..... 136.725005: __filemap_get_folio <-iomap_write_begin dd-828 [000] ..... 136.725006: filemap_get_entry <-__filemap_get_folio dd-828 [000] ..... 136.725007: __rcu_read_lock <-filemap_get_entry dd-828 [000] ..... 136.725008: __rcu_read_unlock <-filemap_get_entry dd-828 [000] ..... 136.725009: inode_to_bdi <-__filemap_get_folio dd-828 [000] ..... 136.725010: I_BDEV <-inode_to_bdi dd-828 [000] ..... 136.725012: __folio_alloc_noprof <-__filemap_get_folio dd-828 [000] ..... 136.725013: __alloc_pages_noprof <-__folio_alloc_noprof dd-828 [000] ..... 136.725014: get_page_from_freelist <-__alloc_pages_noprof dd-828 [000] ..... 136.725015: node_dirty_ok <-get_page_from_freelist dd-828 [000] ..... 136.725016: preempt_count_add <-get_page_from_freelist dd-828 [000] ...1. 136.725018: _raw_spin_trylock <-get_page_from_freelist dd-828 [000] ...1. 136.725019: preempt_count_add <-_raw_spin_trylock dd-828 [000] ...2. 136.725021: _raw_spin_lock_irqsave <-__rmqueue_pcplist dd-828 [000] d..2. 136.725022: preempt_count_add <-_raw_spin_lock_irqsave dd-828 [000] d..3. 136.725025: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725028: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725029: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725031: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725032: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725034: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725036: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725037: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725038: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725039: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725040: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725042: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725043: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725044: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725046: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725047: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725048: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725050: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725051: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725052: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725053: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725055: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725056: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725057: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725058: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725060: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725061: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725062: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725063: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725065: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725066: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725067: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725068: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725070: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725071: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725072: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725073: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725075: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725076: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725077: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725079: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725080: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725081: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725082: __mod_zone_page_state <-__rmqueue_pcplist dd-828 [000] d..3. 136.725083: __mod_zone_page_state <-__rmqueue_pcplist ... dd-828 [000] d..3. 136.820279: find_suitable_fallback <-__rmqueue_pcplist dd-828 [000] d..3. 136.820280: find_suitable_fallback <-__rmqueue_pcplist dd-828 [000] d..3. 136.820281: find_suitable_fallback <-__rmqueue_pcplist dd-828 [000] d..3. 136.820282: find_suitable_fallback <-__rmqueue_pcplist dd-828 [000] d..3. 136.820283: find_suitable_fallback <-__rmqueue_pcplist dd-828 [000] d..3. 136.820284: find_suitable_fallback <-__rmqueue_pcplist dd-828 [000] d..3. 136.820285: find_suitable_fallback <-__rmqueue_pcplist dd-828 [000] d..3. 136.820286: find_suitable_fallback <-__rmqueue_pcplist dd-828 [000] d..3. 136.820287: find_suitable_fallback <-__rmqueue_pcplist dd-828 [000] d..3. 136.820289: find_suitable_fallback <-__rmqueue_pcplist dd-828 [000] d..3. 136.820290: steal_suitable_fallback <-__rmqueue_pcplist dd-828 [000] d..3. 136.820525: __mod_zone_page_state <-steal_suitable_fallback ... dd-828 [000] d..3. 136.966945: __mod_zone_page_state <-steal_suitable_fallback dd-828 [000] d..3. 136.966946: find_suitable_fallback <-__rmqueue_pcplist dd-828 [000] d..3. 136.966947: find_suitable_fallback <-__rmqueue_pcplist dd-828 [000] d..3. 136.966948: find_suitable_fallback <-__rmqueue_pcplist dd-828 [000] d..3. 136.966950: find_suitable_fallback <-__rmqueue_pcplist dd-828 [000] d..3. 136.966951: find_suitable_fallback <-__rmqueue_pcplist dd-828 [000] d..3. 136.966952: find_suitable_fallback <-__rmqueue_pcplist dd-828 [000] d..3. 136.966953: find_suitable_fallback <-__rmqueue_pcplist dd-828 [000] d..3. 136.966954: find_suitable_fallback <-__rmqueue_pcplist dd-828 [000] d..3. 136.966955: find_suitable_fallback <-__rmqueue_pcplist dd-828 [000] d..3. 136.966956: find_suitable_fallback <-__rmqueue_pcplist dd-828 [000] d..3. 136.966957: find_suitable_fallback <-__rmqueue_pcplist dd-828 [000] d..3. 136.966958: find_suitable_fallback <-__rmqueue_pcplist dd-828 [000] d..3. 136.966959: find_suitable_fallback <-__rmqueue_pcplist dd-828 [000] d..3. 136.966961: find_suitable_fallback <-__rmqueue_pcplist dd-828 [000] d..3. 136.966962: _raw_spin_unlock_irqrestore <-__rmqueue_pcplist dd-828 [000] d..3. 136.966965: irq_enter <-generic_handle_arch_irq dd-828 [000] d..3. 136.966966: irq_enter_rcu <-generic_handle_arch_irq dd-828 [000] d..3. 136.966967: preempt_count_add <-irq_enter_rcu dd-828 [000] d.h3. 136.966969: gic_handle_irq <-generic_handle_arch_irq dd-828 [000] d.h3. 136.966970: generic_handle_domain_irq <-gic_handle_irq dd-828 [000] d.h3. 136.966971: __irq_resolve_mapping <-generic_handle_domain_irq dd-828 [000] d.h3. 136.966972: __rcu_read_lock <-__irq_resolve_mapping dd-828 [000] d.h3. 136.966974: __rcu_read_unlock <-__irq_resolve_mapping dd-828 [000] d.h3. 136.966975: handle_irq_desc <-gic_handle_irq dd-828 [000] d.h3. 136.966976: handle_percpu_devid_irq <-handle_irq_desc dd-828 [000] d.h3. 136.966978: ipi_handler <-handle_percpu_devid_irq dd-828 [000] d.h3. 136.966979: do_handle_IPI <-ipi_handler dd-828 [000] d.h3. 136.966981: generic_smp_call_function_single_interrupt <-do_handle_IPI dd-828 [000] d.h3. 136.966982: __flush_smp_call_function_queue <-do_handle_IPI dd-828 [000] d.h3. 136.966986: sched_ttwu_pending <-__flush_smp_call_function_queue dd-828 [000] d.h3. 136.966987: preempt_count_add <-sched_ttwu_pending dd-828 [000] d.h4. 136.966990: _raw_spin_lock <-sched_ttwu_pending dd-828 [000] d.h4. 136.966991: preempt_count_add <-_raw_spin_lock dd-828 [000] d.h5. 136.966993: preempt_count_sub <-sched_ttwu_pending dd-828 [000] d.h4. 136.966994: update_rq_clock.part.0 <-sched_ttwu_pending dd-828 [000] d.h4. 136.966996: ttwu_do_activate <-sched_ttwu_pending dd-828 [000] d.h4. 136.966997: activate_task <-ttwu_do_activate dd-828 [000] d.h4. 136.966998: enqueue_task_fair <-activate_task dd-828 [000] d.h4. 136.967000: update_curr <-enqueue_task_fair dd-828 [000] d.h4. 136.967001: update_min_vruntime <-update_curr dd-828 [000] d.h4. 136.967003: __cgroup_account_cputime <-update_curr BR Carlos Song > > On Thu, Mar 20, 2025 at 11:07:41AM +0000, Carlos Song wrote: > > > Hi, all > > > > > > I found a 300ms~600ms IRQ off when writing 1Gb data to mmc device at > > I.MX7d SDB board at Linux-kernel-v6.14. > > > But I test the same case at Linux-kernel-v6.7, this longest IRQ off > > > time is only > > 1ms~2ms. So the issue is introduced from v6.7~v6.14. > > > > > > Run this cmd to test: > > > dd if=/dev/zero of=/dev/mmcblk2p4 bs=4096 seek=12500 count=256000 > > > conv=fsync > > > > > > This issue looks from blkdev_buffered_write() function. Because when > > > I run this cmd with "oflag=direct" to use blkdev_direct_write(), I > > > can not see > > any long time IRQ off. > > > > > > Then I use Ftrace irqoff tracer to trace the longest IRQ off event, > > > I found some > > differences between v6.7 and v6.14: > > > In iomap_file_buffered_write(), __folio_alloc (in v6.7) is replaced > > > by > > _folio_alloc_noprof (in v6.14) by this patch. > > > The spinlock disabled IRQ ~300ms+. It looks there are some fixes for > > > this patch, > > but I still can see IRQ off 300ms+ at 6.14.0-rc7-next-20250319. > > > > > > Do I trigger one bug? I know little about mem so I have to report it > > > and hope I > > can get some help or guide. > > > I put my ftrace log at the mail tail to help trace and explain. > > > > Did you track down which spinlock? > > > > > > > Hi, > > Sorry for my late reply and thank you for your quick reply! > From the trace log, I think the spinlock is from here like this: > > __alloc_frozen_pages_noprof ==> get_page_from_freelist==>spinclok > > Do you need other log not only this? > > > > => get_page_from_freelist" > > > => __alloc_frozen_pages_noprof > > > => __folio_alloc_noprof > > > => __filemap_get_folio > > > > > commit b951aaff503502a7fe066eeed2744ba8a6413c89 > > > Author: Suren Baghdasaryan > > surenb@google.com<mailto:surenb@google.com> > > > Date: Thu Mar 21 09:36:40 2024 -0700 > > > > > > mm: enable page allocation tagging > > > > > > Redefine page allocators to record allocation tags upon their > invocation. > > > Instrument post_alloc_hook and free_pages_prepare to modify > current > > > allocation tag. > > > > > > [surenb@google.com: undo _noprof additions in the documentation] > > > Link: > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flkml > > .kern > > > el.org%2Fr%2F20240326231453.1206227-3-surenb%40google.com&data=05% > > > 7C02%7Ccarlos.song%40nxp.com%7Cdf027bbb97074fc2cde808dd67a1d6c5%7 > > > C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C638780666983046738% > > > 7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDA > > > wMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C > > > &sdata=Aby13KyQjF5pcbYW%2BkfsEJLiaPmS2ZiJLUHJ%2BCr2JXM%3D&reserve > > d=0 > > > Link: > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flkml > > .kern > > > el.org%2Fr%2F20240321163705.3067592-19-surenb%40google.com&data=05 > > %7C02%7Ccarlos.song%40nxp.com%7Cdf027bbb97074fc2cde808dd67a1d6c5 > % > > > 7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C638780666983065790 > > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuM > D > > > AwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7 > > > C&sdata=n0hjs2fhjYs%2BcnbrxHy4vFK6D4GFVL4%2Fu72anOOLiEI%3D&reserve > > d=0 > > > Signed-off-by: Suren Baghdasaryan > > surenb@google.com<mailto:surenb@google.com> > > > Co-developed-by: Kent Overstreet > > kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev> > > > Signed-off-by: Kent Overstreet > > kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev> > > > Reviewed-by: Kees Cook > > keescook@chromium.org<mailto:keescook@chromium.org> > > > Tested-by: Kees Cook > > keescook@chromium.org<mailto:keescook@chromium.org> > > > Cc: Alexander Viro > > viro@zeniv.linux.org.uk<mailto:viro@zeniv.linux.org.uk> > > > Cc: Alex Gaynor > > alex.gaynor@gmail.com<mailto:alex.gaynor@gmail.com> > > > Cc: Alice Ryhl aliceryhl@google.com<mailto:aliceryhl@google.com> > > > Cc: Andreas Hindborg > > a.hindborg@samsung.com<mailto:a.hindborg@samsung.com> > > > Cc: Benno Lossin > > benno.lossin@proton.me<mailto:benno.lossin@proton.me> > > > Cc: "Björn Roy Baron" > > bjorn3_gh@protonmail.com<mailto:bjorn3_gh@protonmail.com> > > > Cc: Boqun Feng > > boqun.feng@gmail.com<mailto:boqun.feng@gmail.com> > > > Cc: Christoph Lameter cl@linux.com<mailto:cl@linux.com> > > > Cc: Dennis Zhou dennis@kernel.org<mailto:dennis@kernel.org> > > > Cc: Gary Guo gary@garyguo.net<mailto:gary@garyguo.net> > > > Cc: Miguel Ojeda ojeda@kernel.org<mailto:ojeda@kernel.org> > > > Cc: Pasha Tatashin > > pasha.tatashin@soleen.com<mailto:pasha.tatashin@soleen.com> > > > Cc: Peter Zijlstra > peterz@infradead.org<mailto:peterz@infradead.org> > > > Cc: Tejun Heo tj@kernel.org<mailto:tj@kernel.org> > > > Cc: Vlastimil Babka vbabka@suse.cz<mailto:vbabka@suse.cz> > > > Cc: Wedson Almeida Filho > > wedsonaf@gmail.com<mailto:wedsonaf@gmail.com> > > > Signed-off-by: Andrew Morton > > > akpm@linux-foundation.org<mailto:akpm@linux-foundation.org> > > > > > > > > > Ftrace irqoff tracer shows detail: > > > At v6.14: > > > # tracer: irqsoff > > > # > > > # irqsoff latency trace v1.1.5 on 6.14.0-rc7-next-20250319 # > > > -------------------------------------------------------------------- > > > # latency: 279663 us, #21352/21352, CPU#0 | (M:NONE VP:0, KP:0, SP:0 > > > HP:0 > > #P:2) > > > # ----------------- > > > # | task: dd-805 (uid:0 nice:0 policy:0 rt_prio:0) > > > # ----------------- > > > # => started at: __rmqueue_pcplist > > > # => ended at: _raw_spin_unlock_irqrestore > > > # > > > # > > > # _------=> CPU# > > > # / _-----=> irqs-off/BH-disabled > > > # | / _----=> need-resched > > > # || / _---=> hardirq/softirq > > > # ||| / _--=> preempt-depth > > > # |||| / _-=> migrate-disable > > > # ||||| / delay > > > # cmd pid |||||| time | caller > > > # \ / |||||| \ | / > > > dd-805 0d.... 1us : __rmqueue_pcplist > > > dd-805 0d.... 3us : _raw_spin_trylock > > <-__rmqueue_pcplist > > > dd-805 0d.... 7us : __mod_zone_page_state > > <-__rmqueue_pcplist > > > dd-805 0d.... 10us : __mod_zone_page_state > > <-__rmqueue_pcplist > > > dd-805 0d.... 12us : __mod_zone_page_state > > <-__rmqueue_pcplist > > > dd-805 0d.... 15us : __mod_zone_page_state > > <-__rmqueue_pcplist > > > dd-805 0d.... 17us : __mod_zone_page_state > > <-__rmqueue_pcplist > > > dd-805 0d.... 19us : __mod_zone_page_state > > <-__rmqueue_pcplist > > > ... > > > dd-805 0d.... 1535us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-805 0d.... 1538us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-805 0d.... 1539us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-805 0d.... 1542us+: try_to_claim_block > > <-__rmqueue_pcplist > > > dd-805 0d.... 1597us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-805 0d.... 1599us+: try_to_claim_block > > <-__rmqueue_pcplist > > > dd-805 0d.... 1674us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-805 0d.... 1676us+: try_to_claim_block > > <-__rmqueue_pcplist > > > dd-805 0d.... 1716us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-805 0d.... 1718us+: try_to_claim_block > > <-__rmqueue_pcplist > > > dd-805 0d.... 1801us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-805 0d.... 1803us+: try_to_claim_block > > <-__rmqueue_pcplist > > > ... > > > dd-805 0d.... 279555us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-805 0d.... 279556us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-805 0d.... 279558us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-805 0d.... 279560us+: try_to_claim_block > > <-__rmqueue_pcplist > > > dd-805 0d.... 279616us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-805 0d.... 279618us : __mod_zone_page_state > > <-__rmqueue_pcplist > > > dd-805 0d.... 279620us : find_suitable_fallback > > <-__rmqueue_pcplist > > > ... > > > dd-805 0d.... 279658us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-805 0d.... 279660us : _raw_spin_unlock_irqrestore > > <-__rmqueue_pcplist > > > dd-805 0d.... 279662us : _raw_spin_unlock_irqrestore > > > dd-805 0d.... 279666us+: trace_hardirqs_on > > <-_raw_spin_unlock_irqrestore > > > dd-805 0d.... 279712us : <stack trace> > > > => get_page_from_freelist > > > => __alloc_frozen_pages_noprof > > > => __folio_alloc_noprof > > > => __filemap_get_folio > > > => iomap_write_begin > > > => iomap_file_buffered_write > > > => blkdev_write_iter > > > => vfs_write > > > => ksys_write > > > => ret_fast_syscall > > > > > > At v6.7: > > > # tracer: irqsoff > > > # > > > # irqsoff latency trace v1.1.5 on 6.7.0 # > > > -------------------------------------------------------------------- > > > # latency: 2477 us, #146/146, CPU#0 | (M:server VP:0, KP:0, SP:0 HP:0 #P:2) > > > # ----------------- > > > # | task: dd-808 (uid:0 nice:0 policy:0 rt_prio:0) > > > # ----------------- > > > # => started at: _raw_spin_lock_irqsave > > > # => ended at: _raw_spin_unlock_irqrestore > > > # > > > # > > > # _------=> CPU# > > > # / _-----=> irqs-off/BH-disabled > > > # | / _----=> need-resched > > > # || / _---=> hardirq/softirq > > > # ||| / _--=> preempt-depth > > > # |||| / _-=> migrate-disable > > > # ||||| / delay > > > # cmd pid |||||| time | caller > > > # \ / |||||| \ | / > > > dd-808 0d.... 1us!: _raw_spin_lock_irqsave > > > dd-808 0d.... 186us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 189us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 191us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 192us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 194us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 196us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 199us : steal_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 203us!: move_freepages_block > > <-steal_suitable_fallback > > > dd-808 0d.... 330us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 332us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 334us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 336us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 338us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 339us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 341us : steal_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 343us!: move_freepages_block > > <-steal_suitable_fallback > > > dd-808 0d.... 479us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 481us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 483us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 485us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 486us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 488us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 490us : steal_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 492us!: move_freepages_block > > <-steal_suitable_fallback > > > dd-808 0d.... 630us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 632us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 634us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 636us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 638us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 640us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 642us : steal_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 644us!: move_freepages_block > > <-steal_suitable_fallback > > > dd-808 0d.... 771us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 773us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 775us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 777us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 778us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 780us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 782us : steal_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 784us!: move_freepages_block > > <-steal_suitable_fallback > > > dd-808 0d.... 911us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 913us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 915us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 916us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 918us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 920us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 922us : steal_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 924us!: move_freepages_block > > <-steal_suitable_fallback > > > dd-808 0d.... 1055us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1058us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1059us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1061us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1063us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1065us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1066us : steal_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1068us!: move_freepages_block > > <-steal_suitable_fallback > > > dd-808 0d.... 1194us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1196us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1198us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1200us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1202us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1203us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1205us : steal_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1208us!: move_freepages_block > > <-steal_suitable_fallback > > > dd-808 0d.... 1333us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1335us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1337us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1339us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1341us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1342us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1344us : steal_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1346us!: move_freepages_block > > <-steal_suitable_fallback > > > dd-808 0d.... 1480us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1482us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1484us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1486us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1488us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1490us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1492us : steal_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1494us!: move_freepages_block > > <-steal_suitable_fallback > > > dd-808 0d.... 1621us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1623us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1625us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1627us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1629us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1630us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1632us : steal_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1634us!: move_freepages_block > > <-steal_suitable_fallback > > > dd-808 0d.... 1761us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1763us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1765us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1766us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1768us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1770us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1772us : steal_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1774us!: move_freepages_block > > <-steal_suitable_fallback > > > dd-808 0d.... 1900us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1902us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1903us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1905us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1907us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1909us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1911us : steal_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 1913us!: move_freepages_block > > <-steal_suitable_fallback > > > dd-808 0d.... 2038us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2040us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2042us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2044us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2046us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2047us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2049us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2051us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2053us : steal_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2055us!: move_freepages_block > > <-steal_suitable_fallback > > > dd-808 0d.... 2175us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2176us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2178us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2180us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2182us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2183us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2185us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2187us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2189us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2191us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2192us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2194us : steal_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2196us!: move_freepages_block > > <-steal_suitable_fallback > > > dd-808 0d.... 2323us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2325us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2327us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2328us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2330us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2332us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2334us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2335us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2337us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2339us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2341us : find_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2343us : steal_suitable_fallback > > <-__rmqueue_pcplist > > > dd-808 0d.... 2345us!: move_freepages_block > > <-steal_suitable_fallback > > > dd-808 0d.... 2470us : __mod_zone_page_state > > <-__rmqueue_pcplist > > > dd-808 0d.... 2473us : _raw_spin_unlock_irqrestore > > <-__rmqueue_pcplist > > > dd-808 0d.... 2476us : _raw_spin_unlock_irqrestore > > > dd-808 0d.... 2479us+: tracer_hardirqs_on > > <-_raw_spin_unlock_irqrestore > > > dd-808 0d.... 2520us : <stack trace> > > > => get_page_from_freelist > > > => __alloc_pages > > > => __folio_alloc > > > => __filemap_get_folio > > > => iomap_write_begin > > > => iomap_file_buffered_write > > > => blkdev_write_iter > > > => vfs_write > > > => ksys_write > > > => ret_fast_syscall > > > > > > Best Regard > > > Carlos Song > > > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Ask help about this patch b951aaff5035 " mm: enable page allocation tagging" 2025-03-27 11:02 ` Carlos Song @ 2025-03-31 20:01 ` Suren Baghdasaryan 2025-04-02 2:30 ` Carlos Song 0 siblings, 1 reply; 8+ messages in thread From: Suren Baghdasaryan @ 2025-03-31 20:01 UTC (permalink / raw) To: Carlos Song Cc: Kent Overstreet, akpm, willy, linux-mm, linux-kernel, linux-fsdevel On Thu, Mar 27, 2025 at 4:02 AM Carlos Song <carlos.song@nxp.com> wrote: > > Hi, > > I get the function calling step log, I hope it can help debug little. Hi Carlos, Since you do not enable CONFIG_MEM_ALLOC_PROFILING, "mm: enable page allocation tagging" patch would most definitely not cause this issue. From the traces you posted I think the spinlock involved here is zone->lock inside rmqueue_bulk(): https://elixir.bootlin.com/linux/v6.14-rc6/source/mm/page_alloc.c#L2310 I quickly scanned the code within that loop comparing 6.7 and 6.14 and while I did not find anything obvious, steal_suitable_fallback() function seems quite different. I would suggest profiling this path to see which part might have regressed and go from there. BTW, are you using pure upstream code or it has some out-of-tree modifications? Thanks, Suren. > > The spinlock_irqsave from 136.725021 to 136.966962_raw_spin_unlock_irqrestore. > Around 260ms? It is Just one case, sometimes it will spend more time. > > dd-828 [000] ..... 136.724980: fdget_pos <-ksys_write > dd-828 [000] ..... 136.724982: vfs_write <-ksys_write > dd-828 [000] ..... 136.724983: blkdev_write_iter <-vfs_write > dd-828 [000] ..... 136.724984: I_BDEV <-blkdev_write_iter > dd-828 [000] ..... 136.724985: file_update_time <-blkdev_write_iter > dd-828 [000] ..... 136.724986: inode_needs_update_time <-file_update_time > dd-828 [000] ..... 136.724987: ktime_get_coarse_real_ts64 <-inode_needs_update_time > dd-828 [000] ..... 136.724989: timestamp_truncate <-inode_needs_update_time > dd-828 [000] ..... 136.724990: iomap_file_buffered_write <-blkdev_write_iter > dd-828 [000] ..... 136.724992: iomap_iter <-iomap_file_buffered_write > dd-828 [000] ..... 136.724993: blkdev_iomap_begin <-iomap_iter > dd-828 [000] ..... 136.724994: I_BDEV <-blkdev_iomap_begin > dd-828 [000] ..... 136.724995: balance_dirty_pages_ratelimited_flags <-iomap_file_buffered_write > dd-828 [000] ..... 136.724996: inode_to_bdi <-balance_dirty_pages_ratelimited_flags > dd-828 [000] ..... 136.724997: I_BDEV <-inode_to_bdi > dd-828 [000] ..... 136.724998: preempt_count_add <-balance_dirty_pages_ratelimited_flags > dd-828 [000] ...1. 136.725000: preempt_count_sub <-balance_dirty_pages_ratelimited_flags > dd-828 [000] ..... 136.725001: fault_in_readable <-fault_in_iov_iter_readable > dd-828 [000] ..... 136.725003: iomap_write_begin <-iomap_file_buffered_write > dd-828 [000] ..... 136.725004: iomap_get_folio <-iomap_write_begin > dd-828 [000] ..... 136.725005: __filemap_get_folio <-iomap_write_begin > dd-828 [000] ..... 136.725006: filemap_get_entry <-__filemap_get_folio > dd-828 [000] ..... 136.725007: __rcu_read_lock <-filemap_get_entry > dd-828 [000] ..... 136.725008: __rcu_read_unlock <-filemap_get_entry > dd-828 [000] ..... 136.725009: inode_to_bdi <-__filemap_get_folio > dd-828 [000] ..... 136.725010: I_BDEV <-inode_to_bdi > dd-828 [000] ..... 136.725012: __folio_alloc_noprof <-__filemap_get_folio > dd-828 [000] ..... 136.725013: __alloc_pages_noprof <-__folio_alloc_noprof > dd-828 [000] ..... 136.725014: get_page_from_freelist <-__alloc_pages_noprof > dd-828 [000] ..... 136.725015: node_dirty_ok <-get_page_from_freelist > dd-828 [000] ..... 136.725016: preempt_count_add <-get_page_from_freelist > dd-828 [000] ...1. 136.725018: _raw_spin_trylock <-get_page_from_freelist > dd-828 [000] ...1. 136.725019: preempt_count_add <-_raw_spin_trylock > dd-828 [000] ...2. 136.725021: _raw_spin_lock_irqsave <-__rmqueue_pcplist > dd-828 [000] d..2. 136.725022: preempt_count_add <-_raw_spin_lock_irqsave > dd-828 [000] d..3. 136.725025: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725028: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725029: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725031: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725032: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725034: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725036: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725037: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725038: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725039: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725040: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725042: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725043: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725044: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725046: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725047: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725048: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725050: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725051: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725052: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725053: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725055: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725056: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725057: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725058: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725060: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725061: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725062: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725063: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725065: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725066: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725067: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725068: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725070: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725071: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725072: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725073: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725075: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725076: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725077: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725079: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725080: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725081: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725082: __mod_zone_page_state <-__rmqueue_pcplist > dd-828 [000] d..3. 136.725083: __mod_zone_page_state <-__rmqueue_pcplist > ... > dd-828 [000] d..3. 136.820279: find_suitable_fallback <-__rmqueue_pcplist > dd-828 [000] d..3. 136.820280: find_suitable_fallback <-__rmqueue_pcplist > dd-828 [000] d..3. 136.820281: find_suitable_fallback <-__rmqueue_pcplist > dd-828 [000] d..3. 136.820282: find_suitable_fallback <-__rmqueue_pcplist > dd-828 [000] d..3. 136.820283: find_suitable_fallback <-__rmqueue_pcplist > dd-828 [000] d..3. 136.820284: find_suitable_fallback <-__rmqueue_pcplist > dd-828 [000] d..3. 136.820285: find_suitable_fallback <-__rmqueue_pcplist > dd-828 [000] d..3. 136.820286: find_suitable_fallback <-__rmqueue_pcplist > dd-828 [000] d..3. 136.820287: find_suitable_fallback <-__rmqueue_pcplist > dd-828 [000] d..3. 136.820289: find_suitable_fallback <-__rmqueue_pcplist > dd-828 [000] d..3. 136.820290: steal_suitable_fallback <-__rmqueue_pcplist > dd-828 [000] d..3. 136.820525: __mod_zone_page_state <-steal_suitable_fallback > ... > dd-828 [000] d..3. 136.966945: __mod_zone_page_state <-steal_suitable_fallback > dd-828 [000] d..3. 136.966946: find_suitable_fallback <-__rmqueue_pcplist > dd-828 [000] d..3. 136.966947: find_suitable_fallback <-__rmqueue_pcplist > dd-828 [000] d..3. 136.966948: find_suitable_fallback <-__rmqueue_pcplist > dd-828 [000] d..3. 136.966950: find_suitable_fallback <-__rmqueue_pcplist > dd-828 [000] d..3. 136.966951: find_suitable_fallback <-__rmqueue_pcplist > dd-828 [000] d..3. 136.966952: find_suitable_fallback <-__rmqueue_pcplist > dd-828 [000] d..3. 136.966953: find_suitable_fallback <-__rmqueue_pcplist > dd-828 [000] d..3. 136.966954: find_suitable_fallback <-__rmqueue_pcplist > dd-828 [000] d..3. 136.966955: find_suitable_fallback <-__rmqueue_pcplist > dd-828 [000] d..3. 136.966956: find_suitable_fallback <-__rmqueue_pcplist > dd-828 [000] d..3. 136.966957: find_suitable_fallback <-__rmqueue_pcplist > dd-828 [000] d..3. 136.966958: find_suitable_fallback <-__rmqueue_pcplist > dd-828 [000] d..3. 136.966959: find_suitable_fallback <-__rmqueue_pcplist > dd-828 [000] d..3. 136.966961: find_suitable_fallback <-__rmqueue_pcplist > dd-828 [000] d..3. 136.966962: _raw_spin_unlock_irqrestore <-__rmqueue_pcplist > dd-828 [000] d..3. 136.966965: irq_enter <-generic_handle_arch_irq > dd-828 [000] d..3. 136.966966: irq_enter_rcu <-generic_handle_arch_irq > dd-828 [000] d..3. 136.966967: preempt_count_add <-irq_enter_rcu > dd-828 [000] d.h3. 136.966969: gic_handle_irq <-generic_handle_arch_irq > dd-828 [000] d.h3. 136.966970: generic_handle_domain_irq <-gic_handle_irq > dd-828 [000] d.h3. 136.966971: __irq_resolve_mapping <-generic_handle_domain_irq > dd-828 [000] d.h3. 136.966972: __rcu_read_lock <-__irq_resolve_mapping > dd-828 [000] d.h3. 136.966974: __rcu_read_unlock <-__irq_resolve_mapping > dd-828 [000] d.h3. 136.966975: handle_irq_desc <-gic_handle_irq > dd-828 [000] d.h3. 136.966976: handle_percpu_devid_irq <-handle_irq_desc > dd-828 [000] d.h3. 136.966978: ipi_handler <-handle_percpu_devid_irq > dd-828 [000] d.h3. 136.966979: do_handle_IPI <-ipi_handler > dd-828 [000] d.h3. 136.966981: generic_smp_call_function_single_interrupt <-do_handle_IPI > dd-828 [000] d.h3. 136.966982: __flush_smp_call_function_queue <-do_handle_IPI > dd-828 [000] d.h3. 136.966986: sched_ttwu_pending <-__flush_smp_call_function_queue > dd-828 [000] d.h3. 136.966987: preempt_count_add <-sched_ttwu_pending > dd-828 [000] d.h4. 136.966990: _raw_spin_lock <-sched_ttwu_pending > dd-828 [000] d.h4. 136.966991: preempt_count_add <-_raw_spin_lock > dd-828 [000] d.h5. 136.966993: preempt_count_sub <-sched_ttwu_pending > dd-828 [000] d.h4. 136.966994: update_rq_clock.part.0 <-sched_ttwu_pending > dd-828 [000] d.h4. 136.966996: ttwu_do_activate <-sched_ttwu_pending > dd-828 [000] d.h4. 136.966997: activate_task <-ttwu_do_activate > dd-828 [000] d.h4. 136.966998: enqueue_task_fair <-activate_task > dd-828 [000] d.h4. 136.967000: update_curr <-enqueue_task_fair > dd-828 [000] d.h4. 136.967001: update_min_vruntime <-update_curr > dd-828 [000] d.h4. 136.967003: __cgroup_account_cputime <-update_curr > > BR > Carlos Song > > > > On Thu, Mar 20, 2025 at 11:07:41AM +0000, Carlos Song wrote: > > > > Hi, all > > > > > > > > I found a 300ms~600ms IRQ off when writing 1Gb data to mmc device at > > > I.MX7d SDB board at Linux-kernel-v6.14. > > > > But I test the same case at Linux-kernel-v6.7, this longest IRQ off > > > > time is only > > > 1ms~2ms. So the issue is introduced from v6.7~v6.14. > > > > > > > > Run this cmd to test: > > > > dd if=/dev/zero of=/dev/mmcblk2p4 bs=4096 seek=12500 count=256000 > > > > conv=fsync > > > > > > > > This issue looks from blkdev_buffered_write() function. Because when > > > > I run this cmd with "oflag=direct" to use blkdev_direct_write(), I > > > > can not see > > > any long time IRQ off. > > > > > > > > Then I use Ftrace irqoff tracer to trace the longest IRQ off event, > > > > I found some > > > differences between v6.7 and v6.14: > > > > In iomap_file_buffered_write(), __folio_alloc (in v6.7) is replaced > > > > by > > > _folio_alloc_noprof (in v6.14) by this patch. > > > > The spinlock disabled IRQ ~300ms+. It looks there are some fixes for > > > > this patch, > > > but I still can see IRQ off 300ms+ at 6.14.0-rc7-next-20250319. > > > > > > > > Do I trigger one bug? I know little about mem so I have to report it > > > > and hope I > > > can get some help or guide. > > > > I put my ftrace log at the mail tail to help trace and explain. > > > > > > Did you track down which spinlock? > > > > > > > > > > > Hi, > > > > Sorry for my late reply and thank you for your quick reply! > > From the trace log, I think the spinlock is from here like this: > > > > __alloc_frozen_pages_noprof ==> get_page_from_freelist==>spinclok > > > > Do you need other log not only this? > > > > > > => get_page_from_freelist" > > > > => __alloc_frozen_pages_noprof > > > > => __folio_alloc_noprof > > > > => __filemap_get_folio > > > > > > > > commit b951aaff503502a7fe066eeed2744ba8a6413c89 > > > > Author: Suren Baghdasaryan > > > surenb@google.com<mailto:surenb@google.com> > > > > Date: Thu Mar 21 09:36:40 2024 -0700 > > > > > > > > mm: enable page allocation tagging > > > > > > > > Redefine page allocators to record allocation tags upon their > > invocation. > > > > Instrument post_alloc_hook and free_pages_prepare to modify > > current > > > > allocation tag. > > > > > > > > [surenb@google.com: undo _noprof additions in the documentation] > > > > Link: > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flkml > > > .kern > > > > > el.org%2Fr%2F20240326231453.1206227-3-surenb%40google.com&data=05% > > > > > 7C02%7Ccarlos.song%40nxp.com%7Cdf027bbb97074fc2cde808dd67a1d6c5%7 > > > > > C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C638780666983046738% > > > > > 7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMDA > > > > > wMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7C > > > > > &sdata=Aby13KyQjF5pcbYW%2BkfsEJLiaPmS2ZiJLUHJ%2BCr2JXM%3D&reserve > > > d=0 > > > > Link: > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2Flkml > > > .kern > > > > > el.org%2Fr%2F20240321163705.3067592-19-surenb%40google.com&data=05 > > > %7C02%7Ccarlos.song%40nxp.com%7Cdf027bbb97074fc2cde808dd67a1d6c5 > > % > > > > > 7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C638780666983065790 > > > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuM > > D > > > > > AwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7 > > > > > C&sdata=n0hjs2fhjYs%2BcnbrxHy4vFK6D4GFVL4%2Fu72anOOLiEI%3D&reserve > > > d=0 > > > > Signed-off-by: Suren Baghdasaryan > > > surenb@google.com<mailto:surenb@google.com> > > > > Co-developed-by: Kent Overstreet > > > kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev> > > > > Signed-off-by: Kent Overstreet > > > kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev> > > > > Reviewed-by: Kees Cook > > > keescook@chromium.org<mailto:keescook@chromium.org> > > > > Tested-by: Kees Cook > > > keescook@chromium.org<mailto:keescook@chromium.org> > > > > Cc: Alexander Viro > > > viro@zeniv.linux.org.uk<mailto:viro@zeniv.linux.org.uk> > > > > Cc: Alex Gaynor > > > alex.gaynor@gmail.com<mailto:alex.gaynor@gmail.com> > > > > Cc: Alice Ryhl aliceryhl@google.com<mailto:aliceryhl@google.com> > > > > Cc: Andreas Hindborg > > > a.hindborg@samsung.com<mailto:a.hindborg@samsung.com> > > > > Cc: Benno Lossin > > > benno.lossin@proton.me<mailto:benno.lossin@proton.me> > > > > Cc: "Björn Roy Baron" > > > bjorn3_gh@protonmail.com<mailto:bjorn3_gh@protonmail.com> > > > > Cc: Boqun Feng > > > boqun.feng@gmail.com<mailto:boqun.feng@gmail.com> > > > > Cc: Christoph Lameter cl@linux.com<mailto:cl@linux.com> > > > > Cc: Dennis Zhou dennis@kernel.org<mailto:dennis@kernel.org> > > > > Cc: Gary Guo gary@garyguo.net<mailto:gary@garyguo.net> > > > > Cc: Miguel Ojeda ojeda@kernel.org<mailto:ojeda@kernel.org> > > > > Cc: Pasha Tatashin > > > pasha.tatashin@soleen.com<mailto:pasha.tatashin@soleen.com> > > > > Cc: Peter Zijlstra > > peterz@infradead.org<mailto:peterz@infradead.org> > > > > Cc: Tejun Heo tj@kernel.org<mailto:tj@kernel.org> > > > > Cc: Vlastimil Babka vbabka@suse.cz<mailto:vbabka@suse.cz> > > > > Cc: Wedson Almeida Filho > > > wedsonaf@gmail.com<mailto:wedsonaf@gmail.com> > > > > Signed-off-by: Andrew Morton > > > > akpm@linux-foundation.org<mailto:akpm@linux-foundation.org> > > > > > > > > > > > > Ftrace irqoff tracer shows detail: > > > > At v6.14: > > > > # tracer: irqsoff > > > > # > > > > # irqsoff latency trace v1.1.5 on 6.14.0-rc7-next-20250319 # > > > > -------------------------------------------------------------------- > > > > # latency: 279663 us, #21352/21352, CPU#0 | (M:NONE VP:0, KP:0, SP:0 > > > > HP:0 > > > #P:2) > > > > # ----------------- > > > > # | task: dd-805 (uid:0 nice:0 policy:0 rt_prio:0) > > > > # ----------------- > > > > # => started at: __rmqueue_pcplist > > > > # => ended at: _raw_spin_unlock_irqrestore > > > > # > > > > # > > > > # _------=> CPU# > > > > # / _-----=> irqs-off/BH-disabled > > > > # | / _----=> need-resched > > > > # || / _---=> hardirq/softirq > > > > # ||| / _--=> preempt-depth > > > > # |||| / _-=> migrate-disable > > > > # ||||| / delay > > > > # cmd pid |||||| time | caller > > > > # \ / |||||| \ | / > > > > dd-805 0d.... 1us : __rmqueue_pcplist > > > > dd-805 0d.... 3us : _raw_spin_trylock > > > <-__rmqueue_pcplist > > > > dd-805 0d.... 7us : __mod_zone_page_state > > > <-__rmqueue_pcplist > > > > dd-805 0d.... 10us : __mod_zone_page_state > > > <-__rmqueue_pcplist > > > > dd-805 0d.... 12us : __mod_zone_page_state > > > <-__rmqueue_pcplist > > > > dd-805 0d.... 15us : __mod_zone_page_state > > > <-__rmqueue_pcplist > > > > dd-805 0d.... 17us : __mod_zone_page_state > > > <-__rmqueue_pcplist > > > > dd-805 0d.... 19us : __mod_zone_page_state > > > <-__rmqueue_pcplist > > > > ... > > > > dd-805 0d.... 1535us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-805 0d.... 1538us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-805 0d.... 1539us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-805 0d.... 1542us+: try_to_claim_block > > > <-__rmqueue_pcplist > > > > dd-805 0d.... 1597us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-805 0d.... 1599us+: try_to_claim_block > > > <-__rmqueue_pcplist > > > > dd-805 0d.... 1674us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-805 0d.... 1676us+: try_to_claim_block > > > <-__rmqueue_pcplist > > > > dd-805 0d.... 1716us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-805 0d.... 1718us+: try_to_claim_block > > > <-__rmqueue_pcplist > > > > dd-805 0d.... 1801us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-805 0d.... 1803us+: try_to_claim_block > > > <-__rmqueue_pcplist > > > > ... > > > > dd-805 0d.... 279555us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-805 0d.... 279556us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-805 0d.... 279558us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-805 0d.... 279560us+: try_to_claim_block > > > <-__rmqueue_pcplist > > > > dd-805 0d.... 279616us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-805 0d.... 279618us : __mod_zone_page_state > > > <-__rmqueue_pcplist > > > > dd-805 0d.... 279620us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > ... > > > > dd-805 0d.... 279658us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-805 0d.... 279660us : _raw_spin_unlock_irqrestore > > > <-__rmqueue_pcplist > > > > dd-805 0d.... 279662us : _raw_spin_unlock_irqrestore > > > > dd-805 0d.... 279666us+: trace_hardirqs_on > > > <-_raw_spin_unlock_irqrestore > > > > dd-805 0d.... 279712us : <stack trace> > > > > => get_page_from_freelist > > > > => __alloc_frozen_pages_noprof > > > > => __folio_alloc_noprof > > > > => __filemap_get_folio > > > > => iomap_write_begin > > > > => iomap_file_buffered_write > > > > => blkdev_write_iter > > > > => vfs_write > > > > => ksys_write > > > > => ret_fast_syscall > > > > > > > > At v6.7: > > > > # tracer: irqsoff > > > > # > > > > # irqsoff latency trace v1.1.5 on 6.7.0 # > > > > -------------------------------------------------------------------- > > > > # latency: 2477 us, #146/146, CPU#0 | (M:server VP:0, KP:0, SP:0 HP:0 #P:2) > > > > # ----------------- > > > > # | task: dd-808 (uid:0 nice:0 policy:0 rt_prio:0) > > > > # ----------------- > > > > # => started at: _raw_spin_lock_irqsave > > > > # => ended at: _raw_spin_unlock_irqrestore > > > > # > > > > # > > > > # _------=> CPU# > > > > # / _-----=> irqs-off/BH-disabled > > > > # | / _----=> need-resched > > > > # || / _---=> hardirq/softirq > > > > # ||| / _--=> preempt-depth > > > > # |||| / _-=> migrate-disable > > > > # ||||| / delay > > > > # cmd pid |||||| time | caller > > > > # \ / |||||| \ | / > > > > dd-808 0d.... 1us!: _raw_spin_lock_irqsave > > > > dd-808 0d.... 186us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 189us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 191us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 192us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 194us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 196us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 199us : steal_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 203us!: move_freepages_block > > > <-steal_suitable_fallback > > > > dd-808 0d.... 330us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 332us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 334us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 336us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 338us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 339us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 341us : steal_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 343us!: move_freepages_block > > > <-steal_suitable_fallback > > > > dd-808 0d.... 479us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 481us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 483us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 485us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 486us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 488us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 490us : steal_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 492us!: move_freepages_block > > > <-steal_suitable_fallback > > > > dd-808 0d.... 630us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 632us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 634us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 636us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 638us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 640us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 642us : steal_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 644us!: move_freepages_block > > > <-steal_suitable_fallback > > > > dd-808 0d.... 771us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 773us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 775us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 777us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 778us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 780us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 782us : steal_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 784us!: move_freepages_block > > > <-steal_suitable_fallback > > > > dd-808 0d.... 911us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 913us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 915us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 916us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 918us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 920us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 922us : steal_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 924us!: move_freepages_block > > > <-steal_suitable_fallback > > > > dd-808 0d.... 1055us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1058us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1059us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1061us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1063us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1065us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1066us : steal_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1068us!: move_freepages_block > > > <-steal_suitable_fallback > > > > dd-808 0d.... 1194us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1196us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1198us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1200us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1202us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1203us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1205us : steal_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1208us!: move_freepages_block > > > <-steal_suitable_fallback > > > > dd-808 0d.... 1333us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1335us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1337us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1339us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1341us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1342us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1344us : steal_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1346us!: move_freepages_block > > > <-steal_suitable_fallback > > > > dd-808 0d.... 1480us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1482us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1484us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1486us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1488us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1490us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1492us : steal_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1494us!: move_freepages_block > > > <-steal_suitable_fallback > > > > dd-808 0d.... 1621us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1623us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1625us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1627us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1629us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1630us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1632us : steal_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1634us!: move_freepages_block > > > <-steal_suitable_fallback > > > > dd-808 0d.... 1761us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1763us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1765us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1766us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1768us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1770us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1772us : steal_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1774us!: move_freepages_block > > > <-steal_suitable_fallback > > > > dd-808 0d.... 1900us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1902us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1903us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1905us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1907us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1909us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1911us : steal_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 1913us!: move_freepages_block > > > <-steal_suitable_fallback > > > > dd-808 0d.... 2038us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2040us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2042us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2044us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2046us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2047us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2049us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2051us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2053us : steal_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2055us!: move_freepages_block > > > <-steal_suitable_fallback > > > > dd-808 0d.... 2175us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2176us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2178us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2180us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2182us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2183us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2185us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2187us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2189us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2191us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2192us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2194us : steal_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2196us!: move_freepages_block > > > <-steal_suitable_fallback > > > > dd-808 0d.... 2323us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2325us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2327us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2328us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2330us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2332us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2334us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2335us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2337us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2339us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2341us : find_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2343us : steal_suitable_fallback > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2345us!: move_freepages_block > > > <-steal_suitable_fallback > > > > dd-808 0d.... 2470us : __mod_zone_page_state > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2473us : _raw_spin_unlock_irqrestore > > > <-__rmqueue_pcplist > > > > dd-808 0d.... 2476us : _raw_spin_unlock_irqrestore > > > > dd-808 0d.... 2479us+: tracer_hardirqs_on > > > <-_raw_spin_unlock_irqrestore > > > > dd-808 0d.... 2520us : <stack trace> > > > > => get_page_from_freelist > > > > => __alloc_pages > > > > => __folio_alloc > > > > => __filemap_get_folio > > > > => iomap_write_begin > > > > => iomap_file_buffered_write > > > > => blkdev_write_iter > > > > => vfs_write > > > > => ksys_write > > > > => ret_fast_syscall > > > > > > > > Best Regard > > > > Carlos Song > > > > ^ permalink raw reply [flat|nested] 8+ messages in thread
* Re: Ask help about this patch b951aaff5035 " mm: enable page allocation tagging" 2025-03-31 20:01 ` Suren Baghdasaryan @ 2025-04-02 2:30 ` Carlos Song 0 siblings, 0 replies; 8+ messages in thread From: Carlos Song @ 2025-04-02 2:30 UTC (permalink / raw) To: Suren Baghdasaryan Cc: Kent Overstreet, akpm, willy, linux-mm, linux-kernel, linux-fsdevel > -----Original Message----- > From: Suren Baghdasaryan <surenb@google.com> > Sent: Tuesday, April 1, 2025 4:02 AM > To: Carlos Song <carlos.song@nxp.com> > Cc: Kent Overstreet <kent.overstreet@linux.dev>; akpm@linux-foundation.org; > willy@infradead.org; linux-mm@kvack.org; linux-kernel@vger.kernel.org; > linux-fsdevel@vger.kernel.org > Subject: [EXT] Re: Ask help about this patch b951aaff5035 " mm: enable page > allocation tagging" > > Caution: This is an external email. Please take care when clicking links or > opening attachments. When in doubt, report the message using the 'Report > this email' button > > > On Thu, Mar 27, 2025 at 4:02 AM Carlos Song <carlos.song@nxp.com> wrote: > > > > Hi, > > > > I get the function calling step log, I hope it can help debug little. > > Hi Carlos, > Since you do not enable CONFIG_MEM_ALLOC_PROFILING, "mm: enable page > allocation tagging" patch would most definitely not cause this issue. > > From the traces you posted I think the spinlock involved here is > zone->lock inside rmqueue_bulk(): > https://elixir.bo/ > otlin.com%2Flinux%2Fv6.14-rc6%2Fsource%2Fmm%2Fpage_alloc.c%23L2310 > &data=05%7C02%7Ccarlos.song%40nxp.com%7Cb83b5b83b6f94339e2f108d > d708edead%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C638790 > 481110586829%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydW > UsIlYiOiIwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3 > D%7C0%7C%7C%7C&sdata=gXVB6wbfdbT6kH1QBc7tcfyeUnGpChGDc5IpVTcuz > MM%3D&reserved=0 > I quickly scanned the code within that loop comparing 6.7 and 6.14 and while I > did not find anything obvious, steal_suitable_fallback() function seems quite > different. I would suggest profiling this path to see which part might have > regressed and go from there. > BTW, are you using pure upstream code or it has some out-of-tree > modifications? > Thanks, > Suren. > > Hi, Suren Thank you for your help, I found the regression is really from steal_suitable_fallback() function. When add this patch, the spinlock time will become 100ms from 1ms, I will write a new email to discuss this. commit c0cd6f557b9090525d288806cccbc73440ac235a Author: Johannes Weiner <hannes@cmpxchg.org> Date: Wed Mar 20 14:02:11 2024 -0400 mm: page_alloc: fix freelist movement during block conversion Currently, page block type conversion during fallbacks, atomic reservations and isolation can strand various amounts of free pages on incorrect freelists. For example, fallback stealing moves free pages in the block to the new type's freelists, but then may not actually claim the block for that type if there aren't enough compatible pages already allocated. In all cases, free page moving might fail if the block straddles more than one zone, in which case no free pages are moved at all, but the block type is changed anyway. This is detrimental to type hygiene on the freelists. It encourages incompatible page mixing down the line (ask for one type, get another) and thus contributes to long-term fragmentation. Split the process into a proper transaction: check first if conversion will happen, then try to move the free pages, and only if that was successful convert the block to the new type. [baolin.wang@linux.alibaba.com: fix allocation failures with CONFIG_CMA] Link: https://lkml.kernel.org/r/a97697e0-45b0-4f71-b087-fdc7a1d43c0e@linux.alibaba.com Link: https://lkml.kernel.org/r/20240320180429.678181-7-hannes@cmpxchg.org Signed-off-by: Johannes Weiner <hannes@cmpxchg.org> Signed-off-by: Baolin Wang <baolin.wang@linux.alibaba.com> Tested-by: "Huang, Ying" <ying.huang@intel.com> Reviewed-by: Vlastimil Babka <vbabka@suse.cz> Tested-by: Baolin Wang <baolin.wang@linux.alibaba.com> Cc: David Hildenbrand <david@redhat.com> Cc: Mel Gorman <mgorman@techsingularity.net> Cc: Zi Yan <ziy@nvidia.com> Signed-off-by: Andrew Morton <akpm@linux-foundation.org> > > > > The spinlock_irqsave from 136.725021 to > 136.966962_raw_spin_unlock_irqrestore. > > Around 260ms? It is Just one case, sometimes it will spend more time. > > > > dd-828 [000] ..... 136.724980: > fdget_pos <-ksys_write > > dd-828 [000] ..... 136.724982: vfs_write > <-ksys_write > > dd-828 [000] ..... 136.724983: blkdev_write_iter > <-vfs_write > > dd-828 [000] ..... 136.724984: I_BDEV > <-blkdev_write_iter > > dd-828 [000] ..... 136.724985: file_update_time > <-blkdev_write_iter > > dd-828 [000] ..... 136.724986: > inode_needs_update_time <-file_update_time > > dd-828 [000] ..... 136.724987: > ktime_get_coarse_real_ts64 <-inode_needs_update_time > > dd-828 [000] ..... 136.724989: timestamp_truncate > <-inode_needs_update_time > > dd-828 [000] ..... 136.724990: > iomap_file_buffered_write <-blkdev_write_iter > > dd-828 [000] ..... 136.724992: iomap_iter > <-iomap_file_buffered_write > > dd-828 [000] ..... 136.724993: blkdev_iomap_begin > <-iomap_iter > > dd-828 [000] ..... 136.724994: I_BDEV > <-blkdev_iomap_begin > > dd-828 [000] ..... 136.724995: > balance_dirty_pages_ratelimited_flags <-iomap_file_buffered_write > > dd-828 [000] ..... 136.724996: inode_to_bdi > <-balance_dirty_pages_ratelimited_flags > > dd-828 [000] ..... 136.724997: I_BDEV > <-inode_to_bdi > > dd-828 [000] ..... 136.724998: preempt_count_add > <-balance_dirty_pages_ratelimited_flags > > dd-828 [000] ...1. 136.725000: preempt_count_sub > <-balance_dirty_pages_ratelimited_flags > > dd-828 [000] ..... 136.725001: fault_in_readable > <-fault_in_iov_iter_readable > > dd-828 [000] ..... 136.725003: iomap_write_begin > <-iomap_file_buffered_write > > dd-828 [000] ..... 136.725004: iomap_get_folio > <-iomap_write_begin > > dd-828 [000] ..... 136.725005: __filemap_get_folio > <-iomap_write_begin > > dd-828 [000] ..... 136.725006: filemap_get_entry > <-__filemap_get_folio > > dd-828 [000] ..... 136.725007: __rcu_read_lock > <-filemap_get_entry > > dd-828 [000] ..... 136.725008: __rcu_read_unlock > <-filemap_get_entry > > dd-828 [000] ..... 136.725009: inode_to_bdi > <-__filemap_get_folio > > dd-828 [000] ..... 136.725010: I_BDEV > <-inode_to_bdi > > dd-828 [000] ..... 136.725012: __folio_alloc_noprof > <-__filemap_get_folio > > dd-828 [000] ..... 136.725013: > __alloc_pages_noprof <-__folio_alloc_noprof > > dd-828 [000] ..... 136.725014: > get_page_from_freelist <-__alloc_pages_noprof > > dd-828 [000] ..... 136.725015: node_dirty_ok > <-get_page_from_freelist > > dd-828 [000] ..... 136.725016: preempt_count_add > <-get_page_from_freelist > > dd-828 [000] ...1. 136.725018: _raw_spin_trylock > <-get_page_from_freelist > > dd-828 [000] ...1. 136.725019: preempt_count_add > <-_raw_spin_trylock > > dd-828 [000] ...2. 136.725021: > _raw_spin_lock_irqsave <-__rmqueue_pcplist > > dd-828 [000] d..2. 136.725022: > preempt_count_add <-_raw_spin_lock_irqsave > > dd-828 [000] d..3. 136.725025: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725028: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725029: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725031: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725032: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725034: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725036: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725037: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725038: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725039: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725040: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725042: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725043: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725044: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725046: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725047: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725048: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725050: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725051: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725052: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725053: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725055: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725056: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725057: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725058: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725060: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725061: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725062: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725063: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725065: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725066: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725067: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725068: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725070: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725071: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725072: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725073: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725075: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725076: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725077: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725079: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725080: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725081: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725082: > __mod_zone_page_state <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.725083: > __mod_zone_page_state <-__rmqueue_pcplist > > ... > > dd-828 [000] d..3. 136.820279: > find_suitable_fallback <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.820280: > find_suitable_fallback <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.820281: > find_suitable_fallback <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.820282: > find_suitable_fallback <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.820283: > find_suitable_fallback <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.820284: > find_suitable_fallback <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.820285: > find_suitable_fallback <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.820286: > find_suitable_fallback <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.820287: > find_suitable_fallback <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.820289: > find_suitable_fallback <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.820290: > steal_suitable_fallback <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.820525: > __mod_zone_page_state <-steal_suitable_fallback > > ... > > dd-828 [000] d..3. 136.966945: > __mod_zone_page_state <-steal_suitable_fallback > > dd-828 [000] d..3. 136.966946: > find_suitable_fallback <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.966947: > find_suitable_fallback <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.966948: > find_suitable_fallback <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.966950: > find_suitable_fallback <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.966951: > find_suitable_fallback <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.966952: > find_suitable_fallback <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.966953: > find_suitable_fallback <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.966954: > find_suitable_fallback <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.966955: > find_suitable_fallback <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.966956: > find_suitable_fallback <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.966957: > find_suitable_fallback <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.966958: > find_suitable_fallback <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.966959: > find_suitable_fallback <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.966961: > find_suitable_fallback <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.966962: > _raw_spin_unlock_irqrestore <-__rmqueue_pcplist > > dd-828 [000] d..3. 136.966965: irq_enter > <-generic_handle_arch_irq > > dd-828 [000] d..3. 136.966966: irq_enter_rcu > <-generic_handle_arch_irq > > dd-828 [000] d..3. 136.966967: > preempt_count_add <-irq_enter_rcu > > dd-828 [000] d.h3. 136.966969: gic_handle_irq > <-generic_handle_arch_irq > > dd-828 [000] d.h3. 136.966970: > generic_handle_domain_irq <-gic_handle_irq > > dd-828 [000] d.h3. 136.966971: > __irq_resolve_mapping <-generic_handle_domain_irq > > dd-828 [000] d.h3. 136.966972: __rcu_read_lock > <-__irq_resolve_mapping > > dd-828 [000] d.h3. 136.966974: __rcu_read_unlock > <-__irq_resolve_mapping > > dd-828 [000] d.h3. 136.966975: handle_irq_desc > <-gic_handle_irq > > dd-828 [000] d.h3. 136.966976: > handle_percpu_devid_irq <-handle_irq_desc > > dd-828 [000] d.h3. 136.966978: ipi_handler > <-handle_percpu_devid_irq > > dd-828 [000] d.h3. 136.966979: do_handle_IPI > <-ipi_handler > > dd-828 [000] d.h3. 136.966981: > generic_smp_call_function_single_interrupt <-do_handle_IPI > > dd-828 [000] d.h3. 136.966982: > __flush_smp_call_function_queue <-do_handle_IPI > > dd-828 [000] d.h3. 136.966986: > sched_ttwu_pending <-__flush_smp_call_function_queue > > dd-828 [000] d.h3. 136.966987: > preempt_count_add <-sched_ttwu_pending > > dd-828 [000] d.h4. 136.966990: _raw_spin_lock > <-sched_ttwu_pending > > dd-828 [000] d.h4. 136.966991: > preempt_count_add <-_raw_spin_lock > > dd-828 [000] d.h5. 136.966993: > preempt_count_sub <-sched_ttwu_pending > > dd-828 [000] d.h4. 136.966994: > update_rq_clock.part.0 <-sched_ttwu_pending > > dd-828 [000] d.h4. 136.966996: ttwu_do_activate > <-sched_ttwu_pending > > dd-828 [000] d.h4. 136.966997: activate_task > <-ttwu_do_activate > > dd-828 [000] d.h4. 136.966998: enqueue_task_fair > <-activate_task > > dd-828 [000] d.h4. 136.967000: update_curr > <-enqueue_task_fair > > dd-828 [000] d.h4. 136.967001: > update_min_vruntime <-update_curr > > dd-828 [000] d.h4. 136.967003: > __cgroup_account_cputime <-update_curr > > > > BR > > Carlos Song > > > > > > On Thu, Mar 20, 2025 at 11:07:41AM +0000, Carlos Song wrote: > > > > > Hi, all > > > > > > > > > > I found a 300ms~600ms IRQ off when writing 1Gb data to mmc > > > > > device at > > > > I.MX7d SDB board at Linux-kernel-v6.14. > > > > > But I test the same case at Linux-kernel-v6.7, this longest IRQ > > > > > off time is only > > > > 1ms~2ms. So the issue is introduced from v6.7~v6.14. > > > > > > > > > > Run this cmd to test: > > > > > dd if=/dev/zero of=/dev/mmcblk2p4 bs=4096 seek=12500 > > > > > count=256000 conv=fsync > > > > > > > > > > This issue looks from blkdev_buffered_write() function. Because > > > > > when I run this cmd with "oflag=direct" to use > > > > > blkdev_direct_write(), I can not see > > > > any long time IRQ off. > > > > > > > > > > Then I use Ftrace irqoff tracer to trace the longest IRQ off > > > > > event, I found some > > > > differences between v6.7 and v6.14: > > > > > In iomap_file_buffered_write(), __folio_alloc (in v6.7) is > > > > > replaced by > > > > _folio_alloc_noprof (in v6.14) by this patch. > > > > > The spinlock disabled IRQ ~300ms+. It looks there are some fixes > > > > > for this patch, > > > > but I still can see IRQ off 300ms+ at 6.14.0-rc7-next-20250319. > > > > > > > > > > Do I trigger one bug? I know little about mem so I have to > > > > > report it and hope I > > > > can get some help or guide. > > > > > I put my ftrace log at the mail tail to help trace and explain. > > > > > > > > Did you track down which spinlock? > > > > > > > > > > > > > > > Hi, > > > > > > Sorry for my late reply and thank you for your quick reply! > > > From the trace log, I think the spinlock is from here like this: > > > > > > __alloc_frozen_pages_noprof ==> get_page_from_freelist==>spinclok > > > > > > Do you need other log not only this? > > > > > > > > => get_page_from_freelist" > > > > > => __alloc_frozen_pages_noprof > > > > > => __folio_alloc_noprof > > > > > => __filemap_get_folio > > > > > > > > > > > commit b951aaff503502a7fe066eeed2744ba8a6413c89 > > > > > Author: Suren Baghdasaryan > > > > surenb@google.com<mailto:surenb@google.com> > > > > > Date: Thu Mar 21 09:36:40 2024 -0700 > > > > > > > > > > mm: enable page allocation tagging > > > > > > > > > > Redefine page allocators to record allocation tags upon > > > > > their > > > invocation. > > > > > Instrument post_alloc_hook and free_pages_prepare to modify > > > current > > > > > allocation tag. > > > > > > > > > > [surenb@google.com: undo _noprof additions in the > documentation] > > > > > Link: > > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2F > > > > > lkml%2F&data=05%7C02%7Ccarlos.song%40nxp.com%7Cb83b5b83b6f94339 > e2f > > > > > 108dd708edead%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C63 > 87904 > > > > > 81110616161%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUs > IlYiOi > > > > > IwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0 > > > > %7C%7C%7C&sdata=Ebsi8btyFmX6xHr%2BvVEUHIFbsPKqETE%2Brf5Hs41 > %2Ff8o% > > > > 3D&reserved=0 > > > > .kern > > > > > > > > el.org%2Fr%2F20240326231453.1206227-3-surenb%40google.com&data=05 > % > > > > > > > > 7C02%7Ccarlos.song%40nxp.com%7Cdf027bbb97074fc2cde808dd67a1d6c5% > 7 > > > > > > > > C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C63878066698304673 > 8% > > > > > > > > 7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwLjAuMD > A > > > > > > > > wMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C%7 > C > > > > > > > > &sdata=Aby13KyQjF5pcbYW%2BkfsEJLiaPmS2ZiJLUHJ%2BCr2JXM%3D&reserv > e > > > > d=0 > > > > > Link: > > > > https://eur01.safelinks.protection.outlook.com/?url=https%3A%2F%2F > > > > > lkml%2F&data=05%7C02%7Ccarlos.song%40nxp.com%7Cb83b5b83b6f94339 > e2f > > > > > 108dd708edead%7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C63 > 87904 > > > > > 81110632328%7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUs > IlYiOi > > > > > IwLjAuMDAwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0 > > > > %7C%7C%7C&sdata=f5xbAEifpWVbOVuxJ5LV5FS1TmaTD5T4Q%2BhtX2Ju > lsw%3D&r > > > > eserved=0 > > > > .kern > > > > > > > > el.org%2Fr%2F20240321163705.3067592-19-surenb%40google.com&data=0 > 5 > > > > %7C02%7Ccarlos.song%40nxp.com%7Cdf027bbb97074fc2cde808dd67a > 1d6c5 > > > % > > > > > > > > 7C686ea1d3bc2b4c6fa92cd99c5c301635%7C0%7C0%7C6387806669830657 > 90 > > > > %7CUnknown%7CTWFpbGZsb3d8eyJFbXB0eU1hcGkiOnRydWUsIlYiOiIwL > jAuM > > > D > > > > > > > > AwMCIsIlAiOiJXaW4zMiIsIkFOIjoiTWFpbCIsIldUIjoyfQ%3D%3D%7C0%7C%7C% > 7 > > > > > > > > C&sdata=n0hjs2fhjYs%2BcnbrxHy4vFK6D4GFVL4%2Fu72anOOLiEI%3D&reserv > e > > > > d=0 > > > > > Signed-off-by: Suren Baghdasaryan > > > > surenb@google.com<mailto:surenb@google.com> > > > > > Co-developed-by: Kent Overstreet > > > > kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev> > > > > > Signed-off-by: Kent Overstreet > > > > kent.overstreet@linux.dev<mailto:kent.overstreet@linux.dev> > > > > > Reviewed-by: Kees Cook > > > > keescook@chromium.org<mailto:keescook@chromium.org> > > > > > Tested-by: Kees Cook > > > > keescook@chromium.org<mailto:keescook@chromium.org> > > > > > Cc: Alexander Viro > > > > viro@zeniv.linux.org.uk<mailto:viro@zeniv.linux.org.uk> > > > > > Cc: Alex Gaynor > > > > alex.gaynor@gmail.com<mailto:alex.gaynor@gmail.com> > > > > > Cc: Alice Ryhl > aliceryhl@google.com<mailto:aliceryhl@google.com> > > > > > Cc: Andreas Hindborg > > > > a.hindborg@samsung.com<mailto:a.hindborg@samsung.com> > > > > > Cc: Benno Lossin > > > > benno.lossin@proton.me<mailto:benno.lossin@proton.me> > > > > > Cc: "Björn Roy Baron" > > > > bjorn3_gh@protonmail.com<mailto:bjorn3_gh@protonmail.com> > > > > > Cc: Boqun Feng > > > > boqun.feng@gmail.com<mailto:boqun.feng@gmail.com> > > > > > Cc: Christoph Lameter cl@linux.com<mailto:cl@linux.com> > > > > > Cc: Dennis Zhou dennis@kernel.org<mailto:dennis@kernel.org> > > > > > Cc: Gary Guo gary@garyguo.net<mailto:gary@garyguo.net> > > > > > Cc: Miguel Ojeda ojeda@kernel.org<mailto:ojeda@kernel.org> > > > > > Cc: Pasha Tatashin > > > > pasha.tatashin@soleen.com<mailto:pasha.tatashin@soleen.com> > > > > > Cc: Peter Zijlstra > > > peterz@infradead.org<mailto:peterz@infradead.org> > > > > > Cc: Tejun Heo tj@kernel.org<mailto:tj@kernel.org> > > > > > Cc: Vlastimil Babka vbabka@suse.cz<mailto:vbabka@suse.cz> > > > > > Cc: Wedson Almeida Filho > > > > wedsonaf@gmail.com<mailto:wedsonaf@gmail.com> > > > > > Signed-off-by: Andrew Morton > > > > > akpm@linux-foundation.org<mailto:akpm@linux-foundation.org> > > > > > > > > > > > > > > > Ftrace irqoff tracer shows detail: > > > > > At v6.14: > > > > > # tracer: irqsoff > > > > > # > > > > > # irqsoff latency trace v1.1.5 on 6.14.0-rc7-next-20250319 # > > > > > ---------------------------------------------------------------- > > > > > ---- # latency: 279663 us, #21352/21352, CPU#0 | (M:NONE VP:0, > > > > > KP:0, SP:0 > > > > > HP:0 > > > > #P:2) > > > > > # ----------------- > > > > > # | task: dd-805 (uid:0 nice:0 policy:0 rt_prio:0) > > > > > # ----------------- > > > > > # => started at: __rmqueue_pcplist > > > > > # => ended at: _raw_spin_unlock_irqrestore > > > > > # > > > > > # > > > > > # _------=> CPU# > > > > > # / _-----=> irqs-off/BH-disabled > > > > > # | / _----=> need-resched > > > > > # || / _---=> hardirq/softirq > > > > > # ||| / _--=> preempt-depth > > > > > # |||| / _-=> migrate-disable > > > > > # ||||| / delay > > > > > # cmd pid |||||| time | caller > > > > > # \ / |||||| \ | / > > > > > dd-805 0d.... 1us : __rmqueue_pcplist > > > > > dd-805 0d.... 3us : _raw_spin_trylock > > > > <-__rmqueue_pcplist > > > > > dd-805 0d.... 7us : __mod_zone_page_state > > > > <-__rmqueue_pcplist > > > > > dd-805 0d.... 10us : __mod_zone_page_state > > > > <-__rmqueue_pcplist > > > > > dd-805 0d.... 12us : __mod_zone_page_state > > > > <-__rmqueue_pcplist > > > > > dd-805 0d.... 15us : __mod_zone_page_state > > > > <-__rmqueue_pcplist > > > > > dd-805 0d.... 17us : __mod_zone_page_state > > > > <-__rmqueue_pcplist > > > > > dd-805 0d.... 19us : __mod_zone_page_state > > > > <-__rmqueue_pcplist > > > > > ... > > > > > dd-805 0d.... 1535us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-805 0d.... 1538us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-805 0d.... 1539us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-805 0d.... 1542us+: try_to_claim_block > > > > <-__rmqueue_pcplist > > > > > dd-805 0d.... 1597us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-805 0d.... 1599us+: try_to_claim_block > > > > <-__rmqueue_pcplist > > > > > dd-805 0d.... 1674us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-805 0d.... 1676us+: try_to_claim_block > > > > <-__rmqueue_pcplist > > > > > dd-805 0d.... 1716us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-805 0d.... 1718us+: try_to_claim_block > > > > <-__rmqueue_pcplist > > > > > dd-805 0d.... 1801us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-805 0d.... 1803us+: try_to_claim_block > > > > <-__rmqueue_pcplist > > > > > ... > > > > > dd-805 0d.... 279555us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-805 0d.... 279556us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-805 0d.... 279558us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-805 0d.... 279560us+: try_to_claim_block > > > > <-__rmqueue_pcplist > > > > > dd-805 0d.... 279616us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-805 0d.... 279618us : __mod_zone_page_state > > > > <-__rmqueue_pcplist > > > > > dd-805 0d.... 279620us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > ... > > > > > dd-805 0d.... 279658us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-805 0d.... 279660us : _raw_spin_unlock_irqrestore > > > > <-__rmqueue_pcplist > > > > > dd-805 0d.... 279662us : _raw_spin_unlock_irqrestore > > > > > dd-805 0d.... 279666us+: trace_hardirqs_on > > > > <-_raw_spin_unlock_irqrestore > > > > > dd-805 0d.... 279712us : <stack trace> > > > > > => get_page_from_freelist > > > > > => __alloc_frozen_pages_noprof > > > > > => __folio_alloc_noprof > > > > > => __filemap_get_folio > > > > > => iomap_write_begin > > > > > => iomap_file_buffered_write > > > > > => blkdev_write_iter > > > > > => vfs_write > > > > > => ksys_write > > > > > => ret_fast_syscall > > > > > > > > > > At v6.7: > > > > > # tracer: irqsoff > > > > > # > > > > > # irqsoff latency trace v1.1.5 on 6.7.0 # > > > > > ---------------------------------------------------------------- > > > > > ---- # latency: 2477 us, #146/146, CPU#0 | (M:server VP:0, KP:0, > > > > > SP:0 HP:0 #P:2) > > > > > # ----------------- > > > > > # | task: dd-808 (uid:0 nice:0 policy:0 rt_prio:0) > > > > > # ----------------- > > > > > # => started at: _raw_spin_lock_irqsave > > > > > # => ended at: _raw_spin_unlock_irqrestore > > > > > # > > > > > # > > > > > # _------=> CPU# > > > > > # / _-----=> irqs-off/BH-disabled > > > > > # | / _----=> need-resched > > > > > # || / _---=> hardirq/softirq > > > > > # ||| / _--=> preempt-depth > > > > > # |||| / _-=> migrate-disable > > > > > # ||||| / delay > > > > > # cmd pid |||||| time | caller > > > > > # \ / |||||| \ | / > > > > > dd-808 0d.... 1us!: _raw_spin_lock_irqsave > > > > > dd-808 0d.... 186us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 189us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 191us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 192us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 194us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 196us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 199us : steal_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 203us!: move_freepages_block > > > > <-steal_suitable_fallback > > > > > dd-808 0d.... 330us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 332us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 334us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 336us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 338us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 339us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 341us : steal_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 343us!: move_freepages_block > > > > <-steal_suitable_fallback > > > > > dd-808 0d.... 479us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 481us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 483us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 485us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 486us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 488us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 490us : steal_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 492us!: move_freepages_block > > > > <-steal_suitable_fallback > > > > > dd-808 0d.... 630us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 632us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 634us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 636us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 638us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 640us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 642us : steal_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 644us!: move_freepages_block > > > > <-steal_suitable_fallback > > > > > dd-808 0d.... 771us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 773us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 775us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 777us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 778us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 780us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 782us : steal_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 784us!: move_freepages_block > > > > <-steal_suitable_fallback > > > > > dd-808 0d.... 911us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 913us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 915us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 916us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 918us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 920us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 922us : steal_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 924us!: move_freepages_block > > > > <-steal_suitable_fallback > > > > > dd-808 0d.... 1055us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1058us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1059us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1061us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1063us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1065us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1066us : steal_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1068us!: move_freepages_block > > > > <-steal_suitable_fallback > > > > > dd-808 0d.... 1194us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1196us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1198us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1200us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1202us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1203us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1205us : steal_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1208us!: move_freepages_block > > > > <-steal_suitable_fallback > > > > > dd-808 0d.... 1333us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1335us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1337us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1339us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1341us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1342us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1344us : steal_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1346us!: move_freepages_block > > > > <-steal_suitable_fallback > > > > > dd-808 0d.... 1480us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1482us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1484us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1486us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1488us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1490us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1492us : steal_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1494us!: move_freepages_block > > > > <-steal_suitable_fallback > > > > > dd-808 0d.... 1621us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1623us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1625us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1627us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1629us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1630us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1632us : steal_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1634us!: move_freepages_block > > > > <-steal_suitable_fallback > > > > > dd-808 0d.... 1761us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1763us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1765us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1766us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1768us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1770us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1772us : steal_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1774us!: move_freepages_block > > > > <-steal_suitable_fallback > > > > > dd-808 0d.... 1900us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1902us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1903us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1905us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1907us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1909us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1911us : steal_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 1913us!: move_freepages_block > > > > <-steal_suitable_fallback > > > > > dd-808 0d.... 2038us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2040us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2042us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2044us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2046us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2047us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2049us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2051us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2053us : steal_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2055us!: move_freepages_block > > > > <-steal_suitable_fallback > > > > > dd-808 0d.... 2175us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2176us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2178us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2180us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2182us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2183us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2185us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2187us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2189us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2191us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2192us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2194us : steal_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2196us!: move_freepages_block > > > > <-steal_suitable_fallback > > > > > dd-808 0d.... 2323us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2325us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2327us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2328us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2330us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2332us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2334us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2335us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2337us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2339us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2341us : find_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2343us : steal_suitable_fallback > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2345us!: move_freepages_block > > > > <-steal_suitable_fallback > > > > > dd-808 0d.... 2470us : __mod_zone_page_state > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2473us : _raw_spin_unlock_irqrestore > > > > <-__rmqueue_pcplist > > > > > dd-808 0d.... 2476us : _raw_spin_unlock_irqrestore > > > > > dd-808 0d.... 2479us+: tracer_hardirqs_on > > > > <-_raw_spin_unlock_irqrestore > > > > > dd-808 0d.... 2520us : <stack trace> > > > > > => get_page_from_freelist > > > > > => __alloc_pages > > > > > => __folio_alloc > > > > > => __filemap_get_folio > > > > > => iomap_write_begin > > > > > => iomap_file_buffered_write > > > > > => blkdev_write_iter > > > > > => vfs_write > > > > > => ksys_write > > > > > => ret_fast_syscall > > > > > > > > > > Best Regard > > > > > Carlos Song > > > > > ^ permalink raw reply [flat|nested] 8+ messages in thread
end of thread, other threads:[~2025-04-02 2:31 UTC | newest] Thread overview: 8+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2025-03-20 11:07 Ask help about this patch b951aaff5035 " mm: enable page allocation tagging" Carlos Song 2025-03-20 11:24 ` Kent Overstreet 2025-03-20 15:50 ` Suren Baghdasaryan 2025-03-27 8:20 ` [EXT] " Carlos Song 2025-03-27 8:25 ` Carlos Song 2025-03-27 11:02 ` Carlos Song 2025-03-31 20:01 ` Suren Baghdasaryan 2025-04-02 2:30 ` Carlos Song
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox