* Re: [Bug 220605] New: Very slow access on HDD when a swap partition is used on this HDD on kernel 6.16
2025-09-29 23:07 ` [Bug 220605] New: Very slow access on HDD when a swap partition is used on this HDD on kernel 6.16 Andrew Morton
@ 2025-10-10 16:06 ` Pierre Juhen
2025-10-14 6:29 ` Kairui Song
2025-10-15 13:00 ` Pierre Juhen
2025-10-16 8:04 ` Pierre Juhen
2 siblings, 1 reply; 6+ messages in thread
From: Pierre Juhen @ 2025-10-10 16:06 UTC (permalink / raw)
To: Andrew Morton; +Cc: bugzilla-daemon, linux-mm
[-- Attachment #1: Type: text/plain, Size: 4238 bytes --]
Hi Andrew and people reading those lists,
I still struggle to understand what is wrong with me.
I rebuilt my mdadm + bcache array to be sure it was not something linked
to a (too) old configuration. Upgrading to 6.16.10 didn't change anything.
One symptom of the problem I identified is the time of the swapoff
command (I have 32G, 12G are free and there is less than 1G in the swap)
MiB Mem : 31952,0 total, 12646,2 free, 2226,8 used, 17587,5 buff/cache
MiB Swap: 36000,0 total, 35212,2 free, 787,8 used. 29725,2 avail Mem
*$ time sudo swapoff -a
* *
real6m41,276s
user0m0,005s
sys0m0,028s*
Waiting for more than 6 minutes elapsed for 33 ms of work is bizarre.
During that time, if I try to launch another tab with *iotop*, nothing
is displayed during at least one minute.
During that time, my computer is almost idle.
So it looks like swapoff and iotop are stucked, waiting for some kind of
a semaphore inside the kernel.
Any idea to dig further ?
Thanks,
Pierre
Le 30/09/2025 à 01:07, Andrew Morton a écrit :
> (switched to email. Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
>
> On Sat, 27 Sep 2025 08:07:59 +0000bugzilla-daemon@kernel.org wrote:
>
>> https://bugzilla.kernel.org/show_bug.cgi?id=220605
>>
>> Bug ID: 220605
>> Summary: Very slow access on HDD when a swap partition is used
>> on this HDD on kernel 6.16
>> Product: Memory Management
>> Version: 2.5
>> Hardware: Intel
>> OS: Linux
>> Status: NEW
>> Severity: high
>> Priority: P3
>> Component: Page Allocator
>> Assignee:akpm@linux-foundation.org
>> Reporter:pierre.juhen@orange.fr
>> Regression: No
>>
>> Hello everybody,
>>
>> I have a strange behavior on my system (Fedora 42 up to date => kernel 6.16.8).
> You certainly do. I wonder what we did to cause this.
>
> I'll cc the MM development list. Please prefer to report issues over
> email this way - we don't use bugzilla and it's rather annoying that MM
> bugzilla even exists.
>
>
>> I have many disks but but my two swap partitions are on 2 HDD, sda and sdb,
>> which are identical (SEAGATE Barracuda 2To). Swap partition on sda has a higher
>> priority than the one on sdb.
>>
>> I don't use zram.
>>
>> When a swap partion is used, the disk on wich it seats is very slow.
>>
>> $ sudo hdparm -t /dev/sda
>> /dev/sda:
>> Timing buffered disk reads: 24 MB in 3.72 seconds = 6.46 MB/sec
>>
>> $ sudo hdparm -t /dev/sdb
>> /dev/sdb:
>> Timing buffered disk reads: 540 MB in 3.16 seconds = 170.85 MB/sec
>> pierre@pierre:~$
>>
>> Sometimes, speed gets down to less than 2 MB/s.
>>
>> Lets turn swap off => swapoff -a
>>
>> $ sudo hdparm -t /dev/sdb
>> /dev/sdb:
>> Timing buffered disk reads: 462 MB in 3.00 seconds = 153.75 MB/sec
>>
>> $ sudo hdparm -t /dev/sda
>>
>> /dev/sda:
>> Timing buffered disk reads: 474 MB in 3.00 seconds = 157.84 MB/sec
>> pierre@pierre:~$
>>
>> Now speed is normal.
>>
>> Lets put swap on again => sudo swapon -a
>>
>> Speed is still normal :
>>
>> $ sudo hdparm -t /dev/sda
>> /dev/sda:
>> Timing buffered disk reads: 612 MB in 3.01 seconds = 203.56 MB/sec
>> $ sudo hdparm -t /dev/sdb
>> /dev/sdb:
>> Timing buffered disk reads: 570 MB in 3.00 seconds = 189.95 MB/sec
>> pierre@pierre:~$
>>
>> But swap is not used
>>
>> MiB Mem : 31952,0 total, 27091,7 free, 3057,6 used, 2343,0 buff/cache
>> MiB Swap: 36000,0 total, 36000,0 free, 0,0 used. 28894,4 avail Mem
>>
>>
>> So I suspect that, when the swap is used, there is some sort of lock that slows
>> the whole disk that contains the swap partition that is being used.
>>
>> Logically, the system becomes very slow.
>>
>> There is nothing in the journal that seems related to this problem.
>>
>> Since the virtual memory has been modified in 6.16, I suspect a nasty bug.
>>
>> This affects 6.16.5 to 6.16.8.
>>
>> This appeared "recently", probably with 6.16
>>
>> Thanks,
>>
>> Regards,
>>
>> Pierre
>>
>> --
>> You may reply to this email to add a comment.
>>
>> You are receiving this mail because:
>> You are the assignee for the bug.
[-- Attachment #2: Type: text/html, Size: 8537 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Bug 220605] New: Very slow access on HDD when a swap partition is used on this HDD on kernel 6.16
2025-10-10 16:06 ` Pierre Juhen
@ 2025-10-14 6:29 ` Kairui Song
2025-10-14 6:57 ` Kairui Song
0 siblings, 1 reply; 6+ messages in thread
From: Kairui Song @ 2025-10-14 6:29 UTC (permalink / raw)
To: Pierre Juhen; +Cc: Andrew Morton, bugzilla-daemon, linux-mm
On Sat, Oct 11, 2025 at 3:42 AM Pierre Juhen <pierre.juhen@orange.fr> wrote:
>
> Hi Andrew and people reading those lists,
Hi Pierre,
>
> I still struggle to understand what is wrong with me.
>
> I rebuilt my mdadm + bcache array to be sure it was not something linked to a (too) old configuration. Upgrading to 6.16.10 didn't change anything.
>
> One symptom of the problem I identified is the time of the swapoff command (I have 32G, 12G are free and there is less than 1G in the swap)
>
> MiB Mem : 31952,0 total, 12646,2 free, 2226,8 used, 17587,5 buff/cache
> MiB Swap: 36000,0 total, 35212,2 free, 787,8 used. 29725,2 avail Mem
>
>
> $ time sudo swapoff -a
>
> real 6m41,276s
> user 0m0,005s
> sys 0m0,028s
>
> Waiting for more than 6 minutes elapsed for 33 ms of work is bizarre.
>
> During that time, if I try to launch another tab with iotop, nothing is displayed during at least one minute.
>
> During that time, my computer is almost idle.
>
> So it looks like swapoff and iotop are stucked, waiting for some kind of a semaphore inside the kernel.
>
> Any idea to dig further ?
>
> Thanks,
>
> Pierre
>
> Le 30/09/2025 à 01:07, Andrew Morton a écrit :
>
> (switched to email. Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
>
> On Sat, 27 Sep 2025 08:07:59 +0000 bugzilla-daemon@kernel.org wrote:
>
> https://bugzilla.kernel.org/show_bug.cgi?id=220605
>
> Bug ID: 220605
> Summary: Very slow access on HDD when a swap partition is used
> on this HDD on kernel 6.16
> Product: Memory Management
> Version: 2.5
> Hardware: Intel
> OS: Linux
> Status: NEW
> Severity: high
> Priority: P3
> Component: Page Allocator
> Assignee: akpm@linux-foundation.org
> Reporter: pierre.juhen@orange.fr
> Regression: No
>
> Hello everybody,
>
> I have a strange behavior on my system (Fedora 42 up to date => kernel 6.16.8).
>
> You certainly do. I wonder what we did to cause this.
>
> I'll cc the MM development list. Please prefer to report issues over
> email this way - we don't use bugzilla and it's rather annoying that MM
> bugzilla even exists.
>
>
> I have many disks but but my two swap partitions are on 2 HDD, sda and sdb,
> which are identical (SEAGATE Barracuda 2To). Swap partition on sda has a higher
> priority than the one on sdb.
>
> I don't use zram.
>
> When a swap partion is used, the disk on wich it seats is very slow.
>
> $ sudo hdparm -t /dev/sda
> /dev/sda:
> Timing buffered disk reads: 24 MB in 3.72 seconds = 6.46 MB/sec
>
> $ sudo hdparm -t /dev/sdb
> /dev/sdb:
> Timing buffered disk reads: 540 MB in 3.16 seconds = 170.85 MB/sec
> pierre@pierre:~$
>
> Sometimes, speed gets down to less than 2 MB/s.
>
> Lets turn swap off => swapoff -a
>
> $ sudo hdparm -t /dev/sdb
> /dev/sdb:
> Timing buffered disk reads: 462 MB in 3.00 seconds = 153.75 MB/sec
>
> $ sudo hdparm -t /dev/sda
>
> /dev/sda:
> Timing buffered disk reads: 474 MB in 3.00 seconds = 157.84 MB/sec
> pierre@pierre:~$
>
> Now speed is normal.
>
> Lets put swap on again => sudo swapon -a
>
> Speed is still normal :
>
> $ sudo hdparm -t /dev/sda
> /dev/sda:
> Timing buffered disk reads: 612 MB in 3.01 seconds = 203.56 MB/sec
> $ sudo hdparm -t /dev/sdb
> /dev/sdb:
> Timing buffered disk reads: 570 MB in 3.00 seconds = 189.95 MB/sec
> pierre@pierre:~$
>
> But swap is not used
>
> MiB Mem : 31952,0 total, 27091,7 free, 3057,6 used, 2343,0 buff/cache
> MiB Swap: 36000,0 total, 36000,0 free, 0,0 used. 28894,4 avail Mem
>
So the disk is already having a performance regression after swapon,
while there is no memory pressure and SWAP is not used yet?
Are you using swapon on the whole disk / partition or swapon on a file
on top of the disk / partition?
>
> So I suspect that, when the swap is used, there is some sort of lock that slows
> the whole disk that contains the swap partition that is being used.
>
> Logically, the system becomes very slow.
This is really strange if swap is not actually used. I just tested
with HDD locally and didn't find anything wrong about it.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Bug 220605] New: Very slow access on HDD when a swap partition is used on this HDD on kernel 6.16
2025-10-14 6:29 ` Kairui Song
@ 2025-10-14 6:57 ` Kairui Song
0 siblings, 0 replies; 6+ messages in thread
From: Kairui Song @ 2025-10-14 6:57 UTC (permalink / raw)
To: Pierre Juhen; +Cc: Andrew Morton, bugzilla-daemon, linux-mm
On Tue, Oct 14, 2025 at 2:29 PM Kairui Song <ryncsn@gmail.com> wrote:
>
> On Sat, Oct 11, 2025 at 3:42 AM Pierre Juhen <pierre.juhen@orange.fr> wrote:
> >
> > Hi Andrew and people reading those lists,
>
> Hi Pierre,
>
> > $ sudo hdparm -t /dev/sda
> > /dev/sda:
> > Timing buffered disk reads: 612 MB in 3.01 seconds = 203.56 MB/sec
> > $ sudo hdparm -t /dev/sdb
> > /dev/sdb:
> > Timing buffered disk reads: 570 MB in 3.00 seconds = 189.95 MB/sec
> > pierre@pierre:~$
> >
> > But swap is not used
> >
> > MiB Mem : 31952,0 total, 27091,7 free, 3057,6 used, 2343,0 buff/cache
> > MiB Swap: 36000,0 total, 36000,0 free, 0,0 used. 28894,4 avail Mem
> >
>
> So the disk is already having a performance regression after swapon,
> while there is no memory pressure and SWAP is not used yet?
>
> Are you using swapon on the whole disk / partition or swapon on a file
> on top of the disk / partition?
>
> >
> > So I suspect that, when the swap is used, there is some sort of lock that slows
> > the whole disk that contains the swap partition that is being used.
> >
> > Logically, the system becomes very slow.
>
> This is really strange if swap is not actually used. I just tested
> with HDD locally and didn't find anything wrong about it.
I just checked the bugzilla page more carefully, sorry I had some
misunderstanding of what is going on just now.
So usually when there is swap IO on an HDD, it will cause performance
regression because HDDs are slow and really bad at random IO. So if
you are seeing a lower performance with recent kernels, could you help
check if that's because the memory pressure is higher, or the kernel
prefers to do SWAP more aggressively now, or the performance is
just worse when swap is working? Maybe you can check and compare the
swap usage after the same workload before / after the regression?
I'll also try to test again locally with some slight pressure on swap.
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Bug 220605] New: Very slow access on HDD when a swap partition is used on this HDD on kernel 6.16
2025-09-29 23:07 ` [Bug 220605] New: Very slow access on HDD when a swap partition is used on this HDD on kernel 6.16 Andrew Morton
2025-10-10 16:06 ` Pierre Juhen
@ 2025-10-15 13:00 ` Pierre Juhen
2025-10-16 8:04 ` Pierre Juhen
2 siblings, 0 replies; 6+ messages in thread
From: Pierre Juhen @ 2025-10-15 13:00 UTC (permalink / raw)
To: Andrew Morton, Kairui Song; +Cc: bugzilla-daemon, linux-mm
[-- Attachment #1.1: Type: text/plain, Size: 14523 bytes --]
Hi, Andrew, Kairui,
I finally partially found the cause of the issue.
The guilty process was baloo_file_extract (part of the KDE file
indexer). The index was abnormally large, probably corrupted. I stopped
it, removed the index and restarted it, and it's now OK.
I found that by looking to the number of major page faults. In 2 hours,
the process had more than 300k major page faults, whilst a few other
processes (chrome, thunderbird) had only a few hundreds.
The program itself is small : 160k.
The question is now, why this faulty process was putting such a pressure
on the swap disk, whilst there was a lot of free memory ?
Use of mmap for the index ?
When mmap gets a page fault, it goes through the swap instead of looking
directly into the file ?
Thanks,
Regards,
Pierre
*_Iostat -x before _*
pierre@pierre:~$ iostat -x
Linux 6.16.11-200.fc42.x86_64 (pierre.juhen) 15/10/2025 _x86_64_
(12 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0,83 0,08 0,63 *13,28* 0,00 85,17
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz
w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s
drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
bcache0 87,97 3264,68 0,00 0,00 4,63 37,11 17,88
212,65 0,00 0,00 74,93 11,89 0,55 0,00 0,00 0,00
155,10 0,00 0,00 0,00 1,83 30,36
dm-0 0,21 4,82 0,00 0,00 53,06 23,42 0,01
0,03 0,00 0,00 32,43 2,86 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,01 0,38
dm-1 1,35 9,15 0,00 0,00 49,42 6,80 0,01
0,03 0,00 0,00 52,43 2,86 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,07 2,35
dm-10 50,24 891,59 0,00 0,00 6,53 17,75 11,12
224,31 0,00 0,00 192,55 20,16 0,40 542,69 0,00 0,00
165,62 1357,06 0,00 0,00 2,54 23,27
dm-2 0,04 2,55 0,00 0,00 23,14 70,10 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,08
dm-3 1,46 11,72 0,00 0,00 0,42 8,01 0,01
0,03 0,00 0,00 33,00 2,86 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,05
dm-4 50,28 893,54 0,00 0,00 6,50 17,77 11,12
224,31 0,00 0,00 76,19 20,16 0,40 542,69 0,00 0,00
165,62 1357,06 0,00 0,00 1,24 23,26
dm-5 36,18 2355,75 0,00 0,00 2,22 65,11 6,74
101,36 0,00 0,00 72,94 15,03 0,15 93,81 0,00 0,00
127,33 619,31 0,00 0,00 0,59 15,11
dm-6 0,08 1,81 0,00 0,00 0,28 21,39 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00
dm-7 0,08 1,81 0,00 0,00 0,15 21,39 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00
dm-8 0,13 2,84 0,00 0,00 0,95 21,55 0,00
0,00 0,00 0,00 11,00 2,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,01
*dm-9 84,74 341,28 0,00 0,00 18,21 4,03 110,88
443,53 0,00 0,00 175,27 4,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 20,98 95,69*
md126 5,36 388,18 0,00 0,00 95,94 72,40 17,66
325,71 0,00 0,00 63,39 18,44 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 1,63 17,62
md127 1,62 19,41 0,00 0,00 48,64 11,96 0,02
0,07 0,00 0,00 48,10 4,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,08 2,57
nvme0n1 66,66 2910,01 31,04 31,77 0,28 43,65 12,96
379,13 7,71 37,31 0,57 29,25 1,93 989,62 0,00 0,00
0,64 512,69 1,78 0,43 0,03 0,53
sda 60,79 557,38 28,51 31,92 21,92 9,17 33,52
615,59 87,88 72,39 40,19 18,37 0,44 318,50 0,00 0,00
116,42 731,53 0,95 233,14 2,95 92,36
sdb 2,60 204,98 0,46 14,97 11,26 78,94 4,41
153,72 4,19 48,69 42,32 34,84 0,25 318,00 0,00 0,00
22,67 1258,16 0,98 80,22 0,30 7,48
sdc 0,28 6,72 0,00 0,00 0,20 23,82 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00
sdd 0,09 2,00 0,00 0,00 0,25 22,45 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00
*_iostat -x now _*
iostat -x
Linux 6.16.11-200.fc42.x86_64 (pierre.juhen) 15/10/2025 _x86_64_(12 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0,75 0,01 0,45 2,04 0,00 96,74
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s
wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s
%drqm d_await dareq-sz f/s f_await aqu-sz %util
bcache0 35,15 1114,60 0,00 0,00 5,57 31,71 12,76
152,85 0,00 0,00 34,75 11,98 0,41 0,00 0,00
0,00 52,19 0,00 0,00 0,00 0,66 13,69
dm-0 0,08 1,84 0,00 0,00 48,12 22,33 0,00 0,00
0,00 0,00 36,50 1,20 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,27
dm-1 0,56 3,83 0,00 0,00 48,28 6,81 0,00 0,00
0,00 0,00 36,00 1,20 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,03 0,37
dm-10 20,26 234,41 0,00 0,00 7,64 11,57 9,76
155,66 0,00 0,00 45,01 15,95 0,35 312,56 0,00
0,00 44,31 889,08 0,00 0,00 0,61 10,78
dm-2 0,02 1,08 0,00 0,00 9,55 70,10 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,01
dm-3 0,62 4,42 0,00 0,00 0,32 7,15 0,00 0,00
0,00 0,00 45,60 1,20 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,02
dm-4 20,28 235,23 0,00 0,00 7,62 11,60 9,76
155,66 0,00 0,00 24,69 15,95 0,35 312,56 0,00
0,00 44,30 889,08 0,00 0,00 0,41 10,78
dm-5 14,24 873,40 0,00 0,00 2,89 61,35 3,00
44,04 0,00 0,00 67,53 14,70 0,06 50,70 0,00
0,00 99,84 870,69 0,00 0,00 0,25 5,45
dm-6 0,04 0,77 0,00 0,00 0,24 21,39 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00
dm-7 0,04 0,77 0,00 0,00 0,14 21,39 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00
dm-8 0,06 1,20 0,00 0,00 1,41 21,55 0,00 0,00
0,00 0,00 14,50 2,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,01
*dm-9 0,06 1,20 0,00 0,00 4,55 21,55 0,00 0,00
0,00 0,00 41,00 2,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,03*
md126 0,67 7,97 0,00 0,00 46,55
11,82 0,00 0,01 0,00 0,00 11,83 4,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,03 0,58
md127 1,67 104,20 0,00 0,00 156,95 62,51 12,44
199,71 0,00 0,00 26,66 16,06 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,59 5,12
nvme0n1 27,87 1025,06 11,83 29,79 0,29 36,78 8,69
168,45 5,10 36,99 0,75 19,38 0,83 424,68 0,00
0,00 0,69 512,68 1,42 0,39 0,02 0,28
sda 1,54 282,54 0,32 17,10 87,19 183,83 4,69 108,07
2,81 37,49 45,70 23,04 0,33 184,46 0,00
0,00 44,90 551,69 0,81 123,50 0,46 9,53
sdb 0,69 58,24 0,29 29,17 37,97 83,97 3,45
91,65 2,69 43,74 35,91 26,53 0,18 178,80 0,00
0,00 17,37 1006,71 0,81 89,78 0,23 6,83
sdc 0,12 2,84 0,00 0,00 0,20 23,82 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00
sdd 0,04 0,85 0,00 0,00 0,25 22,45 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00
Le 30/09/2025 à 01:07, Andrew Morton a écrit :
> (switched to email. Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
>
> On Sat, 27 Sep 2025 08:07:59 +0000bugzilla-daemon@kernel.org wrote:
>
>> https://bugzilla.kernel.org/show_bug.cgi?id=220605
>>
>> Bug ID: 220605
>> Summary: Very slow access on HDD when a swap partition is used
>> on this HDD on kernel 6.16
>> Product: Memory Management
>> Version: 2.5
>> Hardware: Intel
>> OS: Linux
>> Status: NEW
>> Severity: high
>> Priority: P3
>> Component: Page Allocator
>> Assignee:akpm@linux-foundation.org
>> Reporter:pierre.juhen@orange.fr
>> Regression: No
>>
>> Hello everybody,
>>
>> I have a strange behavior on my system (Fedora 42 up to date => kernel 6.16.8).
> You certainly do. I wonder what we did to cause this.
>
> I'll cc the MM development list. Please prefer to report issues over
> email this way - we don't use bugzilla and it's rather annoying that MM
> bugzilla even exists.
>
>
>> I have many disks but but my two swap partitions are on 2 HDD, sda and sdb,
>> which are identical (SEAGATE Barracuda 2To). Swap partition on sda has a higher
>> priority than the one on sdb.
>>
>> I don't use zram.
>>
>> When a swap partion is used, the disk on wich it seats is very slow.
>>
>> $ sudo hdparm -t /dev/sda
>> /dev/sda:
>> Timing buffered disk reads: 24 MB in 3.72 seconds = 6.46 MB/sec
>>
>> $ sudo hdparm -t /dev/sdb
>> /dev/sdb:
>> Timing buffered disk reads: 540 MB in 3.16 seconds = 170.85 MB/sec
>> pierre@pierre:~$
>>
>> Sometimes, speed gets down to less than 2 MB/s.
>>
>> Lets turn swap off => swapoff -a
>>
>> $ sudo hdparm -t /dev/sdb
>> /dev/sdb:
>> Timing buffered disk reads: 462 MB in 3.00 seconds = 153.75 MB/sec
>>
>> $ sudo hdparm -t /dev/sda
>>
>> /dev/sda:
>> Timing buffered disk reads: 474 MB in 3.00 seconds = 157.84 MB/sec
>> pierre@pierre:~$
>>
>> Now speed is normal.
>>
>> Lets put swap on again => sudo swapon -a
>>
>> Speed is still normal :
>>
>> $ sudo hdparm -t /dev/sda
>> /dev/sda:
>> Timing buffered disk reads: 612 MB in 3.01 seconds = 203.56 MB/sec
>> $ sudo hdparm -t /dev/sdb
>> /dev/sdb:
>> Timing buffered disk reads: 570 MB in 3.00 seconds = 189.95 MB/sec
>> pierre@pierre:~$
>>
>> But swap is not used
>>
>> MiB Mem : 31952,0 total, 27091,7 free, 3057,6 used, 2343,0 buff/cache
>> MiB Swap: 36000,0 total, 36000,0 free, 0,0 used. 28894,4 avail Mem
>>
>>
>> So I suspect that, when the swap is used, there is some sort of lock that slows
>> the whole disk that contains the swap partition that is being used.
>>
>> Logically, the system becomes very slow.
>>
>> There is nothing in the journal that seems related to this problem.
>>
>> Since the virtual memory has been modified in 6.16, I suspect a nasty bug.
>>
>> This affects 6.16.5 to 6.16.8.
>>
>> This appeared "recently", probably with 6.16
>>
>> Thanks,
>>
>> Regards,
>>
>> Pierre
>>
>> --
>> You may reply to this email to add a comment.
>>
>> You are receiving this mail because:
>> You are the assignee for the bug.
[-- Attachment #1.2: Type: text/html, Size: 32522 bytes --]
[-- Attachment #2: Copie d'écran_20251015_115726.png --]
[-- Type: image/png, Size: 296656 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread
* Re: [Bug 220605] New: Very slow access on HDD when a swap partition is used on this HDD on kernel 6.16
2025-09-29 23:07 ` [Bug 220605] New: Very slow access on HDD when a swap partition is used on this HDD on kernel 6.16 Andrew Morton
2025-10-10 16:06 ` Pierre Juhen
2025-10-15 13:00 ` Pierre Juhen
@ 2025-10-16 8:04 ` Pierre Juhen
2 siblings, 0 replies; 6+ messages in thread
From: Pierre Juhen @ 2025-10-16 8:04 UTC (permalink / raw)
To: Andrew Morton, Kairui Song; +Cc: bugzilla-daemon, linux-mm
[-- Attachment #1.1: Type: text/plain, Size: 15438 bytes --]
HI Andrew, Kairui,
I dug a bit around this issue.
The key-value database engine behind baloo is lmdb
<https://github.com/LMDB/lmdb> (https://github.com/LMDB/lmdb).
lmdb uses mmap.
The problems of baloo are not new. If you search "baloo_file CPU", you
will find hundreds of messages complaining about an heavy load on CPU
and RAM. (but not on the swap).
I already had the problem, that was quite easy to find with top.
Now something seems to have changed, with 6.16.
Instead of having high CPU usage we have a strong pressure on the swap disk.
It could mean that, when there is a filemap_fault (that's normal because
the index is not loaded in memory in one pass), it induces a page fault
that involves the swap.
The swap is therefore overloaded, and the system slows down dramatically.
So the key issue is to understand why we shift from a filemap_fault to a
major page fault.
Regards,
Pierre
Hi, Andrew, Kairui,
I finally partially found the cause of the issue.
The guilty process was baloo_file_extract (part of the KDE file
indexer). The index was abnormally large, probably corrupted. I stopped
it, removed the index and restarted it, and it's now OK.
I found that by looking to the number of major page faults. In 2 hours,
the process had more than 300k major page faults, whilst a few other
processes (chrome, thunderbird) had only a few hundreds.
The program itself is small : 160k.
The question is now, why this faulty process was putting such a pressure
on the swap disk, whilst there was a lot of free memory ?
Use of mmap for the index ?
When mmap gets a page fault, it goes through the swap instead of looking
directly into the file ?
Thanks,
Regards,
Pierre
*_Iostat -x before _*
pierre@pierre:~$ iostat -x
Linux 6.16.11-200.fc42.x86_64 (pierre.juhen) 15/10/2025 _x86_64_
(12 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0,83 0,08 0,63 *13,28* 0,00 85,17
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz
w/s wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s
drqm/s %drqm d_await dareq-sz f/s f_await aqu-sz %util
bcache0 87,97 3264,68 0,00 0,00 4,63 37,11 17,88
212,65 0,00 0,00 74,93 11,89 0,55 0,00 0,00 0,00
155,10 0,00 0,00 0,00 1,83 30,36
dm-0 0,21 4,82 0,00 0,00 53,06 23,42 0,01
0,03 0,00 0,00 32,43 2,86 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,01 0,38
dm-1 1,35 9,15 0,00 0,00 49,42 6,80 0,01
0,03 0,00 0,00 52,43 2,86 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,07 2,35
dm-10 50,24 891,59 0,00 0,00 6,53 17,75 11,12
224,31 0,00 0,00 192,55 20,16 0,40 542,69 0,00 0,00
165,62 1357,06 0,00 0,00 2,54 23,27
dm-2 0,04 2,55 0,00 0,00 23,14 70,10 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,08
dm-3 1,46 11,72 0,00 0,00 0,42 8,01 0,01
0,03 0,00 0,00 33,00 2,86 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,05
dm-4 50,28 893,54 0,00 0,00 6,50 17,77 11,12
224,31 0,00 0,00 76,19 20,16 0,40 542,69 0,00 0,00
165,62 1357,06 0,00 0,00 1,24 23,26
dm-5 36,18 2355,75 0,00 0,00 2,22 65,11 6,74
101,36 0,00 0,00 72,94 15,03 0,15 93,81 0,00 0,00
127,33 619,31 0,00 0,00 0,59 15,11
dm-6 0,08 1,81 0,00 0,00 0,28 21,39 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00
dm-7 0,08 1,81 0,00 0,00 0,15 21,39 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00
dm-8 0,13 2,84 0,00 0,00 0,95 21,55 0,00
0,00 0,00 0,00 11,00 2,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,01
*dm-9 84,74 341,28 0,00 0,00 18,21 4,03 110,88
443,53 0,00 0,00 175,27 4,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 20,98 95,69*
md126 5,36 388,18 0,00 0,00 95,94 72,40 17,66
325,71 0,00 0,00 63,39 18,44 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 1,63 17,62
md127 1,62 19,41 0,00 0,00 48,64 11,96 0,02
0,07 0,00 0,00 48,10 4,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,08 2,57
nvme0n1 66,66 2910,01 31,04 31,77 0,28 43,65 12,96
379,13 7,71 37,31 0,57 29,25 1,93 989,62 0,00 0,00
0,64 512,69 1,78 0,43 0,03 0,53
sda 60,79 557,38 28,51 31,92 21,92 9,17 33,52
615,59 87,88 72,39 40,19 18,37 0,44 318,50 0,00 0,00
116,42 731,53 0,95 233,14 2,95 92,36
sdb 2,60 204,98 0,46 14,97 11,26 78,94 4,41
153,72 4,19 48,69 42,32 34,84 0,25 318,00 0,00 0,00
22,67 1258,16 0,98 80,22 0,30 7,48
sdc 0,28 6,72 0,00 0,00 0,20 23,82 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00
sdd 0,09 2,00 0,00 0,00 0,25 22,45 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00
*_iostat -x now _*
iostat -x
Linux 6.16.11-200.fc42.x86_64 (pierre.juhen) 15/10/2025 _x86_64_(12 CPU)
avg-cpu: %user %nice %system %iowait %steal %idle
0,75 0,01 0,45 2,04 0,00 96,74
Device r/s rkB/s rrqm/s %rrqm r_await rareq-sz w/s
wkB/s wrqm/s %wrqm w_await wareq-sz d/s dkB/s drqm/s
%drqm d_await dareq-sz f/s f_await aqu-sz %util
bcache0 35,15 1114,60 0,00 0,00 5,57 31,71 12,76
152,85 0,00 0,00 34,75 11,98 0,41 0,00 0,00
0,00 52,19 0,00 0,00 0,00 0,66 13,69
dm-0 0,08 1,84 0,00 0,00 48,12 22,33 0,00 0,00
0,00 0,00 36,50 1,20 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,27
dm-1 0,56 3,83 0,00 0,00 48,28 6,81 0,00 0,00
0,00 0,00 36,00 1,20 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,03 0,37
dm-10 20,26 234,41 0,00 0,00 7,64 11,57 9,76
155,66 0,00 0,00 45,01 15,95 0,35 312,56 0,00
0,00 44,31 889,08 0,00 0,00 0,61 10,78
dm-2 0,02 1,08 0,00 0,00 9,55 70,10 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,01
dm-3 0,62 4,42 0,00 0,00 0,32 7,15 0,00 0,00
0,00 0,00 45,60 1,20 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,02
dm-4 20,28 235,23 0,00 0,00 7,62 11,60 9,76
155,66 0,00 0,00 24,69 15,95 0,35 312,56 0,00
0,00 44,30 889,08 0,00 0,00 0,41 10,78
dm-5 14,24 873,40 0,00 0,00 2,89 61,35 3,00
44,04 0,00 0,00 67,53 14,70 0,06 50,70 0,00
0,00 99,84 870,69 0,00 0,00 0,25 5,45
dm-6 0,04 0,77 0,00 0,00 0,24 21,39 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00
dm-7 0,04 0,77 0,00 0,00 0,14 21,39 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00
dm-8 0,06 1,20 0,00 0,00 1,41 21,55 0,00 0,00
0,00 0,00 14,50 2,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,01
*dm-9 0,06 1,20 0,00 0,00 4,55 21,55 0,00 0,00
0,00 0,00 41,00 2,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,03*
md126 0,67 7,97 0,00 0,00 46,55
11,82 0,00 0,01 0,00 0,00 11,83 4,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,03 0,58
md127 1,67 104,20 0,00 0,00 156,95 62,51 12,44
199,71 0,00 0,00 26,66 16,06 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,59 5,12
nvme0n1 27,87 1025,06 11,83 29,79 0,29 36,78 8,69
168,45 5,10 36,99 0,75 19,38 0,83 424,68 0,00
0,00 0,69 512,68 1,42 0,39 0,02 0,28
sda 1,54 282,54 0,32 17,10 87,19 183,83 4,69 108,07
2,81 37,49 45,70 23,04 0,33 184,46 0,00
0,00 44,90 551,69 0,81 123,50 0,46 9,53
sdb 0,69 58,24 0,29 29,17 37,97 83,97 3,45
91,65 2,69 43,74 35,91 26,53 0,18 178,80 0,00
0,00 17,37 1006,71 0,81 89,78 0,23 6,83
sdc 0,12 2,84 0,00 0,00 0,20 23,82 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00
sdd 0,04 0,85 0,00 0,00 0,25 22,45 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00 0,00 0,00
0,00 0,00 0,00 0,00 0,00 0,00
Le 30/09/2025 à 01:07, Andrew Morton a écrit :
> (switched to email. Please respond via emailed reply-to-all, not via the
> bugzilla web interface).
>
> On Sat, 27 Sep 2025 08:07:59 +0000bugzilla-daemon@kernel.org wrote:
>
>> https://bugzilla.kernel.org/show_bug.cgi?id=220605
>>
>> Bug ID: 220605
>> Summary: Very slow access on HDD when a swap partition is used
>> on this HDD on kernel 6.16
>> Product: Memory Management
>> Version: 2.5
>> Hardware: Intel
>> OS: Linux
>> Status: NEW
>> Severity: high
>> Priority: P3
>> Component: Page Allocator
>> Assignee:akpm@linux-foundation.org
>> Reporter:pierre.juhen@orange.fr
>> Regression: No
>>
>> Hello everybody,
>>
>> I have a strange behavior on my system (Fedora 42 up to date => kernel 6.16.8).
> You certainly do. I wonder what we did to cause this.
>
> I'll cc the MM development list. Please prefer to report issues over
> email this way - we don't use bugzilla and it's rather annoying that MM
> bugzilla even exists.
>
>
>> I have many disks but but my two swap partitions are on 2 HDD, sda and sdb,
>> which are identical (SEAGATE Barracuda 2To). Swap partition on sda has a higher
>> priority than the one on sdb.
>>
>> I don't use zram.
>>
>> When a swap partion is used, the disk on wich it seats is very slow.
>>
>> $ sudo hdparm -t /dev/sda
>> /dev/sda:
>> Timing buffered disk reads: 24 MB in 3.72 seconds = 6.46 MB/sec
>>
>> $ sudo hdparm -t /dev/sdb
>> /dev/sdb:
>> Timing buffered disk reads: 540 MB in 3.16 seconds = 170.85 MB/sec
>> pierre@pierre:~$
>>
>> Sometimes, speed gets down to less than 2 MB/s.
>>
>> Lets turn swap off => swapoff -a
>>
>> $ sudo hdparm -t /dev/sdb
>> /dev/sdb:
>> Timing buffered disk reads: 462 MB in 3.00 seconds = 153.75 MB/sec
>>
>> $ sudo hdparm -t /dev/sda
>>
>> /dev/sda:
>> Timing buffered disk reads: 474 MB in 3.00 seconds = 157.84 MB/sec
>> pierre@pierre:~$
>>
>> Now speed is normal.
>>
>> Lets put swap on again => sudo swapon -a
>>
>> Speed is still normal :
>>
>> $ sudo hdparm -t /dev/sda
>> /dev/sda:
>> Timing buffered disk reads: 612 MB in 3.01 seconds = 203.56 MB/sec
>> $ sudo hdparm -t /dev/sdb
>> /dev/sdb:
>> Timing buffered disk reads: 570 MB in 3.00 seconds = 189.95 MB/sec
>> pierre@pierre:~$
>>
>> But swap is not used
>>
>> MiB Mem : 31952,0 total, 27091,7 free, 3057,6 used, 2343,0 buff/cache
>> MiB Swap: 36000,0 total, 36000,0 free, 0,0 used. 28894,4 avail Mem
>>
>>
>> So I suspect that, when the swap is used, there is some sort of lock that slows
>> the whole disk that contains the swap partition that is being used.
>>
>> Logically, the system becomes very slow.
>>
>> There is nothing in the journal that seems related to this problem.
>>
>> Since the virtual memory has been modified in 6.16, I suspect a nasty bug.
>>
>> This affects 6.16.5 to 6.16.8.
>>
>> This appeared "recently", probably with 6.16
>>
>> Thanks,
>>
>> Regards,
>>
>> Pierre
>>
>> --
>> You may reply to this email to add a comment.
>>
>> You are receiving this mail because:
>> You are the assignee for the bug.
[-- Attachment #1.2: Type: text/html, Size: 34090 bytes --]
[-- Attachment #2: Copie d'écran_20251015_115726.png --]
[-- Type: image/png, Size: 296656 bytes --]
^ permalink raw reply [flat|nested] 6+ messages in thread