Hard system lock-ups when using encrypted swap and RAM is exhausted

Hard system lock-ups when using encrypted swap and RAM is exhausted

[Aaron Rainbolt]

[-- Attachment #1: Type: text/plain, Size: 3530 bytes --]

Not sure if this is a memory management issue, a LUKS issue, or both,
so I wrote both mailing lists.

I'm seeing an issue with both the latest mainline kernel (6.18-rc5) and
Debian 13's 6.12 kernel package. When physical memory fills up, the
entire system locks up hard, as if it hit rather severe thrashing,
despite the fact that there appears to be disk cache that can still be
evicted, and there is ample amounts of swap space remaining (gigabytes
of it). This issue did not occur with the 6.1 kernel in Debian 12. I'm
seeing this occur in very low-memory Debian VMs, with between 512 and
900 MB RAM, running under VirtualBox and KVM. (I suspect, but have not
verified, that I'm seeing similar behavior under Xen as well.) These
VMs generally use a swappiness of 1, though I have seen a lockup occur
even with a swappiness of 60. The filesystem in use, in case it
matters, is ext4.

To reproduce on a system running Linux 6.18-rc5, with :

* Follow the steps from
  https://gitlab.com/cryptsetup/cryptsetup/-/wikis/FrequentlyAskedQuestions,
  section "2.3 How do I set up encrypted swap?", but creating a
  swapfile rather than a swap partition. I created an 8 GB swapfile
  with fallocate. Reboot the system when done.
* In a TTY, open a terminal multiplexer (or something you can abuse as
  one, Vim works well), and open two terminals. In one terminal, run
  `htop` so you can observe memory and swap usage.
* In the `htop` terminal, sort by M_RESIDENT.
* In the other terminal, create a new file `test.py`, that will
  gradually fill memory at a relatively fast pace and print an
  indicator that it's still alive. I used the following code for this:

    import time

    count = 0
    mem_list = []
    while True:
        mem_list.append([x for x in range(2048)])
        count += 1
        time.sleep(0.002)
        print(count)

* Run the script with `python3 test.py`.
* While the script runs, observe the growing memory usage in `htop`.
  Swap usage should start at or near 0, RAM usage will gradually
  increase. Once RAM usage starts getting high, some data will start
  being swapped out as expected, but after a short while the whole VM
  will lock up despite there being gigabytes of swap left. (On my KVM
  VM, the last time htop updated its screen, it showed RAM usage of
  712M/846M, and swap usage of 328M/7.40G. The python3 process
  running the script was consuming 551M memory. The VM is entirely
  unresponsive. Incidentally, the python3 process also was in
  uninterruptible sleep when htop last updated its screen, but that
  could mean nothing since it might have come out of sleep between the
  last screen update and the VM lockup.)

Under Bookworm with Linux 6.1, the Python script would occasionally
freeze, but the VM would remain responsive, and the script would
eventually resume. Even with kernel 6.12, both unencrypted swapfiles and
swapfiles that are technically unencrypted but live on a LUKS volume
both behave as expected. It's only swapfiles that are themselves
encrypted that seem to trigger these lockups.

I haven't looked at the code at all, but it seems like maybe memory
LUKS needs available in order to operate is being consumed, thus
making it impossible to swap anything in and out of the swapfile? That
seems like it would cause these symptoms or similar, though I don't
know.

Let me know if I can provide any further information on the issue. I'm
happy to bisect the kernel if it will help.

--
Aaron

[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Hard system lock-ups when using encrypted swap and RAM is exhausted

[Milan Broz]

Hi,

On 11/12/25 6:18 AM, Aaron Rainbolt wrote:
> Not sure if this is a memory management issue, a LUKS issue, or both,
> so I wrote both mailing lists.

It is not a LUKS issue; cryptsetup/LUKS activates the encrypted device,
so it is only the kernel/dm-crypt handling IOs.

Adding cc to dm-devel as this would be another combination device-mapper
and encrypted swap that could cause issues...

However, could you please specify exactly your storage configuration?

 From the subject, I expected you to have an encrypted swap, but it is
not clear if there are other encrypted devices.

Please paste at least lsblk, lsblk -f output, and also luksDump
(or crypttab if it is not LUKS) for LUKS/dm-crypt configuration.

Thanks,
Milan


> 
> I'm seeing an issue with both the latest mainline kernel (6.18-rc5) and
> Debian 13's 6.12 kernel package. When physical memory fills up, the
> entire system locks up hard, as if it hit rather severe thrashing,
> despite the fact that there appears to be disk cache that can still be
> evicted, and there is ample amounts of swap space remaining (gigabytes
> of it). This issue did not occur with the 6.1 kernel in Debian 12. I'm
> seeing this occur in very low-memory Debian VMs, with between 512 and
> 900 MB RAM, running under VirtualBox and KVM. (I suspect, but have not
> verified, that I'm seeing similar behavior under Xen as well.) These
> VMs generally use a swappiness of 1, though I have seen a lockup occur
> even with a swappiness of 60. The filesystem in use, in case it
> matters, is ext4.
> 
> To reproduce on a system running Linux 6.18-rc5, with :
> 
> * Follow the steps from
>    https://gitlab.com/cryptsetup/cryptsetup/-/wikis/FrequentlyAskedQuestions,
>    section "2.3 How do I set up encrypted swap?", but creating a
>    swapfile rather than a swap partition. I created an 8 GB swapfile
>    with fallocate. Reboot the system when done.
> * In a TTY, open a terminal multiplexer (or something you can abuse as
>    one, Vim works well), and open two terminals. In one terminal, run
>    `htop` so you can observe memory and swap usage.
> * In the `htop` terminal, sort by M_RESIDENT.
> * In the other terminal, create a new file `test.py`, that will
>    gradually fill memory at a relatively fast pace and print an
>    indicator that it's still alive. I used the following code for this:
> 
>      import time
> 
>      count = 0
>      mem_list = []
>      while True:
>          mem_list.append([x for x in range(2048)])
>          count += 1
>          time.sleep(0.002)
>          print(count)
> 
> * Run the script with `python3 test.py`.
> * While the script runs, observe the growing memory usage in `htop`.
>    Swap usage should start at or near 0, RAM usage will gradually
>    increase. Once RAM usage starts getting high, some data will start
>    being swapped out as expected, but after a short while the whole VM
>    will lock up despite there being gigabytes of swap left. (On my KVM
>    VM, the last time htop updated its screen, it showed RAM usage of
>    712M/846M, and swap usage of 328M/7.40G. The python3 process
>    running the script was consuming 551M memory. The VM is entirely
>    unresponsive. Incidentally, the python3 process also was in
>    uninterruptible sleep when htop last updated its screen, but that
>    could mean nothing since it might have come out of sleep between the
>    last screen update and the VM lockup.)
> 
> Under Bookworm with Linux 6.1, the Python script would occasionally
> freeze, but the VM would remain responsive, and the script would
> eventually resume. Even with kernel 6.12, both unencrypted swapfiles and
> swapfiles that are technically unencrypted but live on a LUKS volume
> both behave as expected. It's only swapfiles that are themselves
> encrypted that seem to trigger these lockups.
> 
> I haven't looked at the code at all, but it seems like maybe memory
> LUKS needs available in order to operate is being consumed, thus
> making it impossible to swap anything in and out of the swapfile? That
> seems like it would cause these symptoms or similar, though I don't
> know.
> 
> Let me know if I can provide any further information on the issue. I'm
> happy to bisect the kernel if it will help.
> 
> --
> Aaron

Re: Hard system lock-ups when using encrypted swap and RAM is exhausted

[Mikulas Patocka]

On Thu, 27 Nov 2025, Milan Broz wrote:

> Hi,
> 
> On 11/12/25 6:18 AM, Aaron Rainbolt wrote:
> > Not sure if this is a memory management issue, a LUKS issue, or both,
> > so I wrote both mailing lists.
> 
> It is not a LUKS issue; cryptsetup/LUKS activates the encrypted device,
> so it is only the kernel/dm-crypt handling IOs.
> 
> Adding cc to dm-devel as this would be another combination device-mapper
> and encrypted swap that could cause issues...
> 
> However, could you please specify exactly your storage configuration?
> 
> From the subject, I expected you to have an encrypted swap, but it is
> not clear if there are other encrypted devices.
> 
> Please paste at least lsblk, lsblk -f output, and also luksDump
> (or crypttab if it is not LUKS) for LUKS/dm-crypt configuration.
> 
> Thanks,
> Milan
> 
> 
> > 
> > I'm seeing an issue with both the latest mainline kernel (6.18-rc5) and
> > Debian 13's 6.12 kernel package. When physical memory fills up, the
> > entire system locks up hard, as if it hit rather severe thrashing,
> > despite the fact that there appears to be disk cache that can still be
> > evicted, and there is ample amounts of swap space remaining (gigabytes
> > of it). This issue did not occur with the 6.1 kernel in Debian 12. I'm
> > seeing this occur in very low-memory Debian VMs, with between 512 and
> > 900 MB RAM, running under VirtualBox and KVM. (I suspect, but have not
> > verified, that I'm seeing similar behavior under Xen as well.) These
> > VMs generally use a swappiness of 1, though I have seen a lockup occur
> > even with a swappiness of 60. The filesystem in use, in case it
> > matters, is ext4.
> > 
> > To reproduce on a system running Linux 6.18-rc5, with :
> > 
> > * Follow the steps from
> >    https://gitlab.com/cryptsetup/cryptsetup/-/wikis/FrequentlyAskedQuestions,
> >    section "2.3 How do I set up encrypted swap?", but creating a
> >    swapfile rather than a swap partition.

Hi

Encrypted swap file is not supposed to work. It uses the loop device that 
routes the requests to a filesystem and the filesystem needs to allocate 
memory to process requests.

So, this is what happened to you - the machine runs out of memory, it 
needs to swap out some pages, dm-crypt encrypts the pages and generates 
write bios, the write bios are directed to the loop device, the loop 
device directs them to the filesystem, the filesystem attempts to allocate 
more memory => deadlock.

I got the deadlock with 6.18-rc4 when I used dm-crypt on a file and I 
didn't get the deadlock when I used dm-crypt on a SCSI block device. That 
is expected behavior.

Note that the in-kernel OOM killer sometimes doesn't kill the application 
and discards read-only program pages (which generates big I/O churn and 
general system slowdown) - if you are hitting this problem, I recommend 
installing userspace OOM killer, such as earlyoom.

Mikulas

Re: Hard system lock-ups when using encrypted swap and RAM is exhausted

[Aaron Rainbolt]

[-- Attachment #1: Type: text/plain, Size: 4161 bytes --]

On Thu, 27 Nov 2025 18:54:04 +0100 (CET)
Mikulas Patocka <mpatocka@redhat.com> wrote:

> On Thu, 27 Nov 2025, Milan Broz wrote:
> 
> > Hi,
> > 
> > On 11/12/25 6:18 AM, Aaron Rainbolt wrote:  
> > > Not sure if this is a memory management issue, a LUKS issue, or
> > > both, so I wrote both mailing lists.  
> > 
> > It is not a LUKS issue; cryptsetup/LUKS activates the encrypted
> > device, so it is only the kernel/dm-crypt handling IOs.
> > 
> > Adding cc to dm-devel as this would be another combination
> > device-mapper and encrypted swap that could cause issues...
> > 
> > However, could you please specify exactly your storage
> > configuration?
> > 
> > From the subject, I expected you to have an encrypted swap, but it
> > is not clear if there are other encrypted devices.
> > 
> > Please paste at least lsblk, lsblk -f output, and also luksDump
> > (or crypttab if it is not LUKS) for LUKS/dm-crypt configuration.
> > 
> > Thanks,
> > Milan
> > 
> >   
> > > 
> > > I'm seeing an issue with both the latest mainline kernel
> > > (6.18-rc5) and Debian 13's 6.12 kernel package. When physical
> > > memory fills up, the entire system locks up hard, as if it hit
> > > rather severe thrashing, despite the fact that there appears to
> > > be disk cache that can still be evicted, and there is ample
> > > amounts of swap space remaining (gigabytes of it). This issue did
> > > not occur with the 6.1 kernel in Debian 12. I'm seeing this occur
> > > in very low-memory Debian VMs, with between 512 and 900 MB RAM,
> > > running under VirtualBox and KVM. (I suspect, but have not
> > > verified, that I'm seeing similar behavior under Xen as well.)
> > > These VMs generally use a swappiness of 1, though I have seen a
> > > lockup occur even with a swappiness of 60. The filesystem in use,
> > > in case it matters, is ext4.
> > > 
> > > To reproduce on a system running Linux 6.18-rc5, with :
> > > 
> > > * Follow the steps from
> > >    https://gitlab.com/cryptsetup/cryptsetup/-/wikis/FrequentlyAskedQuestions,
> > >    section "2.3 How do I set up encrypted swap?", but creating a
> > >    swapfile rather than a swap partition.  
> 
> Hi
> 
> Encrypted swap file is not supposed to work. It uses the loop device
> that routes the requests to a filesystem and the filesystem needs to
> allocate memory to process requests.
> 
> So, this is what happened to you - the machine runs out of memory, it 
> needs to swap out some pages, dm-crypt encrypts the pages and
> generates write bios, the write bios are directed to the loop device,
> the loop device directs them to the filesystem, the filesystem
> attempts to allocate more memory => deadlock.
> 
> I got the deadlock with 6.18-rc4 when I used dm-crypt on a file and I 
> didn't get the deadlock when I used dm-crypt on a SCSI block device.
> That is expected behavior.

Is it only expected behavior since some time after kernel 6.1, or has
it always been expected behavior and encrypted swapfiles simply worked
by accident with kernel 6.1? Is there any reasonable way to reserve
some memory for in-kernel filesystem code (at least for some
filesystems like ext4 in the event it's not feasible for all of them)
that will ensure it has enough memory to handle I/O operations even if
the system is completely out of memory from userspace's perspective?
I'd be happy to try to contribute a fix if possible.

With kernel 6.1, this was working reliably, and Kicksecure (a
security-focused Debian derivative with a relatively sizable userbase)
was using encrypted swapfiles by default. We never got any reports of
lockups like we're seeing now with kernel 6.12. Whether this was
intended to work or not before, it seemed to work very well, and this
seems like a regression from our standpoint.

--
Aaron

> Note that the in-kernel OOM killer sometimes doesn't kill the
> application and discards read-only program pages (which generates big
> I/O churn and general system slowdown) - if you are hitting this
> problem, I recommend installing userspace OOM killer, such as
> earlyoom.
> 
> Mikulas
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Hard system lock-ups when using encrypted swap and RAM is exhausted

[Kurt Fitzner]

On 2025-11-27 13:54, Mikulas Patocka wrote:

> Encrypted swap file is not supposed to work.

Do you have a reference for this?  The concept of encrypted swap files 
has been a valid workflow for a very long time.

> So, this is what happened to you - the machine runs out of memory, it
> needs to swap out some pages, dm-crypt encrypts the pages and generates
> write bios, the write bios are directed to the loop device, the loop
> device directs them to the filesystem, the filesystem attempts to 
> allocate
> more memory => deadlock.

If it's the filesystem trying to allocate memory on writes to a swap 
file that is causing a memory allocation/swap race, then any write to a 
swap file would engender the same result, regardless of encryption. The 
encryption layer is redundant under the failure mode you propose.

I can confirm I have put kernels up to and including 6.14 under heavy 
memory stress and have never encountered anything that feels like a 
memory allocation race.  All my systems have encrypted swap files.

I can't speak toward later kernels, or any bugs that may or may not be 
presesnt, but I know of nothing to suggest that encrypted swap files 
remain anything other than an intended feature.

     Kurt

Re: Hard system lock-ups when using encrypted swap and RAM is exhausted

[Mikulas Patocka]

On Thu, 27 Nov 2025, Kurt Fitzner wrote:

> On 2025-11-27 13:54, Mikulas Patocka wrote:
> 
> > Encrypted swap file is not supposed to work.
> 
> Do you have a reference for this?  The concept of encrypted swap files has
> been a valid workflow for a very long time.
> 
> > So, this is what happened to you - the machine runs out of memory, it
> > needs to swap out some pages, dm-crypt encrypts the pages and generates
> > write bios, the write bios are directed to the loop device, the loop
> > device directs them to the filesystem, the filesystem attempts to allocate
> > more memory => deadlock.
> 
> If it's the filesystem trying to allocate memory on writes to a swap file that
> is causing a memory allocation/swap race, then any write to a swap file would
> engender the same result, regardless of encryption. The encryption layer is
> redundant under the failure mode you propose.

No - when you swap to unencrypted file, the kernel bypasses the 
filesystem. It creates a block map of the file at swap activation time and 
when it swaps to the file it uses this block map and it doesn't call any 
filesystem functions.

This bypass doesn't work when you put other block device drivers over the 
file.

> I can confirm I have put kernels up to and including 6.14 under heavy memory
> stress and have never encountered anything that feels like a memory allocation
> race.  All my systems have encrypted swap files.
> 
> I can't speak toward later kernels, or any bugs that may or may not be
> presesnt, but I know of nothing to suggest that encrypted swap files remain
> anything other than an intended feature.
> 
>     Kurt

So, it works by chance, not by design.

Mikulas

Re: Hard system lock-ups when using encrypted swap and RAM is exhausted

[Aaron Rainbolt]

[-- Attachment #1: Type: text/plain, Size: 6975 bytes --]

On Thu, 27 Nov 2025 08:59:13 +0100
Milan Broz <gmazyland@gmail.com> wrote:

> Hi,
> 
> On 11/12/25 6:18 AM, Aaron Rainbolt wrote:
> > Not sure if this is a memory management issue, a LUKS issue, or
> > both, so I wrote both mailing lists.  
> 
> It is not a LUKS issue; cryptsetup/LUKS activates the encrypted
> device, so it is only the kernel/dm-crypt handling IOs.
> 
> Adding cc to dm-devel as this would be another combination
> device-mapper and encrypted swap that could cause issues...
> 
> However, could you please specify exactly your storage configuration?
> 
>  From the subject, I expected you to have an encrypted swap, but it is
> not clear if there are other encrypted devices.
>
> Please paste at least lsblk, lsblk -f output, and also luksDump
> (or crypttab if it is not LUKS) for LUKS/dm-crypt configuration.

Mikulas seems to have reproduced the issue already, but just in case
it's still useful, here's the requested data, along with /etc/fstab.
(/dev/sda2 is a BIOS boot partition.)

-----

[sysmaint ~]% lsblk    
NAME   MAJ:MIN RM  SIZE RO TYPE  MOUNTPOINTS
loop0    7:0    0    4G  0 loop  
└─swap 254:0    0    4G  0 crypt [SWAP]
sda      8:0    0  100G  0 disk  
├─sda1   8:1    0  100M  0 part  /boot/efi
├─sda2   8:2    0    1M  0 part  
└─sda3   8:3    0 99.9G  0 part  /

-----

[sysmaint ~]% lsblk -f
NAME   FSTYPE FSVER LABEL UUID                                 FSAVAIL FSUSE% MOUNTPOINTS
loop0  swap   1           0a06fb5a-39f5-4a89-b8a8-fdc2470ee0f3    
└─swap swap   1     swap  f9aa44bb-2214-4a32-ac7e-34360b0f313b                [SWAP]
sda    
├─sda1 vfat   FAT32 EFI   6907-5000                                           /boot/efi
├─sda2    
└─sda3 ext4   1.0         26ada0c0-1165-4098-884d-aafd2220c2c6   64.7G    29% /

-----

[sysmaint ~]% cat /etc/crypttab    
# <target name> <source device>         <key file>      <options>
swap /swapfile /dev/urandom swap,noearly

-----

[sysmaint ~]% cat /etc/fstab    
# /etc/fstab - created by grml-debootstrap on Sun Nov  2 20:23:37 UTC 2025
# Accessible filesystems, by reference, are maintained under '/dev/disk/'.
# See man pages fstab(5), findfs(8), mount(8) and/or blkid(8) for more info.
#
# After editing this file, run 'systemctl daemon-reload' to update systemd
# units generated from this file.
#
/dev/disk/by-uuid/26ada0c0-1165-4098-884d-aafd2220c2c6 /  auto    defaults,errors=remount-ro,x-systemd.growfs 0   1   
UUID=6907-5000  /boot/efi       vfat    umask=0077      0       1   
proc           /proc        proc    defaults                      0   0   
/dev/cdrom     /mnt/cdrom0  iso9660 ro,user,noauto                0   0   
# some other examples:
# /dev/sda2       none         swap    sw,pri=0             0   0   
# /dev/hda1       /Grml        ext3    dev,suid,user,noauto 0  2
# //1.2.3.4/pub   /smb/pub     cifs    user,noauto,uid=grml,gid=grml 0 0 
# linux:/pub      /beer        nfs     defaults             0  0
# tmpfs           /tmp         tmpfs   size=300M            0  0
# /dev/sda5       none         swap    sw                   0  0
/dev/mapper/swap none swap sw 0 0

--
Aaron

> Thanks,
> Milan
> 
> 
> > 
> > I'm seeing an issue with both the latest mainline kernel (6.18-rc5)
> > and Debian 13's 6.12 kernel package. When physical memory fills up,
> > the entire system locks up hard, as if it hit rather severe
> > thrashing, despite the fact that there appears to be disk cache
> > that can still be evicted, and there is ample amounts of swap space
> > remaining (gigabytes of it). This issue did not occur with the 6.1
> > kernel in Debian 12. I'm seeing this occur in very low-memory
> > Debian VMs, with between 512 and 900 MB RAM, running under
> > VirtualBox and KVM. (I suspect, but have not verified, that I'm
> > seeing similar behavior under Xen as well.) These VMs generally use
> > a swappiness of 1, though I have seen a lockup occur even with a
> > swappiness of 60. The filesystem in use, in case it matters, is
> > ext4.
> > 
> > To reproduce on a system running Linux 6.18-rc5, with :
> > 
> > * Follow the steps from
> >    https://gitlab.com/cryptsetup/cryptsetup/-/wikis/FrequentlyAskedQuestions,
> >    section "2.3 How do I set up encrypted swap?", but creating a
> >    swapfile rather than a swap partition. I created an 8 GB swapfile
> >    with fallocate. Reboot the system when done.
> > * In a TTY, open a terminal multiplexer (or something you can abuse
> > as one, Vim works well), and open two terminals. In one terminal,
> > run `htop` so you can observe memory and swap usage.
> > * In the `htop` terminal, sort by M_RESIDENT.
> > * In the other terminal, create a new file `test.py`, that will
> >    gradually fill memory at a relatively fast pace and print an
> >    indicator that it's still alive. I used the following code for
> > this:
> > 
> >      import time
> > 
> >      count = 0
> >      mem_list = []
> >      while True:
> >          mem_list.append([x for x in range(2048)])
> >          count += 1
> >          time.sleep(0.002)
> >          print(count)
> > 
> > * Run the script with `python3 test.py`.
> > * While the script runs, observe the growing memory usage in `htop`.
> >    Swap usage should start at or near 0, RAM usage will gradually
> >    increase. Once RAM usage starts getting high, some data will
> > start being swapped out as expected, but after a short while the
> > whole VM will lock up despite there being gigabytes of swap left.
> > (On my KVM VM, the last time htop updated its screen, it showed RAM
> > usage of 712M/846M, and swap usage of 328M/7.40G. The python3
> > process running the script was consuming 551M memory. The VM is
> > entirely unresponsive. Incidentally, the python3 process also was in
> >    uninterruptible sleep when htop last updated its screen, but that
> >    could mean nothing since it might have come out of sleep between
> > the last screen update and the VM lockup.)
> > 
> > Under Bookworm with Linux 6.1, the Python script would occasionally
> > freeze, but the VM would remain responsive, and the script would
> > eventually resume. Even with kernel 6.12, both unencrypted
> > swapfiles and swapfiles that are technically unencrypted but live
> > on a LUKS volume both behave as expected. It's only swapfiles that
> > are themselves encrypted that seem to trigger these lockups.
> > 
> > I haven't looked at the code at all, but it seems like maybe memory
> > LUKS needs available in order to operate is being consumed, thus
> > making it impossible to swap anything in and out of the swapfile?
> > That seems like it would cause these symptoms or similar, though I
> > don't know.
> > 
> > Let me know if I can provide any further information on the issue.
> > I'm happy to bisect the kernel if it will help.
> > 
> > --
> > Aaron  
> 


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]