From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id DCFD1C25B76 for ; Mon, 3 Jun 2024 12:02:30 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 58D8E6B0093; Mon, 3 Jun 2024 08:02:30 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 53D236B0095; Mon, 3 Jun 2024 08:02:30 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 3DDA06B0098; Mon, 3 Jun 2024 08:02:30 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 1FA7B6B0093 for ; Mon, 3 Jun 2024 08:02:30 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id B6777C0E13 for ; Mon, 3 Jun 2024 12:02:29 +0000 (UTC) X-FDA: 82189440018.14.7A42E30 Received: from mail-ed1-f52.google.com (mail-ed1-f52.google.com [209.85.208.52]) by imf14.hostedemail.com (Postfix) with ESMTP id A4A2D100028 for ; Mon, 3 Jun 2024 12:02:27 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=ISE9TsBL; spf=pass (imf14.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.208.52 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1717416147; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WMOJmLS/Nn9J0SYKlSWYVE6j+qu58H+1OHJ/cgq+xrA=; b=aLKM9RhsAvh7KIiZhsyUkSMIveZbiCrH3sloBrdZ6gxziqD3mRcdkr3LrT2nOWv1YQTXZW xKeC2MVdNab+u1O/3E2iozmXIL+k16RKJNuWJ+pPkIHDE4/X06Qc/MN9SxlGZMtqeZPNZ7 UM53rTasF1OBc5t5kegYRON6gEIJwC4= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=rivosinc-com.20230601.gappssmtp.com header.s=20230601 header.b=ISE9TsBL; spf=pass (imf14.hostedemail.com: domain of alexghiti@rivosinc.com designates 209.85.208.52 as permitted sender) smtp.mailfrom=alexghiti@rivosinc.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1717416147; a=rsa-sha256; cv=none; b=dP0o+aoyKPweVQav0mK4DrxRl/XG2e9MlVLZK0VP+leY7gyz26rKB4czhvitBSj2F4d7kw zkdwWru6iy+eIGWflAOmvN1xHN8LShxgOvpzE1wzhdaXaaF74Rta5FJKiQSBEFUPCrh3aM FAH2ON3oeKyqPJuq250WTUpciuBw6Ow= Received: by mail-ed1-f52.google.com with SMTP id 4fb4d7f45d1cf-57a2ed9af7dso4224028a12.1 for ; Mon, 03 Jun 2024 05:02:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=rivosinc-com.20230601.gappssmtp.com; s=20230601; t=1717416146; x=1718020946; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=WMOJmLS/Nn9J0SYKlSWYVE6j+qu58H+1OHJ/cgq+xrA=; b=ISE9TsBLPvrYQmpK7po88Ubk+qkCTzC2/TQT6iq7Wa6xTF09hqbi5MK9VauAeNRlPa hJP0c3pExVWTEAglo60YV6pbYKCx97pOVmGtGMkxnKGlRXyVrVrSVEGW29d9+0vK0eqT RSYpHrMX7xhGUeGvXOD8dGrxQKR+L1VZtRH8PqPsvs+EuBdM1qmVeN9hllE9Sl6zfSg6 iMWDpBozreAHJLJ0dGk6RMFK4CN7/Ldi/R/ThrS5UD4WhRxdE/DUFRBIuF9B9rOvxy3+ KWoFna5/RTvUYR9aXie3lYxW9RO2uAKZFcmKels3eVCSs2HfFnRiiP81vUAWgcmrqXa8 T2Gw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1717416146; x=1718020946; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WMOJmLS/Nn9J0SYKlSWYVE6j+qu58H+1OHJ/cgq+xrA=; b=JT+FMjFLiG2EF004NeOG0ezcY/NpWuU7P0I/CuBtrN3Y8l3Z27suk4oChYoKKYgGlp 5D7tz0T02gJEkKq6GtIjxwsTYqMfxeNUSUiKTrOeyVJ5N44J4ZDX2qb45s7H90s3XHeT Lj4jjCSfO7x+MN83UbdxvWqMQBHwf6ew3QpcOnKQZwUptU+9rtXLy8MI9GR8JDypyPY0 bktrcbKOLq5ZT18nG31Tg32CpAJZWsQcHKE+hIVZ8/rnRuCwhNSoHPIUGtxJTmlHRy5C N9dxiUWk2uOKfvABE/OvlwnwxRsqDoRCvLlrkDEAIaf9LLi8o2FwWmBHRWUdXkP1AhZK 6SMQ== X-Forwarded-Encrypted: i=1; AJvYcCUiTF2ngnMDI4Z+rIQ8ilF5RF89wj3WQ1zQAAJ/rG7iWqe0F3Bmka7+vLU62roL9fOzM1qYxRwZRMtI3PMgA66xjgA= X-Gm-Message-State: AOJu0YxzCDjDxQei+L9KOo/leG0Dz10RrS1XQqf0j/Kjy6IJW84nF6Xd n/ojiBPnwlYt9jD4bjZlNO7gbBA2xltSriAnZ9PYk3pQwdCMx5q54i+pxtTSPcvLZ9GNYIHL57A 7L5gl77lA1Cp4b4ueRCncHzYfbXJIjQUSlBIxcw== X-Google-Smtp-Source: AGHT+IEcNuSPY/6zmXkbJ2ykkTAVjgIdBcvRO/PSDLvgQzilSvNQKgMTHA7oFyKBa29xVYDm6MJBhlpDtDdGT6zZAIU= X-Received: by 2002:a50:a414:0:b0:579:7b6c:5ab3 with SMTP id 4fb4d7f45d1cf-57a364e3615mr7699348a12.22.1717416145533; Mon, 03 Jun 2024 05:02:25 -0700 (PDT) MIME-Version: 1.0 References: <20240131155929.169961-1-alexghiti@rivosinc.com> <20240131155929.169961-4-alexghiti@rivosinc.com> In-Reply-To: From: Alexandre Ghiti Date: Mon, 3 Jun 2024 14:02:14 +0200 Message-ID: Subject: Re: [External] [PATCH RFC/RFT v2 3/4] riscv: Stop emitting preventive sfence.vma for new vmalloc mappings To: yunhui cui Cc: Catalin Marinas , Will Deacon , Thomas Bogendoerfer , Michael Ellerman , Nicholas Piggin , Christophe Leroy , Paul Walmsley , Palmer Dabbelt , Albert Ou , Andrew Morton , Ved Shanbhogue , Matt Evans , Dylan Jhong , linux-arm-kernel@lists.infradead.org, linux-kernel@vger.kernel.org, linux-mips@vger.kernel.org, linuxppc-dev@lists.ozlabs.org, linux-riscv@lists.infradead.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: A4A2D100028 X-Stat-Signature: 7thw7yx4qb6a495f7fpcefb717xhhd38 X-Rspam-User: X-HE-Tag: 1717416147-753230 X-HE-Meta: U2FsdGVkX18U201/s/kMsZahLh+SCT41wXoMajAVslF8EVYJn446h+F+AweKyo983xo7e1lI1cG7m3GInjIqlzQBZd0ojz9B8jpMBx5UvuRhn+q+bAxFxio7ZKmzEwgFtHAgnCrRretrRM7HXuuGsUG25Gh+QBr+rEL8EvxPB/DswpVDNWP45lIWBXJq3EHLd2aV192U325LmI6CebzFzPhF4hvbgcXjOzuLglHDiHYVs8XDmCDHd2fF268GFms2fG4VHAmHET3RQks5BwQ5Nq7TWyP2O1g9O+kTBF23zAw5YCxr71owBr5TzyA7gASheuOogzQaly1zlMAHCPTiE+tJAhBdR/6K+9pU/TxL9iCicR+7hvyMDPmnF3TsUweb6NtDhizscwhwC+tDCehJ4ez/aB1/pCisd7OZr/912ZbypDOFDuA2e0yc9rSb8wWAvKUE53oxgmJGia0CBkt9sIjrEduTAf9b+m/HoZQs3BJ2R7Zo9JJxeFNAvMYgveD3y1BQM5trukf3N2Az4W8qSkX2ypoFYw3kJQo1CWC24NIi1TVtU2/yJHB9qJs1BBAykGsipkHHBKjruiGIG23bwtP2APWlEJzjZRl8O5oQjBtBM2Q4SFaFAdIXg8JXiiD8NfOV6VCQgFXf9Smfzoye3k1Zte3vIk+x8arkLgqApNecEm2j4LAUo6Ut+K7JCuOa1jReGybpeIeceX1w5JlzSIFsJ2vH/S9lo1wAyBQvvYDfD6VPMJB9xrqXfRF3kqLx5xxYdsRTiwPNjd8ollbnqrO1w5RyLDv6BEQGFS34138viS0D5J9Ydz8Rzl63VugtDwcazdOY0HzGPPC2fci1YLDSihy6ZC4sEdxWTC0dOxxYRsW0GvN6yDudvrTypvFK/Qz1smxSk30h/hNlyNLXRTT/YNkqCAEzUaCVXT/Ejhqkdc27o2arJD/OjLEPv78w9BPvwZc5JB/9ZXLBHz2 fCo4+rim lN3Pv5hIcvykS3uKQiemqOmqMIEY60/mDoMVNmJQvcNOBzfLrBonVotHG31AGLkn6awJS4J5onxvh/g4KdJ/7SOaog5cSahaYhOplZZ6OO4KypgBS1bX6hdNsOSryb5VADxGHdhKsFBkdByqjBRNan4RNf0lTTfeTN3pEzcMy1Gm5zdrb/J/unZRpx3F6QNDnRnHFAEM+AZbKg/lrACNIVUhYTz6q3aLIliM4KeQHxW9D3u/JuCe/7FeXzcGapyWCl0vBOLG0CAR5UAGzdOXVb19Nbr/I8jJWS4PLmwJQR5Y5YoQyRuY41GzUFrh2jTWVvSoOjFB5JBClgZY40zM/gdyVjXfXS2KAX4BpKDCLOYG/cqONr4F6FtVvi0XFS0j4X1KFxyMiKRQSVUW1ca6t5cpfbpinws7copgSSw5HGuVqOt5nTrent/4YGqKoTGiZT+qkeb4duX0zXzM0UnSoHMXizoY9czOQ0EZwpOeSDE9sywPAvUrGECFFgN9Wj9QfePwP/lBFGH07fYQ8+rX+57wVMAyY0BtHMPBNQttoN+TbEhej11B1PCvt8BjJv7YHzfgA X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi Yunhui, On Mon, Jun 3, 2024 at 4:26=E2=80=AFAM yunhui cui = wrote: > > Hi Alexandre, > > On Thu, Feb 1, 2024 at 12:03=E2=80=AFAM Alexandre Ghiti wrote: > > > > In 6.5, we removed the vmalloc fault path because that can't work (see > > [1] [2]). Then in order to make sure that new page table entries were > > seen by the page table walker, we had to preventively emit a sfence.vma > > on all harts [3] but this solution is very costly since it relies on IP= I. > > > > And even there, we could end up in a loop of vmalloc faults if a vmallo= c > > allocation is done in the IPI path (for example if it is traced, see > > [4]), which could result in a kernel stack overflow. > > > > Those preventive sfence.vma needed to be emitted because: > > > > - if the uarch caches invalid entries, the new mapping may not be > > observed by the page table walker and an invalidation may be needed. > > - if the uarch does not cache invalid entries, a reordered access > > could "miss" the new mapping and traps: in that case, we would actual= ly > > only need to retry the access, no sfence.vma is required. > > > > So this patch removes those preventive sfence.vma and actually handles > > the possible (and unlikely) exceptions. And since the kernel stacks > > mappings lie in the vmalloc area, this handling must be done very early > > when the trap is taken, at the very beginning of handle_exception: this > > also rules out the vmalloc allocations in the fault path. > > > > Link: https://lore.kernel.org/linux-riscv/20230531093817.665799-1-bjorn= @kernel.org/ [1] > > Link: https://lore.kernel.org/linux-riscv/20230801090927.2018653-1-dyla= n@andestech.com [2] > > Link: https://lore.kernel.org/linux-riscv/20230725132246.817726-1-alexg= hiti@rivosinc.com/ [3] > > Link: https://lore.kernel.org/lkml/20200508144043.13893-1-joro@8bytes.o= rg/ [4] > > Signed-off-by: Alexandre Ghiti > > --- > > arch/riscv/include/asm/cacheflush.h | 18 +++++- > > arch/riscv/include/asm/thread_info.h | 5 ++ > > arch/riscv/kernel/asm-offsets.c | 5 ++ > > arch/riscv/kernel/entry.S | 84 ++++++++++++++++++++++++++++ > > arch/riscv/mm/init.c | 2 + > > 5 files changed, 113 insertions(+), 1 deletion(-) > > > > diff --git a/arch/riscv/include/asm/cacheflush.h b/arch/riscv/include/a= sm/cacheflush.h > > index a129dac4521d..b0d631701757 100644 > > --- a/arch/riscv/include/asm/cacheflush.h > > +++ b/arch/riscv/include/asm/cacheflush.h > > @@ -37,7 +37,23 @@ static inline void flush_dcache_page(struct page *pa= ge) > > flush_icache_mm(vma->vm_mm, 0) > > > > #ifdef CONFIG_64BIT > > -#define flush_cache_vmap(start, end) flush_tlb_kernel_range(= start, end) > > +extern u64 new_vmalloc[NR_CPUS / sizeof(u64) + 1]; > > +extern char _end[]; > > +#define flush_cache_vmap flush_cache_vmap > > +static inline void flush_cache_vmap(unsigned long start, unsigned long= end) > > +{ > > + if (is_vmalloc_or_module_addr((void *)start)) { > > + int i; > > + > > + /* > > + * We don't care if concurrently a cpu resets this valu= e since > > + * the only place this can happen is in handle_exceptio= n() where > > + * an sfence.vma is emitted. > > + */ > > + for (i =3D 0; i < ARRAY_SIZE(new_vmalloc); ++i) > > + new_vmalloc[i] =3D -1ULL; > > + } > > +} > > #define flush_cache_vmap_early(start, end) local_flush_tlb_kernel_= range(start, end) > > #endif > > > > diff --git a/arch/riscv/include/asm/thread_info.h b/arch/riscv/include/= asm/thread_info.h > > index 5d473343634b..32631acdcdd4 100644 > > --- a/arch/riscv/include/asm/thread_info.h > > +++ b/arch/riscv/include/asm/thread_info.h > > @@ -60,6 +60,11 @@ struct thread_info { > > void *scs_base; > > void *scs_sp; > > #endif > > + /* > > + * Used in handle_exception() to save a0, a1 and a2 before know= ing if we > > + * can access the kernel stack. > > + */ > > + unsigned long a0, a1, a2; > > }; > > > > #ifdef CONFIG_SHADOW_CALL_STACK > > diff --git a/arch/riscv/kernel/asm-offsets.c b/arch/riscv/kernel/asm-of= fsets.c > > index a03129f40c46..939ddc0e3c6e 100644 > > --- a/arch/riscv/kernel/asm-offsets.c > > +++ b/arch/riscv/kernel/asm-offsets.c > > @@ -35,6 +35,8 @@ void asm_offsets(void) > > OFFSET(TASK_THREAD_S9, task_struct, thread.s[9]); > > OFFSET(TASK_THREAD_S10, task_struct, thread.s[10]); > > OFFSET(TASK_THREAD_S11, task_struct, thread.s[11]); > > + > > + OFFSET(TASK_TI_CPU, task_struct, thread_info.cpu); > > OFFSET(TASK_TI_FLAGS, task_struct, thread_info.flags); > > OFFSET(TASK_TI_PREEMPT_COUNT, task_struct, thread_info.preempt_= count); > > OFFSET(TASK_TI_KERNEL_SP, task_struct, thread_info.kernel_sp); > > @@ -42,6 +44,9 @@ void asm_offsets(void) > > #ifdef CONFIG_SHADOW_CALL_STACK > > OFFSET(TASK_TI_SCS_SP, task_struct, thread_info.scs_sp); > > #endif > > + OFFSET(TASK_TI_A0, task_struct, thread_info.a0); > > + OFFSET(TASK_TI_A1, task_struct, thread_info.a1); > > + OFFSET(TASK_TI_A2, task_struct, thread_info.a2); > > > > OFFSET(TASK_TI_CPU_NUM, task_struct, thread_info.cpu); > > OFFSET(TASK_THREAD_F0, task_struct, thread.fstate.f[0]); > > diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S > > index 9d1a305d5508..c1ffaeaba7aa 100644 > > --- a/arch/riscv/kernel/entry.S > > +++ b/arch/riscv/kernel/entry.S > > @@ -19,6 +19,78 @@ > > > > .section .irqentry.text, "ax" > > > > +.macro new_vmalloc_check > > + REG_S a0, TASK_TI_A0(tp) > > + REG_S a1, TASK_TI_A1(tp) > > + REG_S a2, TASK_TI_A2(tp) > > + > > + csrr a0, CSR_CAUSE > > + /* Exclude IRQs */ > > + blt a0, zero, _new_vmalloc_restore_context > > + /* Only check new_vmalloc if we are in page/protection fault */ > > + li a1, EXC_LOAD_PAGE_FAULT > > + beq a0, a1, _new_vmalloc_kernel_address > > + li a1, EXC_STORE_PAGE_FAULT > > + beq a0, a1, _new_vmalloc_kernel_address > > + li a1, EXC_INST_PAGE_FAULT > > + bne a0, a1, _new_vmalloc_restore_context > > + > > +_new_vmalloc_kernel_address: > > + /* Is it a kernel address? */ > > + csrr a0, CSR_TVAL > > + bge a0, zero, _new_vmalloc_restore_context > > + > > + /* Check if a new vmalloc mapping appeared that could explain t= he trap */ > > + > > + /* > > + * Computes: > > + * a0 =3D &new_vmalloc[BIT_WORD(cpu)] > > + * a1 =3D BIT_MASK(cpu) > > + */ > > + REG_L a2, TASK_TI_CPU(tp) > > + /* > > + * Compute the new_vmalloc element position: > > + * (cpu / 64) * 8 =3D (cpu >> 6) << 3 > > + */ > > + srli a1, a2, 6 > > + slli a1, a1, 3 > > + la a0, new_vmalloc > > + add a0, a0, a1 > > + /* > > + * Compute the bit position in the new_vmalloc element: > > + * bit_pos =3D cpu % 64 =3D cpu - (cpu / 64) * 64 =3D cpu - (cp= u >> 6) << 6 > > + * =3D cpu - ((cpu >> 6) << 3) << 3 > > + */ > > + slli a1, a1, 3 > > + sub a1, a2, a1 > > + /* Compute the "get mask": 1 << bit_pos */ > > + li a2, 1 > > + sll a1, a2, a1 > > + > > + /* Check the value of new_vmalloc for this cpu */ > > + REG_L a2, 0(a0) > > + and a2, a2, a1 > > + beq a2, zero, _new_vmalloc_restore_context > > + > > + /* Atomically reset the current cpu bit in new_vmalloc */ > > + amoxor.w a0, a1, (a0) > > + > > + /* Only emit a sfence.vma if the uarch caches invalid entries *= / > > + ALTERNATIVE("sfence.vma", "nop", 0, RISCV_ISA_EXT_SVVPTC, 1) > > + > > + REG_L a0, TASK_TI_A0(tp) > > + REG_L a1, TASK_TI_A1(tp) > > + REG_L a2, TASK_TI_A2(tp) > > + csrw CSR_SCRATCH, x0 > > + sret > > + > > +_new_vmalloc_restore_context: > > + REG_L a0, TASK_TI_A0(tp) > > + REG_L a1, TASK_TI_A1(tp) > > + REG_L a2, TASK_TI_A2(tp) > > +.endm > > + > > + > > SYM_CODE_START(handle_exception) > > /* > > * If coming from userspace, preserve the user thread pointer a= nd load > > @@ -30,6 +102,18 @@ SYM_CODE_START(handle_exception) > > > > .Lrestore_kernel_tpsp: > > csrr tp, CSR_SCRATCH > > + > > + /* > > + * The RISC-V kernel does not eagerly emit a sfence.vma after e= ach > > + * new vmalloc mapping, which may result in exceptions: > > + * - if the uarch caches invalid entries, the new mapping would= not be > > + * observed by the page table walker and an invalidation is n= eeded. > > + * - if the uarch does not cache invalid entries, a reordered a= ccess > > + * could "miss" the new mapping and traps: in that case, we o= nly need > > + * to retry the access, no sfence.vma is required. > > + */ > > + new_vmalloc_check > > + > > REG_S sp, TASK_TI_KERNEL_SP(tp) > > > > #ifdef CONFIG_VMAP_STACK > > diff --git a/arch/riscv/mm/init.c b/arch/riscv/mm/init.c > > index eafc4c2200f2..54c9fdeda11e 100644 > > --- a/arch/riscv/mm/init.c > > +++ b/arch/riscv/mm/init.c > > @@ -36,6 +36,8 @@ > > > > #include "../kernel/head.h" > > > > +u64 new_vmalloc[NR_CPUS / sizeof(u64) + 1]; > > + > > struct kernel_mapping kernel_map __ro_after_init; > > EXPORT_SYMBOL(kernel_map); > > #ifdef CONFIG_XIP_KERNEL > > -- > > 2.39.2 > > > > > > Can we consider using new_vmalloc as a percpu variable, so that we > don't need to add a0/1/2 in thread_info? At first, I used percpu variables. But then I realized that percpu areas are allocated in the vmalloc area, so if somehow we take a trap when accessing the new_vmalloc percpu variable, we could not recover from this as we would trap forever in new_vmalloc_check. But admittedly, not sure that can happen. And how would that remove a0, a1 and a2 from thread_info? We'd still need to save some registers somewhere to access the percpu variable right? > Also, try not to do too much > calculation logic in new_vmalloc_check, after all, handle_exception is > a high-frequency path. In this case, can we consider writing > new_vmalloc_check in C language to increase readability? If we write that in C, we don't have the control over the allocated registers and then we can't correctly save the context. Thanks for your interest in this patchset :) Alex > > Thanks, > Yunhui