From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id AD42FC83F0F for ; Tue, 8 Jul 2025 10:07:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 3D77F6B017A; Tue, 8 Jul 2025 06:07:44 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 3AE946B033C; Tue, 8 Jul 2025 06:07:44 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 29E4B6B033A; Tue, 8 Jul 2025 06:07:44 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 122A66B0338 for ; Tue, 8 Jul 2025 06:07:44 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 914FCC0527 for ; Tue, 8 Jul 2025 10:07:43 +0000 (UTC) X-FDA: 83640670806.27.8126A9D Received: from mail-oi1-f171.google.com (mail-oi1-f171.google.com [209.85.167.171]) by imf30.hostedemail.com (Postfix) with ESMTP id B91A88000C for ; Tue, 8 Jul 2025 10:07:40 +0000 (UTC) Authentication-Results: imf30.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=lf1q+xX4; spf=pass (imf30.hostedemail.com: domain of cuiyunhui@bytedance.com designates 209.85.167.171 as permitted sender) smtp.mailfrom=cuiyunhui@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1751969261; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=ShBg34hVsoMJ1vlPxNf3mH3/jDb2DnYrFBCkYKMK3pY=; b=wDgSs8w6OwD82trf2eMH3gqixmHd0H0JZymyevOJTarSWmbxIzdnG/m3ZPk8CWKgDoE6k2 /KHZackA6DhUE+P4VCttAIKYG/oXkrWXLiE5qJHr0gfR9bdKYVriDGjgHWanq+U6RbBk0U Ax3NxcgaEmatA6M87SXeIiTnpQzpT68= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1751969261; a=rsa-sha256; cv=none; b=UjXbXT56TKDn1JXIRFVRZvciD+t8dX/uswYtUUpTAis0aAl1fUa25aQYaYEXo674j1cZMG FNDMOm1Mha9gSs7jOmA8DWV8mKd+MgFtYqZFYF8gRFa8V/RCN2NRx6goCi1g1UUpNT/23z x2hruPbVDFS67rNn1yIc/wHMGJ6B6hU= ARC-Authentication-Results: i=1; imf30.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=lf1q+xX4; spf=pass (imf30.hostedemail.com: domain of cuiyunhui@bytedance.com designates 209.85.167.171 as permitted sender) smtp.mailfrom=cuiyunhui@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com Received: by mail-oi1-f171.google.com with SMTP id 5614622812f47-40b859461easo2282603b6e.1 for ; Tue, 08 Jul 2025 03:07:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1751969259; x=1752574059; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=ShBg34hVsoMJ1vlPxNf3mH3/jDb2DnYrFBCkYKMK3pY=; b=lf1q+xX42i7UmjaBXgBID8ZJR2mR24gpDdkrszC86Q4OMfoo47Th9CNqDWRp0tGKNQ lrlMykL/AoCbPIKmaeWlN5f1CVN/HN+AGx1sC9QGkY50vJTUo5gT2Sx29BWKWHcB0yZH CtmH3/OGpyl004eN23HppMvfHoAapvDHolyPV/JM8o0sDNL2HF1qC/m+37wUjP2POGiR gIHiUCO3kC8nBiHD7wtub91yk88oYbvSlQcu2bHqxKHsYsU93OUSOV2wgO2zZ+GaWXD2 x5cphS9nN2gB9aq45d7zgaJ1EXJRAJGQFV5EfszbNyyNubvew9Bd/aBZBv5k4bP/hdEj gMFA== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1751969259; x=1752574059; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=ShBg34hVsoMJ1vlPxNf3mH3/jDb2DnYrFBCkYKMK3pY=; b=RJw0f/W4ugFwl4t2RU6GLNF6fKpuAHj0Pa9XLphkJh1wmGoVPG2TvfgYrwdfQdmqtj Y8eFnelmuD2qsZtlvOXs6EmGZynug5EGudoEUUt8S6kjxvZtYyvVTFRehTpee1mxciRa xnYd6uCzxC/FGoHm8qNCk8yNjKW0/S4DrZyt9xzHXDmGverD+sfCQa1kv2Lr9uqnFQRd BTkc8Lx7XYu1yeazFC/i9COe7ss8OOZ8Px8V+zmLxl8zAohBrUNgE9r3M07EKlMtlYxO LaQhBQSqNu+9ZBEUGyH/JR2GnobMRRa+n3Ejd8gvvhWANorVNJkWHR4lNer+ZBLJHB4C FGkw== X-Forwarded-Encrypted: i=1; AJvYcCUICn9+yqbZ7gtfj9MSELY1gMIsrBgVMiK8Pdghlge+1Ui9mCcVBv0NOOz7QbpzWeFY6H0uiLglpg==@kvack.org X-Gm-Message-State: AOJu0YyV1ID3NoMAntallIFlTzfE9iWbOl6Qh7FaTVP8PQCOSqxxu62q arcexit5SZchFVxjyZTLUxQ67JAm5WzF8S559scUNBPnTapssMBglWcJa9I2gnwkDzgxLQ+aR8g vrjcsPQJYst2kPQRXblrS6wTzr6kpjbY66dN2K6x0mw== X-Gm-Gg: ASbGnctece22p/SvvtCAUKMEc5uLc/BjIc5tdql1YSql0tcpsM1iWGPeGHLrQkGTqXC RzjFJUwE71sblTt1CzdTbItOrHhlrV8AwIjlhevTV2UBauceeD+T/bqJ2ydBalQnz8Hd7PLn/fW AdwYgn7IJnfbWt+k996L6/WxVQTnj+Y+UlGvy+jzJTV+E3kA== X-Google-Smtp-Source: AGHT+IFYEYqs2WMKAMHJi+n4eGUgFs64PQCx+HQ60GIjFCu+jrmCDIf42ANQM6HjwmKcez+B2RP2YUVdr28AeKuyiV4= X-Received: by 2002:a05:6808:4a52:20b0:404:ed0d:79e5 with SMTP id 5614622812f47-4113f408987mr1144175b6e.30.1751969259482; Tue, 08 Jul 2025 03:07:39 -0700 (PDT) MIME-Version: 1.0 References: <20250704084500.62688-1-cuiyunhui@bytedance.com> In-Reply-To: From: yunhui cui Date: Tue, 8 Jul 2025 18:07:27 +0800 X-Gm-Features: Ac12FXysBXsB5Ps-NOvgjudnKpiBJody5f6VY0xxRLYTEmLwWFyQR8cdqQJ11iE Message-ID: Subject: Re: [External] [PATCH] RISC-V: store percpu offset in CSR_SCRATCH To: =?UTF-8?B?UmFkaW0gS3LEjW3DocWZ?= Cc: masahiroy@kernel.org, nathan@kernel.org, nicolas.schier@linux.dev, dennis@kernel.org, tj@kernel.org, cl@gentwo.org, paul.walmsley@sifive.com, palmer@dabbelt.com, aou@eecs.berkeley.edu, alex@ghiti.fr, andybnac@gmail.com, bjorn@rivosinc.com, cyrilbur@tenstorrent.com, rostedt@goodmis.org, puranjay@kernel.org, ben.dooks@codethink.co.uk, zhangchunyan@iscas.ac.cn, ruanjinjie@huawei.com, jszhang@kernel.org, charlie@rivosinc.com, cleger@rivosinc.com, antonb@tenstorrent.com, ajones@ventanamicro.com, debug@rivosinc.com, haibo1.xu@intel.com, samuel.holland@sifive.com, linux-kbuild@vger.kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, linux-riscv@lists.infradead.org, linux-riscv , wangziang.ok@bytedance.com Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Server: rspam12 X-Rspamd-Queue-Id: B91A88000C X-Stat-Signature: edixh83qyeqjxosrc8g4yk1ga6ib4e6b X-HE-Tag: 1751969260-796209 X-HE-Meta: U2FsdGVkX19IIMK8Jb8krYka7nirx6r8rS6dAPzbFIE/LXDM7B0NukNameePslFt4XBme9UGtAVxge2Xckb/18BtPppKgpIIjOfYwB+xKwPjEE9z4OtglFMtTyV0DxMj7HRwyISasCkD1IVlks9rbTfZrrB9Il3hTAOuSuvqy8DlyKFFNnuTBAkBkQA8VDbmjf4/mxNya6zoueO9nQfNXkr1tUIfGHH4B3Je6Rk5ud8u9xAtrxsVPH52orUmlv4e7rbytAXzVdXeat/x4PNyPa/yLOZ7EkuT+H8lzW2EymkSEk57hwy1e3r8FwOYrLbG84XQ7iwDRfn/MjLO1Rmn7in840NQPHFV0EoQ0a9mjmlp2VjMi7XMN+o0LYHVUC/s0WJrPKxLTXmWsARqqklWsQSAkxW5AjlJK3srkEMOlIcgVg5KYYSn+UEs5jvgEfJvvRSKWYlihVR1FZ8O6AnX/yTwh6wTqdpJb7s04zGtmJRCc/R24da/kFU/Iwm7LAt9r7llQZ/LJPZ4Wl5vnPEcjpLRXZAd36PGlekarXE1sDHWlN1hrrettjSPf0HApSqbHG27ICui+0cxMv5BTGqjqhhm0oDDDxEDRF7pR6f80MlwTFCvJ/4+/EQJy8gyrwSbbZ8pzMdR+5Rf66FQKKx0As9Vdwp6EV7oI/RN0TskE7pUidquwvoGyXpdFRMh5Kef0A3Xl6wcUvu5BxrnwTZpR0RZbNd3+cZHKrQaSLdTzrZ8/uwdCfukdGD1jOxj7oWRyaQmhSxJMJClTcJ7h5egZBOzCA+Q9e5vzytZVd2oI61tjNuw6R3dmmxOottFUpFgIhSSYPWfMAbXsCw2qIkg4HhgJsfKrz183cCeWeTEfvn0xttviw1NysafTk0HK8GxW0V5QJUjJXt4Rbt/bz7tfgb21U0p+VAnSIt86nlXCUmWhFiCeQ/WBI5DaNDnnLMNiSzUeJRFNCnaQHUe0Ob NKShMGws b+cAKZ0KB9R0/UKrbiqiJ0xJSMcnjP/2zGC/g561cZ436Hys4B6+6YfK+Eas0pdJ4WU75IQ/XmpUr3Oj2pFVviwN65zFbgim6ffmcq1dSCayyDfDK6xbjb2lJJBnxmf12wmFw6xW8JrKqEDJ/FnhAhbIZO1/tcQ6zJDBF2vukLX90tgA8EGj4gpCRKbp/Z3M0GttoGR9CycN6cpHyOl/A35qOa1p3rbhP7meTSkdhDqL05O6GCdrzPTt4WE3/AZnjBcW7TtAu5kJxdF5wtmDdR90zbHPjtopTylE26DCwbT0xH6Tr7xm/B8ZzUzh39A+AYDAM7+GnLvrAyynx5tcY7fGkfBRcKG0+xbaaiUAeSuph5YGHuemGPaYph5ngEVbOUzKazf+483iZVBk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hi Radim, On Mon, Jul 7, 2025 at 8:50=E2=80=AFPM Radim Kr=C4=8Dm=C3=A1=C5=99 wrote: > > 2025-07-04T16:45:00+08:00, Yunhui Cui : > > The following data was collected from tests conducted on the > > Spacemit(R) X60 using the fixed register method: > > [...] > > The fixed register method reduced performance by 5.29%. > > The per-CPU offset optimization improved performance by 2.52%. > > What is the performance if you use the scratch register? > > The patch below is completely unoptimized as I didn't want to shuffle > code around too much, but it could give a rough idea. > > Thanks. > > ---8<--- > The scratch register currently denotes the mode before exception, but we > can just use two different exception entry points to provide the same > information, which frees the scratch register for the percpu offset. > > The user/kernel entry paths need more through rewrite, because they are > terribly wasteful right now. > --- > Applies on top of d7b8f8e20813f0179d8ef519541a3527e7661d3a (v6.16-rc5) > > arch/riscv/include/asm/percpu.h | 13 ++++++++++ > arch/riscv/kernel/entry.S | 46 ++++++++++++++++++++------------- > arch/riscv/kernel/head.S | 7 +---- > arch/riscv/kernel/smpboot.c | 7 +++++ > arch/riscv/kernel/stacktrace.c | 4 +-- > 5 files changed, 51 insertions(+), 26 deletions(-) > create mode 100644 arch/riscv/include/asm/percpu.h > > diff --git a/arch/riscv/include/asm/percpu.h b/arch/riscv/include/asm/per= cpu.h > new file mode 100644 > index 000000000000..2c838514e3ea > --- /dev/null > +++ b/arch/riscv/include/asm/percpu.h > @@ -0,0 +1,13 @@ > +#ifndef __ASM_PERCPU_H > +#define __ASM_PERCPU_H > + > +static inline void set_my_cpu_offset(unsigned long off) > +{ > + csr_write(CSR_SCRATCH, off); > +} > + > +#define __my_cpu_offset csr_read(CSR_SCRATCH) > + > +#include > + > +#endif > diff --git a/arch/riscv/kernel/entry.S b/arch/riscv/kernel/entry.S > index 75656afa2d6b..e48c553d6779 100644 > --- a/arch/riscv/kernel/entry.S > +++ b/arch/riscv/kernel/entry.S > @@ -91,18 +91,8 @@ > REG_L a0, TASK_TI_A0(tp) > .endm > > - > -SYM_CODE_START(handle_exception) > - /* > - * If coming from userspace, preserve the user thread pointer and= load > - * the kernel thread pointer. If we came from the kernel, the sc= ratch > - * register will contain 0, and we should continue on the current= TP. > - */ > - csrrw tp, CSR_SCRATCH, tp > - bnez tp, .Lsave_context > - > -.Lrestore_kernel_tpsp: > - csrr tp, CSR_SCRATCH > +SYM_CODE_START(handle_kernel_exception) > + csrw CSR_SCRATCH, tp > > #ifdef CONFIG_64BIT > /* > @@ -126,8 +116,22 @@ SYM_CODE_START(handle_exception) > bnez sp, handle_kernel_stack_overflow > REG_L sp, TASK_TI_KERNEL_SP(tp) > #endif > + j handle_exception > +ASM_NOKPROBE(handle_kernel_exception) > +SYM_CODE_END(handle_kernel_exception) > > -.Lsave_context: > +SYM_CODE_START(handle_user_exception) > + /* > + * If coming from userspace, preserve the user thread pointer and= load > + * the kernel thread pointer. > + */ > + csrrw tp, CSR_SCRATCH, tp > + j handle_exception > + > +SYM_CODE_END(handle_user_exception) > +ASM_NOKPROBE(handle_user_exception) > + > +SYM_CODE_START_NOALIGN(handle_exception) > REG_S sp, TASK_TI_USER_SP(tp) > REG_L sp, TASK_TI_KERNEL_SP(tp) > addi sp, sp, -(PT_SIZE_ON_STACK) > @@ -158,11 +162,15 @@ SYM_CODE_START(handle_exception) > REG_S s4, PT_CAUSE(sp) > REG_S s5, PT_TP(sp) > > - /* > - * Set the scratch register to 0, so that if a recursive exceptio= n > - * occurs, the exception vector knows it came from the kernel > - */ > - csrw CSR_SCRATCH, x0 > + REG_L s0, TASK_TI_CPU(tp) > + slli s0, s0, 3 > + la s1, __per_cpu_offset > + add s1, s1, s0 > + REG_L s1, 0(s1) > + > + csrw CSR_SCRATCH, s1 This patch cleverly differentiates whether an exception originates from user mode or kernel mode. However, there's still an issue with using CSR_SCRATCH: each time handle_exception() is called, the following instructions must be executed: REG_L s0, TASK_TI_CPU(tp) slli s0, s0, 3 la s1, __per_cpu_offset add s1, s1, s0 REG_L s1, 0(s1) csrw CSR_SCRATCH, s1 Should we consider adding a dedicated CSR (e.g., CSR_SCRATCH2) to store the percpu offset instead? See: https://lists.riscv.org/g/tech-privileged/topic/113437553#msg2506 > + la s1, handle_kernel_exception > + csrw CSR_TVEC, s1 > > /* Load the global pointer */ > load_global_pointer > @@ -236,6 +244,8 @@ SYM_CODE_START_NOALIGN(ret_from_exception) > * structures again. > */ > csrw CSR_SCRATCH, tp > + la a0, handle_user_exception > + csrw CSR_TVEC, a0 > 1: > #ifdef CONFIG_RISCV_ISA_V_PREEMPTIVE > move a0, sp > diff --git a/arch/riscv/kernel/head.S b/arch/riscv/kernel/head.S > index bdf3352acf4c..d8858334af2d 100644 > --- a/arch/riscv/kernel/head.S > +++ b/arch/riscv/kernel/head.S > @@ -188,14 +188,9 @@ secondary_start_sbi: > .align 2 > .Lsetup_trap_vector: > /* Set trap vector to exception handler */ > - la a0, handle_exception > + la a0, handle_kernel_exception > csrw CSR_TVEC, a0 > > - /* > - * Set sup0 scratch register to 0, indicating to exception vector= that > - * we are presently executing in kernel. > - */ > - csrw CSR_SCRATCH, zero > ret > > SYM_CODE_END(_start) > diff --git a/arch/riscv/kernel/smpboot.c b/arch/riscv/kernel/smpboot.c > index 601a321e0f17..2db44b10bedb 100644 > --- a/arch/riscv/kernel/smpboot.c > +++ b/arch/riscv/kernel/smpboot.c > @@ -41,6 +41,11 @@ > > static DECLARE_COMPLETION(cpu_running); > > +void __init smp_prepare_boot_cpu(void) > +{ > + set_my_cpu_offset(per_cpu_offset(smp_processor_id())); > +} > + > void __init smp_prepare_cpus(unsigned int max_cpus) > { > int cpuid; > @@ -225,6 +230,8 @@ asmlinkage __visible void smp_callin(void) > mmgrab(mm); > current->active_mm =3D mm; > > + set_my_cpu_offset(per_cpu_offset(curr_cpuid)); > + > store_cpu_topology(curr_cpuid); > notify_cpu_starting(curr_cpuid); > > diff --git a/arch/riscv/kernel/stacktrace.c b/arch/riscv/kernel/stacktrac= e.c > index 3fe9e6edef8f..69b2f390a2d4 100644 > --- a/arch/riscv/kernel/stacktrace.c > +++ b/arch/riscv/kernel/stacktrace.c > @@ -16,7 +16,7 @@ > > #ifdef CONFIG_FRAME_POINTER > > -extern asmlinkage void handle_exception(void); > +extern asmlinkage void handle_kernel_exception(void); > extern unsigned long ret_from_exception_end; > > static inline int fp_is_valid(unsigned long fp, unsigned long sp) > @@ -72,7 +72,7 @@ void notrace walk_stackframe(struct task_struct *task, = struct pt_regs *regs, > fp =3D frame->fp; > pc =3D ftrace_graph_ret_addr(current, &graph_idx,= frame->ra, > &frame->ra); > - if (pc >=3D (unsigned long)handle_exception && > + if (pc >=3D (unsigned long)handle_kernel_exceptio= n && > pc < (unsigned long)&ret_from_exception_end) = { > if (unlikely(!fn(arg, pc))) > break; Thanks, Yunhui