From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 99122C5B549 for ; Fri, 30 May 2025 09:33:53 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 36F606B008A; Fri, 30 May 2025 05:33:53 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 2F80D6B00AE; Fri, 30 May 2025 05:33:53 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 1C07D6B00B9; Fri, 30 May 2025 05:33:53 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id E653E6B008A for ; Fri, 30 May 2025 05:33:52 -0400 (EDT) Received: from smtpin02.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay06.hostedemail.com (Postfix) with ESMTP id AA2A3EC854 for ; Fri, 30 May 2025 09:33:52 +0000 (UTC) X-FDA: 83499062304.02.77579AB Received: from mail-pl1-f172.google.com (mail-pl1-f172.google.com [209.85.214.172]) by imf28.hostedemail.com (Postfix) with ESMTP id AD20DC0002 for ; Fri, 30 May 2025 09:33:50 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="VPE/IVhG"; spf=pass (imf28.hostedemail.com: domain of libo.gcs85@bytedance.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=libo.gcs85@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1748597630; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=+2NFrx3BFqUZITaGnYPssFKiBOpe/dz6IJRWesRURms=; b=tTdwe6eloLoa5w3X93OC870fYr7cFfToANgzX/VTLEwwSjP8cY7WGvGSFH7z2+Z7JlQYZg RzaGJrxzII771TZcImt5r7rZWuA2xfEHj43aBxYlSE7TQWfeLXU6R/QGjyCfQ9tvHlskWM 7UGj/vkkBtkxCWICYGkU3iB28znKo0c= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b="VPE/IVhG"; spf=pass (imf28.hostedemail.com: domain of libo.gcs85@bytedance.com designates 209.85.214.172 as permitted sender) smtp.mailfrom=libo.gcs85@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1748597630; a=rsa-sha256; cv=none; b=0YCdgWjdxurKVjvaWdEliUT9IVV3vEJ0Xyhptcvr38SItOQZdCKzA/NY6RaApI98ALKSp2 r4whuFqr1U5s91t5EhjSEMoD6VYutgrXQMcruDJRhcypdh1R8N2K/pmXQKUrZvCTSO8I1h bz23jtfvHRbVoDM86trAc0ogCv1DZMc= Received: by mail-pl1-f172.google.com with SMTP id d9443c01a7336-23228b9d684so20714095ad.1 for ; Fri, 30 May 2025 02:33:50 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1748597629; x=1749202429; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=+2NFrx3BFqUZITaGnYPssFKiBOpe/dz6IJRWesRURms=; b=VPE/IVhG9TAV4DKinnWpV6EnER9SKqxEF0kv8ZXbcHVJbA2B/0z1Az+rI/Zn8gdQrs rWgm/CoSPUi6byGUguspRbGKtNjaN/4bbOu00sdlMtUF9ICVAHyDoy0iPAEjyVToyKVu diYw3rUNqa1y7KlwLnNKgIBm7MXVPZWejYAEKT0KPq1Aj4/wh/AYYoOLxhMakMTXHR9v pDqlJU08Z2fZmImGouCSvtHOqvNZ4miVWnJVe6nrdjJiFLXes8WKW/tJqpK1OwuIqv1D CcfPlVfGNcHDR67yXBE1tt+rGwvl07hUUysYxW8nSO5UhiGY9wTkf4W4LyxNBaUduKY7 ajwQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748597629; x=1749202429; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=+2NFrx3BFqUZITaGnYPssFKiBOpe/dz6IJRWesRURms=; b=n6T7TjkgA23b1x80LW23WMELPOvRTCC6VxHeAURWmGnUe7BR8ubAIfql3Ev/mTNuwn h8LAPS3vNP5FzX4ngdxdE6XSrqvkl84qoqFxpqeGv6h0M/Jtv+kFR1LxncdnkdQEiK0y vrxl3CgupFWZK6rn0WyC2wZqZy+rfQxYY0+ji5y6X8gnRT9zIjrCJzVsPi9xU1XlVYXT FgDzT8UbY26P4LPYS/Np7SGjBXmbdujWJnueEku+ZZUzB2pdOcXLw+0m5Ht99EavJwlf g8UJZ8toKh1GmrXr9N3TcDf8PnnDazOZr3+dhCERzXuAPJXeLNtYw03MaKq/8bGzTGrp luIg== X-Forwarded-Encrypted: i=1; AJvYcCUvniy+Praw/Jw3v1xc9Oari8S8gZlQlLGAbola+cBVmsc+lebchx+MvUuMyqnoemMeTkZaQRvs2Q==@kvack.org X-Gm-Message-State: AOJu0Yw2q5FmOwb/UyMFm/KPr/aq5+KqmXunqLEQphhGlNstFSXssGfM GMOXsZCy5T2PrTshtdF1E7Q55N54ETlozIdngt+HNOkJNewCi6XuL2FIev0+RMEs7Xg= X-Gm-Gg: ASbGnct8aQHkZ6g8Dc1V6OmDWblzeFtyL/xPTv8Ps4fHjy8xR1Ci6SiIajkOME3AEyr O31cT44ju2glVIIoCenV/D9N5NfDSHBgd3qnBBxtsRPxaSQVWJDwyIacMijKU482rzLVkHiOIrW 8QSHY2DcFc3uxy+6cfX6kHZqTkQWRBBs5LyaoWZskLfhpQc9vaR1kymX9/xRBaiFRWYaW4THDbh ebixKBOipwAnjXDu9pURWfaWPfAB/vAXaIe4C1lKRncq9OSKWLOiFuxzOVstlGHdAvOKJ+9grRd a5wS0ND5NAjCCpY3pjaHAGwNft/YjZFk/UPuLnI5onrt+qSELRHdUa2DMSKvZcmcej6YVxEp5jl wwrm1rqHpEw== X-Google-Smtp-Source: AGHT+IHkejFlF7JacpAFc31d67Aa0abDbDTBPS+AvGZCJTX8xpAHAqZtwkMExSrIMlmiA7Dd0bS1gw== X-Received: by 2002:a17:90b:3ec3:b0:310:cea4:e3b9 with SMTP id 98e67ed59e1d1-31250452c5fmr1772636a91.34.1748597629413; Fri, 30 May 2025 02:33:49 -0700 (PDT) Received: from FQ627FTG20.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-3124e29f7b8sm838724a91.2.2025.05.30.02.33.34 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 30 May 2025 02:33:49 -0700 (PDT) From: Bo Li To: tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, luto@kernel.org, kees@kernel.org, akpm@linux-foundation.org, david@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org, peterz@infradead.org Cc: dietmar.eggemann@arm.com, hpa@zytor.com, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com, viro@zeniv.linux.org.uk, brauner@kernel.org, jack@suse.cz, lorenzo.stoakes@oracle.com, Liam.Howlett@oracle.com, vbabka@suse.cz, rppt@kernel.org, surenb@google.com, mhocko@suse.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, jannh@google.com, pfalcato@suse.de, riel@surriel.com, harry.yoo@oracle.com, linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, yinhongbo@bytedance.com, dengliang.1214@bytedance.com, xieyongji@bytedance.com, chaiwen.cc@bytedance.com, songmuchun@bytedance.com, yuanzhu@bytedance.com, chengguozhu@bytedance.com, sunjiadong.lff@bytedance.com, Bo Li Subject: [RFC v2 21/35] RPAL: add kernel entry handling for lazy switch Date: Fri, 30 May 2025 17:27:49 +0800 Message-Id: <924aa7959502c4c3271cb311632eb505e894e26e.1748594841.git.libo.gcs85@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam01 X-Rspamd-Queue-Id: AD20DC0002 X-Stat-Signature: n8c9j4u7zeqjwk14pi55tbscnzkuah9t X-Rspam-User: X-HE-Tag: 1748597630-549039 X-HE-Meta: U2FsdGVkX1/nCAX45IxZld2eYUbUqP69DN2VmjMglEWGGmE8Q5ZSbmiz7gVNfeYcRWGSeg4ZPlsQ2ZaiXHR7RtvYaChDYKpE75b9Fdl0IckMNbFrcto6qdaFUumOs/kXdvyPal2LjzA/VH7CHqvkMuNBhwXGHMVfuqVM5HLHhJcLDgnRaDiDmW91i0k3Yow/jFP+hNDlAuyOnMdjLtyLUx5yvA9SjmH7bnb+0ROEqO+WbxGVNL98DcycCa4F3HUVRaZ+fTponCGyDM3DIh4O10xFjvYqET/eXT/EyKwG7vmb5e7ywt2Fxq1pIcUFSF6KUFjkEilppIMWiwnccsmaPt15JhXmZB5rbN3MHGgsGHnV5J9zZuQk+cox4AyQgr0l+ksLzuEVTPuJ9kNeKkU2+R1ULaoJH4mBTnUVA3PjMvzel4/59m2hYlb8O9lEvj1W52IvoD9pksjP8F5T9ExMUTNbksdqhi2L4LNKYgq6yaNXfNTJ7RrWHabTj2YffPtxUBbPx4iqwCi4JcnG2FlaM4T19b798/AJGp1jH/tZidFBkyrp6UiRoajaSnRHDb5aasmnDCFRIixYeToBWVVWtl2wyVW/KsJG6FKex5zCoyO4huRYN3lm1OWQH0YgZYHntDLlaNdOYWGQ0vXYQ4oj7gCxYghJdJElR8Q7BpVPnMmQlghvm5hV+FkIqRxLsis71EQ8scfCrst9bq68Nt6er0kSKnkrUamlGocsexhRJ8/w8w1DD6PB8e26GMXqwrztwN8UmrCnVba7nIawvZF6MROkVO90SgxhsST0WWLjloXGYm01JYoMjX1sKcYmV65rN4v+6i4A14G0dy8t9k86v+eaMXO9TeUvI6Bg6B4ytPcOXZ5+UShnQV0ZLr3Ia/MAJZgLZyrTAbym5qLNJyhRmegYf0SvuopTS4CqmhrT9Ms3n6DoUzowaNYpDYWxr7eM4jec3yFfYDmHaKIe5Gr UL+LIjEG 6rGbh/ApjUA2sriL5DRdyz9L7SFM6G2xkB0AqzBMnZIAJvWc1gnpmxA24wcLZybe3wjVodcsuLBwQGUJrUihwYKc14mMUufi7390IX27ZMHcs5r8s2wN+/26aoqaMX/03/wWcVRUqNBRqAsCHjGR9ae82kEWnxQNi3JLY0ty+u0r6ETdAcbLggYncP8ObazQIVkp6T0hsnL8A27SZNwrecKJDYdHsQOxcVCBFboH4+Ce8LYxLrOIuUagYnE4JtlAotVP407ZkWOuVcAqsIlHl74hQ4C9YhTRfMPXezD+bE7qzYjqe2uyrGKEKNQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: At the kernel entry point, RPAL performs a lazy switch. Therefore, it is necessary to hook all kernel entry points to execute the logic related to the lazy switch. At the kernel entry, apart from some necessary operations related to the lazy switch (such as ensuring that the general-purpose registers remain unchanged before and after the lazy switch), the task before the lazy switch will lose its user mode context (which is passed to the task after the lazy switch). Therefore, the kernel entry also needs to handle the issue of the previous task losing its user mode context. This patch hooks all locations where the transition from user mode to kernel mode occurs, including entry_SYSCALL_64, error_entry, and asm_exc_nmi. When the kernel detects a mismatch between the kernel-mode and user mode contexts, it executes the logic related to the lazy switch. Taking the switch from the sender to the receiver as an example, the receiver thread is first locked to the CPU where the sender is located. Then, the receiver thread in the CALL state is woken up through rpal_try_to_wake_up(). The general purpose register state (pt_regs) of the sender is copied to the receiver, and rpal_schedule() is executed to complete the lazy switch. Regarding the issue of the sender losing its context, the kernel loads the pre-saved user mode context of the sender into the sender's pt_regs and constructs the kernel stack frame of the sender in a manner similar to the fork operation. The handling of the switch from the receiver to the sender is similar, except that the receiver will be unlocked from the current CPU, and the receiver can only return to the user mode through the kernel return method. Signed-off-by: Bo Li --- arch/x86/entry/entry_64.S | 137 ++++++++++++++++++++++++++++++++++ arch/x86/kernel/asm-offsets.c | 3 + arch/x86/rpal/core.c | 137 ++++++++++++++++++++++++++++++++++ include/linux/rpal.h | 6 ++ 4 files changed, 283 insertions(+) diff --git a/arch/x86/entry/entry_64.S b/arch/x86/entry/entry_64.S index 13b4d0684575..59c38627510d 100644 --- a/arch/x86/entry/entry_64.S +++ b/arch/x86/entry/entry_64.S @@ -118,6 +118,20 @@ SYM_INNER_LABEL(entry_SYSCALL_64_after_hwframe, SYM_L_GLOBAL) UNTRAIN_RET CLEAR_BRANCH_HISTORY +#ifdef CONFIG_RPAL + /* + * We first check if it is a RPAL sender/receiver with + * current->rpal_cd. For non-RPAL task, we just skip it. + * For rpal task, We may need to check if it needs to do + * lazy switch. + */ + movq PER_CPU_VAR(current_task), %r13 + movq TASK_rpal_cd(%r13), %rax + testq %rax, %rax + jz _do_syscall + jmp do_rpal_syscall +_do_syscall: +#endif call do_syscall_64 /* returns with IRQs disabled */ /* @@ -190,6 +204,101 @@ SYM_CODE_START(rpal_ret_from_lazy_switch) jmp swapgs_restore_regs_and_return_to_usermode #endif SYM_CODE_END(rpal_ret_from_lazy_switch) + +/* return address offset of stack frame */ +#define RPAL_FRAME_RET_ADDR_OFFSET -56 + +SYM_CODE_START(do_rpal_syscall) + movq %rsp, %r14 + call rpal_syscall_64_context_switch + testq %rax, %rax + jz 1f + + /* + * When we come here, everything but stack switching is finished. + * This makes current task use another task's kernel stack. Thus, + * we need to do stack switching here. + * + * At the meanwhile, the previous task's stack content is corrupted, + * we also need to rebuild its stack frames, so that it will jump to + * rpal_ret_from_lazy_switch when it is scheduled in. This is inspired + * by ret_from_fork. + */ + movq TASK_threadsp(%rax), %rsp +#ifdef CONFIG_STACKPROTECTOR + movq TASK_stack_canary(%rax), %rbx + movq %rbx, PER_CPU_VAR(__stack_chk_guard) +#endif + /* rebuild src's frame */ + movq $rpal_ret_from_lazy_switch, -8(%r14) + leaq RPAL_FRAME_RET_ADDR_OFFSET(%r14), %rbx + movq %rbx, TASK_threadsp(%r13) + + movq %r13, %rdi + /* + * Everything of task switch is done, but we still need to do + * a little extra things for lazy switch. + */ + call rpal_lazy_switch_tail + +1: + movq ORIG_RAX(%rsp), %rsi + movq %rsp, %rdi + jmp _do_syscall +SYM_CODE_END(do_rpal_syscall) + +SYM_CODE_START(do_rpal_error) + popq %r12 + movq %rax, %rsp + movq %rax, %r14 + movq %rax, %rdi + call rpal_exception_context_switch + testq %rax, %rax + jz 1f + + movq TASK_threadsp(%rax), %rsp + ENCODE_FRAME_POINTER +#ifdef CONFIG_STACKPROTECTOR + movq TASK_stack_canary(%rax), %rbx + movq %rbx, PER_CPU_VAR(__stack_chk_guard) +#endif + /* rebuild src's frame */ + movq $rpal_ret_from_lazy_switch, -8(%r14) + leaq RPAL_FRAME_RET_ADDR_OFFSET(%r14), %rbx + movq %rbx, TASK_threadsp(%r13) + + movq %r13, %rdi + call rpal_lazy_switch_tail +1: + movq %rsp, %rax + pushq %r12 + jmp _do_error +SYM_CODE_END(do_rpal_error) + +SYM_CODE_START(do_rpal_nmi) + movq %rsp, %r14 + movq %rsp, %rdi + call rpal_nmi_context_switch + testq %rax, %rax + jz 1f + + movq TASK_threadsp(%rax), %rsp + ENCODE_FRAME_POINTER +#ifdef CONFIG_STACKPROTECTOR + movq TASK_stack_canary(%rax), %rbx + movq %rbx, PER_CPU_VAR(__stack_chk_guard) +#endif + /* rebuild src's frame */ + movq $rpal_ret_from_lazy_switch, -8(%r14) + leaq RPAL_FRAME_RET_ADDR_OFFSET(%r14), %rbx + movq %rbx, TASK_threadsp(%r13) + + movq %r13, %rdi + call rpal_lazy_switch_tail + +1: + jmp _do_nmi +SYM_CODE_END(do_rpal_nmi) #endif /* @@ -1047,7 +1156,22 @@ SYM_CODE_START(error_entry) leaq 8(%rsp), %rdi /* arg0 = pt_regs pointer */ /* Put us onto the real thread stack. */ +#ifdef CONFIG_RPAL + call sync_regs + /* + * Check whether we need to perform lazy switch after we + * switch to the real thread stack. + */ + movq PER_CPU_VAR(current_task), %r13 + movq TASK_rpal_cd(%r13), %rdi + testq %rdi, %rdi + jz _do_error + jmp do_rpal_error +_do_error: + RET +#else jmp sync_regs +#endif /* * There are two places in the kernel that can potentially fault with @@ -1206,6 +1330,19 @@ SYM_CODE_START(asm_exc_nmi) IBRS_ENTER UNTRAIN_RET +#ifdef CONFIG_RPAL + /* + * Check whether we need to perform lazy switch only when + * we come from userspace. + */ + movq PER_CPU_VAR(current_task), %r13 + movq TASK_rpal_cd(%r13), %rax + testq %rax, %rax + jz _do_nmi + jmp do_rpal_nmi +_do_nmi: +#endif + /* * At this point we no longer need to worry about stack damage * due to nesting -- we're on the normal thread stack and we're diff --git a/arch/x86/kernel/asm-offsets.c b/arch/x86/kernel/asm-offsets.c index 6259b474073b..010202c31b37 100644 --- a/arch/x86/kernel/asm-offsets.c +++ b/arch/x86/kernel/asm-offsets.c @@ -46,6 +46,9 @@ static void __used common(void) #ifdef CONFIG_STACKPROTECTOR OFFSET(TASK_stack_canary, task_struct, stack_canary); #endif +#ifdef CONFIG_RPAL + OFFSET(TASK_rpal_cd, task_struct, rpal_cd); +#endif BLANK(); OFFSET(pbe_address, pbe, address); diff --git a/arch/x86/rpal/core.c b/arch/x86/rpal/core.c index ed4c11e6838c..c48df1ce4324 100644 --- a/arch/x86/rpal/core.c +++ b/arch/x86/rpal/core.c @@ -7,6 +7,7 @@ */ #include +#include #include #include "internal.h" @@ -39,6 +40,20 @@ static inline void rpal_unlock_cpu_kernel_ret(struct task_struct *tsk) rpal_set_cpus_allowed_ptr(tsk, false, true); } +void rpal_lazy_switch_tail(struct task_struct *tsk) +{ + struct rpal_receiver_call_context *rcc; + + if (rpal_test_task_thread_flag(current, RPAL_LAZY_SWITCHED_BIT)) { + rcc = current->rpal_rd->rcc; + atomic_cmpxchg(&rcc->receiver_state, rpal_build_call_state(tsk->rpal_sd), + RPAL_RECEIVER_STATE_LAZY_SWITCH); + } else { + rpal_unlock_cpu(tsk); + rpal_unlock_cpu(current); + } +} + void rpal_kernel_ret(struct pt_regs *regs) { struct task_struct *tsk; @@ -76,6 +91,87 @@ void rpal_kernel_ret(struct pt_regs *regs) } } +static inline void rebuild_stack(struct rpal_task_context *ctx, + struct pt_regs *regs) +{ + regs->r12 = ctx->r12; + regs->r13 = ctx->r13; + regs->r14 = ctx->r14; + regs->r15 = ctx->r15; + regs->bx = ctx->rbx; + regs->bp = ctx->rbp; + regs->ip = ctx->rip; + regs->sp = ctx->rsp; +} + +static inline void rebuild_sender_stack(struct rpal_sender_data *rsd, + struct pt_regs *regs) +{ + rebuild_stack(&rsd->scc->rtc, regs); +} + +static inline void rebuild_receiver_stack(struct rpal_receiver_data *rrd, + struct pt_regs *regs) +{ + rebuild_stack(&rrd->rcc->rtc, regs); +} + +static inline void update_dst_stack(struct task_struct *next, + struct pt_regs *src) +{ + struct pt_regs *dst; + + dst = task_pt_regs(next); + *dst = *src; + next->thread.sp = (unsigned long)dst; +} + +/* + * rpal_do_kernel_context_switch - the main routine of RPAL lazy switch + * @next: task to switch to + * @regs: the user pt_regs saved in kernel entry + * + * This function performs the lazy switch. When switch from sender to + * receiver, we need to lock both task to current CPU to avoid double + * control flow when we perform lazy switch and after then. + */ +static struct task_struct * +rpal_do_kernel_context_switch(struct task_struct *next, struct pt_regs *regs) +{ + struct task_struct *prev = current; + + if (rpal_test_task_thread_flag(next, RPAL_LAZY_SWITCHED_BIT)) { + current->rpal_sd->receiver = next; + rpal_lock_cpu(current); + rpal_lock_cpu(next); + rpal_try_to_wake_up(next); + update_dst_stack(next, regs); + /* + * When a lazy switch occurs, we need to set the sender's + * user-mode context to a predefined state by the sender. + * Otherwise, sender's user context will be corrupted. + */ + rebuild_sender_stack(current->rpal_sd, regs); + rpal_schedule(next); + } else { + update_dst_stack(next, regs); + /* + * When a lazy switch occurs, we need to set the receiver's + * user-mode context to a predefined state by the receiver. + * Otherwise, sender's user context will be corrupted. + */ + rebuild_receiver_stack(current->rpal_rd, regs); + rpal_schedule(next); + rpal_clear_task_thread_flag(prev, RPAL_LAZY_SWITCHED_BIT); + prev->rpal_rd->sender = NULL; + } + if (unlikely(!irqs_disabled())) { + local_irq_disable(); + rpal_err("%s: irq is enabled\n", __func__); + } + return next; +} + static inline struct task_struct *rpal_get_sender_task(void) { struct task_struct *next; @@ -123,6 +219,18 @@ static inline struct task_struct *rpal_misidentify(void) return next; } +static inline struct task_struct * +rpal_kernel_context_switch(struct pt_regs *regs) +{ + struct task_struct *next = NULL; + + next = rpal_misidentify(); + if (unlikely(next != NULL)) + next = rpal_do_kernel_context_switch(next, regs); + + return next; +} + struct task_struct *rpal_find_next_task(unsigned long fsbase) { struct rpal_service *cur = rpal_current_service(); @@ -147,6 +255,35 @@ struct task_struct *rpal_find_next_task(unsigned long fsbase) return tsk; } +__visible struct task_struct * +rpal_syscall_64_context_switch(struct pt_regs *regs, unsigned long nr) +{ + struct task_struct *next; + + next = rpal_kernel_context_switch(regs); + + return next; +} + +__visible struct task_struct * +rpal_exception_context_switch(struct pt_regs *regs) +{ + struct task_struct *next; + + next = rpal_kernel_context_switch(regs); + + return next; +} + +__visible struct task_struct *rpal_nmi_context_switch(struct pt_regs *regs) +{ + struct task_struct *next; + + next = rpal_kernel_context_switch(regs); + + return next; +} + static bool check_hardware_features(void) { if (!boot_cpu_has(X86_FEATURE_FSGSBASE)) { diff --git a/include/linux/rpal.h b/include/linux/rpal.h index 01b582fa821e..b24176f3f245 100644 --- a/include/linux/rpal.h +++ b/include/linux/rpal.h @@ -479,7 +479,13 @@ struct rpal_service *rpal_get_mapped_service_by_id(struct rpal_service *rs, int rpal_rebuild_sender_context_on_fault(struct pt_regs *regs, unsigned long addr, int error_code); struct mm_struct *rpal_pf_get_real_mm(unsigned long address, int *rebuild); +__visible struct task_struct * +rpal_syscall_64_context_switch(struct pt_regs *regs, unsigned long nr); +__visible struct task_struct * +rpal_exception_context_switch(struct pt_regs *regs); +__visible struct task_struct *rpal_nmi_context_switch(struct pt_regs *regs); struct task_struct *rpal_find_next_task(unsigned long fsbase); +void rpal_lazy_switch_tail(struct task_struct *tsk); void rpal_kernel_ret(struct pt_regs *regs); extern void rpal_pick_mmap_base(struct mm_struct *mm, -- 2.20.1