From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 1A5DCC5B549 for ; Fri, 30 May 2025 09:32:21 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B34F96B00AD; Fri, 30 May 2025 05:32:20 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id AE5446B00AE; Fri, 30 May 2025 05:32:20 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 986DC6B00AF; Fri, 30 May 2025 05:32:20 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 6D3396B00AD for ; Fri, 30 May 2025 05:32:20 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 2CE66142B04 for ; Fri, 30 May 2025 09:32:20 +0000 (UTC) X-FDA: 83499058440.05.51E6420 Received: from mail-pj1-f51.google.com (mail-pj1-f51.google.com [209.85.216.51]) by imf14.hostedemail.com (Postfix) with ESMTP id 30B8210000C for ; Fri, 30 May 2025 09:32:18 +0000 (UTC) Authentication-Results: imf14.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=HTRhY4Ht; spf=pass (imf14.hostedemail.com: domain of libo.gcs85@bytedance.com designates 209.85.216.51 as permitted sender) smtp.mailfrom=libo.gcs85@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1748597538; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=WgdxFQILtv1ZFBwGDlokBNLY9DZwyLg+1V8wfJcAxqs=; b=VvDlMAeIEs+pBI7NvJzfWr7GQQ7kjfnvnLWGnVBuSLSxYA8xhFXP/cXdHfr0BB/m7jlVuJ XNAqpj8i3EYalcUdf9VHz6g6G+bEAyf/Z4g36w1Y8E04wRAI4o8KcanugwEqU4jWisTPjS quaK7f39HtTLloilxpo6q7yImePx4rA= ARC-Authentication-Results: i=1; imf14.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=HTRhY4Ht; spf=pass (imf14.hostedemail.com: domain of libo.gcs85@bytedance.com designates 209.85.216.51 as permitted sender) smtp.mailfrom=libo.gcs85@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1748597538; a=rsa-sha256; cv=none; b=KVM+egxc2j0OiHjMeY+cQk2gi3Tm4+pdnuNKXR4hK+dtli4/FX0gLc9HspKg5wc3OZFClu xFFjMPD7dV/T+/4ecJVBMAwhUKHdNvQLrMzc+GyLiCEVSyjTu0boVmOmiV22vg3Qu2SZ0J 4u70ekdPEtp4jYR4MNEsx5/6qaBnRhQ= Received: by mail-pj1-f51.google.com with SMTP id 98e67ed59e1d1-311a6236effso1361696a91.2 for ; Fri, 30 May 2025 02:32:17 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1748597537; x=1749202337; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=WgdxFQILtv1ZFBwGDlokBNLY9DZwyLg+1V8wfJcAxqs=; b=HTRhY4HtXpBR/JDpHy0B/r1mDaKJy94EGNImjpmhAUB6Ghp6WMjEtjUxlJhAvYLPJN OqJ0qHDVebBFAAXQHId8Now6RWD4fiAuPHkGa9sfEr2yTC15tee7CQH1YNIXVpUZcVMT sSbnKpaKATLPaMpTaN4Zutkv3DM6umvbppjU0dnKKdmthqTd5asxt9Xm9NVNJ4cs9ME+ PCSD9XHI91ZTzT8dudjVI/fTHIQ2vS5r9ZcKNshf7MlawTHOuA7Cdyh19ml7ux2w15mq HMPk78G+KgPxzoF2MjV6gO4ZD15rHwn0tnHN2y6aVS5DUEfcKBJtsUStYxZC1ZUZ6Tr2 BNiw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748597537; x=1749202337; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=WgdxFQILtv1ZFBwGDlokBNLY9DZwyLg+1V8wfJcAxqs=; b=ZEKmY1kUQivofSpaVz7/4zD8fGCLmyCYcUC5qRxDX1JFqDmKhrBjjM0PZNb6br5gTR e65C4gHxqq75WILWROOyzHMPM9LWkKdP4Jg3m90YGnbqS/S8BaQsdWzIG+bMqxAThJR5 wGlpdf4LYzDrswAMhK7eWVCo/v5PUSGFfOVJ+HjwrDke67sMTciL+qqrsHboSXu5aMfP IZ6hoEze6BSni77JDU3CxUC6VJGJNweB3Nvro0dzztbUGcIYuG0LDqNq+//yUNmOcmwD Cg9+wQpEKJ/q+mryiieRJULCkxwXpTpdHYiyx2LDo9nsCIOXuCqq3Uqo0Lw2dgPhn1Z6 aCAA== X-Forwarded-Encrypted: i=1; AJvYcCU0rZY+Oa4QzDY0Fq9NJd5MphJ2G8NEG7kdtD77s11I0F+9/XRbGZXRzYT6gbdzsfDD3w2epkGCIg==@kvack.org X-Gm-Message-State: AOJu0YyesAL83VKpLw4ZZkR8m4BCb/MRc6KUlRH7vCSYj+qx7ZTu8nxf 9vKWCb1LGUPQzpLRBMxPZvlcRiqmKsvkiYEra2MLh+aHQpUDgcwB5wEYAlU1MT+mwuw= X-Gm-Gg: ASbGnct7TAAseC5Ew2NMl0bBLXH35+OLiup4dYNQv984nfs2s2MjN2pbczFs+0HMmeE tzVSG3NVuGj94bgt0vEX7bOv8bWUQmb2sH42jQp0LnB62mMz0NQbG06L05d46lt6mRUDhykWmiZ 6u2JleI0EIw0mpAq66bkDsU3WXqmxrnZttrDQC9nM1V1ohdzQgiQ3uu3E9NVvHHLSsCWBpXPTJJ JW7U4SyZwDD8LgJ4YWw7PKt5dKsJSgwKDxqGl8U5rfuDMGHxZq5rPBjhqK+/Qx3AUqj6FK9sfM9 OYHzmAJGY3F4nfaMSS0sqby7/6XV248zoqM1swGDLbX0aUypZJ6NbASXCqoy/hnko2e9lcKhOWp n4ST8s5KQCg== X-Google-Smtp-Source: AGHT+IEJreThnxFmdrlfjlpwKZv59IFtRtCaH2wQICekYkicVpLMTveAapFUeJrWIU6RsHlorrZhig== X-Received: by 2002:a17:90b:314d:b0:311:c1ec:7d0c with SMTP id 98e67ed59e1d1-312504437b2mr2132223a91.27.1748597536883; Fri, 30 May 2025 02:32:16 -0700 (PDT) Received: from FQ627FTG20.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-3124e29f7b8sm838724a91.2.2025.05.30.02.32.02 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 30 May 2025 02:32:16 -0700 (PDT) From: Bo Li To: tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, luto@kernel.org, kees@kernel.org, akpm@linux-foundation.org, david@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org, peterz@infradead.org Cc: dietmar.eggemann@arm.com, hpa@zytor.com, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com, viro@zeniv.linux.org.uk, brauner@kernel.org, jack@suse.cz, lorenzo.stoakes@oracle.com, Liam.Howlett@oracle.com, vbabka@suse.cz, rppt@kernel.org, surenb@google.com, mhocko@suse.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, jannh@google.com, pfalcato@suse.de, riel@surriel.com, harry.yoo@oracle.com, linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, yinhongbo@bytedance.com, dengliang.1214@bytedance.com, xieyongji@bytedance.com, chaiwen.cc@bytedance.com, songmuchun@bytedance.com, yuanzhu@bytedance.com, chengguozhu@bytedance.com, sunjiadong.lff@bytedance.com, Bo Li Subject: [RFC v2 15/35] RPAL: add sender/receiver state Date: Fri, 30 May 2025 17:27:43 +0800 Message-Id: <6582d600063dd2176558bdf2b62a6a143bd594e2.1748594840.git.libo.gcs85@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Stat-Signature: 8wfag7qkkhfxygo7qpaq3w3scn1izoq8 X-Rspamd-Queue-Id: 30B8210000C X-Rspamd-Server: rspam11 X-HE-Tag: 1748597537-501401 X-HE-Meta: U2FsdGVkX1/ydUr2+au4sUrSt/7JUichRDD+KJT9x7EiauJ0JDU72QDgiCo7/TjG2Mx/yGcW+fDsXfg8wKEI1Io1SXN6xuVakjtsfQqQEou7gZXXgZhqEF3fmcpyVcJPZHS616MKIEKeWeSFc4N8cs9Z6DEwJFTe6lCD9BEpfCgFV2+RXfQLiRYBkoBcZaS3VZtVu52Oyf0Jr/6ksi12RsNM2cXKm/fkgVEHst257C5KiCUuDA9C8lyLbY9xkC0ujizPlaBZ13aK8OKN9V3NJ9nunvKJodaEkgagyfENTuvAeZBcnbWq/4ubkmZBYxllSYvXjw89CktvWOIQhxcHOsW/3qfpSRVKB7ODaiaql4MHDoTDLewaHDnA4AwnRaE8BXDyWGNULwUX83LGe6kR0vrcPAWCf2m1Bw6NR7se3m/oeXNluUAcsxIvhaMHqTPMGpuUfwO9q99SNHawzcw6IFDRMcRTQzpTPPXZmCoIxtDWH3eHRgPG9HEuXZnA8qTyvC2hbaB0F+POGyxfcZN7zNiqS0e29uY66P1X51q+CFPUehcfpKnxHZM7P0ZEkpkvWRYIKSStQTzDTs+bPPNCKM5sAnPJbQeTz52F9hfAbzj+oiMfIVEoNE3qryL3S7GR/qtp9XtKBknfds08OPUGXIsX+K74vTnt3jc07+ZqMVQ24OufFp77mFeMw+I+sLk/HDAAguDlDDjifeqXp4UK2HwpLtmCBFMHSYwTO75mI203MhgEVQEaBYm2lvNDpvjBWAsR45mAH1Pw6XiZ4Go1113+jRiLJqF2HZkcfa4ZNoV1+mDgN3ayIiycsZBoDRnHSRzcXHE3mfCMX9xC9Xdp9iDEg3zKFHOeIezor7hzw7lIzNl4PEYfWl0qYjtodWPcZa/iXioxIrzYaM15R56SdsClMUb8x8doAKAf/7SQtFNxmdtfzm3brJlgbNGe8h4kKbIzs/ocl7h3aiSlE42 1jxAYnLa HgMZHjubiNHrXH9FrNHRBc4AwQfc/B4Q8Vzr4uKUZ4YwP7a+BZ528Kn4nJcJDEduANsMxvkKtHtOuwWO9COgo/TiWNcDeGVWBD/Zpv7V6fiYmvEMGJ71cCBS24jM2J9neB7pTdEahAITUdi1nFVFiiUBerm3pmTvG12LIoSLzIfbQ7L43GIv0+cJ0S8+EZ26T1t1OhqNjgVMfmYCa/ZXpSeRRSWq6+AqvOl5iz7nLHzbgDnNWNzr7D2rrlBqdpT7XIx/kwGmGud2QZ6RWHMpa2Ib62xQaa2s0vGwsip90rAqihkJlJf/2LGOPJQ== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: The lazy switch defines six receiver states, and their state transitions are as follows: |<--->READY<----> WAIT <----> CALL ----> LAZY_SWITCH ---> KERNEL_RET | | | RUNNING <----------------------------------------|---------------| The receiver thread initially starts in the RUNNING state and can transition to the WAIT state voluntarily. The READY state is a temporary state before entering WAIT state. For a receiver in the WAIT state, it must be in the TASK_INTERRUPTIBLE state. If the receiver thread is woken up, the WAIT state can transition to the RUNNING state. Once the receiver is in the WAIT state, the sender thread can initiate an RPAL call, causing the receiver to enter the CALL state. A receiver thread in the CALL state cannot be awakened until a lazy switch occurs or its state changes. The call state carries additional service_id and sender_id information. If the sender completes executing the receiver's code without entering the kernel after issuing the RPAL call, the receiver transitions back from the CALL state to the WAIT state. Conversely, if the sender enters the kernel during the RPAL call, the receiver's state changes to LAZY_SWITCH. >From the LAZY_SWITCH state, the receiver thread has two possible state transitions: When the receiver thread finishes execution and switches back to the sender via a lazy switch, it first enters the KERNEL_RET state and then transitions to the RUNNING state. If the receiver thread runs for too long and the scheduler resumes the sender, the receiver directly transitions to the RUNNING state. Transitions to the RUNNING state can be done in userspace. The lazy switch mechanism defines three states for the sender thread: - RUNNING: The sender starts in this state. When the sender initiates an RPAL call to switch from user mode to the receiver, it transitions to the CALL state. - CALL: The sender remains in this state while the receiver is executing the code triggered by the RPAL call. When the receiver switches back to the sender from user mode, the sender returns to the RUNNING state. - KERNEL_RET: If the receiver takes an extended period to switch back to the sender after a lazy switch, the scheduler may preempt the sender to run other tasks. In this case, the sender enters the KERNEL_RET state while in the kernel. Once the sender resumes execution in user mode, it transitions back to the RUNNING state. This patch implements the handling and transition of the receiver's state. When a receiver leaves the run queue in the READY state, its state transitions to the WAIT state; otherwise, it transitions to the RUNNING state. The patch also modifies try_to_wake_up() to handling different states: for the READY and WAIT states, try_to_wake_up() causes the state to change to the RUNNING state. For the CALL state, try_to_wake_up() cannot wake up the task. The patch provides a special interface, rpal_try_to_wake_up(), to wake up tasks in the CALL state, which can be used for lazy switches. Signed-off-by: Bo Li --- arch/x86/kernel/process_64.c | 43 ++++++++++++ arch/x86/rpal/internal.h | 7 ++ include/linux/rpal.h | 50 ++++++++++++++ kernel/sched/core.c | 130 +++++++++++++++++++++++++++++++++++ 4 files changed, 230 insertions(+) diff --git a/arch/x86/kernel/process_64.c b/arch/x86/kernel/process_64.c index f39ff02e498d..4830e9215de7 100644 --- a/arch/x86/kernel/process_64.c +++ b/arch/x86/kernel/process_64.c @@ -40,6 +40,7 @@ #include #include #include +#include #include #include @@ -596,6 +597,36 @@ void compat_start_thread(struct pt_regs *regs, u32 new_ip, u32 new_sp, bool x32) } #endif +#ifdef CONFIG_RPAL +static void rpal_receiver_enter_wait(struct task_struct *prev_p) +{ + if (READ_ONCE(prev_p->__state) == TASK_INTERRUPTIBLE) { + atomic_cmpxchg(&prev_p->rpal_rd->rcc->receiver_state, + RPAL_RECEIVER_STATE_READY, + RPAL_RECEIVER_STATE_WAIT); + } else { + /* + * Simply check RPAL_RECEIVER_STATE_READY is not enough. It is + * possible task's state is TASK_RUNNING. Consider following case: + * + * CPU 0(prev_p) CPU 1(waker) + * set TASK_INTERRUPTIBLE + * set RPAL_RECEIVER_STATE_READY + * check TASK_INTERRUPTIBLE + * clear RPAL_RECEIVER_STATE_READY + * clear TASK_INTERRUPTIBLE + * set TASK_INTERRUPTIBLE + * set RPAL_RECEIVER_STATE_READY + * ttwu_runnable() + * schedule() + */ + atomic_cmpxchg(&prev_p->rpal_rd->rcc->receiver_state, + RPAL_RECEIVER_STATE_READY, + RPAL_RECEIVER_STATE_RUNNING); + } +} +#endif + /* * switch_to(x,y) should switch tasks from x to y. * @@ -704,6 +735,18 @@ __switch_to(struct task_struct *prev_p, struct task_struct *next_p) loadsegment(ss, __KERNEL_DS); } +#ifdef CONFIG_RPAL + /* + * When we come to here, the stack switching is finished. Therefore, + * the receiver thread is prepared for a lazy switch. We then change + * the receiver_state from RPAL_RECEIVER_STATE_REAY to + * RPAL_RECEIVER_STATE_WAIT and other thread is able to call it with + * RPAL call. + */ + if (rpal_test_task_thread_flag(prev_p, RPAL_RECEIVER_BIT)) + rpal_receiver_enter_wait(prev_p); +#endif + /* Load the Intel cache allocation PQR MSR. */ resctrl_sched_in(next_p); diff --git a/arch/x86/rpal/internal.h b/arch/x86/rpal/internal.h index cf6d608a994a..6256172bb79e 100644 --- a/arch/x86/rpal/internal.h +++ b/arch/x86/rpal/internal.h @@ -47,3 +47,10 @@ int rpal_unregister_sender(void); int rpal_register_receiver(unsigned long addr); int rpal_unregister_receiver(void); void exit_rpal_thread(void); + +static inline unsigned long +rpal_build_call_state(const struct rpal_sender_data *rsd) +{ + return ((rsd->rcd.service_id << RPAL_SID_SHIFT) | + (rsd->scc->sender_id << RPAL_ID_SHIFT) | RPAL_RECEIVER_STATE_CALL); +} diff --git a/include/linux/rpal.h b/include/linux/rpal.h index 3310d222739e..4f4719bb7eae 100644 --- a/include/linux/rpal.h +++ b/include/linux/rpal.h @@ -87,6 +87,13 @@ enum { #define RPAL_ERROR_MAGIC 0x98CC98CC +#define RPAL_SID_SHIFT 24 +#define RPAL_ID_SHIFT 8 +#define RPAL_RECEIVER_STATE_MASK ((1 << RPAL_ID_SHIFT) - 1) +#define RPAL_SID_MASK (~((1 << RPAL_SID_SHIFT) - 1)) +#define RPAL_ID_MASK (~(0 | RPAL_RECEIVER_STATE_MASK | RPAL_SID_MASK)) +#define RPAL_MAX_ID ((1 << (RPAL_SID_SHIFT - RPAL_ID_SHIFT)) - 1) + extern unsigned long rpal_cap; enum rpal_task_flag_bits { @@ -94,6 +101,22 @@ enum rpal_task_flag_bits { RPAL_RECEIVER_BIT, }; +enum rpal_receiver_state { + RPAL_RECEIVER_STATE_RUNNING, + RPAL_RECEIVER_STATE_KERNEL_RET, + RPAL_RECEIVER_STATE_READY, + RPAL_RECEIVER_STATE_WAIT, + RPAL_RECEIVER_STATE_CALL, + RPAL_RECEIVER_STATE_LAZY_SWITCH, + RPAL_RECEIVER_STATE_MAX, +}; + +enum rpal_sender_state { + RPAL_SENDER_STATE_RUNNING, + RPAL_SENDER_STATE_CALL, + RPAL_SENDER_STATE_KERNEL_RET, +}; + /* * user_meta will be sent to other service when requested. */ @@ -215,6 +238,8 @@ struct rpal_task_context { struct rpal_receiver_call_context { struct rpal_task_context rtc; int receiver_id; + atomic_t receiver_state; + atomic_t sender_state; }; /* recovery point for sender */ @@ -390,11 +415,35 @@ static inline bool rpal_test_current_thread_flag(unsigned long bit) { return test_bit(bit, ¤t->rpal_flag); } + +static inline bool rpal_test_task_thread_flag(struct task_struct *tsk, + unsigned long bit) +{ + return test_bit(bit, &tsk->rpal_flag); +} + +static inline void rpal_set_task_thread_flag(struct task_struct *tsk, + unsigned long bit) +{ + set_bit(bit, &tsk->rpal_flag); +} + +static inline void rpal_clear_task_thread_flag(struct task_struct *tsk, + unsigned long bit) +{ + clear_bit(bit, &tsk->rpal_flag); +} #else static inline struct rpal_service *rpal_current_service(void) { return NULL; } static inline void rpal_set_current_thread_flag(unsigned long bit) { } static inline void rpal_clear_current_thread_flag(unsigned long bit) { } static inline bool rpal_test_current_thread_flag(unsigned long bit) { return false; } +static inline bool rpal_test_task_thread_flag(struct task_struct *tsk, + unsigned long bit) { return false; } +static inline void rpal_set_task_thread_flag(struct task_struct *tsk, + unsigned long bit) { } +static inline void rpal_clear_task_thread_flag(struct task_struct *tsk, + unsigned long bit) { } #endif void rpal_unregister_service(struct rpal_service *rs); @@ -414,4 +463,5 @@ struct mm_struct *rpal_pf_get_real_mm(unsigned long address, int *rebuild); extern void rpal_pick_mmap_base(struct mm_struct *mm, struct rlimit *rlim_stack); +int rpal_try_to_wake_up(struct task_struct *p); #endif diff --git a/kernel/sched/core.c b/kernel/sched/core.c index 62b3416f5e43..045e92ee2e3b 100644 --- a/kernel/sched/core.c +++ b/kernel/sched/core.c @@ -67,6 +67,7 @@ #include #include #include +#include #ifdef CONFIG_PREEMPT_DYNAMIC # ifdef CONFIG_GENERIC_ENTRY @@ -3820,6 +3821,40 @@ static int ttwu_runnable(struct task_struct *p, int wake_flags) return ret; } +#ifdef CONFIG_RPAL +static bool rpal_check_state(struct task_struct *p) +{ + bool ret = true; + + if (rpal_test_task_thread_flag(p, RPAL_RECEIVER_BIT)) { + struct rpal_receiver_call_context *rcc = p->rpal_rd->rcc; + int state; + +retry: + state = atomic_read(&rcc->receiver_state) & RPAL_RECEIVER_STATE_MASK; + switch (state) { + case RPAL_RECEIVER_STATE_READY: + case RPAL_RECEIVER_STATE_WAIT: + if (state != atomic_cmpxchg(&rcc->receiver_state, state, + RPAL_RECEIVER_STATE_RUNNING)) + goto retry; + break; + case RPAL_RECEIVER_STATE_KERNEL_RET: + case RPAL_RECEIVER_STATE_LAZY_SWITCH: + case RPAL_RECEIVER_STATE_RUNNING: + break; + case RPAL_RECEIVER_STATE_CALL: + ret = false; + break; + default: + rpal_err("%s: invalid state: %d\n", __func__, state); + break; + } + } + return ret; +} +#endif + #ifdef CONFIG_SMP void sched_ttwu_pending(void *arg) { @@ -3841,6 +3876,11 @@ void sched_ttwu_pending(void *arg) if (WARN_ON_ONCE(task_cpu(p) != cpu_of(rq))) set_task_cpu(p, cpu_of(rq)); +#ifdef CONFIG_RPAL + if (!rpal_check_state(p)) + continue; +#endif + ttwu_do_activate(rq, p, p->sched_remote_wakeup ? WF_MIGRATED : 0, &rf); } @@ -4208,6 +4248,17 @@ int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) if (!ttwu_state_match(p, state, &success)) goto out; +#ifdef CONFIG_RPAL + /* + * For rpal thread, we need to check if it can be woken up. If not, + * we do not wake it up here but wake it up later by kernel worker. + * + * For normal thread, nothing happens. + */ + if (!rpal_check_state(p)) + goto out; +#endif + trace_sched_waking(p); ttwu_do_wakeup(p); goto out; @@ -4224,6 +4275,11 @@ int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) if (!ttwu_state_match(p, state, &success)) break; +#ifdef CONFIG_RPAL + if (!rpal_check_state(p)) + break; +#endif + trace_sched_waking(p); /* @@ -4344,6 +4400,56 @@ int try_to_wake_up(struct task_struct *p, unsigned int state, int wake_flags) return success; } +#ifdef CONFIG_RPAL +int rpal_try_to_wake_up(struct task_struct *p) +{ + guard(preempt)(); + int cpu, success = 0; + int wake_flags = WF_TTWU; + + BUG_ON(READ_ONCE(p->__state) == TASK_RUNNING); + + scoped_guard (raw_spinlock_irqsave, &p->pi_lock) { + smp_mb__after_spinlock(); + if (!ttwu_state_match(p, TASK_NORMAL, &success)) + break; + + trace_sched_waking(p); + /* see try_to_wake_up() */ + smp_rmb(); + +#ifdef CONFIG_SMP + smp_acquire__after_ctrl_dep(); + WRITE_ONCE(p->__state, TASK_WAKING); + /* see try_to_wake_up() */ + if (smp_load_acquire(&p->on_cpu) && + ttwu_queue_wakelist(p, task_cpu(p), wake_flags)) + break; + smp_cond_load_acquire(&p->on_cpu, !VAL); + + cpu = select_task_rq(p, p->wake_cpu, &wake_flags); + if (task_cpu(p) != cpu) { + if (p->in_iowait) { + delayacct_blkio_end(p); + atomic_dec(&task_rq(p)->nr_iowait); + } + + wake_flags |= WF_MIGRATED; + psi_ttwu_dequeue(p); + set_task_cpu(p, cpu); + } +#else + cpu = task_cpu(p); +#endif + } + ttwu_queue(p, cpu, wake_flags); + if (success) + ttwu_stat(p, task_cpu(p), wake_flags); + + return success; +} +#endif + static bool __task_needs_rq_lock(struct task_struct *p) { unsigned int state = READ_ONCE(p->__state); @@ -6574,6 +6680,18 @@ pick_next_task(struct rq *rq, struct task_struct *prev, struct rq_flags *rf) #define SM_PREEMPT 1 #define SM_RTLOCK_WAIT 2 +#ifdef CONFIG_RPAL +static inline void rpal_check_ready_state(struct task_struct *tsk, int state) +{ + if (rpal_test_task_thread_flag(tsk, RPAL_RECEIVER_BIT)) { + struct rpal_receiver_call_context *rcc = tsk->rpal_rd->rcc; + + atomic_cmpxchg(&rcc->receiver_state, state, + RPAL_RECEIVER_STATE_RUNNING); + } +} +#endif + /* * Helper function for __schedule() * @@ -6727,7 +6845,19 @@ static void __sched notrace __schedule(int sched_mode) goto picked; } } else if (!preempt && prev_state) { +#ifdef CONFIG_RPAL + if (!try_to_block_task(rq, prev, &prev_state)) { + /* + * As the task enter TASK_RUNNING state, we should clean up + * RPAL_RECEIVER_STATE_READY status. Therefore, the receiver's + * state will not be change to RPAL_RECEIVER_STATE_WAIT. Thus, + * there is no RPAL call when a receiver is at TASK_RUNNING state. + */ + rpal_check_ready_state(prev, RPAL_RECEIVER_STATE_READY); + } +#else try_to_block_task(rq, prev, &prev_state); +#endif switch_count = &prev->nvcsw; } -- 2.20.1