From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BE8A2C5B549 for ; Fri, 30 May 2025 09:32:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 62CC46B00B1; Fri, 30 May 2025 05:32:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5DCB36B00B2; Fri, 30 May 2025 05:32:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 47DD46B00B3; Fri, 30 May 2025 05:32:51 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 24F896B00B1 for ; Fri, 30 May 2025 05:32:51 -0400 (EDT) Received: from smtpin17.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id D1A88C2A8F for ; Fri, 30 May 2025 09:32:50 +0000 (UTC) X-FDA: 83499059700.17.2222CFD Received: from mail-pl1-f182.google.com (mail-pl1-f182.google.com [209.85.214.182]) by imf20.hostedemail.com (Postfix) with ESMTP id 05BB01C0005 for ; Fri, 30 May 2025 09:32:48 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=K1O+fR63; spf=pass (imf20.hostedemail.com: domain of libo.gcs85@bytedance.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=libo.gcs85@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1748597569; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=7odL+uROHJjULwRoyw6w9QE9qQsYAMCGXEhdmggZkMI=; b=t1cIfsNmkJ8nyxztTLRKwCq1QaZNRxrlDNNRni1t/z39JSWRiWQM6Iut8KUpi4ymB4WCS2 hH+L1f0pK1P7azuLmd/5+S5xFuEySq+Qr8j5ppk+HCCn8Y3fk1bGqpeJigHRPUBh8hsYxa zmHAvpPn3xfJ6yLZX1InyPcKteJFtlg= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=bytedance.com header.s=google header.b=K1O+fR63; spf=pass (imf20.hostedemail.com: domain of libo.gcs85@bytedance.com designates 209.85.214.182 as permitted sender) smtp.mailfrom=libo.gcs85@bytedance.com; dmarc=pass (policy=quarantine) header.from=bytedance.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1748597569; a=rsa-sha256; cv=none; b=HFcYMxzuvrhbbz0AfEK5IqyisARidRYyjB8b2zAubf0vOMsUI8CZkpTayvKg4XTR9skMqV +qgpOrNfLhrSUv5XKtYkqyOfNGNmNOsEyogZlvja1uKHP0KNMauOIU1ShEuoX+iGmPdW0M 4QNhRtXWIbqmLoeF3EHqtGwUDjp0+5U= Received: by mail-pl1-f182.google.com with SMTP id d9443c01a7336-22e09f57ed4so26541035ad.0 for ; Fri, 30 May 2025 02:32:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=bytedance.com; s=google; t=1748597568; x=1749202368; darn=kvack.org; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:from:to:cc:subject:date :message-id:reply-to; bh=7odL+uROHJjULwRoyw6w9QE9qQsYAMCGXEhdmggZkMI=; b=K1O+fR63Z1oDGKdkNJT2SsE/KzFmk+5iAip8N1JoEX14I0Ub7G0JYvXL2PTCx1dWaG bnR+0R3hAEhuKErO/FC5cYZjfmxeRX3YBuYuthuIhPgQFlYPMnuSp7sSn0df8h5L6jNs fjWunQQXAQ/8+apQ7rP2VUZJ53tPBWG8UkUAD03Kj2iESEnFSPsQbHNniwaJ7XKIt8ni exIV68cvZGkdvgcb8/+BhYife0IBERhDT9t9GRaCmo+Luhlt61gKN2dttHngxrpOsnE7 adxFWgw4XqS9NaH1I2DPl2c7gHhzSvJJIwQYCfrR/mX6QDnps9XhsJNT/3a2IU0N4NYw nN2A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1748597568; x=1749202368; h=content-transfer-encoding:mime-version:references:in-reply-to :message-id:date:subject:cc:to:from:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=7odL+uROHJjULwRoyw6w9QE9qQsYAMCGXEhdmggZkMI=; b=otfstjnDYu5abjNEMLKdLH4zso37PoqZro62ZYSlJOkQ5EnrpTWJUg7gR/CfgQfMaM Ot5O/Hlzm7KmLA2ahq1npVpzDdxHkP2cKalOaHf6O6T0PzuNx33HuqqlW7Bgm8UxIOIU gBU12xbllOOPKTZ5wiDmmbvuvTq+kR9j5AAhkR6iMBVslKoc7gtcGprG1oKCRWCe2p3C eZ12wfp/oKBTry3B2AlTHuowrqES7zPekjBhm5KHPTHxLePxK4Mqg9ZBn1PpJTALuL9B Zr5eCI7tv2A1JtX6qA9D7RSGn5Hokz1oMl0NdNw3joYIFHgBNZRskJoWZn+iGC0ZC72F +zHw== X-Forwarded-Encrypted: i=1; AJvYcCWTHK/H7OvQ5Y95oM9MLlkJZrJPu0EMCd3I2XWoXp+FydGpjcUNSQI3+3+4PyykjY066vuYybzXKQ==@kvack.org X-Gm-Message-State: AOJu0YxnGK0SXpbHJHPENqEIlilUZHzSWmOqr7JLh8iZCyGsVe0kYB3a cKfyIN2K1Pmv5sN4ZDXrELKuRODGFvniiQs1T3YjWtP+ecQWtFhWopkygeB3O7KF4Ss= X-Gm-Gg: ASbGncsGnjLztRCeK+dGjPm2HxfNW3jof8LpLp+BUY0ibF9cS560DXCnFBO0YF0JK2h nwjWsaLfmBFbwrEmNJwJ/Se4pFN2XCTO67UZ4TFwQvN8pKc65C2KOb3TghCzy3l0zwGQcbsZ/eq Otfdr7kqev9pf7nMEgxvjXF3aI+Oy4LY2szRZwbSo6lFJv5bZIyBLpWypOcRiRqd1Lg1pv7MNvc JJikiQ+0fTnAqYz55dfYE6k9jdCY+kiOCwHOuJRoKVA2hgTj/KWyYhk3cTSACR6wqQ/aeGqVxog UA/SqnuFVWFYtGzzwFdcrxx37WEb7FCdZDE5mP9xxEFoaS+x2FqUElWamDn/Qkxb8WaE9COKju5 1H1VDrS43Sg== X-Google-Smtp-Source: AGHT+IE+I3OxPDYMOIkiL8a1fAG5KorJApLQ8jYCZgv/H0F4D9V+Weoec5Tdc5pVnNzU/vxuroLuaA== X-Received: by 2002:a17:90b:3eca:b0:30e:5c74:53c9 with SMTP id 98e67ed59e1d1-31246453833mr3702580a91.11.1748597567790; Fri, 30 May 2025 02:32:47 -0700 (PDT) Received: from FQ627FTG20.bytedance.net ([63.216.146.178]) by smtp.gmail.com with ESMTPSA id 98e67ed59e1d1-3124e29f7b8sm838724a91.2.2025.05.30.02.32.32 (version=TLS1_3 cipher=TLS_CHACHA20_POLY1305_SHA256 bits=256/256); Fri, 30 May 2025 02:32:47 -0700 (PDT) From: Bo Li To: tglx@linutronix.de, mingo@redhat.com, bp@alien8.de, dave.hansen@linux.intel.com, x86@kernel.org, luto@kernel.org, kees@kernel.org, akpm@linux-foundation.org, david@redhat.com, juri.lelli@redhat.com, vincent.guittot@linaro.org, peterz@infradead.org Cc: dietmar.eggemann@arm.com, hpa@zytor.com, acme@kernel.org, namhyung@kernel.org, mark.rutland@arm.com, alexander.shishkin@linux.intel.com, jolsa@kernel.org, irogers@google.com, adrian.hunter@intel.com, kan.liang@linux.intel.com, viro@zeniv.linux.org.uk, brauner@kernel.org, jack@suse.cz, lorenzo.stoakes@oracle.com, Liam.Howlett@oracle.com, vbabka@suse.cz, rppt@kernel.org, surenb@google.com, mhocko@suse.com, rostedt@goodmis.org, bsegall@google.com, mgorman@suse.de, vschneid@redhat.com, jannh@google.com, pfalcato@suse.de, riel@surriel.com, harry.yoo@oracle.com, linux-kernel@vger.kernel.org, linux-perf-users@vger.kernel.org, linux-fsdevel@vger.kernel.org, linux-mm@kvack.org, duanxiongchun@bytedance.com, yinhongbo@bytedance.com, dengliang.1214@bytedance.com, xieyongji@bytedance.com, chaiwen.cc@bytedance.com, songmuchun@bytedance.com, yuanzhu@bytedance.com, chengguozhu@bytedance.com, sunjiadong.lff@bytedance.com, Bo Li Subject: [RFC v2 17/35] RPAL: add a mapping between fsbase and tasks Date: Fri, 30 May 2025 17:27:45 +0800 Message-Id: <964eab3190221c0c880ee9a52957865512c8571c.1748594841.git.libo.gcs85@bytedance.com> X-Mailer: git-send-email 2.39.5 (Apple Git-154) In-Reply-To: References: MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspam-User: X-Stat-Signature: n7h68kqgm1u7rmnuk65tprmpkbzzr1ao X-Rspamd-Queue-Id: 05BB01C0005 X-Rspamd-Server: rspam11 X-HE-Tag: 1748597568-674051 X-HE-Meta: U2FsdGVkX190td8jBhuQNW+jkpVghERrkfgy76qktAF0rqMZZAcZmuYA7QO/SZR7nw8xmkllbe77TGmg2mCN+6RNkMR6uTe2cwKQbff5p2gduY+QoSByN4XX5aTtZNz6TiNJJqAG1NoNRfkW6SZ1IpDmq7RUvTcYslAF/e6wAO+ySGffaw9aAXCoVmkVN8HeQ6b4+z79/0qr6za8+fOlocrD1E7lfCU30CBdHP21jYkNw4ucpLM+YSP8mHsvFJTU9F6Ygl1OrP67xMn3sl7q74eieMqOAwoXVvQ1rDXNgyjlstJZO+k6Usb+ExizyZWhHKt5RXWFXIwC/VkW9VNAP7t59FTYjkKlA1BXN5qVof79+1ELjDJeOqkvLkIIE3RZX1jmlqpCWLAvclijtranAR1MKKMIUSkWZ80gO4fWaQYOpR2zHI7usuO0aTxxbkTxzHmE24FXE/TvYjlbf6M5ekRdxZO2aFQ+GptZbGgBggKKy4eRPoHlmSo6J6uIywayCrE7c0660qptW4gimrP/tZ12UHhH0e+AQmuxauEafx7S5xCkhWH2rTcnlqAmvXVSqv+c2PsktIOGlXcp0WPsFmGdruAB+qVeE04jNq1/o0zZownFEcotXKxWl8QIhSvglbywG6D8xRu3fjEbLiLKa4LhPldB8tZV5EbEbeFqkvx+WeiWFruOxfeKBCfJWDks2yLjBAfSXo+/AS/2tZvJMBV+oJF33CIOcWsuuBFF6oOdJOYnqWa1iUoZfy6cytinutlfrcy2yVeFzfbRNYBwAFWtU+aTMFkILc0Ho71OhDBmUH09LET6Um93UI5tS6LafT7k9Eqb+k3rFPUkA6sTaF3g7yZQ2x00h+gVAq+wCw+xTMh4Nudl7Wl/0t/1JLnRJA4DXiVExELKJDmceGCzjhgdgYPtre/Qi8/Wg4li9pqIuYKWunx2QEw4KUDISF7xrOd8KTc15XaSrOzmbLM uvX6XhVA F0spOMii+gGI5Tz5ZzT8dex1abmAf8Fc66nna4SBkXKbwfHnrWSU1WarBbcZ4pgQOQ71Zym5YB2+7L2SJP/y3yAXrkyt5QDjGCLN7iE+4zkv7pyomMbAMT12ofe0QoPjFe7ByZkrrjY5NW2//7G6t1QhWCl1jP67WP8iiD1mgiOgZzUX3DHvjK/+6wBHsiENn+bw7iRw8zF0PxDOZkwZ4/NmHj5kMI1D8mgdQ5i4ubeylTH1HO0QSiXqOnhFzecuug5a/oE6Tuze+JrqBzeOGZNZAlAfAW61+QJQeIPWV7L05oOO1BmLezHJmZA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: RPAL relies on the value of the fsbase register to determine whether a lazy switch is necessary. Therefore, a mapping between fsbase and tasks must be established. This patch allows a thread to register its fsbase value when it is registered as a receiver. The rpal_find_next_task() interface is used to locate the receiver corresponding to a given fsbase value. Additionally, a new rpal_misidentify() interface has been added to check if the current fsbase value matches the current task. If they do not match, the task corresponding to the fsbase is identified, the RPAL_LAZY_SWITCHED_BIT flag is set, and the current task is recorded. The kernel can later use this flag and the recorded task to backtrack to the task before the lazy switch. Signed-off-by: Bo Li --- arch/x86/rpal/core.c | 85 ++++++++++++++++++++++++++++++++++++++++++ arch/x86/rpal/thread.c | 57 +++++++++++++++++++++++++++- include/linux/rpal.h | 15 ++++++++ 3 files changed, 156 insertions(+), 1 deletion(-) diff --git a/arch/x86/rpal/core.c b/arch/x86/rpal/core.c index c185a453c1b2..19c4ef38bca3 100644 --- a/arch/x86/rpal/core.c +++ b/arch/x86/rpal/core.c @@ -7,6 +7,7 @@ */ #include +#include #include "internal.h" @@ -33,12 +34,96 @@ static inline void rpal_unlock_cpu(struct task_struct *tsk) } } + +static inline struct task_struct *rpal_get_sender_task(void) +{ + struct task_struct *next; + + next = current->rpal_rd->sender; + current->rpal_rd->sender = NULL; + + return next; +} + +/* + * RPAL uses the value of fsbase (which libc uses as the base + * address for thread-local storage) to determine whether a + * lazy switch should be performed. + */ +static inline struct task_struct *rpal_misidentify(void) +{ + struct task_struct *next = NULL; + struct rpal_service *cur = rpal_current_service(); + unsigned long fsbase; + + fsbase = rdfsbase(); + if (unlikely(!rpal_is_correct_address(cur, fsbase))) { + if (rpal_test_current_thread_flag(RPAL_LAZY_SWITCHED_BIT)) { + /* current is receiver, next is sender */ + next = rpal_get_sender_task(); + if (unlikely(next == NULL)) { + rpal_err("cannot find sender task\n"); + goto out; + } + } else { + /* current is sender, next is receiver */ + next = rpal_find_next_task(fsbase); + if (unlikely(next == NULL)) { + rpal_err( + "cannot find receiver task, fsbase: 0x%016lx\n", + fsbase); + goto out; + } + rpal_set_task_thread_flag(next, RPAL_LAZY_SWITCHED_BIT); + next->rpal_rd->sender = current; + } + } +out: + return next; +} + +struct task_struct *rpal_find_next_task(unsigned long fsbase) +{ + struct rpal_service *cur = rpal_current_service(); + struct rpal_service *tgt; + struct task_struct *tsk = NULL; + int i; + + tgt = rpal_get_mapped_service_by_addr(cur, fsbase); + if (unlikely(!tgt)) { + pr_debug("rpal debug: cannot find legal rs, fsbase: 0x%016lx\n", + fsbase); + return NULL; + } + for (i = 0; i < RPAL_MAX_RECEIVER_NUM; ++i) { + if (tgt->fs_tsk_map[i].fsbase == fsbase) { + tsk = tgt->fs_tsk_map[i].tsk; + break; + } + } + rpal_put_service(tgt); + + return tsk; +} + +static bool check_hardware_features(void) +{ + if (!boot_cpu_has(X86_FEATURE_FSGSBASE)) { + rpal_err("no fsgsbase feature\n"); + return false; + } + return true; +} + int __init rpal_init(void) { int ret = 0; rpal_cap = 0; + if (!check_hardware_features()) + goto fail; + ret = rpal_service_init(); if (ret) goto fail; diff --git a/arch/x86/rpal/thread.c b/arch/x86/rpal/thread.c index bc203e9c6e5e..db3b13ff82be 100644 --- a/arch/x86/rpal/thread.c +++ b/arch/x86/rpal/thread.c @@ -7,9 +7,53 @@ */ #include +#include #include "internal.h" +static bool set_fs_tsk_map(void) +{ + struct rpal_service *cur = rpal_current_service(); + struct rpal_fsbase_tsk_map *ftm; + unsigned long fsbase = rdfsbase(); + bool success = false; + int i = 0; + + for (i = 0; i < RPAL_MAX_RECEIVER_NUM; ++i) { + ftm = &cur->fs_tsk_map[i]; + if (ftm->fsbase == 0 && + cmpxchg64(&ftm->fsbase, 0, fsbase) == 0) { + ftm->tsk = current; + success = true; + break; + } + } + + return success; +} + +static bool clear_fs_tsk_map(void) +{ + struct rpal_service *cur = rpal_current_service(); + struct rpal_fsbase_tsk_map *ftm; + unsigned long fsbase = rdfsbase(); + bool success = false; + int i = 0; + + for (i = 0; i < RPAL_MAX_RECEIVER_NUM; ++i) { + ftm = &cur->fs_tsk_map[i]; + if (ftm->fsbase == fsbase) { + ftm->tsk = NULL; + barrier(); + ftm->fsbase = 0; + success = true; + break; + } + } + + return success; +} + static void rpal_common_data_init(struct rpal_common_data *rcd) { rcd->bp_task = current; @@ -54,6 +98,7 @@ int rpal_register_sender(unsigned long addr) rsd->rsp = rsp; rsd->scc = (struct rpal_sender_call_context *)(addr - rsp->user_start + rsp->kernel_start); + rsd->receiver = NULL; current->rpal_sd = rsd; rpal_set_current_thread_flag(RPAL_SENDER_BIT); @@ -122,15 +167,21 @@ int rpal_register_receiver(unsigned long addr) goto put_shared_page; } + if (!set_fs_tsk_map()) { + ret = -EAGAIN; + goto free_rrd; + } + rpal_common_data_init(&rrd->rcd); if (rpal_init_thread_pending(&rrd->rcd)) { ret = -ENOMEM; - goto free_rrd; + goto clear_fs; } rrd->rsp = rsp; rrd->rcc = (struct rpal_receiver_call_context *)(addr - rsp->user_start + rsp->kernel_start); + rrd->sender = NULL; current->rpal_rd = rrd; rpal_set_current_thread_flag(RPAL_RECEIVER_BIT); @@ -139,6 +190,8 @@ int rpal_register_receiver(unsigned long addr) return 0; +clear_fs: + clear_fs_tsk_map(); free_rrd: kfree(rrd); put_shared_page: @@ -158,6 +211,8 @@ int rpal_unregister_receiver(void) goto out; } + clear_fs_tsk_map(); + rpal_put_shared_page(rrd->rsp); rpal_clear_current_thread_flag(RPAL_RECEIVER_BIT); rpal_free_thread_pending(&rrd->rcd); diff --git a/include/linux/rpal.h b/include/linux/rpal.h index 5b115be14a55..45137770fac6 100644 --- a/include/linux/rpal.h +++ b/include/linux/rpal.h @@ -80,6 +80,9 @@ /* No more than 15 services can be requested due to limitation of MPK. */ #define MAX_REQUEST_SERVICE 15 +/* We allow at most 16 receiver thread in one process */ +#define RPAL_MAX_RECEIVER_NUM 16 + enum { RPAL_REQUEST_MAP, RPAL_REVERSE_MAP, @@ -100,6 +103,7 @@ enum rpal_task_flag_bits { RPAL_SENDER_BIT, RPAL_RECEIVER_BIT, RPAL_CPU_LOCKED_BIT, + RPAL_LAZY_SWITCHED_BIT, }; enum rpal_receiver_state { @@ -145,6 +149,11 @@ struct rpal_poll_data { wait_queue_head_t rpal_waitqueue; }; +struct rpal_fsbase_tsk_map { + unsigned long fsbase; + struct task_struct *tsk; +}; + /* * Each RPAL process (a.k.a RPAL service) should have a pointer to * struct rpal_service in all its tasks' task_struct. @@ -202,6 +211,9 @@ struct rpal_service { /* Notify service is released by others */ struct rpal_poll_data rpd; + /* fsbase / pid map */ + struct rpal_fsbase_tsk_map fs_tsk_map[RPAL_MAX_RECEIVER_NUM]; + /* delayed service put work */ struct delayed_work delayed_put_work; @@ -283,12 +295,14 @@ struct rpal_receiver_data { struct rpal_common_data rcd; struct rpal_shared_page *rsp; struct rpal_receiver_call_context *rcc; + struct task_struct *sender; }; struct rpal_sender_data { struct rpal_common_data rcd; struct rpal_shared_page *rsp; struct rpal_sender_call_context *scc; + struct task_struct *receiver; }; enum rpal_command_type { @@ -465,6 +479,7 @@ struct rpal_service *rpal_get_mapped_service_by_id(struct rpal_service *rs, int rpal_rebuild_sender_context_on_fault(struct pt_regs *regs, unsigned long addr, int error_code); struct mm_struct *rpal_pf_get_real_mm(unsigned long address, int *rebuild); +struct task_struct *rpal_find_next_task(unsigned long fsbase); extern void rpal_pick_mmap_base(struct mm_struct *mm, struct rlimit *rlim_stack); -- 2.20.1