From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DB0FC61DA3 for ; Fri, 24 Feb 2023 10:11:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5BA1C6B009F; Fri, 24 Feb 2023 05:11:50 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 569766B00A1; Fri, 24 Feb 2023 05:11:50 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 47E956B00A2; Fri, 24 Feb 2023 05:11:50 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 39B346B009F for ; Fri, 24 Feb 2023 05:11:50 -0500 (EST) Received: from smtpin12.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 08CFF160220 for ; Fri, 24 Feb 2023 10:11:50 +0000 (UTC) X-FDA: 80501769180.12.7BBC98A Received: from mail3-166.sinamail.sina.com.cn (mail3-166.sinamail.sina.com.cn [202.108.3.166]) by imf01.hostedemail.com (Postfix) with ESMTP id 9484940019 for ; Fri, 24 Feb 2023 10:11:45 +0000 (UTC) Authentication-Results: imf01.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf01.hostedemail.com: domain of hdanton@sina.com designates 202.108.3.166 as permitted sender) smtp.mailfrom=hdanton@sina.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1677233508; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=s2M5GV9SZwKmFGKc1wMq4zKa2miesZf5Yrw08MZoXjU=; b=OqWYx7DKwYUVos5tCtW8DKuqjNgXq1CGmmFhqOxRmkrgrIQYxRwFXM331j09Byf/iT58Dy AiWhZNvx0fjk8jEhBFdwdai42Dz+gbtfxebKw4D/tj+kto8ndgQTJqL8aFBxDZt8GnYsu7 9FqhYR8zwSnsuWI1a9JGtGLj2tdfDu0= ARC-Authentication-Results: i=1; imf01.hostedemail.com; dkim=none; dmarc=none; spf=pass (imf01.hostedemail.com: domain of hdanton@sina.com designates 202.108.3.166 as permitted sender) smtp.mailfrom=hdanton@sina.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1677233508; a=rsa-sha256; cv=none; b=pF2rJaRhOgeDojRDsWll7DrS9YhBQwwVibcqYHm9BYPkGip+gqyqbQI6xurApZocHnAIxM eWZRonh/ypG3SvILkpdVv6cNdDt7M46hNX0mh7ljDLktTAJlR2uymr1eFCMfV4L9FwAThd xyGDLvxrkQITKOyLylqmrtnUXyyIQE0= Received: from unknown (HELO localhost.localdomain)([114.249.61.130]) by sina.com (172.16.97.23) with ESMTP id 63F88D540002004D; Fri, 24 Feb 2023 18:11:34 +0800 (CST) X-Sender: hdanton@sina.com X-Auth-ID: hdanton@sina.com X-SMAIL-MID: 21304654919424 From: Hillf Danton To: Peter Zijlstra Cc: longman@redhat.com, mingo@redhat.com, will@kernel.org, linux-kernel@vger.kernel.org, linux-mm@kvack.org, boqun.feng@gmail.com Subject: Re: [PATCH 3/6] locking/rwsem: Rework writer wakeup Date: Fri, 24 Feb 2023 18:11:29 +0800 Message-Id: <20230224101129.3020-1-hdanton@sina.com> In-Reply-To: <20230223123319.487908155@infradead.org> References: <20230223122642.491637862@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: 9484940019 X-Rspam-User: X-Stat-Signature: owm854bomr8jggd9exprm9mym9xqfgfn X-HE-Tag: 1677233505-349231 X-HE-Meta: U2FsdGVkX1+oluPM8Y0vZT6DTfM0LdY1UiwsX5juAA9mVHiw5AwOjiWQG+wx2hDhV4KOcsPXFDOoXf7Gjw82UoMsUu99afPs1YLxWNXbRNmn9YdAHWcaYQssdnarAJfXbqlP6p36odPR0Q9PRiT6GalI3nO5KJStt6lgSeGHk9j55LkXxpGi47XO4izr+Xvcq8tOT4v6PElEnwq8oDHlZxGEQekzY5i3zE+uNWHUeU58EiVols2s8FvaXEWa8/2F9wbqc9VTtSSsodE45fIEmffJMp1vhL0AkeIIDT5mAHFIkBnSeOMQ387goqzZhfnz9sxpuVKJLT2fNT9LLGiuKb+4k5YS8MD39MbSsdschcRv+qGTo3fs5DtEUdVrqqDNFXLYf4d3njhMxNz1KtwadvZFvKDkq5kggLmDj3PXYwqwib3+6wrmiRWQS93ob9JFYzYGEkN9gb7i8dRWbkvKgvLGFqX9QEi6SEIqKup915JfwyEBrMNsYts2ItaE3vgWr9xjzROcvuU6H7qDXh212dON1zhmT7fkiOyHHwmhKZa2gePTXtl4g8t6I09BV8nNeh60zgwyZX+nNmNiVpMXcf9OKMUxXSnO2DoyET+br6dg4GLeGuJWnfmh9rSZHMSvX8YXKOO0qtg0ooImPRoYCQEVdnaEX1XYgMGKIh3fLpUm/zzwCjemhMLYGvJZPjC35MxVePl+Sm5VGUoxCnJ2D07w6UDWjRLXfHBgRgVK5qpK7R52GEo1f69w/K41mfYy9PBavMaqDHNF2N872fV5vF5RS7bOxtXaWUPjpyGhCsTTTJJK2DFVXVZseYCW8UdemKuZwWpdEpZzhIxmRypZcG3xrkWf013RyN7uIg91bLXe0duyPTrdgBkDVgvQA9mTNj/43LUNd4gMKACcqoaXfdFo3yylkcM75R2IHgiivINmnhn+3HosnDUNJL4toq6e1Vj5wAVpCNW9i1aCsxk 59Z6g+UU YAAKrUymZ9AgZIfix9rRdEkIWjt7EH++d5kfiXqxqpMr9auCDCB2wKv+kileNajU+Fo+IDTfGEuifiDe1Y+sqDpp4fCo0gmKvpLBMLc8hJyII20Kq13GbpdKvrn6n+DyHC6/LwUBSSw9mGu/JplVd/5uH6NEq2L1huF1e8Mh2SdtyYNvjSt7/3OfsmNfzWOggTSEtfYXkkGBNesusbnjsA5lpuSHfUTaHK/Uj X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, 23 Feb 2023 13:26:45 +0100 Peter Zijlstra > +static void rwsem_writer_wake(struct rw_semaphore *sem, > + struct rwsem_waiter *waiter, > + struct wake_q_head *wake_q) > +{ > + struct rwsem_waiter *first = rwsem_first_waiter(sem); > + long count, new; > + > + lockdep_assert_held(&sem->wait_lock); > + > + count = atomic_long_read(&sem->count); > + do { > + bool has_handoff = !!(count & RWSEM_FLAG_HANDOFF); > + > + if (has_handoff) { > + /* > + * Honor handoff bit and yield only when the first > + * waiter is the one that set it. Otherwisee, we > + * still try to acquire the rwsem. > + */ > + if (first->handoff_set && (waiter != first)) > + return; Given that HANDOFF disables all spinning and stealing in 2/6, what sense made by still trying to acquire the rwsem? The answer is perhaps it is called at wake time. > + } > + > + new = count; > + > + if (count & RWSEM_LOCK_MASK) { If it is only called at wake time, the chance for a transient RWSEM_READER_BIAS to ruin the wakeup is not zero. > + /* > + * A waiter (first or not) can set the handoff bit > + * if it is an RT task or wait in the wait queue > + * for too long. > + */ > + if (has_handoff || (!rt_task(waiter->task) && > + !time_after(jiffies, waiter->timeout))) > + return; > + > + new |= RWSEM_FLAG_HANDOFF; > + } else { > + new |= RWSEM_WRITER_LOCKED; > + new &= ~RWSEM_FLAG_HANDOFF; > + > + if (list_is_singular(&sem->wait_list)) > + new &= ~RWSEM_FLAG_WAITERS; > + } > + } while (!atomic_long_try_cmpxchg_acquire(&sem->count, &count, new)); > + > + /* > + * We have either acquired the lock with handoff bit cleared or set > + * the handoff bit. Only the first waiter can have its handoff_set > + * set here to enable optimistic spinning in slowpath loop. > + */ > + if (new & RWSEM_FLAG_HANDOFF) { > + first->handoff_set = true; > + lockevent_inc(rwsem_wlock_handoff); > + return; > + } > + > + /* > + * Have rwsem_writer_wake() fully imply rwsem_del_waiter() on > + * success. > + */ > + list_del(&waiter->list); > + rwsem_set_owner(sem); > + rwsem_waiter_wake(waiter, wake_q); > +} > + > /* > * handle the lock release when processes blocked on it that can now run > * - if we come here from up_xxxx(), then the RWSEM_FLAG_WAITERS bit must > @@ -424,23 +518,12 @@ static void rwsem_mark_wake(struct rw_se > */ > waiter = rwsem_first_waiter(sem); > > - if (waiter->type != RWSEM_WAITING_FOR_WRITE) > - goto wake_readers; > - > - if (wake_type == RWSEM_WAKE_ANY) { > - /* > - * Mark writer at the front of the queue for wakeup. > - * Until the task is actually later awoken later by > - * the caller, other writers are able to steal it. > - * Readers, on the other hand, will block as they > - * will notice the queued writer. > - */ > - wake_q_add(wake_q, waiter->task); > - lockevent_inc(rwsem_wake_writer); > + if (waiter->type == RWSEM_WAITING_FOR_WRITE) { > + if (wake_type == RWSEM_WAKE_ANY) > + rwsem_writer_wake(sem, waiter, wake_q); > + return;