From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 09FBCEB64D9 for ; Thu, 6 Jul 2023 11:31:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 5BE9D8D0002; Thu, 6 Jul 2023 07:31:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 594DD8D0001; Thu, 6 Jul 2023 07:31:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 436428D0002; Thu, 6 Jul 2023 07:31:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 339588D0001 for ; Thu, 6 Jul 2023 07:31:26 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id D970DC058E for ; Thu, 6 Jul 2023 11:31:25 +0000 (UTC) X-FDA: 80980971330.27.349B455 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf29.hostedemail.com (Postfix) with ESMTP id B3916120005 for ; Thu, 6 Jul 2023 11:31:22 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QMlKVccu; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf29.hostedemail.com: domain of vschneid@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1688643083; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=YUOZEoZh3tniSwu0Lc1VH/leg3NuusrIfXtvdOdkp+U=; b=RDb07WcN2bHprS9xxrmMioU80v8aD67CHwSU2yb849WRnG78ZMdM8XFr4xTmbt4h0oLAc6 izXEMiAWVWRtiZN1sSJEpNlFMCQ8X/BBSpSJhYFHBfBuibLLLhKyM7eyoO2aFJef4K5pLo BXdLr4/nXCcDKhGOB0S/qwt5oXIdJW0= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=QMlKVccu; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf29.hostedemail.com: domain of vschneid@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=vschneid@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1688643083; a=rsa-sha256; cv=none; b=A0fz9Iqmfmvb+3ooX3q9xQhFPhwKIgOasXqoQ7d71UgrOn8aqIN1SK8jrjgBVEqil/RV3o BkWReIfzvmuOxwndAiUl/lkU7xLD+Hu7T6ub7euUYPnFxOu7PGlcefQJcZc/2AJMA48uWg XudGEiTAfVzz/WFfaBYbr8lx2h59oXw= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1688643082; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=YUOZEoZh3tniSwu0Lc1VH/leg3NuusrIfXtvdOdkp+U=; b=QMlKVccuabu4t7QebzfosLOGm6Ws1XRPSSlmxZ3lRWxJNuZd+YkMu5aNGSnQhmtalhs/At f91SDwrFh4cK2DTUklmtzYbnVekDNN9R/xu0jDMDpcomIOC6L/ra9mAs35pUIOceRAlXqA VchNzW4sv+do2xdkWZgX/stRrzT3iDo= Received: from mail-qv1-f70.google.com (mail-qv1-f70.google.com [209.85.219.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-171-kER_2FzJPXKllQrWbYqOfw-1; Thu, 06 Jul 2023 07:31:19 -0400 X-MC-Unique: kER_2FzJPXKllQrWbYqOfw-1 Received: by mail-qv1-f70.google.com with SMTP id 6a1803df08f44-635e91cba88so8491296d6.0 for ; Thu, 06 Jul 2023 04:31:19 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1688643079; x=1691235079; h=mime-version:message-id:date:references:in-reply-to:subject:cc:to :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=YUOZEoZh3tniSwu0Lc1VH/leg3NuusrIfXtvdOdkp+U=; b=jZhQ3P7GXjWE/TyL9CN9sxPOB1tsJRkEYvQuK7W+gdlYfFW8B7GuMIV3FCbtNZLPlC EqKXcpyXcZHnBOhkEiF37ZH42HlwymtSwCAXmUIh2TJg8CmH2qtxEtYlQpsJHm0t0+yp bkzQdAZrh3ul8O216AQ6mVF0tAee6f2UJWPdsm1ewq/ft9nlvIOuhfHVFzs2OMbkuNgX rHJKlYX/LLOx5VmNrDWSuQZeaGZ39R0vFcAeyipetMAf+37cmG+FnnvXF/ZtnysBwZw8 URI1VF0HbqLt91oFay4CkUkxwhMGfLF45H9ex0GVmsQd0j5dq/GKe0uGMgrZaAFHDc7i ZVww== X-Gm-Message-State: ABy/qLZir1rp4c7LPsD+2pwFerJ2jVFE8zBD6/Fmt8qIs1gpqqkH5hQO 731CHNUw/bK3sOXObewqNHfH8uOkkBnbRD15/qAq3Wus0v8+exLX1F+ixSrZA+g3QEVKALx3Cnb lQyu24pV+Nt8= X-Received: by 2002:a0c:f014:0:b0:636:e56c:eedb with SMTP id z20-20020a0cf014000000b00636e56ceedbmr1572748qvk.34.1688643079063; Thu, 06 Jul 2023 04:31:19 -0700 (PDT) X-Google-Smtp-Source: APBJJlEBNT72yhfIkKfRmJowXJfm9QDc9/aizAtdc3cEmmOmAM0nOogaJ0lzClntXieI3e1t66iJTw== X-Received: by 2002:a0c:f014:0:b0:636:e56c:eedb with SMTP id z20-20020a0cf014000000b00636e56ceedbmr1572731qvk.34.1688643078801; Thu, 06 Jul 2023 04:31:18 -0700 (PDT) Received: from vschneid.remote.csb ([154.57.232.159]) by smtp.gmail.com with ESMTPSA id oo23-20020a05620a531700b007673f8803c3sm624918qkn.96.2023.07.06.04.31.12 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Thu, 06 Jul 2023 04:31:18 -0700 (PDT) From: Valentin Schneider To: Peter Zijlstra Cc: linux-kernel@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-doc@vger.kernel.org, kvm@vger.kernel.org, linux-mm@kvack.org, bpf@vger.kernel.org, x86@kernel.org, Nicolas Saenz Julienne , Steven Rostedt , Masami Hiramatsu , Jonathan Corbet , Thomas Gleixner , Ingo Molnar , Borislav Petkov , Dave Hansen , "H. Peter Anvin" , Paolo Bonzini , Wanpeng Li , Vitaly Kuznetsov , Andy Lutomirski , Frederic Weisbecker , "Paul E. McKenney" , Andrew Morton , Uladzislau Rezki , Christoph Hellwig , Lorenzo Stoakes , Josh Poimboeuf , Kees Cook , Sami Tolvanen , Ard Biesheuvel , Nicholas Piggin , Juerg Haefliger , Nicolas Saenz Julienne , "Kirill A. Shutemov" , Nadav Amit , Dan Carpenter , Chuang Wang , Yang Jihong , Petr Mladek , "Jason A. Donenfeld" , Song Liu , Julian Pidancet , Tom Lendacky , Dionna Glaze , Thomas =?utf-8?Q?Wei=C3=9Fschuh?= , Juri Lelli , Daniel Bristot de Oliveira , Marcelo Tosatti , Yair Podemsky Subject: Re: [RFC PATCH 11/14] context-tracking: Introduce work deferral infrastructure In-Reply-To: <20230705223956.GD2813335@hirez.programming.kicks-ass.net> References: <20230705181256.3539027-1-vschneid@redhat.com> <20230705181256.3539027-12-vschneid@redhat.com> <20230705223956.GD2813335@hirez.programming.kicks-ass.net> Date: Thu, 06 Jul 2023 12:31:11 +0100 Message-ID: MIME-Version: 1.0 X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Type: text/plain X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: B3916120005 X-Stat-Signature: gn45w4figb4uaymp7h3paq3cxok7fzd9 X-Rspam-User: X-HE-Tag: 1688643082-854477 X-HE-Meta: U2FsdGVkX1+U0oshLbhspZswLiNskd4klRZqWmvqZi2dER3U3qVfmPWZUnhOEv0I44Oi185lG0SiyT6VCEnIo9+JqnD37ps7/xkjq/KeF0jaa6K+NdwL1AZXXge0Tzqra5jqq+cVDehMjut7o1b7HFjpPsjIfo81zPCwyegKW4wppwpWumgNmh4TG40LANLZA7hKeAOhdkV5ALWd4dSjnP9eygXvT9TH8/oidnONy3qGSIrJSCPtFu+rb5b/w+yba5RATmvGFAygaae2OjDdu62RFf1B70o9+bF24ZhI0Ib1fvvMXnOC7tIxNuJRgQzkJJs1pvXMnp/HouguB33jvsS3DaP5m3J3qMwZp1/5R/UYTJLKefTW3UFanM7A1PGREgkfKyw8yN920+/Ms4dhWD/wc+ePBmGLcCzMfBaBH6llGNfL+ewFSnkV+9Jix+z7rR2spJuRMnotk71vdOg+6fwN+KFwpIb4d9kGrzcKz49RHFlwHSUCZxOIMTlmGCXRajyaRAyp+tB+Rwg1K8x3cyExdDarpzuYkuhdCXPrSwtLZ3YbsAl43qskDT/epjo/sAVAgR/C08NqXKBzI7viACn0L3g7r10eL7ecXoV64MDAOXTx7l5nwTB/3M1jxwdJgA5OGTb4fB1TPNq0idhxlKPScIn1Bctu6nMMQvJFQtvzpEG+2MHgsp4r96GCV/xWRPf+/pRnvIQ1hLRm0HgioTcockGcEfYQTNnEg7Cc5n8rL4NV//TrwXxWQmOY2bj4XZ+PL38AeFrXUjyRc44NiCHS2d4dylqLgT/hYP882iEUbXnkUuwAIIZa7WiSuzdB+opIXnyLtiCfOcdwXIp9TuFRtg0fJeSHLYsR2fezi6o7nt4Nbd4Yl9m01+1q1hC630xF/pm80yJRWF7TSKXTjj/verCY6qXeObo86fkhlrhYpt6r+03UjJgB7nz8zKNGseRPBGUA+O5H/mj3YNE VILgyzlZ 6eKm+ZO8z4aNi28k+ug8V4kJeAZyiCVBxycWGUoxt6etZndhX/7f4llM7xRsASktS0VGetMldpw9iO2jW1q/t4Wph7Ar8yxmSs0TFy8Ml2DR4zaK1Dft9R1Y+wpAUmAHUkuR7lI9Wotp8h8ZE1u046e7d/VpAM8xZB3EdRsgIuv7cyFfT1VQYu4E1GYoFt3e3XoUsMtZdWuSaPTv/JzIXMdnU6wfUqiRSlDW6KBTDRRS+543rVPpitXss3cjwqAjpvGMJ6xrs6+e5gg2ytsuPo7bKlg== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 06/07/23 00:39, Peter Zijlstra wrote: > On Wed, Jul 05, 2023 at 07:12:53PM +0100, Valentin Schneider wrote: > >> Note: A previous approach by PeterZ [1] used an extra bit in >> context_tracking.state to flag the presence of deferred callbacks to >> execute, and the actual callbacks were stored in a separate atomic >> variable. >> >> This meant that the atomic read of context_tracking.state was sufficient to >> determine whether there are any deferred callbacks to execute. >> Unfortunately, it presents a race window. Consider the work setting >> function as: >> >> preempt_disable(); >> seq = atomic_read(&ct->seq); >> if (__context_tracking_seq_in_user(seq)) { >> /* ctrl-dep */ >> atomic_or(work, &ct->work); >> ret = atomic_try_cmpxchg(&ct->seq, &seq, seq|CT_SEQ_WORK); >> } >> preempt_enable(); >> >> return ret; >> >> Then the following can happen: >> >> CPUx CPUy >> CT_SEQ_WORK \in context_tracking.state >> atomic_or(WORK_N, &ct->work); >> ct_kernel_enter() >> ct_state_inc(); >> atomic_try_cmpxchg(&ct->seq, &seq, seq|CT_SEQ_WORK); >> >> The cmpxchg() would fail, ultimately causing an IPI for WORK_N to be >> sent. Unfortunately, the work bit would remain set, and it can't be sanely >> cleared in case another CPU set it concurrently - this would ultimately >> lead to a double execution of the callback, one as a deferred callback and >> one in the IPI. As not all IPI callbacks are idempotent, this is >> undesirable. > > So adding another atomic is arguably worse. > > The thing is, if the NOHZ_FULL CPU is actually doing context transitions > (SYSCALLs etc..) then everything is fundamentally racy, there is no > winning that game, we could find the remote CPU is in-kernel, send an > IPI, the remote CPU does return-to-user and receives the IPI. > > And then the USER is upset... because he got an IPI. > Yeah, that part is inevitably racy. The thing I was especially worried about was the potential double executions (once in IPI, again in deferred work). It's not /too/ bad as the only two deferred callbacks I'm introducing here are costly-but-stateless, but IMO is a bad foundation. But it seems like we can reuse the existing atomic and squeeze some bits in there, so let's see how that goes :-)