From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2F8CAC3DA64 for ; Tue, 6 Aug 2024 05:08:44 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8F0FD6B007B; Tue, 6 Aug 2024 01:08:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 8A1676B0082; Tue, 6 Aug 2024 01:08:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 78F826B0083; Tue, 6 Aug 2024 01:08:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 576C46B007B for ; Tue, 6 Aug 2024 01:08:43 -0400 (EDT) Received: from smtpin03.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id BED70A04DB for ; Tue, 6 Aug 2024 05:08:42 +0000 (UTC) X-FDA: 82420640484.03.42ECBC7 Received: from mail-qt1-f176.google.com (mail-qt1-f176.google.com [209.85.160.176]) by imf16.hostedemail.com (Postfix) with ESMTP id EA085180017 for ; Tue, 6 Aug 2024 05:08:40 +0000 (UTC) Authentication-Results: imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=MZyOct3R; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of yuzhao@google.com designates 209.85.160.176 as permitted sender) smtp.mailfrom=yuzhao@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1722920851; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Jjc5PvFQAIDEJuasUzjAKu0+WbtJ/LimH4KncpiNJ2k=; b=m0qUvfA69J0PhA5sYHDfhbSwu5EvRTcAL+y/s89iTKq5PY8nVlvzj0PQH3Ine8VRcmZhj9 lBiA4bfXkgJGPbWrVPSJArOUGnjhNaS9iOfz6cp2sFmWt/d/aWVcYgBqImSW2DNtu6tQR2 y/zVfbdXpM/riqV1/vQVKIV8y8wnJ9o= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1722920851; a=rsa-sha256; cv=none; b=At1Dvnka8VdNsB7Uj5lh3JIqF+stWNIFUdyc7b/0ikfHOHXQKVmyCgoA0+bOEsMZSwJ88K evlxQVO8OpyQ5i3agFDcB2lLIZKczb89XvP+m+zG0k/pwgMN5BKc3aC+gzFtFLEjWrBv1i DHdWpVS0cj5tszjK5kM4C6N0txS5lcc= ARC-Authentication-Results: i=1; imf16.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=MZyOct3R; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf16.hostedemail.com: domain of yuzhao@google.com designates 209.85.160.176 as permitted sender) smtp.mailfrom=yuzhao@google.com Received: by mail-qt1-f176.google.com with SMTP id d75a77b69052e-44fee2bfd28so147941cf.1 for ; Mon, 05 Aug 2024 22:08:40 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1722920920; x=1723525720; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=Jjc5PvFQAIDEJuasUzjAKu0+WbtJ/LimH4KncpiNJ2k=; b=MZyOct3RtxhUDMUfHZVYrsDxTNy+zhjKKmv974zKmFRLBaUos8AnK7FeGjPndO97xP vQ+Qckbxl3qmrODPtB2s7q9GGoSRlN6/VjYApnrjJVUw25za3/qlBkLjl6m6gkLkZTEh nEynoEJAcH+hubClS721W9jaJ1q25lGQUxS2Q1Vru3x1PjOKcEaUKgJr1ACcQDzAVCf0 mjGSRme8ga54LgL/Wph0a4QJhMfY3o423yJaPf4ws8HhxK/yuFogykg2IKk0vk1/5oO6 gH8LE8rT79lhXgxVtjmOwHXdYcmIKQRqZf5vwmnAftN0zJOEeK+9CEJKlJ4DWVNVf4Vd qA1Q== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1722920920; x=1723525720; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=Jjc5PvFQAIDEJuasUzjAKu0+WbtJ/LimH4KncpiNJ2k=; b=pxJ7ydgAjrQotVMPoSS1r858w1I0X/5+azLLnEwkUM9UZkxoKLGOOe70Or4REI0kXD bhe4ElCxOhTqpffZuJzsIRd37uDGVKIuIWaN6gAn5Qb17C4OihYtG5O5i/vun3oDpYfz W5o/YUhCb4GbsTXsxJ2IM0OspYlVOqRvU+wD4nsDCWd/mhQzxmc7QxBbnmjtuZr/xCJ2 Igqhwz/FaCw23d5TYjBpV8iIjvzth4xHk0S334+1LoFyRmRnScpA5oIjz2fm/9iwJf6E N+WQaUlfITStURIGpalS8QbnJHsxdJbg7Sk+5CgaKzzwT5S70ea2B+CsgT/UrtBFo1OP 8Izg== X-Forwarded-Encrypted: i=1; AJvYcCXWXQ1JkXlgQbvql6GhGLklSZatwllOrRiL3omX3s69WtiHjkQGNpVL0BR9QZScMD9tK444SFNuqnqEihuwIVVSBY0= X-Gm-Message-State: AOJu0YxTbiLD/VeeZE8HyLgmipbRAnbEEItoXu6++j2ORt/+OmZvNQWV vwAzGnPHolBLrZ1vNpbZG8jxUsATzSQZdU9jxMzmZUwutgcnY/eYJt7y65FVQArttbOmYzJ0MZ6 m/kjs9VRLC/Mji0L5FysF4oiYtKT7lNHns+Og X-Google-Smtp-Source: AGHT+IGe+hbnq8FqqpBq6h2fG7LKEjlEXbgQq7QkXIaGvlbHxBcjruOiSCjcQRxF74y9Lk6vbB+gLm+PKJ0ZasuFnCc= X-Received: by 2002:ac8:7fd3:0:b0:447:d7ff:961d with SMTP id d75a77b69052e-451bd1ea570mr1055671cf.9.1722920919750; Mon, 05 Aug 2024 22:08:39 -0700 (PDT) MIME-Version: 1.0 References: <20240806022114.3320543-1-yuzhao@google.com> <20240806022114.3320543-4-yuzhao@google.com> In-Reply-To: <20240806022114.3320543-4-yuzhao@google.com> From: Yu Zhao Date: Mon, 5 Aug 2024 23:08:01 -0600 Message-ID: Subject: Re: [RFC PATCH 3/4] arm64: pause remote CPUs to update vmemmap To: Catalin Marinas , Will Deacon Cc: Andrew Morton , David Rientjes , Douglas Anderson , Frank van der Linden , Mark Rutland , Muchun Song , Nanyong Sun , Yang Shi , linux-arm-kernel@lists.infradead.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: EA085180017 X-Stat-Signature: zzmktrsdwi6ucqbem5aiz8k5q88ihu76 X-Rspam-User: X-HE-Tag: 1722920920-639099 X-HE-Meta: U2FsdGVkX1+YlNDl853sdIatw9cfCRFah5pcHX78MFl0NVCJt6xd1YfgtM7LU/k1exoftLt5wp6CBqio8oq+h/BN5PvgCwafoMkuTNM+La3mbVbQGZ/vBJA7g4w0jMRQ9zoOvbqIdHTGLL0be5/+q355GPv574DHC3Ci4chs+NJSWHEJAYXFuH5GWSJk0SxSKa6ONPgZryO4J5dVwcPyiG+YY8B/9Eo/V5qIKrLRg7oDaGxkvo3yZyyjolnqNd3oXiSGnNcpQp+bJL6PoIn+OXHOHhuoOrqDi7MepNh6rw4QwQKa+jHKJzfJoXKzgZV/xOpFeSqCJXmArdx47o9H/fEoN59kMLjeqxlxZA8uC5VbC4Fhzx1g9XhDdfz5d5y/4NGIDZFojeIBEvJLgqgmzM2EvdixmD+vAaskTibQtWdV5ZrkP1ygu5WMvVK4AwI1KOS1VNJ9oclm6CwUowuWHRmneU8emOYpLsYcqZd/xYuE0ToRILFayseca1yJkW6xg/olotY8vwTTRc4RFs0K4H4xdEUmtxuwT7Z6/uNlNpGknmQNqmwMtTkDdLhFNmPz/zvKAYJ3BsxANnJwB60nEiiqWYzF6Ep6RPJ0Odx2aHtfhc4osjifHmaCx3spP7s0VZcQ/5R4HAuS1ULS1/JNAXw7c6EPBsDJlIpvi7z9ntq/uBKQccb87n2dCj5hx7PdFMqwsxZMY0OlL7BRuk3Q79OOSTh8xA11fUGn6o0N+bWqoZd8kARDuvfPxsu46qbCrj8mWI+Cpg84wvA+KNFLMgGeWErXqA2+dn7RRKtSSQr0zMWvvJUXyjoi1PfDGp9Yl67rhUSENNL7/tZasV4/0it4SHgkXharQUT2hKvmnz449po4Ck94EY1iYRNq7D6FkRgt9icYe2zW6dpA/Ib8Vup7slkr9ARfxlYnE1L858qsfiT4GrzFK4LKEkhisNkCA52WvvB2pOOEnhFXNF3 ETIWxdBn a8pIt0y4FgbURHHoSys03ywA1l2Nm2SUWUjnGwmRSQ7tRtRP2TT2ZG/pbSqE/0ZXTm7EV+M4f7ffjCFThOv6MBYikzNwxfzNkdf4bV8BFigEf3ThjKhYXiJgsMhBocRUb+DqybUqg9YX00WrcS6jn7gOdFPYQjAgWPLAa5LqqQm7VO8B49EAFQdqMTamkQsld1IcD X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Mon, Aug 5, 2024 at 8:21=E2=80=AFPM Yu Zhao wrote: > > Pause remote CPUs so that the local CPU can follow the proper BBM > sequence to safely update the vmemmap mapping `struct page` areas. > > While updating the vmemmap, it is guaranteed that neither the local > CPU nor the remote ones will access the `struct page` area being > updated, and therefore they will not trigger kernel PFs. > > Signed-off-by: Yu Zhao > --- > arch/arm64/include/asm/pgalloc.h | 55 ++++++++++++++++++++++++++++++++ > mm/hugetlb_vmemmap.c | 14 ++++++++ > 2 files changed, 69 insertions(+) > > diff --git a/arch/arm64/include/asm/pgalloc.h b/arch/arm64/include/asm/pg= alloc.h > index 8ff5f2a2579e..1af1aa34a351 100644 > --- a/arch/arm64/include/asm/pgalloc.h > +++ b/arch/arm64/include/asm/pgalloc.h > @@ -12,6 +12,7 @@ > #include > #include > #include > +#include > > #define __HAVE_ARCH_PGD_FREE > #define __HAVE_ARCH_PUD_FREE > @@ -137,4 +138,58 @@ pmd_populate(struct mm_struct *mm, pmd_t *pmdp, pgta= ble_t ptep) > __pmd_populate(pmdp, page_to_phys(ptep), PMD_TYPE_TABLE | PMD_TAB= LE_PXN); > } > > +#ifdef CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP > + > +#define vmemmap_update_lock vmemmap_update_lock > +static inline void vmemmap_update_lock(void) > +{ > + cpus_read_lock(); > +} > + > +#define vmemmap_update_unlock vmemmap_update_unlock > +static inline void vmemmap_update_unlock(void) > +{ > + cpus_read_unlock(); > +} > + > +#define vmemmap_update_pte vmemmap_update_pte > +static inline void vmemmap_update_pte(unsigned long addr, pte_t *ptep, p= te_t pte) > +{ > + preempt_disable(); > + pause_remote_cpus(); > + > + pte_clear(&init_mm, addr, ptep); > + flush_tlb_kernel_range(addr, addr + PAGE_SIZE); > + set_pte_at(&init_mm, addr, ptep, pte); > + > + resume_remote_cpus(); > + preempt_enable(); > +} Note that I kept this API from Nanyong for the sake of discussion. What I actually plan to test in our production is: #define vmemmap_update_pte_range_start vmemmap_update_pte_range_start static inline void vmemmap_update_pte_range_start(pte_t *pte, unsigned long start, unsigned long end) { preempt_disable(); pause_remote_cpus(); for (; start !=3D end; start +=3D PAGE_SIZE, pte++) pte_clear(&init_mm, start, pte); flush_tlb_kernel_range(start, end); } #define vmemmap_update_pte_range_end vmemmap_update_pte_range_end static inline void vmemmap_update_pte_range_end(void) { resume_remote_cpus(); preempt_enable(); } > +#define vmemmap_update_pmd vmemmap_update_pmd > +static inline void vmemmap_update_pmd(unsigned long addr, pmd_t *pmdp, p= te_t *ptep) > +{ > + preempt_disable(); > + pause_remote_cpus(); > + > + pmd_clear(pmdp); > + flush_tlb_kernel_range(addr, addr + PMD_SIZE); > + pmd_populate_kernel(&init_mm, pmdp, ptep); > + > + resume_remote_cpus(); > + preempt_enable(); > +} > + > +#define vmemmap_flush_tlb_all vmemmap_flush_tlb_all > +static inline void vmemmap_flush_tlb_all(void) > +{ > +} > + > +#define vmemmap_flush_tlb_range vmemmap_flush_tlb_range > +static inline void vmemmap_flush_tlb_range(unsigned long start, unsigned= long end) > +{ > +} > + > +#endif /* CONFIG_HUGETLB_PAGE_OPTIMIZE_VMEMMAP */ > + > #endif > diff --git a/mm/hugetlb_vmemmap.c b/mm/hugetlb_vmemmap.c > index 2dd92e58f304..893c73493d9c 100644 > --- a/mm/hugetlb_vmemmap.c > +++ b/mm/hugetlb_vmemmap.c > @@ -46,6 +46,18 @@ struct vmemmap_remap_walk { > unsigned long flags; > }; > > +#ifndef vmemmap_update_lock > +static void vmemmap_update_lock(void) > +{ > +} > +#endif > + > +#ifndef vmemmap_update_unlock > +static void vmemmap_update_unlock(void) > +{ > +} > +#endif > + > #ifndef vmemmap_update_pmd > static inline void vmemmap_update_pmd(unsigned long addr, > pmd_t *pmdp, pte_t *ptep) > @@ -194,10 +206,12 @@ static int vmemmap_remap_range(unsigned long start,= unsigned long end, > > VM_BUG_ON(!PAGE_ALIGNED(start | end)); > > + vmemmap_update_lock(); > mmap_read_lock(&init_mm); > ret =3D walk_page_range_novma(&init_mm, start, end, &vmemmap_rema= p_ops, > NULL, walk); > mmap_read_unlock(&init_mm); > + vmemmap_update_unlock(); > if (ret) > return ret; > > -- > 2.46.0.rc2.264.g509ed76dc8-goog >