From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 63491E77188 for ; Wed, 8 Jan 2025 05:06:46 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id EA8416B0096; Wed, 8 Jan 2025 00:06:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E587C6B0098; Wed, 8 Jan 2025 00:06:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id CF9416B0099; Wed, 8 Jan 2025 00:06:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id ACBA46B0096 for ; Wed, 8 Jan 2025 00:06:45 -0500 (EST) Received: from smtpin22.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 5E782A0F4B for ; Wed, 8 Jan 2025 05:06:45 +0000 (UTC) X-FDA: 82983099570.22.2AD0ED3 Received: from mail-ua1-f47.google.com (mail-ua1-f47.google.com [209.85.222.47]) by imf29.hostedemail.com (Postfix) with ESMTP id 8179B120005 for ; Wed, 8 Jan 2025 05:06:43 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=FsVGvXwc; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf29.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.222.47 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1736312803; a=rsa-sha256; cv=none; b=eSQ8TiSetFaCnqI2CNVRxCetQb/3rHDXQySfNxm/si0fwJV6rfcpcs/2/Dde8l+AEgAWzE Gjv3M2tCm73beeQTK618ogYwq55OKQMB+cPITUp1lZDbX0g7SiE4bP8jHxQUwV6UMTg0Tm SdO5U0zEEtFUhS6ikd15ta/jvM0ABwM= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=gmail.com header.s=20230601 header.b=FsVGvXwc; dmarc=pass (policy=none) header.from=gmail.com; spf=pass (imf29.hostedemail.com: domain of 21cnbao@gmail.com designates 209.85.222.47 as permitted sender) smtp.mailfrom=21cnbao@gmail.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1736312803; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=GuHUj4dIaDgAm9+h4UiMj4vEuRqX6naNBIjrtSVnTqo=; b=p7z6I3Gox05Ff73/lnkENJdwxNxcPQKDzK/Ap1HdZrSd5Ru2FrMNCWHMlkCgVhqNCraS3U P42bktORx7X5oxgp1BCVjHeM9VHkkV1qswcng4RjZwfx89Kobj006GAYdXwL7hVeOIMQMC Cy5dvBkl2kvBIwSn9irkkCeZMTWsH3Y= Received: by mail-ua1-f47.google.com with SMTP id a1e0cc1a2514c-85c4557fce0so3256091241.3 for ; Tue, 07 Jan 2025 21:06:43 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20230601; t=1736312802; x=1736917602; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=GuHUj4dIaDgAm9+h4UiMj4vEuRqX6naNBIjrtSVnTqo=; b=FsVGvXwcgXspHWAzf0tSekvi3Pf3YQCJSsFLKRLPVGx2/+K7oOBgg/DqUOfh5DNcWL 3qqtdisPTPhOa2Wr5I5rG9ZNz293C8VKWr4T2UxXKN1HJxMfPuwMNOvKYCme9YlLpToZ fCeDEtMTf3t+80MfdsCNU1da3VTEyYJ7d0jtk50gPCV8vRHf9SlCzHSL/BqTeVaz37kG U2TkbMJkJVf5l9JkqAjIY2Xos/xCnlsL5ggGVP3C3asz/1Awm1G4tahdejIwaLPGoWfU SM7jFSy/wJzO0QV/KW9Y7voNxxkcrpvuQGB3r3vaS3sCjFCHOYNnM114DNQfGsWVUSrk E8og== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1736312802; x=1736917602; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=GuHUj4dIaDgAm9+h4UiMj4vEuRqX6naNBIjrtSVnTqo=; b=sdZ3So4vmAtE8LQjpAdsI3iVqwNnJEH2U0/zphnHaKssnnAurp9AZ+dogZp3K9U+Yl 8yJ8UAIjyqgWf/amCoi1fdmHKKes0M1zRJjqTe94Udu+rls2HdLMLWaPFsDpjDZPl0Rj T2JCan+iN3T31qgF+JxeBjvUk3VOtmjeqDpGWHF56u6Cdlxu2bT99bI0h7Mkff1Z1NQ9 1Ff5NLiwRTtWNOnpSUac9o6kWL746il3Ne1w1GJ/p3Dqz9fdsfm5K9Pxnnz07toVil8G rc3LPwTGHx+9wSlmjqQOynK4UkTBO1auRYCDvK1411ai7lA3FrARfmLbCu/bKB5sdLD5 AQmw== X-Forwarded-Encrypted: i=1; AJvYcCU58wYzeC+hkTJ4U/614QAXaWD8cfvZBmF/OLKgxU/HUxOvBa/TE/QCT4Ty+Rxe7PdKLwzizXdyVQ==@kvack.org X-Gm-Message-State: AOJu0YzRgrvDqeiDyYj8Kw1RZwYXFGv3mH+7LGoTTRFfLuVy+DiBieTs of988xLXFndivhF4iLYnKmhnxkKv6W8VfU7DpksTEJeAoUiy57OFlrfB20qYPDyol1IxwxvTC7R j6jMazbTB0fGzH+dRiKT5Gq1Fb3s= X-Gm-Gg: ASbGnct2hlCaiPss+2W1nP1kUmhTBe514T6Gn1hlCrcfgjOdLkgw1dQ18xNSBw8PPnG zPF0lzNZClZLCyxKF0JIZCGVEZfHrAG/K0jkjYIcQ1ZgwMyPEBSdNPu5+AiIjdPqZKgWy338= X-Google-Smtp-Source: AGHT+IHYkcgXVqBzp0wYwNKe0EAxYkg33lq3J3t749SzgatOZl9b7IA4Mdmzev4G/+1KqiLskIdTHphYBVcf5HHV20A= X-Received: by 2002:a05:6102:2928:b0:4af:c31d:b4e8 with SMTP id ada2fe7eead31-4b3d0dd756emr1261673137.14.1736312802546; Tue, 07 Jan 2025 21:06:42 -0800 (PST) MIME-Version: 1.0 References: <20250107222236.2715883-1-yosryahmed@google.com> <20250107222236.2715883-2-yosryahmed@google.com> In-Reply-To: From: Barry Song <21cnbao@gmail.com> Date: Wed, 8 Jan 2025 18:06:30 +1300 X-Gm-Features: AbW1kvYysiX0vlU67zpkRf7DvH5Z0lqmGE1f8gfsIHCYq6jHWED5sm2VnBKu7WU Message-ID: Subject: Re: [PATCH v2 2/2] mm: zswap: disable migration while using per-CPU acomp_ctx To: Nhat Pham Cc: Yosry Ahmed , Johannes Weiner , Andrew Morton , Chengming Zhou , Vitaly Wool , Sam Sun , "linux-mm@kvack.org" , "linux-kernel@vger.kernel.org" , "stable@vger.kernel.org" , "Sridhar, Kanchana P" Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Stat-Signature: 5r8368wrn4iyns99tmt8i3wcts85utai X-Rspam-User: X-Rspamd-Queue-Id: 8179B120005 X-Rspamd-Server: rspam08 X-HE-Tag: 1736312803-255622 X-HE-Meta: U2FsdGVkX1/UIl6cGS5eeH7HQyj2cgn7r7AGud4yJXP13FGxZTINVbrvMec6++E0GIiwya+t2vRTkhaVND0t3T2kejvdYvJfsebSkqhOWAz3OjLFRD/5aSwKjPs9jkVGNLdK6TOYGkQbGrfW3fUDl0nK5N0ppzAu3FrHGGCnw9mpYF8itHMw98Ul8JR9AIy3AVtMupTA+kUuBCLtBUH+CLDCnyvA4xybw/KAZ9DZ9QLY6w7K8uBJdgi82mE0Ar5ji/CMp+jy/+X0swo0ooU8GWlog0C7X2FHmWyl0xrw4F5mL5HmhICvkTV6Q1ETmYeUDjtc2Tfqy4+Z3MBq5Q4VnBM0Uy+8MMogzqhuI3Ci0uSOggzC1IeHic2QEWdg8YuX/sHdv+Ma016suNyw/UvHa8z6r2g8Tq92u9M09GDbF5CSrUKxlHYxUU8C0Hm9dv3zKG2kaZnpu/Loi8Hx2ev6Fh3uf8pVwkrdPmvz+h88acXplOzn9GLy6XYgIBmXylH4P3yuKyOPtkUGyrVU7hb4mKU5hUbKYM0soa2UV6fQ2DiEudG5SiBnNq8aeNWNMcoUzFizSEWZ6l6YgqH2mu0yl43zAUBrwJq1SzjApnW9klEvsHiNRzmps8rLg4He+Xfg8JUyQL+VU+tjOdmtRrOXQyot4CQJ6FhLdaJsKXCkp3xxdz3RjyAlBnWG6FJZ9jcMLxF4SOhZAqVtyGW67wmz3wcHJlfGuu2Dg509CAs1A3ruvzKlneXeRF03IWkL3szvucHELqeHW2/TDTcLcxUlPy1PGfotdThe42gWjkBHV8eG13chST57z6LlPwvIpBtOMUj4DStt7xAodp/AIsfj+SJ2QQh5/ZbnChR0bZT5uaMNRVCExAXRE2R/yub/o2yyDsQ0Hd3ABE/YtnD4Cmw3V8xyEmDVBgKDH4ld2f4+qrSS9e9xoTX5jTLM2b0sTtCK+sjVTiGhrFGB2s6VwdC DJgDV79Z cPPo3/yHfx7P3yxhh0oGs1kM6n/n5ssUPg9aCTPgMtsULG6SxB/N4d+I3O/JdkSRYB+ykiH6bT3FEN+LnvZcCP9xgnH9+egMUkSuNwstMx0/u2QVmY7LjlCaZO5LaMfgy/BkU0CDtoGaTsBrBiwwXSYzXnigWViabb+UUwYABO5ooMnD8ePwMRriGbppEVBZgyTrg4FOWgW4reG9ZcfJiS15jUFc/6wYwRi2PtNC/fzWETRpN896Q8kwfqWCLhF1q5E9LVEJcr76d1ySN7tqtzVMv3gazYr3QD2eQIViLIiVP288yPkIppsi1Jj5hMCfvEGycZJ+l/0VNkefv1OavTxYbh0AYA+CYVaRdafWDbcmHg4aDjtsV7oKhmVp+AQTeF7/5Rw404MkB+oxBgwqtIM9V5JewSH6GaVGYVLNfUZpp44O+UDdpKuZGeiyDXGP7qrDXil53u6nUSYY= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, Jan 8, 2025 at 5:46=E2=80=AFPM Nhat Pham wrote: > > On Wed, Jan 8, 2025 at 9:34=E2=80=AFAM Yosry Ahmed wrote: > > > > > > Actually, using the mutex to protect against CPU hotunplug is not too > > complicated. The following diff is one way to do it (lightly tested). > > Johannes, Nhat, any preferences between this patch (disabling > > migration) and the following diff? > > I mean if this works, this over migration diasbling any day? :) > > > > > diff --git a/mm/zswap.c b/mm/zswap.c > > index f6316b66fb236..4d6817c679a54 100644 > > --- a/mm/zswap.c > > +++ b/mm/zswap.c > > @@ -869,17 +869,40 @@ static int zswap_cpu_comp_dead(unsigned int cpu, > > struct hlist_node *node) > > struct zswap_pool *pool =3D hlist_entry(node, struct zswap_pool= , node); > > struct crypto_acomp_ctx *acomp_ctx =3D per_cpu_ptr(pool->acomp_= ctx, cpu); > > > > + mutex_lock(&acomp_ctx->mutex); > > if (!IS_ERR_OR_NULL(acomp_ctx)) { > > if (!IS_ERR_OR_NULL(acomp_ctx->req)) > > acomp_request_free(acomp_ctx->req); > > + acomp_ctx->req =3D NULL; > > if (!IS_ERR_OR_NULL(acomp_ctx->acomp)) > > crypto_free_acomp(acomp_ctx->acomp); > > kfree(acomp_ctx->buffer); > > } > > + mutex_unlock(&acomp_ctx->mutex); > > > > return 0; > > } > > > > +static struct crypto_acomp_ctx *acomp_ctx_get_cpu_locked( > > + struct crypto_acomp_ctx __percpu *acomp_ctx) > > +{ > > + struct crypto_acomp_ctx *ctx; > > + > > + for (;;) { > > + ctx =3D raw_cpu_ptr(acomp_ctx); > > + mutex_lock(&ctx->mutex); > > I'm a bit confused. IIUC, ctx is per-cpu right? What's protecting this > cpu-local data (including the mutex) from being invalidated under us > while we're sleeping and waiting for the mutex? > > If it is somehow protected, then yeah this seems quite elegant :) thought about this again. Could it be the following? bool cpus_is_read_locked(void) { return percpu_is_read_locked(&cpu_hotplug_lock); } in zswap: bool locked =3D cpus_is_read_locked(); if (!locked) cpus_read_lock(); .... // do our job if (!locked) cpus_read_unlock(); This seems to resolve all three problems: 1. if our context has held read lock, we won't hold it again; 2. if other contexts are holding write lock, we wait for the completion of cpuhotplug by acquiring read lock 3. if our context hasn't held a read lock, we hold it. > > > + if (likely(ctx->req)) > > + return ctx; > > + /* Raced with zswap_cpu_comp_dead() on CPU hotunplug */ > > + mutex_unlock(&ctx->mutex); > > + } > > +} > > + > > +static void acomp_ctx_put_unlock(struct crypto_acomp_ctx *ctx) > > +{ > > + mutex_unlock(&ctx->mutex); > > +} > > + > > static bool zswap_compress(struct page *page, struct zswap_entry *entr= y, > > struct zswap_pool *pool) > > { > > @@ -893,10 +916,7 @@ static bool zswap_compress(struct page *page, > > struct zswap_entry *entry, > > gfp_t gfp; > > u8 *dst; > > > > - acomp_ctx =3D raw_cpu_ptr(pool->acomp_ctx); > > - > > - mutex_lock(&acomp_ctx->mutex); > > - > > + acomp_ctx =3D acomp_ctx_get_cpu_locked(pool->acomp_ctx); > > dst =3D acomp_ctx->buffer; > > sg_init_table(&input, 1); > > sg_set_page(&input, page, PAGE_SIZE, 0); > > @@ -949,7 +969,7 @@ static bool zswap_compress(struct page *page, > > struct zswap_entry *entry, > > else if (alloc_ret) > > zswap_reject_alloc_fail++; > > > > - mutex_unlock(&acomp_ctx->mutex); > > + acomp_ctx_put_unlock(acomp_ctx); > > return comp_ret =3D=3D 0 && alloc_ret =3D=3D 0; > > } > > > > @@ -960,9 +980,7 @@ static void zswap_decompress(struct zswap_entry > > *entry, struct folio *folio) > > struct crypto_acomp_ctx *acomp_ctx; > > u8 *src; > > > > - acomp_ctx =3D raw_cpu_ptr(entry->pool->acomp_ctx); > > - mutex_lock(&acomp_ctx->mutex); > > - > > + acomp_ctx =3D acomp_ctx_get_cpu_locked(entry->pool->acomp_ctx); > > src =3D zpool_map_handle(zpool, entry->handle, ZPOOL_MM_RO); > > /* Thanks Barry