From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id BB685E7718B for ; Wed, 25 Dec 2024 23:03:22 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9A19B6B007B; Wed, 25 Dec 2024 18:03:21 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 92A606B0083; Wed, 25 Dec 2024 18:03:21 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 7A46C6B0085; Wed, 25 Dec 2024 18:03:21 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id 5AE666B007B for ; Wed, 25 Dec 2024 18:03:21 -0500 (EST) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id BC1A080C56 for ; Wed, 25 Dec 2024 23:03:20 +0000 (UTC) X-FDA: 82935008436.05.C75879D Received: from mail-pl1-f171.google.com (mail-pl1-f171.google.com [209.85.214.171]) by imf27.hostedemail.com (Postfix) with ESMTP id 6957440006 for ; Wed, 25 Dec 2024 23:02:34 +0000 (UTC) Authentication-Results: imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nkubK9uv; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of rientjes@google.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=rientjes@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1735167779; a=rsa-sha256; cv=none; b=Rz/soX3Un13ACKWPzeTxDkpJ5A4FLz8jq0oCT6nmS2boZFET93JxMho1mTybc6i/jvuIlb iJMj5c9kpMCoCCmvKZFdu7OyCQ5Jqp24/YOkNVuTy6VEX3HpxuawTA4vgMat61mdJbPFIi uymt6jY81nVQgWPYrw6UmP+0gDoBNkY= ARC-Authentication-Results: i=1; imf27.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=nkubK9uv; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf27.hostedemail.com: domain of rientjes@google.com designates 209.85.214.171 as permitted sender) smtp.mailfrom=rientjes@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1735167779; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=0+QmCJq9N7M47iIpTN+M28rX8zFiCWZOHP68NqTtyXQ=; b=AJNpH7dK2qDHRcwPg1FD1EYn60CD7P64dCkqhelSxZdR/mMvi3M5YHL4gL2CwQRogwsa5c +boDwzf7SFnkhwfW6SJPSU3NcylQTmwGM7TP+awg4N43TOUsyB3FXxVx++MA7+9AAt6r3R hcTgF0mGfk4uSXVWvw/lqrNUUZtAnhE= Received: by mail-pl1-f171.google.com with SMTP id d9443c01a7336-21625b4f978so614535ad.0 for ; Wed, 25 Dec 2024 15:03:18 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1735167797; x=1735772597; darn=kvack.org; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:from:to:cc:subject:date:message-id:reply-to; bh=0+QmCJq9N7M47iIpTN+M28rX8zFiCWZOHP68NqTtyXQ=; b=nkubK9uvLIA6Io3HuIffJtPz6AQz5MUtXjmel0xR7mkZTy5+dffkTxEVL3htoWFuSc 6aJnFFq14iF3S3dtN3BuGNiEUsJ/Uj6t3NkwnF94ORFLGF2C6QNr9UQ0WUsrz0SX40cf HwRlke6q7mM49qg4tngWoo2pIb63PXpn44IolHubhlRmNHY4rzmh6pfBcDqnvoc1fzXU 0yzWnPSzHHRUJA4vwmxVY9IUfGGoUj/i+3t4b8z3t3qlxkMb6BIITgRzEz3PdYDi/hFx nXT9u6gb8FzBrYbXnMAuF898U3/6c0vO1SmYdYYTnV70KUvrID0wkHQZ0G7PoxZ79WfJ QShQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1735167797; x=1735772597; h=mime-version:references:message-id:in-reply-to:subject:cc:to:from :date:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=0+QmCJq9N7M47iIpTN+M28rX8zFiCWZOHP68NqTtyXQ=; b=fld4OlGKS4Ur8aqPDCzkRMC712saZBMhe9yl/kRJ8W0E4gYOCoo9PSUKnXJ3A8zUTu ezpxaYs342e5pPyChonXyLoeS0S38kUcyJKc+iSlWBkgPKKWOvxTaauFwQjHsZ1k8p0R YSaLoWumFMYjFsReeqcuGx/ACwZ13DqFfvPrA/xpCVS98/1HTEA0GvkDVfhgTf/zehoH zR1ApgbR4UOQCVR1Vew7NX5smU6oIexIO+Qn5qJ/kq7zy2adVG1Q+GKuCCC+JRZl7JLJ 8WSTMBVO/uh3aLS/Bh7DasO/gyLUI+hD13NlSmaHQdlsZ3hKkrDJRo61CaV+/pc7K4hA gDOg== X-Forwarded-Encrypted: i=1; AJvYcCXGztp55XHkFhrL+lgYVqmoeEoXYp16/3uTVuWRDvwey1PE/rMjFYZwW1KqtJNXRZc4L/iMURNacg==@kvack.org X-Gm-Message-State: AOJu0Yx1eS7+mzt4c0p2oxkVRtP1iKHR3jKc973rhSrLnQuC7F9Tr5Y4 qr9jn9SuLH5tuoFt9/UIIftM+Kn2qk8R/VwFxIFVs/M/JScO2UDzc9VY+UTSfg== X-Gm-Gg: ASbGncs0/z/JwOD1eLfy6W0fLFcBEJly3hayQIm0vbzRSzgyoFdux5Mp/Q1VX/R3Y2I uTuysy+0u6hpzHvnLRDH1O5+UNEBOjZCzeGzoXAMyPugr/yau4iGTFhEypaQGarLWSoYNi/xR8u FaiNxbGIjg1HhMvjLsQVUEANmcK2p0yWHRvwciZ5xOJBNR2VEWIg/FXIN7SBYk6oE00XLpUu5Ff HFbIp3WQ8w0J8P4PZiXwaU/w+6cl1DlXeXzSvl+2yP7sUfS212xl9k5cyDaM5KFV28RTVNvB0t6 6EjcpBKwDEc= X-Google-Smtp-Source: AGHT+IHnwErpW2GfaYRm3YefdzmidBuWRPmovq4erXqlV5iLeGay9nulCCM4TuX4VD9pew7rvrUjdg== X-Received: by 2002:a17:903:244f:b0:216:607d:c867 with SMTP id d9443c01a7336-219e777a523mr9151635ad.29.1735167797350; Wed, 25 Dec 2024 15:03:17 -0800 (PST) Received: from [2620:0:1008:15:336e:f011:c69b:3eea] ([2620:0:1008:15:336e:f011:c69b:3eea]) by smtp.gmail.com with ESMTPSA id d2e1a72fcca58-72aad8fbac8sm11720386b3a.159.2024.12.25.15.03.16 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 25 Dec 2024 15:03:16 -0800 (PST) Date: Wed, 25 Dec 2024 15:03:16 -0800 (PST) From: David Rientjes To: mengensun88@gmail.com cc: akpm@linux-foundation.org, linux-mm@kvack.org, alexjlzheng@tencent.com, MengEn Sun Subject: Re: [PATCH] mm/page_alloc: add cond_resched in __drain_all_pages() In-Reply-To: <1735107961-9938-1-git-send-email-mengensun@tencent.com> Message-ID: <3b000941-b1b6-befa-4ec9-2bff63d557c1@google.com> References: <1735107961-9938-1-git-send-email-mengensun@tencent.com> MIME-Version: 1.0 Content-Type: text/plain; charset=US-ASCII X-Rspam-User: X-Rspamd-Server: rspam03 X-Rspamd-Queue-Id: 6957440006 X-Stat-Signature: z1hc6nxcydtbrh89yxo3knz9zsnb64zr X-HE-Tag: 1735167754-980589 X-HE-Meta: U2FsdGVkX1+VsAKJKcFaV1xsJg3gfgghEVFl9ZDnNr32EgkWmDLoCdAyepD+MIkPy11NOjVy1wINPjT+OCLrWixNh2wUBr/lXMo+AAhf6stfdDbud87Lb8dXSjk8xAcD3E0oIWjTb5R2jhThg3VffT4t3n4ZXOT8Eue3gFiuKfIec1fUgUMgt61ZJ2Mq258BeFaPaYoN3K3NHzGkD7XJYqsLzEDJ5DG3joLeUzRUGdP1fZtgUL2AezpxZqRdjQHXExbVSWquEHlvkxb9KDWeQ4JwHK/0NxpKjjwZGPQHGkytOFzm3kRKSvhdOt8QlMcCYBSDqAFzL1XRQbJtYjpLMiYwhxbp0rn9sq+sHpoNk1hh62Y9Pmnwjw/k63PqChrTxwPl/I7np36r3ckPmHWvlGRCjOfcP4dzdKoTEs5njdSpiju83So0oWUoORnR3ypld2gUkQ5RVeh4nVlvcnKgP7MMcZZntkMn3cQXMZ+FpX+QgTBGmKF8szxVzxW9j1PVWv/HczAoobEVrx46EcsscC55b0eT6L1B16hWodpryFI6sDaKXGlhwilYqt8+/7SyTmTRhOEanStY/t7K5q9yLXxZCFjkmSKpRsNB8fBtWv8xbgAZdeYfKPndqceYZmQJHLv3VwcM+SRD+adySW5pCu8j3krA0H9M3lRvlW6Gm4zPsAL2HcHjyiLscWgjTsSc2dqu/KeDugjABxXD1JlYDmcP/4LK2QI1S0qCaStzm+fYNYE3DgLPlV3Dz4gMY5RiN+V6BAaebw7yfaa4LCzbxU5fhoDJnPnkYqPDzQIPKGmb1x2Ffpq7OJUIn1IVzkSpItMpUSyc1GIe/Rn/D543U1V5A8xKWs3pUYtgn2EoS+Z7IYBXDzttUYZ94wsCIm3vzmoJKa2o/P/uf43E5mSfUENrGKYDnx58+F/Wjs/0NK35vgcWgKk9p6diKcgkHUx91dpbPt5OsJOPC08HF6o QqUDI392 pdiK6FmxVZS5lPzFdp0X47EU4Vj4r5jDZo3cZteIVYPdjMnYwz0GReYIg7SrDQnuvG92nYeQH5jZuZqJ269sZxlDP8QPgU+Jl5YmXJMF+valuXKmPcJonlCNKUmHtGOThF6kq2w/1dlfPlrYoU3F3HddCMlrwcp0IP9M7V+lZofsBdfwhwXjqdEQ8ZTmevyN8H/JQnE8FDyFs/rm8PJkWow2ET7MygcatA2louqTtBFH10qXqxJkf3Ts1Vi67s3l3fQdQJ/5BYbANMXPvTBvIDEsSf49VoCLPeFR3wds5MxvuouzC0QMjgFevL5f+LYWzKvoUp112FMNYamCxnnZ5FWsx1tPDE1ryBE5CSRD6gv2J5kPDJ48EFdoHNpjAmvetxd7r1oTlEtUhWvDFITTq2Ogh8OhLhNxmnfjXOYExb0SK9gWv3MiIKmd6osSLwSuWiNE7bSK6nCTsTE+6KO1vtdju4w== X-Bogosity: Ham, tests=bogofilter, spamicity=0.187441, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Wed, 25 Dec 2024, mengensun88@gmail.com wrote: > From: MengEn Sun > > Since version v5.19-rc7, draining remote per-CPU pools (PCP) no > longer relies on workqueues; instead, the current CPU is > responsible for draining the PCPs of all CPUs. > > However, due to the lack of scheduling points in the > __drain_all_pages function, this can lead to soft locks in > some extreme cases. > > We observed the following soft-lockup stack on a 64-core, > 223GB machine during testing: > watchdog: BUG: soft lockup - CPU#29 stuck for 23s! [stress-ng-vm] > RIP: 0010:native_queued_spin_lock_slowpath+0x5b/0x1c0 > _raw_spin_lock > drain_pages_zone > drain_pages > drain_all_pages > __alloc_pages_slowpath > __alloc_pages_nodemask > alloc_pages_vma > do_huge_pmd_anonymous_page > handle_mm_fault > > Fixes: <443c2accd1b66> ("mm/page_alloc: remotely drain per-cpu lists") The < > would be removed. > Reviewed-by: JinLiang Zheng > Signed-off-by: MengEn Sun > --- > mm/page_alloc.c | 1 + > 1 file changed, 1 insertion(+) > > diff --git a/mm/page_alloc.c b/mm/page_alloc.c > index c6c7bb3ea71b..d05b32ec1e40 100644 > --- a/mm/page_alloc.c > +++ b/mm/page_alloc.c > @@ -2487,6 +2487,7 @@ static void __drain_all_pages(struct zone *zone, bool force_all_cpus) > drain_pages_zone(cpu, zone); > else > drain_pages(cpu); > + cond_resched(); > } > > mutex_unlock(&pcpu_drain_mutex); This is another example of a soft lockup that we haven't observed and we have systems with many more cores than 64. Is this happening because of contention on pcp->lock or zone->lock? I would assume the latter, but best to confirm. I think this is just papering over a scalability problem with zone->lock. How many NUMA nodes and zones does this 223GB system have? If this is a problem with zone->lock, this problem should likely be addressed more holistically.