From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 52383C47422 for ; Fri, 26 Jan 2024 22:52:09 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B74D76B0078; Fri, 26 Jan 2024 17:52:08 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id B24916B007B; Fri, 26 Jan 2024 17:52:08 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9EC496B007D; Fri, 26 Jan 2024 17:52:08 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 8ED196B0078 for ; Fri, 26 Jan 2024 17:52:08 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 596964063E for ; Fri, 26 Jan 2024 22:52:08 +0000 (UTC) X-FDA: 81722961936.26.5F99322 Received: from mail-ed1-f41.google.com (mail-ed1-f41.google.com [209.85.208.41]) by imf07.hostedemail.com (Postfix) with ESMTP id BE16640015 for ; Fri, 26 Jan 2024 22:52:06 +0000 (UTC) Authentication-Results: imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=htU3M7TO; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of zokeefe@google.com designates 209.85.208.41 as permitted sender) smtp.mailfrom=zokeefe@google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1706309526; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=nv6CuAe+GsEY4s10INbXlJWjCo/P1ReTexRtJp+fto4=; b=FJifHsxOI1ObZD/FSJmfqSHbHnUqLsSDR3hZfEIJNr0m+0KKCIUhyVExTG8CXNetbRvIYR NbiGOBStVOAko9LEuICHMWr2V8zMqSU8Pw7dV3DljB4WgAO9dE4tfqj9W0cqbAZ/x//YWw eFwNzd66tHmm4L+bnwKYKn7bHcV/CBs= ARC-Authentication-Results: i=1; imf07.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=htU3M7TO; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf07.hostedemail.com: domain of zokeefe@google.com designates 209.85.208.41 as permitted sender) smtp.mailfrom=zokeefe@google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1706309526; a=rsa-sha256; cv=none; b=wZAAp8xZbqC4/eixfMgKozBWWBKZzJ0CtdTW0EMR5FaOa+TcHJgNUFhCWoIYRG7ZdCfCOC QKVvrH/W5Tz4oa07qPPLUlWA7WUieb2ElXKSR2hkb5gyrFpLAzmOLjIZEM0XRlAJ/B8vXd PPqJlSFhbH2Dwo+xbxZoeK1RDBX0otI= Received: by mail-ed1-f41.google.com with SMTP id 4fb4d7f45d1cf-55818b7053eso5213a12.0 for ; Fri, 26 Jan 2024 14:52:06 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1706309525; x=1706914325; darn=kvack.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=nv6CuAe+GsEY4s10INbXlJWjCo/P1ReTexRtJp+fto4=; b=htU3M7TOkeOeCs7J5Z+6re+CT5GIIYJBpmo5qdgeXXsh7HXMhy8erjj6kW0eNfGyDH +PhWb3dui1nogT1L6VO/Z5L3VccAifqJCA+hAmFZCWNv1GD0WzJCpKAaFUQtgMH9e3XM 2Lov/oQXZz8YNvlU6JEaYxveMtYL2d3q2wjw17L8YWLk5yn/2Q5al/fQ6s5x9Nj/wZ3Z 09Z3gmNuUKwD+vCDozfRTAFlT69TEMP+/I/mFQtbmtXOZUaXgx5svL2cI4nGOmOzEGiZ Dsmb2XFrPd/rqkQgzO63X3GXMhAUCJ5h3jvqdCGXVMLkn5dqz/DrCU0Ve7OFoWhj8KHR 2x7g== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1706309525; x=1706914325; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=nv6CuAe+GsEY4s10INbXlJWjCo/P1ReTexRtJp+fto4=; b=R93O0/XRC/OYLh9UbqVId8FQJ/S1mk6G01HJwKcUJ99uO31jAYWErN3eAKb0tcYHWV 7QUuy5tbkznfv64RWry47Pp3khEcvUdlYf4TJUJz47b7AueckQWEFMbirxHHKUp7Nv6K lKCVUlhpm11nKcSXJKT2jJNtzX0cz+LbKKadBhtYCeEkjF0rxA05wL96DOR9e1ofJNe5 hJIVqW4cx7bVMVc2/n7+hTC+s48lORdRrgek+/gPzg4wvm5fKz1YFZS2Kaannxf+wC8p Ub4munPDqYrOfG8v8P0ISLlkjwJCB/giU8TR1yf3o3zhveyU9fVrF7S5YuabhzJKqayc hoTg== X-Gm-Message-State: AOJu0YwmF9BJiev8F916B4x/BtYGt9m9S2bspTfX0Bip2DsRqtSBuGUO d1Y0bWrPh3p/yNW+RQkYlZQJS/sfcChT3NZ53KAVR+s5MYJwP2Zh51/JFdTL06bnAcRzHBmNWE7 brWH1O/6naHKRHGm/3rxLt/DB/e2BtULlRvvQ X-Google-Smtp-Source: AGHT+IGaDtH7FpFYdEdiLIZCdiJulfrgpMDcdlLq96DsjIw+rzeyCE7D1PNmdFGY3OHVKPpNWFRD6bNf66BchO9id9w= X-Received: by 2002:a05:6402:a43:b0:55c:8eda:6e19 with SMTP id bt3-20020a0564020a4300b0055c8eda6e19mr254822edb.2.1706309524959; Fri, 26 Jan 2024 14:52:04 -0800 (PST) MIME-Version: 1.0 References: <5c7f25f9-f86b-8e15-8603-e212b9911cac@quicinc.com> <342a8854-eef5-f68a-15e5-275de70e3f01@quicinc.com> <5adb12eb-8403-5860-28eb-5f6ab12f3c04@quicinc.com> In-Reply-To: From: "Zach O'Keefe" Date: Fri, 26 Jan 2024 14:51:26 -0800 Message-ID: Subject: Re: [PATCH V3 3/3] mm: page_alloc: drain pcp lists before oom kill To: Michal Hocko Cc: Charan Teja Kalla , akpm@linux-foundation.org, mgorman@techsingularity.net, david@redhat.com, vbabka@suse.cz, hannes@cmpxchg.org, quic_pkondeti@quicinc.com, linux-mm@kvack.org, linux-kernel@vger.kernel.org, Axel Rasmussen , Yosry Ahmed , David Rientjes Content-Type: text/plain; charset="UTF-8" X-Rspam-User: X-Stat-Signature: f9jrt8ptmuk5o3ws7wyg67f7eyqyxgoc X-Rspamd-Server: rspam07 X-Rspamd-Queue-Id: BE16640015 X-HE-Tag: 1706309526-373821 X-HE-Meta: U2FsdGVkX18q8A8R6G7rwzjY5TqGJRc0poO6vpELM3Kz3r6Q/qJB3nDhCyn9WStHCIvVEonbBikPHNbxUwyFy2086IiGO7Cqmso0kbWuzLw0zLLX9ASVBrqtyYVQ3V1oOPWKFvSot5KR8aZXWLCkrSrrwsLjFbdmit37AeriDpkHwePKbskzCVu2aK76XtYDpx+xPx+xt39WhyoVcrcn+BUeyPkI0ipBLEiSGXmaR+rVZfUjOlhUxdPW9cA6skR/Fce3Gr3thtqfj8oJsezbnWZ1hu+8vgA5+yxuKz5JZggVjb6Xh2/2JXT97ee4uYUPsKULT6t3gIQxloaW1Pt6dbMCJYOcnjlH6QeYpA1cwT7kzl/cg02tuGMuA6q1l8A2swUTdLKxbwyW+oMDcE8QOuGbcPiYi562jNZwDpJW691CO/+mpID+4iHw2QXpVALUg2UpLomjb3MGb1+HGT0ffiQhnRZRevW/m1P/4AQMopD2HZOpuh8Qm6QCnAsiBoLjIoPd6uA3Gi0c9cemPa4zBmmygQVgNEo6lByLV+o3iOwYCVFPR/tHUpAQIZOJDkIeEar+sFnoBbZh/96W+4gGK2FekJNdH6CA4xCWJHiC56IzwCB0zdnu5hpqu6cXjNztKPddwsffjOGyU6raMXknzoob5AC1Ipy0A/Qr9hTkt9dwH0ZWSv2W1ZawO60vsoRQQhvc+3hojU4Kxrph4HZAq6hgUKYahGhQq3XLB9m8YXa6KNEA/qi6iUicVrB/BWGCezWtRqNyZhVJOPx+SStncVmOu7d5ldjCcRC7nXCJy8iz99wIV2vbKlCZ0itrBd6xH0ZDN/Mg0Way87e4mkFM7tm1NfbCdaf8Ci6IlbaVmtxYBT58odTsUXVkQrSI6kzp6y44RloTaCVJrtpj1ckhYDZr6gSpv7ine+2oDEODuE57fi2IeyHBx2Eg3NuIWjxnHAFLcyxsWmabyZgkxHc qgsJFYn0 GEQdKam5+xvlMVkuVIsCmNbboGL6FH/RsMWjfH2ybExHqMnc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000127, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: Hey Michal, > Do you have any example OOM reports? [..] Sure, here is one on a 1TiB, 128-physical core machine running a 5.10-based kernel (sorry, it reads pretty awkwardly when wrapped): ---8<--- mytask invoked oom-killer: gfp_mask=0x100cca(GFP_HIGHUSER_MOVABLE), order=0, oom_score_adj=0 <...> oom-kill:constraint=CONSTRAINT_NONE,nodemask=(null),cpuset=sdc,mems_allowed=0-1,global_oom,task_memcg=/sdc,task=mytask,pid=835214,uid=0 Out of memory: Killed process 835214 (mytask) total-vm:787716604kB, anon-rss:787536152kB, file-rss:64kB, shmem-rss:0kB, UID:0 pgtables:1541224kB oom_score_adj:0, hugetlb-usage:0kB Mem-Info: active_anon:320 inactive_anon:198083493 isolated_anon:0 active_file:128283 inactive_file:290086 isolated_file:0 unevictable:3525 dirty:15 writeback:0 slab_reclaimable:35505 slab_unreclaimable:272917 mapped:46414 shmem:822 pagetables:64085088 sec_pagetables:0 bounce:0 kernel_misc_reclaimable:0 free:325793 free_pcp:263277 free_cma:0 Node 0 active_anon:1112kB inactive_anon:268172556kB active_file:270992kB inactive_file:254612kB unevictable:12404kB isolated(anon):0kB isolated(file):0kB mapped:147240kB dirty:52kB writeback:0kB shmem:304kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:1310720kB writeback_tmp:0kB kernel_stack:32000kB pagetables:255483108kB sec_pagetables:0kB all_unreclaimable? yes Node 1 active_anon:168kB inactive_anon:524161416kB active_file:242140kB inactive_file:905732kB unevictable:1696kB isolated(anon):0kB isolated(file):0kB mapped:38416kB dirty:8kB writeback:0kB shmem:2984kB shmem_thp:0kB shmem_pmdmapped:0kB anon_thp:267732992kB writeback_tmp:0kB kernel_stack:8520kB pagetables:857244kB sec_pagetables:0kB all_unreclaimable? yes Node 0 Crash free:72kB min:108kB low:220kB high:332kB reserved_highatomic:0KB active_anon:0kB inactive_anon:111940kB active_file:280kB inactive_file:316kB unevictable:0kB writepending:4kB present:114284kB managed:114196kB mlocked:0kB bounce:0kB free_pcp:1528kB local_pcp:24kB free_cma:0kB lowmem_reserve[]: 0 0 0 0 Node 0 DMA32 free:66592kB min:2580kB low:5220kB high:7860kB reserved_highatomic:0KB active_anon:8kB inactive_anon:19456kB active_file:4kB inactive_file:224kB unevictable:0kB writepending:0kB present:2643512kB managed:2643512kB mlocked:0kB bounce:0kB free_pcp:8040kB local_pcp:244kB free_cma:0kB lowmem_reserve[]: 0 0 16029 16029 Node 0 Normal free:513048kB min:513192kB low:1038700kB high:1564208kB reserved_highatomic:0KB active_anon:1104kB inactive_anon:268040520kB active_file:270708kB inactive_file:254072kB unevictable:12404kB writepending:48kB present:533969920kB managed:525510968kB mlocked:12344kB bounce:0kB free_pcp:790040kB local_pcp:7060kB free_cma:0kB lowmem_reserve[]: 0 0 0 0 Node 1 Normal free:723460kB min:755656kB low:1284080kB high:1812504kB reserved_highatomic:0KB active_anon:168kB inactive_anon:524161416kB active_file:242140kB inactive_file:905732kB unevictable:1696kB writepending:8kB present:536866816kB managed:528427664kB mlocked:1588kB bounce:0kB free_pcp:253500kB local_pcp:12kB free_cma:0kB lowmem_reserve[]: 0 0 0 0 Node 0 Crash: 0*4kB 0*8kB 1*16kB (M) 0*32kB 0*64kB 0*128kB 0*256kB 0*512kB 0*1024kB 0*2048kB 0*4096kB = 16kB Node 0 DMA32: 80*4kB (UME) 74*8kB (UE) 23*16kB (UME) 21*32kB (UME) 40*64kB (UE) 35*128kB (UME) 3*256kB (UE) 9*512kB (UME) 13*1024kB (UM) 19*2048kB (UME) 0*4096kB = 66592kB Node 0 Normal: 1999*4kB (UE) 259*8kB (UM) 465*16kB (UM) 114*32kB (UE) 54*64kB (UME) 14*128kB (U) 74*256kB (UME) 128*512kB (UE) 96*1024kB (U) 56*2048kB (U) 46*4096kB (U) = 512292kB Node 1 Normal: 2280*4kB (UM) 12667*8kB (UM) 8859*16kB (UME) 5221*32kB (UME) 1631*64kB (UME) 899*128kB (UM) 330*256kB (UME) 0*512kB 0*1024kB 0*2048kB 0*4096kB = 723208kB Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Node 0 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=1048576kB Node 1 hugepages_total=0 hugepages_free=0 hugepages_surp=0 hugepages_size=2048kB 420675 total pagecache pages 0 pages in swap cache Swap cache stats: add 0, delete 0, find 0/0 Free swap = 268435456kB Total swap = 268435456kB ---8<--- Node 0/1 Normal free memory is below respective min watermarks, with 790040kB+253500kB ~= 1GiB of memory on pcp lists. With this patch, the GFP_HIGHUSER_MOVABLE + unrestricted mems_allowed allocation would have allowed us to access all that memory, very likely avoiding the oom. > [..] There were recent changes to scale > the pcp pages and it would be good to know whether they work reasonably > well even under memory pressure. I'm not familiar with these changes, but a quick check of recent activity points to v6.7 commit fa8c4f9a665b ("mm: fix draining remote pageset") ; is this what you are referring to? Thanks, and have a great day, Zach > > I am not objecting to the patch discussed here but it would be really > good to understand the underlying problem and the scale of it. > > Thanks! > -- > Michal Hocko > SUSE Labs