From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-17.0 required=3.0 tests=BAYES_00,DKIMWL_WL_MED, DKIM_SIGNED,DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS, INCLUDES_PATCH,MAILING_LIST_MULTI,NORMAL_HTTP_TO_IP,NUMERIC_HTTP_ADDR, SPF_HELO_NONE,SPF_PASS,USER_IN_DEF_DKIM_WL autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4E939C433B4 for ; Fri, 30 Apr 2021 06:24:29 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id BAF09613F4 for ; Fri, 30 Apr 2021 06:24:28 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org BAF09613F4 Authentication-Results: mail.kernel.org; dmarc=fail (p=reject dis=none) header.from=google.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 4A3846B00AB; Fri, 30 Apr 2021 02:24:28 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 453CD6B00AE; Fri, 30 Apr 2021 02:24:28 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 2CD056B00B0; Fri, 30 Apr 2021 02:24:28 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0193.hostedemail.com [216.40.44.193]) by kanga.kvack.org (Postfix) with ESMTP id 13FC96B00AB for ; Fri, 30 Apr 2021 02:24:28 -0400 (EDT) Received: from smtpin29.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id BE6A14DA2 for ; Fri, 30 Apr 2021 06:24:27 +0000 (UTC) X-FDA: 78088044174.29.EF729C7 Received: from mail-wm1-f42.google.com (mail-wm1-f42.google.com [209.85.128.42]) by imf17.hostedemail.com (Postfix) with ESMTP id 787DF40002E3 for ; Fri, 30 Apr 2021 06:24:23 +0000 (UTC) Received: by mail-wm1-f42.google.com with SMTP id b19-20020a05600c06d3b029014258a636e8so999204wmn.2 for ; Thu, 29 Apr 2021 23:24:27 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20161025; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc:content-transfer-encoding; bh=YFIkLOXwzPtOenJjCQiiKg1nIMhir/w9NnvjOy6hjlk=; b=ssrM2lZ3tHZqS5AjLeEgyQQ34R3rHatMp6+MaePG4kjsYQb2YvILD7OIkqg/fcWMMh 2S5E0vY6Rh2EKSthEMMdne3EN+VQBnEmYZ5gjsLfBf3H7dU4r0hYDIVWdK8Sx/R8t3sI 771PXtnadp3N/ThrAafRjJcZhm0sxjClvZLXL2r/rYDE9k7fvB0JAtp9drqJzXFx4k1d hMIJyx+vSEMcIFKGA5Jl6bLxNqNrtVBJa44OpKT+jtdpvsTllmKebHI4M/RarngQfhrN 23U1eCAiwuv4nDd7dmFs2Q87t/SIG5j+5mQ+CGt2jk7z06yxH8sGTS3zkCk1xEdLumS6 R79w== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc:content-transfer-encoding; bh=YFIkLOXwzPtOenJjCQiiKg1nIMhir/w9NnvjOy6hjlk=; b=RVWAtlsfHpzVa7IBPSu3u+IJluWEIr3697QGHt+EaEcVEo7zOyJSjZbjCoRJTJJopN ALzPIWuVYeuI2eVySMxstxITEmS5/UxBRlp89UNC329L23RP1zK3jISNPBmDtzX2Bn/i 40q7Z7E2ZEk/YbUZtuBm3UJVKkvl7fJxcFyP8pUBbOUXR0rGr31fTjyGBGfotJHHIVcg i/b3/cHK/0q4qdV+x53uF9E7J8e9ln+qizXE1axUUEwRP1X5n5OLmhZjzj+iMWF7OF2U d+nSTrB/iyfMoDhawRVAxDKuofhs7Hmbr2gyM2zKDZ9/O5G73Uy2zQvEp9HI7Ji20TVH WE+g== X-Gm-Message-State: AOAM530+hVngwZgcfO2dgyhF7sSKqUItMp9atlD/QiTcydcvJQlPdY/Y nom8JGmhZESnI54AUOhyHbUemBmUzAyDF2gj+M66Ww== X-Google-Smtp-Source: ABdhPJw2+y5IZ9nZ1JQ3YFqC95nxaDPwH63eKPyBTSKO6z3ePEydo5FduNfLn2criWdcaRROs52ule3mPitdFMNYZcw= X-Received: by 2002:a7b:cf12:: with SMTP id l18mr4234009wmg.37.1619763865958; Thu, 29 Apr 2021 23:24:25 -0700 (PDT) MIME-Version: 1.0 References: <20210416023536.168632-1-zhengjun.xing@linux.intel.com> <7b7a1c09-3d16-e199-15d2-ccea906d4a66@linux.intel.com> <7a0fecab-f9e1-ad39-d55e-01e574a35484@linux.intel.com> In-Reply-To: From: Yu Zhao Date: Fri, 30 Apr 2021 00:24:14 -0600 Message-ID: Subject: Re: [RFC] mm/vmscan.c: avoid possible long latency caused by too_many_isolated() To: Xing Zhengjun Cc: Andrew Morton , Linux-MM , linux-kernel , Huang Ying , Tim Chen , Shakeel Butt , Michal Hocko , wfg@mail.ustc.edu.cn Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable Authentication-Results: imf17.hostedemail.com; dkim=pass header.d=google.com header.s=20161025 header.b=ssrM2lZ3; dmarc=pass (policy=reject) header.from=google.com; spf=pass (imf17.hostedemail.com: domain of yuzhao@google.com designates 209.85.128.42 as permitted sender) smtp.mailfrom=yuzhao@google.com X-Rspamd-Server: rspam03 X-Stat-Signature: ti9fnxsf8gkh88gcxephfd5pe9ke4ato X-Rspamd-Queue-Id: 787DF40002E3 Received-SPF: none (google.com>: No applicable sender policy available) receiver=imf17; identity=mailfrom; envelope-from=""; helo=mail-wm1-f42.google.com; client-ip=209.85.128.42 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1619763863-561316 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Apr 29, 2021 at 11:57 PM Xing Zhengjun wrote: > > Hi Yu, > > On 4/24/2021 4:23 AM, Yu Zhao wrote: > > On Fri, Apr 23, 2021 at 02:57:07PM +0800, Xing Zhengjun wrote: > >> On 4/23/2021 1:13 AM, Yu Zhao wrote: > >>> On Thu, Apr 22, 2021 at 04:36:19PM +0800, Xing Zhengjun wrote: > >>>> Hi, > >>>> > >>>> In the system with very few file pages (nr_active_file + nr_ina= ctive_file > >>>> < 100), it is easy to reproduce "nr_isolated_file > nr_inactive_file= ", then > >>>> too_many_isolated return true, shrink_inactive_list enter "msleep(10= 0)", the > >>>> long latency will happen. > >>>> > >>>> The test case to reproduce it is very simple: allocate many huge pag= es(near > >>>> the DRAM size), then do free, repeat the same operation many times. > >>>> In the test case, the system with very few file pages (nr_active_fil= e + > >>>> nr_inactive_file < 100), I have dumpped the numbers of > >>>> active/inactive/isolated file pages during the whole test(see in the > >>>> attachments) , in shrink_inactive_list "too_many_isolated" is very e= asy to > >>>> return true, then enter "msleep(100)",in "too_many_isolated" sc->gfp= _mask is > >>>> 0x342cca ("_GFP_IO" and "__GFP_FS" is masked) , it is also very easy= to > >>>> enter =E2=80=9Cinactive >>=3D3=E2=80=9D, then =E2=80=9Cisolated > in= active=E2=80=9D will be true. > >>>> > >>>> So I have a proposal to set a threshold number for the total file p= ages to > >>>> ignore the system with very few file pages, and then bypass the 100m= s sleep. > >>>> It is hard to set a perfect number for the threshold, so I just give= an > >>>> example of "256" for it. > >>>> > >>>> I appreciate it if you can give me your suggestion/comments. Thanks. > >>> > >>> Hi Zhengjun, > >>> > >>> It seems to me using the number of isolated pages to keep a lid on > >>> direct reclaimers is not a good solution. We shouldn't keep going > >>> that direction if we really want to fix the problem because migration > >>> can isolate many pages too, which in turn blocks page reclaim. > >>> > >>> Here is something works a lot better. Please give it a try. Thanks. > >> > >> Thanks, I will try it with my test cases. > > > > Thanks. I took care my sloppiness from yesterday and tested the > > following. It should apply cleanly and work well. Please let me know. > > > > diff --git a/include/linux/mmzone.h b/include/linux/mmzone.h > > index 47946cec7584..48bb2b77389e 100644 > > --- a/include/linux/mmzone.h > > +++ b/include/linux/mmzone.h > > @@ -832,6 +832,7 @@ typedef struct pglist_data { > > #endif > > > > /* Fields commonly accessed by the page reclaim scanner */ > > + atomic_t nr_reclaimers; > > > > /* > > * NOTE: THIS IS UNUSED IF MEMCG IS ENABLED. > > diff --git a/mm/vmscan.c b/mm/vmscan.c > > index 562e87cbd7a1..3fcdfbee89c7 100644 > > --- a/mm/vmscan.c > > +++ b/mm/vmscan.c > > @@ -1775,43 +1775,6 @@ int isolate_lru_page(struct page *page) > > return ret; > > } > > > > -/* > > - * A direct reclaimer may isolate SWAP_CLUSTER_MAX pages from the LRU = list and > > - * then get rescheduled. When there are massive number of tasks doing = page > > - * allocation, such sleeping direct reclaimers may keep piling up on e= ach CPU, > > - * the LRU list will go small and be scanned faster than necessary, le= ading to > > - * unnecessary swapping, thrashing and OOM. > > - */ > > -static int too_many_isolated(struct pglist_data *pgdat, int file, > > - struct scan_control *sc) > > -{ > > - unsigned long inactive, isolated; > > - > > - if (current_is_kswapd()) > > - return 0; > > - > > - if (!writeback_throttling_sane(sc)) > > - return 0; > > - > > - if (file) { > > - inactive =3D node_page_state(pgdat, NR_INACTIVE_FILE); > > - isolated =3D node_page_state(pgdat, NR_ISOLATED_FILE); > > - } else { > > - inactive =3D node_page_state(pgdat, NR_INACTIVE_ANON); > > - isolated =3D node_page_state(pgdat, NR_ISOLATED_ANON); > > - } > > - > > - /* > > - * GFP_NOIO/GFP_NOFS callers are allowed to isolate more pages, s= o they > > - * won't get blocked by normal direct-reclaimers, forming a circu= lar > > - * deadlock. > > - */ > > - if ((sc->gfp_mask & (__GFP_IO | __GFP_FS)) =3D=3D (__GFP_IO | __G= FP_FS)) > > - inactive >>=3D 3; > > - > > - return isolated > inactive; > > -} > > - > > /* > > * move_pages_to_lru() moves pages from private @list to appropriate = LRU list. > > * On return, @list is reused as a list of pages to be freed by the c= aller. > > @@ -1911,20 +1874,6 @@ shrink_inactive_list(unsigned long nr_to_scan, s= truct lruvec *lruvec, > > bool file =3D is_file_lru(lru); > > enum vm_event_item item; > > struct pglist_data *pgdat =3D lruvec_pgdat(lruvec); > > - bool stalled =3D false; > > - > > - while (unlikely(too_many_isolated(pgdat, file, sc))) { > > - if (stalled) > > - return 0; > > - > > - /* wait a bit for the reclaimer. */ > > - msleep(100); > > - stalled =3D true; > > - > > - /* We are about to die and free our memory. Return now. *= / > > - if (fatal_signal_pending(current)) > > - return SWAP_CLUSTER_MAX; > > - } > > > > lru_add_drain(); > > > > @@ -2903,6 +2852,8 @@ static void shrink_zones(struct zonelist *zonelis= t, struct scan_control *sc) > > unsigned long nr_soft_scanned; > > gfp_t orig_mask; > > pg_data_t *last_pgdat =3D NULL; > > + bool should_retry =3D false; > > + int nr_cpus =3D num_online_cpus(); > > > > /* > > * If the number of buffer_heads in the machine exceeds the maxim= um > > @@ -2914,9 +2865,18 @@ static void shrink_zones(struct zonelist *zoneli= st, struct scan_control *sc) > > sc->gfp_mask |=3D __GFP_HIGHMEM; > > sc->reclaim_idx =3D gfp_zone(sc->gfp_mask); > > } > > - > > +retry: > > for_each_zone_zonelist_nodemask(zone, z, zonelist, > > sc->reclaim_idx, sc->nodemask) { > > + /* > > + * Shrink each node in the zonelist once. If the zonelist= is > > + * ordered by zone (not the default) then a node may be s= hrunk > > + * multiple times but in that case the user prefers lower= zones > > + * being preserved. > > + */ > > + if (zone->zone_pgdat =3D=3D last_pgdat) > > + continue; > > + > > /* > > * Take care memory controller reclaiming has small influ= ence > > * to global LRU. > > @@ -2941,16 +2901,28 @@ static void shrink_zones(struct zonelist *zonel= ist, struct scan_control *sc) > > sc->compaction_ready =3D true; > > continue; > > } > > + } > > > > - /* > > - * Shrink each node in the zonelist once. If the > > - * zonelist is ordered by zone (not the default) = then a > > - * node may be shrunk multiple times but in that = case > > - * the user prefers lower zones being preserved. > > - */ > > - if (zone->zone_pgdat =3D=3D last_pgdat) > > - continue; > > + /* > > + * A direct reclaimer may isolate SWAP_CLUSTER_MAX pages = from > > + * the LRU list and then get rescheduled. When there are = massive > > + * number of tasks doing page allocation, such sleeping d= irect > > + * reclaimers may keep piling up on each CPU, the LRU lis= t will > > + * go small and be scanned faster than necessary, leading= to > > + * unnecessary swapping, thrashing and OOM. > > + */ > > + VM_BUG_ON(current_is_kswapd()); > > > > + if (!atomic_add_unless(&zone->zone_pgdat->nr_reclaimers, = 1, nr_cpus)) { > > + should_retry =3D true; > > + continue; > > + } > > + > > + if (last_pgdat) > > + atomic_dec(&last_pgdat->nr_reclaimers); > > + last_pgdat =3D zone->zone_pgdat; > > + > > + if (!cgroup_reclaim(sc)) { > > /* > > * This steals pages from memory cgroups over sof= tlimit > > * and returns the number of reclaimed pages and > > @@ -2966,13 +2938,20 @@ static void shrink_zones(struct zonelist *zonel= ist, struct scan_control *sc) > > /* need some check for avoid more shrink_zone() *= / > > } > > > > - /* See comment about same check for global reclaim above = */ > > - if (zone->zone_pgdat =3D=3D last_pgdat) > > - continue; > > - last_pgdat =3D zone->zone_pgdat; > > shrink_node(zone->zone_pgdat, sc); > > } > > > > + if (last_pgdat) > > + atomic_dec(&last_pgdat->nr_reclaimers); > > + else if (should_retry) { > > + /* wait a bit for the reclaimer. */ > > + if (!schedule_timeout_killable(HZ / 10)) > > + goto retry; > > + > > + /* We are about to die and free our memory. Return now. *= / > > + sc->nr_reclaimed +=3D SWAP_CLUSTER_MAX; > > + } > > + > > /* > > * Restore to original mask to avoid the impact on the caller if = we > > * promoted it to __GFP_HIGHMEM. > > @@ -4189,6 +4168,15 @@ static int __node_reclaim(struct pglist_data *pg= dat, gfp_t gfp_mask, unsigned in > > set_task_reclaim_state(p, &sc.reclaim_state); > > > > if (node_pagecache_reclaimable(pgdat) > pgdat->min_unmapped_pages= ) { > > + int nr_cpus =3D num_online_cpus(); > > + > > + VM_BUG_ON(current_is_kswapd()); > > + > > + if (!atomic_add_unless(&pgdat->nr_reclaimers, 1, nr_cpus)= ) { > > + schedule_timeout_killable(HZ / 10); > > + goto out; > > + } > > + > > /* > > * Free memory by calling shrink node with increasing > > * priorities until we have enough memory freed. > > @@ -4196,8 +4184,10 @@ static int __node_reclaim(struct pglist_data *pg= dat, gfp_t gfp_mask, unsigned in > > do { > > shrink_node(pgdat, &sc); > > } while (sc.nr_reclaimed < nr_pages && --sc.priority >=3D= 0); > > - } > > > > + atomic_dec(&pgdat->nr_reclaimers); > > + } > > +out: > > set_task_reclaim_state(p, NULL); > > current->flags &=3D ~PF_SWAPWRITE; > > memalloc_noreclaim_restore(noreclaim_flag); > > > > I use my compaction test case to test it, test more than 30 times can > not reproduce the 100ms sleep. I find that applies the patch, direct > reclaim path latency reduces much, but the direct compact path latency > double compares with it before. Thanks for the update. I think both the speedup and the slowdown are good news. Why? My theory is that this patch speeds up the reclaim path, which takes the lru lock. This in turn induces the lock contention and causes the slowdown in the compaction path. The compaction path also takes the lru lock: compact_zone_order() -> isolate_migratepages() -> isolate_migratepages_block() -> mem_cgroup_page_lruvec(). This theory can be proven by checking whether the slowdown in the path above. Also did you compare the following stats with and without this patch? $ grep compact /proc/vmstat compact_migrate_scanned 5114098 compact_free_scanned 93606243 compact_isolated 3629513 compact_stall 66 compact_fail 61 compact_success 5 compact_daemon_wake 390 compact_daemon_migrate_scanned 288911 compact_daemon_free_scanned 9835224 You might see a higher number of isolated pages. > 24) | __alloc_pages_direct_compact() { > 24) | try_to_compact_pages() { > 24) 0.131 us | __next_zones_zonelist(); > 24) @ 184008.2 us | compact_zone_order(); > 24) 0.189 us | __next_zones_zonelist(); > 24) 0.547 us | compact_zone_order(); > 24) 0.225 us | __next_zones_zonelist(); > 24) 0.592 us | compact_zone_order(); > 24) 0.146 us | __next_zones_zonelist(); > 24) @ 184012.3 us | } > 24) | get_page_from_freelist() { > 24) 0.160 us | __zone_watermark_ok(); > 24) 0.140 us | __next_zones_zonelist(); > 24) 0.141 us | __zone_watermark_ok(); > 24) 0.134 us | __next_zones_zonelist(); > 24) 0.121 us | __zone_watermark_ok(); > 24) 0.123 us | __next_zones_zonelist(); > 24) 1.688 us | } > 24) 0.130 us | ___might_sleep(); > 24) | __cond_resched() { > 24) 0.123 us | rcu_all_qs(); > 24) 0.370 us | } > 24) @ 184015.2 us | } > 24) | /* mm_page_alloc: page=3D0000000000000000 pfn=3D0 > order=3D9 migratetype=3D1 > gfp_flags=3DGFP_HIGHUSER_MOVABLE|__GFP_NOWARN|__GFP_COMP|__GFP_THISNODE *= / > 24) | /* memlatency: lat=3D184716 order=3D9 > gfp_flags=3D342cca > (GFP_HIGHUSER_MOVABLE|__GFP_NOWARN|__GFP_COMP|__GFP_THISNODE|0x812a3c6000= 000000^@)migratetype=3D1 > */ > /*The memlatency measures the latency of "__alloc_pages_nodemask" */ > > > -- > Zhengjun Xing