From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.6 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4C6B4C433E0 for ; Mon, 1 Jun 2020 15:56:50 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D5DCA20734 for ; Mon, 1 Jun 2020 15:56:49 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=cmpxchg-org.20150623.gappssmtp.com header.i=@cmpxchg-org.20150623.gappssmtp.com header.b="HKghj+So" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D5DCA20734 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=cmpxchg.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6355680007; Mon, 1 Jun 2020 11:56:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 5E6F08E0006; Mon, 1 Jun 2020 11:56:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 4D74380007; Mon, 1 Jun 2020 11:56:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0105.hostedemail.com [216.40.44.105]) by kanga.kvack.org (Postfix) with ESMTP id 34B358E0006 for ; Mon, 1 Jun 2020 11:56:49 -0400 (EDT) Received: from smtpin05.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay05.hostedemail.com (Postfix) with ESMTP id D930F181AC9BF for ; Mon, 1 Jun 2020 15:56:48 +0000 (UTC) X-FDA: 76881096096.05.chess92_1330f1eba520 Received: from filter.hostedemail.com (10.5.16.251.rfc1918.com [10.5.16.251]) by smtpin05.hostedemail.com (Postfix) with ESMTP id BBC8F180143D9 for ; Mon, 1 Jun 2020 15:56:48 +0000 (UTC) X-HE-Tag: chess92_1330f1eba520 X-Filterd-Recvd-Size: 10045 Received: from mail-qk1-f194.google.com (mail-qk1-f194.google.com [209.85.222.194]) by imf42.hostedemail.com (Postfix) with ESMTP for ; Mon, 1 Jun 2020 15:56:48 +0000 (UTC) Received: by mail-qk1-f194.google.com with SMTP id w3so9490635qkb.6 for ; Mon, 01 Jun 2020 08:56:47 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=S8Qvy+/UE9YUsxgqFdhQ9/Oks0dEcBg3u0ktuSB1aWQ=; b=HKghj+SoRXku+7iNGt3tonDA2yUKOrwEGfoGCl2h1CSqMx7rfRKQRinLQQOnABCvFj GKZzX1yDugQuVj4idGs/6sRfXP14CktFckDGTm4xYUqucKsXpUcEb+fMCixFBk6VL7yS /BhxLIey46Kfmg8nuW0cophr0n5IemIAOvEosUTFaf07FNBNL4jibYEl6MCKTk3dIlSs VxbHNMwWpihiEHxX2Aqorug9uDtNstX7ySL1xsC7wjTVR1AzXhLyBtjPBpXuCto5Nw5w JkKxWU+kfeoicifgwCbj8B1NomFJAFFn9mBK/fxPT67MZ4kkEapJgrmVksmy1JhZ7zx7 2hkw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=S8Qvy+/UE9YUsxgqFdhQ9/Oks0dEcBg3u0ktuSB1aWQ=; b=YdIVUNjwWJdZEJIWmxXraF1GSaaVEW1us2ND7n82yOcvpzM7F+ZOX5Q3LRf5zztHI2 iIfZqccQjNQaSRT7IZJe21XjXkRSgplzKplyrA+b/U5pqNCJ7OaA9LcV8DfXjEPKNrm6 A2lsGPwv0DsTdln8iXQOYhTH9tmHGeRDl/LEIQij+9hKx/gtQJXbMxORYezQfDtS/eNW UYzH+BrLnpr7PiHpGuUlGPnvW2jssRN9hjwAlxkoKTVwsJggazXddp0Yo8dxuCngEFuX aS1+MYSR4VIYwU/lydQlLweGILC4EuhQAg/Z9ehxQuXdDIcBvCYfF4+TUWzwU66VUH+v bfoQ== X-Gm-Message-State: AOAM532WoFlIZnyEtREvp26hDzeFWHLYkmRwFJ5grnXmolOKzNJzC9C8 UedXW/6BTGSdkXhjqjzI1v45/itwn8M= X-Google-Smtp-Source: ABdhPJyPcOk7Pk8cRbO8oW58+QzceCI/0v8VQvUZVlfcX1wsNn/dDGgYF706DM4fb4f4yi6woF3gpQ== X-Received: by 2002:a05:620a:526:: with SMTP id h6mr20241138qkh.338.1591027007243; Mon, 01 Jun 2020 08:56:47 -0700 (PDT) Received: from localhost ([2620:10d:c091:480::1:d1fb]) by smtp.gmail.com with ESMTPSA id w204sm15250387qka.41.2020.06.01.08.56.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Jun 2020 08:56:46 -0700 (PDT) Date: Mon, 1 Jun 2020 11:56:15 -0400 From: Johannes Weiner To: Joonsoo Kim Cc: Linux Memory Management List , Rik van Riel , Minchan Kim , Michal Hocko , Andrew Morton , Joonsoo Kim , LKML , kernel-team@fb.com Subject: Re: [PATCH 05/14] mm: workingset: let cache workingset challenge anon Message-ID: <20200601155615.GA131075@cmpxchg.org> References: <20200520232525.798933-1-hannes@cmpxchg.org> <20200520232525.798933-6-hannes@cmpxchg.org> <20200527134333.GF6781@cmpxchg.org> <20200528170155.GA69521@cmpxchg.org> <20200529151228.GA92892@cmpxchg.org> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: X-Rspamd-Queue-Id: BBC8F180143D9 X-Spamd-Result: default: False [0.00 / 100.00] X-Rspamd-Server: rspam02 Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000029, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Jun 01, 2020 at 03:14:24PM +0900, Joonsoo Kim wrote: > 2020=EB=85=84 5=EC=9B=94 30=EC=9D=BC (=ED=86=A0) =EC=98=A4=EC=A0=84 12:= 12, Johannes Weiner =EB=8B=98=EC=9D=B4 =EC=9E=91=EC=84= =B1: > > > > On Fri, May 29, 2020 at 03:48:00PM +0900, Joonsoo Kim wrote: > > > 2020=EB=85=84 5=EC=9B=94 29=EC=9D=BC (=EA=B8=88) =EC=98=A4=EC=A0=84= 2:02, Johannes Weiner =EB=8B=98=EC=9D=B4 =EC=9E=91=EC= =84=B1: > > > > On Thu, May 28, 2020 at 04:16:50PM +0900, Joonsoo Kim wrote: > > > > > 2020=EB=85=84 5=EC=9B=94 27=EC=9D=BC (=EC=88=98) =EC=98=A4=ED=9B= =84 10:43, Johannes Weiner =EB=8B=98=EC=9D=B4 =EC=9E=91= =EC=84=B1: > > > > > > On Wed, May 27, 2020 at 11:06:47AM +0900, Joonsoo Kim wrote: > > > > The only way they could get reclaimed is if their access distance= ends > > > > up bigger than the file cache. But if that's the case, then the > > > > workingset is overcommitted, and none of the pages qualify for re= claim > > > > protection. Picking a subset to protect against the rest is arbit= rary. > > > > > > In the fixed example, although other file (500 MB) is repeatedly ac= cessed, > > > it's not workingset. If we have unified list (file + anon), access = distance of > > > Pn will be larger than whole memory size. Therefore, it's not overc= ommitted > > > workingset and this patch wrongly try to activate it. As I said bef= ore, > > > without considering inactive_age for anon list, this calculation ca= n not be > > > correct. > > > > You're right. If we don't take anon age into account, the activations > > could be over-eager; however, so would counting IO cost and exerting > > pressure on anon be, which means my previous patch to split these two > > wouldn't fix fundamental the problem you're pointing out. We simply >=20 > Splitting would not fix the fundamental problem (over-eager) but it wou= ld > greatly weaken the problem. Just counting IO cost doesn't break the > active/inactive separation in file list. It does cause more scan on ano= n list > but I think that it's endurable. I think the split is a good idea. The only thing I'm not sure yet is if we can get away without an additional page flag if the active flag cannot be reused to denote thrashing. I'll keep at it, maybe I can figure something out. But I think it would be follow-up work. > > have to take anon age into account for the refaults to be comparable. >=20 > Yes, taking anon age into account is also a good candidate to fix the p= roblem. Okay, good. > > However, your example cannot have a completely silent stable state. A= s > > we stop workingset aging, the refault distances will slowly increase > > again. We will always have a bit of churn, and rightfully so, because > > the workingset *could* go stale. > > > > That's the same situation in my cache-only example above. Anytime you > > have a subset of pages that by itself could fit into memory, but can'= t > > because of an established workingset, ongoing sampling is necessary. > > > > But the rate definitely needs to reduce as we detect that in-memory > > pages are indeed hot. Otherwise we cause more churn than is required > > for an appropriate rate of workingset sampling. > > > > How about the patch below? It looks correct, but I will have to re-ru= n > > my tests to make sure I / we are not missing anything. >=20 > Much better! It may solve my concern mostly. Okay thanks for confirming. I'll send a proper version to Andrew. > But, I still think that modified refault activation equation isn't > safe. The next > problem I found is related to the scan ratio limit patch ("limit the ra= nge of > LRU type balancing") on this series. See the below example. >=20 > anon: Hot (X M) > file: Hot (200 M) / dummy (200 M) > P: 1200 M (3 parts, each one 400 M, P1, P2, P3) > Access Pattern: A -> F(H) -> P1 -> A -> F(H) -> P2 -> ... -> >=20 > Without this patch, A and F(H) are kept on the memory and look like > it's correct. >=20 > With this patch and below fix, refault equation for Pn would be: >=20 > Refault dist of Pn =3D 1200 (from file non-resident) + 1200 * anon scan > ratio (from anon non-resident) > anon + active file =3D X + 200 > 1200 + 1200 * anon scan ratio (0.5 ~ 2) < X + 200 That doesn't look quite right to me. The anon part of the refault distance is driven by X, so the left-hand of this formula contains X as well. 1000 file (1200M reuse distance, 200M in-core size) + F(H) reactivations = + X * scan ratio < X + 1000 Activations persist as long as anon isn't fully scanned and it isn't established yet that it's fully hot. Meaning, we optimistically assume the refaulting pages can be workingset until we're proven wrong. > According to the size of X, Pn's refault result would be different. Pn = could > be activated with large enough X and then F(H) could be evicted. In ide= al > case (unified list), for this example, Pn should not be activated in an= y X. Yes. The active/iocost split would allow us to be smarter about it. > This is a fundamental problem since we have two list type (file/anon) a= nd > scan ratio limit is required. Anyway, we need to take care of this real= ity and > the way most safe is to count IO cost instead of doing activation in th= is > 'non-resident dist < (active + anon list)' case. Agreed here again. > Again, for this patch, I'm not confident myself so please let me know i= f I'm > wrong. As far as this patch goes, I think it's important to look at the bigger picture. We need to have convergence first before being able to worry about optimizing. Stable states are optimizations, but false stable states are correctness problems. For the longest time, we scanned active pages unconditionally during page reclaim. This was always safe in the sense that it wouldn't get stuck on a stale workingset, but it incurs unnecessary workingset churn when reclaim is driven by use-once patterns. We optimized the latter too aggressively, and as a result caused situations where we indefinitely fail to cache the hottest data. That's not really a workable trade-off. With the active/iocost split you're suggesting, we can reasonably optimize your example scenario. But we can't do it if the flipside means complete failure to transition between in-memory sets. So I think we should go ahead with this patch (with the anon age recognition fixed, because that's a correctness issue), and follow it up with the stable state optimization to shrink anon first.