From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.6 required=3.0 tests=DKIM_INVALID,DKIM_SIGNED, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B1F6EC10F25 for ; Fri, 6 Mar 2020 14:54:52 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 69A2A20848 for ; Fri, 6 Mar 2020 14:54:52 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=fail reason="signature verification failed" (2048-bit key) header.d=cmpxchg-org.20150623.gappssmtp.com header.i=@cmpxchg-org.20150623.gappssmtp.com header.b="gRRfcW+L" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 69A2A20848 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=cmpxchg.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 04A736B0005; Fri, 6 Mar 2020 09:54:52 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id F3C0A6B0006; Fri, 6 Mar 2020 09:54:51 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E2B496B0007; Fri, 6 Mar 2020 09:54:51 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0165.hostedemail.com [216.40.44.165]) by kanga.kvack.org (Postfix) with ESMTP id C779C6B0005 for ; Fri, 6 Mar 2020 09:54:51 -0500 (EST) Received: from smtpin01.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id 6CD792C14 for ; Fri, 6 Mar 2020 14:54:51 +0000 (UTC) X-FDA: 76565234382.01.scarf49_47682cf33e012 X-HE-Tag: scarf49_47682cf33e012 X-Filterd-Recvd-Size: 8491 Received: from mail-qv1-f67.google.com (mail-qv1-f67.google.com [209.85.219.67]) by imf33.hostedemail.com (Postfix) with ESMTP for ; Fri, 6 Mar 2020 14:54:50 +0000 (UTC) Received: by mail-qv1-f67.google.com with SMTP id s15so1026919qvn.8 for ; Fri, 06 Mar 2020 06:54:50 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:content-transfer-encoding:in-reply-to; bh=Bqd6ZYkkbQZsjUZxOCU8aqreR8WWA0WqwI2veaijy9M=; b=gRRfcW+LPsQTVdeJewFIIRnlA202F8zGYP2AgclMBTeb0CBxB1TbU0/qiLa9bcWTNV ZYUkdFX63Ngz0qQYmBBAsGy750bWhxRAb1MoZ7Pq1akjCaHE2OoE36CcoAhYypW1cZP6 4u8ShZYAvWn3C7OxQcK6b5spWaL/Cs4q/ZfbkevewKc9RIu1xoVTzTGT1scVhrdvoTss m/BJDhoiPFtdoKoGm7PFC20AWbeUO7zOJ3IUGNaJPCxI6kJtfVd962xNN7zqEsoKS6db 8MGRymHyT9+8k7YXy8dkFFtqgl6nhSBT5b8pzTKiQEa3pPvUSWUSmBUW/M2e5p1rJpIC jiCg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:content-transfer-encoding :in-reply-to; bh=Bqd6ZYkkbQZsjUZxOCU8aqreR8WWA0WqwI2veaijy9M=; b=Dz653DBEI4TaCuRMqSR1yIv7R7DSVLtekgUlw1wder7i1iIyXnVTcXjCOxT0tSL0OV gYgjW3P55SB4+O8fB5KoTdrm1AXpGXZFcUv5XOWFXyOpu3l7fyVcCF/DfK6ynw1hG/E0 +sTr6K9m1XICbpmFadutOSmsrTyuALdF4dhnr9CojW372dBQ6JTWzAgD859ofaI5oXF0 ECi5xBahomewcK0qfmbLK94bi+eWevGIWVawR+LvXVlCt+wm/DHOWY5yumDkrubYfH4y AC6Dp4DEbkJqL7Tq9n0jaL4RD1Ay0RoGSovm1O8/xrkx3a3espFlb2kRVvfpSmnDQlDs Zb1g== X-Gm-Message-State: ANhLgQ23hGB+04270TbUDpDQCFbpgeEbszbQziNuKYAYhrtEJfH8armO 2eEnzd7IB7LqmTzy33vhEI4BPQ== X-Google-Smtp-Source: ADFU+vtpqlVu9kBXDV+OWxw1CwkNjTyc3KvT1BBOrBwFLLi3fGybwAnVngU+gPyyWiBeEgltwxQACA== X-Received: by 2002:a05:6214:1351:: with SMTP id b17mr3209331qvw.73.1583506489751; Fri, 06 Mar 2020 06:54:49 -0800 (PST) Received: from localhost (70.44.39.90.res-cmts.bus.ptd.net. [70.44.39.90]) by smtp.gmail.com with ESMTPSA id 4sm6355655qkm.22.2020.03.06.06.54.48 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 06 Mar 2020 06:54:48 -0800 (PST) Date: Fri, 6 Mar 2020 09:54:48 -0500 From: Johannes Weiner To: Hugh Dickins Cc: Qian Cai , Matthew Wilcox , LKML , Andrew Morton , aarcange@redhat.com, Alex Shi , daniel.m.jordan@oracle.com, khlebnikov@yandex-team.ru, kirill@shutemov.name, kravetz@us.ibm.com, mhocko@kernel.org, mm-commits@vger.kernel.org, tj@kernel.org, vdavydov.dev@gmail.com, yang.shi@linux.alibaba.com, linux-mm@kvack.org Subject: Re: [failures] mm-vmscan-remove-unnecessary-lruvec-adding.patch removed from -mm tree Message-ID: <20200306145448.GC2508@cmpxchg.org> References: <20200306025041.rERhvnYmB%akpm@linux-foundation.org> <211632B1-2D6F-4BFA-A5A0-3030339D3D2A@lca.pw> <20200306033850.GO29971@bombadil.infradead.org> <97EE83E1-FEC9-48B6-98E8-07FB3FECB961@lca.pw> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline In-Reply-To: Content-Transfer-Encoding: quoted-printable X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Mar 05, 2020 at 08:17:46PM -0800, Hugh Dickins wrote: > On Tue, 3 Mar 2020, Alex Shi wrote: > > =E5=9C=A8 2020/3/3 =E4=B8=8A=E5=8D=886:12, Andrew Morton =E5=86=99=E9= =81=93: > > >> Thanks for Testing support from Intel 0day and Rong Chen, Fengguan= g Wu, > > >> and Yun Wang. > > > I'm not seeing a lot of evidence of review and test activity yet. = But > > > I think I'll grab patches 01-06 as they look like fairly > > > straightforward improvements. > >=20 > > cc Fengguang and Rong Chen > >=20 > > I did some local functional testing and kselftest, they all look fine= . > > 0day only warn me if some case failed. Is it no news is good news? :) >=20 > And now the bad news. >=20 > Andrew, please revert those six (or seven as they ended up in mmotm). > 5.6-rc4-mm1 without them runs my tmpfs+loop+swapping+memcg+ksm kernel > build loads fine (did four hours just now), but 5.6-rc4-mm1 itself > crashed just after starting - seconds or minutes I didn't see, > but it did not complete an iteration. >=20 > I thought maybe those six would be harmless (though I've not looked > at them at all); but knew already that the full series is not good yet: > I gave it a try over 5.6-rc4 on Monday, and crashed very soon on simple= r > testing, in different ways from what hits mmotm. >=20 > The first thing wrong with the full set was when I tried tmpfs+loop+ > swapping kernel builds in "mem=3D700M cgroup_disabled=3Dmemory", of cou= rse > with CONFIG_DEBUG_LIST=3Dy. That soon collapsed in a splurge of OOM kil= ls > and list_del corruption messages: __list_del_entry_valid < list_del < > __page_cache_release < __put_page < put_page < __try_to_reclaim_swap < > free_swap_and_cache < shmem_free_swap < shmem_undo_range. >=20 > When I next tried with "mem=3D1G" and memcg enabled (but not being used= ), > that managed some iterations, no OOM kills, no list_del warnings (was > it swapping? perhaps, perhaps not, I was trying to go easy on it just > to see if "cgroup_disabled=3Dmemory" had been the problem); but when > rebooting after that, again list_del corruption messages and crash > (I didn't note them down). >=20 > So I didn't take much notice of what the mmotm crash backtrace showed > (but IIRC shmem and swap were in it). >=20 > Alex, I'm afraid you're focusing too much on performance results, > without doing the basic testing needed - I thought we had given you > some hints on the challenging areas (swapping, move_charge_at_immigrate= , > page migration) when we attached a *correctly working* 5.3 version back > on 23rd August: >=20 > https://lore.kernel.org/linux-mm/alpine.LSU.2.11.1908231736001.16920@eg= gly.anvils/ >=20 > (Correctly working, except missing two patches I'd mistakenly dropped > as unnecessary in earlier rebases: but our discussions with Johannes > later showed to be very necessary, though their races rarely seen.) > > I have not had the time (and do not expect to have the time) to review > your series: maybe it's one or two small fixes away from being complete= , > or maybe it's still fundamentally flawed, I do not know. I had naively > hoped that you would help with a patchset that worked, rather than > cutting it down into something which does not. I'm a bit confused by this. I, and I believe Alex, kept going down a different path because it didn't sound like there was a solution to the compaction race. As I remember, the conversation ended on this: : Your race here (again, lruvec lock taken then PageLRU observed, but : page->mem_cgroup changed in between) really questions my whole scheme: : I am not going to propose a solution now, I'll have to go back and : recheck my assumptions all over. Certainly isolate_migratepage_block() : has a harder job than any other, but I need to re-review it all. https://lore.kernel.org/lkml/alpine.LSU.2.11.1911221616580.1144@eggly.anv= ils/ That's certainly why I kept looking and eventually proposed using PageLRU clearing as a lock. Maybe there is a better way to do it, but I didn't see it. An LRU list corruption in page_cache_release() suggests a bug in the way this new locking scheme works or is applied - rather than a gratuitous divergence from your series that could have been avoided. > Submitting your series to routine testing is much easier for me than > reviewing it: but then, yes, it's a pity that I don't find the time > to report the results on intervening versions, which also crashed. >=20 > What I have to do now, is set aside time today and tomorrow, to package > up the old scripts I use, describe them and their environment, and send > them to you (cc akpm in case I fall under a bus): so that you can > reproduce the crashes for yourself, and get to work on them. I think that would be very useful. tmpfs+loop+swapping+memcg+ksm kernel builds aren't exactly a go-to test case for most mm developers (although maybe they should be!)