From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-0.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 735BBC2D0DB for ; Wed, 29 Jan 2020 16:40:00 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 27C9C20716 for ; Wed, 29 Jan 2020 16:39:59 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (2048-bit key) header.d=cmpxchg-org.20150623.gappssmtp.com header.i=@cmpxchg-org.20150623.gappssmtp.com header.b="NF6ZpH4S" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 27C9C20716 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=cmpxchg.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 9976B6B000A; Wed, 29 Jan 2020 11:39:59 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 9482D6B000C; Wed, 29 Jan 2020 11:39:59 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 8854B6B000D; Wed, 29 Jan 2020 11:39:59 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0041.hostedemail.com [216.40.44.41]) by kanga.kvack.org (Postfix) with ESMTP id 72D4F6B000A for ; Wed, 29 Jan 2020 11:39:59 -0500 (EST) Received: from smtpin06.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 2C071180AD801 for ; Wed, 29 Jan 2020 16:39:59 +0000 (UTC) X-FDA: 76431233718.06.bean73_91473ddc3eb3b X-HE-Tag: bean73_91473ddc3eb3b X-Filterd-Recvd-Size: 6171 Received: from mail-wm1-f67.google.com (mail-wm1-f67.google.com [209.85.128.67]) by imf50.hostedemail.com (Postfix) with ESMTP for ; Wed, 29 Jan 2020 16:39:58 +0000 (UTC) Received: by mail-wm1-f67.google.com with SMTP id q9so396257wmj.5 for ; Wed, 29 Jan 2020 08:39:57 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=cmpxchg-org.20150623.gappssmtp.com; s=20150623; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=6AM4f6F2mCpdk67MO056iRrphhgMrWUH20+VH5sckE0=; b=NF6ZpH4SL0Z1rx5gfKmwAGhhdaj6edSdXv9U2bsz9DcvwH5aIagYM5q9u853WH+O3e HRnQyXPCmpRBUYEJADHF8Qj4x9SexXmfGrdw93/EtHiIWoFDS4zpny4mhKft3a/UpNhW 2nhpF3GbWlX7EapGylmMt9Bo8T/dm3V+tuNZluBYgHPKO44Ju40YILm9i+Ro1Bsg4pk7 qVOkBT12eEz3NGlEh7PYvVLHLukGSlcbw8M3wkUUx6Y31m4MzZT4W7fub3dDtFCM5tDI MqMwnHGhY3c/07sOs/EC8J8Pr85u0YuaY8ZbMsBOR5s5yUFZ70cfc/mbt5fO2wd/ohEz a6Hw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=6AM4f6F2mCpdk67MO056iRrphhgMrWUH20+VH5sckE0=; b=OrAeMVdan55esDl9gTXaUuMl1c+gSgho/vbTV4TfiYq1TgbFGB6dLjxygsrcRZpU/d siHTYyMfpqlFijKC3pq8aD87ncCg5oet4KY6t5sWCJ/hyes7MvZO9OYj3lxmiFRG3q4m KCgFqsCihg57554fLEco4GOmt/wZjtyNLjhX4lep5pi640Oi4YRWtXf4r1D1Df58POmL OnWbX8yYwPOaYoDNQeVl2xf+sq9blvX643XWlDVDGpOPMK8BK71KTGAZAbS3cYaoOi1W DXgKlL4/pQrOMNK7U/hzmQ+0Jl3JxIMVlNborJhwnkA8WnYXJQLd0H80oqF2qfY43xIz 9WnQ== X-Gm-Message-State: APjAAAUgvg5G4o1gwzYg+4a9bSUYouHtlJbrlJQsVlL+mwMiw8o4NIWI UUtPzovoDL7+cLBdpqD6NMN4mg== X-Google-Smtp-Source: APXvYqw5NkYdL5mzG4D47oAR6WF65k146b1ZKafwqSSWgp6T+MOQXkWHj3Jq2Kctk6AiNNEHxuFSkQ== X-Received: by 2002:a1c:2089:: with SMTP id g131mr85613wmg.63.1580315996729; Wed, 29 Jan 2020 08:39:56 -0800 (PST) Received: from localhost ([2620:10d:c092:200::1:cb03]) by smtp.gmail.com with ESMTPSA id a13sm3506468wrp.93.2020.01.29.08.39.56 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 29 Jan 2020 08:39:56 -0800 (PST) Date: Wed, 29 Jan 2020 16:39:54 +0000 From: Johannes Weiner To: Michal Hocko Cc: Chris Edwards , "linux-mm@kvack.org" Subject: Re: Paging out when free memory is low but not exhausted (and available memory remains high) Message-ID: <20200129163954.GA210679@cmpxchg.org> References: <20200123123127.GK29276@dhcp22.suse.cz> <1579844599463.32567@otago.ac.nz> <20200124100423.GP29276@dhcp22.suse.cz> <20200127100646.GA203985@cmpxchg.org> <1580181722920.30551@otago.ac.nz> <1580187538078.61819@otago.ac.nz> <1580195997590.47770@otago.ac.nz> <1580297769621.48601@otago.ac.nz> <20200129133911.GM24244@dhcp22.suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20200129133911.GM24244@dhcp22.suse.cz> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Jan 29, 2020 at 02:39:11PM +0100, Michal Hocko wrote: > On Wed 29-01-20 11:36:09, Chris Edwards wrote: > > Following the idea that it's some interaction with the X server, I further noticed that switching out of X to a text virtual console makes the page-outs stop. Going back to the X VT, the page-outs resume. > > > > I've attached another set of vmstat logs for the following timeline: > > > > 1580290800 System is running Xorg, stress to limit memory, and dd to exercise the buffer cache, with constant page-outs > > 1580290810 Switch to a text virtual console - page-outs stop > > 1580290830 Switch back to X VT - page-outs resume > > > > I'm vaguely suspecting something to do with the way Xorg handles old-fashioned programs that do CPU-driven bitmap-based rendering, as my desktop does typically have a lot of these (urxvt instances, xosview, the Notion window manager itself) - maybe they cause some particular pattern of memory churn in the X server, and perhaps only with certain video drivers...could Xorg perhaps wrongly madvise() the kernel about certain memory? It seems notable that having glxgears should cause the page-outs to stop. > > > > However, even a minimal X session with a sakura or qterminal window seems to show some degree of needless page-outs with low memory and busy cache, though not as severe - however, it's difficult to avoid observer effects! There did seem to be a notable pattern of increasing swap utilisation when switching away from the X VT, and a drop in swap utilisation when switching back to X. > > > > Should I perhaps take it up with the Xorg people instead? > > It is quite possible that those GUI applications are over allocating and > talking to Xorg people might give you some hints how to pursue debugging > in that direction. > > From the MM kernel POV it is still very interesting to find out why the > anonymous memory is evicted while there is a lot of clean page cache. > I didn't get to look at your recent vmstat data though and will vanish > on vacation during the weekend. Chris mentioned in a previous email that he's seeing stalls in the i915 driver. It could well be their shrinker doing direct writepage calls on the object pages, which would explain why changing the VM policy on anon/file had no impact on what Chris is seeing. The vmstat logs show pages moving around on the unevictable list, but there are no mlocked pages. That would also match i915 driver activity: they mark the shmem mapping unevictable to keep reclaim control over those pages inside the shrinker rather than the VM. Chris, could you trace the i915 shrinker? Enable the shrinker trace point: # echo 1 >/sys/kernel/debug/tracing/events/i915/i915_gem_shrink/enable Then watch for events while the swapping is occuring: # cat /sys/kernel/debug/tracing/trace