From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-6.8 required=3.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_PATCH,MAILING_LIST_MULTI, MSGID_FROM_MTA_HEADER,SIGNED_OFF_BY,SPF_HELO_NONE,SPF_PASS,URIBL_BLOCKED autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id D61D1C352A4 for ; Tue, 11 Feb 2020 01:31:31 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 7C46D20733 for ; Tue, 11 Feb 2020 01:31:31 +0000 (UTC) Authentication-Results: mail.kernel.org; dkim=pass (1024-bit key) header.d=fb.com header.i=@fb.com header.b="K4jN5OI6"; dkim=pass (1024-bit key) header.d=fb.onmicrosoft.com header.i=@fb.onmicrosoft.com header.b="DrOI7zDb" DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 7C46D20733 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=fb.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 0E5696B0250; Mon, 10 Feb 2020 20:31:31 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 096566B0251; Mon, 10 Feb 2020 20:31:31 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id EA07E6B0252; Mon, 10 Feb 2020 20:31:30 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0133.hostedemail.com [216.40.44.133]) by kanga.kvack.org (Postfix) with ESMTP id CF0F46B0250 for ; Mon, 10 Feb 2020 20:31:30 -0500 (EST) Received: from smtpin09.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay01.hostedemail.com (Postfix) with ESMTP id 68171180AD802 for ; Tue, 11 Feb 2020 01:31:30 +0000 (UTC) X-FDA: 76476118740.09.flag26_64609e8aacd4f X-HE-Tag: flag26_64609e8aacd4f X-Filterd-Recvd-Size: 14775 Received: from mx0a-00082601.pphosted.com (mx0a-00082601.pphosted.com [67.231.145.42]) by imf39.hostedemail.com (Postfix) with ESMTP for ; Tue, 11 Feb 2020 01:31:29 +0000 (UTC) Received: from pps.filterd (m0044010.ppops.net [127.0.0.1]) by mx0a-00082601.pphosted.com (8.16.0.42/8.16.0.42) with SMTP id 01B1QJux029422; Mon, 10 Feb 2020 17:31:27 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.com; h=date : from : to : cc : subject : message-id : references : content-type : in-reply-to : mime-version; s=facebook; bh=XNLI6THpkhVbN4KPd3CMCtur/UFco0ZFntMH/HFsIks=; b=K4jN5OI6cVJQre0bNzV4cBJGirZ6YfmlA/fNzS3knzz+eBn9Y5IV9LjyTw1qxDR+pjFn veoVhxkURpXN2TFWSF4639C0o7OHH80n+tlBDHXWYdekKLQr4s5Bp8bP8Ru36+fl9mZ3 HiCQp1NLCNE1vKx6YzKxdLH4QbFDQjgZHTs= Received: from mail.thefacebook.com ([163.114.132.120]) by mx0a-00082601.pphosted.com with ESMTP id 2y2deurbb8-2 (version=TLSv1.2 cipher=ECDHE-RSA-AES128-GCM-SHA256 bits=128 verify=NOT); Mon, 10 Feb 2020 17:31:27 -0800 Received: from NAM11-BN8-obe.outbound.protection.outlook.com (100.104.98.9) by o365-in.thefacebook.com (100.104.94.199) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_128_GCM_SHA256) id 15.1.1779.2; Mon, 10 Feb 2020 17:31:26 -0800 ARC-Seal: i=1; a=rsa-sha256; s=arcselector9901; d=microsoft.com; cv=none; b=XJwsrLyIA+znufmUpftadQv40RNCze/dA/QqihQWMLGeGFa4IA/SAg8eaXoiljxRLbgFBfEtrcE5o3mHoqcLolUBUap9cpFDnfvTOdNRny32ZDgwN7QF1tZ0VqmLfrBzG2LC0x5C9v9qlzv1vAa2nXrXK07cjhquI3P39DEeNEQ2HXh5IsNWEOzBunvbY3gLfUh8OqAVuSjVVt8dLL/bh0w1jhOYrKw7q6hbyucPTmlWJwgP0azxxua9wOWUEgneB/TdsBljRz/ynM/rTve1k9ZzD6bMEyRjCktlPxfijpYze03PyO40oe1aWtzVwMwujASxIg4KWFCdoq6PMVLIlA== ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=microsoft.com; s=arcselector9901; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XNLI6THpkhVbN4KPd3CMCtur/UFco0ZFntMH/HFsIks=; b=kzx60eOcHxRgZwdFb+w4gmEGEdXyVEIviZdxNtp39dJf6i9E7jK8dw+8KjKfFM0puDipF5QixbjYPhRVKDX+u/yuSRsld6vBelTVVtwW0iY6HvFFIg4HK79TXyOTCeQXIsFMeJF+j1p7EPk1FQ/IZUw8cPPppg9KhL06rbLwDWsgZhXhlV1KRqoXVJSqWopLzSxFoDZ/Y+vCtcw8ND9E5/gHTvKf11jfCs9fAC3CSmY0o/Y4zG5R4qJuF8Qc8KosJxQY1eBXzp2bjDQAj3Kx64kBF91vMXtvPEl1VMa+JB/KZJweCnmRx4alzgSWRZs9AbvpmvR56E/4c0h56YNlwQ== ARC-Authentication-Results: i=1; mx.microsoft.com 1; spf=pass smtp.mailfrom=fb.com; dmarc=pass action=none header.from=fb.com; dkim=pass header.d=fb.com; arc=none DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=fb.onmicrosoft.com; s=selector2-fb-onmicrosoft-com; h=From:Date:Subject:Message-ID:Content-Type:MIME-Version:X-MS-Exchange-SenderADCheck; bh=XNLI6THpkhVbN4KPd3CMCtur/UFco0ZFntMH/HFsIks=; b=DrOI7zDb0INqR0F0UAD481w2RmMhVTcyk2WFVqeCWrcJQPW4b5mwBkoOT0jRI5PytaRbxyvnckQuouqwepQUwZJA+VAMZ00bUQfiDr9wMhWr9+Q+VMEwmHv7GLoiqqMK7j5hplWavRfHbtw9IGkKgE6qwHi3uUS2Vq/TCFr1V2M= Received: from BYAPR15MB2631.namprd15.prod.outlook.com (20.179.155.147) by BYAPR15MB3286.namprd15.prod.outlook.com (20.179.59.140) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2707.21; Tue, 11 Feb 2020 01:31:25 +0000 Received: from BYAPR15MB2631.namprd15.prod.outlook.com ([fe80::ccb6:a331:77d8:d308]) by BYAPR15MB2631.namprd15.prod.outlook.com ([fe80::ccb6:a331:77d8:d308%7]) with mapi id 15.20.2707.030; Tue, 11 Feb 2020 01:31:25 +0000 Date: Mon, 10 Feb 2020 17:31:18 -0800 From: Roman Gushchin To: Gavin Shan CC: , , , , Subject: Re: [RFC PATCH] mm/vmscan: Don't round up scan size for online memory cgroup Message-ID: <20200211013118.GA147346@carbon.lan> References: <20200210121445.711819-1-gshan@redhat.com> <20200210161721.GA167254@tower.DHCP.thefacebook.com> <9919b674-244d-0a55-c842-b0661585f9e2@redhat.com> Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <9919b674-244d-0a55-c842-b0661585f9e2@redhat.com> X-ClientProxiedBy: BL0PR05CA0004.namprd05.prod.outlook.com (2603:10b6:208:91::14) To BYAPR15MB2631.namprd15.prod.outlook.com (2603:10b6:a03:150::19) MIME-Version: 1.0 Received: from carbon.lan (2620:10d:c091:480::31b9) by BL0PR05CA0004.namprd05.prod.outlook.com (2603:10b6:208:91::14) with Microsoft SMTP Server (version=TLS1_2, cipher=TLS_ECDHE_RSA_WITH_AES_256_GCM_SHA384) id 15.20.2729.13 via Frontend Transport; Tue, 11 Feb 2020 01:31:23 +0000 X-Originating-IP: [2620:10d:c091:480::31b9] X-MS-PublicTrafficType: Email X-MS-Office365-Filtering-Correlation-Id: 8ed90bfa-d75f-4b44-395a-08d7ae921b9d X-MS-TrafficTypeDiagnostic: BYAPR15MB3286: X-Microsoft-Antispam-PRVS: X-FB-Source: Internal X-MS-Oob-TLC-OOBClassifiers: OLM:1107; X-Forefront-PRVS: 0310C78181 X-Forefront-Antispam-Report: SFV:NSPM;SFS:(10019020)(39860400002)(396003)(376002)(346002)(366004)(136003)(189003)(199004)(53546011)(6506007)(186003)(6916009)(16526019)(86362001)(2906002)(966005)(52116002)(7696005)(478600001)(36756003)(5660300002)(55016002)(66946007)(66476007)(66556008)(33656002)(9686003)(4326008)(1076003)(316002)(81166006)(8936002)(6666004)(8676002)(8886007)(81156014)(27376004);DIR:OUT;SFP:1102;SCL:1;SRVR:BYAPR15MB3286;H:BYAPR15MB2631.namprd15.prod.outlook.com;FPR:;SPF:None;LANG:en;PTR:InfoNoRecords;MX:1;A:1; Received-SPF: None (protection.outlook.com: fb.com does not designate permitted sender hosts) X-MS-Exchange-SenderADCheck: 1 X-Microsoft-Antispam: BCL:0; X-Microsoft-Antispam-Message-Info: MjNBjPRbna9uO3/wPb5Zp/k2i94mOgR9h5XdZpb8+95Cfz6E8I+bXpSsOuGzc1hGHWZxnc2NQHZ0ZNiwJui4JNILS6SRM8CHOmU5XlrZq2rYx8t3MEKCFFzDYSH2+XVKomfC5lRPogkmF3ercImcVLBwZSLIeUgw4yZcCWZXVpf0+PoDnsJOZCcA3OlMI7ffSQoCau5BoqiAFYMFraTxNeXHkVmx4jwF2RcCfPlcLPexkzgJG9ym+n2ljtbvMPFR22CY6A8nWyc3OpY1GPpPKXcXYkrCxcqrS7EvBfr8PLoLNpiR7xAXHZUZAqlmrx/SS1v7Vg1wJkgGsUSJiAa/XE1TOYSv30iGdMspjNU+tErt4qUEKpEOJ1KfzlhC8hKMVV7KbYkRfoYPOTDpO7SML0bID/ksAcoLQT9wKjRoQLZfczd2O8SzwsekN1f9YxT67Hq1xqwTIUV5j9+0i99GpRcr4MZFXzo4bA5ClXzt2/pcJxdIGUDLsshIf8xX/5uOK9counVCoFPHBO2gATYhqjefh/JAzDkctWvdpW4V0vyQGZbdi9uZ92fpNL/cLEYO X-MS-Exchange-AntiSpam-MessageData: QVMr8mFR3Wzw1bkjr9D7QB6PbOSjUI/7AP+qoR1XROe7CSkBZLu+Fo3B8TICrixLvNwd2Q0+/1AkYDC9ktf6/tbOrnsQlH4jNLNV7US2NQ/pYl8hVLdqIR9zshckWIpB7UfEMRrMpMjrCutHk/0xBM0GUGqurvmKeaJl5sUVr8E= X-MS-Exchange-CrossTenant-Network-Message-Id: 8ed90bfa-d75f-4b44-395a-08d7ae921b9d X-MS-Exchange-CrossTenant-OriginalArrivalTime: 11 Feb 2020 01:31:24.9750 (UTC) X-MS-Exchange-CrossTenant-FromEntityHeader: Hosted X-MS-Exchange-CrossTenant-Id: 8ae927fe-1255-47a7-a2af-5f3a069daaa2 X-MS-Exchange-CrossTenant-MailboxType: HOSTED X-MS-Exchange-CrossTenant-UserPrincipalName: VMmcPmn8mlv9PMjyh2NHNleCmZcXnv5wn0l41LHYOTJ6QloZZGJe0m4tI7ZiYwyv X-MS-Exchange-Transport-CrossTenantHeadersStamped: BYAPR15MB3286 X-OriginatorOrg: fb.com X-Proofpoint-Virus-Version: vendor=fsecure engine=2.50.10434:6.0.138,18.0.572 definitions=2020-02-10_08:2020-02-10,2020-02-10 signatures=0 X-Proofpoint-Spam-Details: rule=fb_default_notspam policy=fb_default score=0 suspectscore=0 phishscore=0 priorityscore=1501 adultscore=0 mlxlogscore=999 lowpriorityscore=0 mlxscore=0 bulkscore=0 spamscore=0 clxscore=1015 impostorscore=0 malwarescore=0 classifier=spam adjust=0 reason=mlx scancount=1 engine=8.12.0-2001150001 definitions=main-2002110006 X-FB-Internal: deliver X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Feb 11, 2020 at 10:55:53AM +1100, Gavin Shan wrote: > Hi Roman, > > On 2/11/20 3:17 AM, Roman Gushchin wrote: > > Hello, Gavin! > > > > On Mon, Feb 10, 2020 at 11:14:45PM +1100, Gavin Shan wrote: > > > commit 68600f623d69 ("mm: don't miss the last page because of round-off > > > error") makes the scan size round up to @denominator regardless of the > > > memory cgroup's state, online or offline. This affects the overall > > > reclaiming behavior: The corresponding LRU list is eligible for reclaiming > > > only when its size logically right shifted by @sc->priority is bigger than > > > zero in the former formula (non-roundup one). > > > > Not sure I fully understand, but wasn't it so before 68600f623d69 too? > > > > It's correct that "(non-roundup one)" is typo and should have been dropped. > Will be corrected in v2 if needed. Thanks! > > > > For example, the inactive > > > anonymous LRU list should have at least 0x4000 pages to be eligible for > > > reclaiming when we have 60/12 for swappiness/priority and without taking > > > scan/rotation ratio into account. After the roundup is applied, the > > > inactive anonymous LRU list becomes eligible for reclaiming when its > > > size is bigger than or equal to 0x1000 in the same condition. > > > > > > (0x4000 >> 12) * 60 / (60 + 140 + 1) = 1 > > > ((0x1000 >> 12) * 60) + 200) / (60 + 140 + 1) = 1 > > > > > > aarch64 has 512MB huge page size when the base page size is 64KB. The > > > memory cgroup that has a huge page is always eligible for reclaiming in > > > that case. The reclaiming is likely to stop after the huge page is > > > reclaimed, meaing the subsequent @sc->priority and memory cgroups will be > > > skipped. It changes the overall reclaiming behavior. This fixes the issue > > > by applying the roundup to offlined memory cgroups only, to give more > > > preference to reclaim memory from offlined memory cgroup. It sounds > > > reasonable as those memory is likely to be useless. > > > > So is the problem that relatively small memory cgroups are getting reclaimed > > on default prio, however before they were skipped? > > > > Yes, you're correct. There are two dimensions for global reclaim: priority > (sc->priority) and memory cgroup. The scan/reclaim is carried out by iterating > from these two dimensions until the reclaimed pages are enough. If the roundup > is applied to current memory cgroup and occasionally helps to reclaim enough > memory, the subsequent priority and memory cgroup will be skipped. > > > > > > > The issue was found by starting up 8 VMs on a Ampere Mustang machine, > > > which has 8 CPUs and 16 GB memory. Each VM is given with 2 vCPUs and 2GB > > > memory. 784MB swap space is consumed after these 8 VMs are completely up. > > > Note that KSM is disable while THP is enabled in the testing. With this > > > applied, the consumed swap space decreased to 60MB. > > > > > > total used free shared buff/cache available > > > Mem: 16196 10065 2049 16 4081 3749 > > > Swap: 8175 784 7391 > > > total used free shared buff/cache available > > > Mem: 16196 11324 3656 24 1215 2936 > > > Swap: 8175 60 8115 > > > > Does it lead to any performance regressions? Or it's only about increased > > swap usage? > > > > Apart from swap usage, it also had performance downgrade for my case. With > your patch (68600f623d69) included, it took 264 seconds to bring up 8 VMs. > However, 236 seconds are used to do same thing with my patch applied on top > of yours. There is 10% performance downgrade. It's the reason why I had a > stable tag. I see... > > > > > > > Fixes: 68600f623d69 ("mm: don't miss the last page because of round-off error") > > > Cc: # v4.20+ > > > Signed-off-by: Gavin Shan > > > --- > > > mm/vmscan.c | 9 ++++++--- > > > 1 file changed, 6 insertions(+), 3 deletions(-) > > > > > > diff --git a/mm/vmscan.c b/mm/vmscan.c > > > index c05eb9efec07..876370565455 100644 > > > --- a/mm/vmscan.c > > > +++ b/mm/vmscan.c > > > @@ -2415,10 +2415,13 @@ static void get_scan_count(struct lruvec *lruvec, struct scan_control *sc, > > > /* > > > * Scan types proportional to swappiness and > > > * their relative recent reclaim efficiency. > > > - * Make sure we don't miss the last page > > > - * because of a round-off error. > > > + * Make sure we don't miss the last page on > > > + * the offlined memory cgroups because of a > > > + * round-off error. > > > */ > > > - scan = DIV64_U64_ROUND_UP(scan * fraction[file], > > > + scan = mem_cgroup_online(memcg) ? > > > + div64_u64(scan * fraction[file], denominator) : > > > + DIV64_U64_ROUND_UP(scan * fraction[file], > > > denominator); > > > > It looks a bit strange to round up for offline and basically down for > > everything else. So maybe it's better to return to something like > > the very first version of the patch: > > https://urldefense.proofpoint.com/v2/url?u=https-3A__www.spinics.net_lists_kernel_msg2883146.html&d=DwIC-g&c=5VD0RTtNlTh3ycd41b3MUw&r=jJYgtDM7QT-W-Fz_d29HYQ&m=urGWFxpEgETD4pryLqIYaKdVUk1Munj_zLpJthvrreM&s=k2RDZGNcvb_Sia2tZwcMPZ79Mad5dw1oT8JdIy0rkGY&e= ? > > For memcg reclaim reasons we do care only about an edge case with few pages. > > > > But overall it's not obvious to me, why rounding up is worse than rounding down. > > Maybe we should average down but accumulate the reminder? > > Creating an implicit bias for small memory cgroups sounds groundless. > > > > I don't think v1 path works for me either. The logic in v1 isn't too much > different from commit 68600f623d69. v1 has selective roundup, but current > code is having a forced roundup. With 68600f623d69 reverted and your v1 > patch applied, it took 273 seconds to bring up 8 VMs and 1752MB swap is used. > It looks more worse than 68600f623d69. > > Yeah, it's not reasonable to have a bias on all memory cgroups regardless > their states. I do think it's still right to give bias to offlined memory > cgroups. I don't think so, it really depends on the workload. Imagine systemd restarting a service due to some update or with other arguments. Almost entire pagecache is relevant and can be reused by a new cgroup. > So the point is we need take care of the memory cgroup's state > and apply the bias to offlined ones only. The offlined memory cgroup is > going to die and has been dead. It's unlikely for its memory to be used > by someone, but still possible. So it's reasonable to hardly squeeze the > used memory of offlined memory cgroup if possible. Anyway, I think your version is good to mitigate the regression. So, please feel free to add Acked-by: Roman Gushchin But I think we need something more clever long-term: e.g. accumulate the leftover from the division and add it to the next calculation. If you can test such an approach on your workload, that would be nice. Thanks!