From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 211D5C3DA6E for ; Wed, 3 Jan 2024 21:30:45 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 7BD386B03B3; Wed, 3 Jan 2024 16:30:45 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 76CE46B03B4; Wed, 3 Jan 2024 16:30:45 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 634396B03B5; Wed, 3 Jan 2024 16:30:45 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id 542416B03B3 for ; Wed, 3 Jan 2024 16:30:45 -0500 (EST) Received: from smtpin08.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay03.hostedemail.com (Postfix) with ESMTP id 2960CA0914 for ; Wed, 3 Jan 2024 21:30:45 +0000 (UTC) X-FDA: 81639294450.08.CD9B503 Received: from mail-ed1-f44.google.com (mail-ed1-f44.google.com [209.85.208.44]) by imf04.hostedemail.com (Postfix) with ESMTP id 4137240022 for ; Wed, 3 Jan 2024 21:30:42 +0000 (UTC) Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=gooddata.com header.s=google header.b=Z79726hT; spf=pass (imf04.hostedemail.com: domain of jaroslav.pulchart@gooddata.com designates 209.85.208.44 as permitted sender) smtp.mailfrom=jaroslav.pulchart@gooddata.com; dmarc=pass (policy=none) header.from=gooddata.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1704317443; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=j6sqxnRsU2JjXEjn1CXukwdIlAoQXeCYFzpgQAvshx4=; b=TPnDdvrQdVGb4RQgPS2Srqvn62SsOkP/n221AvD0/HadKOiV3kx+GcijcqH+Qhnym2nS5M /lR0lp1WQjQ4m415X/V1bYJFTSTZozM2DIz7hyvH2Dh43luUWncuErvVKw/Wonw6bDAE82 ID5Od4Xkg7yk3Kwlp9yDKqmlrYPhaeM= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=gooddata.com header.s=google header.b=Z79726hT; spf=pass (imf04.hostedemail.com: domain of jaroslav.pulchart@gooddata.com designates 209.85.208.44 as permitted sender) smtp.mailfrom=jaroslav.pulchart@gooddata.com; dmarc=pass (policy=none) header.from=gooddata.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1704317443; a=rsa-sha256; cv=none; b=7tIOUDQ/j++bM5D8Z1pjXm8624Rd9Jr5DNJc/qjgt4yVupS3VpjHYuot+dk7cDdaWRCN71 5vLCm7/tihmyr+U2o2rc48SnOfQ+YRKid+NjNZ47gfgAiFBHtvI7RvqnYqWZ249P0pk3AT 15MlxZEFYnhcrJWzvZHOsSnWasZAXFQ= Received: by mail-ed1-f44.google.com with SMTP id 4fb4d7f45d1cf-5565b66e9c5so2781847a12.3 for ; Wed, 03 Jan 2024 13:30:42 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gooddata.com; s=google; t=1704317441; x=1704922241; darn=kvack.org; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:from:to:cc:subject:date:message-id:reply-to; bh=j6sqxnRsU2JjXEjn1CXukwdIlAoQXeCYFzpgQAvshx4=; b=Z79726hT/exVE+EBjurEwfsVvyrRTVx6VxLD9kd4M6Rh42A7JdInF7/vYPP2rEqfAb x0KhyfyD0J6PTG2Izv2VawMQ8H0QArh2bRUuQQnDgUrdHi/5JvzMJ3s9kuXGwoWxIxV7 wu6uyHwFgf3qoXbRpgk+xN/wcYaEG6Hfp2iUs= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1704317441; x=1704922241; h=cc:to:subject:message-id:date:from:in-reply-to:references :mime-version:x-gm-message-state:from:to:cc:subject:date:message-id :reply-to; bh=j6sqxnRsU2JjXEjn1CXukwdIlAoQXeCYFzpgQAvshx4=; b=bkJGgU1SToqPVEqNGXG1h/gPxVGzzWokhP7X+c9DU6vncW1ZSocsWS5a9A6NJ4nZN2 hbe8vyXMwEUSlhxNlMhvyiuuX33VcTPOGIbH+gU0tzaqf+1iP4JiI/lVm+N7xKiv/rlh VgULgPA290DRdOCgcFoTVTo1chdu7aP8GYWAPAlpPoSCMuu816yGFFUUDoH0AuLAdTJZ PMdFr6WkhsYBmfFJHGSW6WoG2P3vCVy0UgidqALuIHLYILNyTjWqj8CXfCv3sNdCFnM+ 9wLRAETpjFyr5WUV6jk6ZeHGKEjWNZxhGddQQhdX0EOGpGOhKLUBNhUoZqu7iPlREWFu yO1Q== X-Gm-Message-State: AOJu0Yxm5dssnm4lAFBDDfvkZp9yVFVbw/TY7YmygbBLzu2Yq96R1lQP bCOvkrqgrKn6V7jFOJSZsmnm+rQyKFNBYvYUWt4rswUDTkM+ X-Google-Smtp-Source: AGHT+IFhzHp+xHfSBlhvyBXbvBTlZalwie/CZ67BDJLjQDzeP28bHSeJi7chyvKq/5KWpV5ZhEkB7mDIsrBCtkV0tZw= X-Received: by 2002:a17:906:1099:b0:a27:462e:e356 with SMTP id u25-20020a170906109900b00a27462ee356mr4749487eju.72.1704317441633; Wed, 03 Jan 2024 13:30:41 -0800 (PST) MIME-Version: 1.0 References: <7df7e478-bd93-03df-5b10-19308f416e95@quicinc.com> In-Reply-To: From: Jaroslav Pulchart Date: Wed, 3 Jan 2024 22:30:14 +0100 Message-ID: Subject: Re: high kswapd CPU usage with symmetrical swap in/out pattern with multi-gen LRU To: Yu Zhao Cc: Daniel Secik , Charan Teja Kalla , Igor Raits , Kalesh Singh , akpm@linux-foundation.org, linux-mm@kvack.org Content-Type: text/plain; charset="UTF-8" X-Rspamd-Queue-Id: 4137240022 X-Rspam-User: X-Stat-Signature: k9a1n5ab71tu85gkycq13b4pyauu9myk X-Rspamd-Server: rspam01 X-HE-Tag: 1704317442-727425 X-HE-Meta: U2FsdGVkX1+7IEZfOOAutmWuPL1ugLRowSs7NRYPMyofX6L0iAKBldJDrr3sfxLg3ybZLPxVIFjxKstVZ+Z9mV8Cdq/D45VX3OCvmFGkzBVj91opBtgARzVud8k/nrlKV2RTEVfnH2Xte9f4GRgIoI3SpVYOCUs8B921ZCZskGzZVjCMz9PHwjwHGCYZHccY52Gdf736e0F3puxQCJp5WYDM2rX2Hb0VIUvaHJVda2xsYGzetyPJLurW2UB+B7xMFtjVGciH8vbhNSniGVoDvOAuIUGrOKt7yvEzgsn6Sdm4FtlVCP9KcR5qHOJZykr3KR3CQt5LlBA3nUHLmcDPEdG7YCGxYy4vG8D6PGq5il5Z/Eao01QB1oim8jiX5wxNdjaR2D7+RTbeTG7WljEemN3e/A7621OswOVCsk3rdm1utk42mPe10j3eozkXKvHsK4pF1zbeSDfAdtXD4lbTGQnT3vp3oD+xN2yTawB3QkH9hlOIWQplznamcx8PJ+IyXV8Au4Tr61VBtSsOWymxvZILbvGm2eVsqSRoIKeFy728ZzOjauaxvJz+EJ0tp/0Qdla0WPr7yx8/1E1nz8gl/rdk5otc+ylOFEwXw3MqNz3FfQLLFfE5hXNtLIBlU+FgHcgMjw3d6iPNItCo+xP1i09pXmUp8WTVRVHs4B6IoqU0DH1NbEKYSFUjKKn1lXoNvfpfzZWpKca4Qqbm+s3qzNk+DAixhiHGzx0QmhUo3cQ0BUo8QkxVZ6A0SEg8vrFzeiMCKRvss52wJqkU6KJSzYkeOfMPztZvUff0GBeB/5XSaA2VK7TAq6un/4osUfZexkQXoFsbCN2QUFlVjSyEkT2g1/8TAFpM5y8ebiP07pXZcyR92W10HdPr/NtIP7s7ftxaUg3QWRxURFMyUtCXYGF2R27ULNBleH0mROVmnVACmniR2LIRf/fCRUtMIjBzAPHTHW9t2tjbJ42xLkc AlZmpDI1 JLspqq8AAg60hd9vYJ+fm9SKFqDjiT69ovt2t7TjwLgOF0LOCyBLW9dlWOhcc1W9u+gM+EdjN0whG5B8nhakJ6zUGJVP/KYJKhDFPFowUYJeYH+08EeeJ4g719KQF65sjc53nrppQtK5VQsk= X-Bogosity: Ham, tests=bogofilter, spamicity=0.024321, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: > > > > > Hi yu, > > > > On 12/2/2023 5:22 AM, Yu Zhao wrote: > > > Charan, does the fix previously attached seem acceptable to you? Any > > > additional feedback? Thanks. > > > > First, thanks for taking this patch to upstream. > > > > A comment in code snippet is checking just 'high wmark' pages might > > succeed here but can fail in the immediate kswapd sleep, see > > prepare_kswapd_sleep(). This can show up into the increased > > KSWAPD_HIGH_WMARK_HIT_QUICKLY, thus unnecessary kswapd run time. > > @Jaroslav: Have you observed something like above? > > I do not see any unnecessary kswapd run time, on the contrary it is > fixing the kswapd continuous run issue. > > > > > So, in downstream, we have something like for zone_watermark_ok(): > > unsigned long size = wmark_pages(zone, mark) + MIN_LRU_BATCH << 2; > > > > Hard to convince of this 'MIN_LRU_BATCH << 2' empirical value, may be we > > should atleast use the 'MIN_LRU_BATCH' with the mentioned reasoning, is > > what all I can say for this patch. > > > > + mark = sysctl_numa_balancing_mode & NUMA_BALANCING_MEMORY_TIERING ? > > + WMARK_PROMO : WMARK_HIGH; > > + for (i = 0; i <= sc->reclaim_idx; i++) { > > + struct zone *zone = lruvec_pgdat(lruvec)->node_zones + i; > > + unsigned long size = wmark_pages(zone, mark); > > + > > + if (managed_zone(zone) && > > + !zone_watermark_ok(zone, sc->order, size, sc->reclaim_idx, 0)) > > + return false; > > + } > > > > > > Thanks, > > Charan > > > > -- > Jaroslav Pulchart > Sr. Principal SW Engineer > GoodData Hello, today we try to update servers to 6.6.9 which contains the mglru fixes (from 6.6.8) and the server behaves much much worse. I got multiple kswapd* load to ~100% imediatelly. 555 root 20 0 0 0 0 R 99.7 0.0 4:32.86 kswapd1 554 root 20 0 0 0 0 R 99.3 0.0 3:57.76 kswapd0 556 root 20 0 0 0 0 R 97.7 0.0 3:42.27 kswapd2 are the changes in upstream different compared to the initial patch which I tested? Best regards, Jaroslav Pulchart