From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 559FAC001B0 for ; Thu, 13 Jul 2023 08:58:41 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 4A07A8D0002; Thu, 13 Jul 2023 04:58:40 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 44F588D0001; Thu, 13 Jul 2023 04:58:40 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 318078D0002; Thu, 13 Jul 2023 04:58:40 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 1B7D18D0001 for ; Thu, 13 Jul 2023 04:58:40 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id E56CAC0182 for ; Thu, 13 Jul 2023 08:58:39 +0000 (UTC) X-FDA: 81005987958.15.CA1F86B Received: from mga18.intel.com (mga18.intel.com [134.134.136.126]) by imf28.hostedemail.com (Postfix) with ESMTP id 9833BC0008 for ; Thu, 13 Jul 2023 08:58:37 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Plys1xbY; spf=pass (imf28.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1689238717; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=5n7Tbx5Iw+n8YFHN4RbMCgopMFCeTx09BmlJbQl7Wxg=; b=ZdVGbnVu130MTDySfXxk9MvhUMAHH3Bce4qaJL2ABzS/PdciyNPrun3Z0FIdzCvH5+9hCV 6KHN37TXkWhoD6urum2ZOcf2mE5A6aNT5BEtJfDG3TZ46PtVEvswl3xPi7iCbNjG9CdANi fT0wWIwfrIp1W9vO/7u5K4vXxzTK20k= ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1689238717; a=rsa-sha256; cv=none; b=GkJ8qewA0P+bYe2jhccgb3hRGg4L7s7EMuKjin+AZc0CBS/kPy+ICXoycx0pnj/iUILLTZ EXvksDMPkEmVMWundeiMT9A1mEm/s0lDL5xi7YRQpW5dg3Mlj9UvDCPc+Ac+exuX0eXGjp XEWeLw+gfN8V1BiaSgGg/8VLZG6o35Y= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=intel.com header.s=Intel header.b=Plys1xbY; spf=pass (imf28.hostedemail.com: domain of ying.huang@intel.com designates 134.134.136.126 as permitted sender) smtp.mailfrom=ying.huang@intel.com; dmarc=pass (policy=none) header.from=intel.com DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=intel.com; i=@intel.com; q=dns/txt; s=Intel; t=1689238716; x=1720774716; h=from:to:cc:subject:references:date:in-reply-to: message-id:mime-version; bh=e1gg+ki8Lj41hSt6qxash9SMcqxtMlarpdfmZTvcSI0=; b=Plys1xbYe2Js2ERNeTLXsO0653m3aGzHqjlxO+XjUH6mktMGod9nU8ry 80qHASjhSzsJQbAbbUk5yfaYloxZ7KFky4AUO8Cj4zIztrHzxgo4S1/OJ Z476fD6Wm6YY6bAfZZ9zjdv6o9W0AM4HA5CEn50gv+SseY6su7IEUxLDS Q8AimcVpRsyGqzZ+JuzA0quFetEh2SyUDNzXpFJrjee78bHpZMg4FtsVb x/f9sm5Y5HlqFAebQV2FqdFaV7IvC7I+G4pxIRL1gLzCFzpCPjB3g5JIx TnGZdJyrcDFEQknbNPoalqFjcc4U56Rh0WOgSaW1WgdTZ1lLa50UosMCz g==; X-IronPort-AV: E=McAfee;i="6600,9927,10769"; a="349983486" X-IronPort-AV: E=Sophos;i="6.01,202,1684825200"; d="scan'208";a="349983486" Received: from orsmga005.jf.intel.com ([10.7.209.41]) by orsmga106.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Jul 2023 01:58:34 -0700 X-ExtLoop1: 1 X-IronPort-AV: E=McAfee;i="6600,9927,10769"; a="895931771" X-IronPort-AV: E=Sophos;i="6.01,202,1684825200"; d="scan'208";a="895931771" Received: from yhuang6-desk2.sh.intel.com (HELO yhuang6-desk2.ccr.corp.intel.com) ([10.238.208.55]) by orsmga005-auth.jf.intel.com with ESMTP/TLS/ECDHE-RSA-AES256-GCM-SHA384; 13 Jul 2023 01:58:31 -0700 From: "Huang, Ying" To: Mel Gorman Cc: Michal Hocko , , , Arjan Van De Ven , Andrew Morton , Vlastimil Babka , David Hildenbrand , Johannes Weiner , Dave Hansen , Pavel Tatashin , Matthew Wilcox Subject: Re: [RFC 2/2] mm: alloc/free depth based PCP high auto-tuning References: <20230710065325.290366-1-ying.huang@intel.com> <20230710065325.290366-3-ying.huang@intel.com> <20230712090526.thk2l7sbdcdsllfi@techsingularity.net> Date: Thu, 13 Jul 2023 16:56:54 +0800 In-Reply-To: <20230712090526.thk2l7sbdcdsllfi@techsingularity.net> (Mel Gorman's message of "Wed, 12 Jul 2023 10:05:26 +0100") Message-ID: <871qhcdwa1.fsf@yhuang6-desk2.ccr.corp.intel.com> User-Agent: Gnus/5.13 (Gnus v5.13) Emacs/28.2 (gnu/linux) MIME-Version: 1.0 Content-Type: text/plain; charset=ascii X-Rspamd-Queue-Id: 9833BC0008 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: eneqt1zga6skw89hudohjmchtierncfn X-HE-Tag: 1689238717-261755 X-HE-Meta: U2FsdGVkX19mVWCuf0q6tjRQUgyycAW9k9II6wqt9n8HK4Kmrf2azpz35UDDLgJ7MhjmIiXYGF5oJQKlqzKBzYm8mHE/UfOUlI8YNfuZZCof03l6bFDS/TGR7mCKyxFfKJiPuNhQv3aGgkvzEqh0+6slbwjDnVHG2Nq39syGM4rY71QFvColxtBMfb+Tm5LyV9hdr9gUxTbnsawGZ/qKcuzqKPdDwrjCSga0Y/WkiXkY+/Syz8ll5k6lbsLk77d2LZT5WxM/GIY3EPIxB8PN1ygil3NqRBSRdWO0Fop2FtXHY9mEI0K67gg3BGlbSXVIgPT2ls4gmZQ4zlq8u6Oqdoe865R5qICwsGdYhHPkycEx7FiX24VfUb5GRijqPf/oJqxY5XbW7xqdjfYaocrGnmqgg2VCFixRyRMa27nYql4E65bTRKPciP5WZxu7mhkYs1FJqum/wtijn/Qb8gbfNUg/0v+c075QDW5xPXFHRFFOBgymZmOZd1pjC8lr/17GDJYeDaAbWQqWJn8/7m5Vk8EDae5DzRa747GJf9CWFsB7v46/kBKbVWXTvKqhRgaqEEtgV9tU2IuRvQ07/QfHLgS69clSWTl93D5taTwRGNULtGwuzblZUPQkjc5KHga3vC1gdPqpDo7bzIV1BcVM6v/EsbXEZzpy1QZtPrbvbfvrAbkxPeG0efBJYjb3pmWzuXqjS4XZfF1WHuoodQi192qrxefpOJyS7osvXrEDTMAfFw6TufjN1YsGrEcIY7j8eXH9RCsjsLnY50mGRfwLmPldM32+5P9w9keBKYPj16qngJkDUlepM2A7n1I/xwNl5qaryiKhIPjLtDgHhH6ax36mayVsJ7U8k03cG6zJw2BdEb7kGEmSRu6K1tJTy6vq+eP6PQ0EtvZjc9tAqUc4CwNsuz0g/wjhftUVyqQkjXtqgtQLQBKlp1FM8YZUbS+oshUKAv98MCrA0VGJdY3 YbzUTlN3 c0qH+CltFHt0dwshhzSsDeQzDnKP7tWS72ZMLqNFjhqyQYRgkXp9m37sE3RnGmfmAXriPTMdb0WtfIVWDUmXfC2NJlllKjGNa85l9tXm7MXd3J4Us5z/oLRBZZZJre+anGBZ+cV2RKw7SkRpCOMmI//dXidAtUD6/bvRWcc4leugFFERrGRSxt3s4wkxwGOGVZUeqtJJ57daX42jzCj1IWbjXQ4xJXltdBODQ+LT6VKNCQe0F4eN+4jJSsw6eya0POeUyRwRYf4Vf5m0SS7B62oKeKSZGfbPtQGdY9ckPnio9Z/w= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Mel Gorman writes: > On Tue, Jul 11, 2023 at 01:19:46PM +0200, Michal Hocko wrote: >> On Mon 10-07-23 14:53:25, Huang Ying wrote: >> > To auto-tune PCP high for each CPU automatically, an >> > allocation/freeing depth based PCP high auto-tuning algorithm is >> > implemented in this patch. >> > >> > The basic idea behind the algorithm is to detect the repetitive >> > allocation and freeing pattern with short enough period (about 1 >> > second). The period needs to be short to respond to allocation and >> > freeing pattern changes quickly and control the memory wasted by >> > unnecessary caching. >> >> 1s is an ethernity from the allocation POV. Is a time based sampling >> really a good choice? I would have expected a natural allocation/freeing >> feedback mechanism. I.e. double the batch size when the batch is >> consumed and it requires to be refilled and shrink it under memory >> pressure (GFP_NOWAIT allocation fails) or when the surplus grows too >> high over batch (e.g. twice as much). Have you considered something as >> simple as that? >> Quite honestly I am not sure time based approach is a good choice >> because memory consumptions tends to be quite bulky (e.g. application >> starts or workload transitions based on requests). >> > > I tend to agree. Tuning based on the recent allocation pattern without > frees would make more sense and also be symmetric with how free_factor > works. This sounds good to me. I will give it a try to tune PCP batch. Have some questions about how to tune PCP high based on that. Details are in the following. > I suspect that time-based may be heavily orientated around the > will-it-scale benchmark. I totally agree that will-it-scale isn't a real workload. So, I tried to find some more practical ones. I found that a repetitive allocation/freeing several hundreds MB pages pattern exists in kernel building and netperf/SCTP_STREAM_MANY workloads. More details can be found in my reply to Michal as follows, https://lore.kernel.org/linux-mm/877cr4dydo.fsf@yhuang6-desk2.ccr.corp.intel.com/ > While I only glanced at this, a few things jumped out > > 1. Time-based heuristics are not ideal. congestion_wait() and > friends was an obvious case where time-based heuristics fell apart even > before the event it waited on was removed. For congestion, it happened to > work for slow storage for a while but that was about it. For allocation > stream detection, it has a similar problem. If a process is allocating > heavily, then fine, if it's in bursts of less than a second more than one > second apart then it will not adapt. While I do not think it is explicitly > mentioned anywhere, my understanding was that heuristics like this within > mm/ should be driven by explicit events as much as possible and not time. The proposed heuristics can be changed to be not time-based. When it is found that the allocation/freeing depth is larger than previous value, the PCP high can be increased immediately. We use time-based implementation to try to reduce the overhead. And, we mainly targeted long-time allocation pattern before. > 2. If time was to be used, it would be cheaper to have the simpliest possible > state tracking in the fast paths and decay any resizing of the PCP > within the vmstat updates (reuse pcp->expire except it applies to local > pcps). Even this is less than ideal as the PCP may be too large for short > periods of time but it may also act as a backstop for worst-case behaviour This sounds reasonable. Thanks! We will try this if we choose to use time-based implementation. > 3. free_factor is an existing mechanism for detecting recent patterns > and adapting the PCP sizes. The allocation side should be symmetric > and the events that should drive it are "refills" on the alloc side and > "drains" on the free side. Initially it might be easier to have a single > parameter that scales batch and high up to a limit For example, when a workload is started, several GB pages will be allocated. We will observe many "refills", and almost no "drains". So, we will scales batch and high up to a limit. When the workload exits, large number of pages of the workload will be put in PCP because the PCP high is increased. When should we decrease the PCP batch and high? > 4. The amount of state tracked seems excessive and increases the size of > the per-cpu structure by more than 1 cache line. That in itself may not > be a problem but the state is tracked on every page alloc/free that goes > through the fast path and it's relatively complex to track. That is > a constant penalty in fast paths that may not may not be relevant to the > workload and only sustained bursty allocation streams may offset the > cost. Yes. Thanks for pointing this out. We will optimize this if the other aspects of the basic idea could be accepted. > 5. Memory pressure and reclaim activity does not appear to be accounted > for and it's not clear if pcp->high is bounded or if it's possible for > a single PCP to hide a large number of pages from other CPUs sharing the > same node. The max size of the PCP should probably be explicitly clamped. As in my reply to Michal's email, previously I thought ZONE_RECLAIM_ACTIVE will be set for kswap reclaiming, and PCP high will be decreased to (batch * 4) effectively. Or I miss something? There's a upper limit for PCP high, which comes from MIN_PERCPU_PAGELIST_HIGH_FRACTION. That is, we calculate the PCP high as if percpu_pagelist_high_fraction is set to MIN_PERCPU_PAGELIST_HIGH_FRACTION (8). Then, we use that value as the upper limit for PCP high. Best Regards, Huang, Ying