From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id B99B6C7EE23 for ; Tue, 23 May 2023 18:40:14 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 10A2D900003; Tue, 23 May 2023 14:40:14 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0BA93900002; Tue, 23 May 2023 14:40:14 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E9D97900003; Tue, 23 May 2023 14:40:13 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0013.hostedemail.com [216.40.44.13]) by kanga.kvack.org (Postfix) with ESMTP id D58AD900002 for ; Tue, 23 May 2023 14:40:13 -0400 (EDT) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 8637B1407B8 for ; Tue, 23 May 2023 18:40:13 +0000 (UTC) X-FDA: 80822384706.15.453ABD3 Received: from galois.linutronix.de (Galois.linutronix.de [193.142.43.55]) by imf29.hostedemail.com (Postfix) with ESMTP id 8EA05120026 for ; Tue, 23 May 2023 18:40:11 +0000 (UTC) Authentication-Results: imf29.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b=to6mfsuK; dkim=pass header.d=linutronix.de header.s=2020e header.b=uiOoO4iR; spf=pass (imf29.hostedemail.com: domain of tglx@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=tglx@linutronix.de; dmarc=pass (policy=none) header.from=linutronix.de ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684867212; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=iw9JRBN6ov8kZsawNlaeZLyfKYqzxHmDYNyHCKPcp4E=; b=nvJNiulXcTUyv/IyZag2hlLtTlB0FLZy60QLcrI6VJCm7uI1/gcVI7bKHyEjKuT9MQTi8W 1EV3eYmEneaOVigkVjXkl+nw5+F0Ahdi7GCXkuDLOQZxnP/dApLonLn53zPk1OIJ57jCYC qv4oX64sHbuEBXUHBZyLINwLY8TP3mo= ARC-Authentication-Results: i=1; imf29.hostedemail.com; dkim=pass header.d=linutronix.de header.s=2020 header.b=to6mfsuK; dkim=pass header.d=linutronix.de header.s=2020e header.b=uiOoO4iR; spf=pass (imf29.hostedemail.com: domain of tglx@linutronix.de designates 193.142.43.55 as permitted sender) smtp.mailfrom=tglx@linutronix.de; dmarc=pass (policy=none) header.from=linutronix.de ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684867212; a=rsa-sha256; cv=none; b=mty89p+XJuGsNtvfOlxPbRNRU8hqSw8V+yWdY7HbBBxBZAbQ2GjQ+H8gM2aiUL54suSQ58 N+rqelWxLmyAZDhJwYrUC4ke2zNWFKUxfgUSr+3EITfstEDkdsgdWJrX9b8X4ESFjYrPjB NdAQkyVaZOnTTo8pSVsakhv/KfyqYmk= From: Thomas Gleixner DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020; t=1684867209; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iw9JRBN6ov8kZsawNlaeZLyfKYqzxHmDYNyHCKPcp4E=; b=to6mfsuK3gDZIqdQgROzYs3xZvprTC2yVH0hywTnjTt0zoWWA2rDtH9u3fbIRoidsegcH4 ZLzxC8anTcUXXDJP1VL3uMq+jkTHWxjyGtcI80ItattV7ufUVO1PLBFPJNglo01K1V9WbK s9/QrzXFHlkTgu0ZH4/+6Saa9ojTUTqDJVn291/8KJjt1tA74n4yzkqbKINul7/2u+kZZg S6RFcnuLehozKkmCzXUoGSh5t6dtBxToWSyaDsxElcv4P8T1siKUtBYK2Et8x9nSKwLfyO mKtO6aLfW3pUc7zyLgIhSfG4zgZq+b11tXpYPUovX9tGFU1Z+Vv6SNl3x6TwLw== DKIM-Signature: v=1; a=ed25519-sha256; c=relaxed/relaxed; d=linutronix.de; s=2020e; t=1684867209; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=iw9JRBN6ov8kZsawNlaeZLyfKYqzxHmDYNyHCKPcp4E=; b=uiOoO4iRd4JoSny0YKCeNTQSmJSwybLuHPqeUv4klDyFB9Kv0oXPakudNyogXb8xZysxBb Vlruka8owqT6+gDg== To: Uladzislau Rezki Cc: linux-mm@kvack.org, Andrew Morton , Christoph Hellwig , Lorenzo Stoakes , Peter Zijlstra , Baoquan He Subject: Re: [patch 0/6] mm/vmalloc: Assorted fixes and improvements In-Reply-To: References: <20230523135902.517032811@linutronix.de> <878rdfuf54.ffs@tglx> <875y8jueup.ffs@tglx> Date: Tue, 23 May 2023 20:40:07 +0200 Message-ID: <87353mvqm0.ffs@tglx> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 8EA05120026 X-Rspam-User: X-Stat-Signature: fmme6de6ymquzhjaayp8uqndhxmgde3u X-Rspamd-Server: rspam01 X-HE-Tag: 1684867211-807 X-HE-Meta: U2FsdGVkX18ACnktKsstA5xtRzszv761xFeTyV/0+gsw1tKZd4Ey8a2tWFiE31wSd8Rp+hTQK4/+b3H7RRXNJsyj6FAR+SNs2xvUmflJl2ujV9TvRCh9iTmuKzfxf23WayPMjnAKrL3k7FWOMwzcyM3Ess8PzGQaQMc837TUKs8pRheZ7zG1+fDmGBYw5RU5oir6CaQccVtD1CNOr0WrgEQL/J+y1Fdinrj+7i/5nBturFP2O4o+KNOtKAUQCeXLmNK8adR8ySZxRhSgCloGB986lTjduhZuQDdwNtOv0vKwl77M6+G8FlNC30QLBnxFKNvJral6MDDMjgnMhV0Pb6thxhewbDHz1whriYnF6Ow5BbdlV6wCXDHOAEXuB3CKNXuESab80dFxTPmso4vLilQSuQB0bzYnaaCUToAJT+rMaw1eOPw79JmseoGng8/GlwjObSDjfOQ0YNyr3MXIU0wvMkbXLOxOVVn+kL4Tkbic7N1yIQuza6YlF+r88RHBisVOt3/MSOWL1vaYmGbp5rFFJMHB6FYGKc6RiuBy0H72V6A+UTrMoHbSDtb6MFjhNUO3h+WiuKo3dsBzsZa592p9BY88sZYjJwTv4DvqDfrGPApVkQ1HO+xoAzIEq7ELmE/JfI6VwTPfsp3/pyvP4DNEun3H1ANT5/Y2QdvFfUKthsD0YGpPE4e8hAJxyUNLMJZZKlOXJ2X+obXKY/1QOgGiGDcP9A/gDJV3mzQrG/u3aE3gR5qinVCVDJqcXwvbTIq1kg8fhwSzOGL2U/rqzYysgvz3eT9pkT+asO25zrt9p8hvt2h6tBnVR++nwgFq6CCds+WOwo+UVXpR3AS6RqwOwBELiZvXd8Mqyf8hb2ahhzVqfczW0I0N5ovi9qyI5ym/5hMDIbVD4OW+22FrzMk77cSkPQiLZuLuEEJlLKiMK14lxfxlEPu0uWOVm2bJ8IQBM7xrcl0OZWsbHAQ QBh1+gLW PlTeFQV5hergU4oRJkmv10vSVEzpCTujyRNf0E/zwCOe1rMo+YIXe2+8mx6R/nxWusHdKkibD297zOyMB6e2XWBckglR6pyOT/ylX X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, May 23 2023 at 19:55, Uladzislau Rezki wrote: > On Tue, May 23, 2023 at 07:48:09PM +0200, Uladzislau Rezki wrote: >> So, v6.3 does not contain that patch. I have to use the next instead. >>=20 > next-20230523: It works against linus tree just fine. > mm/vmalloc.c: In function =E2=80=98_vm_unmap_aliases=E2=80=99: > mm/vmalloc.c:2280:9: error: too few arguments to function =E2=80=98purge_= fragmented_block=E2=80=99 > 2280 | if (!purge_fragmented_block(vb, vbq, &purge_list) && > | ^~~~~~~~~~~~~~~~~~~~~~ > mm/vmalloc.c:2095:13: note: declared here > 2095 | static bool purge_fragmented_block(struct vmap_block *vb, struct = vmap_block_queue *vbq, > | ^~~~~~~~~~~~~~~~~~~~~~ > CC drivers/acpi/pmic/intel_pmic_bxtwc.o >=20=20 > > there is only one complain in fact. That's the one where I failed to pull the refreshed patch back from the test machine. --- Subject: mm/vmalloc: Don't purge usable blocks unnecessarily From: Thomas Gleixner Date: Mon, 22 May 2023 19:38:32 +0200 Purging fragmented blocks is done unconditionally in several contexts: 1) From drain_vmap_area_work(), when the number of lazy to be freed vmap_areas reached the threshold 2) Reclaiming vmalloc address space from pcpu_get_vm_areas() 3) _unmap_aliases() #1 There is no reason to zap fragmented vmap blocks unconditionally, simply because reclaiming all lazy areas drains at least 32MB * fls(num_online_cpus()) per invocation which is plenty. #2 Reclaiming when running out of space or due to memory pressure makes a lot of sense #3 _unmap_aliases() requires to touch everything because the caller has no clue which vmap_area used a particular page last and the vmap_area lost that information too. Except for the vfree + VM_FLUSH_RESET_PERMS case, which removes the vmap area first and then cares about the flush. That in turn requires a full walk of _all_ vmap areas including the one which was just added to the purge list. But as this has to be flushed anyway this is an opportunity to combine outstanding TLB flushes and do the housekeeping of purging freed areas, but like #1 there is no real good reason to zap usable vmap blocks unconditionally. Add a @force_purge argument to the relevant functions and if not true only purge fragmented blocks which have less than 1/4 of their capacity left. Signed-off-by: Thomas Gleixner --- mm/vmalloc.c | 36 +++++++++++++++++++++++------------- 1 file changed, 23 insertions(+), 13 deletions(-) --- a/mm/vmalloc.c +++ b/mm/vmalloc.c @@ -791,7 +791,7 @@ get_subtree_max_size(struct rb_node *nod RB_DECLARE_CALLBACKS_MAX(static, free_vmap_area_rb_augment_cb, struct vmap_area, rb_node, unsigned long, subtree_max_size, va_size) =20 -static void purge_vmap_area_lazy(void); +static void purge_vmap_area_lazy(bool force_purge); static BLOCKING_NOTIFIER_HEAD(vmap_notify_list); static void drain_vmap_area_work(struct work_struct *work); static DECLARE_WORK(drain_vmap_work, drain_vmap_area_work); @@ -1649,7 +1649,7 @@ static struct vmap_area *alloc_vmap_area =20 overflow: if (!purged) { - purge_vmap_area_lazy(); + purge_vmap_area_lazy(true); purged =3D 1; goto retry; } @@ -1717,7 +1717,7 @@ static atomic_long_t vmap_lazy_nr =3D ATOM static DEFINE_MUTEX(vmap_purge_lock); =20 /* for per-CPU blocks */ -static void purge_fragmented_blocks_allcpus(void); +static void purge_fragmented_blocks_allcpus(bool force_purge); =20 /* * Purges all lazily-freed vmap areas. @@ -1787,10 +1787,10 @@ static bool __purge_vmap_area_lazy(unsig /* * Kick off a purge of the outstanding lazy areas. */ -static void purge_vmap_area_lazy(void) +static void purge_vmap_area_lazy(bool force_purge) { mutex_lock(&vmap_purge_lock); - purge_fragmented_blocks_allcpus(); + purge_fragmented_blocks_allcpus(force_purge); __purge_vmap_area_lazy(ULONG_MAX, 0); mutex_unlock(&vmap_purge_lock); } @@ -1908,6 +1908,12 @@ static struct vmap_area *find_unlink_vma =20 #define VMAP_BLOCK_SIZE (VMAP_BBMAP_BITS * PAGE_SIZE) =20 +/* + * Purge threshold to prevent overeager purging of fragmented blocks for + * regular operations: Purge if vb->free is less than 1/4 of the capacity. + */ +#define VMAP_PURGE_THRESHOLD (VMAP_BBMAP_BITS / 4) + #define VMAP_RAM 0x1 /* indicates vm_map_ram area*/ #define VMAP_BLOCK 0x2 /* mark out the vmap_block sub-type*/ #define VMAP_FLAGS_MASK 0x3 @@ -2087,12 +2093,16 @@ static void free_vmap_block(struct vmap_ } =20 static bool purge_fragmented_block(struct vmap_block *vb, struct vmap_bloc= k_queue *vbq, - struct list_head *purge_list) + struct list_head *purge_list, bool force_purge) { if (!(vb->free + vb->dirty =3D=3D VMAP_BBMAP_BITS && vb->dirty !=3D VMAP_= BBMAP_BITS)) return false; =20 - /* prevent further allocs after releasing lock */ + /* Don't overeagerly purge usable blocks unless requested */ + if (!force_purge && vb->free < VMAP_PURGE_THRESHOLD) + return false; + + /* prevent further allocs after releasing lock */ WRITE_ONCE(vb->free, 0); /* prevent purging it again */ WRITE_ONCE(vb->dirty, VMAP_BBMAP_BITS); @@ -2115,7 +2125,7 @@ static void free_purged_blocks(struct li } } =20 -static void purge_fragmented_blocks(int cpu) +static void purge_fragmented_blocks(int cpu, bool force_purge) { LIST_HEAD(purge); struct vmap_block *vb; @@ -2130,19 +2140,19 @@ static void purge_fragmented_blocks(int continue; =20 spin_lock(&vb->lock); - purge_fragmented_block(vb, vbq, &purge); + purge_fragmented_block(vb, vbq, &purge, force_purge); spin_unlock(&vb->lock); } rcu_read_unlock(); free_purged_blocks(&purge); } =20 -static void purge_fragmented_blocks_allcpus(void) +static void purge_fragmented_blocks_allcpus(bool force_purge) { int cpu; =20 for_each_possible_cpu(cpu) - purge_fragmented_blocks(cpu); + purge_fragmented_blocks(cpu, force_purge); } =20 static void *vb_alloc(unsigned long size, gfp_t gfp_mask) @@ -2267,7 +2277,7 @@ static void _vm_unmap_aliases(unsigned l * not purgeable, check whether there is dirty * space to be flushed. */ - if (!purge_fragmented_block(vb, vbq, &purge_list) && + if (!purge_fragmented_block(vb, vbq, &purge_list, false) && vb->dirty_max && vb->dirty !=3D VMAP_BBMAP_BITS) { unsigned long va_start =3D vb->va->va_start; unsigned long s, e; @@ -4173,7 +4183,7 @@ struct vm_struct **pcpu_get_vm_areas(con overflow: spin_unlock(&free_vmap_area_lock); if (!purged) { - purge_vmap_area_lazy(); + purge_vmap_area_lazy(true); purged =3D true; =20 /* Before "retry", check if we recover. */