From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 2B5E8C77B7D for ; Thu, 18 May 2023 17:16:19 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 9B985900005; Thu, 18 May 2023 13:16:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 96A0C900003; Thu, 18 May 2023 13:16:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 83125900005; Thu, 18 May 2023 13:16:18 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0016.hostedemail.com [216.40.44.16]) by kanga.kvack.org (Postfix) with ESMTP id 6F9EE900003 for ; Thu, 18 May 2023 13:16:18 -0400 (EDT) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 30BD41A08FC for ; Thu, 18 May 2023 17:16:18 +0000 (UTC) X-FDA: 80804029236.20.DBC7A51 Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf19.hostedemail.com (Postfix) with ESMTP id 02C281A0013 for ; Thu, 18 May 2023 17:16:15 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=g+N7l4bQ; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf19.hostedemail.com: domain of song@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=song@kernel.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1684430176; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=JIuCOT65Zm7jU2KqBrdHn+xdU/MeZEypGnW1k7dHQy4=; b=sZiwGIEulUj09QPmQX2IUwUy8t9PgPvWXJ7D2tZtWp2FLpVvgnOnrAdNd0KVgz3f36nCX6 rJuXoCHK7DVzpnI59rcHXiO7IvQTHjtnvmyt9CAOJccuZeQu+3OOhW39hVH5jNKYn8c3mx DauYYU1tCl+cAKc4X5HxYuLfNA7t0as= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=kernel.org header.s=k20201202 header.b=g+N7l4bQ; dmarc=pass (policy=none) header.from=kernel.org; spf=pass (imf19.hostedemail.com: domain of song@kernel.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=song@kernel.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1684430176; a=rsa-sha256; cv=none; b=moZMeVmH5Hf3IHavGyM73CxIAJCIt6KsIhmuapS/7gdv9/XvqaTkNPJBGM4WtKfOzxrA/d Jb9SEQADx5pb6lOsCuhlcYA9aWrHbKroc0wsHBcw5v1d1OzBlzv0rpoBHdUmeRT2e6/MYH INUCIDzE19cO26OGljWDp0REn846zKU= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id D9EF265110 for ; Thu, 18 May 2023 17:16:14 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id BE76FC433A1 for ; Thu, 18 May 2023 17:16:13 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=kernel.org; s=k20201202; t=1684430173; bh=BicXuHAH0L0Nj1CK7fc4znKiUcAy3bHgVFwXS6rqpv8=; h=References:In-Reply-To:From:Date:Subject:To:Cc:From; b=g+N7l4bQMeSJN0riQz8JUa1UFMvnLPw7I8F5CtkcYIl1N/qZ3b+wJzcxyTm9sRmjb VJ/1NYk6XWhS0SXHSwO52p1p89b7VeIXH/Gkkr/4ltSEj7jOVzoqtxPwhcV1ZdvaGv w2ohtiEc1P0hq17iDppKdn/PkU7xNThfDLdSHKUG5aYJaH/obFB6DiuIz8B05tk/hF Lyo0aPt4/lV3JMkvu2rBeHWOGm9YGJl/jno0CedZpWkWUQpnM/wG3pb9J+WII4rDOu dsqhbMDu+myTLJeupN5foFRefVFoz65wN1k5f8QuP2EpfO2d+U7C+jT1wdxNNj0uOR O/LCwAC4FVWsg== Received: by mail-lj1-f176.google.com with SMTP id 38308e7fff4ca-2af1ae3a21fso7969491fa.0 for ; Thu, 18 May 2023 10:16:13 -0700 (PDT) X-Gm-Message-State: AC+VfDyDbnFrZv1AOD95JQlBkq4AHVWPU0b7l5ATlTYshn35zIJt6kUj 0wRzgF+MpW2uJHYqxPLWhH2fpdKTN1eeI28UTGg= X-Google-Smtp-Source: ACHHUZ5G/pNUXKf8HN8qep0pHvgI1/meZ1dwee9HVbTYCVYba+8igYppAwpm5kHbrxX3hnmw5giGIqyU1BcxOR8GW48= X-Received: by 2002:ac2:4a65:0:b0:4f1:3d7d:409e with SMTP id q5-20020ac24a65000000b004f13d7d409emr1540743lfp.0.1684430171727; Thu, 18 May 2023 10:16:11 -0700 (PDT) MIME-Version: 1.0 References: <20230308094106.227365-1-rppt@kernel.org> <20230308094106.227365-2-rppt@kernel.org> <20230518152354.GD4967@kernel.org> In-Reply-To: From: Song Liu Date: Thu, 18 May 2023 10:15:59 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [RFC PATCH 1/5] mm: intorduce __GFP_UNMAPPED and unmapped_alloc() To: Kent Overstreet Cc: Mike Rapoport , linux-mm@kvack.org, Andrew Morton , Dave Hansen , Peter Zijlstra , Rick Edgecombe , Thomas Gleixner , Vlastimil Babka , linux-kernel@vger.kernel.org, x86@kernel.org Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspamd-Queue-Id: 02C281A0013 X-Rspam-User: X-Rspamd-Server: rspam05 X-Stat-Signature: kiiybh4bey9ot18qb738e6frj8jezqx3 X-HE-Tag: 1684430175-593324 X-HE-Meta: U2FsdGVkX1+Jxt07oei79286yFHcqZRPyx59CpTtCBSZT2lb0BUoFcGXizGPZd3x/9cwhQY+ZtyHMdEfe+HMN+e35aF6K21ad8BDciRhaG0QRnCo4b/QFFQKBw8RRUUN2Vu8GmgZan8XFDZUiEIHPxxm9tjINHiEGvRL1/HvrhxPhXPnvFvJr8Q8mCoEnsQY8kGJ4cjLUNulTz6VBsFY/2kIDJfdMDZ2zPoQCOhgUqOW2DPRYxQK4yX+lzA97e/J2cAMuILyrRzJX0ANuy1j6CakKxctiPk5sWO42BrKTRt52gvRL803jOwKs9g6DMQTp1Y7Q+5rgRM3d2B6XbS0LXYjmwJT+rzKfpJIQNPipOckw1TeIWA2bSy4jn9mSFXZDxZUgHVCC6eUe2MRVGj35KronthYkD9ceCkgWy9PWPIbIrnUCOH9b3Z1dKZaxii/eiVAQkeMy7eyXux7WhYVmwF2ySP9P6Ww8/kQ2JhB00Ecrvq9eyJ2g8JYhqRpaa61CWybiTHErKaLkgdNCrnJm42+BINZJlSTisgKgStI1TTGNtNyGOv/n2LT4sot5iWk4jMuX8O/GGuS2wYO0u62mNeiLPvlOTt1lUMXeEEdF9/YHkAaexu3n6p+tap638D+RCt80sc/eMG7+dYMA6IOVFQecOPScN0MkDmJJMwOdDYOoEZl1QNRvJ4zI76kzq7G1vegzNBxsgdHWRbyn0YaqSSNWV0yf9Gbo6Ztkhz+6guwqKB+78xDoRxGuidJUx/kOpEc8sWSxfjHpgtz37U++Mg4IoLlr+Iluh9PAA+eTqyYR/Xym1gdxnsgSIEESb2BR91A3h3UyolW0RhSXLtFuOJvVhsHbf40YidEp6EX3f87HgXbEGQ+NS9KLGImOX0dJoQrDQoFG+BNfSJtLFTtZhmXplHsOgVR3RhCHGCjnD5Zxk2TGqZP0/B/10W41YVc24zWS9pzER6B+ue59F/ LBGAz7pm cc24RgI/r2zE6R21ZA0vAsAu1rfEO/QsE/WoC5GHlyaUWSmsbjemCxA6HgNRt2yEHRcPnCmKZMgj0pwltMcLNh3DFKk6qam5mAacnkk+uTUNL6qgAeEp9zXMLhO6yN48EGv9hOhyxeGg50BzG8Pc9YNsP4jO+SzIiskm5SETucYhNk78enpTRfHRbaeSXNvupkpLzBYW4AZYyvVr4yMOs/eAdH5iuwLgT3h46urautX1zBE7mFgWBWpEfC1u6RX9Rjj3E6blo/9qmSbePah5CnhwMwEa6HnIbWgzjnMsyWxMYTNir2Xg2LuGs4ewwcMi6h0V6SXgQgH+gBxlPsVDKKY0z+RtNw4ESu/iUu37VznwszCs= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, May 18, 2023 at 9:58=E2=80=AFAM Kent Overstreet wrote: > > On Thu, May 18, 2023 at 09:33:20AM -0700, Song Liu wrote: > > I am working on patches based on the discussion in [1]. I am planning t= o > > send v1 for review in a week or so. > > For reference, here's my own (early, but functioning :) slab allocator: > > Look forward to comparing! This looks similar to the bpf_prog_pack allocator we use for BPF at the moment. (search for bpf_prog_pack in kernel/bpf/core.c). I guess we can also use bpf_prog_pack for bcachefs for now. Thanks, Song > -->-- > From 6eeb6b8ef4271ea1a8d9cac7fbaeeb7704951976 Mon Sep 17 00:00:00 2001 > From: Kent Overstreet > Date: Wed, 17 May 2023 01:22:06 -0400 > Subject: [PATCH] mm: jit/text allocator > > This provides a new, very simple slab allocator for jit/text, i.e. bpf, > ftrace trampolines, or bcachefs unpack functions. > > With this API we can avoid ever mapping pages both writeable and > executable (not implemented in this patch: need to tweak > module_alloc()), and it also supports sub-page sized allocations. > > Signed-off-by: Kent Overstreet > > diff --git a/include/linux/jitalloc.h b/include/linux/jitalloc.h > new file mode 100644 > index 0000000000..f1549d60e8 > --- /dev/null > +++ b/include/linux/jitalloc.h > @@ -0,0 +1,9 @@ > +/* SPDX-License-Identifier: GPL-2.0 */ > +#ifndef _LINUX_JITALLOC_H > +#define _LINUX_JITALLOC_H > + > +void jit_update(void *buf, void *new_buf, size_t len); > +void jit_free(void *buf); > +void *jit_alloc(void *buf, size_t len); > + > +#endif /* _LINUX_JITALLOC_H */ > diff --git a/mm/Kconfig b/mm/Kconfig > index 4751031f3f..ff26a4f0c9 100644 > --- a/mm/Kconfig > +++ b/mm/Kconfig > @@ -1202,6 +1202,9 @@ config LRU_GEN_STATS > This option has a per-memcg and per-node memory overhead. > # } > > +config JITALLOC > + bool > + > source "mm/damon/Kconfig" > > endmenu > diff --git a/mm/Makefile b/mm/Makefile > index c03e1e5859..25e82db9e8 100644 > --- a/mm/Makefile > +++ b/mm/Makefile > @@ -138,3 +138,4 @@ obj-$(CONFIG_IO_MAPPING) +=3D io-mapping.o > obj-$(CONFIG_HAVE_BOOTMEM_INFO_NODE) +=3D bootmem_info.o > obj-$(CONFIG_GENERIC_IOREMAP) +=3D ioremap.o > obj-$(CONFIG_SHRINKER_DEBUG) +=3D shrinker_debug.o > +obj-$(CONFIG_JITALLOC) +=3D jitalloc.o > diff --git a/mm/jitalloc.c b/mm/jitalloc.c > new file mode 100644 > index 0000000000..7c4d621802 > --- /dev/null > +++ b/mm/jitalloc.c > @@ -0,0 +1,187 @@ > +// SPDX-License-Identifier: GPL-2.0 > + > +#include > +#include > +#include > +#include > +#include > +#include > +#include > +#include > + > +#include > + > +static DEFINE_MUTEX(jit_alloc_lock); > + > +struct jit_cache { > + unsigned obj_size_bits; > + unsigned objs_per_slab; > + struct list_head partial; > +}; > + > +#define JITALLOC_MIN_SIZE 16 > +#define NR_JIT_CACHES ilog2(PAGE_SIZE / JITALLOC_MIN_SIZE) > + > +static struct jit_cache jit_caches[NR_JIT_CACHES]; > + > +struct jit_slab { > + unsigned long __page_flags; > + > + struct jit_cache *cache; > + void *executably_mapped;; > + unsigned long *objs_allocated; /* bitmap of free object= s */ > + struct list_head list; > +}; > + > +#define folio_jit_slab(folio) (_Generic((folio), = \ > + const struct folio *: (const struct jit_slab *)(folio),= \ > + struct folio *: (struct jit_slab *)(folio))) > + > +#define jit_slab_folio(s) (_Generic((s), = \ > + const struct jit_slab *: (const struct folio *)s, = \ > + struct jit_slab *: (struct folio *)s)) > + > +static struct jit_slab *jit_slab_alloc(struct jit_cache *cache) > +{ > + void *executably_mapped =3D module_alloc(PAGE_SIZE); > + struct page *page; > + struct folio *folio; > + struct jit_slab *slab; > + unsigned long *objs_allocated; > + > + if (!executably_mapped) > + return NULL; > + > + objs_allocated =3D kcalloc(BITS_TO_LONGS(cache->objs_per_slab), s= izeof(unsigned long), GFP_KERNEL); > + if (!objs_allocated ) { > + vfree(executably_mapped); > + return NULL; > + } > + > + set_vm_flush_reset_perms(executably_mapped); > + set_memory_rox((unsigned long) executably_mapped, 1); > + > + page =3D vmalloc_to_page(executably_mapped); > + folio =3D page_folio(page); > + > + __folio_set_slab(folio); > + slab =3D folio_jit_slab(folio); > + slab->cache =3D cache; > + slab->executably_mapped =3D executably_mapped; > + slab->objs_allocated =3D objs_allocated; > + INIT_LIST_HEAD(&slab->list); > + > + return slab; > +} > + > +static void *jit_cache_alloc(void *buf, size_t len, struct jit_cache *ca= che) > +{ > + struct jit_slab *s =3D > + list_first_entry_or_null(&cache->partial, struct jit_slab= , list) ?: > + jit_slab_alloc(cache); > + unsigned obj_idx, nr_allocated; > + > + if (!s) > + return NULL; > + > + obj_idx =3D find_first_zero_bit(s->objs_allocated, cache->objs_pe= r_slab); > + > + BUG_ON(obj_idx >=3D cache->objs_per_slab); > + __set_bit(obj_idx, s->objs_allocated); > + > + nr_allocated =3D bitmap_weight(s->objs_allocated, s->cache->objs_= per_slab); > + > + if (nr_allocated =3D=3D s->cache->objs_per_slab) { > + list_del_init(&s->list); > + } else if (nr_allocated =3D=3D 1) { > + list_del(&s->list); > + list_add(&s->list, &s->cache->partial); > + } > + > + return s->executably_mapped + (obj_idx << cache->obj_size_bits); > +} > + > +void jit_update(void *buf, void *new_buf, size_t len) > +{ > + text_poke_copy(buf, new_buf, len); > +} > +EXPORT_SYMBOL_GPL(jit_update); > + > +void jit_free(void *buf) > +{ > + struct page *page; > + struct folio *folio; > + struct jit_slab *s; > + unsigned obj_idx, nr_allocated; > + size_t offset; > + > + if (!buf) > + return; > + > + page =3D vmalloc_to_page(buf); > + folio =3D page_folio(page); > + offset =3D offset_in_folio(folio, buf); > + > + if (!folio_test_slab(folio)) { > + vfree(buf); > + return; > + } > + > + s =3D folio_jit_slab(folio); > + > + mutex_lock(&jit_alloc_lock); > + obj_idx =3D offset >> s->cache->obj_size_bits; > + > + __clear_bit(obj_idx, s->objs_allocated); > + > + nr_allocated =3D bitmap_weight(s->objs_allocated, s->cache->objs_= per_slab); > + > + if (nr_allocated =3D=3D 0) { > + list_del(&s->list); > + kfree(s->objs_allocated); > + folio_put(folio); > + } else if (nr_allocated + 1 =3D=3D s->cache->objs_per_slab) { > + list_del(&s->list); > + list_add(&s->list, &s->cache->partial); > + } > + > + mutex_unlock(&jit_alloc_lock); > +} > +EXPORT_SYMBOL_GPL(jit_free); > + > +void *jit_alloc(void *buf, size_t len) > +{ > + unsigned jit_cache_idx =3D ilog2(roundup_pow_of_two(len) / 16); > + void *p; > + > + if (jit_cache_idx < NR_JIT_CACHES) { > + mutex_lock(&jit_alloc_lock); > + p =3D jit_cache_alloc(buf, len, &jit_caches[jit_cache_idx= ]); > + mutex_unlock(&jit_alloc_lock); > + } else { > + p =3D module_alloc(len); > + if (p) { > + set_vm_flush_reset_perms(p); > + set_memory_rox((unsigned long) p, DIV_ROUND_UP(le= n, PAGE_SIZE)); > + } > + } > + > + if (p && buf) > + jit_update(p, buf, len); > + > + return p; > +} > +EXPORT_SYMBOL_GPL(jit_alloc); > + > +static int __init jit_alloc_init(void) > +{ > + for (unsigned i =3D 0; i < ARRAY_SIZE(jit_caches); i++) { > + jit_caches[i].obj_size_bits =3D ilog2(JITALLOC_MIN_SI= ZE) + i; > + jit_caches[i].objs_per_slab =3D PAGE_SIZE >> jit_cach= es[i].obj_size_bits; > + > + INIT_LIST_HEAD(&jit_caches[i].partial); > + } > + > + return 0; > +} > +core_initcall(jit_alloc_init); >