From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3E27C00144 for ; Mon, 1 Aug 2022 14:37:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id B6D358E0001; Mon, 1 Aug 2022 10:37:50 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id B1C526B0072; Mon, 1 Aug 2022 10:37:50 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 9BCE38E0001; Mon, 1 Aug 2022 10:37:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id 8CE856B0071 for ; Mon, 1 Aug 2022 10:37:50 -0400 (EDT) Received: from smtpin11.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id 609CD1A0769 for ; Mon, 1 Aug 2022 14:37:50 +0000 (UTC) X-FDA: 79751277900.11.8ECABDB Received: from mail-pg1-f179.google.com (mail-pg1-f179.google.com [209.85.215.179]) by imf28.hostedemail.com (Postfix) with ESMTP id E63D6C00E2 for ; Mon, 1 Aug 2022 14:37:49 +0000 (UTC) Received: by mail-pg1-f179.google.com with SMTP id h132so9846597pgc.10 for ; Mon, 01 Aug 2022 07:37:49 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20210112; h=date:from:to:cc:subject:message-id:references:mime-version :content-disposition:in-reply-to; bh=4La+/TTqE7pmoTbyCn6Aak17r/OUWM6KiH/lhDYcZds=; b=UK3zY+SWxC3NVdhU/RyG7uoqtQEcFWVSKlN/ILHs2hpAcXCIu6KsnOe0fU2RPKKgbQ a164rGtltIJmtuHZ7aMkALsu4ZALa8ibcbyCLi8kXX1FvHSbffxAgjGGKOp2WCU9tzK6 xawkHda9pxd8+xRxGQzWwNuJxhCLIul7WJFwQumqTBeeCBxfQTWhE0mjogg9hyPx1daF X1PDNCCAbEXN69u0HGdnquCpibNSLnNzX9cNSQDPnNqM7Tp42fZIHwGdB9KoLhdwEAxC mEG2n1Rt1ZZGjOivISWk3HvwzFlv7B00thDs2bgR5AOOA5q56+I0eMEhn8BwNXGsMtdd kcXg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=4La+/TTqE7pmoTbyCn6Aak17r/OUWM6KiH/lhDYcZds=; b=fBbky0ALTzkXGr++Bk5ZmjSgRKWIjOdLTX/1sxS9GIx72nCEqpiJlpi61rG/iIWOFU pMCMpWoeJS6FHfzXy4SMk3lkA8LyCYteTBpAkNZ6E0k7WG8oqNxcwYKwOqYNfXGMyNkK 0shfMPBaUxOLRxtJNKaNyeKi9P63tzkWyXRw4YoHYMfrV6M7RJH/5y+T47Yz7hX/IsgR FKmGM5H3NRpNEvZiO4zhb0Vl+fcX9X3GaYCXYWCXON6PetgS6uOYGSHh2hN+HGuYUUZ+ KcyYbeDEBxdLkcellBj1oRVJTp//wL8VcddXBln5VOceXqNe9FTOTLcl9vcUTPPdPS/Y GV5Q== X-Gm-Message-State: AJIora9OH0EyneoW54E77p9/O+EZ+/3EPUiVft7bZ47cN3RJWBjT4i+T pjg3RZf18/lqCQsT7caxhXo= X-Google-Smtp-Source: AGRyM1sJFoxULt0qX+w1Z+xBlsAYIQQgLwcQ2Pbr5wjpQVXw0VRgWBUlQbmmACH8lew5/5Vkq9o4gw== X-Received: by 2002:a05:6a00:a8b:b0:4e1:52db:9e5c with SMTP id b11-20020a056a000a8b00b004e152db9e5cmr16391691pfl.38.1659364668567; Mon, 01 Aug 2022 07:37:48 -0700 (PDT) Received: from hyeyoo ([114.29.91.56]) by smtp.gmail.com with ESMTPSA id k5-20020a170902ce0500b0016bc947c5b7sm9707228plg.38.2022.08.01.07.37.44 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Mon, 01 Aug 2022 07:37:47 -0700 (PDT) Date: Mon, 1 Aug 2022 23:37:41 +0900 From: Hyeonggon Yoo <42.hyeyoo@gmail.com> To: Vlastimil Babka Cc: Christoph Lameter , Pekka Enberg , David Rientjes , Joonsoo Kim , Andrew Morton , Roman Gushchin , Joe Perches , Vasily Averin , Matthew WilCox , linux-kernel@vger.kernel.org, linux-mm@kvack.org Subject: Re: [PATCH v3 08/15] mm/slab_common: kmalloc_node: pass large requests to page allocator Message-ID: References: <20220712133946.307181-1-42.hyeyoo@gmail.com> <20220712133946.307181-9-42.hyeyoo@gmail.com> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=UK3zY+SW; spf=pass (imf28.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.215.179 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1659364670; a=rsa-sha256; cv=none; b=cqTvdJnxfQUG6NkkLzPfXxQEE+VqwduLj50lVY0bwWTZYExu5DcWPyPH7JhU0AeALqTVrW T1NZP6yjiMrIvUeTzXRJD6dCukdmiEMADK/f61R0II2kZatIA5+On15y4Cz5/CYSiKwvSv MwjtmSKDFsM9FHG5bkjUCVkWp5Zwj4s= ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1659364670; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=4La+/TTqE7pmoTbyCn6Aak17r/OUWM6KiH/lhDYcZds=; b=QthtTmJ+W5JIfMcyQsQWJWqmv61i2hFvn5Iyu0G+S8clSFI4ae7J8pvgQJK7O8Hjaw4Q89 utrgWX/VeN4v6W7Hv9ZW1wq2YCemhWLLNLPhULsToypglYKLCAf4cCRs3F/f7vRk6Jjok5 fO8ybEVOWXqnmkQ7hpedwrUTaKHn43Y= X-Stat-Signature: bc3ww1rtktzuoifceprb5bqsqjhh46xs X-Rspamd-Queue-Id: E63D6C00E2 X-Rspam-User: Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=gmail.com header.s=20210112 header.b=UK3zY+SW; spf=pass (imf28.hostedemail.com: domain of 42.hyeyoo@gmail.com designates 209.85.215.179 as permitted sender) smtp.mailfrom=42.hyeyoo@gmail.com; dmarc=pass (policy=none) header.from=gmail.com X-Rspamd-Server: rspam04 X-HE-Tag: 1659364669-302983 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Jul 28, 2022 at 06:09:27PM +0200, Vlastimil Babka wrote: > On 7/12/22 15:39, Hyeonggon Yoo wrote: > > Now that kmalloc_large_node() is in common code, pass large requests > > to page allocator in kmalloc_node() using kmalloc_large_node(). > > > > One problem is that currently there is no tracepoint in > > kmalloc_large_node(). Instead of simply putting tracepoint in it, > > use kmalloc_large_node{,_notrace} depending on its caller to show > > useful address for both inlined kmalloc_node() and > > __kmalloc_node_track_caller() when large objects are allocated. > > > > Signed-off-by: Hyeonggon Yoo <42.hyeyoo@gmail.com> > > Reviewed-by: Vlastimil Babka > Thanks! > Nit below: > > > --- > > v3: > > This patch is new in v3 and this avoids > > missing caller in __kmalloc_large_node_track_caller() > > when kmalloc_large_node() is called. > > > > include/linux/slab.h | 26 +++++++++++++++++++------- > > mm/slab.h | 2 ++ > > mm/slab_common.c | 17 ++++++++++++++++- > > mm/slub.c | 2 +- > > 4 files changed, 38 insertions(+), 9 deletions(-) > > > > diff --git a/include/linux/slab.h b/include/linux/slab.h > > index 082499306098..fd2e129fc813 100644 > > --- a/include/linux/slab.h > > +++ b/include/linux/slab.h > > @@ -571,23 +571,35 @@ static __always_inline __alloc_size(1) void *kmalloc(size_t size, gfp_t flags) > > return __kmalloc(size, flags); > > } > > > > +#ifndef CONFIG_SLOB > > static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t flags, int node) > > { > > -#ifndef CONFIG_SLOB > > - if (__builtin_constant_p(size) && > > - size <= KMALLOC_MAX_CACHE_SIZE) { > > - unsigned int i = kmalloc_index(size); > > + if (__builtin_constant_p(size)) { > > + unsigned int index; > > > > - if (!i) > > + if (size > KMALLOC_MAX_CACHE_SIZE) > > + return kmalloc_large_node(size, flags, node); > > + > > + index = kmalloc_index(size); > > + > > + if (!index) > > return ZERO_SIZE_PTR; > > > > return kmem_cache_alloc_node_trace( > > - kmalloc_caches[kmalloc_type(flags)][i], > > + kmalloc_caches[kmalloc_type(flags)][index], > > flags, node, size); > > } > > -#endif > > return __kmalloc_node(size, flags, node); > > } > > +#else > > +static __always_inline __alloc_size(1) void *kmalloc_node(size_t size, gfp_t flags, int node) > > +{ > > + if (__builtin_constant_p(size) && size > KMALLOC_MAX_CACHE_SIZE) > > + return kmalloc_large_node(size, flags, node); > > + > > + return __kmalloc_node(size, flags, node); > > +} > > +#endif > > > > /** > > * kmalloc_array - allocate memory for an array. > > diff --git a/mm/slab.h b/mm/slab.h > > index a8d5eb1c323f..7cb51ff44f0c 100644 > > --- a/mm/slab.h > > +++ b/mm/slab.h > > @@ -273,6 +273,8 @@ void create_kmalloc_caches(slab_flags_t); > > > > /* Find the kmalloc slab corresponding for a certain size */ > > struct kmem_cache *kmalloc_slab(size_t, gfp_t); > > + > > +void *kmalloc_large_node_notrace(size_t size, gfp_t flags, int node); > > #endif > > > > gfp_t kmalloc_fix_flags(gfp_t flags); > > diff --git a/mm/slab_common.c b/mm/slab_common.c > > index 6f855587b635..dc872e0ef0fc 100644 > > --- a/mm/slab_common.c > > +++ b/mm/slab_common.c > > @@ -956,7 +956,8 @@ void *kmalloc_large(size_t size, gfp_t flags) > > } > > EXPORT_SYMBOL(kmalloc_large); > > > > -void *kmalloc_large_node(size_t size, gfp_t flags, int node) > > +static __always_inline > > I don't think we need to inline, compiler should be able to make > kmalloc_large_node(_notrace) quite efficient anyway. You mean s/static __always_inline/static/g, or like this? kmalloc_large_node_notrace(): fold __kmalloc_large_node_notrace() into here kmalloc_large_node(): kmalloc_large_node_notrace() trace_kmalloc() I have no strong opinion. IMO It's unlikely that there would be workloads that are meaningfully affected by inlining or not inlining __kmalloc_large_node_notrace(). Just wanted to avoid adding even tiny of overhead by this series. > > > +void *__kmalloc_large_node_notrace(size_t size, gfp_t flags, int node) > > { > > struct page *page; > > void *ptr = NULL; > > @@ -976,6 +977,20 @@ void *kmalloc_large_node(size_t size, gfp_t flags, int node) > > > > return ptr; > > } > > + > > +void *kmalloc_large_node_notrace(size_t size, gfp_t flags, int node) > > +{ > > + return __kmalloc_large_node_notrace(size, flags, node); > > +} > > + > > +void *kmalloc_large_node(size_t size, gfp_t flags, int node) > > +{ > > + void *ret = __kmalloc_large_node_notrace(size, flags, node); > > + > > + trace_kmalloc_node(_RET_IP_, ret, NULL, size, > > + PAGE_SIZE << get_order(size), flags, node); > > + return ret; > > +} > > EXPORT_SYMBOL(kmalloc_large_node); > > > > #ifdef CONFIG_SLAB_FREELIST_RANDOM > > diff --git a/mm/slub.c b/mm/slub.c > > index f22a84dd27de..3d02cf44adf7 100644 > > --- a/mm/slub.c > > +++ b/mm/slub.c > > @@ -4401,7 +4401,7 @@ void *__do_kmalloc_node(size_t size, gfp_t flags, int node, unsigned long caller > > void *ret; > > > > if (unlikely(size > KMALLOC_MAX_CACHE_SIZE)) { > > - ret = kmalloc_large_node(size, flags, node); > > + ret = kmalloc_large_node_notrace(size, flags, node); > > > > trace_kmalloc_node(caller, ret, NULL, > > size, PAGE_SIZE << get_order(size), > -- Thanks, Hyeonggon