From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B15AC46467 for ; Thu, 19 Jan 2023 18:09:07 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id CCF7D6B0071; Thu, 19 Jan 2023 13:09:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id C7FD66B0073; Thu, 19 Jan 2023 13:09:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id B20826B0074; Thu, 19 Jan 2023 13:09:06 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0014.hostedemail.com [216.40.44.14]) by kanga.kvack.org (Postfix) with ESMTP id A2D616B0071 for ; Thu, 19 Jan 2023 13:09:06 -0500 (EST) Received: from smtpin20.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id 4941480173 for ; Thu, 19 Jan 2023 18:09:06 +0000 (UTC) X-FDA: 80372335092.20.9DF43DC Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by imf19.hostedemail.com (Postfix) with ESMTP id 439C81A0009 for ; Thu, 19 Jan 2023 18:09:04 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ZI83TLxk; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf19.hostedemail.com: domain of jbrouer@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=jbrouer@redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1674151744; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=b/gO3d3tfSy1H0tWMkbcVJW8vhTKCMyI1UuEUKguo/Y=; b=EaH+qo7nEaBjc9GvnWGl2cC74LYGxgJHymbMRNLLiRO/m7Te4oOihWBvgxj/n/+9RxDDDC nlM4Xd87Bn5MxQZjKbMXK2vMpW6VwJSa/WcQyk/DdtINsaJOt7HsNbFZmRuEy3XGOE344l zk/P6TSlx0u0WOYV6tAlohLBbPReBao= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=ZI83TLxk; dmarc=pass (policy=none) header.from=redhat.com; spf=pass (imf19.hostedemail.com: domain of jbrouer@redhat.com designates 170.10.133.124 as permitted sender) smtp.mailfrom=jbrouer@redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1674151744; a=rsa-sha256; cv=none; b=yZ4ZtUwg4kSptDYQ71GI2HmU67FGluaitkr0XNkJduvixQ0aCJNUS1cQN8IX2E2d3h352X gHaR3Aawg2WoV8o6mtBvSbTLbh8HELAO0frJUK5K5nSyrHTdbDw8bfjBcKmLBowV5tlMPw CA9uGTXSi5gUs+H+Dw5WeDrm9oC6er8= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1674151743; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=b/gO3d3tfSy1H0tWMkbcVJW8vhTKCMyI1UuEUKguo/Y=; b=ZI83TLxk3oozqVive3rSkHj5RwSvFfZZhMfHfipXysxACNN9htd1N9BpxNdhr0GQ2ZjAkw fZwvrRPfgz7nFslwISM8E2n5D4JZjglbavwWdvP7wqENB7aQQ0qzw2YbEm9d9S5CW7r3cO TUR5Tx0l/Xm5ZKpetqmRqHmqDXR8u5g= Received: from mail-ed1-f70.google.com (mail-ed1-f70.google.com [209.85.208.70]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_128_GCM_SHA256) id us-mta-184-hKGJpVLZNXCWo_cT_FVztg-1; Thu, 19 Jan 2023 13:09:02 -0500 X-MC-Unique: hKGJpVLZNXCWo_cT_FVztg-1 Received: by mail-ed1-f70.google.com with SMTP id z18-20020a05640235d200b0049d84165065so2141758edc.18 for ; Thu, 19 Jan 2023 10:09:02 -0800 (PST) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=content-transfer-encoding:in-reply-to:references:to :content-language:subject:cc:user-agent:mime-version:date:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=b/gO3d3tfSy1H0tWMkbcVJW8vhTKCMyI1UuEUKguo/Y=; b=NwBVELGlpl7WDbZ+0O3ROAHaKVv3Sy47YKOSsGogzF55xHDqVltGalrCYFB6nZo26/ UfrTvAfpEPhR1kseC58C6q3LTU09T4hMwtkpqFrd1Sx4PmkLTUZX+Dxwt+fQ/VBUA1PR lAziEiOW0BCAExT+n4gKKlfWcb7ZNZkKSX3cpROF67gtQA5EPsV+BZ1ikBCVWRFU0ysi 76F8QbgJD4/ow/qqkby2u/YK5LyAyilKctf0CJCC2JXAlTNkYl9oRoPHsPOuQzKnijd9 cy4ECH0+NYlvbU0MzeYGUMyeo8oS+8TQ5z4GgxdXlTEP6gl6BFft+5JUiQyviGbR9Juq 0B2w== X-Gm-Message-State: AFqh2kputIeeyo3EiK86l+PgX3se4AH5sswM1QcJA/qAZF1xKzRistmS CUZ4idTY8Y7sAF3RF7SJT/rhk1x1LcS0/+ryl7vPPNLgtlUNnxgGPVEvIJjeDk4G3r+jlFZNyAi EsP8saLWglMc= X-Received: by 2002:a17:906:6313:b0:872:6508:190 with SMTP id sk19-20020a170906631300b0087265080190mr13152931ejc.6.1674151740773; Thu, 19 Jan 2023 10:09:00 -0800 (PST) X-Google-Smtp-Source: AMrXdXskSUFliZvu7tLh4HONJuC8MzSqGjXFZUGF0GwT6v3T+8rLu4AkXCUJD4g5uFbFtEzsZMuhpA== X-Received: by 2002:a17:906:6313:b0:872:6508:190 with SMTP id sk19-20020a170906631300b0087265080190mr13152901ejc.6.1674151740535; Thu, 19 Jan 2023 10:09:00 -0800 (PST) Received: from [192.168.41.200] (83-90-141-187-cable.dk.customer.tdc.net. [83.90.141.187]) by smtp.gmail.com with ESMTPSA id k22-20020a17090646d600b0085fc3dec567sm11281206ejs.175.2023.01.19.10.08.58 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Thu, 19 Jan 2023 10:08:59 -0800 (PST) From: Jesper Dangaard Brouer X-Google-Original-From: Jesper Dangaard Brouer Message-ID: Date: Thu, 19 Jan 2023 19:08:58 +0100 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.6.0 Cc: brouer@redhat.com, netdev@vger.kernel.org, linux-mm@kvack.org, Andrew Morton , Mel Gorman , Joonsoo Kim , penberg@kernel.org, vbabka@suse.cz, Jakub Kicinski , "David S. Miller" , edumazet@google.com, pabeni@redhat.com Subject: Re: [PATCH RFC] mm+net: allow to set kmem_cache create flag for SLAB_NEVER_MERGE To: Matthew Wilcox , Christoph Lameter References: <167396280045.539803.7540459812377220500.stgit@firesoul> <36f5761f-d4d9-4ec9-a64-7a6c6c8b956f@gentwo.de> In-Reply-To: X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 7bit X-Rspamd-Queue-Id: 439C81A0009 X-Rspamd-Server: rspam09 X-Rspam-User: X-Stat-Signature: r7ifhn9ahmku4gfi5zoxu1ww8ujfozus X-HE-Tag: 1674151744-817907 X-HE-Meta: U2FsdGVkX1/5B7Qc7qTqnF9JgWzxhcsrYFKxO4vd4gdfnJCkxa7EWzAbqb38GJhljHBU8SO3JKeW0A33EuuvUsv9I8ES54bARcmj1y4CukGFIvWAiMoCw1WsEbmux5vvHOmNDJ2YP28eMZlojnpT20Lx8qQ8AOymULaxTnhNisH7u8UPApa8vlZGJZCsOZpWi16JGqNkc2BtDG/vTYM3Q4o/O8YQh+TFZH6T/SjdUFxwO6jKhMKku2mV/TaLverU8Lcn+MzKPZCxetf368Gq8r+tUE4OCpeOirEU6XMoV1CL5sVB6QkACwbgDlLgIDxc8xHRhxclZ9agHgqSIdgbua6pQfp1gR610T0gP7vqvyKilCoguszqxBFhYOQ9+kD/LZb1RAoBNmeTaXNxey3BLDB9Gu8+VyZ6CwLmnFyvvZhoRwzcTPKZ8UtG0V2keH8Ie6Z3L49gjTT0EMss992PI8ZUD6C9lCdiDowC6EgZTNIrluMTSlMri0LbtadGTWCPkC3rKTVUt2jvo2YK/+CUxp+rfy0HZwVcBn2UDhAXJwaw595efii7MZdhW1qSZ0nph9g9EjDTle4n4BbwOSh4HmYe7SGD/2JIQx+5/eIwPlWY41O92PMVC2E2hYCPVXmTBkPZhjdQadHb+5ygWdwW4dIXHIhv7Tgz9UsXrs1K4r2GqdrG5gRNuG2vfVHXPWnSbXh6hzcMi+7CkDChrpwzB3NDMCGN2fXLAK2IYURsOK2UqxJoVeB/+iPmPjYMgtCs42fArDSiLXR3eNneKxBo9+zXhAafAemJ+FmlFGM/V6YtDdyID8GTwuj9Eq0RXK1k7uyRc1rgniXESyTXp+1HyMUF8mBlAesB7jsfUNM0TZ53nZj/oPEEU6ZWjmUUVFZhhW4H0nIjUDkb73FeipWIQ7HQ4QzgNYR1v2w1akfUAKYRfiMjr1WRCW+4WQPceZoVIicREdlZD+fd1t+Lca+ UUQGZiOl fiZNXgMfvaakki53iDKzwyzXxvUSgIhguyGa70BPn0G+l+oMjlC9X00go+ta9K9ncpkrT/jJbAT0rAr6vLkjm8LiwMCED/CLJExHmUWMmNNYucsU4fPZqkGVvv/y2UTk6jgqoFNg0fd8Fyq67mqEivm/fCrICmbt8SbgN X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 18/01/2023 06.17, Matthew Wilcox wrote: > On Tue, Jan 17, 2023 at 03:54:34PM +0100, Christoph Lameter wrote: >> On Tue, 17 Jan 2023, Jesper Dangaard Brouer wrote: >> >>> When running different network performance microbenchmarks, I started >>> to notice that performance was reduced (slightly) when machines had >>> longer uptimes. I believe the cause was 'skbuff_head_cache' got >>> aliased/merged into the general slub for 256 bytes sized objects (with >>> my kernel config, without CONFIG_HARDENED_USERCOPY). >> >> Well that is a common effect that we see in multiple subsystems. This is >> due to general memory fragmentation. Depending on the prior load the >> performance could actually be better after some runtime if the caches are >> populated avoiding the page allocator etc. > > The page allocator isn't _that_ expensive. I could see updating several > slabs being more expensive than allocating a new page. > For 10Gbit/s wirespeed small frames I have 201 cycles as budget. I prefer to measure things, so lets see what page alloc cost, but also relate this to how much this is per 4096 bytes. alloc_pages order:0(4096B/x1) 246 cycles per-4096B 246 cycles alloc_pages order:1(8192B/x2) 300 cycles per-4096B 150 cycles alloc_pages order:2(16384B/x4) 328 cycles per-4096B 82 cycles alloc_pages order:3(32768B/x8) 357 cycles per-4096B 44 cycles alloc_pages order:4(65536B/x16) 516 cycles per-4096B 32 cycles alloc_pages order:5(131072B/x32) 801 cycles per-4096B 25 cycles I looked back at my MM-presentation[2016][2017], and notice that in [2017] I reported that Mel have improved order-0 page cost to 143 cycles in kernel 4.11-rc1. According to above measurements kernel have regressed in performance. [2016] https://people.netfilter.org/hawk/presentations/MM-summit2016/generic_page_pool_mm_summit2016.pdf [2017] https://people.netfilter.org/hawk/presentations/MM-summit2017/MM-summit2017-JesperBrouer.pdf >> The merging could actually be beneficial since there may be more partial >> slabs to allocate from and thus avoiding expensive calls to the page >> allocator. > > What might be more effective is allocating larger order slabs. I see > that kmalloc-256 allocates a pair of pages and manages 32 objects within > that pair. It should perform better in Jesper's scenario if it allocated > 4 pages and managed 64 objects per slab. > > Simplest way to test that should be booting a kernel with > 'slub_min_order=2'. Does that help matters at all, Jesper? You could > also try slub_min_order=3. Going above that starts to get a bit sketchy. > I have tried this slub_min_order trick before, and it did help. I've not tested it is recently. --Jesper