From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id A3DFBC19F4F for ; Tue, 30 Apr 2024 18:29:51 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1015B6B00B7; Tue, 30 Apr 2024 14:29:51 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 0B2096B00BB; Tue, 30 Apr 2024 14:29:51 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id E93A36B00BE; Tue, 30 Apr 2024 14:29:50 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id CCFA46B00B7 for ; Tue, 30 Apr 2024 14:29:50 -0400 (EDT) Received: from smtpin19.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 7D27216019E for ; Tue, 30 Apr 2024 18:29:50 +0000 (UTC) X-FDA: 82067036940.19.3C27302 Received: from mail-lj1-f171.google.com (mail-lj1-f171.google.com [209.85.208.171]) by imf19.hostedemail.com (Postfix) with ESMTP id 9FC9A1A0005 for ; Tue, 30 Apr 2024 18:29:48 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=FNiJaiDY; spf=pass (imf19.hostedemail.com: domain of almasrymina@google.com designates 209.85.208.171 as permitted sender) smtp.mailfrom=almasrymina@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1714501788; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=l1aj9TYKSBdPXW7U/K10knZmptyDbqlwkvpIjAvkFLw=; b=fWnPqbDji1Lj7y1azk/5R4TXWiMKUyshDmcBJXeEB5UzdRwnmnVu03qfvDjy/msFUQVg6I 2lGoktf/MipTKtivuRPMOk3PXiwo1aAUL0X+tXK6Jc1TY6azzpALk5jj+wFcVwHSLT7Oc8 ZFPkf8nAMswbg9VZC3XYCcFTL8XnwtE= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=google.com header.s=20230601 header.b=FNiJaiDY; spf=pass (imf19.hostedemail.com: domain of almasrymina@google.com designates 209.85.208.171 as permitted sender) smtp.mailfrom=almasrymina@google.com; dmarc=pass (policy=reject) header.from=google.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1714501788; a=rsa-sha256; cv=none; b=GBV7B+CVXsxBAuzIXHfcRCnCdTCH6Cv3BLGeAgZy4u1yZ4YknvP0wza10d2ExkcUPF6ua0 gHRpQ+bpMPktW2E2QTjHJ8cxKBocSj5HVHb50uF20nkxGhxgY+XesOaaHAkQouk0z1qppp id6dpt4PcdOs+ziPaYNEigUwExUd3i8= Received: by mail-lj1-f171.google.com with SMTP id 38308e7fff4ca-2d8a2cbe1baso80052661fa.0 for ; Tue, 30 Apr 2024 11:29:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20230601; t=1714501787; x=1715106587; darn=kvack.org; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:from:to:cc:subject:date :message-id:reply-to; bh=l1aj9TYKSBdPXW7U/K10knZmptyDbqlwkvpIjAvkFLw=; b=FNiJaiDYp97Nhjpt9ej3wy1mrANRc4WXEgM2lmuAWEdxHbgNZY8qpxIJMHze/P47aw cgSgljcGrcK3fyoXAI1usyvGWe3JUM8REpQSiIq4Ay02M6ufwDxTXEETQ3oBigKOCpPS 6yNOle1apcdNfRmqVnTFLf6L1WY09FCrwCCHKJxe3NEjXSUooWLC0wM64+MhFVEyWd7v LrhHia/FETHTCNCCko8nXII8A0/UlPTfkFeg4AnwPx+A1GCELN3IYjPGUaSA0fW2GD/1 u78v1z/m2zN/OEbtWnDnfU98WaW4pbhLrMXjmee1vKtcPTO+28vGM512eC4O/1f5AGZX XaHw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1714501787; x=1715106587; h=content-transfer-encoding:cc:to:subject:message-id:date:from :in-reply-to:references:mime-version:x-gm-message-state:from:to:cc :subject:date:message-id:reply-to; bh=l1aj9TYKSBdPXW7U/K10knZmptyDbqlwkvpIjAvkFLw=; b=Zj+fgiPauoj36OgoTEBIq1mp20aU/4h6E93f4fyP9EFoVv+dGiG7EQMCnkF5Ut0U7E cPKRXNVh4eNsFk5Oq5uNm2AFt96WStKyyDMAnVDo3kFEOluBK+SIQb4/gMMEeX4WyZkW o8AyJblQvkX8MRkpc9NuRoy8/Ub7zobQgDjjRX0EOLzhCYkV2rHzzpUQYrU/Al+FDQCc zJiKcsJGFHbWDD8dO4lyeaA0446UsO36P8WXgVfIP7ZSY14If2yGVjYwSycDI7c2pNsm Vy/iOb3MDrKThpdgS05IEveHVpFIVTyLiogsjctQpC4X1d7X1B/1hIuY7pEm/72oRfZV b0MA== X-Forwarded-Encrypted: i=1; AJvYcCXLMKF0qSeg4O+5fEkskshbI+v7HW8h2pM7AHdcPzSgMxBX7PKbbfBwpV4vFWoiOIVR1p2PjzQos4nTSVGPSI9dh7Y= X-Gm-Message-State: AOJu0Yy6ll+5kowNfZ6jrfO8mIsayVyP0VDG3WVlJ+hltG2O2ZsEG1X8 ZLW8qkDQ1tCagNywZ5SuFb3RURZb1mxAPNy5sCnrUKPWx9j0hdMQbz1M2xGd7HwmH5Cg49LYHAm xWhSxZf1rtwSOpsUAnxLvqyGB+UaEnj/mpCve X-Google-Smtp-Source: AGHT+IH0gaB2E/IC4/ciB9fJSDhMnVEQNygzcW8LVJwvG8xEFCDgTF/mhBRXlaBuFxJh5ugqGtCSIbfkywn/lGfB5fQ= X-Received: by 2002:a19:ca1b:0:b0:51c:d05b:6b8d with SMTP id a27-20020a19ca1b000000b0051cd05b6b8dmr194410lfg.23.1714501786252; Tue, 30 Apr 2024 11:29:46 -0700 (PDT) MIME-Version: 1.0 References: <20240403002053.2376017-1-almasrymina@google.com> <20240403002053.2376017-8-almasrymina@google.com> <8357256a-f0e9-4640-8fec-23341fc607db@davidwei.uk> In-Reply-To: From: Mina Almasry Date: Tue, 30 Apr 2024 11:29:31 -0700 Message-ID: Subject: Re: [RFC PATCH net-next v8 07/14] page_pool: devmem support To: Jens Axboe Cc: David Wei , netdev@vger.kernel.org, linux-kernel@vger.kernel.org, linux-doc@vger.kernel.org, linux-alpha@vger.kernel.org, linux-mips@vger.kernel.org, linux-parisc@vger.kernel.org, sparclinux@vger.kernel.org, linux-trace-kernel@vger.kernel.org, linux-arch@vger.kernel.org, bpf@vger.kernel.org, linux-kselftest@vger.kernel.org, linux-media@vger.kernel.org, dri-devel@lists.freedesktop.org, "David S. Miller" , Eric Dumazet , Jakub Kicinski , Paolo Abeni , Jonathan Corbet , Richard Henderson , Ivan Kokshaysky , Matt Turner , Thomas Bogendoerfer , "James E.J. Bottomley" , Helge Deller , Andreas Larsson , Jesper Dangaard Brouer , Ilias Apalodimas , Steven Rostedt , Masami Hiramatsu , Mathieu Desnoyers , Arnd Bergmann , Alexei Starovoitov , Daniel Borkmann , Andrii Nakryiko , Martin KaFai Lau , Eduard Zingerman , Song Liu , Yonghong Song , John Fastabend , KP Singh , Stanislav Fomichev , Hao Luo , Jiri Olsa , Steffen Klassert , Herbert Xu , David Ahern , Willem de Bruijn , Shuah Khan , Sumit Semwal , =?UTF-8?Q?Christian_K=C3=B6nig?= , Amritha Nambiar , Maciej Fijalkowski , Alexander Mikhalitsyn , Kaiyuan Zhang , Christian Brauner , Simon Horman , David Howells , Florian Westphal , Yunsheng Lin , Kuniyuki Iwashima , Arseniy Krasnov , Aleksander Lobakin , Michael Lass , Jiri Pirko , Sebastian Andrzej Siewior , Lorenzo Bianconi , Richard Gobert , Sridhar Samudrala , Xuan Zhuo , Johannes Berg , Abel Wu , Breno Leitao , Pavel Begunkov , Jason Gunthorpe , Shailend Chand , Harshitha Ramamurthy , Shakeel Butt , Jeroen de Borst , Praveen Kaligineedi , linux-mm@kvack.org, Matthew Wilcox Content-Type: text/plain; charset="UTF-8" Content-Transfer-Encoding: quoted-printable X-Rspam-User: X-Rspamd-Queue-Id: 9FC9A1A0005 X-Rspamd-Server: rspam06 X-Stat-Signature: kcao4etxrg48hs7g59rks9r1cw1umifi X-HE-Tag: 1714501788-884710 X-HE-Meta: U2FsdGVkX19LJ3RVH3+3qKd1YU6XDOz0rILEYi2btHZvhKt48CHK0SZHsIckNCLCOAy9dNycYgiAUmQS5VPLGWNzj0piPi5HcFLJ9yZz/vbZ7wot23mVDIa62Uw140zWASRyBDZEx0gzUtn3pBkOy/w7R8ma/FgvNDX0zmrOfGGfIemUCnhkdc88Ioa23X2QYAeyhl/aJmcfwqXVz6a8Vi4CoGILX3dlVRSwSzuNCBMnRlS0UgdZf4bfUIhjSXyKU64A0FRIDw/becAPgEvNevpWp0uJsF3gu4zuHWCm8S8+8P/oXgidFLDhSxvIpASO8KObC/F9pqnS+cuvHdW4+bHfwuamp0AyLlguM91Eavs+1b9e1LiDYAsTUxTr6dhdWE9PXSQ7kW8Pw+VWtujkMQuUXx4hfEIFnHEjhcCedZ8QPwubSiYxFH8WK7dsvmPC1Uo4ci8hU6K+Vi7QYNS5FCgOrmYJZXdL92+tbO43/YJq4TjUM8aqEJZFwdK+qj+60Y10/HSow8QE02hOf3s3lJmMQt1bIx8D8lHhcqbRrGn/AfkBHh+9L+zQCkbL/caHcBUr6MfwvfufW95vZgCClNnZSS7xbJo4MAFG9b+OcVjja+Mqx9RBca43aTB76Tt9mM7xuEqdCmhqXU6SmJEHTtaifXMrdZvCV2SyDI+mg/HgbNSa43CRiePABH5PgK5iwg3RZTr42E9OQu2mTzL68leH0cTl2Jx9UuoZjy52igUj9+PBo/Iy9QJudw3rdKlVr+LgUsWZKQcksKY9uCJREFhqMWCv2sgEBvd9Alq/zXBaxQEkLAMVCOBJAEB15Tk8UHaKgF3BFpkUlOmKVEhFnEbXa3VW+otF7HJ2NlmV1+twdfv1k2D0zfddYCWxzUhMGoXKvMLayALfDrXfKqK06dSgYdxINg4sDNKdOhuU6QA4Q9KP+TM+ts74PAG8HXrBNyLEqRpyZ1OpMthkxFe HNEG1vhU HAHa3eVDn/EqylIm4s7IwbytPx3BZ5ydk/nrdWOPMxc0Bv6wWtcAmqCe+ahto5EnL+//zDv4eUS7NJvqwfiseZlUgxbV1PfUkHEm7ouGD0K2y2ub4kPihkb4W5+TmVq1ZLXnB7SleGIUYMBIpM/WZL6dI1+u9Z32cR0ynj6obS4BsAExKuzuWUSBW5pHykHffrFRz5zqy02em5UIet6mqoRroZLRRSWRmBKSqA0So88ptr72zMCwrhj7L81IoBARZl7h73D/bqwn8eDqe63286QSjTiEDNSQkZrKFE35RwobZMu3OQ1jigpbCeoZp+1SzltqFLIfAc/gcnVoo97Mvla6z8Aw5pbGKhaZ1Rr+NdYfKkywhIgVbFqK1bchNAbRnm8hZ2zAfNMJQZ32BHy2kvG9t6X9jD5PjMoBsA5czg/ofSnc= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, Apr 30, 2024 at 6:46=E2=80=AFAM Jens Axboe wrote: > > On 4/26/24 8:11 PM, Mina Almasry wrote: > > On Fri, Apr 26, 2024 at 5:18?PM David Wei wrote: > >> > >> On 2024-04-02 5:20 pm, Mina Almasry wrote: > >>> @@ -69,20 +106,26 @@ net_iov_binding(const struct net_iov *niov) > >>> */ > >>> typedef unsigned long __bitwise netmem_ref; > >>> > >>> +static inline bool netmem_is_net_iov(const netmem_ref netmem) > >>> +{ > >>> +#if defined(CONFIG_PAGE_POOL) && defined(CONFIG_DMA_SHARED_BUFFER) > >> > >> I am guessing you added this to try and speed up the fast path? It's > >> overly restrictive for us since we do not need dmabuf necessarily. I > >> spent a bit too much time wondering why things aren't working only to > >> find this :( > > > > My apologies, I'll try to put the changelog somewhere prominent, or > > notify you when I do something that I think breaks you. > > > > Yes, this is a by-product of a discussion with regards to the > > page_pool benchmark regressions due to adding devmem. There is some > > background on why this was added and the impact on the > > bench_page_pool_simple tests in the cover letter. > > > > For you, I imagine you want to change this to something like: > > > > #if defined(CONFIG_PAGE_POOL) > > #if defined(CONFIG_DMA_SHARED_BUFFER) || defined(CONFIG_IOURING) > > > > or something like that, right? Not sure if this is something I should > > do here or if something more appropriate to be in the patches you > > apply on top. > > In general, attempting to hide overhead behind config options is always > a losing proposition. It merely serves to say "look, if these things > aren't enabled, the overhead isn't there", while distros blindly enable > pretty much everything and then you're back where you started. > The history there is that this check adds 1 cycle regression to the page_pool fast path benchmark. The regression last I measured is 8->9 cycles, so in % wise it's a quite significant 12.5% (more details in the cover letter[1]). I doubt I can do much better than that to be honest. There was a desire not to pay this overhead in setups that will likely not care about devmem, like embedded devices maybe, or setups without GPUs. Adding a CONFIG check here seemed like very low hanging fruit, but yes it just hides the overhead in some configs, not really removes it. There was a discussion about adding this entire netmem/devmem work under a new CONFIG. There was pushback particularly from Willem that at the end of the day what is enabled on most distros is what matters and we added code churn and CONFIG churn for little value. If there is significant pushback to the CONFIG check I can remove it. I don't feel like it's critical, it just mirco-optimizes some setups that doesn't really care about this work area. [1] https://lore.kernel.org/netdev/20240403002053.2376017-1-almasrymina@goo= gle.com/ --=20 Thanks, Mina