From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 51337C77B75 for ; Wed, 3 May 2023 15:49:40 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id DE1EC6B0081; Wed, 3 May 2023 11:49:39 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id D91DB6B0083; Wed, 3 May 2023 11:49:39 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C58D56B0085; Wed, 3 May 2023 11:49:39 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.133.124]) by kanga.kvack.org (Postfix) with ESMTP id A48A96B0081 for ; Wed, 3 May 2023 11:49:39 -0400 (EDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1683128979; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=O4VP89Zv90qrgsob/vJ4a462qWannnDrJro4rCmwa5M=; b=Sln4Meck3wZl9xDXzZjqNARUMKjAMEV1n3snFP+qEtLlUBXtYABCtKk89yQF4W3gM1XQxM PQw4XppS5BS4WzdekMB4+DUu/fRhBVUQeimkq2/R5fBVIXUwG9uuAT8XjlMtI+jPk/wYLO h+2i+b3tO/jgxQOC8t4vACZERG+4ilA= Received: from mail-ej1-f69.google.com (mail-ej1-f69.google.com [209.85.218.69]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-154-3UtnRAfpOPOf6-gn5huWfQ-1; Wed, 03 May 2023 11:49:38 -0400 X-MC-Unique: 3UtnRAfpOPOf6-gn5huWfQ-1 Received: by mail-ej1-f69.google.com with SMTP id a640c23a62f3a-95c3f20df2dso518068866b.1 for ; Wed, 03 May 2023 08:49:38 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1683128977; x=1685720977; h=content-transfer-encoding:in-reply-to:references:to :content-language:subject:cc:user-agent:mime-version:date:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=O4VP89Zv90qrgsob/vJ4a462qWannnDrJro4rCmwa5M=; b=OcxgYvbQEUNpuwfI+lbPvZAwKWJ3Rf8dyaJEPDntiiqnJngQW+TngcSPHUGyowML4L jrtZdxPr0tkG/Y0QZb2yJ5uGnVHrmx/5alGc9BfdSyWI8l41AhMSmty7ysLu8gr6UrMb PGf/AAR7NmSJXX21I3fj2RP2FJfuSNEdp8mPR2WxcKsn1LHpi2kqySHPBywAgY/zaT84 JVEoUWgvm6LDcQAGgnP5CHZLuxmx9u/0Pazo4UeOhqbBJB3ekVKGuN91GSBe/fm0b2+n 5vIn8/evEUkbO40LeRTszj8Q1wunm3MU4OnrJ7fAg2jsARQVkakGvslY/I/OKF/P7ZDd Ha3A== X-Gm-Message-State: AC+VfDzTEsV38rGh4jdi2VnZLMd0v/uBJ38BgKwBhwI0w7UH2kg3JsOz K+iahlSUjWLN/4mBcc96YvhmKuliyG/LVyRhGoPm6ECJQBQ7LoRof77EVEd9g+t9RyW6QC8emmg e/8kOxG8mX+o= X-Received: by 2002:a17:907:c21:b0:933:3814:e0f4 with SMTP id ga33-20020a1709070c2100b009333814e0f4mr4712163ejc.16.1683128977009; Wed, 03 May 2023 08:49:37 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ6e5zv7cZWE/jkHWJTvri6PxUa3jx686lEP7V5PP9ooaEOwZDZGGsjXhQil4vBJmqj+KSXbvg== X-Received: by 2002:a17:907:c21:b0:933:3814:e0f4 with SMTP id ga33-20020a1709070c2100b009333814e0f4mr4712134ejc.16.1683128976703; Wed, 03 May 2023 08:49:36 -0700 (PDT) Received: from [192.168.42.222] (cgn-cgn9-185-107-14-3.static.kviknet.net. [185.107.14.3]) by smtp.gmail.com with ESMTPSA id th7-20020a1709078e0700b009596e7e0dbasm13598187ejc.162.2023.05.03.08.49.35 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 03 May 2023 08:49:36 -0700 (PDT) From: Jesper Dangaard Brouer X-Google-Original-From: Jesper Dangaard Brouer Message-ID: <3a5a28c4-01a3-793c-6969-475aba3ff3b5@redhat.com> Date: Wed, 3 May 2023 17:49:34 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.9.1 Cc: brouer@redhat.com, Ilias Apalodimas , netdev@vger.kernel.org, Eric Dumazet , linux-mm@kvack.org, Mel Gorman , lorenzo@kernel.org, linyunsheng@huawei.com, bpf@vger.kernel.org, "David S. Miller" , Paolo Abeni , Andrew Morton , willy@infradead.org Subject: Re: [PATCH RFC net-next/mm V3 1/2] page_pool: Remove workqueue in new shutdown scheme To: =?UTF-8?Q?Toke_H=c3=b8iland-J=c3=b8rgensen?= , Jakub Kicinski References: <168269854650.2191653.8465259808498269815.stgit@firesoul> <168269857929.2191653.13267688321246766547.stgit@firesoul> <20230502193309.382af41e@kernel.org> <87ednxbr3c.fsf@toke.dk> In-Reply-To: <87ednxbr3c.fsf@toke.dk> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 03/05/2023 13.18, Toke Høiland-Jørgensen wrote: > Jakub Kicinski writes: > >> On Fri, 28 Apr 2023 18:16:19 +0200 Jesper Dangaard Brouer wrote: >>> This removes the workqueue scheme that periodically tests when >>> inflight reach zero such that page_pool memory can be freed. >>> >>> This change adds code to fast-path free checking for a shutdown flags >>> bit after returning PP pages. >> >> We can remove the warning without removing the entire delayed freeing >> scheme. I definitely like the SHUTDOWN flag and patch 2 but I'm a bit >> less clear on why the complexity of datapath freeing is justified. >> Can you explain? > > You mean just let the workqueue keep rescheduling itself every minute > for the (potentially) hours that skbs will stick around? Seems a bit > wasteful, doesn't it? :) I agree that this workqueue that keeps rescheduling is wasteful. It actually reschedules every second, even more wasteful. NIC drivers will have many HW RX-queues, with separate PP instances, that each can start a workqueue that resched every sec. Eric have convinced me that SKBs can "stick around" for longer than the assumptions in PP. The old PP assumptions came from XDP-return path. It is time to cleanup. > > We did see an issue where creating and tearing down lots of page pools > in a short period of time caused significant slowdowns due to the > workqueue mechanism. Lots being "thousands per second". This is possible > using the live packet mode of bpf_prog_run() for XDP, which will setup > and destroy a page pool for each syscall... Yes, the XDP live packet mode of bpf_prog_run is IMHO abusing the page_pool API. We should fix that somehow, at least the case where live packet mode is only injecting a single packet, but still creates a PP instance. The PP in live packet mode IMHO only makes sense when repeatedly sending packets that gets recycles and are pre-inited by PP. This use of PP does exemplify why is it problematic to keep the workqueue. I have considered (and could be convinced) delaying the free via call_rcu, but it also create an unfortunate backlog of work in the case of live packet mode of bpf_prog_run. --Jesper