From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 9012FC77B60 for ; Fri, 28 Apr 2023 15:48:18 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1AE926B0085; Fri, 28 Apr 2023 11:48:18 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 139976B0087; Fri, 28 Apr 2023 11:48:18 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F419F6B0088; Fri, 28 Apr 2023 11:48:17 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id E01B56B0085 for ; Fri, 28 Apr 2023 11:48:17 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay09.hostedemail.com (Postfix) with ESMTP id B2DD780180 for ; Fri, 28 Apr 2023 15:48:17 +0000 (UTC) X-FDA: 80731231434.14.22EA0B7 Received: from us-smtp-delivery-124.mimecast.com (us-smtp-delivery-124.mimecast.com [170.10.129.124]) by imf28.hostedemail.com (Postfix) with ESMTP id 767A3C001A for ; Fri, 28 Apr 2023 15:48:15 +0000 (UTC) Authentication-Results: imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gdWyjA4H; spf=pass (imf28.hostedemail.com: domain of jbrouer@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=jbrouer@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682696895; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=/hcBafirGtbsKs3va60aDlPh2Tf2tewUCDGORUeB8Xo=; b=Ya7zfGSpIK9qmgkl+ks4fxT1oE9o0lP5WcNrnM+TmGwmL7hqjIzZuj9NCdagu38/hXLwuP vcQOX8JcDeCCMjTRs6djv5CFpP6iQY8l7oF+/2hE6x8OxkDscXtgDa9URtsU9OVcUCOTy3 rih8EksrBgAtD0cQKltT5T45kNwiQco= ARC-Authentication-Results: i=1; imf28.hostedemail.com; dkim=pass header.d=redhat.com header.s=mimecast20190719 header.b=gdWyjA4H; spf=pass (imf28.hostedemail.com: domain of jbrouer@redhat.com designates 170.10.129.124 as permitted sender) smtp.mailfrom=jbrouer@redhat.com; dmarc=pass (policy=none) header.from=redhat.com ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682696895; a=rsa-sha256; cv=none; b=QhujBPKbvD5MNJHijUrEnAhoSVTZPCdL9nnyO5sHKOOqP9MQO5OVRAQET9/vlDr1I7eFu0 gc5TqYnXHUGfvdS5tYWWfzHPdSkJ7psmuTPffbKYUfkMUDMYXkfYEgH5SfRPtIFelkLKpf O/zlOzhh3yKm6aq74hTebeRDOgZv6XA= DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=redhat.com; s=mimecast20190719; t=1682696894; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=/hcBafirGtbsKs3va60aDlPh2Tf2tewUCDGORUeB8Xo=; b=gdWyjA4HU9eTNjnCg9nNFgAHjjO+8xorlBycpSm1Aat42nXCB2ezDfP+H/4THhrqlYf943 YqfKP9SCFfAdU6HAFCkj7Wpxa7N9g+MZjSvKgl+wh9tJOHNISKRGVrv1GZw5fdphj+axGu hXVdo9gKlN9W0L7SfVU1Ua9iAaBgny8= Received: from mail-ej1-f71.google.com (mail-ej1-f71.google.com [209.85.218.71]) by relay.mimecast.com with ESMTP with STARTTLS (version=TLSv1.3, cipher=TLS_AES_256_GCM_SHA384) id us-mta-25-vz9_Hs8hOsG_t0ePlp6Mew-1; Fri, 28 Apr 2023 11:46:58 -0400 X-MC-Unique: vz9_Hs8hOsG_t0ePlp6Mew-1 Received: by mail-ej1-f71.google.com with SMTP id a640c23a62f3a-94ecfab8993so1223971366b.1 for ; Fri, 28 Apr 2023 08:46:49 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20221208; t=1682696808; x=1685288808; h=content-transfer-encoding:in-reply-to:references:to :content-language:subject:cc:user-agent:mime-version:date:message-id :from:x-gm-message-state:from:to:cc:subject:date:message-id:reply-to; bh=/hcBafirGtbsKs3va60aDlPh2Tf2tewUCDGORUeB8Xo=; b=jOGUPsN78eA4N0REBC7jRGpXVYygXWc2mcZ1hI0KWNRKdYIYUv7iyjEXpGqK9B28+w pgs0YEdEgBtDbVC7VRBssOyHSrg2eK7F0Y3LVPuYj2biPYQllp64cDVa1yZ897mCj2pI StRdHtUEl6WIUYSUcrTwODIzsQklUG8tiSClGkKNdPf7PZWWUS2OE0XLBaDsByFYSBbu WtmZFjXw2CWumbLgz3UU754zoxoitO+TR9AN8P/bCijOwpRtMMW7YaRGHeGSow0cqT89 HTEwNU1Ui4LDzKKJffErTXQuxcKVt5iKAOrVR8fJXuR+1fhYaMhQZ/4z2U6PQWutXdvx GT0Q== X-Gm-Message-State: AC+VfDzUrbpTlFfLmppXBQZwsh1kTrVQYAB7BqaHJy4Aqrp7DHiwkpNo VW1V9jPEy0Oifw1/vPmPmyx7d9WLSu5CsCuas54rR4n+KA1y2Tk0oylFAnxw5bbzcsFwuqDqqcK 9uhN2My2w2ZE= X-Received: by 2002:a17:907:1629:b0:94a:5ecb:6ea7 with SMTP id hb41-20020a170907162900b0094a5ecb6ea7mr5527462ejc.43.1682696808494; Fri, 28 Apr 2023 08:46:48 -0700 (PDT) X-Google-Smtp-Source: ACHHUZ7rpAV07rm98ZXqi9U6WhnhyHfOeQi41kKjdlW0G3BuiV1Nj46oZf/7aUeX4PuytnJiOaiEjA== X-Received: by 2002:a17:907:1629:b0:94a:5ecb:6ea7 with SMTP id hb41-20020a170907162900b0094a5ecb6ea7mr5527435ejc.43.1682696808186; Fri, 28 Apr 2023 08:46:48 -0700 (PDT) Received: from [192.168.42.222] (194-45-78-10.static.kviknet.net. [194.45.78.10]) by smtp.gmail.com with ESMTPSA id o23-20020a170906769700b0094e9f87c6d4sm11191093ejm.192.2023.04.28.08.46.46 (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Fri, 28 Apr 2023 08:46:47 -0700 (PDT) From: Jesper Dangaard Brouer X-Google-Original-From: Jesper Dangaard Brouer Message-ID: Date: Fri, 28 Apr 2023 17:46:46 +0200 MIME-Version: 1.0 User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:102.0) Gecko/20100101 Thunderbird/102.9.1 Cc: brouer@redhat.com, lorenzo@kernel.org, linyunsheng@huawei.com, bpf@vger.kernel.org, "David S. Miller" , Jakub Kicinski , Paolo Abeni , Andrew Morton , willy@infradead.org Subject: Re: [PATCH RFC net-next/mm V2 1/2] page_pool: Remove workqueue in new shutdown scheme To: =?UTF-8?Q?Toke_H=c3=b8iland-J=c3=b8rgensen?= , Ilias Apalodimas , netdev@vger.kernel.org, Eric Dumazet , linux-mm@kvack.org, Mel Gorman References: <168262348084.2036355.16294550378793036683.stgit@firesoul> <168262351129.2036355.1136491155595493268.stgit@firesoul> <871qk582tn.fsf@toke.dk> In-Reply-To: <871qk582tn.fsf@toke.dk> X-Mimecast-Spam-Score: 0 X-Mimecast-Originator: redhat.com Content-Language: en-US Content-Type: text/plain; charset=UTF-8; format=flowed Content-Transfer-Encoding: 8bit X-Rspam-User: X-Rspamd-Server: rspam03 X-Stat-Signature: aqcwrcr8kfcj87t5kbw6n5i93bu6jkjb X-Rspamd-Queue-Id: 767A3C001A X-HE-Tag: 1682696895-142260 X-HE-Meta: U2FsdGVkX1+hfgV9z+BJ20amXI//bC8knp8CBiEWRV0yKLv65hLopG8dwnV7WpvcYTeHwVqPW8/qq+qfET8liVY/jmf5F2n++Bow/XPUZWSF0fo7HQuDGalT82WyjOkRszDLRrkbVMjg4//RPhmFL01vPag0kyRrtxhA4lc2uLCXFZQ40Q4Hrg9Sk1aNAFJgQrXz6euS30zAlwbCqUFiLGQuU3uUQt3BvOObejM6QNcBKQP9vK/QztdoXoRYCZY4a0AGN2bECII7Asr9I+A/wFJKaWdKxxtaKFi0wSFirokSyS9MUW41kNY/uKTV7llBgRW/yK/oV466pTQ5vrVqBvGxznfhWmyVIfk42u1aspgWp4kn9dC1cgEboS2cQWQ2i/+KEZ79/hjP4eGseTdHUWvYX2yfZTwg0GJhz1zqgEbv9sjL5Z2gdAKcuNra9UeJY+blao4qbse4/T/4fE1OsFBCLeHTcZELF5ozwmcrOYtXWstc+p/24c4rclHiAad0liZPPnrRYzdmn1Augrkv0forc40Ld6ybce2kiHlzd9k5B2qqeCsP9yx8BNYcCWikjeNdM91EKvhlDNtN83fT72Ez114yTkmj+RSLiDHRm3Ck0KoZWzFLnEyLrlTSBajMm9DMul6+bsbgaqqZlRz1J33qnhRR9L71fVSIEr5OeWOjht2L4kQCp3zGsO8PNflxMqGpmQjzV501z4ymEteIcJtZGTW4aB3fn5DZJ+jDQPVn5UB79SqQgXpvbclHDlRVscyNF6KrNPQNFmv4a2ACwef1KY/kbdIl3dWxWShC58n4/NuSHJ08Mok9qM1LH3CA8tE6hEN4URxyFnHjdtYgI2FszgTBwmOPnQEGzlPtYHzSZ+a5rmkYpLXXnZzATs9fgoeVOc/Df9W/BYzLB05OJlYWQPjr8GyoZQipSe4buGTpdzPmV2Kr7GbLCIrJ33RRcCEHx1bTGQvBbNQvzYh 7meq4T09 VidKwLqNIsEsaUxFo0qgRmoJ25tjZGAtvV36KnwchUhyy2XdTZjiWkjKudt3UVwky98s2yjXftFGFrIPch0PUbo8FFUvoqi6psNtFEJicPU6XF5d8BGEi5dL+AsEeeKinX8eBpcFzgyHqxh6LCxKU27NgtoCA5kzecoM95GCY+HvbPc6s/aPhdcX9jIzyM550yY/ouD/UNAhQRBioUzYPH43RYwkiRQGt/iDOiVsF/AUdOeQNhdnRnlqvaxp5Q3k+Tnej0k/sybDTxOFRGYKdi0j2aYcMry1LwO44+DHWdUvyw5JuXXsFV6RU9CI3yEo3nCRAV2fl7Lr6TkOPVW9dwGp91HhHcW3elPSZWZAkN6Y3nd1LEe38HX2jIWJ7kRsREuHo42paUZJzOXjR8XZK9oWL+nvFw3dvVe3qsm4uR2nDrEZJ3DIUW6Y2IdcPJU33pHDx8asGF54NFe7dwnzOvQKoZRzKu8Ah/0oDMd3KguAFtAo= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On 27/04/2023 22.53, Toke Høiland-Jørgensen wrote: >> @@ -868,11 +890,13 @@ void page_pool_destroy(struct page_pool *pool) >> if (!page_pool_release(pool)) >> return; >> >> - pool->defer_start = jiffies; >> - pool->defer_warn = jiffies + DEFER_WARN_INTERVAL; >> + /* PP have pages inflight, thus cannot immediately release memory. >> + * Enter into shutdown phase. >> + */ >> + pool->p.flags |= PP_FLAG_SHUTDOWN; > > I think there's another race here: once the flag is set in this line > (does this need a memory barrier, BTW?), another CPU can return the last > outstanding page, read the flag and call page_pool_empty_ring(). If this > happens before the call to page_pool_empty_ring() below, you'll get a > use-after-free. > > To avoid this, we could artificially bump the pool->hold_cnt *before* > setting the flag above; that way we know that the page_pool_empty_ring() > won't trigger a release, because inflight pages will never go below 1. > And then, below the page_pool_empty_ring() call below, we can add an > artificial bump of the release_cnt as well, which means we'll get proper > atomic semantics on the counters and only ever release once. I.e.,: > >> - INIT_DELAYED_WORK(&pool->release_dw, page_pool_release_retry); >> - schedule_delayed_work(&pool->release_dw, DEFER_TIME); >> + /* Concurrent CPUs could have returned last pages into ptr_ring */ >> + page_pool_empty_ring(pool); > release_cnt = atomic_inc_return(&pool->pages_state_release_cnt); > page_pool_free_attempt(pool, release_cnt); > I agree and I've implemented this solution (see V3 soon). I've used smp_store_release() instead of WRITE_ONCE(), because AFAIK smp_store_release() adds the memory barriers. --Jesper