From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id E428BC433F5 for ; Mon, 7 Mar 2022 16:49:06 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 12C098D0002; Mon, 7 Mar 2022 11:49:06 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 0DC1F8D0001; Mon, 7 Mar 2022 11:49:06 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id F0C708D0002; Mon, 7 Mar 2022 11:49:05 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (relay.hostedemail.com [64.99.140.26]) by kanga.kvack.org (Postfix) with ESMTP id E1D8A8D0001 for ; Mon, 7 Mar 2022 11:49:05 -0500 (EST) Received: from smtpin07.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id ADB2521F89 for ; Mon, 7 Mar 2022 16:49:05 +0000 (UTC) X-FDA: 79218175050.07.7866C7F Received: from mail-yb1-f182.google.com (mail-yb1-f182.google.com [209.85.219.182]) by imf21.hostedemail.com (Postfix) with ESMTP id 2D59B1C000A for ; Mon, 7 Mar 2022 16:49:05 +0000 (UTC) Received: by mail-yb1-f182.google.com with SMTP id l2so12699890ybe.8 for ; Mon, 07 Mar 2022 08:49:04 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=google.com; s=20210112; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=wOaEmSLT8Hs8VbykY7UZSpJotv9MXZiDNybb91mB0/A=; b=tkjRQRTQjIx8Brz4GulR3BnbRaCpEJmULkorenyHZmon6Eu1SEIeV38HFh3LGAKi5+ juJvYGVmuC/0a/kntYr1BjDzZKQ10Fr+ESwqmkd++wkKI77bD3c1ztQHm5t3GU0p/1/c Z3V+RGoyxvJxXbxDygUsntM2raFKNCS1/cryvUkXbqPJ3zfEOCGiQT9zGoSbV6BSKMod 14Q5rea9Q+Ju2Xgm0RPTicufRonSNumVfaIjKCO+52jYo/ZtHPIj/pZ+q2fDw1qrtbUP yrIabyYVpPfMdBbMp8IhTAMwylN6tujsMC/Xixdt4uh0aAu5uTBw5aWWG5pV7TdXZgua hvoQ== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20210112; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=wOaEmSLT8Hs8VbykY7UZSpJotv9MXZiDNybb91mB0/A=; b=DdPjMjBssKMzErj9KcZJXi6hoKEumXnVd84IXy61ChRtImZCZZx+3IKWztuSfSxQ0P vALIQ1oIaShPNlLgYxuF/y7v3rmXHKOTTb9veedFQc55+hsJmUIPhvbR9DJ7e/YuSxc+ LMJzK0QFw8aJJKMYI3kNuThkQZtnDM2feMY6Z75ERlPmmtWa/6XWNemPJkbkuVa6E08F aIMTaqLs9FhJt9cd6IdUdYj27D1hXWi3Z2U+jAkTrOX0X+9JNGfRUCtwCqxg2EYepU5C 11A6dsLe15xEX0d1Y/V7GWldPA43hTjBeio+ZZmndtIx5Gb9Btsmjo2rEhqeO/apnQM0 Go5A== X-Gm-Message-State: AOAM530lvHmFu6RqPIjIoVCeqqKwR5DGeEXfOUWMVS500ItXfmX436uo y4vCe7ncxy4qWZgtn5R/FpXwlD48ryrd0xiqmN86aA== X-Google-Smtp-Source: ABdhPJxz/CwlVLg/63PDH3wr31b34EO7lwt0G6vNhxIH2XU6AVZ/PzAYFFsFGdG+2+FCNpMRQGhrpokwxBOtzZfCbNQ= X-Received: by 2002:a25:1906:0:b0:61d:9576:754e with SMTP id 6-20020a251906000000b0061d9576754emr8661270ybz.426.1646671744163; Mon, 07 Mar 2022 08:49:04 -0800 (PST) MIME-Version: 1.0 References: <20220225012819.1807147-1-surenb@google.com> <20220302002150.2113-1-hdanton@sina.com> In-Reply-To: From: Suren Baghdasaryan Date: Mon, 7 Mar 2022 08:48:53 -0800 Message-ID: Subject: Re: [RFC 1/1] mm: page_alloc: replace mm_percpu_wq with kthreads in drain_all_pages To: Petr Mladek Cc: Hillf Danton , linux-mm , LKML , Mel Gorman , Michal Hocko , Tim Murray , Minchan Kim , Johannes Weiner Content-Type: text/plain; charset="UTF-8" X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 2D59B1C000A X-Stat-Signature: z6n8y9paqj19jgqfqcanxkkxxdyyryzh Authentication-Results: imf21.hostedemail.com; dkim=pass header.d=google.com header.s=20210112 header.b=tkjRQRTQ; spf=pass (imf21.hostedemail.com: domain of surenb@google.com designates 209.85.219.182 as permitted sender) smtp.mailfrom=surenb@google.com; dmarc=pass (policy=reject) header.from=google.com X-Rspam-User: X-HE-Tag: 1646671745-720332 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000043, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Mon, Mar 7, 2022 at 8:35 AM Petr Mladek wrote: > > On Wed 2022-03-02 15:06:24, Suren Baghdasaryan wrote: > > On Tue, Mar 1, 2022 at 4:22 PM Hillf Danton wrote: > > > > > > On Thu, 24 Feb 2022 17:28:19 -0800 Suren Baghdasaryan wrote: > > > > Sending as an RFC to confirm if this is the right direction and to > > > > clarify if other tasks currently executed on mm_percpu_wq should be > > > > also moved to kthreads. The patch seems stable in testing but I want > > > > to collect more performance data before submitting a non-RFC version. > > > > > > > > > > > > Currently drain_all_pages uses mm_percpu_wq to drain pages from pcp > > > > list during direct reclaim. The tasks on a workqueue can be delayed > > > > by other tasks in the workqueues using the same per-cpu worker pool. > > > > > > The pending works may be freeing a couple of slabs/pages each. Who knows? > > > > If we are talking about work specifically scheduled on mm_percpu_wq > > then apart from drain_all_pages, mm_percpu_wq is used to execute > > vmstat_update and lru_add_drain_cpu for drainig pagevecs. If OTOH what > > you mean is that the work might be blocked by say kswapd, which is > > freeing memory, then sure, who knows... > > Note that the same worker pool is used by many workqueues. And > work items in per-cpu workqueues are serialized on a single worker. > Another worker is used only when a work goes into a sleeping wait. > > I want to say that "drain_all_pages" are not blocked only by other > works using "mm_percpu_wq" but also by works from many other > workqueues, including "system_wq". > > These works might do anything, including memory allocation, freeing. Ah, I didn't know this (I think you mentioned it in one of your previous replies but I missed it). Thank you for clarifying! > > > > > > > > This results in sizable delays in drain_all_pages when cpus are highly > > > > contended. > > > > Memory management operations designed to relieve memory pressure should > > > > not be allowed to block by other tasks, especially if the task in direct > > > > reclaim has higher priority than the blocking tasks. > > > > > > Wonder why priority is the right cure to tight memory - otherwise it was > > > not a problem given a direct reclaimer of higher priority. > > > > > > Off topic question - why is it making sense from begining for a task of > > > lower priority to peel pages off from another of higher priority if > > > priority is considered in direct reclaim? > > > > The way I understood your question is that you are asking why we have > > to use workqueues of potentially lower priority to drain pages for a > > potentially higher priority process in direct reclaim (which is > > blocked waiting for workqueues to complete draining)? > > If so, IIUC this mechanism was introduced in > > https://lore.kernel.org/all/20170117092954.15413-4-mgorman@techsingularity.net > > to avoid draining from IPI context (CC'ing Mel Gorman to correct me if > > I'm wrong). > > I think the issue here is that in the process we are losing > > information about the priority of the process in direct reclaim, which > > might lead to priority inversion. > > Note that priority of workqueue workers is static. It is defined > by the workqueue parameters. > > kthread_worker API allows to create custom kthreads. The user could > modify the priority as needed. It allows to prevent priority > inversion. It can hardly be achieved by workques where the workers > are heavily shared by unrelated tasks. Yes but I suspect we would not want to dynamically change the priority of the kthreads performing drain_all_pages? I guess we could adjust it based on the highest priority among the tasks waiting for drain_all_pages and that would eliminate priority inversion. However I'm not sure about the possible overhead associated with such dynamic priority adjustments. My RFC sets up the kthreads to be low priority FIFO threads. It's a simplification and I'm not sure that is the right approach here... > > Best Regards, > Petr