From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-4.0 required=3.0 tests=INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4AB00C2BA16 for ; Tue, 7 Apr 2020 07:24:44 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 05F822064A for ; Tue, 7 Apr 2020 07:24:43 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 05F822064A Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=kernel.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 885038E0011; Tue, 7 Apr 2020 03:24:43 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 835368E0001; Tue, 7 Apr 2020 03:24:43 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 724BF8E0011; Tue, 7 Apr 2020 03:24:43 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0195.hostedemail.com [216.40.44.195]) by kanga.kvack.org (Postfix) with ESMTP id 5BC338E0001 for ; Tue, 7 Apr 2020 03:24:43 -0400 (EDT) Received: from smtpin21.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id 1C38D801ABC2 for ; Tue, 7 Apr 2020 07:24:43 +0000 (UTC) X-FDA: 76680221646.21.help58_2922d1b3d1020 X-HE-Tag: help58_2922d1b3d1020 X-Filterd-Recvd-Size: 6361 Received: from mail-wr1-f65.google.com (mail-wr1-f65.google.com [209.85.221.65]) by imf35.hostedemail.com (Postfix) with ESMTP for ; Tue, 7 Apr 2020 07:24:42 +0000 (UTC) Received: by mail-wr1-f65.google.com with SMTP id k1so2643545wrm.3 for ; Tue, 07 Apr 2020 00:24:42 -0700 (PDT) X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:to:cc:subject:message-id:references :mime-version:content-disposition:in-reply-to; bh=n6FnIuOZPi/ruTZydug6G1Y7/h7rJsp6Y+RjBmISzyY=; b=juV4++ALi85DmLwf+lOvFATt+CCubElsjpy+AZa19EMNXY8RDS2ssfVU/jG17Iuu6M aSCOfrYGq/x8LclzghoJ2lsI8mYSJ7VAYJSO/gDX4GcpAWAKSXgK9h7ycE8iwGihlRgM fhrGK3kRqe5X2yG5KyOiXpqYimXf6RRVla/2OnZ2ajVhiYY5BsJ3Vx475VWnssoQWYII EJYZZRepC5XA2KW20V1I4aQ6hiWuJo9M18qmx1IJvxBAlqfXEJnaWqrNCJA8zV8hQiBd KtqxPg0tGGfafZJTI02Cimt/pu7FLUGb05wf2fkyCr7I+rI10k1SWFMTW10mKrYI6oe0 n5Cw== X-Gm-Message-State: AGi0PubFFnqtzg7WFhA4HGkeRPa/N1Ao4CZOuH1/ok6Lcqb5ARKO92Yg /ZLkpc9argrGHjSA55BuME0= X-Google-Smtp-Source: APiQypKfOL4a/WUufFiUv2U4j0JDcB1NNo6X9GIZA69ARU1CAw7jrsKOaFLG0D0a7uyrutia1pxTzA== X-Received: by 2002:adf:9e8c:: with SMTP id a12mr1120975wrf.273.1586244281615; Tue, 07 Apr 2020 00:24:41 -0700 (PDT) Received: from localhost (ip-37-188-180-223.eurotel.cz. [37.188.180.223]) by smtp.gmail.com with ESMTPSA id n64sm1076078wme.45.2020.04.07.00.24.40 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Tue, 07 Apr 2020 00:24:40 -0700 (PDT) Date: Tue, 7 Apr 2020 09:24:39 +0200 From: Michal Hocko To: NeilBrown Cc: John Hubbard , David Rientjes , Andrew Morton , Joel Fernandes , "Paul E. McKenney" , linux-mm@kvack.org, LKML Subject: Re: [PATCH 1/2] mm: clarify __GFP_MEMALLOC usage Message-ID: <20200407072439.GG18914@dhcp22.suse.cz> References: <20200403083543.11552-1-mhocko@kernel.org> <20200403083543.11552-2-mhocko@kernel.org> <87blo8xnz2.fsf@notabene.neil.brown.name> <20200406070137.GC19426@dhcp22.suse.cz> <4f861f07-4b47-8ddc-f783-10201ea302d3@nvidia.com> <875zecw1n6.fsf@notabene.neil.brown.name> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <875zecw1n6.fsf@notabene.neil.brown.name> X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue 07-04-20 11:00:29, Neil Brown wrote: > On Mon, Apr 06 2020, John Hubbard wrote: > > > On 4/6/20 12:01 AM, Michal Hocko wrote: [...] > >> diff --git a/include/linux/gfp.h b/include/linux/gfp.h > >> index e5b817cb86e7..9cacef1a3ee0 100644 > >> --- a/include/linux/gfp.h > >> +++ b/include/linux/gfp.h > >> @@ -110,6 +110,11 @@ struct vm_area_struct; > >> * the caller guarantees the allocation will allow more memory to be freed > >> * very shortly e.g. process exiting or swapping. Users either should > >> * be the MM or co-ordinating closely with the VM (e.g. swap over NFS). > >> + * Users of this flag have to be extremely careful to not deplete the reserve > >> + * completely and implement a throttling mechanism which controls the consumption > >> + * of the reserve based on the amount of freed memory. > >> + * Usage of a pre-allocated pool (e.g. mempool) should be always considered before > >> + * using this flag. > > I think this version is pretty good. Thanks! I will stick with it then. [...] > I think it is hard to write rules because the rules are a bit spongey. Exactly! And the more specific we are the more likely people are going to follow literally. And we do not want that. We want people to be aware of the limitation but want them to think hard before using the flag. > With mempools, we have a nice clear rule. When you allocate from a > mempool you must have a clear path to freeing that allocation which will > not block on memory allocation except from a subordinate mempool. This > implies a partial ordering between mempools. When you have layered > block devices the path through the layers from filesystem down to > hardware defines the order. It isn't enforced, but it is quite easy to > reason about. > > GFP_MEMALLOC effectively provides multiple mempools. So it could > theoretically deadlock if multiple long dependency chains > happened. i.e. if 1000 threads each make a GFP_MEMALLOC allocation and > then need to make another one before the first can be freed - then you > hit problems. There is no formal way to guarantee that this doesn't > happen. We just say "be gentle" and minimize the users of this flag, > and keep more memory in reserve than we really need. > Note that 'threads' here might not be Linux tasks. If you have an IO > request that proceed asynchronously, moving from queue to queue and > being handled by different task, then each one is a "thread" for the > purpose of understanding mem-alloc dependency. > > So maybe what I really should focus on is not how quickly things happen, > but how many happen concurrently. The idea of throttling is to allow > previous requests to complete before we start too many more. > > With Swap-over-NFS, some of the things that might need to be allocated > are routing table entries. These scale with the number of NFS servers > rather than the number of IO requests, so they are not going to cause > concurrency problems. > We also need memory to store replies, but these never exceed the number > of pending requests, so there is limited concurrency there. > NFS can send a lot of requests in parallel, but the main limit is the > RPC "slot table" and while that grows dynamically, it does so with > GFP_NOFS, so it can block or fail (I wonder if that should explicitly > disable the use of the reserves). > > So there a limit on concurrency imposed by non-GFP_MEMALLOC allocations This really makes sense to mention in the allocation manual (Documentation/core-api/memory-allocation.rst) as suggested by John. Care to make it into a patch? -- Michal Hocko SUSE Labs