From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-5.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,HEADER_FROM_DIFFERENT_DOMAINS,MAILING_LIST_MULTI, SPF_HELO_NONE,SPF_PASS autolearn=no autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id B6A93C433DB for ; Wed, 17 Mar 2021 20:03:26 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id 3372564F2A for ; Wed, 17 Mar 2021 20:03:26 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org 3372564F2A Authentication-Results: mail.kernel.org; dmarc=none (p=none dis=none) header.from=linux-foundation.org Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 6A8D36B006E; Wed, 17 Mar 2021 16:03:25 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 67EF86B0071; Wed, 17 Mar 2021 16:03:25 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 520966B0073; Wed, 17 Mar 2021 16:03:25 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0168.hostedemail.com [216.40.44.168]) by kanga.kvack.org (Postfix) with ESMTP id 3496D6B006E for ; Wed, 17 Mar 2021 16:03:25 -0400 (EDT) Received: from smtpin19.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay04.hostedemail.com (Postfix) with ESMTP id DC2976116 for ; Wed, 17 Mar 2021 20:03:24 +0000 (UTC) X-FDA: 77930440728.19.53F3589 Received: from mail-lf1-f41.google.com (mail-lf1-f41.google.com [209.85.167.41]) by imf29.hostedemail.com (Postfix) with ESMTP id 2FE6B68A7B for ; Wed, 17 Mar 2021 19:27:11 +0000 (UTC) Received: by mail-lf1-f41.google.com with SMTP id b83so667441lfd.11 for ; Wed, 17 Mar 2021 12:27:10 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux-foundation.org; s=google; h=mime-version:references:in-reply-to:from:date:message-id:subject:to :cc; bh=g7PfSQ478H4gL+ePUwNmtoQqfcCoytqCVvRBHHB4kK4=; b=HysRjg5Vm8PBkWcawhMQ9e8rTUA0qSCOQ5N7lqw7nKYmH4Jh2LscIStkT50FshfSWy S+lRXzzxscxhDGbI5SfGjeRh+XLEC5PZHDk6kM9yLX+sRcXCX1J5yrzAm6KGX0JWnuFJ VrHwd5Si03mYfj8Fd7mivuEsEwmiRHXQAZ9g0= X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:mime-version:references:in-reply-to:from:date :message-id:subject:to:cc; bh=g7PfSQ478H4gL+ePUwNmtoQqfcCoytqCVvRBHHB4kK4=; b=W76O1o0Y7dRckfeww7cZIIjXVzCQ513UW+VIOvai5j/muPJ6z6vHl5d/IVrqe2KYWp nCeyZ/AkHwPUGftliArk9cWniqIDX69k9Gd20eveejr/XUW6n75uUC8lJHLTWX3LQsyR fxUAAXZ0hO4sVuaPWwnRuwgIYAw7Lp0S+4WYgEkATWn4CtbXZ3kp8QDn4YuqNJVMjJry S/Rsv/ZWajB3Ug2yLwJoIxfATAOGM9WhFx1uVB0emTkNtK+y6TmoXdGziMPAq5h8PtFf AvtdeXjCFhvL6M/QyRKGQ/p8KoXDP47xkOl1JOAyZrAuCelQ9qv1F8d1vSUoPOK7gZoT zHrA== X-Gm-Message-State: AOAM533jP8/DiXgfcZ3ek/OtQWBDBNH7pq5cn4wf/BlOUOuVH9CDS2pl VZL1WVCpRZYuimeLB/uqzQySFUoZuOLmJA== X-Google-Smtp-Source: ABdhPJyCIQgOS6SO9978jnO7MbFRb6qHBg4wO/rIcu2HBiBx+ReqjfJIx0huNcJ1uBYF2l5QxXQWdQ== X-Received: by 2002:a05:6512:607:: with SMTP id b7mr3166213lfe.463.1616009229014; Wed, 17 Mar 2021 12:27:09 -0700 (PDT) Received: from mail-lj1-f177.google.com (mail-lj1-f177.google.com. [209.85.208.177]) by smtp.gmail.com with ESMTPSA id w28sm3485986lfk.185.2021.03.17.12.27.07 for (version=TLS1_3 cipher=TLS_AES_128_GCM_SHA256 bits=128/128); Wed, 17 Mar 2021 12:27:07 -0700 (PDT) Received: by mail-lj1-f177.google.com with SMTP id u10so4627408lju.7 for ; Wed, 17 Mar 2021 12:27:07 -0700 (PDT) X-Received: by 2002:a2e:864d:: with SMTP id i13mr3180718ljj.48.1616009226982; Wed, 17 Mar 2021 12:27:06 -0700 (PDT) MIME-Version: 1.0 References: <20210317075427.587806-1-npiggin@gmail.com> <89cb49c0-2736-dd4c-f401-4b88c22cc11f@rasmusvillemoes.dk> <1615977781.fijkhk7ep5.astroid@bobo.none> In-Reply-To: <1615977781.fijkhk7ep5.astroid@bobo.none> From: Linus Torvalds Date: Wed, 17 Mar 2021 12:26:51 -0700 X-Gmail-Original-Message-ID: Message-ID: Subject: Re: [PATCH v2] Increase page and bit waitqueue hash size To: Nicholas Piggin Cc: Linux Kernel Mailing List , Rasmus Villemoes , Andrew Morton , Anton Blanchard , Linux-MM , Ingo Molnar Content-Type: text/plain; charset="UTF-8" X-Stat-Signature: 7r5fgduxxatax9xiraybbfiu65p9jisd X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 2FE6B68A7B Received-SPF: none (linuxfoundation.org>: No applicable sender policy available) receiver=imf29; identity=mailfrom; envelope-from=""; helo=mail-lf1-f41.google.com; client-ip=209.85.167.41 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1616009231-133550 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Wed, Mar 17, 2021 at 3:44 AM Nicholas Piggin wrote: > > Argh, because I didn't test small. Sorry I had the BASE_SMALL setting in > another patch and thought it would be a good idea to mash them together. > In hindsight probably not even if it did build. I was going to complain about that code in general. First complaining about the hash being small, and then adding a config option to make it ridiculously much *smaller* seemed wrong to begin with, and didn't make any sense. So no, please don't smash together. In fact, I'd like to see this split up, and with more numbers: - separate out the bit_waitqueue thing that is almost certainly not remotely as critical (and maybe not needed at all) - show the profile number _after_ the patch(es) - explain why you picked the random scaling numbers (21 and 22 for the two different cases)? - give an estimate of how big the array now ends up being for different configurations. I think it ends up using that "scale" factor of 21, and basically being "memory size >> 21" and then rounding up to a power of two. And honestly, I'm not sure that makes much sense. So for a 1GB machine we get the same as we used to for the bit waitqueue (twice as many for the page waitqueue) , but if you run on some smaller setup, you apparently can end up with just a couple of buckets. So I'd feel a lot better about this if I saw the numbers, and got the feeling that the patch actually tries to take legacy machines into account. And even on a big machine, what's the advantage of scaling perfectly with memory. If you have a terabyte of RAM, why would you need half a million hash entries (if I did the math right), and use 4GB of memory on it? The contention doesn't go up by amount of memory, it goes up roughly by number of threads, and the two are very seldom really all that linearly connected. So honestly, I'd like to see more reasonable numbers. I'd like to see what the impact of just raising the hash bit size from 8 to 16 is on that big machine. Maybe still using alloc_large_system_hash(), but using a low-imit of 8 (our traditional very old number that hasn't been a problem even on small machines), and a high-limit of 16 or something. And if you want even more, I really really want that justified by the performance / profile numbers. And does does that "bit_waitqueue" really merit updating AT ALL? It's almost entirely unused these days. I think maybe the page lock code used to use that, but then realized it had more specialized needs, so now it's separate. So can we split that bit-waitqueue thing up from the page waitqueue changes? They have basically nothing in common except for a history, and I think they should be treated separately (including the explanation for what actually hits the bottleneck). Linus