From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org X-Spam-Level: X-Spam-Status: No, score=-12.8 required=3.0 tests=BAYES_00,DKIM_SIGNED, DKIM_VALID,DKIM_VALID_AU,FREEMAIL_FORGED_FROMDOMAIN,FREEMAIL_FROM, HEADER_FROM_DIFFERENT_DOMAINS,INCLUDES_CR_TRAILER,INCLUDES_PATCH, MAILING_LIST_MULTI,SPF_HELO_NONE,SPF_PASS autolearn=ham autolearn_force=no version=3.4.0 Received: from mail.kernel.org (mail.kernel.org [198.145.29.99]) by smtp.lore.kernel.org (Postfix) with ESMTP id 60357C433E0 for ; Wed, 17 Mar 2021 10:44:50 +0000 (UTC) Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by mail.kernel.org (Postfix) with ESMTP id D3B4264F21 for ; Wed, 17 Mar 2021 10:44:49 +0000 (UTC) DMARC-Filter: OpenDMARC Filter v1.3.2 mail.kernel.org D3B4264F21 Authentication-Results: mail.kernel.org; dmarc=fail (p=none dis=none) header.from=gmail.com Authentication-Results: mail.kernel.org; spf=pass smtp.mailfrom=owner-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix) id 549A46B006E; Wed, 17 Mar 2021 06:44:49 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 4FA1B6B0070; Wed, 17 Mar 2021 06:44:49 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 39A696B0071; Wed, 17 Mar 2021 06:44:49 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from forelay.hostedemail.com (smtprelay0189.hostedemail.com [216.40.44.189]) by kanga.kvack.org (Postfix) with ESMTP id 1E58F6B006E for ; Wed, 17 Mar 2021 06:44:49 -0400 (EDT) Received: from smtpin22.hostedemail.com (10.5.19.251.rfc1918.com [10.5.19.251]) by forelay03.hostedemail.com (Postfix) with ESMTP id D19448249980 for ; Wed, 17 Mar 2021 10:44:48 +0000 (UTC) X-FDA: 77929033056.22.C461680 Received: from mail-pl1-f175.google.com (mail-pl1-f175.google.com [209.85.214.175]) by imf29.hostedemail.com (Postfix) with ESMTP id 5BDD9DA for ; Wed, 17 Mar 2021 10:44:48 +0000 (UTC) Received: by mail-pl1-f175.google.com with SMTP id s7so577298plg.5 for ; Wed, 17 Mar 2021 03:44:48 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=date:from:subject:to:cc:references:in-reply-to:mime-version :message-id:content-transfer-encoding; bh=QspAW94trZprGCB1fixf21vQlfCFlRs4FPYE7jk2m+Q=; b=au1LkU25+cE4xEc1QYxw+rChmMwCu3PhGyEsX/Sv+lzn5c4GA/+CXIcwy7CfBmsb2u g43At1dmgA9Iesqa1t07W4RMsQ5cHbsr1BAHEhKWK90q8YkCNcL4h1ulo2LmyADA3wGo 8MXRa6TFRyGNNadAve8uuLJzTaK0Hr54zAcE+G4smQM0CDsa2dbXrmo9L7JiJ+Ta3U+9 sMUb1EEP4LmvpSyBpxvPLTZhh4lZKjFCKdhisFfnlFF7csmQEIXNuBpGZC7qQVffP8s/ 1NlUWYpp9U5oQKo9KCvveNGOoEhdBUU2xLpVZiifCRRjSAtDqty37yOOBbk7FYJ5t02i sSrw== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:date:from:subject:to:cc:references:in-reply-to :mime-version:message-id:content-transfer-encoding; bh=QspAW94trZprGCB1fixf21vQlfCFlRs4FPYE7jk2m+Q=; b=qYSYWrt/0dKN3m8bYDZZrHeSHCyMbkXbWN74RAN60E7TC0irx5gc+h6bXY8LYODUkQ Ryc7tCzJkHG+5HmMcHBA9FZOOUTYKRRgft9bPKY9d2RUX07h4M4XAbWFZlfNInNWRGQ8 dnz0s2gNPlk2ilEJrPnN6wT+V/LjS2D1A7EzBPRzd8A66Wz1EW6zi+9M2ND8TjBYDyNO snW2Ay8OmpfFApo7LNez5vFtq137i5p+t7hv/qhusGrxRJgXj5JQuAHD9ZlK2vi/loXa U7s1tAD2pqTgqqgSlE9clMdjDd/FimVWBfOslDFWnM2LhTNKN4jkJ2Iiznbq9ST17q8G hJDQ== X-Gm-Message-State: AOAM531X39LppEuUgueuuDyRG6wV6GzxWr923JwW1ZrDi1i/pS+c3Vxa IOVNMfpgZBVgnnxOn4rC5rI= X-Google-Smtp-Source: ABdhPJzvjphxuko2KMq6oGvr48Yr1UWJ/A9mnABdlXJWrxnsuWUwTbgHhXiD+YRtg37oHJAPq8PGSw== X-Received: by 2002:a17:90b:909:: with SMTP id bo9mr3904206pjb.107.1615977887240; Wed, 17 Mar 2021 03:44:47 -0700 (PDT) Received: from localhost (58-6-239-121.tpgi.com.au. [58.6.239.121]) by smtp.gmail.com with ESMTPSA id k17sm13281555pfa.68.2021.03.17.03.44.45 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Wed, 17 Mar 2021 03:44:46 -0700 (PDT) Date: Wed, 17 Mar 2021 20:44:41 +1000 From: Nicholas Piggin Subject: Re: [PATCH v2] Increase page and bit waitqueue hash size To: linux-kernel@vger.kernel.org, Rasmus Villemoes Cc: Andrew Morton , Anton Blanchard , linux-mm@kvack.org, Ingo Molnar , Linus Torvalds References: <20210317075427.587806-1-npiggin@gmail.com> <89cb49c0-2736-dd4c-f401-4b88c22cc11f@rasmusvillemoes.dk> In-Reply-To: <89cb49c0-2736-dd4c-f401-4b88c22cc11f@rasmusvillemoes.dk> MIME-Version: 1.0 Message-Id: <1615977781.fijkhk7ep5.astroid@bobo.none> Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable X-Stat-Signature: xaqppsi1c68j4txh8tiugasjkxip4tys X-Rspamd-Server: rspam05 X-Rspamd-Queue-Id: 5BDD9DA Received-SPF: none (gmail.com>: No applicable sender policy available) receiver=imf29; identity=mailfrom; envelope-from=""; helo=mail-pl1-f175.google.com; client-ip=209.85.214.175 X-HE-DKIM-Result: pass/pass X-HE-Tag: 1615977888-898384 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: Excerpts from Rasmus Villemoes's message of March 17, 2021 8:12 pm: > On 17/03/2021 08.54, Nicholas Piggin wrote: >=20 >> +#if CONFIG_BASE_SMALL >> +static const unsigned int page_wait_table_bits =3D 4; >> static wait_queue_head_t page_wait_table[PAGE_WAIT_TABLE_SIZE] __cachel= ine_aligned; >=20 >> =20 >> + if (!CONFIG_BASE_SMALL) { >> + page_wait_table =3D alloc_large_system_hash("page waitqueue hash", >> + sizeof(wait_queue_head_t), >> + 0, >=20 > So, how does the compiler not scream at you for assigning to an array, > even if it's inside an if (0)? >=20 Argh, because I didn't test small. Sorry I had the BASE_SMALL setting in=20 another patch and thought it would be a good idea to mash them together.=20 In hindsight probably not even if it did build. Thanks, Nick -- [PATCH v3] Increase page and bit waitqueue hash size The page waitqueue hash is a bit small (256 entries) on very big systems. A 16 socket 1536 thread POWER9 system was found to encounter hash collisions and excessive time in waitqueue locking at times. This was intermittent and hard to reproduce easily with the setup we had (very little real IO capacity). The theory is that sometimes (depending on allocation luck) important pages would happen to collide a lot in the hash, slowing down pag= e locking, causing the problem to snowball. An small test case was made where threads would write and fsync different pages, generating just a small amount of contention across many pages. Increasing page waitqueue hash size to 262144 entries increased throughput by 182% while also reducing standard deviation 3x. perf before the increase= : 36.23% [k] _raw_spin_lock_irqsave - - | |--34.60%--wake_up_page_bit | 0 | iomap_write_end.isra.38 | iomap_write_actor | iomap_apply | iomap_file_buffered_write | xfs_file_buffered_aio_write | new_sync_write 17.93% [k] native_queued_spin_lock_slowpath - - | |--16.74%--_raw_spin_lock_irqsave | | | --16.44%--wake_up_page_bit | iomap_write_end.isra.38 | iomap_write_actor | iomap_apply | iomap_file_buffered_write | xfs_file_buffered_aio_write This patch uses alloc_large_system_hash to allocate a bigger system hash that scales somewhat with memory size. The bit/var wait-queue is also changed to keep code matching, albiet with a smaller scale factor. This hash could be made per-node, which may help reduce remote accesses on well localised workloads, but that adds some complexity with indexing and hotplug, so until we get a less artificial workload to test with, keep it simple. Signed-off-by: Nicholas Piggin --- kernel/sched/wait_bit.c | 25 ++++++++++++++++++------- mm/filemap.c | 16 ++++++++++++++-- 2 files changed, 32 insertions(+), 9 deletions(-) diff --git a/kernel/sched/wait_bit.c b/kernel/sched/wait_bit.c index 02ce292b9bc0..3cc5fa552516 100644 --- a/kernel/sched/wait_bit.c +++ b/kernel/sched/wait_bit.c @@ -2,19 +2,20 @@ /* * The implementation of the wait_bit*() and related waiting APIs: */ +#include #include "sched.h" =20 -#define WAIT_TABLE_BITS 8 -#define WAIT_TABLE_SIZE (1 << WAIT_TABLE_BITS) - -static wait_queue_head_t bit_wait_table[WAIT_TABLE_SIZE] __cacheline_align= ed; +#define BIT_WAIT_TABLE_SIZE (1 << BIT_WAIT_TABLE_BITS) +#define BIT_WAIT_TABLE_BITS bit_wait_table_bits +static unsigned int bit_wait_table_bits __ro_after_init; +static wait_queue_head_t *bit_wait_table __ro_after_init; =20 wait_queue_head_t *bit_waitqueue(void *word, int bit) { const int shift =3D BITS_PER_LONG =3D=3D 32 ? 5 : 6; unsigned long val =3D (unsigned long)word << shift | bit; =20 - return bit_wait_table + hash_long(val, WAIT_TABLE_BITS); + return bit_wait_table + hash_long(val, BIT_WAIT_TABLE_BITS); } EXPORT_SYMBOL(bit_waitqueue); =20 @@ -152,7 +153,7 @@ EXPORT_SYMBOL(wake_up_bit); =20 wait_queue_head_t *__var_waitqueue(void *p) { - return bit_wait_table + hash_ptr(p, WAIT_TABLE_BITS); + return bit_wait_table + hash_ptr(p, BIT_WAIT_TABLE_BITS); } EXPORT_SYMBOL(__var_waitqueue); =20 @@ -246,6 +247,16 @@ void __init wait_bit_init(void) { int i; =20 - for (i =3D 0; i < WAIT_TABLE_SIZE; i++) + bit_wait_table =3D alloc_large_system_hash("bit waitqueue hash", + sizeof(wait_queue_head_t), + 0, + 22, + 0, + &bit_wait_table_bits, + NULL, + 0, + 0); + + for (i =3D 0; i < BIT_WAIT_TABLE_SIZE; i++) init_waitqueue_head(bit_wait_table + i); } diff --git a/mm/filemap.c b/mm/filemap.c index 43700480d897..df5a3ef4031b 100644 --- a/mm/filemap.c +++ b/mm/filemap.c @@ -34,6 +34,7 @@ #include #include #include +#include #include #include #include @@ -990,9 +991,10 @@ EXPORT_SYMBOL(__page_cache_alloc); * at a cost of "thundering herd" phenomena during rare hash * collisions. */ -#define PAGE_WAIT_TABLE_BITS 8 #define PAGE_WAIT_TABLE_SIZE (1 << PAGE_WAIT_TABLE_BITS) -static wait_queue_head_t page_wait_table[PAGE_WAIT_TABLE_SIZE] __cacheline= _aligned; +#define PAGE_WAIT_TABLE_BITS page_wait_table_bits +static unsigned int page_wait_table_bits __ro_after_init; +static wait_queue_head_t *page_wait_table __ro_after_init; =20 static wait_queue_head_t *page_waitqueue(struct page *page) { @@ -1003,6 +1005,16 @@ void __init pagecache_init(void) { int i; =20 + page_wait_table =3D alloc_large_system_hash("page waitqueue hash", + sizeof(wait_queue_head_t), + 0, + 21, + 0, + &page_wait_table_bits, + NULL, + 0, + 0); + for (i =3D 0; i < PAGE_WAIT_TABLE_SIZE; i++) init_waitqueue_head(&page_wait_table[i]); =20 --=20 2.23.0 > Rasmus >=20