From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 71EADCCA479 for ; Fri, 17 Jun 2022 17:50:27 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id E5CCF6B0083; Fri, 17 Jun 2022 13:50:26 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id DAEB76B007B; Fri, 17 Jun 2022 13:50:26 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id C28226B0083; Fri, 17 Jun 2022 13:50:26 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id A4C6F6B007B for ; Fri, 17 Jun 2022 13:50:26 -0400 (EDT) Received: from smtpin13.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay01.hostedemail.com (Postfix) with ESMTP id 6CC1F60925 for ; Fri, 17 Jun 2022 17:50:26 +0000 (UTC) X-FDA: 79588467252.13.6EAD7DF Received: from casper.infradead.org (casper.infradead.org [90.155.50.34]) by imf04.hostedemail.com (Postfix) with ESMTP id DACAB4008E for ; Fri, 17 Jun 2022 17:50:25 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=infradead.org; s=casper.20170209; h=Content-Transfer-Encoding:MIME-Version: References:In-Reply-To:Message-Id:Date:Subject:Cc:To:From:Sender:Reply-To: Content-Type:Content-ID:Content-Description; bh=Y585jLSB8I2ngwz0lCZ0YAGr1TvrbiJzuvo1jnhD1YU=; b=eJI8isJNzuJyQw7rsv8YJ9Hhvf 7x1M7YShOlN7guXpfNsD9ShLxWMn3Os9KXUe+cWZo6rqYWM4ioMO/aYbrwwaM6R7qZ+tFiPB8YOc4 Tet8AVUmRAQ0YLTKQK2qbdo7/hWm+rd1SWtHt0qqRm46rdYTIoE+g6Ns5pQjFUDjR2YrEnf5u5gOm ZbSOV4qsQiaR9CVXfphaFqtHtPnLdpzHTGwyen7ykdkOdEV/dqh88dvAy4zZ+sb7Gn6X5UNmaGYCA nPJEOluSfVyVQWWBCnQojiRTYaGyx7AqtZmL34fSo+wcRC/LteBdDxWKDT3G9OUhR/OjpYLNG9ryd P3wIeUgg==; Received: from willy by casper.infradead.org with local (Exim 4.94.2 #2 (Red Hat Linux)) id 1o2G6y-0030aT-E4; Fri, 17 Jun 2022 17:50:24 +0000 From: "Matthew Wilcox (Oracle)" To: Andrew Morton Cc: "Matthew Wilcox (Oracle)" , linux-mm@kvack.org Subject: [PATCH 10/22] mm/swap: Pull the CPU conditional out of __lru_add_drain_all() Date: Fri, 17 Jun 2022 18:50:08 +0100 Message-Id: <20220617175020.717127-11-willy@infradead.org> X-Mailer: git-send-email 2.31.1 In-Reply-To: <20220617175020.717127-1-willy@infradead.org> References: <20220617175020.717127-1-willy@infradead.org> MIME-Version: 1.0 Content-Transfer-Encoding: 8bit ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1655488226; a=rsa-sha256; cv=none; b=VcUMT/I/OY8KI81we3/N2YaJLGcktbZkroosojeK63DMtLu24ewJ6DnQLYFKmG7as2C+7U CgXrYrsa8XAxNapIeU43+t1MtedagVw1KrSwy+oBa5EEVk2sCOAqLTjyI11WJoahIDnWBb hvNTeFoQKIM/ZxrJh74KQGQXXp4qwxU= ARC-Authentication-Results: i=1; imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=eJI8isJN; dmarc=none; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1655488226; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=Y585jLSB8I2ngwz0lCZ0YAGr1TvrbiJzuvo1jnhD1YU=; b=26+Yd82/paMgSr6kXGEmkl+x0qeoCmNJhfjUkOrNE/+zjQsb10a34mQnqudf6U+NwicuML QA3fkTapCfqJGhpj3ug/nRGMc6+ZyNsMTunLtV7+AEjCXKDEtzc8l/iaNqdC6n86FuBkjh fXyNUOTTivCY7zoAo2ZiuSNpEGGS8Ao= Authentication-Results: imf04.hostedemail.com; dkim=pass header.d=infradead.org header.s=casper.20170209 header.b=eJI8isJN; dmarc=none; spf=none (imf04.hostedemail.com: domain of willy@infradead.org has no SPF policy when checking 90.155.50.34) smtp.mailfrom=willy@infradead.org X-Rspam-User: X-Stat-Signature: m5duo7qykf476rdk9it13osmj1oqpiuy X-Rspamd-Queue-Id: DACAB4008E X-Rspamd-Server: rspam08 X-HE-Tag: 1655488225-115642 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The function is too long, so pull this complicated conditional out into cpu_needs_drain(). This ends up shrinking the text by 14 bytes, by allowing GCC to cache the result of calling per_cpu() instead of relocating each lookup individually. Signed-off-by: Matthew Wilcox (Oracle) --- mm/swap.c | 24 ++++++++++++++++-------- 1 file changed, 16 insertions(+), 8 deletions(-) diff --git a/mm/swap.c b/mm/swap.c index e65a195c184b..7966aa6bdd3f 100644 --- a/mm/swap.c +++ b/mm/swap.c @@ -765,6 +765,21 @@ static void lru_add_drain_per_cpu(struct work_struct *dummy) lru_add_and_bh_lrus_drain(); } +static bool cpu_needs_drain(unsigned int cpu) +{ + struct cpu_fbatches *fbatches = &per_cpu(cpu_fbatches, cpu); + + /* Check these in order of likelihood that they're not zero */ + return folio_batch_count(&fbatches->lru_add) || + data_race(folio_batch_count(&per_cpu(lru_rotate.fbatch, cpu))) || + folio_batch_count(&fbatches->lru_deactivate_file) || + folio_batch_count(&fbatches->lru_deactivate) || + folio_batch_count(&fbatches->lru_lazyfree) || + folio_batch_count(&fbatches->activate) || + need_mlock_page_drain(cpu) || + has_bh_in_lru(cpu, NULL); +} + /* * Doesn't need any cpu hotplug locking because we do rely on per-cpu * kworkers being shut down before our page_alloc_cpu_dead callback is @@ -849,14 +864,7 @@ static inline void __lru_add_drain_all(bool force_all_cpus) for_each_online_cpu(cpu) { struct work_struct *work = &per_cpu(lru_add_drain_work, cpu); - if (folio_batch_count(&per_cpu(cpu_fbatches.lru_add, cpu)) || - data_race(folio_batch_count(&per_cpu(lru_rotate.fbatch, cpu))) || - folio_batch_count(&per_cpu(cpu_fbatches.lru_deactivate_file, cpu)) || - folio_batch_count(&per_cpu(cpu_fbatches.lru_deactivate, cpu)) || - folio_batch_count(&per_cpu(cpu_fbatches.lru_lazyfree, cpu)) || - folio_batch_count(&per_cpu(cpu_fbatches.activate, cpu)) || - need_mlock_page_drain(cpu) || - has_bh_in_lru(cpu, NULL)) { + if (cpu_needs_drain(cpu)) { INIT_WORK(work, lru_add_drain_per_cpu); queue_work_on(cpu, mm_percpu_wq, work); __cpumask_set_cpu(cpu, &has_work); -- 2.35.1