From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 8B320C05027 for ; Wed, 1 Feb 2023 16:27:34 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id ECF676B0072; Wed, 1 Feb 2023 11:27:33 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id E7F526B0073; Wed, 1 Feb 2023 11:27:33 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id D47996B0074; Wed, 1 Feb 2023 11:27:33 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id C6B446B0072 for ; Wed, 1 Feb 2023 11:27:33 -0500 (EST) Received: from smtpin24.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id A2EFE1A0F0B for ; Wed, 1 Feb 2023 16:27:33 +0000 (UTC) X-FDA: 80419253586.24.5AC7716 Received: from mailout2n.rrzn.uni-hannover.de (mailout2n.rrzn.uni-hannover.de [130.75.2.113]) by imf18.hostedemail.com (Postfix) with ESMTP id E944D1C0011 for ; Wed, 1 Feb 2023 16:27:30 +0000 (UTC) Authentication-Results: imf18.hostedemail.com; dkim=none; spf=pass (imf18.hostedemail.com: domain of halbuer@sra.uni-hannover.de designates 130.75.2.113 as permitted sender) smtp.mailfrom=halbuer@sra.uni-hannover.de; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1675268851; a=rsa-sha256; cv=none; b=gGsLnck2YBHAvKMJ8l3ez0/+L7L3qZH3vlIQMfuQmCebWF5mbGCkUYJzNCoVdKZKzWGOGQ pi/2jA1u/JUliYOfMVjEbR1gsrk+ZY1xAEF5+rAma9SxSZasjppjw3qNhcXwn4wnMtgVqg WPILdaq3Xd8iOdGKa094V4gLTwiOT8o= ARC-Authentication-Results: i=1; imf18.hostedemail.com; dkim=none; spf=pass (imf18.hostedemail.com: domain of halbuer@sra.uni-hannover.de designates 130.75.2.113 as permitted sender) smtp.mailfrom=halbuer@sra.uni-hannover.de; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1675268851; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-transfer-encoding:content-transfer-encoding: in-reply-to:references; bh=eOGQxS4tS8N2yYWy5jyACnXBKsCltGqL20U9YX4QZHc=; b=NvZNkTu8nv6AhdxnTVtJFLOFrvE7JaxJBoxbx8LZ5DqeneFk7PaaYDLwxqL5rI6NmbvBJc 2KBfBnYyYKwsKKRUW6peSYuqeNTA9rRZ0idjEP5rSxZN6X5wuBe5ekZsZysP8shNP8UW4G chxQshd2P1TRyZy9w+eLBscmZfmYiss= Received: from beifus.sra.uni-hannover.de (lab.sra.uni-hannover.de [130.75.33.87]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature RSA-PSS (2048 bits) server-digest SHA256) (No client certificate requested) by mailout2n.rrzn.uni-hannover.de (Postfix) with ESMTPSA id CD1991F4B8; Wed, 1 Feb 2023 17:27:28 +0100 (CET) From: Alexander Halbuer To: akpm@linux-foundation.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org Cc: Alexander Halbuer Subject: [PATCH] mm: reduce lock contention of pcp buffer refill Date: Wed, 1 Feb 2023 17:25:49 +0100 Message-Id: <20230201162549.68384-1-halbuer@sra.uni-hannover.de> X-Mailer: git-send-email 2.39.1 MIME-Version: 1.0 Content-Transfer-Encoding: 8bit X-Virus-Scanned: clamav-milter 0.103.7 at mailout2n X-Virus-Status: Clean X-Rspam-User: X-Rspamd-Queue-Id: E944D1C0011 X-Rspamd-Server: rspam01 X-Stat-Signature: 3tyxtr4tiehyp3t7qc8t86dg88dwbps9 X-HE-Tag: 1675268850-900010 X-HE-Meta: U2FsdGVkX19S5dP/Ank5RvQc8saYoFi4uubXwsk2Wes5cBJEYm69GPl7w0g7+zPnWCNWfWWbFVvGs7eVZuWnSBYP+P6/AxXiNh8tEk8+PWWeCFh/guY2wxFwpxrwnz3pYZm9V6jTRzCclNi/JyxnjnIJSZUenNaYOBRez8KC5+eI4cvkbvIGjRQpWuOJ922fJkaSosnBPnA6DTU66nGtp+uBPkFH5tXA0F/AqmH+xYQwGgWkCiJBszpgOQ/RDj6vdDVV2HpopXLHn4slgsxdDJCIM8HDxQo0o01pEvvJ4M/h+xLSBwYYv0avgRKUd7SjPJoR7p37xSgbrM9OGF5xLY8dG8l9yndRBY6V6fl+jtb57yOO/VOo61CGIriVr8EFk2FXWJu/rCm3/pRiTgBxOVpnPMMc/hJJWmRHItZ5XrcTv38csbaL/bsbajq8TwPKeYli9HezYBBHCQiH7ifLDxW30dF/u69gFHflGfFqBVBmkP8zww+zW+aTGbOKPySvz0ABJ5VHJVToonPTBHByJecSbfJxSPdUVgwh76/NNSrN+EWHO3ymT6hgLQ/NSlZaYpMuunc1PMJ3Pt2TcNIDfdHGAyAJFtywt233ntUnttyDBExkjkArMYjvJxdjxHgv50SFQOiyqKLMOYGOtlNVGMqNZpGvkpnHN3Qdg4iLRkcYagwa5MNfHk6c07UNZjAbi+cpHH6QpBxe/RHSgAOyga9MFC1WmJ3pNWcLfpnrSIGVv5Yee70cXXNt3D+ov+6CJQt/Nzbqpae6QiIDfIo2GEXevUurpnyLEa99gAZNrcyzcUXEHfL9BFst/JX9+Oeo3fK4dH8uciqSUFkhRmWzZ/2QtSg31H9L5HkuNH3fF0uJaK6on1V+3v3LIcK0X7xr1//YCWo7T+Ex4vunbJgORuHKoPDiltkHnk0ZgQyLq//YaqSqdl8LEDrj6Ku+oE2yRqb8Loa6xxWPq/dY9Ii ENOcbz4m TK5XZjbctB6r307ZiaACWQg8rPcdNNimJ0JM8lySF+iYbokM89MuVug2liJ25ZNEV9xKC1XJhUKDX+ymO57TcICj0lMQ0nogsier1 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: The `rmqueue_bulk` function batches the allocation of multiple elements to refill the per-CPU buffers into a single hold of the zone lock. Each element is allocated and checked using the `check_pcp_refill` function. The check touches every related struct page which is especially expensive for higher order allocations (huge pages). This patch reduces the time holding the lock by moving the check out of the critical section similar to the `rmqueue_buddy` function which allocates a single element. Measurements of parallel allocation-heavy workloads show a reduction of the average huge page allocation latency of 50 percent for two cores and nearly 90 percent for 24 cores. Signed-off-by: Alexander Halbuer --- mm/page_alloc.c | 22 ++++++++++++++++++---- 1 file changed, 18 insertions(+), 4 deletions(-) diff --git a/mm/page_alloc.c b/mm/page_alloc.c index 0745aedebb37..4b80438b1f59 100644 --- a/mm/page_alloc.c +++ b/mm/page_alloc.c @@ -3119,6 +3119,8 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, { unsigned long flags; int i, allocated = 0; + struct list_head *prev_tail = list->prev; + struct page *pos, *n; spin_lock_irqsave(&zone->lock, flags); for (i = 0; i < count; ++i) { @@ -3127,9 +3129,6 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, if (unlikely(page == NULL)) break; - if (unlikely(check_pcp_refill(page, order))) - continue; - /* * Split buddy pages returned by expand() are received here in * physical page order. The page is added to the tail of @@ -3141,7 +3140,6 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, * pages are ordered properly. */ list_add_tail(&page->pcp_list, list); - allocated++; if (is_migrate_cma(get_pcppage_migratetype(page))) __mod_zone_page_state(zone, NR_FREE_CMA_PAGES, -(1 << order)); @@ -3155,6 +3153,22 @@ static int rmqueue_bulk(struct zone *zone, unsigned int order, */ __mod_zone_page_state(zone, NR_FREE_PAGES, -(i << order)); spin_unlock_irqrestore(&zone->lock, flags); + + /* + * Pages are appended to the pcp list without checking to reduce the + * time holding the zone lock. Checking the appended pages happens right + * after the critical section while still holding the pcp lock. + */ + pos = list_first_entry(prev_tail, struct page, pcp_list); + list_for_each_entry_safe_from(pos, n, list, pcp_list) { + if (unlikely(check_pcp_refill(pos, order))) { + list_del(&pos->pcp_list); + continue; + } + + allocated++; + } + return allocated; } -- 2.39.1