From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id D2500C4332F for ; Tue, 20 Dec 2022 23:56:47 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 20AD38E0002; Tue, 20 Dec 2022 18:56:47 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 1BBC78E0001; Tue, 20 Dec 2022 18:56:47 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 0A9E58E0002; Tue, 20 Dec 2022 18:56:47 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0015.hostedemail.com [216.40.44.15]) by kanga.kvack.org (Postfix) with ESMTP id F0AB78E0001 for ; Tue, 20 Dec 2022 18:56:46 -0500 (EST) Received: from smtpin15.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay08.hostedemail.com (Postfix) with ESMTP id 9D808140564 for ; Tue, 20 Dec 2022 23:56:46 +0000 (UTC) X-FDA: 80264347212.15.B5B809E Received: from out-154.mta0.migadu.com (out-154.mta0.migadu.com [91.218.175.154]) by imf06.hostedemail.com (Postfix) with ESMTP id C92A7180008 for ; Tue, 20 Dec 2022 23:56:44 +0000 (UTC) Authentication-Results: imf06.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=hsswEVk0; spf=pass (imf06.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.154 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1671580605; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pzTn8ur1XtUN2Eo2Nx2CrqTa1RIIIWnV7wMZmTTSeVA=; b=iYAKPVo/VrmPJdS2LCw+/Jbh4ZTEAE3Z9qA6uqjjhDrtJc3H3F0E74oTwfZLz2dFnm8k82 Qku0gJWiOwxYYhH887JrgISt+z6gB6EDWIM984kXtj/IxE/pCvIXIseRyk5DdlrlRi/BII KKnWQLPLU903l/gqtPzDaIWbYbhxuOY= ARC-Authentication-Results: i=1; imf06.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=hsswEVk0; spf=pass (imf06.hostedemail.com: domain of roman.gushchin@linux.dev designates 91.218.175.154 as permitted sender) smtp.mailfrom=roman.gushchin@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1671580605; a=rsa-sha256; cv=none; b=3LWlIA4dc+l7qtv+UEPNzt/O46gqwxinW0vqvJ5SgW/ue/XOwNAtVh/MEJEEz5WtJbSHdU TNU3JC+RHRK5HPN279o4VO4XzcTrYdhb/ORlWnupkhZKuUw/6emvXPY6fUM29aq5OVrZ9p McrZH75pDZ4NDgHlcfU1xUkpA4YaKvw= Date: Tue, 20 Dec 2022 15:56:37 -0800 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1671580602; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=pzTn8ur1XtUN2Eo2Nx2CrqTa1RIIIWnV7wMZmTTSeVA=; b=hsswEVk0//IyZFc2+E1WqZAHxe0CFtd4/DVo+x3UvNn/+X9mxF5n56d2exL+WLYeNkoin4 RV2YPPvXag2wbDau0BBIZQAVrRT6bPBeabW/xRTj4cmq2EguhpTRKggNZhdUfVWd8erRnf dyDxVHaidT492nQGMk1Ux/lVIr0RqRI= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Roman Gushchin To: Shakeel Butt Cc: Vlastimil Babka , linux-mm@kvack.org, linux-kernel@vger.kernel.org, Johannes Weiner , Michal Hocko , Muchun Song , Andrew Morton , Waiman Long , Sven Luther Subject: Re: [PATCH RFC] ipc/mqueue: introduce msg cache Message-ID: References: <20221220184813.1908318-1-roman.gushchin@linux.dev> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Migadu-Flow: FLOW_OUT X-Rspam-User: X-Rspamd-Server: rspam04 X-Rspamd-Queue-Id: C92A7180008 X-Stat-Signature: 99g845b8jhw3qghy3ikh6p6e7tzw6cmt X-HE-Tag: 1671580604-25942 X-HE-Meta: U2FsdGVkX1+tjQrN0GYYSfjNlRfuDKS2VFuqxlLkjbVwLvev4HX+k71JwXdiWKQHkTRL2pFlQLuGack42XNJgmowKy/P/FgVkF5DNwFinM0qFgqwcQDK2nNizi0v/FcKkgYlRQ4oHB93sq9z9A1r+cG+jpOXFfIqAnquYbu3ncw/TXiWvJRxfArwks8BFyTiu4M6aFLLY4E+oIJPRCmYIwSS3AuDIUhkEtMJv4DBn93OKA+56p5mebLO8GKpmc4grA1xY6BfNRm0Nd7Nw33pMJXiYjyGKDEvtskKeND+Z8b17HqYb0/2iUCu1ds/g+I4j0L4jhauah4B0EP4wBe76QYrZS6TNbm1IsFv1dOCylk7k6IsqBkJd0DXCzF+Q1eB2tkc77EkEhOBYFF8+5yiwqwf8cbRc3SvB8Bg96WCc251vvC3xu5v1lbMlL2eFIIDrreAvYT2EwxCDwLUR+PTqKdOlNcb9UzwT6wiiH+E9VQu5pbwQxr5oVRY+pTrUf/TPA2cxrbYQJdeD4/5Lv8eTKbVwEp0+9oNFOH426ueecnnSHyzJob3KTmTRpOeQHFJIZ9WQpBABOKRvTt2RSsNrcKMLaONH87GoLz9D8lZbu/SGT8smw1Er20xtxgYmGaP0jZOYS6yLrvAWtaMaoSo1VO1t3BuDs/2wYn53yk7RPcbO6XeiKuUAfH88uMdGBv9s7bQusEcPGPlrKIJStVNQxkjhAMD9RoVCPqoRtTHSDOWYFCZ1xUme7ic+uvi3Zdp+lYRlLYXUYSOSnoIgPi9u/me+rLFPDjBhY2wluirbcRllejpwyCoRrlD3ql8dCtph8DXmLnFnlrcVbvPVitgdSmB3TsP17osl6F3ti6klUBbdMQ6POS8VA== X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Tue, Dec 20, 2022 at 03:28:11PM -0800, Shakeel Butt wrote: > On Tue, Dec 20, 2022 at 12:59 PM Roman Gushchin > wrote: > > > > On Tue, Dec 20, 2022 at 11:53:25AM -0800, Shakeel Butt wrote: > > > +Vlastimil > > > > > > On Tue, Dec 20, 2022 at 10:48 AM Roman Gushchin > > > wrote: > > > > > > > > Sven Luther reported a regression in the posix message queues > > > > performance caused by switching to the per-object tracking of > > > > slab objects introduced by patch series ending with the > > > > commit 10befea91b61 ("mm: memcg/slab: use a single set of kmem_caches for all > > > > allocations"). > > > > > > > > To mitigate the regression cache allocated mqueue messages on a small > > > > percpu cache instead of releasing and re-allocating them every time. > > > > > > > > > > Seems fine with me but I am wondering what is stopping us to do this > > > caching in the slab layer for all accounted allocations? Does this > > > only make sense for specific scenarios/use-cases? > > > > It's far from trivial, unfortunately. Here we have an mqueue object to associate > > a percpu cache with and the hit rate is expected to be high, assuming the mqueue > > will be used to pass a lot of messages. > > > > With a generic slab cache we return to the necessity of managing > > the per-cgroup x per-slab-cache x per-cpu free list (or some other data structure), > > which is already far from trivial, based on the previous experience. It can > > easily lead to a significant memory waste, which will fully compensate all perf > > wins. > > > > So probably we need some heuristics to allocate caches only for really hot slab > > caches and use some sort of a hash map (keyed by cgroup and slab cache) to > > access freelists. What I miss to commit more time to this project (aside from not > > having it), is the lack of real workloads which will noticeably win from this work. > > > > Sven provided a good example and benchmark to reproduce the regression, so it > > was easy to justify the work. > > > > Thanks for the explanation. I think we should add this to the commit > message as well. I do think we should have a general framework for > such caching as there are other users (e.g. io_uring) doing the same > and some future users can take advantage as well e.g. I think this > type of caching will be helpful for filelock_cache as well. Anyways > that can be done in future. I agree. One way I'm thinking about is to provide an API for creating objcg-specific slab caches. All objects belonging to a such slab cache will belong to the same objcg. In this case the accounting can be even faster than the previous per-page accounting. And the user of such API will be responsible for the lifetime of the cache, as well as the decision whether to use the local or global slab cache. Thanks!