From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 21976C54E5D for ; Thu, 14 Mar 2024 19:49:35 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 97218800DF; Thu, 14 Mar 2024 15:49:34 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9221A800B4; Thu, 14 Mar 2024 15:49:34 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 810E2800DF; Thu, 14 Mar 2024 15:49:34 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0012.hostedemail.com [216.40.44.12]) by kanga.kvack.org (Postfix) with ESMTP id 6EEB3800B4 for ; Thu, 14 Mar 2024 15:49:34 -0400 (EDT) Received: from smtpin09.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay10.hostedemail.com (Postfix) with ESMTP id 1A273C0657 for ; Thu, 14 Mar 2024 19:49:34 +0000 (UTC) X-FDA: 81896684268.09.E97A6B3 Received: from out-170.mta1.migadu.com (out-170.mta1.migadu.com [95.215.58.170]) by imf15.hostedemail.com (Postfix) with ESMTP id 5F508A000D for ; Thu, 14 Mar 2024 19:49:32 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=vyVBxkEC; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf15.hostedemail.com: domain of kent.overstreet@linux.dev designates 95.215.58.170 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1710445772; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=oQhKBgFBwcHxMMVU5RZZVIyABjVMSjCmIgF/D8RmMd0=; b=3g3GadAtrgETyZ6xagvP6DHI5G99edPH0+k33krEocaPUkU2rqFPpvJDieMbvnecjdTlXx Vr1wjTzEthZFVAwFVR/syO/jjAyeZufwvagGXBy7exa3gRoLT+xsLpgqcgKEDCYxiDvGcC Sztr5C5pSQxWLVF0RsV58glUy9iwqWk= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b=vyVBxkEC; dmarc=pass (policy=none) header.from=linux.dev; spf=pass (imf15.hostedemail.com: domain of kent.overstreet@linux.dev designates 95.215.58.170 as permitted sender) smtp.mailfrom=kent.overstreet@linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1710445772; a=rsa-sha256; cv=none; b=we/Pko4EeIJpnfAlWjWhke6TvL4p5y/DSL0xY3tUPB69Qd6zWYjpFItpNNN25IKGHw6lyD K8B87aqGBNY0hO9JA8dgPahhKPiw5vuW4wsbD3P/Z4hd2N9b3OPyKYEEGlARhh7k1hDHIU GGZxyptuvbjv7UbOuZVycPrJVZWyfK0= Date: Thu, 14 Mar 2024 15:49:22 -0400 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1710445769; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: content-transfer-encoding:content-transfer-encoding: in-reply-to:in-reply-to:references:references; bh=oQhKBgFBwcHxMMVU5RZZVIyABjVMSjCmIgF/D8RmMd0=; b=vyVBxkECVqaeyjcPV4xlon4xwRuqkYRismZotCyCVT8L9bU6/2JAk/tIYs0I0psufu68LS juN/mOOUJVXdHan06zn33FPTtVqRAfcOxriZVnKnmyKM/q1RTUi2qyNDsw+OAhAPmYsvp1 72SJigfNEAjBd0ZzHdWYSxTd7iQcRCs= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Kent Overstreet To: Pasha Tatashin Cc: "H. Peter Anvin" , linux-kernel@vger.kernel.org, linux-mm@kvack.org, akpm@linux-foundation.org, x86@kernel.org, bp@alien8.de, brauner@kernel.org, bristot@redhat.com, bsegall@google.com, dave.hansen@linux.intel.com, dianders@chromium.org, dietmar.eggemann@arm.com, eric.devolder@oracle.com, hca@linux.ibm.com, hch@infradead.org, jacob.jun.pan@linux.intel.com, jgg@ziepe.ca, jpoimboe@kernel.org, jroedel@suse.de, juri.lelli@redhat.com, kinseyho@google.com, kirill.shutemov@linux.intel.com, lstoakes@gmail.com, luto@kernel.org, mgorman@suse.de, mic@digikod.net, michael.christie@oracle.com, mingo@redhat.com, mjguzik@gmail.com, mst@redhat.com, npiggin@gmail.com, peterz@infradead.org, pmladek@suse.com, rick.p.edgecombe@intel.com, rostedt@goodmis.org, surenb@google.com, tglx@linutronix.de, urezki@gmail.com, vincent.guittot@linaro.org, vschneid@redhat.com Subject: Re: [RFC 00/14] Dynamic Kernel Stacks Message-ID: References: <20240311164638.2015063-1-pasha.tatashin@soleen.com> <2cb8f02d-f21e-45d2-afe2-d1c6225240f3@zytor.com> MIME-Version: 1.0 Content-Type: text/plain; charset=utf-8 Content-Disposition: inline Content-Transfer-Encoding: 8bit In-Reply-To: X-Migadu-Flow: FLOW_OUT X-Rspamd-Server: rspam09 X-Rspamd-Queue-Id: 5F508A000D X-Stat-Signature: jq75hsj5zy7qgfmk4oxtdwzic4doqnn5 X-Rspam-User: X-HE-Tag: 1710445772-835979 X-HE-Meta: U2FsdGVkX18TJO35r8PF7nlXLnC71MBAazIllkWH/Es2cFu7pc+ITlnfmAWAZxJdyGoEWKRWQ6WCBhBKM5Ip6gYGmNQGbPQHfkA6Dqll1lQ6+aR1nI1cTmGt0UX0W8T7sC2oa/Vk/t4rpKzSN2D4vsxwguD99XfG7NTkpJDzcrCkf2+9UVQpFq4IeoZ+AzDwBWN6em0YOS50/USpLfT+bQvqlVSwd8NY6A9n9WJ+Jjc8uaotbc5UtWnDwPJ8SogzuAwevNByUuygd2h5ya85IyJ3X8Y88mzGEdExWkpnXwN+sz+Lzo2WruZyNINngFVz3tvV1ZtR03IU9fqZiVNRE8G5ynmSeoOOazd7W/lWEAWEgcsJijLfmbzF3ObFTkbnFy6MnbFTZuuYdYPgvzEj3ULeZw1JX0c+gORcwb196V9gvAmH1TAggscX5hbsGerOtPqFACLguWyWZJ6PFJE6gUg2i1waDGKmFxQivaXe5rFL1WV19h4cPyABdEe3PWu4j6J9GfGGY+eCOBGfwcIuYs6unkUM1HlkB82ZQPF0sFwdLzHnEX4lpVjIFoH0JsNXHaiBs6HSHSxH7+2N5HNO9aoAQFxMMrKWZeNdqkAxTbwVGIkixnmpGluju741CLCDvYI/Mil7xz+UqRZVmeccft1ZfYiTQnFe0q4YbZRgVt4NWgeEt6PIKAf3ggAUvSMrDaJ9iY93eT0QcnoI0i0NhtJ14KXCkNUWAQgVlANy+8ijHQnVN8uaHI79V7Scqv3ceTSXOSkoAnpv9VumB528IFU9MTh3MPqYxDOQwtR38Ms= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Thu, Mar 14, 2024 at 03:34:03PM -0400, Pasha Tatashin wrote: > On Thu, Mar 14, 2024 at 3:29 PM Kent Overstreet > wrote: > > > > On Thu, Mar 14, 2024 at 03:23:08PM -0400, Pasha Tatashin wrote: > > > > > > > > > > My point is that what matters is total memory use, not just memory used in > > > > > the kernel. Amdahl's law. > > > > > > > > If userspace is running a few processes with many threads and the > > > > userspace stacks are small, kernel stacks could end up dominating. > > > > > > > > I'd like to see some numbers though. > > > > > > The unused kernel stack pages occupy petabytes of memory across the fleet [1]. > > > > Raw number doesn't mean much here (I know how many machines Google has, > > of course it's going to be petabytes ;), percentage of system memory > > would be better. > > > > What I'd _really_ like to see is raw output from memory allocation > > profiling, so we can see how much memory is going to kernel stacks vs. > > other kernel allocations. > > I've heard there is memory profiling working that can help with that... I heard you've tried it out, too :) > While I do not have the data you are asking for, the other kernel > allocations might be useful, but this particular project is targeted > to help with reducing overhead where the memory is not used, or used > in very extreme rare cases. Well, do you think you could gather it? We shouldn't be blindly applying performance optimizations; we need to know where to focus our efforts. e.g. on my laptop I've currently got 356 processes for < 6M of kernel stack out of 32G total ram, so clearly this isn't much use to me. If the ratio is similar on your servers - nah, don't want it. I expect the ratio is not similar and you are burning proportially more memory on kernel stacks, but we still need to gather the data and do the math :) > > > Number of kernel threads vs. number of user threads would also be good > > to know - I've been seeing ps output lately where we've got a lot more > > workqueue workers than we should, perhaps that's something that could be > > addressed. > > Yes, doing other optimizations make sense, reducing the total number > kernel threads if possible might help as well. I will look into this > as well to see how many user threads vs kernel threads we have. Great, that will help too.