From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 32615C77B71 for ; Fri, 21 Apr 2023 15:12:59 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id A17CD6B0071; Fri, 21 Apr 2023 11:12:58 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 9A0E36B0075; Fri, 21 Apr 2023 11:12:58 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 81A426B0078; Fri, 21 Apr 2023 11:12:58 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 6E0C36B0071 for ; Fri, 21 Apr 2023 11:12:58 -0400 (EDT) Received: from smtpin14.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay05.hostedemail.com (Postfix) with ESMTP id 02B8B404D8 for ; Fri, 21 Apr 2023 15:12:57 +0000 (UTC) X-FDA: 80705740836.14.192D03F Received: from dfw.source.kernel.org (dfw.source.kernel.org [139.178.84.217]) by imf15.hostedemail.com (Postfix) with ESMTP id 37699A0013 for ; Fri, 21 Apr 2023 15:12:56 +0000 (UTC) Authentication-Results: imf15.hostedemail.com; dkim=pass header.d=linuxfoundation.org header.s=korg header.b=u5TjJixi; spf=pass (imf15.hostedemail.com: domain of gregkh@linuxfoundation.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=gregkh@linuxfoundation.org; dmarc=pass (policy=none) header.from=linuxfoundation.org ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1682089976; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=IvH0zTQ6Cty6mqOHN6wdD9gh7StcHgd4TIQOXhCbwt8=; b=UAz6aUdzXeZwUZSl4uSzYY08/yIoVdydT5sfACar1IY88nGqhTogXkayRH1twfVnWAeASl 3IJQCMIEiavg+rnn7wuzjw/UsWbLDntpEc1rHwCl6vBVr9toLVdZZQ1D15IY13uy9/p7xI CZBxwLDaL6znta8db+NGoHpTrmQ/6vs= ARC-Authentication-Results: i=1; imf15.hostedemail.com; dkim=pass header.d=linuxfoundation.org header.s=korg header.b=u5TjJixi; spf=pass (imf15.hostedemail.com: domain of gregkh@linuxfoundation.org designates 139.178.84.217 as permitted sender) smtp.mailfrom=gregkh@linuxfoundation.org; dmarc=pass (policy=none) header.from=linuxfoundation.org ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1682089976; a=rsa-sha256; cv=none; b=1hj58L1BqZ4dQZTT74dAJDAR2rYgi9uiJwEtWfnR9+x2JlBHOa4t5eC5V0Ohgor5Sgfp+8 UUwKejX8aB5ggTjsbFu0yaV47GYlblYiSjekccRX1vwTunXKvkLs9H1vesJoUcgH0YSG0s //RpW/ARdGLHAXhCUm2VcD5in0HaKLQ= Received: from smtp.kernel.org (relay.kernel.org [52.25.139.140]) (using TLSv1.2 with cipher ECDHE-RSA-AES256-GCM-SHA384 (256/256 bits)) (No client certificate requested) by dfw.source.kernel.org (Postfix) with ESMTPS id 19C4160FDA; Fri, 21 Apr 2023 15:12:55 +0000 (UTC) Received: by smtp.kernel.org (Postfix) with ESMTPSA id A31CBC433D2; Fri, 21 Apr 2023 15:12:53 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/simple; d=linuxfoundation.org; s=korg; t=1682089974; bh=YxpowVHwsKeFgDfSN4DkprW9h6O8h9UyRh/INgZFhQA=; h=Date:From:To:Cc:Subject:References:In-Reply-To:From; b=u5TjJixiqGqbmI2F/94VhizZOx77gxuv7jWOSb6YCSISr8e7x5V7lE3URl+LlLgfF gJFeUEhWiiI9CPmZsAHPYWMYb+TJuYec0Ejv+XgWSphQe3ph01AHoz+VsHqItKXevB OQd0veR80QyuIlsy0eqg8XzV7CCslr7Qoeh6UXr0= Date: Fri, 21 Apr 2023 17:12:51 +0200 From: Greg KH To: Luis Chamberlain Cc: lucas.demarchi@intel.com, david@redhat.com, patches@lists.linux.dev, linux-modules@vger.kernel.org, linux-mm@kvack.org, linux-kernel@vger.kernel.org, pmladek@suse.com, petr.pavlu@suse.com, prarit@redhat.com, torvalds@linux-foundation.org, rafael@kernel.org, christophe.leroy@csgroup.eu, tglx@linutronix.de, peterz@infradead.org, song@kernel.org, rppt@kernel.org, dave@stgolabs.net, willy@infradead.org, vbabka@suse.cz, mhocko@suse.com, dave.hansen@linux.intel.com, colin.i.king@gmail.com, jim.cromie@gmail.com, catalin.marinas@arm.com, jbaron@akamai.com, rick.p.edgecombe@intel.com, j.granados@samsung.com Subject: Re: [PATCH] module: add debugging auto-load duplicate module support Message-ID: References: <20230418204636.791699-1-mcgrof@kernel.org> <2023041951-evolution-unwitting-1791@gregkh> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: X-Stat-Signature: 7x71h1buxyduge5xi7ijhcpjzd8msipj X-Rspam-User: X-Rspamd-Queue-Id: 37699A0013 X-Rspamd-Server: rspam06 X-HE-Tag: 1682089976-651608 X-HE-Meta: U2FsdGVkX18eA+UE0tR59ea5B9yLf8fR1jRVMGXZcdJe0LZKwFQF3owMccOsJDfGXM7g3mVHkPPeEgPG5kbS1DdOUU3ncly3U9RiPZ1jDNK+h7P3UW2wj5kD7s0XJQi5NK9lssaQs9r8Yxtt9b85Z3kIz6YeF0coSOtAoKyi2Muv2D41N026c4XYxeN56n0K/RwexjD7R/sUUeANw70LgcfBpnLN+pu1I83+noD2R5Fev2WKtjSFAAlFrHUPrj8+aZ4PHCtO/mtSs8hUJkAUG8W1sO2zIg5kd+l+YOchOVDsZIGX9bQTT8e2NmYzpTb4gtG6GAvCVWxmCqBsbRTogKT+Unx/9okTvkX5Ph8JGiArXHGl4ZW/kZ27Dr8DCh1lfJWJE8GbcKLOmUmGtfQi8xAAASNqgVSRDUcfV6Of1WCVilmH5lxoAvve9VLd8XXG4V6q4E0A251BCLRsYkHmCtGtcfkecdGof/ac+mKHpBxygg+pggg0R4a6y5ghrYhD6F6zyLp+klrxh2q7gMrom2KUzHjkkU4ayypLsAQrFxnRd3Au5gArt6SwVXzd5hIYB+TY83Y7nBwIfZofFurjKIPWFl0sl0VYTyM0lPquq0Vf/Dy18sBzW/N92oWLOwxK2wch/KLdIE5XcSJyznFIMIDGn63SJLhVZdXvUsm2thk8z6xIgNLw35KIpKUk8+rWT+evR3ONmchGCBKBkxD5pYXcKtbTasCbO9LDN/gVB3j1Rb5yGUSNZOWuDt7+2QgeU2/LAJBCbXSDJdD72ZYkfOBu97lkuAoMX23BDO294Tidme1ENgM1qM1aiXYmjqycmVmEWzFmY+5moqcgKSehT7LJzLx9BnUa0BNxFTCVZzcqMsFzV7onWNsrIQi5gW8G7Dx9dOLNTwuws4QZDres5beLwYCeWj3vkZsJ3RqOvyzmPbRO4Y1wMGwBifK3pQZEzgoRHWsn/Z8qJckeN1b 5aDcCP2s r6Bq3M4zjWDn8EFpduwNwZhzIhG6n5CaikiefTJei5kGkwrXfyBXb99LfeWW9nxrILQEDR4QzZJGQfmhQwX6/zSzMcinIQBUuGoq1L0QZNdx1O09ZpZTXq07ubDDrmfwOSjA7f9/PySAPo0mqLGhoSY3kVblY2uX5tVmeRHIrdE+pT1Unhaz5NSo86ue/A84jnBktmhXSEIhHv1rZRpychbxDO1GcBZFa2j+9cmApuTMS/aVU8kTkKgUatWJT9c8CE7v+dEA6YUziQIGFGAzY7IA/oCip7yFEQeW7RqCWsSuDbvT74+zO//vZqO3tkvn9dScdZPGHmdTRdI/MzIA9r9OuG0460Yso1AiI X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Thu, Apr 20, 2023 at 02:03:32PM -0700, Luis Chamberlain wrote: > On Thu, Apr 20, 2023 at 07:32:10AM +0200, Greg KH wrote: > > On Wed, Apr 19, 2023 at 04:32:30PM -0700, Luis Chamberlain wrote: > > > > It's not "wasted", as it is returned when the module is determined to be > > > > a duplicate. Otherwise everyone will want this enabled as they think it > > > > will actually save memory. > > > > > > I'll change the language to be clear the issue is memory pressure early > > > on boot. I'll also add a bit of language to help at least guide people > > > to realize that the real value-add for this, ie, I'll have to mention we > > > suspect issue is udev and not module auto-loading and that this however > > > may still help find a few cases we can optimize for. > > > > This isn't udev's "problem", all it is doing is what the kernel asked it > > to do. The kernel said "Here's a new device I found, load a module for > > it please!" > > If you believe that then the issue is still a kernel issue, and the > second part to that sentence "load a module for it" was done without > consideration of the implications, or without optimizations in mind. > Given the implications were perhaps not well understood it is unfair > for us to be hard on ourselves on that. But now we know, ideally if we > could we *should* only issue a request for a module *once* during boot. But there is no mapping between devices and modules other than what is exported in the module info and that is up to userspace to handle. > Where does the kernel actually say "load a module"? The driver core says "hey a new device is now present!" Userspace takes that message and calls kmod with the device information which then determines what module to load by looking at the device aliases. > Isn't that just an implied gesture? Yes. > > And it's the kmod code here, not udev itself doing all of this. > > Yes, IMHO kmod could and *should* be enhanced to share a loading context > during boot so to avoid duplicates too and then udev would have to > embrace such functionality. That's going to take time to propagate, as > you can imagine. udev is just the transport to kmod here, it's not in the job of filtering duplicate messages. > > Why not > > just rate-limit it in userspace if your system can't handle 10's of > > thousands of kmod calls all at once? I think many s390 systems did this > > decades ago when they were controlling 10's of thousands of scsi devices > > and were hit with "device detection storms" at boot like this. > > Boot is a special context and in this particular case I agree userspace > kmod could/should be extended to avoid duplicate module requests in that > context. But likewise the kernel should only have to try to issue a > request for a single module once, if it could easily do that. Are you sure that this is happening at boot in a way that userspace didn't just trigger it on its own after init started up? That happens as a "coldboot" walk of the device tree and all uevent are regenerated. That is userspace asking for this, so there's nothing that the kernel can do. > This does beg the question, why force userspace to rate limit if we > can do better in the kernel? Specially if *we're the ones*, as you say, > that are hinting to userspace to shoot back loading modules for us and we > know we're just going to drop duplicates? Maybe error out of duplicate module loading earlier? I don't know, sorry. > > What specific devices and bus types are the problem here for these systems? > > My best assessment of the situation is that each CPU in udev ends up triggering > a load of duplicate set of modules, not just one, but *a lot*. Not sure > what heuristics udev uses to load a set of modules per CPU. Again, finding which device and bus is causing the problem is going to be key here to try to solve the issue. Are you logging duplicate module loads by name as well? thanks, greg k-h