From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 0DC94C3ABC3 for ; Tue, 13 May 2025 22:17:12 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 1AEFF6B00CC; Tue, 13 May 2025 18:17:11 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 137716B00D6; Tue, 13 May 2025 18:17:11 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id ECED26B00D7; Tue, 13 May 2025 18:17:10 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0010.hostedemail.com [216.40.44.10]) by kanga.kvack.org (Postfix) with ESMTP id C007B6B00CC for ; Tue, 13 May 2025 18:17:10 -0400 (EDT) Received: from smtpin05.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 42E4816187A for ; Tue, 13 May 2025 22:17:11 +0000 (UTC) X-FDA: 83439296262.05.39E3560 Received: from out-173.mta0.migadu.com (out-173.mta0.migadu.com [91.218.175.173]) by imf19.hostedemail.com (Postfix) with ESMTP id 574771A0007 for ; Tue, 13 May 2025 22:17:09 +0000 (UTC) Authentication-Results: imf19.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="wx9K/XDN"; spf=pass (imf19.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.173 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1747174629; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=rahQYu1hjcaoMkOQ2Z62DrFUneDFZMeqj32rynYl3Ak=; b=8VN/FONQ+iY5jId8rrbSlx9VQDQ+JnHt3bF5zK8v7/AbTOqJnQ0+P1+outVuFJKPrL0ePq HeE0b1QKnQ+X3z3r2cVIyTeDZDvfj4SAQyTB7jrR0JJ9i9XaUyY8t7+sjYDcKRtj/7KY77 YMC8LDgjPFpKehEQ8on/i9MNrD7AMrM= ARC-Authentication-Results: i=1; imf19.hostedemail.com; dkim=pass header.d=linux.dev header.s=key1 header.b="wx9K/XDN"; spf=pass (imf19.hostedemail.com: domain of shakeel.butt@linux.dev designates 91.218.175.173 as permitted sender) smtp.mailfrom=shakeel.butt@linux.dev; dmarc=pass (policy=none) header.from=linux.dev ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1747174629; a=rsa-sha256; cv=none; b=NUSgIO1awLdfMr3/BP8iBvGnVwe/ATSHf48WR8AyW2jDQL75FJLQJ4JN8gRNwl9NB0J2U8 HBY4crJVm/2HTXJcapkwugqvVG1XFlM25sVSf76MtXxkyzK2vkCFIET1ETMiKQ1LgSkLPK 6beF5ygc3bUe6INREjVFV8kmMvfyuE0= Date: Tue, 13 May 2025 15:17:00 -0700 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=linux.dev; s=key1; t=1747174626; h=from:from:reply-to:subject:subject:date:date:message-id:message-id: to:to:cc:cc:mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=rahQYu1hjcaoMkOQ2Z62DrFUneDFZMeqj32rynYl3Ak=; b=wx9K/XDNsqu6pyfaRN9KPelr685TgwkIZ1xr5mFHW+x3b9xZQLu1VZrf/0N2qg1XVpfB9O 0Hf3zWfQM3ZlD4p5dlaRY6pGJFUrKBK+9vQaQJjTOPRT8PivyYeNxB1GUui0ARnNei+Os6 I8+Ybr+04GBOUr87R0gPSswu14rvZx4= X-Report-Abuse: Please report any abuse attempt to abuse@migadu.com and include these headers. From: Shakeel Butt To: Peter Zijlstra Cc: Vlastimil Babka , Andrew Morton , Tejun Heo , Johannes Weiner , Michal Hocko , Roman Gushchin , Muchun Song , Alexei Starovoitov , Sebastian Andrzej Siewior , bpf@vger.kernel.org, linux-mm@kvack.org, cgroups@vger.kernel.org, linux-kernel@vger.kernel.org, Meta kernel team , Mathieu Desnoyers Subject: Re: [PATCH 0/4] memcg: nmi-safe kmem charging Message-ID: References: <20250509232859.657525-1-shakeel.butt@linux.dev> <2e2f0568-3687-4574-836d-c23d09614bce@suse.cz> <07e4e8d9-2588-41bf-89d4-328ca6afd263@suse.cz> <20250513114125.GE25763@noisy.programming.kicks-ass.net> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <20250513114125.GE25763@noisy.programming.kicks-ass.net> X-Migadu-Flow: FLOW_OUT X-Rspamd-Queue-Id: 574771A0007 X-Rspam-User: X-Rspamd-Server: rspam11 X-Stat-Signature: koc86j5ntpazjzbp71oncg4dej9o4dtk X-HE-Tag: 1747174629-233532 X-HE-Meta: U2FsdGVkX187eB6s6WrITZkFoYUO7IZ4STIpFhcRmP7wGAZRQ8o9pWOdVMEczBF5DI5WjbSe/yAPasuxb1BJlDMA31+6votPgdA1g4PjXE1+7OD37cofMbHx8vZ3BfTEOXN3/vY3GrND5FHGA7YOLpFlHT44M2HdWMnG4XLi8LLMA+wtAxVNvuukge8vex255Tys8k3II8BNMS4gmCtIVanBnLI/GBiJ8UTLtQipLuVSQEnLp5STpRvRBv31ymv17SHsdKpqQ6vN5p12RytTjwaJ5MT89FBnXDmRL+u/IDf9E4De+AVb1HPUZSi3V7rKPouH4WRWhi3ANnDpkcx/nhFuZyKEwuEph8gZFJrphj3/ZBnzM6hw3docjF72l3EyRAdm7w+a9JH+Ehjl/IZviPsMHf99wUG9Hlrvj0HHsjoFNWGMHWiHjrkRKAfYHqePiskXK1lTLCWMzVei7Pd+18vKFymCgKAWdbMOPF2AtZH/y49KWed/i0RvZAiHF4CfPfrxqJXWFGOuPTkLRZOE6L99vvxWo55pWXZRjxW/Yln9XizMfdcA4nwxiLaitsh4wguoHhVTnPacaKKIwegb11ZxLZza00eOxU2cnF8P6snnMuskd8IHGW3e7437BuBLs3G+efWsdeVqDliNxTeZL7cOt4LBK+sCp+zwsXK7lW99QzKYdn45iKaiyE/vcQjxCF11VhNvMo54JhsgBv+vvhG3OwOlojG4eNZi7M3ovu/DzukSOdvLcPrq0OshKMuGYZQnk87pmUqLBuR6DnMhro9I9xUPE3l0E2Uq66bdu3Wwbnsy6jr13P+bGKGUPBVrGav2TKg15MPVdn0X7U/dq13wxRw37qmUQ1x4UGy/tZ9F+nX/2VIpQGNgak77l35Ijf3kTBO9ngIdzZpZjMnEt5D0NrVgcVj1cGWAAV4mY9a6hDqPN6F8qI5W4y72jXHjnFtXGvRWw52pBSYbaXX oKUuAbis WRso5zManjxZznLUhJiXwytNRzUYRxRrIDcTdD2hm0GMjm/ls0BIUnc/IsPkha/vahtVmifmwT4YN3b3+iLf2xeVPshGEnlkjlflOGuW2ypLt5wICme/X4vt/3QVHuFt7mIwbcV6WHlrM/+IML1b6P3XTtJ8O5CLtXjX8YDifk35A3uYJcolA7SzhUxKE8Dn3mmFnW6LzRcXU54LzcRPIqEeFkJN1Ah/PAq218Y1/pE3jAwg= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Tue, May 13, 2025 at 01:41:25PM +0200, Peter Zijlstra wrote: > On Tue, May 13, 2025 at 09:15:23AM +0200, Vlastimil Babka wrote: > > > >> > The initial prototype tried to make memcg charging infra for kernel > > >> > memory re-entrant against irq and nmi. However upon realizing that > > >> > this_cpu_* operations are not safe on all architectures (Tejun), this > > >> > > >> I assume it was an off-list discussion? > > >> Could we avoid this for the architectures where these are safe, which should > > >> be the major ones I hope? > > IIRC Power64 has issues here, 'funnily' their local_t is NMI safe. > Perhaps we could do the same for their this_cpu_*(), but ideally someone > with actual power hardware should do this ;-) > Is CONFIG_ARCH_HAS_NMI_SAFE_THIS_CPU_OPS the right config to differentiate between such archs? I see Power64 does not have that enabled. > > > Yes it was an off-list discussion. The discussion was more about the > > > this_cpu_* ops vs atomic_* ops as on x86 this_cpu_* does not have lock > > > prefix and how I should prefer this_cpu_* over atomic_* for my series on > > > objcg charging without disabling irqs. Tejun pointed out this_cpu_* are > > > not nmi safe for some archs and it would be better to handle nmi context > > > separately. So, I am not that worried about optimizing for NMI context > > > > Well, we're introducing in_nmi() check and different execution paths to all > > charging. This could be e.g. compiled out for architectures where this_cpu* > > is NMI safe or they don't have NMIs in the first place. > > Very few architectures one would care about do not have NMIs. Risc-V > seems to be the exception here ?!? > > > > but your next comment on generic_atomic64_* ops is giving me headache. > > > > > >> > > >> > series took a different approach targeting only nmi context. Since the > > >> > number of stats that are updated in kernel memory charging path are 3, > > >> > this series added special handling of those stats in nmi context rather > > >> > than making all >100 memcg stats nmi safe. > > >> > > >> Hmm so from patches 2 and 3 I see this relies on atomic64_add(). > > >> But AFAIU lib/atomic64.c has the generic fallback implementation for > > >> architectures that don't know better, and that would be using the "void > > >> generic_atomic64_##op" macro, which AFAICS is doing: > > >> > > >> local_irq_save(flags); \ > > >> arch_spin_lock(lock); \ > > >> v->counter c_op a; \ > > >> arch_spin_unlock(lock); \ > > >> local_irq_restore(flags); \ > > >> > > >> so in case of a nmi hitting after the spin_lock this can still deadlock? > > >> > > >> Hm or is there some assumption that we only use these paths when already > > >> in_nmi() and then another nmi can't come in that context? > > >> > > >> But even then, flush_nmi_stats() in patch 1 isn't done in_nmi() and uses > > >> atomic64_xchg() which in generic_atomic64_xchg() implementation also has the > > >> irq_save+spin_lock. So can't we deadlock there? > > > > > > I was actually assuming that atomic_* ops are safe against nmis for all > > > archs. > > We have HAVE_NMI_SAFE_CMPXCHG for this -- there are architectures where > this is not the case -- but again, those are typically oddball archs you > don't much care about. > > But yes, *64 on 32bit archs is generally not NMI safe. > > > I looked at atomic_* ops in include/asm-generic/atomic.h and it > > > is using arch_cmpxchg() for CONFIG_SMP and it seems like for archs with > > > cmpxchg should be fine against nmi. I am not sure why atomic64_* are not > > > using arch_cmpxchg() instead. I will dig more. > > Not many 32bit architectures have 64bit cmpxchg. We're only now dropping > support for x86 chips without CMPXCHG8b. > As Vlastimil pointed out (last point), I don't think I will need 64bit cmpxchg, 32bit cmpxchg will be fine. > > Yeah I've found https://docs.kernel.org/core-api/local_ops.html and since it > > listed Mathieu we discussed on IRC and he mentioned the same thing that > > atomic_ ops are fine, but the later added 64bit variant isn't, which PeterZ > > (who added it) acknowledged. > > > > But there could be way out if we could somehow compile-time assert that > > either is true: > > - CONFIG_HAVE_NMI=n - we can compile out all the nmi code > > Note that in_nmi() is not depending on HAVE_NMI -- nor can it be. Many > architectures treat various traps as NMI-like, even though they might > not have real NMIs. > > > - this_cpu is safe on that arch - we can also compile out the nmi code > > There is no config symbol for this presently. Hmm what about CONFIG_ARCH_HAS_NMI_SAFE_THIS_CPU_OPS? > > > - (if the above leaves any 64bit arch) its 64bit atomics implementation is safe > > True, only because HPPA does not in fact have NMIs. What is HPPA? > > > - (if there are any 32bit applicable arch left) 32bit atomics should be > > enough for the nmi counters even with >4GB memory as we flush them? and we > > know the 32bit ops are safe > > Older ARM might qualify here. Thanks a lot Vlastimil & Peter for the suggestions. Let me summarize what I plan to do and please point out if I am doing something wrong: #if defined(CONFIG_ARCH_HAS_NMI_SAFE_THIS_CPU_OPS) || !defined(CONFIG_HAVE_NMI) // Do normal this_cpu* ops #elif defined(ARCH_HAVE_NMI_SAFE_CMPXCHG) // Do 32 bit atomic ops with in_nmi() checks #else // Build error or ignore nmi stats?? #endif WDYT? (BTW Vlastimil, I might rebase send out the irq-safe series before this nmi one.) thanks, Shakeel