From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 4A4B0C43334 for ; Fri, 22 Jul 2022 11:04:10 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 6CF9B6B0072; Fri, 22 Jul 2022 07:04:09 -0400 (EDT) Received: by kanga.kvack.org (Postfix, from userid 40) id 67FA26B0073; Fri, 22 Jul 2022 07:04:09 -0400 (EDT) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 548016B0074; Fri, 22 Jul 2022 07:04:09 -0400 (EDT) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0017.hostedemail.com [216.40.44.17]) by kanga.kvack.org (Postfix) with ESMTP id 458BF6B0072 for ; Fri, 22 Jul 2022 07:04:09 -0400 (EDT) Received: from smtpin27.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay07.hostedemail.com (Postfix) with ESMTP id 26E76160BF6 for ; Fri, 22 Jul 2022 11:04:09 +0000 (UTC) X-FDA: 79714451418.27.14C287A Received: from smtp-out2.suse.de (smtp-out2.suse.de [195.135.220.29]) by imf03.hostedemail.com (Postfix) with ESMTP id 67D642006F for ; Fri, 22 Jul 2022 11:04:07 +0000 (UTC) Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by smtp-out2.suse.de (Postfix) with ESMTPS id D8D5C2014D; Fri, 22 Jul 2022 11:04:05 +0000 (UTC) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=suse.com; s=susede1; t=1658487845; h=from:from:reply-to:date:date:message-id:message-id:to:to:cc:cc: mime-version:mime-version:content-type:content-type: in-reply-to:in-reply-to:references:references; bh=wRcI6IS4WCQrefcXgExHWQPA2btCzngFsBLrirnwPjg=; b=CqT89nKjc6F0oNmPNDNQZe4jX5Wt0asx5Kba8ZDKlsEt8phRAmJ5vCG3OGE6TSqm/QzlmA aHzKhhjdjloRUSzikvtDBlRAWWguOIKBfpyECXA/FnXRbv1odRnjIrdrT/5hIt2Xq0qJfl QaPUdcABXfXVmtBdA4ucRgxaqUhO2tU= Received: from imap2.suse-dmz.suse.de (imap2.suse-dmz.suse.de [192.168.254.74]) (using TLSv1.3 with cipher TLS_AES_256_GCM_SHA384 (256/256 bits) key-exchange X25519 server-signature ECDSA (P-521) server-digest SHA512) (No client certificate requested) by imap2.suse-dmz.suse.de (Postfix) with ESMTPS id C760113AB3; Fri, 22 Jul 2022 11:04:05 +0000 (UTC) Received: from dovecot-director2.suse.de ([192.168.254.65]) by imap2.suse-dmz.suse.de with ESMTPSA id gp1xLyWE2mKjEgAAMHmgww (envelope-from ); Fri, 22 Jul 2022 11:04:05 +0000 Date: Fri, 22 Jul 2022 13:04:05 +0200 From: Michal Hocko To: Tetsuo Handa Cc: Andrew Morton , Johannes Weiner , linux-mm Subject: Re: [PATCH v3] mm: memcontrol: fix potential oom_lock recursion deadlock Message-ID: References: <000000000000471c2905e3c2c2c2@google.com> <20220714141813.yi5p4o2tiyvkao6b@quack3> <534fa596-0c29-0f1e-b292-53ad9c3dbbe3@I-love.SAKURA.ne.jp> <20220715013908.ayyimue5yhfwonho@google.com> <03304bf8-d153-698f-0376-9e9a0ec1048e@I-love.SAKURA.ne.jp> <834b896d-68fb-caeb-4316-2e0a2190e3eb@I-love.SAKURA.ne.jp> <86afb39f-8c65-bec2-6cfc-c5e3cd600c0b@I-love.SAKURA.ne.jp> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <86afb39f-8c65-bec2-6cfc-c5e3cd600c0b@I-love.SAKURA.ne.jp> ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1658487847; a=rsa-sha256; cv=none; b=blCI6AYc8tedc7JZrRHOtekqiE+R9hMgx3pJ8RACack79PGDczaGFGTmOdk2kD8Q9LlNSE wmf7t+r0CrPvwlLZbE16Ie3CnRq94hMlv5GKyD6i46pEBFvM7hc4DxyeamAdgzmWZAqxb+ gtyETBwdi7Gfnghy25oaWUTiyX6M9nQ= ARC-Authentication-Results: i=1; imf03.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=CqT89nKj; spf=pass (imf03.hostedemail.com: domain of mhocko@suse.com designates 195.135.220.29 as permitted sender) smtp.mailfrom=mhocko@suse.com; dmarc=pass (policy=quarantine) header.from=suse.com ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1658487847; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=wRcI6IS4WCQrefcXgExHWQPA2btCzngFsBLrirnwPjg=; b=MpczvyiK78TnhW6I5Mv67VIvondlKk1R3xTCUz6/aoCRDDUGe5gZQjLjvGQG9h8Gj7mghZ hoo/FP5y6PN2PGk0P/l7Vbd6yIJBT/yPaLGzzodNJxNdRTc6SJTpKPZWcigBxRWnwcCToI lySgQQyZCTFIo9Mx8YLiY9a/nrDe1LU= Authentication-Results: imf03.hostedemail.com; dkim=pass header.d=suse.com header.s=susede1 header.b=CqT89nKj; spf=pass (imf03.hostedemail.com: domain of mhocko@suse.com designates 195.135.220.29 as permitted sender) smtp.mailfrom=mhocko@suse.com; dmarc=pass (policy=quarantine) header.from=suse.com X-Rspamd-Server: rspam02 X-Rspamd-Queue-Id: 67D642006F X-Stat-Signature: okczbmxkysid9h4dxxqn1kdjrdsa9cwo X-Rspam-User: X-HE-Tag: 1658487847-37471 X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: On Fri 22-07-22 19:45:39, Tetsuo Handa wrote: > syzbot is reporting GFP_KERNEL allocation with oom_lock held when reporting > memcg OOM [1]. Such allocation request might deadlock the system, for > __alloc_pages_may_oom() cannot invoke global OOM killer due to oom_lock > being already held by the caller. OK, I have misunderstood your previous wording and now I have realized that there are 2 issues here. One of them is a (less likely) dead lock on the oom_lock not making a fwd progress (that would require global OOM racing with memcg OOM) and the other is the GFP_NOFS->GFP_KERNEL dependency which can deadlock even without global the above race. Correct? Sorry I could have realized that sooner. > Fix this problem by removing the allocation from memory_stat_format() > completely, and pass static buffer when calling from memcg OOM path. > > Link: https://syzkaller.appspot.com/bug?extid=2d2aeadc6ce1e1f11d45 [1] > Reported-by: syzbot > Suggested-by: Michal Hocko > Fixes: c8713d0b23123759 ("mm: memcontrol: dump memory.stat during cgroup OOM") > Signed-off-by: Tetsuo Handa > Acked-by: Michal Hocko > --- > Changes in v3: > Update patch description. > > Changes in v2: > Use static buffer for OOM reporting, suggested by Michal Hocko . > > mm/memcontrol.c | 22 +++++++++------------- > 1 file changed, 9 insertions(+), 13 deletions(-) > > diff --git a/mm/memcontrol.c b/mm/memcontrol.c > index 618c366a2f07..8092be2fbb7c 100644 > --- a/mm/memcontrol.c > +++ b/mm/memcontrol.c > @@ -1460,14 +1460,12 @@ static inline unsigned long memcg_page_state_output(struct mem_cgroup *memcg, > return memcg_page_state(memcg, item) * memcg_page_state_unit(item); > } > > -static char *memory_stat_format(struct mem_cgroup *memcg) > +static void memory_stat_format(struct mem_cgroup *memcg, char *buf, int bufsize) > { > struct seq_buf s; > int i; > > - seq_buf_init(&s, kmalloc(PAGE_SIZE, GFP_KERNEL), PAGE_SIZE); > - if (!s.buffer) > - return NULL; > + seq_buf_init(&s, buf, bufsize); > > /* > * Provide statistics on the state of the memory subsystem as > @@ -1533,8 +1531,6 @@ static char *memory_stat_format(struct mem_cgroup *memcg) > > /* The above should easily fit into one page */ > WARN_ON_ONCE(seq_buf_has_overflowed(&s)); > - > - return s.buffer; > } > > #define K(x) ((x) << (PAGE_SHIFT-10)) > @@ -1570,7 +1566,10 @@ void mem_cgroup_print_oom_context(struct mem_cgroup *memcg, struct task_struct * > */ > void mem_cgroup_print_oom_meminfo(struct mem_cgroup *memcg) > { > - char *buf; > + /* Use static buffer, for the caller is holding oom_lock. */ > + static char buf[PAGE_SIZE]; > + > + lockdep_assert_held(&oom_lock); > > pr_info("memory: usage %llukB, limit %llukB, failcnt %lu\n", > K((u64)page_counter_read(&memcg->memory)), > @@ -1591,11 +1590,8 @@ void mem_cgroup_print_oom_meminfo(struct mem_cgroup *memcg) > pr_info("Memory cgroup stats for "); > pr_cont_cgroup_path(memcg->css.cgroup); > pr_cont(":"); > - buf = memory_stat_format(memcg); > - if (!buf) > - return; > + memory_stat_format(memcg, buf, sizeof(buf)); > pr_info("%s", buf); > - kfree(buf); > } > > /* > @@ -6335,11 +6331,11 @@ static int memory_events_local_show(struct seq_file *m, void *v) > static int memory_stat_show(struct seq_file *m, void *v) > { > struct mem_cgroup *memcg = mem_cgroup_from_seq(m); > - char *buf; > + char *buf = kmalloc(PAGE_SIZE, GFP_KERNEL); > > - buf = memory_stat_format(memcg); > if (!buf) > return -ENOMEM; > + memory_stat_format(memcg, buf, PAGE_SIZE); > seq_puts(m, buf); > kfree(buf); > return 0; > -- > 2.18.4 -- Michal Hocko SUSE Labs