From mboxrd@z Thu Jan 1 00:00:00 1970 Return-Path: X-Spam-Checker-Version: SpamAssassin 3.4.0 (2014-02-07) on aws-us-west-2-korg-lkml-1.web.codeaurora.org Received: from kanga.kvack.org (kanga.kvack.org [205.233.56.17]) by smtp.lore.kernel.org (Postfix) with ESMTP id 22FF4C0219B for ; Fri, 7 Feb 2025 17:29:23 +0000 (UTC) Received: by kanga.kvack.org (Postfix) id 8D82B6B0092; Fri, 7 Feb 2025 12:29:23 -0500 (EST) Received: by kanga.kvack.org (Postfix, from userid 40) id 889036B0093; Fri, 7 Feb 2025 12:29:23 -0500 (EST) X-Delivered-To: int-list-linux-mm@kvack.org Received: by kanga.kvack.org (Postfix, from userid 63042) id 751496B0095; Fri, 7 Feb 2025 12:29:23 -0500 (EST) X-Delivered-To: linux-mm@kvack.org Received: from relay.hostedemail.com (smtprelay0011.hostedemail.com [216.40.44.11]) by kanga.kvack.org (Postfix) with ESMTP id 5082B6B0092 for ; Fri, 7 Feb 2025 12:29:23 -0500 (EST) Received: from smtpin26.hostedemail.com (a10.router.float.18 [10.200.18.1]) by unirelay04.hostedemail.com (Postfix) with ESMTP id ED2AB1A1CA1 for ; Fri, 7 Feb 2025 17:29:22 +0000 (UTC) X-FDA: 83093834964.26.B2B816A Received: from mail-yb1-f170.google.com (mail-yb1-f170.google.com [209.85.219.170]) by imf20.hostedemail.com (Postfix) with ESMTP id EE4601C000A for ; Fri, 7 Feb 2025 17:29:20 +0000 (UTC) Authentication-Results: imf20.hostedemail.com; dkim=pass header.d=toxicpanda-com.20230601.gappssmtp.com header.s=20230601 header.b="I2p/MRIF"; spf=none (imf20.hostedemail.com: domain of josef@toxicpanda.com has no SPF policy when checking 209.85.219.170) smtp.mailfrom=josef@toxicpanda.com; dmarc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed/relaxed; d=hostedemail.com; s=arc-20220608; t=1738949361; h=from:from:sender:reply-to:subject:subject:date:date: message-id:message-id:to:to:cc:cc:mime-version:mime-version: content-type:content-type:content-transfer-encoding: in-reply-to:in-reply-to:references:references:dkim-signature; bh=pe1yEE3okQrQxigiaI6rQOtX6Rzf5xYLvQlxPdLkV2A=; b=ZJu/ZJtaHp9AYapqZ43hRK28m6T9OaP43hhucjXqQkDHPNKuDwnzo/5OBGBxJmvFiZtNzc nnvAt9lOz6bIeu97PMkAtQapXJSm/w6w893Rp6npElpX3vWxWUm4L+gtDfnX6lDH+ow9Sm yr8kMYcTwVZhezih0eu9rIKppwUSmUc= ARC-Authentication-Results: i=1; imf20.hostedemail.com; dkim=pass header.d=toxicpanda-com.20230601.gappssmtp.com header.s=20230601 header.b="I2p/MRIF"; spf=none (imf20.hostedemail.com: domain of josef@toxicpanda.com has no SPF policy when checking 209.85.219.170) smtp.mailfrom=josef@toxicpanda.com; dmarc=none ARC-Seal: i=1; s=arc-20220608; d=hostedemail.com; t=1738949361; a=rsa-sha256; cv=none; b=76NmiQ/6bEFVTL3uUreO79b/iF1pEMB38UsXgTEuHh+NE6klj62XQt4N1FEv/3/HCINfyf O5I6Qlwq+L1YzZlxNt6XhlkGrU9H0mUHWAb2LokHDLzS5+JPSg1bNNj3bWldAbtsQBwyGB OCMscEOSp2WcmP/O8NZVAUHmZUmEqS0= Received: by mail-yb1-f170.google.com with SMTP id 3f1490d57ef6-e5b1837d182so2880810276.0 for ; Fri, 07 Feb 2025 09:29:20 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=toxicpanda-com.20230601.gappssmtp.com; s=20230601; t=1738949360; x=1739554160; darn=kvack.org; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:from:to:cc:subject:date:message-id:reply-to; bh=pe1yEE3okQrQxigiaI6rQOtX6Rzf5xYLvQlxPdLkV2A=; b=I2p/MRIFcgzUEFF+LgWyiNNjeMmL6fWaPjeRPpzyJ3h4xFYkZoZG/fN3LLzjs5gMXt wUItLkTfMBo4SJB41rcqldLoDYN31GYkiNeoeuY/R+Cxx7cDT0H1jr3/77n2/HWODjX6 mbU4pzBWd5XHFvR7QwEaAUnqtThec7wLUe3OcYV5W0Kx7kXRcIp2UPBttRQTiD1TjFBg ZxO8UfVZw8MS3jfhh26CQsmM/hsi3V/m3lsk8l446MMAgpvZQMb8hMtiG0XzhIwVq9Yi bqLAbSIauQlYP2kHPpIT8ai2GuEUPoh0jbYU/nw3bDiuDmOF4XX6seRW8aK8ZWxa4+pm OWHg== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20230601; t=1738949360; x=1739554160; h=in-reply-to:content-disposition:mime-version:references:message-id :subject:cc:to:from:date:x-gm-message-state:from:to:cc:subject:date :message-id:reply-to; bh=pe1yEE3okQrQxigiaI6rQOtX6Rzf5xYLvQlxPdLkV2A=; b=xAtQLCP9bO/N1bMpitFC9zF5dkc3eYcxTrDGYjwCAZR3y/+FBCr5ef2TjMEFS/9Dtg nMV+uUfeZr8V8rnRoI1xtD6ctabJOzWykD8FhgsHXFkJohLO88c7OsLkXVArCV83mJXP sE/hM+y3JiuO16ecs7ig0GMNjy9gJo0gxwQOubQeA3tUOeUkauq626pBLtajwYt2EVYF BF8b+XjmzT1BqqwsiYlYdTDnZQtIsMHoLTlcokyN4q3DnD0/I5+pRYinKr753f6CnNpv 9MWEPiV5yo9VDp1ew4ybmsFnxbgSeSqwYV0M7FgNWa9UTRsF47VmnJh4MmXvC2GztJDv XiCQ== X-Forwarded-Encrypted: i=1; AJvYcCW41+mVT9cdQUhLltf8BMCfYPj3Jd32CWQr9YJ09JtrkQtcOkFuLyk3/iZ75Oj+L/ML75plywn/3Q==@kvack.org X-Gm-Message-State: AOJu0Yz47WQp88y2RlJoLOir3Wma6noZo7ZgE2qFiG2XU6oghN+EoKQU gpCGjFfBDLj2vCtG6rWVqnSzXe9ig/fximmZkJY5Iyvhkgop4KNnfeCjRmMYi08= X-Gm-Gg: ASbGncsaxoTs9plyyw9qcdlBs8QuudcYqEEandLhS0HfdurvsbXF0yG4iByqJ5VH1dU XJiEDE/T98K32pTLcRzAvgNqYbdksDWewHhTpgbFJc0w5w4JAcWm8Eg1LiXdZsp0SGU6fFuF3Yz 57YW3/p3WxqAqbSahyUPElKJrYXLVBValPywj2hN//IZ6tVsLrCePV8wgAwddvGksgvo8hbDjNL o5GpApAxia2V9BM83h3gqDXJkOewrwN1BnEEiMoWnRkTg/cUhdMzWqQeH4XxhWRLc55dkjvT1L4 /jUtCNaZ7muBDhnGcbyjArDpwD3t7ra2BYVZQ1NbJtf7UrKpiwoA X-Google-Smtp-Source: AGHT+IFwqoWDn7YO/bjFq7nIBquQqHcFTa9mKO/aRiVcSKON9nOTdNqg5xXZEzC8DXFG8NATZDlxfA== X-Received: by 2002:a05:6902:2089:b0:e5b:4a7b:c518 with SMTP id 3f1490d57ef6-e5b4a7bc795mr2727436276.1.1738949359866; Fri, 07 Feb 2025 09:29:19 -0800 (PST) Received: from localhost (syn-076-182-020-124.res.spectrum.com. [76.182.20.124]) by smtp.gmail.com with ESMTPSA id 3f1490d57ef6-e5b437d6841sm667699276.15.2025.02.07.09.29.18 (version=TLS1_3 cipher=TLS_AES_256_GCM_SHA384 bits=256/256); Fri, 07 Feb 2025 09:29:18 -0800 (PST) Date: Fri, 7 Feb 2025 12:29:17 -0500 From: Josef Bacik To: Vlastimil Babka Cc: Miklos Szeredi , Christian Heusel , Miklos Szeredi , regressions@lists.linux.dev, linux-kernel@vger.kernel.org, linux-fsdevel@vger.kernel.org, Joanne Koong , Matthew Wilcox , linux-mm , Mantas =?utf-8?Q?Mikul=C4=97nas?= Subject: Re: [REGRESSION][BISECTED] Crash with Bad page state for FUSE/Flatpak related applications since v6.13 Message-ID: <20250207172917.GA2072771@perftesting> References: <2f681f48-00f5-4e09-8431-2b3dbfaa881e@heusel.eu> <9cd88643-daa8-4379-be0a-bd31de277658@suse.cz> MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: <9cd88643-daa8-4379-be0a-bd31de277658@suse.cz> X-Rspamd-Queue-Id: EE4601C000A X-Stat-Signature: 9ueyaew9bsgioxw5kpweiy99gkq6es6u X-Rspam-User: X-Rspamd-Server: rspam01 X-HE-Tag: 1738949360-151219 X-HE-Meta: U2FsdGVkX1+QfmZF1vkU28akBSPhu6YwaEOeXQcjqs1rm9uw1KZaHkGjrqTFumRDKClROIwa00pncVxmnHbkJ+4dl7PpdYyWMfxh+o9eh+6smd/GG4g02MzeECWP7wwTJccokPM582VoLSDZ5QHOpqMUUaqHdgg01+QoVnaE258AGPljtS7GVtwe56fYolJ71tO5CENfOqfeGg7Ye98pEJbdbWY9UJCCOVMb4TS2ETqt/EXyV2whyw/AvjMmBVeFOxRw2wsd0FLH/ESMsXsHVOq8r0CGjCyGnhAcRrPxBfYPnqor2zWdzulBinRMBK2n8Upmrwba5dgBaCyXPpgi2cQ01rxr0GMmaXZqfZkmAVzQBGRz6tTD1lhjadSDxLQfwsWwu7rAXxrhNPylt9g7qopVwXnCjEtL2KOtpN/GIbHIHklRvWYi0yNtkOedNe2G2Rf7lssAqMXLhIxzfstmA0Yw9MmRuarkPUaEFUR3ltxkvkr431DW6JCESRRkBWxde32Q5Hv3SUsjn69zfeLWtItuQ+94zBoGLcjQ3JeYede2QbBwRC1lb1HAVUIrddpKrpsynyn0LRsIiXfxvUyUspDczzA6PoJAS0+jEDwDgBWg5TR44eR3dunws9mfAaH8NFq03n5fZlonOdCpbbxne+AoXbi7tSJCRx8AfZ2E4K4LQPo4ojCUXNPDPKM42zw5fEP1uCUtdKRztJoE/GXBtiRwawXQeoLRI3D8+6sW81tmHeXnVyaTUUiw/a6AKSQ2t7uJ8rWzaC5gxrlpu6A9zmW4+SWGiHFW/0FQeRzrVGqg814xJD54AyB34TnFO1OzdfbPTiHVFDsWvlV6EdigtdsmYVddq8OxlssfNadgy1PkM/Gwgc3Dh8O/pAg7MYz38lOaM3afFAJB9aa9mqYCQDcXTuZkHFRkZcydV7+DKre53gwmebQ5vczjABctFwSiVJ/1MQDReTyamt7e8iH vSFbQEPN XHAPnk2MKzQdisprFlPNLYLirE4rJx1Eg5pz13HMX83WhrayvRrVXUfDdVD0FaMvDYhKA/HTudvQTQTqGOcSjmAUoxAVTRN6SG03BZbA9m5TaQe1GrTAEMdHpCKWWCLFjF+/cjxwpb3Ds5gjltXoajRoA/O4HHY6JfefsG2qGnkW/fyyDTROOwKq/4zqwtFWDV9MaVSuQplOuGzX0OtgdwC7+EjqGtiohwQsbswoG3tcds1BQr70/cQhiCLHAclyN5itlf6IQF3lPHnuy4WUOHFkXLebuhP3eIBNrIDplDCsnA17j2Qd773tPMxhsA9+3ZA4chcUDsjUh4YCGF4iLujOPILS44jTTQ8mleVCJtxJpTRJvZhZAHUwJ3NTFLRth8mny0uiBsVbnPSpTQGG1g+rQu629eZIAnUCTJZsTg1S1TZtLzS370Xz+Rlo4gKp3WiCZPr65r2fWOjI= X-Bogosity: Ham, tests=bogofilter, spamicity=0.000000, version=1.2.4 Sender: owner-linux-mm@kvack.org Precedence: bulk X-Loop: owner-majordomo@kvack.org List-ID: List-Subscribe: List-Unsubscribe: On Fri, Feb 07, 2025 at 05:49:34PM +0100, Vlastimil Babka wrote: > On 2/7/25 10:34, Miklos Szeredi wrote: > > [Adding Joanne, Willy and linux-mm]. > > > > > > On Thu, 6 Feb 2025 at 11:54, Christian Heusel wrote: > >> > >> Hello everyone, > >> > >> we have recently received [a report][0] on the Arch Linux Gitlab about > >> multiple users having system crashes when using Flatpak programs and > >> related FUSE errors in their dmesg logs. > >> > >> We have subsequently bisected the issue within the mainline kernel tree > >> to the following commit: > >> > >> 3eab9d7bc2f4 ("fuse: convert readahead to use folios") > > I see that commit removes folio_put() from fuse_readpages_end(). Also it now > uses readahead_folio() in fuse_readahead() which does folio_put(). So that's > suspicious to me. It might be storing pointers to pages to ap->pages without > pinning them with a refcount. > > But I don't understand the code enough to know what's the proper fix. A > probably stupid fix would be to use __readahead_folio() instead and keep the > folio_put() in fuse_readpages_end(). Agreed, I'm also confused as to what the right thing is here. It appears the rules are "if the folio is locked, nobody messes with it", so it's not "correct" to hold a reference on the folio while it's being read. I don't love this way of dealing with folios, but that seems to be the way it's always worked. I went and looked at a few of the other file systems and we have NFS which holds it's own reference to the folio while the IO is outstanding, which FUSE is most similar to NFS so this would make sense to do. Btrfs however doesn't do this, BUT we do set_folio_private (or whatever it's called) so that keeps us from being reclaimed since we'll try to lock the folio before we do the reclaim. So perhaps that's the issue here? We need to have a private on the folio + the folio locked to make sure it doesn't get reclaimed while it's out being read? I'm knee deep in other things, if we want a quick fix then I think your suggestion is correct Vlastimil. But I definitely want to know what Willy expects to be the proper order of operations here, and if this is exactly what we're supposed to be doing then something else is going wrong and we should try to reproduce locally and figure out what's happening. Thanks, Josef