Re: [syzbot] WARNING in do_mkdirat

Re: [syzbot] WARNING in do_mkdirat

[Hillf Danton]

On 11 Dec 2022 02:52:57 +0000 Al Viro <viro@zeniv.linux.org.uk>
> On Sat, Dec 10, 2022 at 06:30:22PM -0800, syzbot wrote:
> > Hello,
> > 
> > syzbot has tested the proposed patch but the reproducer is still triggering an issue:
> > WARNING in done_path_create
> 
> How many times does it need to be repeated that ANY BUG REPORTS INVOLVING NTFS3 IN
> REPRODUCER NEED TO BE CCED TO MAINTAINERS OF NTFS3?
> 
> I'm done with any syzbot output.  From now on it's getting triaged
> straight to /dev/null here.

Calm downnnnnn Sir even if this is not the east ender style.

Frankly no interest here at all wasting any network bandwidth just to get you
interrupted if it would take less than 72 hours to discover one of the beatles
you created. And actually more than double check is needed to ensure who
did that.

Re: [syzbot] WARNING in do_mkdirat

[Al Viro]

On Sun, Dec 11, 2022 at 03:56:12PM +0800, Hillf Danton wrote:
> On 11 Dec 2022 02:52:57 +0000 Al Viro <viro@zeniv.linux.org.uk>
> > On Sat, Dec 10, 2022 at 06:30:22PM -0800, syzbot wrote:
> > > Hello,
> > > 
> > > syzbot has tested the proposed patch but the reproducer is still triggering an issue:
> > > WARNING in done_path_create
> > 
> > How many times does it need to be repeated that ANY BUG REPORTS INVOLVING NTFS3 IN
> > REPRODUCER NEED TO BE CCED TO MAINTAINERS OF NTFS3?
> > 
> > I'm done with any syzbot output.  From now on it's getting triaged
> > straight to /dev/null here.
> 
> Calm downnnnnn Sir even if this is not the east ender style.
> 
> Frankly no interest here at all wasting any network bandwidth just to get you
> interrupted if it would take less than 72 hours to discover one of the beatles
> you created. And actually more than double check is needed to ensure who
> did that.

	The first iterations of the same suggestion had been a lot calmer...
One of the earlier examples: https://lore.kernel.org/all/YzEJ2D8kga+ZRDZx@ZenIV/
And I distinctly remember similar attempts from other folks.

	It's really a matter of triage; as it is, syzkaller folks are
expecting that any mail from the bot will be looked into by everyone
on fsdevel, on the off-chance that it's relevant for them.  What's
more, it's not just "read the mail" - information in the mail body
is next to useless in such situations.  So you are asking to
	* start a browser
	* cut'n'paste the URL from MUA
	* dig around in the files linked to the damn thing
... all of that for an fs maintainer to see if his filesystem is
even present?  Seriously?  For each syzbot fsdevel posting?

	I would have looked at it anyway; granted, seeing ntfs3 I'd chalked
it up to ntfs bugs (fs/ntfs3 has not been there for long and it didn't get
outright memory corruptors beaten out of it yet).

	But how the bleeding hell are ntfs folks supposed to guess that
this report might be relevant for them?  Same for XFS, ext4, orangefs,
et sodding cetera - and for most of those any of such reports would've
ended up wasted time for the good and simple reasons that it's not
any fs they'd been involved with.

	What really pisses me off is that on the sending side the
required check is trivial - if you are going to fuzz a filesystem,
put a note into report, preferably in subject.  Sure, it's your
code, you get to decide what to spend your time upon (you == syzkaller
maintainers).  But please keep in mind that for recepients it's
a lot of recurring work, worthless for the majority of those who
end up bothering with it.  Every time they receive a mail from
that source.

	Ignore polite suggestions enough times, earn a mix of
impolite ones and .procmailrc recipes, it's that simple...

Re: [syzbot] WARNING in do_mkdirat

[Hillf Danton]

On 11 Dec 2022 08:39:18 +0000 Al Viro <viro@zeniv.linux.org.uk>
> On Sun, Dec 11, 2022 at 03:56:12PM +0800, Hillf Danton wrote:
> > On 11 Dec 2022 02:52:57 +0000 Al Viro <viro@zeniv.linux.org.uk>
> > > On Sat, Dec 10, 2022 at 06:30:22PM -0800, syzbot wrote:
> > > > Hello,
> > > > 
> > > > syzbot has tested the proposed patch but the reproducer is still triggering an issue:
> > > > WARNING in done_path_create
> > > 
> > > How many times does it need to be repeated that ANY BUG REPORTS INVOLVING NTFS3 IN
> > > REPRODUCER NEED TO BE CCED TO MAINTAINERS OF NTFS3?
> > > 
> > > I'm done with any syzbot output.  From now on it's getting triaged
> > > straight to /dev/null here.
> > 
> > Calm downnnnnn Sir even if this is not the east ender style.
> > 
> > Frankly no interest here at all wasting any network bandwidth just to get you
> > interrupted if it would take less than 72 hours to discover one of the beatles
> > you created. And actually more than double check is needed to ensure who
> > did that.
> 
> 	The first iterations of the same suggestion had been a lot calmer...
> One of the earlier examples: https://lore.kernel.org/all/YzEJ2D8kga+ZRDZx@ZenIV/
> And I distinctly remember similar attempts from other folks.
> 
> 	It's really a matter of triage; as it is, syzkaller folks are
> expecting that any mail from the bot will be looked into by everyone
> on fsdevel, on the off-chance that it's relevant for them.  What's

FILESYSTEMS (VFS and infrastructure)
M:	Alexander Viro <viro@zeniv.linux.org.uk>
L:	linux-fsdevel@vger.kernel.org
S:	Maintained
F:	fs/*
F:	include/linux/fs.h
F:	include/linux/fs_types.h
F:	include/uapi/linux/fs.h
F:	include/uapi/linux/openat2.h

 _> ls fs/* | grep ntfs
fs/ntfs:
ntfs.h
fs/ntfs3:
fsntfs.c
ntfs.h
ntfs_fs.h

Why not change what you really want to cover instead of complaining once more
and opting to triage?

> more, it's not just "read the mail" - information in the mail body
> is next to useless in such situations.  So you are asking to
> 	* start a browser
> 	* cut'n'paste the URL from MUA
> 	* dig around in the files linked to the damn thing
> ... all of that for an fs maintainer to see if his filesystem is
> even present?  Seriously?  For each syzbot fsdevel posting?
> 
> 	I would have looked at it anyway; granted, seeing ntfs3 I'd chalked
> it up to ntfs bugs (fs/ntfs3 has not been there for long and it didn't get
> outright memory corruptors beaten out of it yet).
> 
> 	But how the bleeding hell are ntfs folks supposed to guess that
> this report might be relevant for them?  Same for XFS, ext4, orangefs,
> et sodding cetera - and for most of those any of such reports would've
> ended up wasted time for the good and simple reasons that it's not
> any fs they'd been involved with.
> 
> 	What really pisses me off is that on the sending side the
> required check is trivial - if you are going to fuzz a filesystem,
> put a note into report, preferably in subject.  Sure, it's your
> code, you get to decide what to spend your time upon (you == syzkaller
> maintainers).  But please keep in mind that for recepients it's
> a lot of recurring work, worthless for the majority of those who
> end up bothering with it.  Every time they receive a mail from
> that source.
> 
> 	Ignore polite suggestions enough times, earn a mix of
> impolite ones and .procmailrc recipes, it's that simple...
> 

Re: [syzbot] WARNING in do_mkdirat

[Matthew Wilcox]

On Sun, Dec 11, 2022 at 06:22:08PM +0800, Hillf Danton wrote:
> On 11 Dec 2022 08:39:18 +0000 Al Viro <viro@zeniv.linux.org.uk>
> > On Sun, Dec 11, 2022 at 03:56:12PM +0800, Hillf Danton wrote:
> > > On 11 Dec 2022 02:52:57 +0000 Al Viro <viro@zeniv.linux.org.uk>
> > > > On Sat, Dec 10, 2022 at 06:30:22PM -0800, syzbot wrote:
> > > > > Hello,
> > > > > 
> > > > > syzbot has tested the proposed patch but the reproducer is still triggering an issue:
> > > > > WARNING in done_path_create
> > > > 
> > > > How many times does it need to be repeated that ANY BUG REPORTS INVOLVING NTFS3 IN
> > > > REPRODUCER NEED TO BE CCED TO MAINTAINERS OF NTFS3?
> > > > 
> > > > I'm done with any syzbot output.  From now on it's getting triaged
> > > > straight to /dev/null here.
> > > 
> > > Calm downnnnnn Sir even if this is not the east ender style.
> > > 
> > > Frankly no interest here at all wasting any network bandwidth just to get you
> > > interrupted if it would take less than 72 hours to discover one of the beatles
> > > you created. And actually more than double check is needed to ensure who
> > > did that.
> > 
> > 	The first iterations of the same suggestion had been a lot calmer...
> > One of the earlier examples: https://lore.kernel.org/all/YzEJ2D8kga+ZRDZx@ZenIV/
> > And I distinctly remember similar attempts from other folks.
> > 
> > 	It's really a matter of triage; as it is, syzkaller folks are
> > expecting that any mail from the bot will be looked into by everyone
> > on fsdevel, on the off-chance that it's relevant for them.  What's
> 
> FILESYSTEMS (VFS and infrastructure)
> M:	Alexander Viro <viro@zeniv.linux.org.uk>
> L:	linux-fsdevel@vger.kernel.org
> S:	Maintained
> F:	fs/*
> F:	include/linux/fs.h
> F:	include/linux/fs_types.h
> F:	include/uapi/linux/fs.h
> F:	include/uapi/linux/openat2.h
> 
>  _> ls fs/* | grep ntfs
> fs/ntfs:
> ntfs.h
> fs/ntfs3:
> fsntfs.c
> ntfs.h
> ntfs_fs.h
> 
> Why not change what you really want to cover instead of complaining once more
> and opting to triage?

You've completely misunderstood Al's point.  He's not whining about
being cc'd, he's pointing at this is ONLY USEFUL IF THE NTFS3
MAINTAINERS ARE CC'd.  And they're not.  So this is just noise.
And enough noise means that signal is lost.

> > more, it's not just "read the mail" - information in the mail body
> > is next to useless in such situations.  So you are asking to
> > 	* start a browser
> > 	* cut'n'paste the URL from MUA
> > 	* dig around in the files linked to the damn thing
> > ... all of that for an fs maintainer to see if his filesystem is
> > even present?  Seriously?  For each syzbot fsdevel posting?
> > 
> > 	I would have looked at it anyway; granted, seeing ntfs3 I'd chalked
> > it up to ntfs bugs (fs/ntfs3 has not been there for long and it didn't get
> > outright memory corruptors beaten out of it yet).
> > 
> > 	But how the bleeding hell are ntfs folks supposed to guess that
> > this report might be relevant for them?  Same for XFS, ext4, orangefs,
> > et sodding cetera - and for most of those any of such reports would've
> > ended up wasted time for the good and simple reasons that it's not
> > any fs they'd been involved with.
> > 
> > 	What really pisses me off is that on the sending side the
> > required check is trivial - if you are going to fuzz a filesystem,
> > put a note into report, preferably in subject.  Sure, it's your
> > code, you get to decide what to spend your time upon (you == syzkaller
> > maintainers).  But please keep in mind that for recepients it's
> > a lot of recurring work, worthless for the majority of those who
> > end up bothering with it.  Every time they receive a mail from
> > that source.
> > 
> > 	Ignore polite suggestions enough times, earn a mix of
> > impolite ones and .procmailrc recipes, it's that simple...
> > 
> 

Re: [syzbot] WARNING in do_mkdirat

[Al Viro]

On Sun, Dec 11, 2022 at 03:46:12PM +0000, Matthew Wilcox wrote:

> You've completely misunderstood Al's point.  He's not whining about
> being cc'd, he's pointing at this is ONLY USEFUL IF THE NTFS3
> MAINTAINERS ARE CC'd.  And they're not.  So this is just noise.
> And enough noise means that signal is lost.

Precisely.  Nobody asked for removing fsdevel from Cc, and I do read
what gets posted there.  That is not the problem.

Again, the problem is that extracting the information from those reports
is more inconvenient than it needs to be.

For anything that involves buzzing a specific filesystem the type of
that filesystem is highly relevant.  It's trivial to generate, it had
been (politely) asked for many times over the last months and it keeps
getting nowhere.

A combination of "email" and "cost-shifting" pushes the buttons you
really don't want pushed, because the output is not just occasional
all-caps repeats of original request - quiet .procmailrc rules tend to
stay around for a long time.

Cost-shifting is definitely there - one-time (and trivial, at that)
modification of the bot vs. O(number of fs developers * number of bot
reports that don't get marked).

From what I've seen in various discussions, the assumption of syzkaller
folks seems to be that most of the relevant information is in stack
trace and that's sufficient for practical purposes - anything beyond
that is seen as unwarranted special-casing.  Such assumption does not
match the reality.

For one thing, anything that involves fuzzing the filesystem with
corrupted image (which is a common enough class of reports) is highly
filesystem-specific.  Such reports *are* valuable - this is the area
that doesn't get anywhere near enough routine testing.  However, if
you consider the triage path for such report you get
	* OK, something's clearly wrong with the state of some
filesystem or VFS data structures.  Interesting.
	* Might be something in some specific filesystem, might
be something fs-independent in VFS, might be VFS mishandling a rare
but legitimate behaviour of some method.  Do we have anything to
narrow it down?  Nothing in the stack trace, more's the pity.
	* Has that code just returned from a method?  Yep - complains
about the state of rwsem it unlocks, which happens around the method
calls.  Definitely so in case of mkdir...  Filesystem type has
just become highly relevant - there are *many* ->mkdir() instances,
and each brings along its own set of helpers, etc.
	* No way to tell which fs type it is going by the stuff in
email body.  It's obviously on x86 and it's on mainline tree.  Which
doesn't say much.  Hell with it, the only way to tell is apparently to
follow a link in there.  Which is to say, cut'n'paste an URL from
xterm with ssh session with mutt(1) in it into shell (if I feel
like using lynx(1) or wget(1)) or into a new tab in chromium (which
I had running at the time).
	* ouch.
 https://syzkaller.appspot.com/bug?id=e8da337c58f1a620dce15898ae21af03e0a5ddb3
 # See https://goo.gl/kgGztJ for information about syzkaller reproducers.
 #{"threaded":true,"repeat":true,"procs":1,"slowdown":1,"sandbox":"","sandbox_arg":0,"close_fds":false,"tmpdir":true}
 syz_mount_image$ntfs3(&(0x7f000001f180), &(0x7f00000000c0)='./file2\x00', 0x80008a, &(0x7f0000000300)=ANY=[@ANYRES64=0x0, @ANYRES8=0x0, @ANYBLOB="9bb8aca074f613bfd79072432703c68cf00d"], 0x1, 0x1f195, &(0x7f000001f1c0)=<PAGES AND PAGES OF LINE NOISE>

	Presumably somewhere past those pages of line noise there's
some information about what it does, but you know what?  By that point
I really don't give a rat's arse.  What that thing is doing is, by
the look of it, fuzzing ntfs3.  And there are very good odds that
it might be actually testing how well ntfs3 can cope with image
validation and with failure exits in general.
	fs/ntfs3 is still very rough in those areas; such reports
would be quite useful for NTFS3 maintainers.  Who are (again) not
Cc'ed on that.  Sure, they are probably reading fsdevel, and had
that report been brought to their notice, they might do something
useful with it.  The odds are, they would be able to ask reasonable
questions and would get help if needed.
	What the hell I am supposed to do with that, though?

Option 1: dig through ntfs3 ->mkdir(), try to make a guess about the
expected execution path, see what went wrong, etc.?  Sure, I can do that
- with a few days of RTFS I could probably familiarize myself with their
code enough to serve as emergency co-maintainer.  Been there, done that
for some of unsupported filesystems.  It doesn't scale.

Option 2: hope that ntfs3 folks had seen the same report, did the
same triage and arrived at "useful for us, let's deal with it".
REALLY does not scale, and here's why: it might as well had been
fuzzing xfs instead.  Or btrfs.  Or ext4.  Or any other filesystem.
Are all fs maintainers expected to decode that kind of reports,
even though there's not a damn thing to indicate that they might
be relevant to them?  With "wasted time - not ours" as outcome for
the majority of those people.

Option 3: ping ntfs3 folks, bringing the report to their attention.
Possible, and I'd done just that a couple of times.  Should I *AND*
an unpredictable amount of people on fsdevel take to spamming
ntfs3 folks every time such a report appears?  So every time something
of that sort gets reported, N people do that kind of triage, arrive
to "oh, it's ntfs3 fuzzing" and send N pings to maintainers?  Wonderful,
that...

Option 4: ask syzkaller folks to put the relevant information into
the mailed reports.  With "fuzzing $FILESYSTEM" somewhere it would
be instantly visible, ideally - with *additional* Cc (I'm not asking
to drop fsdevel, etc.) that would bring it to attention of maintainers
in question.  Tried that.  Quite a few times.  Got nowhere.

Option 5: just ignore the syzkaller reports.


That goes for me; similar analysis applies for other people on
fsdevel, except that those whose primary interest is in specific
filesystem are less likely to be interested in reports that refer
to VFS internals.

Face it, the underlying assumption is broken - for a large class
of reports the stack trace does not contain the relevant information.
It needs to be augmented by the data that should be very easy to
get for the bot.  Sure, your code, your priorities, but reports
are only useful when they are not ignored and training people to
ignore those is a bad idea...

Re: [syzbot] WARNING in do_mkdirat

[Hillf Danton]

On 11 Dec 2022 15:46:12 +0000 Matthew Wilcox <willy@infradead.org>
> On Sun, Dec 11, 2022 at 06:22:08PM +0800, Hillf Danton wrote:
> > On 11 Dec 2022 08:39:18 +0000 Al Viro <viro@zeniv.linux.org.uk>
> > > On Sun, Dec 11, 2022 at 03:56:12PM +0800, Hillf Danton wrote:
> > > > On 11 Dec 2022 02:52:57 +0000 Al Viro <viro@zeniv.linux.org.uk>
> > > > > On Sat, Dec 10, 2022 at 06:30:22PM -0800, syzbot wrote:
> > > > > > Hello,
> > > > > > 
> > > > > > syzbot has tested the proposed patch but the reproducer is still triggering an issue:
> > > > > > WARNING in done_path_create
> > > > > 
> > > > > How many times does it need to be repeated that ANY BUG REPORTS INVOLVING NTFS3 IN
> > > > > REPRODUCER NEED TO BE CCED TO MAINTAINERS OF NTFS3?
> > > > > 
> > > > > I'm done with any syzbot output.  From now on it's getting triaged
> > > > > straight to /dev/null here.
> > > > 
> > > > Calm downnnnnn Sir even if this is not the east ender style.
> > > > 
> > > > Frankly no interest here at all wasting any network bandwidth just to get you
> > > > interrupted if it would take less than 72 hours to discover one of the beatles
> > > > you created. And actually more than double check is needed to ensure who
> > > > did that.
> > > 
> > > 	The first iterations of the same suggestion had been a lot calmer...
> > > One of the earlier examples: https://lore.kernel.org/all/YzEJ2D8kga+ZRDZx@ZenIV/
> > > And I distinctly remember similar attempts from other folks.
> > > 
> > > 	It's really a matter of triage; as it is, syzkaller folks are
> > > expecting that any mail from the bot will be looked into by everyone
> > > on fsdevel, on the off-chance that it's relevant for them.  What's
> > 
> > FILESYSTEMS (VFS and infrastructure)
> > M:	Alexander Viro <viro@zeniv.linux.org.uk>
> > L:	linux-fsdevel@vger.kernel.org
> > S:	Maintained
> > F:	fs/*
> > F:	include/linux/fs.h
> > F:	include/linux/fs_types.h
> > F:	include/uapi/linux/fs.h
> > F:	include/uapi/linux/openat2.h
> > 
> >  _> ls fs/* | grep ntfs
> > fs/ntfs:
> > ntfs.h
> > fs/ntfs3:
> > fsntfs.c
> > ntfs.h
> > ntfs_fs.h
> > 
> > Why not change what you really want to cover instead of complaining once more
> > and opting to triage?
> 
> You've completely misunderstood Al's point.  He's not whining about
> being cc'd, he's pointing at this is ONLY USEFUL IF THE NTFS3
> MAINTAINERS ARE CC'd.  And they're not.  So this is just noise.
> And enough noise means that signal is lost.

Call Trace:
 <TASK>
 inode_unlock include/linux/fs.h:761 [inline]
 done_path_create fs/namei.c:3857 [inline]
 do_mkdirat+0x2de/0x550 fs/namei.c:4064
 __do_sys_mkdirat fs/namei.c:4076 [inline]
 __se_sys_mkdirat fs/namei.c:4074 [inline]
 __x64_sys_mkdirat+0x85/0x90 fs/namei.c:4074
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x3d/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x63/0xcd

Given the call trace above, how do you know the ntfs3 guys should be also
Cced in addition to AV? What if it would take more than three months for
syzbot to learn the skills in your mind? What is preventing you routing
the report to ntfs3?

Re: [syzbot] WARNING in do_mkdirat

[Theodore Ts'o]

On Mon, Dec 12, 2022 at 11:29:11AM +0800, Hillf Danton wrote:
> > You've completely misunderstood Al's point.  He's not whining about
> > being cc'd, he's pointing at this is ONLY USEFUL IF THE NTFS3
> > MAINTAINERS ARE CC'd.  And they're not.  So this is just noise.
> > And enough noise means that signal is lost.
> 
> Call Trace:
>  <TASK>
>  inode_unlock include/linux/fs.h:761 [inline]
>  done_path_create fs/namei.c:3857 [inline]
>  do_mkdirat+0x2de/0x550 fs/namei.c:4064
>  __do_sys_mkdirat fs/namei.c:4076 [inline]
>  __se_sys_mkdirat fs/namei.c:4074 [inline]
>  __x64_sys_mkdirat+0x85/0x90 fs/namei.c:4074
>  do_syscall_x64 arch/x86/entry/common.c:50 [inline]
>  do_syscall_64+0x3d/0xb0 arch/x86/entry/common.c:80
>  entry_SYSCALL_64_after_hwframe+0x63/0xcd
> 
> Given the call trace above, how do you know the ntfs3 guys should be also
> Cced in addition to AV? What if it would take more than three months for
> syzbot to learn the skills in your mind? What is preventing you routing
> the report to ntfs3?

If it takes 3 months for syzbot to take a look at the source code in
their own #!@?! reproducer, or just to take a look at the strace link
in the dashboard:

[pid  3639] mount("/dev/loop0", "./file2", "ntfs3", MS_NOSUID|MS_NOEXEC|MS_DIRSYNC|MS_I_VERSION, "") = 0

There's something really wrong.  The point Al has been making (and
I've been making for multiple years) is that Syzbot has the
information, but unfortunately, at the moment, it is only analyzing
the the stack trace, and it is not doing things that really could be
done automatically --- and cloud VM time is cheap, and upstream
maintainer time is expensive.  So by not improving syzbot in a way
that really shouldn't be all that difficult, the syzbot maintainers is
disrespectiving the time of the upstream maintainers.

So sure, we could ask Linus to triage all syzbot reports --- or we
could ask Al to triage all syzbot file system reports --- but that is
not a good use of upstream resources.

And "we didn't know this is super annoying" isn't an excuse, because
I've been asking for things like this *before* the COVID pandemic.  So
if the Syzbot team won't listen to observations by a random Google
engineer who happens to be an ext4 maintainer (or rather, I'm sure
they were listening, but they didn't consider it important enough to
staff and put on the roadmap), maybe something a bit
more.... assertive by Al is something that will inspire them to
prioritize this feature request "above the fold".  :-)

And Al does have a point --- if a lot of upstream maintainers consider
Syzbot reports to be less than useful, they will either auto-file
reports to a junk folder, or just ignore the Syzbot reports because
they are busy and the Probability(Usefulness) is close to zero, then
recovering from that black eye to Syzbot's reputation is going to be a
lot more difficult than if Syzbot was made more respectful of upstream
maintainer time much earlier.

Now, to be fair to the Syzbot team, the Syzbot console has gotten much
better.  You can now download the syzbot trace, and download the
mounted file system, when before, you had to do a lot more work to
extract the file system (which is stored in separate constant C
array's as compressed data) from the C reproducer.  So have things
have gotten better.

But at the same time, characterizing a syzbot report is something to
be done by every file system maintainer who looks as a syzbot report,
because there is no way to add a tag to the syzbot report that this
particular syzbot report *really* is an ntfs3 issue.  So any
information that a single developer figures out when triaging a bug
(is this potentially an ext4 bug, nope, it's an ntfs3 bug) has to be
replicated by every single kernel developer looking at the Syzbot
dashboard.  Which again, is not respectful of upstream maintainers'
time.

						- Ted

Re: [syzbot] WARNING in do_mkdirat

[Marco Elver]

On Mon, 12 Dec 2022 at 19:58, Theodore Ts'o <tytso@mit.edu> wrote:
>
> On Mon, Dec 12, 2022 at 11:29:11AM +0800, Hillf Danton wrote:
> > > You've completely misunderstood Al's point.  He's not whining about
> > > being cc'd, he's pointing at this is ONLY USEFUL IF THE NTFS3
> > > MAINTAINERS ARE CC'd.  And they're not.  So this is just noise.
> > > And enough noise means that signal is lost.
> >
> > Call Trace:
> >  <TASK>
> >  inode_unlock include/linux/fs.h:761 [inline]
> >  done_path_create fs/namei.c:3857 [inline]
> >  do_mkdirat+0x2de/0x550 fs/namei.c:4064
> >  __do_sys_mkdirat fs/namei.c:4076 [inline]
> >  __se_sys_mkdirat fs/namei.c:4074 [inline]
> >  __x64_sys_mkdirat+0x85/0x90 fs/namei.c:4074
> >  do_syscall_x64 arch/x86/entry/common.c:50 [inline]
> >  do_syscall_64+0x3d/0xb0 arch/x86/entry/common.c:80
> >  entry_SYSCALL_64_after_hwframe+0x63/0xcd
> >
> > Given the call trace above, how do you know the ntfs3 guys should be also
> > Cced in addition to AV? What if it would take more than three months for
> > syzbot to learn the skills in your mind? What is preventing you routing
> > the report to ntfs3?
>
> If it takes 3 months for syzbot to take a look at the source code in
> their own #!@?! reproducer, or just to take a look at the strace link
> in the dashboard:
>
> [pid  3639] mount("/dev/loop0", "./file2", "ntfs3", MS_NOSUID|MS_NOEXEC|MS_DIRSYNC|MS_I_VERSION, "") = 0
>
> There's something really wrong.  The point Al has been making (and
> I've been making for multiple years) is that Syzbot has the
> information, but unfortunately, at the moment, it is only analyzing
> the the stack trace, and it is not doing things that really could be
> done automatically --- and cloud VM time is cheap, and upstream
> maintainer time is expensive.  So by not improving syzbot in a way
> that really shouldn't be all that difficult, the syzbot maintainers is
> disrespectiving the time of the upstream maintainers.
>
> So sure, we could ask Linus to triage all syzbot reports --- or we
> could ask Al to triage all syzbot file system reports --- but that is
> not a good use of upstream resources.
>
> And "we didn't know this is super annoying" isn't an excuse, because
> I've been asking for things like this *before* the COVID pandemic.  So
> if the Syzbot team won't listen to observations by a random Google
> engineer who happens to be an ext4 maintainer (or rather, I'm sure
> they were listening, but they didn't consider it important enough to
> staff and put on the roadmap), maybe something a bit
> more.... assertive by Al is something that will inspire them to
> prioritize this feature request "above the fold".  :-)
>
> And Al does have a point --- if a lot of upstream maintainers consider
> Syzbot reports to be less than useful, they will either auto-file
> reports to a junk folder, or just ignore the Syzbot reports because
> they are busy and the Probability(Usefulness) is close to zero, then
> recovering from that black eye to Syzbot's reputation is going to be a
> lot more difficult than if Syzbot was made more respectful of upstream
> maintainer time much earlier.
>
> Now, to be fair to the Syzbot team, the Syzbot console has gotten much
> better.  You can now download the syzbot trace, and download the
> mounted file system, when before, you had to do a lot more work to
> extract the file system (which is stored in separate constant C
> array's as compressed data) from the C reproducer.  So have things
> have gotten better.
>
> But at the same time, characterizing a syzbot report is something to
> be done by every file system maintainer who looks as a syzbot report,
> because there is no way to add a tag to the syzbot report that this
> particular syzbot report *really* is an ntfs3 issue.  So any
> information that a single developer figures out when triaging a bug
> (is this potentially an ext4 bug, nope, it's an ntfs3 bug) has to be
> replicated by every single kernel developer looking at the Syzbot
> dashboard.  Which again, is not respectful of upstream maintainers'
> time.

This is being worked on:
https://github.com/google/syzkaller/issues/3393#issuecomment-1330305227

Teaching a bot the pattern matching skills of a human is non-trivial.
The current design will likely do the simplest thing: regex match
reproducers and map a match to some kernel source dir, for which the
maintainers are Cc'd. If you have better suggestions on how to
mechanize subsystem selection based on a reproducer, please shout.

Re: [syzbot] WARNING in do_mkdirat

[Al Viro]

On Mon, Dec 12, 2022 at 08:29:10PM +0100, Marco Elver wrote:

> > > Given the call trace above, how do you know the ntfs3 guys should be also
> > > Cced in addition to AV? What if it would take more than three months for
> > > syzbot to learn the skills in your mind?

Depends.  If you really are talking about the *BOT* learning to do
that on its own, it certainly would take more than 3 months; strong AI
is hard.  If, OTOH, it is not an AI research project and intervention of
somebody capable of passing the Turing test does not violate the purity
of experiment...  Surely converting "if it mounts an image as filesystem
of type $T, grep the tree for "MODULE_ALIAS_FS($T)" and treat that
as if a function from the resulting file had been found in stack trace"
into something usable for the bot should not take more than 3 months,
should it?

If expressing that rule really takes "more than three months", I would
suggest that something is very wrong with the bot architecture...

> Teaching a bot the pattern matching skills of a human is non-trivial.
> The current design will likely do the simplest thing: regex match
> reproducers and map a match to some kernel source dir, for which the
> maintainers are Cc'd. If you have better suggestions on how to
> mechanize subsystem selection based on a reproducer, please shout.

Er...  Yes?  Look, it's really that simple -
for i in `sed -ne 's/.*syz_mount_image$\([_[:alnum:]]*\).*/\1/p' <$REPRO`; do
	git grep -l "MODULE_ALIAS_FS(\"$i\")"
done | sort | uniq
gets you the list of files.  No, I'm not suggesting to go for that kind
of shell use, but it's clearly doable with regex and search over the source
for fixed strings.  Unless something's drastically wrong with the way the
bot is written, it should be capable of something as basic as that...

If it can't do that kind of mapping, precalculating it for given tree is
also not hard:
git grep 'MODULE_ALIAS_FS("'|sed -ne 's/\(.*\):.*MODULE_ALIAS_FS("\([_[:alnum:]]*\)".*/syz_mount_image$\2:\1/p'
will yield lines like
syz_mount_image$ext2:fs/ext2/super.c
syz_mount_image$ext2:fs/ext4/super.c
syz_mount_image$ext3:fs/ext4/super.c
syz_mount_image$ext4:fs/ext4/super.c
etc.  Surely turning *that* into whatever form the bot wants can't
be terribly hard? [*]

All of that assumes that pattern-matching in syzkaller reproducer is
expressible; if "we must do everything by call trace alone" is
a real limitation, we are SOL; stack trace simply doesn't have
that information.  Is there such an architectural limitation?

[*] depending upon config, ext2 could be mounted by ext2.ko and ext4.ko;
both have the same maillist for bug reports, so this ambiguity doesn't
matter - either match would do.

Re: [syzbot] WARNING in do_mkdirat

[Hillf Danton]

On 13 Dec 2022 01:44:08 +0000 Al Viro <viro@zeniv.linux.org.uk>
> On Mon, Dec 12, 2022 at 08:29:10PM +0100, Marco Elver wrote:
> 
> > > > Given the call trace above, how do you know the ntfs3 guys should be also
> > > > Cced in addition to AV? What if it would take more than three months for
> > > > syzbot to learn the skills in your mind?
> 
> Depends.  If you really are talking about the *BOT* learning to do
> that on its own, it certainly would take more than 3 months; strong AI
> is hard.  If, OTOH, it is not an AI research project and intervention of
> somebody capable of passing the Turing test does not violate the purity
> of experiment...  Surely converting "if it mounts an image as filesystem
> of type $T, grep the tree for "MODULE_ALIAS_FS($T)" and treat that
> as if a function from the resulting file had been found in stack trace"
> into something usable for the bot should not take more than 3 months,
> should it?
> 
> If expressing that rule really takes "more than three months", I would
> suggest that something is very wrong with the bot architecture...

Nope given the fixes to what the bot reported in the last three years,
certainly.

Re: [syzbot] WARNING in do_mkdirat

[Aleksandr Nogikh]

On Tue, Dec 13, 2022 at 2:44 AM Al Viro <viro@zeniv.linux.org.uk> wrote:
>
> On Mon, Dec 12, 2022 at 08:29:10PM +0100, Marco Elver wrote:
>
> > > > Given the call trace above, how do you know the ntfs3 guys should be also
> > > > Cced in addition to AV? What if it would take more than three months for
> > > > syzbot to learn the skills in your mind?
>
> Depends.  If you really are talking about the *BOT* learning to do
> that on its own, it certainly would take more than 3 months; strong AI
> is hard.  If, OTOH, it is not an AI research project and intervention of
> somebody capable of passing the Turing test does not violate the purity
> of experiment...  Surely converting "if it mounts an image as filesystem
> of type $T, grep the tree for "MODULE_ALIAS_FS($T)" and treat that
> as if a function from the resulting file had been found in stack trace"
> into something usable for the bot should not take more than 3 months,
> should it?
>
> If expressing that rule really takes "more than three months", I would
> suggest that something is very wrong with the bot architecture...
>
> > Teaching a bot the pattern matching skills of a human is non-trivial.
> > The current design will likely do the simplest thing: regex match
> > reproducers and map a match to some kernel source dir, for which the
> > maintainers are Cc'd. If you have better suggestions on how to
> > mechanize subsystem selection based on a reproducer, please shout.
>
> Er...  Yes?  Look, it's really that simple -
> for i in `sed -ne 's/.*syz_mount_image$\([_[:alnum:]]*\).*/\1/p' <$REPRO`; do
>         git grep -l "MODULE_ALIAS_FS(\"$i\")"
> done | sort | uniq
> gets you the list of files.  No, I'm not suggesting to go for that kind
> of shell use, but it's clearly doable with regex and search over the source
> for fixed strings.  Unless something's drastically wrong with the way the
> bot is written, it should be capable of something as basic as that...
>
> If it can't do that kind of mapping, precalculating it for given tree is
> also not hard:
> git grep 'MODULE_ALIAS_FS("'|sed -ne 's/\(.*\):.*MODULE_ALIAS_FS("\([_[:alnum:]]*\)".*/syz_mount_image$\2:\1/p'
> will yield lines like
> syz_mount_image$ext2:fs/ext2/super.c
> syz_mount_image$ext2:fs/ext4/super.c
> syz_mount_image$ext3:fs/ext4/super.c
> syz_mount_image$ext4:fs/ext4/super.c
> etc.  Surely turning *that* into whatever form the bot wants can't
> be terribly hard? [*]
>
> All of that assumes that pattern-matching in syzkaller reproducer is
> expressible; if "we must do everything by call trace alone" is
> a real limitation, we are SOL; stack trace simply doesn't have
> that information.  Is there such an architectural limitation?

Thanks for the feedback, and we regret the inconvenience this may have caused.

We've deployed a simple short term solution to the immediate issue:
syzbot will extract the involved filesystems from reproducers and use
this information to construct the email subject line and Cc the
related people/mailing lists. This should take effect starting next
week.

That being said, in response to the original feedback we have already
been planning comprehensive improvements to the subsystem selection
process that will support more than just filesystems. But
unfortunately, this is going to take longer to become available.

--
Aleksandr

>
> [*] depending upon config, ext2 could be mounted by ext2.ko and ext4.ko;
> both have the same maillist for bug reports, so this ambiguity doesn't
> matter - either match would do.

Re: [syzbot] WARNING in do_mkdirat

[Eric Biggers]

On Fri, Dec 16, 2022 at 04:48:34PM +0100, 'Aleksandr Nogikh' via syzkaller-bugs wrote:
> 
> Thanks for the feedback, and we regret the inconvenience this may have caused.
> 
> We've deployed a simple short term solution to the immediate issue:
> syzbot will extract the involved filesystems from reproducers and use
> this information to construct the email subject line and Cc the
> related people/mailing lists. This should take effect starting next
> week.
> 
> That being said, in response to the original feedback we have already
> been planning comprehensive improvements to the subsystem selection
> process that will support more than just filesystems. But
> unfortunately, this is going to take longer to become available.
> 

Thanks Aleksandr.  From what I can see, the fix is working for new filesystem
bugs: the filesystem(s) involved get added to the title and the recipients.

One question: what happens to all the open bugs, like this one ("WARNING in
do_mkdirat") that were reported before the syzbot fix?  Are they going to be
re-reported correctly?  Perhaps any bug whose reproducer includes
"syz_mount_image" and was reported before the date of this fix should be
invalidated more aggressively than usual, so that it can be re-reported?

- Eric

Re: [syzbot] WARNING in do_mkdirat

[Theodore Ts'o]

On Thu, Dec 29, 2022 at 01:17:31PM -0800, Eric Biggers wrote:
> Thanks Aleksandr.  From what I can see, the fix is working for new filesystem
> bugs: the filesystem(s) involved get added to the title and the recipients.
> 
> One question: what happens to all the open bugs, like this one ("WARNING in
> do_mkdirat") that were reported before the syzbot fix?  Are they going to be
> re-reported correctly?  Perhaps any bug whose reproducer includes
> "syz_mount_image" and was reported before the date of this fix should be
> invalidated more aggressively than usual, so that it can be re-reported?

As a related request/wish, it would be nice if those dashboard pages
that were created before the new-style reporting which includes the
file system image, strace otuput, etc., could get regenerated.  For example:

https://syzkaller.appspot.com/bug?id=be6e90ce70987950e6deb3bac8418344ca8b96cd

Even if someone has already submitted a proposed fix, I often like to
double-check that the fix is really fixing the true root cause of the
problem, as opposed to just making a superficial change that blocks
the current syzbot reproducer, but which will eventually be tripped
again because code is still vulnerable.  (For example, we might block
a straightforward reproducer by adding a check at mount time, but if
the superblocks get corrupted during the journal replay, we'd still be
vulnerable.)  And having access to the corrupted file system image,
and other associated reporting data, is often super-helpful in that
regard.

Also, can we at some point have the C reproducer actually using proper
C strings instead of hex digits?  It will make the reproducer much
more human understandable, as well making it easier to edit the string
when the developer is trying to do a better job minimizing the test
case than syzbot.  For example:

  memcpy(
      (void*)0x20000000,
      "\x6e\x6f\x75\x73\x65\x72\x5f\x78\x61\x74\x74\x72\x2c\x61\x63\x6c\x2c\x64"
      "\x65\x62\x75\x67\x5f\x77\x61\x6e\x74\x5f\x65\x78\x74\x72\x61\x5f\x69\x73"
      "\x69\x7a\x65\x3d\x30\x78\x30\x30\x30\x30\x30\x30\x30\x30\x30\x30\x30\x30"
      "\x30\x30\x38\x30\x2c\x6c\x61\x7a\x79\x74\x69\x6d\x65\x2c\x6e\x6f\x62\x68"
      "\x2c\x71\x75\x6f\x74\x61\x2c\x00\x3d\x93\x09\x61\x36\x5d\x73\x58\x9c",
      89);

Would be *much* more understable if it were:

  memcpy(
      (void*)0x20000000,
      "nouser_xattr,acl,debug_want_extra_isize=0x0000000000000080,lazytime,nobh,quota,",
      80);

Of course, something like:

   char mount_options[] = "nouser_xattr,acl,debug_want_extra_isize=0x0000000000000080,lazytime,nobh,quota,";

Would be even better (and more portable) than using random hex
addresses, but just simply using ASCII strings would be a good first
step.

Of course, filling in C structures instead of just a random memcpy of
hex garbage would be even *more* awesome, bunt I'll take what I can
get.  :-)

Another opportunity for improvement is to try minimizing mount
options, so it becomes more obvious which ones are required.  For
example, in the above example, a minimized mount option string would
have been:

  memcpy((void*)0x20000000, "debug_want_extra_isize=0x80,lazytime," 38);

Having a more minimized reproducer would improve the reliability of
the bisect, as well as making it easier for the developer to figure
out the true root cause of the problem.

Cheers,

					- Ted

Re: [syzbot] WARNING in do_mkdirat

[Randy Dunlap]

On 12/31/22 08:57, Theodore Ts'o wrote:
> On Thu, Dec 29, 2022 at 01:17:31PM -0800, Eric Biggers wrote:
>> Thanks Aleksandr.  From what I can see, the fix is working for new filesystem
>> bugs: the filesystem(s) involved get added to the title and the recipients.
>>
>> One question: what happens to all the open bugs, like this one ("WARNING in
>> do_mkdirat") that were reported before the syzbot fix?  Are they going to be
>> re-reported correctly?  Perhaps any bug whose reproducer includes
>> "syz_mount_image" and was reported before the date of this fix should be
>> invalidated more aggressively than usual, so that it can be re-reported?
> 
> As a related request/wish, it would be nice if those dashboard pages
> that were created before the new-style reporting which includes the
> file system image, strace otuput, etc., could get regenerated.  For example:
> 
> https://syzkaller.appspot.com/bug?id=be6e90ce70987950e6deb3bac8418344ca8b96cd
> 
> Even if someone has already submitted a proposed fix, I often like to
> double-check that the fix is really fixing the true root cause of the
> problem, as opposed to just making a superficial change that blocks
> the current syzbot reproducer, but which will eventually be tripped
> again because code is still vulnerable.  (For example, we might block
> a straightforward reproducer by adding a check at mount time, but if
> the superblocks get corrupted during the journal replay, we'd still be
> vulnerable.)  And having access to the corrupted file system image,
> and other associated reporting data, is often super-helpful in that
> regard.
> 
> Also, can we at some point have the C reproducer actually using proper
> C strings instead of hex digits?  It will make the reproducer much
> more human understandable, as well making it easier to edit the string
> when the developer is trying to do a better job minimizing the test
> case than syzbot.  For example:
> 
>   memcpy(
>       (void*)0x20000000,
>       "\x6e\x6f\x75\x73\x65\x72\x5f\x78\x61\x74\x74\x72\x2c\x61\x63\x6c\x2c\x64"
>       "\x65\x62\x75\x67\x5f\x77\x61\x6e\x74\x5f\x65\x78\x74\x72\x61\x5f\x69\x73"
>       "\x69\x7a\x65\x3d\x30\x78\x30\x30\x30\x30\x30\x30\x30\x30\x30\x30\x30\x30"
>       "\x30\x30\x38\x30\x2c\x6c\x61\x7a\x79\x74\x69\x6d\x65\x2c\x6e\x6f\x62\x68"
>       "\x2c\x71\x75\x6f\x74\x61\x2c\x00\x3d\x93\x09\x61\x36\x5d\x73\x58\x9c",
>       89);
> 
> Would be *much* more understable if it were:
> 
>   memcpy(
>       (void*)0x20000000,
>       "nouser_xattr,acl,debug_want_extra_isize=0x0000000000000080,lazytime,nobh,quota,",
>       80);
> 
> Of course, something like:
> 
>    char mount_options[] = "nouser_xattr,acl,debug_want_extra_isize=0x0000000000000080,lazytime,nobh,quota,";
> 
> Would be even better (and more portable) than using random hex
> addresses, but just simply using ASCII strings would be a good first
> step.
> 
> Of course, filling in C structures instead of just a random memcpy of
> hex garbage would be even *more* awesome, bunt I'll take what I can
> get.  :-)
> 
> Another opportunity for improvement is to try minimizing mount
> options, so it becomes more obvious which ones are required.  For
> example, in the above example, a minimized mount option string would
> have been:
> 
>   memcpy((void*)0x20000000, "debug_want_extra_isize=0x80,lazytime," 38);
> 
> Having a more minimized reproducer would improve the reliability of
> the bisect, as well as making it easier for the developer to figure
> out the true root cause of the problem.

Amen to all of that. for so many good reasons.
Thanks.

-- 
~Randy

Re: [syzbot] WARNING in do_mkdirat

[Aleksandr Nogikh]

Hi Eric, Ted,

On Sat, Dec 31, 2022 at 5:58 PM Theodore Ts'o <tytso@mit.edu> wrote:
>
> On Thu, Dec 29, 2022 at 01:17:31PM -0800, Eric Biggers wrote:
> > Thanks Aleksandr.  From what I can see, the fix is working for new filesystem
> > bugs: the filesystem(s) involved get added to the title and the recipients.
> >
> > One question: what happens to all the open bugs, like this one ("WARNING in
> > do_mkdirat") that were reported before the syzbot fix?  Are they going to be
> > re-reported correctly?  Perhaps any bug whose reproducer includes
> > "syz_mount_image" and was reported before the date of this fix should be
> > invalidated more aggressively than usual, so that it can be re-reported?

I fear that the community will not be super excited to see those tons
of fs bug reports again :)

Soon it'll become possible to see the subsystems on the dashboard and
to filter bugs based on them, hopefully this will help those bugs not
get completely lost.

>
> As a related request/wish, it would be nice if those dashboard pages
> that were created before the new-style reporting which includes the
> file system image, strace otuput, etc., could get regenerated.  For example:
>
> https://syzkaller.appspot.com/bug?id=be6e90ce70987950e6deb3bac8418344ca8b96cd

I've deployed the change -- now it should display all the download
links on the bug info page. If we have reported a download link /
strace log in an email, it should be there. For yet older bugs, it's
trickier. We regenerate reproducers once in 100 days, so if the old
bugs keep on happening, the information will appear there over time as
well.

>
> Even if someone has already submitted a proposed fix, I often like to
> double-check that the fix is really fixing the true root cause of the
> problem, as opposed to just making a superficial change that blocks
> the current syzbot reproducer, but which will eventually be tripped
> again because code is still vulnerable.  (For example, we might block
> a straightforward reproducer by adding a check at mount time, but if
> the superblocks get corrupted during the journal replay, we'd still be
> vulnerable.)  And having access to the corrupted file system image,
> and other associated reporting data, is often super-helpful in that
> regard.

Thank you very much for the feedback below!

>
> Also, can we at some point have the C reproducer actually using proper
> C strings instead of hex digits?  It will make the reproducer much
> more human understandable, as well making it easier to edit the string
> when the developer is trying to do a better job minimizing the test
> case than syzbot.  For example:
>
>   memcpy(
>       (void*)0x20000000,
>       "\x6e\x6f\x75\x73\x65\x72\x5f\x78\x61\x74\x74\x72\x2c\x61\x63\x6c\x2c\x64"
>       "\x65\x62\x75\x67\x5f\x77\x61\x6e\x74\x5f\x65\x78\x74\x72\x61\x5f\x69\x73"
>       "\x69\x7a\x65\x3d\x30\x78\x30\x30\x30\x30\x30\x30\x30\x30\x30\x30\x30\x30"
>       "\x30\x30\x38\x30\x2c\x6c\x61\x7a\x79\x74\x69\x6d\x65\x2c\x6e\x6f\x62\x68"
>       "\x2c\x71\x75\x6f\x74\x61\x2c\x00\x3d\x93\x09\x61\x36\x5d\x73\x58\x9c",
>       89);
>
> Would be *much* more understable if it were:
>
>   memcpy(
>       (void*)0x20000000,
>       "nouser_xattr,acl,debug_want_extra_isize=0x0000000000000080,lazytime,nobh,quota,",
>       80);

I've filed an issue to keep track on the progress:
https://github.com/google/syzkaller/issues/3605

>
> Of course, something like:
>
>    char mount_options[] = "nouser_xattr,acl,debug_want_extra_isize=0x0000000000000080,lazytime,nobh,quota,";
>
> Would be even better (and more portable) than using random hex
> addresses, but just simply using ASCII strings would be a good first
> step.
>
> Of course, filling in C structures instead of just a random memcpy of
> hex garbage would be even *more* awesome, bunt I'll take what I can
> get.  :-)
>
> Another opportunity for improvement is to try minimizing mount
> options, so it becomes more obvious which ones are required.  For
> example, in the above example, a minimized mount option string would
> have been:
>
>   memcpy((void*)0x20000000, "debug_want_extra_isize=0x80,lazytime," 38);
>
> Having a more minimized reproducer would improve the reliability of
> the bisect, as well as making it easier for the developer to figure
> out the true root cause of the problem.

I've filed an issue: https://github.com/google/syzkaller/issues/3606

--
Best Regards,
Aleksandr

>
> Cheers,
>
>                                         - Ted
>

Re: [syzbot] WARNING in do_mkdirat

[Hillf Danton]

On 12 Dec 2022 13:58:51 -0500 Theodore Ts'o <tytso@mit.edu>
> On Mon, Dec 12, 2022 at 11:29:11AM +0800, Hillf Danton wrote:
> > > You've completely misunderstood Al's point.  He's not whining about
> > > being cc'd, he's pointing at this is ONLY USEFUL IF THE NTFS3
> > > MAINTAINERS ARE CC'd.  And they're not.  So this is just noise.
> > > And enough noise means that signal is lost.
> > 
> > Call Trace:
> >  <TASK>
> >  inode_unlock include/linux/fs.h:761 [inline]
> >  done_path_create fs/namei.c:3857 [inline]
> >  do_mkdirat+0x2de/0x550 fs/namei.c:4064
> >  __do_sys_mkdirat fs/namei.c:4076 [inline]
> >  __se_sys_mkdirat fs/namei.c:4074 [inline]
> >  __x64_sys_mkdirat+0x85/0x90 fs/namei.c:4074
> >  do_syscall_x64 arch/x86/entry/common.c:50 [inline]
> >  do_syscall_64+0x3d/0xb0 arch/x86/entry/common.c:80
> >  entry_SYSCALL_64_after_hwframe+0x63/0xcd
> > 
> > Given the call trace above, how do you know the ntfs3 guys should be also
> > Cced in addition to AV? What if it would take more than three months for
> > syzbot to learn the skills in your mind? What is preventing you routing
> > the report to ntfs3?
> 
> If it takes 3 months for syzbot to take a look at the source code in
> their own #!@?! reproducer, or just to take a look at the strace link
> in the dashboard:
> 
> [pid  3639] mount("/dev/loop0", "./file2", "ntfs3", MS_NOSUID|MS_NOEXEC|MS_DIRSYNC|MS_I_VERSION, "") = 0
> 
> There's something really wrong.  The point Al has been making (and
> I've been making for multiple years) is that Syzbot has the
> information, but unfortunately, at the moment, it is only analyzing
> the the stack trace, and it is not doing things that really could be
> done automatically --- and cloud VM time is cheap, and upstream

Another case of easy talk instead of patching hard?
What is preventing you from making the syzbot more automatic?
Is it sane to teach the bot to patch ext4 because you are a maintainer?

> maintainer time is expensive.  So by not improving syzbot in a way
> that really shouldn't be all that difficult, the syzbot maintainers is
> disrespectiving the time of the upstream maintainers.

Are upstream maintainers prefering to spend weeks and months creating
ext4 beatles for example over taking a couple of hours scanning the
syzbot reports like this one? Why does the bot kick you if you have a
clean butt? Why are usb people irrelevant in this case?
> 
> So sure, we could ask Linus to triage all syzbot reports --- or we
> could ask Al to triage all syzbot file system reports --- but that is
> not a good use of upstream resources.

Lolll who is preventing you from doing so? Who care/complain if you do?
Why not directly fix today what was reported instead and issue a PR?

Re: [syzbot] WARNING in do_mkdirat

[Al Viro]

On Tue, Dec 13, 2022 at 09:47:16AM +0800, Hillf Danton wrote:

> > maintainer time is expensive.  So by not improving syzbot in a way
> > that really shouldn't be all that difficult, the syzbot maintainers is
> > disrespectiving the time of the upstream maintainers.
> 
> Are upstream maintainers prefering to spend weeks and months creating
> ext4 beatles for example over taking a couple of hours scanning the
> syzbot reports like this one? Why does the bot kick you if you have a
> clean butt? Why are usb people irrelevant in this case?

And to continue with the rethorical questions: has anyone bothered to
inform you that you are an obnoxious cunt?

*plonk*

Re: [syzbot] WARNING in do_mkdirat

[Hillf Danton]

On 13 Dec 2022 03:36:23 +0000 Al Viro <viro@zeniv.linux.org.uk>
> On Tue, Dec 13, 2022 at 09:47:16AM +0800, Hillf Danton wrote:
> 
> > > maintainer time is expensive.  So by not improving syzbot in a way
> > > that really shouldn't be all that difficult, the syzbot maintainers is
> > > disrespectiving the time of the upstream maintainers.
> > 
> > Are upstream maintainers prefering to spend weeks and months creating
> > ext4 beatles for example over taking a couple of hours scanning the
> > syzbot reports like this one? Why does the bot kick you if you have a
> > clean butt? Why are usb people irrelevant in this case?
> 
> And to continue with the rethorical questions: has anyone bothered to
> inform you that you are an obnoxious cunt?

When did cunt go in to your mind? And How?
> 
> *plonk*

Re: [syzbot] WARNING in do_mkdirat

[Alexander Potapenko]

On Tue, Dec 13, 2022 at 5:12 AM Hillf Danton <hdanton@sina.com> wrote:
>
> On 13 Dec 2022 03:36:23 +0000 Al Viro <viro@zeniv.linux.org.uk>
> > On Tue, Dec 13, 2022 at 09:47:16AM +0800, Hillf Danton wrote:
> >
> > > > maintainer time is expensive.  So by not improving syzbot in a way
> > > > that really shouldn't be all that difficult, the syzbot maintainers is
> > > > disrespectiving the time of the upstream maintainers.
> > >
> > > Are upstream maintainers prefering to spend weeks and months creating
> > > ext4 beatles for example over taking a couple of hours scanning the
> > > syzbot reports like this one? Why does the bot kick you if you have a
> > > clean butt? Why are usb people irrelevant in this case?
> >
> > And to continue with the rethorical questions: has anyone bothered to
> > inform you that you are an obnoxious cunt?
>
> When did cunt go in to your mind? And How?
> >
> > *plonk*


I think it's time to take a step back now and try to keep the
discussion constructive.
It's really about bots and automation, no need to engage in personalities.

I myself am not involved in improving syzbot, but as a bystander let
me assure you that the folks do take the feedback seriously and
actually have interest in improving the experience for the upstream
maintainers.



> --
> You received this message because you are subscribed to the Google Groups "syzkaller-bugs" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to syzkaller-bugs+unsubscribe@googlegroups.com.
> To view this discussion on the web visit https://groups.google.com/d/msgid/syzkaller-bugs/20221213041225.3461-1-hdanton%40sina.com.



-- 
Alexander Potapenko
Software Engineer

Google Germany GmbH
Erika-Mann-Straße, 33
80636 München

Geschäftsführer: Paul Manicle, Liana Sebastian
Registergericht und -nummer: Hamburg, HRB 86891
Sitz der Gesellschaft: Hamburg

Re: [syzbot] WARNING in do_mkdirat

[Hillf Danton]

On Sat, 03 Dec 2022 06:52:45 -0800
> syzbot found the following issue on:
> 
> HEAD commit:    ca57f02295f1 afs: Fix fileserver probe RTT handling
> git tree:       upstream
> console output: https://syzkaller.appspot.com/x/log.txt?x=171b06a1880000
> kernel config:  https://syzkaller.appspot.com/x/.config?x=2325e409a9a893e1
> dashboard link: https://syzkaller.appspot.com/bug?extid=919c5a9be8433b8bf201
> compiler:       Debian clang version 13.0.1-++20220126092033+75e33f71c2da-1~exp1~20220126212112.63, GNU ld (GNU Binutils for Debian) 2.35.2
> 
> Unfortunately, I don't have any reproducer for this issue yet.
> 
> Downloadable assets:
> disk image: https://storage.googleapis.com/syzbot-assets/af66f1d3a389/disk-ca57f022.raw.xz
> vmlinux: https://storage.googleapis.com/syzbot-assets/c0c7ec393108/vmlinux-ca57f022.xz
> kernel image: https://storage.googleapis.com/syzbot-assets/ea8871940eaa/bzImage-ca57f022.xz
> 
> IMPORTANT: if you fix the issue, please add the following tag to the commit:
> Reported-by: syzbot+919c5a9be8433b8bf201@syzkaller.appspotmail.com
> 
> WARNING: CPU: 0 PID: 12206 at kernel/locking/rwsem.c:1361 __up_write kernel/locking/rwsem.c:1360 [inline]
> WARNING: CPU: 0 PID: 12206 at kernel/locking/rwsem.c:1361 up_write+0x4f9/0x580 kernel/locking/rwsem.c:1615
> Modules linked in:
> CPU: 0 PID: 12206 Comm: syz-executor.0 Not tainted 6.1.0-rc7-syzkaller-00012-gca57f02295f1 #0
> Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/26/2022
> RIP: 0010:__up_write kernel/locking/rwsem.c:1360 [inline]
> RIP: 0010:up_write+0x4f9/0x580 kernel/locking/rwsem.c:1615
> Code: c7 40 a3 ed 8a 48 c7 c6 e0 a5 ed 8a 48 8b 54 24 28 48 8b 4c 24 18 4d 89 e0 4c 8b 4c 24 30 31 c0 53 e8 1b 83 e8 ff 48 83 c4 08 <0f> 0b e9 6b fd ff ff 48 c7 c1 18 25 76 8e 80 e1 07 80 c1 03 38 c1
> RSP: 0018:ffffc9000338fd40 EFLAGS: 00010292
> RAX: 69eb1955c47aff00 RBX: ffffffff8aeda420 RCX: 0000000000040000
> RDX: ffffc900046f6000 RSI: 0000000000023311 RDI: 0000000000023312
> RBP: ffffc9000338fe10 R08: ffffffff816e560d R09: fffff52000671f61
> R10: fffff52000671f61 R11: 1ffff92000671f60 R12: 0000000000000000
> R13: ffff888027d20a90 R14: 1ffff92000671fb0 R15: dffffc0000000000
> FS:  00007f928b35b700(0000) GS:ffff8880b9800000(0000) knlGS:0000000000000000
> CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
> CR2: 00007f268cbad988 CR3: 000000001e90f000 CR4: 00000000003506f0
> DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
> DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
> Call Trace:
>  <TASK>
>  inode_unlock include/linux/fs.h:761 [inline]
>  done_path_create fs/namei.c:3857 [inline]
>  do_mkdirat+0x2de/0x550 fs/namei.c:4064
>  __do_sys_mkdir fs/namei.c:4081 [inline]
>  __se_sys_mkdir fs/namei.c:4079 [inline]
>  __x64_sys_mkdir+0x6a/0x80 fs/namei.c:4079
>  do_syscall_x64 arch/x86/entry/common.c:50 [inline]
>  do_syscall_64+0x3d/0xb0 arch/x86/entry/common.c:80
>  entry_SYSCALL_64_after_hwframe+0x63/0xcd
> RIP: 0033:0x7f928a68c0d9
> Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 f1 19 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
> RSP: 002b:00007f928b35b168 EFLAGS: 00000246 ORIG_RAX: 0000000000000053
> RAX: ffffffffffffffda RBX: 00007f928a7ac050 RCX: 00007f928a68c0d9
> RDX: 0000000000000000 RSI: 0000000000000000 RDI: 0000000020000400
> RBP: 00007f928a6e7ae9 R08: 0000000000000000 R09: 0000000000000000
> R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
> R13: 00007fffeaab152f R14: 00007f928b35b300 R15: 0000000000022000
>  </TASK>

Add debug info to inode_unlock() to see if this is not a bogus report.

Hillf

--- a/include/linux/fs.h
+++ b/include/linux/fs.h
@@ -758,6 +758,7 @@ static inline void inode_lock(struct ino
 
 static inline void inode_unlock(struct inode *inode)
 {
+	lockdep_assert_held_write(&inode->i_rwsem);
 	up_write(&inode->i_rwsem);
 }
 
--

Re: [syzbot] WARNING in do_mkdirat

[syzbot]

syzbot has found a reproducer for the following issue on:

HEAD commit:    0d1409e4ff08 Merge tag 'drm-fixes-2022-12-09' of git://ano..
git tree:       upstream
console output: https://syzkaller.appspot.com/x/log.txt?x=172b960b880000
kernel config:  https://syzkaller.appspot.com/x/.config?x=d58e7fe7f9cf5e24
dashboard link: https://syzkaller.appspot.com/bug?extid=919c5a9be8433b8bf201
compiler:       Debian clang version 13.0.1-++20220126092033+75e33f71c2da-1~exp1~20220126212112.63, GNU ld (GNU Binutils for Debian) 2.35.2
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=145fde33880000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/9ab0143f95cb/disk-0d1409e4.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/e574d5eaa32f/vmlinux-0d1409e4.xz
kernel image: https://storage.googleapis.com/syzbot-assets/31109436b00b/bzImage-0d1409e4.xz
mounted in repro: https://storage.googleapis.com/syzbot-assets/5cec1c83630e/mount_0.gz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+919c5a9be8433b8bf201@syzkaller.appspotmail.com

WARNING: CPU: 1 PID: 4982 at kernel/locking/rwsem.c:1361 __up_write kernel/locking/rwsem.c:1360 [inline]
WARNING: CPU: 1 PID: 4982 at kernel/locking/rwsem.c:1361 up_write+0x4f9/0x580 kernel/locking/rwsem.c:1615
Modules linked in:
CPU: 1 PID: 4982 Comm: syz-executor.4 Not tainted 6.1.0-rc8-syzkaller-00148-g0d1409e4ff08 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/26/2022
RIP: 0010:__up_write kernel/locking/rwsem.c:1360 [inline]
RIP: 0010:up_write+0x4f9/0x580 kernel/locking/rwsem.c:1615
Code: c7 c0 a3 ed 8a 48 c7 c6 60 a6 ed 8a 48 8b 54 24 28 48 8b 4c 24 18 4d 89 e0 4c 8b 4c 24 30 31 c0 53 e8 ab 7c e8 ff 48 83 c4 08 <0f> 0b e9 6b fd ff ff 48 c7 c1 18 2a 76 8e 80 e1 07 80 c1 03 38 c1
RSP: 0018:ffffc9000564fd40 EFLAGS: 00010292
RAX: 9a2b61996b411800 RBX: ffffffff8aeda4a0 RCX: ffff88801f5d3a80
RDX: 0000000000000000 RSI: 0000000080000000 RDI: 0000000000000000
RBP: ffffc9000564fe10 R08: ffffffff816e5c7d R09: fffff52000ac9f61
R10: fffff52000ac9f61 R11: 1ffff92000ac9f60 R12: 0000000000000000
R13: ffff888069880a90 R14: 1ffff92000ac9fb0 R15: dffffc0000000000
FS:  00007f71b6d62700(0000) GS:ffff8880b9900000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 00007f825d9ff000 CR3: 000000006cef1000 CR4: 00000000003506e0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <TASK>
 inode_unlock include/linux/fs.h:761 [inline]
 done_path_create fs/namei.c:3857 [inline]
 do_mkdirat+0x2de/0x550 fs/namei.c:4064
 __do_sys_mkdirat fs/namei.c:4076 [inline]
 __se_sys_mkdirat fs/namei.c:4074 [inline]
 __x64_sys_mkdirat+0x85/0x90 fs/namei.c:4074
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x3d/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7f71b608c0d9
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 f1 19 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f71b6d62168 EFLAGS: 00000246 ORIG_RAX: 0000000000000102
RAX: ffffffffffffffda RBX: 00007f71b61ac050 RCX: 00007f71b608c0d9
RDX: 0000000000000000 RSI: 0000000020000280 RDI: 0000000000000004
RBP: 00007f71b60e7ae9 R08: 0000000000000000 R09: 0000000000000000
R10: 0000000000000000 R11: 0000000000000246 R12: 0000000000000000
R13: 00007ffeac9136ff R14: 00007f71b6d62300 R15: 0000000000022000
 </TASK>

Re: [syzbot] WARNING in do_mkdirat

[Matthew Wilcox]

On Fri, Dec 09, 2022 at 11:50:41AM -0800, syzbot wrote:
> syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=145fde33880000

I see that ntfs3 is involved.  It's probably the known issue where ntfs3
corrupts the lockdep database.

Re: [syzbot] WARNING in do_mkdirat

[syzbot]

syzbot has found a reproducer for the following issue on:

HEAD commit:    3ecc37918c80 Merge tag 'media/v6.1-4' of git://git.kernel...
git tree:       upstream
console+strace: https://syzkaller.appspot.com/x/log.txt?x=13ae071d880000
kernel config:  https://syzkaller.appspot.com/x/.config?x=d58e7fe7f9cf5e24
dashboard link: https://syzkaller.appspot.com/bug?extid=919c5a9be8433b8bf201
compiler:       Debian clang version 13.0.1-++20220126092033+75e33f71c2da-1~exp1~20220126212112.63, GNU ld (GNU Binutils for Debian) 2.35.2
syz repro:      https://syzkaller.appspot.com/x/repro.syz?x=10aaf2b7880000
C reproducer:   https://syzkaller.appspot.com/x/repro.c?x=17b652b7880000

Downloadable assets:
disk image: https://storage.googleapis.com/syzbot-assets/be14794fd26b/disk-3ecc3791.raw.xz
vmlinux: https://storage.googleapis.com/syzbot-assets/35b850996388/vmlinux-3ecc3791.xz
kernel image: https://storage.googleapis.com/syzbot-assets/0eec0f8f6777/bzImage-3ecc3791.xz
mounted in repro: https://storage.googleapis.com/syzbot-assets/547e98eae9c0/mount_0.gz

IMPORTANT: if you fix the issue, please add the following tag to the commit:
Reported-by: syzbot+919c5a9be8433b8bf201@syzkaller.appspotmail.com

------------[ cut here ]------------
DEBUG_RWSEMS_WARN_ON((rwsem_owner(sem) != current) && !rwsem_test_oflags(sem, RWSEM_NONSPINNABLE)): count = 0x0, magic = 0xffff888072216a70, owner = 0x0, curr 0xffff888078ce57c0, list empty
WARNING: CPU: 0 PID: 4093 at kernel/locking/rwsem.c:1361 __up_write kernel/locking/rwsem.c:1360 [inline]
WARNING: CPU: 0 PID: 4093 at kernel/locking/rwsem.c:1361 up_write+0x4f9/0x580 kernel/locking/rwsem.c:1615
Modules linked in:
CPU: 0 PID: 4093 Comm: syz-executor196 Not tainted 6.1.0-rc8-syzkaller-00152-g3ecc37918c80 #0
Hardware name: Google Google Compute Engine/Google Compute Engine, BIOS Google 10/26/2022
RIP: 0010:__up_write kernel/locking/rwsem.c:1360 [inline]
RIP: 0010:up_write+0x4f9/0x580 kernel/locking/rwsem.c:1615
Code: c7 c0 a3 ed 8a 48 c7 c6 60 a6 ed 8a 48 8b 54 24 28 48 8b 4c 24 18 4d 89 e0 4c 8b 4c 24 30 31 c0 53 e8 ab 7c e8 ff 48 83 c4 08 <0f> 0b e9 6b fd ff ff 48 c7 c1 18 2a 76 8e 80 e1 07 80 c1 03 38 c1
RSP: 0018:ffffc900043ffd40 EFLAGS: 00010292
RAX: 7c48dcb6c422ab00 RBX: ffffffff8aeda4a0 RCX: ffff888078ce57c0
RDX: 0000000000000000 RSI: 0000000080000000 RDI: 0000000000000000
RBP: ffffc900043ffe10 R08: ffffffff816e5c7d R09: fffff5200087ff21
R10: fffff5200087ff21 R11: 1ffff9200087ff20 R12: 0000000000000000
R13: ffff888072216a70 R14: 1ffff9200087ffb0 R15: dffffc0000000000
FS:  00007f68743c0700(0000) GS:ffff8880b9800000(0000) knlGS:0000000000000000
CS:  0010 DS: 0000 ES: 0000 CR0: 0000000080050033
CR2: 0000000000000000 CR3: 0000000026c1b000 CR4: 00000000003506f0
DR0: 0000000000000000 DR1: 0000000000000000 DR2: 0000000000000000
DR3: 0000000000000000 DR6: 00000000fffe0ff0 DR7: 0000000000000400
Call Trace:
 <TASK>
 inode_unlock include/linux/fs.h:761 [inline]
 done_path_create fs/namei.c:3857 [inline]
 do_mkdirat+0x2de/0x550 fs/namei.c:4064
 __do_sys_mkdirat fs/namei.c:4076 [inline]
 __se_sys_mkdirat fs/namei.c:4074 [inline]
 __x64_sys_mkdirat+0x85/0x90 fs/namei.c:4074
 do_syscall_x64 arch/x86/entry/common.c:50 [inline]
 do_syscall_64+0x3d/0xb0 arch/x86/entry/common.c:80
 entry_SYSCALL_64_after_hwframe+0x63/0xcd
RIP: 0033:0x7f687c635589
Code: 28 00 00 00 75 05 48 83 c4 28 c3 e8 71 15 00 00 90 48 89 f8 48 89 f7 48 89 d6 48 89 ca 4d 89 c2 4d 89 c8 4c 8b 4c 24 08 0f 05 <48> 3d 01 f0 ff ff 73 01 c3 48 c7 c1 b8 ff ff ff f7 d8 64 89 01 48
RSP: 002b:00007f68743c02f8 EFLAGS: 00000246 ORIG_RAX: 0000000000000102
RAX: ffffffffffffffda RBX: 00007f687c6d97b0 RCX: 00007f687c635589
RDX: 0000000000000000 RSI: 0000000020000200 RDI: 0000000000000004
RBP: 00007f687c6d97bc R08: 00007f68743c0700 R09: 0000000000000000
R10: 00007f68743c0700 R11: 0000000000000246 R12: 00007f687c6a6258
R13: 0032656c69662f2e R14: 69662f7375622f2e R15: 00007f687c6d97b8
 </TASK>