[Ksummit-discuss] [TECH TOPIC] giving freezer well-defined semantics

[Ksummit-discuss] [TECH TOPIC] giving freezer well-defined semantics

[Jiri Kosina]

Currently, the freezer has rather random semantics and there is no 
rigorous definition that would provide clear guidance which kernel threads 
should be freezable (and what rules they have to follow if they are).

The long and complicated history of freezer resulted in the current state, 
where many kernel threads are marked freezable "just because", with 
freezing points randomly sprinkled all over the place, and it really 
complicates any efforts in cleaning up kthreads (which is why I am 
interested in it -- we'd love to have kthreads cleaned up to make live 
patching easier).

There are various possible aproaches to this. One is basically ad-hoc, 
going over all the existing kthreads one by one, and fixing (probably just 
removing, in most cases) its usage of freezer. The obivous disadvantage is 
that this doesn't prevent anyone to abusing it again the same way in the 
future.

Tejun came up with a different aproach at [1] -- basically getting rid of 
the freezer completely, and rather annotating those I/O requests which are 
needed for writing the hibernation image out, so that they make it through 
all the affected subsystems, while other I/O requests would be frozen.

This would be rather dramatic change both in a way how kthreads work, how 
hibernation works, and it'd be necessary to have means to mark I/O 
requests as "needed for hibernation", therefore I think this would be a 
good cross-subsystem topic.

[1] http://lkml.kernel.org/r/20150613232222.GB346@mtj.duckdns.org

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [TECH TOPIC] giving freezer well-defined semantics

[Rafael J. Wysocki]

On Tuesday, July 07, 2015 10:42:36 PM Jiri Kosina wrote:
> Currently, the freezer has rather random semantics and there is no 
> rigorous definition that would provide clear guidance which kernel threads 
> should be freezable (and what rules they have to follow if they are).

That's with respect to kernel threads, right?

The primary purpose of the freezer is to get user space out of the way
and at least in principle there should be no fundamental need to freeze
any kernel threads.

> The long and complicated history of freezer resulted in the current state, 
> where many kernel threads are marked freezable "just because", with 
> freezing points randomly sprinkled all over the place, and it really 
> complicates any efforts in cleaning up kthreads (which is why I am 
> interested in it -- we'd love to have kthreads cleaned up to make live 
> patching easier).
> 
> There are various possible aproaches to this. One is basically ad-hoc, 
> going over all the existing kthreads one by one, and fixing (probably just 
> removing, in most cases) its usage of freezer. The obivous disadvantage is 
> that this doesn't prevent anyone to abusing it again the same way in the 
> future.
> 
> Tejun came up with a different aproach at [1] -- basically getting rid of 
> the freezer completely, and rather annotating those I/O requests which are 
> needed for writing the hibernation image out, so that they make it through 
> all the affected subsystems, while other I/O requests would be frozen.
> 
> This would be rather dramatic change both in a way how kthreads work, how 
> hibernation works, and it'd be necessary to have means to mark I/O 
> requests as "needed for hibernation", therefore I think this would be a 
> good cross-subsystem topic.
> 
> [1] http://lkml.kernel.org/r/20150613232222.GB346@mtj.duckdns.org

I'm not reading this as a proposal to drop the freezer completely, but
rather to stop using it for kernel threads (I may not understand the
full context correctly, though).

There are two basic resons to freeze user space before suspend in my view.
First, we don't want it to interfere with device driver's suspend/resume
callbacks in any way.  Second, we want to prevent silly stuff (like
active polling in a loop etc.) from running.

Both are valid as far as I can say.

Kernel threads are a totally different matter, however.

Thanks,
Rafael

Re: [Ksummit-discuss] [TECH TOPIC] giving freezer well-defined semantics

[Jiri Kosina]

On Tue, 7 Jul 2015, Rafael J. Wysocki wrote:

> > Currently, the freezer has rather random semantics and there is no 
> > rigorous definition that would provide clear guidance which kernel threads 
> > should be freezable (and what rules they have to follow if they are).
> 
> That's with respect to kernel threads, right?

Yeah, sorry for not being verbose enough. I have been buried in kthread 
interactions with the rest of the kernel for too long, so my world view is 
now a bit distorted.

Yes, this is purely related to try_to_freeze() in !PF_NOFREEZE kernel 
threads.

> The primary purpose of the freezer is to get user space out of the way
> and at least in principle there should be no fundamental need to freeze
> any kernel threads.

Except those which are needed for hibernation image I/O. Which is 
difficult to pre-categorize in a rigorous way.

You need some parts of VM for that. You need the underlying block 
device. You name it.

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [TECH TOPIC] giving freezer well-defined semantics

[Rafael J. Wysocki]

On Tuesday, July 07, 2015 11:13:01 PM Jiri Kosina wrote:
> On Tue, 7 Jul 2015, Rafael J. Wysocki wrote:
> 
> > > Currently, the freezer has rather random semantics and there is no 
> > > rigorous definition that would provide clear guidance which kernel threads 
> > > should be freezable (and what rules they have to follow if they are).
> > 
> > That's with respect to kernel threads, right?
> 
> Yeah, sorry for not being verbose enough. I have been buried in kthread 
> interactions with the rest of the kernel for too long, so my world view is 
> now a bit distorted.
> 
> Yes, this is purely related to try_to_freeze() in !PF_NOFREEZE kernel 
> threads.
> 
> > The primary purpose of the freezer is to get user space out of the way
> > and at least in principle there should be no fundamental need to freeze
> > any kernel threads.
> 
> Except those which are needed for hibernation image I/O.

I don't quite see why there's a need to freeze those particular ones.

OK, it is necessary to ensure that the contents of the image will be consistent
with the state of filesystems on the storage media, so everything that may
change that state should be "frozen" before the image is created, but "frozen"
in terms of "no persistent state changes from now on" rather than in terms of
"no forward progress from now on".

Thanks,
Rafael

Re: [Ksummit-discuss] [TECH TOPIC] giving freezer well-defined semantics

[Jiri Kosina]

On Wed, 8 Jul 2015, Rafael J. Wysocki wrote:

> OK, it is necessary to ensure that the contents of the image will be 
> consistent with the state of filesystems on the storage media, so 
> everything that may change that state should be "frozen" before the 
> image is created, but "frozen" in terms of "no persistent state changes 
> from now on" rather than in terms of "no forward progress from now on".

Yeah. So again, why do we even have freezer for so many kernel threads at 
all? :)

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [TECH TOPIC] giving freezer well-defined semantics

[Rafael J. Wysocki]

On Wednesday, July 08, 2015 10:16:39 AM Jiri Kosina wrote:
> On Wed, 8 Jul 2015, Rafael J. Wysocki wrote:
> 
> > OK, it is necessary to ensure that the contents of the image will be 
> > consistent with the state of filesystems on the storage media, so 
> > everything that may change that state should be "frozen" before the 
> > image is created, but "frozen" in terms of "no persistent state changes 
> > from now on" rather than in terms of "no forward progress from now on".
> 
> Yeah. So again, why do we even have freezer for so many kernel threads at 
> all? :)

Well, one reason may be that we've never grown a decent mechanism for freezing
filesystems (as in "no persistent state changes from now on") and people try to
make up for that by stopping things if they can (but in the kernel space that's
inherently racy).

There also are places where it has been forever and no one is sure what happens
after it's been dropped from there.

And there are places where it is used to provide specific guaranees (like in
the runtime PM framework), but might be replaced with a different mechanism.

Essentially, as I said above, the freezer is only really useful for things
requiring the "no forward progress from now on" rule.  There are things like
that in the kernel too, but I bet there are not too many of them.

Thanks,
Rafael

Re: [Ksummit-discuss] [TECH TOPIC] giving freezer well-defined semantics

[Jan Kara]

On Wed 08-07-15 23:55:09, Rafael J. Wysocki wrote:
> On Wednesday, July 08, 2015 10:16:39 AM Jiri Kosina wrote:
> > On Wed, 8 Jul 2015, Rafael J. Wysocki wrote:
> > 
> > > OK, it is necessary to ensure that the contents of the image will be 
> > > consistent with the state of filesystems on the storage media, so 
> > > everything that may change that state should be "frozen" before the 
> > > image is created, but "frozen" in terms of "no persistent state changes 
> > > from now on" rather than in terms of "no forward progress from now on".
> > 
> > Yeah. So again, why do we even have freezer for so many kernel threads at 
> > all? :)
> 
> Well, one reason may be that we've never grown a decent mechanism for freezing
> filesystems (as in "no persistent state changes from now on") and people try to
> make up for that by stopping things if they can (but in the kernel space that's
> inherently racy).

The most common filesystems - xfs, ext4, ext3, btrfs - handle freezing fine
these days. And the filesystem freezing is used by LVM snapshots,
Virtualization guest snapshots etc. so it is even tested ;).

I know you have proposed to use fs freezing during hibernation some time
ago but I don't remember where it ended up... Do you remember?

								Honza
-- 
Jan Kara <jack@suse.cz>
SUSE Labs, CR

Re: [Ksummit-discuss] [TECH TOPIC] giving freezer well-defined semantics

[Rafael J. Wysocki]

On Thursday, July 09, 2015 01:25:10 PM Jan Kara wrote:
> On Wed 08-07-15 23:55:09, Rafael J. Wysocki wrote:
> > On Wednesday, July 08, 2015 10:16:39 AM Jiri Kosina wrote:
> > > On Wed, 8 Jul 2015, Rafael J. Wysocki wrote:
> > > 
> > > > OK, it is necessary to ensure that the contents of the image will be 
> > > > consistent with the state of filesystems on the storage media, so 
> > > > everything that may change that state should be "frozen" before the 
> > > > image is created, but "frozen" in terms of "no persistent state changes 
> > > > from now on" rather than in terms of "no forward progress from now on".
> > > 
> > > Yeah. So again, why do we even have freezer for so many kernel threads at 
> > > all? :)
> > 
> > Well, one reason may be that we've never grown a decent mechanism for freezing
> > filesystems (as in "no persistent state changes from now on") and people try to
> > make up for that by stopping things if they can (but in the kernel space that's
> > inherently racy).
> 
> The most common filesystems - xfs, ext4, ext3, btrfs - handle freezing fine
> these days. And the filesystem freezing is used by LVM snapshots,
> Virtualization guest snapshots etc. so it is even tested ;).
> 
> I know you have proposed to use fs freezing during hibernation some time
> ago but I don't remember where it ended up... Do you remember?

Pretty much nowhere.

It would be good to revive that patch if the functionality is reliable, though.

Thanks,
Rafael

Re: [Ksummit-discuss] [TECH TOPIC] giving freezer well-defined semantics

[Benjamin Herrenschmidt]

On Tue, 2015-07-07 at 22:42 +0200, Jiri Kosina wrote:
> Tejun came up with a different aproach at [1] -- basically getting rid of 
> the freezer completely, and rather annotating those I/O requests which are 
> needed for writing the hibernation image out, so that they make it through 
> all the affected subsystems, while other I/O requests would be frozen.
> 
> This would be rather dramatic change both in a way how kthreads work, how 
> hibernation works, and it'd be necessary to have means to mark I/O 
> requests as "needed for hibernation", therefore I think this would be a 
> good cross-subsystem topic.

I like that approach. In the early days, when we put together what
eventually became our power management, I was against the freezer to
begin with :-) I always had the feeling that it was an incomplete band
aid which was mostly trying to hide the problem rather than solve it.

In part because just stopping threads isn't a guarantee that things
stop, "driver" requests can be issued in theory from anything such as
timers, external interrupts, etc... threads are only part of the
problem. It's also non obvious whether a given thread is needed to write
the suspend image or not. A single thread can have multiple behaviours
associated with it. It can monitor link health and be necessary for
keeping things working but can also deal with hotplug which might not be
desirable.

So all in all, I agree with Tejun.

Cheers,
Ben.