[Ksummit-discuss] [CORE TOPIC] live patching

[Ksummit-discuss] [CORE TOPIC] live patching

[Jiri Kosina]

[ I was wondering whether to tag this "CORE TOPIC" or "TECH TOPIC", as I 
  really see it on the bordeline; let's see ]

As people are probably aware, the basic infrastructure for live patching 
of the kernel has been merged and there are still quite some loose ends to 
tie up.

On top of original agreement by the two teams working at live kernel 
patching at Red Hat and SUSE, there has been quite some feedback raised 
during upstream review by people not being directly involved in the 
development itself [1].

Tangentially related topics have been spawned in the discussion, namely:

- stack unwinding has so far been deemed purely debugging facility, with 
  no correctness guarantees whatsoever. One of the possible aproaches to
  improving the coverage of live patching infrastructure would assume
  100% reliability of in-kernel stack unwinding.

  Is this something that should be pursued further? Especially opinions 
  and comments from various arch maintainers would be welcome

- it would be beneficial for live patching if kernel threads could be 
  cleaned up; currently it's a mess, with freezer points, signal handling 
  etc. spewed randomly all over the main loop. Generalizing the API (such
  as improving kthread_worker and migrating kthreads to use that) would 
  be beneficial if there are no major objections. See my related TECH 
  TOPIC submission

- Ingo proposed a "checkpoint"-based way for live patching, i.e. forcing 
  all the processess out of kernel to serliaze on a well-defined execution
  point, perform the patching, and resume the execution.

  I see several issues with this aproach, but Ingo seems to be really 
  insisting on this to be the way to go.

I could give a short introduction to how the current in-kernel live 
patching facility looks like and what are the things that are currently 
missing (either because they were considered too controversial by some 
(such as the stack unwinding reliability or lazy-migration) or because 
they just have not been implemented yet) and hopefully create a basis for 
a discussion that would result in more defined opinions on how this 
facility should work to be widely acceptable.

Suggested attendance: folks working on live patching and anyone else 
who's not directly involved, but has strong opinions on how things should 
be done.

[1] http://lkml.kernel.org/r/20150220104418.GD25076@gmail.com

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [CORE TOPIC] live patching

[Masami Hiramatsu]

On 2015/07/08 5:42, Jiri Kosina wrote:
> [ I was wondering whether to tag this "CORE TOPIC" or "TECH TOPIC", as I 
>   really see it on the bordeline; let's see ]
> 
> As people are probably aware, the basic infrastructure for live patching 
> of the kernel has been merged and there are still quite some loose ends to 
> tie up.
> 
> On top of original agreement by the two teams working at live kernel 
> patching at Red Hat and SUSE, there has been quite some feedback raised 
> during upstream review by people not being directly involved in the 
> development itself [1].
> 
> Tangentially related topics have been spawned in the discussion, namely:
> 
> - stack unwinding has so far been deemed purely debugging facility, with 
>   no correctness guarantees whatsoever. One of the possible aproaches to
>   improving the coverage of live patching infrastructure would assume
>   100% reliability of in-kernel stack unwinding.
> 
>   Is this something that should be pursued further? Especially opinions 
>   and comments from various arch maintainers would be welcome

Yeah, at least the live patch would be better to depend on a frame-pointer
to improve correctness...

> 
> - it would be beneficial for live patching if kernel threads could be 
>   cleaned up; currently it's a mess, with freezer points, signal handling 
>   etc. spewed randomly all over the main loop. Generalizing the API (such
>   as improving kthread_worker and migrating kthreads to use that) would 
>   be beneficial if there are no major objections. See my related TECH 
>   TOPIC submission
> 
> - Ingo proposed a "checkpoint"-based way for live patching, i.e. forcing 
>   all the processess out of kernel to serliaze on a well-defined execution
>   point, perform the patching, and resume the execution.

IMO, this is not acceptable especially for most of real-time scheduled tasks
which usually the mission critical systems require. And such systems are
the main target of the live patching feature I think, since those are
sometimes controlling civil infrastructure systems, like trains, plants,
factories and so on.

> 
>   I see several issues with this aproach, but Ingo seems to be really 
>   insisting on this to be the way to go.
> 
> I could give a short introduction to how the current in-kernel live 
> patching facility looks like and what are the things that are currently 
> missing (either because they were considered too controversial by some 
> (such as the stack unwinding reliability or lazy-migration) or because 
> they just have not been implemented yet) and hopefully create a basis for 
> a discussion that would result in more defined opinions on how this 
> facility should work to be widely acceptable.

I'd like to discuss about how to make better(prefer to live patching)
patches, and what kind of patches can not be applied too. If we can
make it clear, it is possible to make a tool to clarify.

Thank you,

> 
> Suggested attendance: folks working on live patching and anyone else 
> who's not directly involved, but has strong opinions on how things should 
> be done.
> 
> [1] http://lkml.kernel.org/r/20150220104418.GD25076@gmail.com
> 
> Thanks,
> 


-- 
Masami HIRAMATSU
Linux Technology Research Center, System Productivity Research Dept.
Center for Technology Innovation - Systems Engineering
Hitachi, Ltd., Research & Development Group
E-mail: masami.hiramatsu.pt@hitachi.com

Re: [Ksummit-discuss] [CORE TOPIC] live patching

[Jiri Kosina]

On Thu, 9 Jul 2015, Masami Hiramatsu wrote:

> > - stack unwinding has so far been deemed purely debugging facility, with 
> >   no correctness guarantees whatsoever. One of the possible aproaches to
> >   improving the coverage of live patching infrastructure would assume
> >   100% reliability of in-kernel stack unwinding.
> > 
> >   Is this something that should be pursued further? Especially opinions 
> >   and comments from various arch maintainers would be welcome
> 
> Yeah, at least the live patch would be better to depend on a frame-pointer
> to improve correctness...

I actually think that Josh is planning to add dwarf2 CFI support as well 
(but yes, first step is getting frame pointer based unwinding to work 
properly).

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [CORE TOPIC] live patching

[Kamezawa Hiroyuki]

On 2015/07/09 1:13, Masami Hiramatsu wrote:
>>
>> - it would be beneficial for live patching if kernel threads could be
>>    cleaned up; currently it's a mess, with freezer points, signal handling
>>    etc. spewed randomly all over the main loop. Generalizing the API (such
>>    as improving kthread_worker and migrating kthreads to use that) would
>>    be beneficial if there are no major objections. See my related TECH
>>    TOPIC submission
>>
>> - Ingo proposed a "checkpoint"-based way for live patching, i.e. forcing
>>    all the processess out of kernel to serliaze on a well-defined execution
>>    point, perform the patching, and resume the execution.
>
> IMO, this is not acceptable especially for most of real-time scheduled tasks
> which usually the mission critical systems require. And such systems are
> the main target of the live patching feature I think, since those are
> sometimes controlling civil infrastructure systems, like trains, plants,
> factories and so on.
>
I'm sorr if this is off-topic.

I think the problem of "freezer v.s. FUSE" will happen if we neet to stop user proceesses.
AFAIK, the problem is a deadlock that fuse daemon freezes even while its client wating for a reply.

example)
    client -> page-fault -> GUP -> fuse daemon -> gethostbyname() -> nscd.8

We can't detect this kind of dependency if we need to freeze userland.

Thanks,
-Kame

Re: [Ksummit-discuss] [CORE TOPIC] live patching

[Jiri Kosina]

On Thu, 9 Jul 2015, Kamezawa Hiroyuki wrote:

> I'm sorr if this is off-topic.
> 
> I think the problem of "freezer v.s. FUSE" will happen if we neet to 
> stop user proceesses. AFAIK, the problem is a deadlock that fuse daemon 
> freezes even while its client wating for a reply.
> 
> example)
>    client -> page-fault -> GUP -> fuse daemon -> gethostbyname() -> nscd.8
> 
> We can't detect this kind of dependency if we need to freeze userland.

Freezing of userspace is happening every time system goes through 
hibernation.

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [CORE TOPIC] live patching

[Kamezawa Hiroyuki]

On 2015/07/09 18:56, Jiri Kosina wrote:
> On Thu, 9 Jul 2015, Kamezawa Hiroyuki wrote:
>
>> I'm sorr if this is off-topic.
>>
>> I think the problem of "freezer v.s. FUSE" will happen if we neet to
>> stop user proceesses. AFAIK, the problem is a deadlock that fuse daemon
>> freezes even while its client wating for a reply.
>>
>> example)
>>     client -> page-fault -> GUP -> fuse daemon -> gethostbyname() -> nscd.8
>>
>> We can't detect this kind of dependency if we need to freeze userland.
>
> Freezing of userspace is happening every time system goes through
> hibernation.
>

yes. so, we need to find a way for fixing deadlock or avoiding it.

AFAIK, Hibernation+FUSE problem has been being reported since several years ago.
I think there are some workaround to stop using FUSE before hibernation by
hand or script or systemd but it seems not to be an option for live patching.

Thanks,
-Kame

Re: [Ksummit-discuss] [CORE TOPIC] live patching

[Theodore Ts'o]

I'd like to ask folks whether they feel we still need further
discussion on Live Patching at the kernel summit.  Not being present
at the Live Kernel Patching miniconf, if folks could describe what
issues still need to be resolved or discussed, I'd really appreciate
it.

Thanks,

					- Ted

Re: [Ksummit-discuss] [CORE TOPIC] live patching

[Jiri Kosina]

On Mon, 12 Oct 2015, Theodore Ts'o wrote:

> I'd like to ask folks whether they feel we still need further
> discussion on Live Patching at the kernel summit.  Not being present
> at the Live Kernel Patching miniconf, if folks could describe what
> issues still need to be resolved or discussed, I'd really appreciate
> it.

Hi Ted,

quite frankly, I think the need for discussing this has temporarily 
diminished, because currently common preparatory steps are being worked on 
(which are generally beneficial even without live patching) -- namely 
Josh's work on reliable stack traces.

Only after that work is finished, the actual implementation of live 
patching improvements (and hence discussions) will resume.

So probably not really appropriate for this year's kernel summit at the 
end of the day, because there is no actual discussion happening at the 
very moment. Will very likely be a up-to-date topic for next year though.

Thanks,

-- 
Jiri Kosina
SUSE Labs

Re: [Ksummit-discuss] [CORE TOPIC] live patching

[Jiri Kosina]

On Tue, 13 Oct 2015, Jiri Kosina wrote:

> quite frankly, I think the need for discussing this has temporarily 
> diminished, because currently common preparatory steps are being worked on 
> (which are generally beneficial even without live patching) -- namely 
> Josh's work on reliable stack traces.

Oh, and also rather important aspect -- the relevance of this topic quite 
depends on the folks that will be present (is the invitee list available 
somewhere by any chance?).

To sum this one up -- I can definitely give a short summary of the status 
quo of the live patching efforts (both what's currently merged and what's 
currently being worked on), but I am not sure how much interesting that's 
for general KS folk. The more controversial discussions are temporarily 
paused for the time being; I expect them to be resume within a few months 
from now.

Thanks,

-- 
Jiri Kosina
SUSE Labs