x86/pkeys in early kernel version

x86/pkeys in early kernel version

[Jeff Xu]

Hello,

I'm investigating if there is a need to backport x86/pkeys fix/feature
into earlier kernel
versions, Chrome is starting to use PKEY in x86, and I hope experts
here can give
advice on this.

For background, ChromeOS regularly syncs with upstream kernel
versions, and has production
that uses 4.4/4.14/4.19/5.4/5.10/5.15.

We plan to use PKEY as Stephen Röttger proposed in
https://docs.google.com/document/d/1O2jwK4dxI3nRcOJuPYkonhTkNQfbmwdvxQMyXgeaRHo/edit?usp=sharing
In short, Chrome will use PKEY to protect a thread and have exclusive
access to some pages.
If there are fixes/features to make this happen, we can backport those
from the ChromeOS branch.

Thanks!
Best regards,
Jeff

Re: x86/pkeys in early kernel version

[Dave Hansen]

On 1/25/23 11:02, Jeff Xu wrote:
> I'm investigating if there is a need to backport x86/pkeys
> fix/feature into earlier kernel versions, Chrome is starting to use
> PKEY in x86, and I hope experts here can give advice on this.
> 
> For background, ChromeOS regularly syncs with upstream kernel 
> versions, and has production that uses 4.4/4.14/4.19/5.4/5.10/5.15.
To be honest, I haven't got the foggiest idea what you need to backport.
 I can barely keep track of mainline.

Are there really production 4.4 kernels that you need to run on
pkey-capable hardware?  That would mean running a 2015-era kernel on a
CPU released in late 2020.  I think Q3'2020 is when the 11th gen CPUs
came out which were the first non-server CPUs that had pkeys.

On a positive note, the pkeys selftest has been pretty consistently
updated as we find bugs.  I'd be curious how well a mainline version of
that selftests runs on old kernels.  But, I'm too scared to find out
what's down that particular rabbit hole myself.

Re: x86/pkeys in early kernel version

[Jeff Xu]

On Wed, Jan 25, 2023 at 11:13 AM Dave Hansen <dave.hansen@intel.com> wrote:
>
> On 1/25/23 11:02, Jeff Xu wrote:
> > I'm investigating if there is a need to backport x86/pkeys
> > fix/feature into earlier kernel versions, Chrome is starting to use
> > PKEY in x86, and I hope experts here can give advice on this.
> >
> > For background, ChromeOS regularly syncs with upstream kernel
> > versions, and has production that uses 4.4/4.14/4.19/5.4/5.10/5.15.
> To be honest, I haven't got the foggiest idea what you need to backport.
>  I can barely keep track of mainline.
>
> Are there really production 4.4 kernels that you need to run on
> pkey-capable hardware?  That would mean running a 2015-era kernel on a
> CPU released in late 2020.  I think Q3'2020 is when the 11th gen CPUs
> came out which were the first non-server CPUs that had pkeys.
>
Thanks!
For 11th gen CPUs, chromebook uses 5.4 and above, so that eliminate
half of the versions.

> On a positive note, the pkeys selftest has been pretty consistently
> updated as we find bugs.  I'd be curious how well a mainline version of
> that selftests runs on old kernels.  But, I'm too scared to find out
> what's down that particular rabbit hole myself.

I can start with 5.10 or 5.15, it seems there are quite some changes though,
for example,  this one by Thomas
https://lore.kernel.org/lkml/20210623120127.327154589@linutronix.de/

My question is, if I have to pick a version that doesn't require a lot
of backporting,
and functionality is stable enough, what version would this be ? 5.4/5.10/5.15 ?

-Jeff

Re: x86/pkeys in early kernel version

[Jeff Xu]

On Wed, Jan 25, 2023 at 5:43 PM Jeff Xu <jeffxu@chromium.org> wrote:
>
> On Wed, Jan 25, 2023 at 11:13 AM Dave Hansen <dave.hansen@intel.com> wrote:
> >
> > On 1/25/23 11:02, Jeff Xu wrote:
> > > I'm investigating if there is a need to backport x86/pkeys
> > > fix/feature into earlier kernel versions, Chrome is starting to use
> > > PKEY in x86, and I hope experts here can give advice on this.
> > >
> > > For background, ChromeOS regularly syncs with upstream kernel
> > > versions, and has production that uses 4.4/4.14/4.19/5.4/5.10/5.15.
> > To be honest, I haven't got the foggiest idea what you need to backport.
> >  I can barely keep track of mainline.
> >
> > Are there really production 4.4 kernels that you need to run on
> > pkey-capable hardware?  That would mean running a 2015-era kernel on a
> > CPU released in late 2020.  I think Q3'2020 is when the 11th gen CPUs
> > came out which were the first non-server CPUs that had pkeys.
> >
> Thanks!
> For 11th gen CPUs, chromebook uses 5.4 and above, so that eliminate
> half of the versions.
>
> > On a positive note, the pkeys selftest has been pretty consistently
> > updated as we find bugs.  I'd be curious how well a mainline version of
> > that selftests runs on old kernels.  But, I'm too scared to find out
> > what's down that particular rabbit hole myself.
>
I took the latest selftest from main and run on 5.15 kernel,
all passed except test_ptrace_modifies_pkru
assert() at protection_keys.c::1623 test_nr: 20 iteration: 1

Is there a bugfix for the ptrace area ?
Thanks




> I can start with 5.10 or 5.15, it seems there are quite some changes though,
> for example,  this one by Thomas
> https://lore.kernel.org/lkml/20210623120127.327154589@linutronix.de/
>
> My question is, if I have to pick a version that doesn't require a lot
> of backporting,
> and functionality is stable enough, what version would this be ? 5.4/5.10/5.15 ?
>
> -Jeff

Re: x86/pkeys in early kernel version

[Kyle Huey]

On Thu, Jan 26, 2023 at 9:36 PM Jeff Xu <jeffxu@chromium.org> wrote:
>
> On Wed, Jan 25, 2023 at 5:43 PM Jeff Xu <jeffxu@chromium.org> wrote:
> >
> > On Wed, Jan 25, 2023 at 11:13 AM Dave Hansen <dave.hansen@intel.com> wrote:
> > >
> > > On 1/25/23 11:02, Jeff Xu wrote:
> > > > I'm investigating if there is a need to backport x86/pkeys
> > > > fix/feature into earlier kernel versions, Chrome is starting to use
> > > > PKEY in x86, and I hope experts here can give advice on this.
> > > >
> > > > For background, ChromeOS regularly syncs with upstream kernel
> > > > versions, and has production that uses 4.4/4.14/4.19/5.4/5.10/5.15.
> > > To be honest, I haven't got the foggiest idea what you need to backport.
> > >  I can barely keep track of mainline.
> > >
> > > Are there really production 4.4 kernels that you need to run on
> > > pkey-capable hardware?  That would mean running a 2015-era kernel on a
> > > CPU released in late 2020.  I think Q3'2020 is when the 11th gen CPUs
> > > came out which were the first non-server CPUs that had pkeys.
> > >
> > Thanks!
> > For 11th gen CPUs, chromebook uses 5.4 and above, so that eliminate
> > half of the versions.
> >
> > > On a positive note, the pkeys selftest has been pretty consistently
> > > updated as we find bugs.  I'd be curious how well a mainline version of
> > > that selftests runs on old kernels.  But, I'm too scared to find out
> > > what's down that particular rabbit hole myself.
> >
> I took the latest selftest from main and run on 5.15 kernel,
> all passed except test_ptrace_modifies_pkru
> assert() at protection_keys.c::1623 test_nr: 20 iteration: 1
>
> Is there a bugfix for the ptrace area ?
> Thanks


What 5.15 series kernel did you run it on? The patches for that didn't
get backported until 5.15.88

- Kyle

>
>
> > I can start with 5.10 or 5.15, it seems there are quite some changes though,
> > for example,  this one by Thomas
> > https://lore.kernel.org/lkml/20210623120127.327154589@linutronix.de/
> >
> > My question is, if I have to pick a version that doesn't require a lot
> > of backporting,
> > and functionality is stable enough, what version would this be ? 5.4/5.10/5.15 ?
> >
> > -Jeff

Re: x86/pkeys in early kernel version

[Dave Hansen]

On 1/25/23 17:43, Jeff Xu wrote:
> My question is, if I have to pick a version that doesn't require a
> lot of backporting, and functionality is stable enough, what version
> would this be ? 5.4/5.10/5.15 ?

Virtually all bugs get fixed in mainline first and then backported to
wherever... sometimes.  If you want the most bug-free kernel, go as
close to upstream as possible.

Re: x86/pkeys in early kernel version

[Jeff Xu]

On Fri, Jan 27, 2023 at 7:22 AM Dave Hansen <dave.hansen@intel.com> wrote:
>
> On 1/25/23 17:43, Jeff Xu wrote:
> > My question is, if I have to pick a version that doesn't require a
> > lot of backporting, and functionality is stable enough, what version
> > would this be ? 5.4/5.10/5.15 ?
>
> Virtually all bugs get fixed in mainline first and then backported to
> wherever... sometimes.  If you want the most bug-free kernel, go as
> close to upstream as possible.
Thanks for the tip.

Re: x86/pkeys in early kernel version

[Jeff Xu]

On Thu, Jan 26, 2023 at 9:55 PM Kyle Huey <me@kylehuey.com> wrote:
>
> On Thu, Jan 26, 2023 at 9:36 PM Jeff Xu <jeffxu@chromium.org> wrote:
> >
> > On Wed, Jan 25, 2023 at 5:43 PM Jeff Xu <jeffxu@chromium.org> wrote:
> > >
> > > On Wed, Jan 25, 2023 at 11:13 AM Dave Hansen <dave.hansen@intel.com> wrote:
> > > >
> > > > On 1/25/23 11:02, Jeff Xu wrote:
> > > > > I'm investigating if there is a need to backport x86/pkeys
> > > > > fix/feature into earlier kernel versions, Chrome is starting to use
> > > > > PKEY in x86, and I hope experts here can give advice on this.
> > > > >
> > > > > For background, ChromeOS regularly syncs with upstream kernel
> > > > > versions, and has production that uses 4.4/4.14/4.19/5.4/5.10/5.15.
> > > > To be honest, I haven't got the foggiest idea what you need to backport.
> > > >  I can barely keep track of mainline.
> > > >
> > > > Are there really production 4.4 kernels that you need to run on
> > > > pkey-capable hardware?  That would mean running a 2015-era kernel on a
> > > > CPU released in late 2020.  I think Q3'2020 is when the 11th gen CPUs
> > > > came out which were the first non-server CPUs that had pkeys.
> > > >
> > > Thanks!
> > > For 11th gen CPUs, chromebook uses 5.4 and above, so that eliminate
> > > half of the versions.
> > >
> > > > On a positive note, the pkeys selftest has been pretty consistently
> > > > updated as we find bugs.  I'd be curious how well a mainline version of
> > > > that selftests runs on old kernels.  But, I'm too scared to find out
> > > > what's down that particular rabbit hole myself.
> > >
> > I took the latest selftest from main and run on 5.15 kernel,
> > all passed except test_ptrace_modifies_pkru
> > assert() at protection_keys.c::1623 test_nr: 20 iteration: 1
> >
> > Is there a bugfix for the ptrace area ?
> > Thanks
>
>
> What 5.15 series kernel did you run it on? The patches for that didn't
> get backported until 5.15.88
>
Thanks! I'm using 5.15.87.
Will this patch set be backported to 5.4 and 5.10 ?
The selftest (from main) also failed on 5.4, in the same test,
but at different line:
assert() at protection_keys.c::1651 test_nr: 20 iteration: 1

- Jeff

> - Kyle
>
> >
> >
> > > I can start with 5.10 or 5.15, it seems there are quite some changes though,
> > > for example,  this one by Thomas
> > > https://lore.kernel.org/lkml/20210623120127.327154589@linutronix.de/
> > >
> > > My question is, if I have to pick a version that doesn't require a lot
> > > of backporting,
> > > and functionality is stable enough, what version would this be ? 5.4/5.10/5.15 ?
> > >
> > > -Jeff

Re: x86/pkeys in early kernel version

[Kyle Huey]

On Fri, Jan 27, 2023 at 11:08 AM Jeff Xu <jeffxu@chromium.org> wrote:
>
> On Thu, Jan 26, 2023 at 9:55 PM Kyle Huey <me@kylehuey.com> wrote:
> >
> > On Thu, Jan 26, 2023 at 9:36 PM Jeff Xu <jeffxu@chromium.org> wrote:
> > >
> > > On Wed, Jan 25, 2023 at 5:43 PM Jeff Xu <jeffxu@chromium.org> wrote:
> > > >
> > > > On Wed, Jan 25, 2023 at 11:13 AM Dave Hansen <dave.hansen@intel.com> wrote:
> > > > >
> > > > > On 1/25/23 11:02, Jeff Xu wrote:
> > > > > > I'm investigating if there is a need to backport x86/pkeys
> > > > > > fix/feature into earlier kernel versions, Chrome is starting to use
> > > > > > PKEY in x86, and I hope experts here can give advice on this.
> > > > > >
> > > > > > For background, ChromeOS regularly syncs with upstream kernel
> > > > > > versions, and has production that uses 4.4/4.14/4.19/5.4/5.10/5.15.
> > > > > To be honest, I haven't got the foggiest idea what you need to backport.
> > > > >  I can barely keep track of mainline.
> > > > >
> > > > > Are there really production 4.4 kernels that you need to run on
> > > > > pkey-capable hardware?  That would mean running a 2015-era kernel on a
> > > > > CPU released in late 2020.  I think Q3'2020 is when the 11th gen CPUs
> > > > > came out which were the first non-server CPUs that had pkeys.
> > > > >
> > > > Thanks!
> > > > For 11th gen CPUs, chromebook uses 5.4 and above, so that eliminate
> > > > half of the versions.
> > > >
> > > > > On a positive note, the pkeys selftest has been pretty consistently
> > > > > updated as we find bugs.  I'd be curious how well a mainline version of
> > > > > that selftests runs on old kernels.  But, I'm too scared to find out
> > > > > what's down that particular rabbit hole myself.
> > > >
> > > I took the latest selftest from main and run on 5.15 kernel,
> > > all passed except test_ptrace_modifies_pkru
> > > assert() at protection_keys.c::1623 test_nr: 20 iteration: 1
> > >
> > > Is there a bugfix for the ptrace area ?
> > > Thanks
> >
> >
> > What 5.15 series kernel did you run it on? The patches for that didn't
> > get backported until 5.15.88
> >
> Thanks! I'm using 5.15.87.
> Will this patch set be backported to 5.4 and 5.10 ?
> The selftest (from main) also failed on 5.4, in the same test,
> but at different line:
> assert() at protection_keys.c::1651 test_nr: 20 iteration: 1

The regression that patch set was intended to fix was introduced in
5.14. I don't know why the test is failing on 5.4 but I have no plans
to investigate it.

- Kyle

> - Jeff
>
> > - Kyle
> >
> > >
> > >
> > > > I can start with 5.10 or 5.15, it seems there are quite some changes though,
> > > > for example,  this one by Thomas
> > > > https://lore.kernel.org/lkml/20210623120127.327154589@linutronix.de/
> > > >
> > > > My question is, if I have to pick a version that doesn't require a lot
> > > > of backporting,
> > > > and functionality is stable enough, what version would this be ? 5.4/5.10/5.15 ?
> > > >
> > > > -Jeff

Re: x86/pkeys in early kernel version

[Kyle Huey]

On Fri, Jan 27, 2023 at 11:22 AM Kyle Huey <me@kylehuey.com> wrote:
>
> On Fri, Jan 27, 2023 at 11:08 AM Jeff Xu <jeffxu@chromium.org> wrote:
> >
> > On Thu, Jan 26, 2023 at 9:55 PM Kyle Huey <me@kylehuey.com> wrote:
> > >
> > > On Thu, Jan 26, 2023 at 9:36 PM Jeff Xu <jeffxu@chromium.org> wrote:
> > > >
> > > > On Wed, Jan 25, 2023 at 5:43 PM Jeff Xu <jeffxu@chromium.org> wrote:
> > > > >
> > > > > On Wed, Jan 25, 2023 at 11:13 AM Dave Hansen <dave.hansen@intel.com> wrote:
> > > > > >
> > > > > > On 1/25/23 11:02, Jeff Xu wrote:
> > > > > > > I'm investigating if there is a need to backport x86/pkeys
> > > > > > > fix/feature into earlier kernel versions, Chrome is starting to use
> > > > > > > PKEY in x86, and I hope experts here can give advice on this.
> > > > > > >
> > > > > > > For background, ChromeOS regularly syncs with upstream kernel
> > > > > > > versions, and has production that uses 4.4/4.14/4.19/5.4/5.10/5.15.
> > > > > > To be honest, I haven't got the foggiest idea what you need to backport.
> > > > > >  I can barely keep track of mainline.
> > > > > >
> > > > > > Are there really production 4.4 kernels that you need to run on
> > > > > > pkey-capable hardware?  That would mean running a 2015-era kernel on a
> > > > > > CPU released in late 2020.  I think Q3'2020 is when the 11th gen CPUs
> > > > > > came out which were the first non-server CPUs that had pkeys.
> > > > > >
> > > > > Thanks!
> > > > > For 11th gen CPUs, chromebook uses 5.4 and above, so that eliminate
> > > > > half of the versions.
> > > > >
> > > > > > On a positive note, the pkeys selftest has been pretty consistently
> > > > > > updated as we find bugs.  I'd be curious how well a mainline version of
> > > > > > that selftests runs on old kernels.  But, I'm too scared to find out
> > > > > > what's down that particular rabbit hole myself.
> > > > >
> > > > I took the latest selftest from main and run on 5.15 kernel,
> > > > all passed except test_ptrace_modifies_pkru
> > > > assert() at protection_keys.c::1623 test_nr: 20 iteration: 1
> > > >
> > > > Is there a bugfix for the ptrace area ?
> > > > Thanks
> > >
> > >
> > > What 5.15 series kernel did you run it on? The patches for that didn't
> > > get backported until 5.15.88
> > >
> > Thanks! I'm using 5.15.87.
> > Will this patch set be backported to 5.4 and 5.10 ?
> > The selftest (from main) also failed on 5.4, in the same test,
> > but at different line:
> > assert() at protection_keys.c::1651 test_nr: 20 iteration: 1
>
> The regression that patch set was intended to fix was introduced in
> 5.14. I don't know why the test is failing on 5.4 but I have no plans
> to investigate it.

Just looking at the line the test is failing on though I would suspect
that when PKRU was being managed by XSAVE (pre-5.14) that the PKRU
register didn't get updated for clearing XSTATE_BV until the XRSTOR
was actually executed (upon return to userspace). So multiple ptrace
calls in succession without userspace code execution would see a stale
PKRU value if the PKRU register was "changed" by clearing the relevant
XSTATE_BV flag. This is an extreme edge case, so I doubt you actually
care about the behavior.

- Kyle

> - Kyle
>
> > - Jeff
> >
> > > - Kyle
> > >
> > > >
> > > >
> > > > > I can start with 5.10 or 5.15, it seems there are quite some changes though,
> > > > > for example,  this one by Thomas
> > > > > https://lore.kernel.org/lkml/20210623120127.327154589@linutronix.de/
> > > > >
> > > > > My question is, if I have to pick a version that doesn't require a lot
> > > > > of backporting,
> > > > > and functionality is stable enough, what version would this be ? 5.4/5.10/5.15 ?
> > > > >
> > > > > -Jeff

Re: x86/pkeys in early kernel version

[Jeff Xu]

On Fri, Jan 27, 2023 at 11:30 AM Kyle Huey <me@kylehuey.com> wrote:
>
> On Fri, Jan 27, 2023 at 11:22 AM Kyle Huey <me@kylehuey.com> wrote:
> >
> > On Fri, Jan 27, 2023 at 11:08 AM Jeff Xu <jeffxu@chromium.org> wrote:
> > >
> > > On Thu, Jan 26, 2023 at 9:55 PM Kyle Huey <me@kylehuey.com> wrote:
> > > >
> > > > On Thu, Jan 26, 2023 at 9:36 PM Jeff Xu <jeffxu@chromium.org> wrote:
> > > > >
> > > > > On Wed, Jan 25, 2023 at 5:43 PM Jeff Xu <jeffxu@chromium.org> wrote:
> > > > > >
> > > > > > On Wed, Jan 25, 2023 at 11:13 AM Dave Hansen <dave.hansen@intel.com> wrote:
> > > > > > >
> > > > > > > On 1/25/23 11:02, Jeff Xu wrote:
> > > > > > > > I'm investigating if there is a need to backport x86/pkeys
> > > > > > > > fix/feature into earlier kernel versions, Chrome is starting to use
> > > > > > > > PKEY in x86, and I hope experts here can give advice on this.
> > > > > > > >
> > > > > > > > For background, ChromeOS regularly syncs with upstream kernel
> > > > > > > > versions, and has production that uses 4.4/4.14/4.19/5.4/5.10/5.15.
> > > > > > > To be honest, I haven't got the foggiest idea what you need to backport.
> > > > > > >  I can barely keep track of mainline.
> > > > > > >
> > > > > > > Are there really production 4.4 kernels that you need to run on
> > > > > > > pkey-capable hardware?  That would mean running a 2015-era kernel on a
> > > > > > > CPU released in late 2020.  I think Q3'2020 is when the 11th gen CPUs
> > > > > > > came out which were the first non-server CPUs that had pkeys.
> > > > > > >
> > > > > > Thanks!
> > > > > > For 11th gen CPUs, chromebook uses 5.4 and above, so that eliminate
> > > > > > half of the versions.
> > > > > >
> > > > > > > On a positive note, the pkeys selftest has been pretty consistently
> > > > > > > updated as we find bugs.  I'd be curious how well a mainline version of
> > > > > > > that selftests runs on old kernels.  But, I'm too scared to find out
> > > > > > > what's down that particular rabbit hole myself.
> > > > > >
> > > > > I took the latest selftest from main and run on 5.15 kernel,
> > > > > all passed except test_ptrace_modifies_pkru
> > > > > assert() at protection_keys.c::1623 test_nr: 20 iteration: 1
> > > > >
> > > > > Is there a bugfix for the ptrace area ?
> > > > > Thanks
> > > >
> > > >
> > > > What 5.15 series kernel did you run it on? The patches for that didn't
> > > > get backported until 5.15.88
> > > >
> > > Thanks! I'm using 5.15.87.
> > > Will this patch set be backported to 5.4 and 5.10 ?
> > > The selftest (from main) also failed on 5.4, in the same test,
> > > but at different line:
> > > assert() at protection_keys.c::1651 test_nr: 20 iteration: 1
> >
> > The regression that patch set was intended to fix was introduced in
> > 5.14. I don't know why the test is failing on 5.4 but I have no plans
> > to investigate it.
>
> Just looking at the line the test is failing on though I would suspect
> that when PKRU was being managed by XSAVE (pre-5.14) that the PKRU
> register didn't get updated for clearing XSTATE_BV until the XRSTOR
> was actually executed (upon return to userspace). So multiple ptrace
> calls in succession without userspace code execution would see a stale
> PKRU value if the PKRU register was "changed" by clearing the relevant
> XSTATE_BV flag. This is an extreme edge case, so I doubt you actually
> care about the behavior.
>
Thank you for the details!
-Jeff

> - Kyle
>
> > - Kyle
> >
> > > - Jeff
> > >
> > > > - Kyle
> > > >
> > > > >
> > > > >
> > > > > > I can start with 5.10 or 5.15, it seems there are quite some changes though,
> > > > > > for example,  this one by Thomas
> > > > > > https://lore.kernel.org/lkml/20210623120127.327154589@linutronix.de/
> > > > > >
> > > > > > My question is, if I have to pick a version that doesn't require a lot
> > > > > > of backporting,
> > > > > > and functionality is stable enough, what version would this be ? 5.4/5.10/5.15 ?
> > > > > >
> > > > > > -Jeff

Re: x86/pkeys in early kernel version

[Jeff Xu]

On Fri, Jan 27, 2023 at 1:12 PM Jeff Xu <jeffxu@chromium.org> wrote:
>
> On Fri, Jan 27, 2023 at 11:30 AM Kyle Huey <me@kylehuey.com> wrote:
> >
> > On Fri, Jan 27, 2023 at 11:22 AM Kyle Huey <me@kylehuey.com> wrote:
> > >
> > > On Fri, Jan 27, 2023 at 11:08 AM Jeff Xu <jeffxu@chromium.org> wrote:
> > > >
> > > > On Thu, Jan 26, 2023 at 9:55 PM Kyle Huey <me@kylehuey.com> wrote:
> > > > >
> > > > > On Thu, Jan 26, 2023 at 9:36 PM Jeff Xu <jeffxu@chromium.org> wrote:
> > > > > >
> > > > > > On Wed, Jan 25, 2023 at 5:43 PM Jeff Xu <jeffxu@chromium.org> wrote:
> > > > > > >
> > > > > > > On Wed, Jan 25, 2023 at 11:13 AM Dave Hansen <dave.hansen@intel.com> wrote:
> > > > > > > >
> > > > > > > > On 1/25/23 11:02, Jeff Xu wrote:
> > > > > > > > > I'm investigating if there is a need to backport x86/pkeys
> > > > > > > > > fix/feature into earlier kernel versions, Chrome is starting to use
> > > > > > > > > PKEY in x86, and I hope experts here can give advice on this.
> > > > > > > > >
> > > > > > > > > For background, ChromeOS regularly syncs with upstream kernel
> > > > > > > > > versions, and has production that uses 4.4/4.14/4.19/5.4/5.10/5.15.
> > > > > > > > To be honest, I haven't got the foggiest idea what you need to backport.
> > > > > > > >  I can barely keep track of mainline.
> > > > > > > >
> > > > > > > > Are there really production 4.4 kernels that you need to run on
> > > > > > > > pkey-capable hardware?  That would mean running a 2015-era kernel on a
> > > > > > > > CPU released in late 2020.  I think Q3'2020 is when the 11th gen CPUs
> > > > > > > > came out which were the first non-server CPUs that had pkeys.
> > > > > > > >
> > > > > > > Thanks!
> > > > > > > For 11th gen CPUs, chromebook uses 5.4 and above, so that eliminate
> > > > > > > half of the versions.
> > > > > > >
> > > > > > > > On a positive note, the pkeys selftest has been pretty consistently
> > > > > > > > updated as we find bugs.  I'd be curious how well a mainline version of
> > > > > > > > that selftests runs on old kernels.  But, I'm too scared to find out
> > > > > > > > what's down that particular rabbit hole myself.
> > > > > > >
> > > > > > I took the latest selftest from main and run on 5.15 kernel,
> > > > > > all passed except test_ptrace_modifies_pkru
> > > > > > assert() at protection_keys.c::1623 test_nr: 20 iteration: 1
> > > > > >
> > > > > > Is there a bugfix for the ptrace area ?
> > > > > > Thanks
> > > > >
> > > > >
> > > > > What 5.15 series kernel did you run it on? The patches for that didn't
> > > > > get backported until 5.15.88
> > > > >
> > > > Thanks! I'm using 5.15.87.
> > > > Will this patch set be backported to 5.4 and 5.10 ?
> > > > The selftest (from main) also failed on 5.4, in the same test,
> > > > but at different line:
> > > > assert() at protection_keys.c::1651 test_nr: 20 iteration: 1
> > >
> > > The regression that patch set was intended to fix was introduced in
> > > 5.14. I don't know why the test is failing on 5.4 but I have no plans
> > > to investigate it.
> >
> > Just looking at the line the test is failing on though I would suspect
> > that when PKRU was being managed by XSAVE (pre-5.14) that the PKRU
> > register didn't get updated for clearing XSTATE_BV until the XRSTOR
> > was actually executed (upon return to userspace). So multiple ptrace
> > calls in succession without userspace code execution would see a stale
> > PKRU value if the PKRU register was "changed" by clearing the relevant
> > XSTATE_BV flag. This is an extreme edge case, so I doubt you actually
> > care about the behavior.
> >

I have another case of test_ptrace_modifier_pkru failure.
This is happening in AMD 5000 CPU and 5.15.98 kernel.
The odd thing about this is:
if I run the whole set of protection_keys (20 cases), it will pass.
If I run the last case (by comment out the others), it will fail with
below error:

has pkeys: 1
startup pkey_reg: 0000000055555550
assert() at protection_keys.c::1623 test_nr: 0 iteration: 1
running abort_hooks()...
errno at assert: 0

And the same test on Intel CPU is passing.

I wonder if this is known or someone has a repro ?

Another question regarding PKRU, may or maynot related to this failure on AMD:
During the thread context switch, will PKRU be saved to the thread's
user space stack? Is this what XSAVE does (pre-5.14), and if we are
not using XSAVE after 5.15, what is used ?

Thanks
-Jeff



> Thank you for the details!
> -Jeff
>
> > - Kyle
> >
> > > - Kyle
> > >
> > > > - Jeff
> > > >
> > > > > - Kyle
> > > > >
> > > > > >
> > > > > >
> > > > > > > I can start with 5.10 or 5.15, it seems there are quite some changes though,
> > > > > > > for example,  this one by Thomas
> > > > > > > https://lore.kernel.org/lkml/20210623120127.327154589@linutronix.de/
> > > > > > >
> > > > > > > My question is, if I have to pick a version that doesn't require a lot
> > > > > > > of backporting,
> > > > > > > and functionality is stable enough, what version would this be ? 5.4/5.10/5.15 ?
> > > > > > >
> > > > > > > -Jeff

Re: x86/pkeys in early kernel version

[Dave Hansen]

On 3/13/23 16:24, Jeff Xu wrote:
> I have another case of test_ptrace_modifier_pkru failure.
> This is happening in AMD 5000 CPU and 5.15.98 kernel.
> The odd thing about this is:
> if I run the whole set of protection_keys (20 cases), it will pass.
> If I run the last case (by comment out the others), it will fail with
> below error:
> 
> has pkeys: 1
> startup pkey_reg: 0000000055555550
> assert() at protection_keys.c::1623 test_nr: 0 iteration: 1
> running abort_hooks()...
> errno at assert: 0
> 
> And the same test on Intel CPU is passing.
> 
> I wonder if this is known or someone has a repro ?
> 
> Another question regarding PKRU, may or maynot related to this failure on AMD:
> During the thread context switch, will PKRU be saved to the thread's
> user space stack? Is this what XSAVE does (pre-5.14), and if we are
> not using XSAVE after 5.15, what is used ?

One Intel vs. AMD delta that I seem to recall is their init trackers.

On Intel, the only "normal" way to get an XSAVE component back to being
tracked as in its init state is with XRSTOR and the right incantations
of XSTATE_BV and RFBM.

On AMD, I think the init tracker is much more magic and aggressive.  In
addition to a careful XRSTOR, WRPKRU(0) _can_ put the PKRU state
component back to being tracked as being in its init state.  Also an
XRSTOR with a PKRU value of 0 can do it.

Basically, on AMD the register state matters for init tracking.  On
Intel, the actual register contents are not taken into account.

It might be interesting to dump xgetbv(1) around the failures to see
what's going on with the init tracker.