I'm working on adding memory protection key support to glibc.

I don't think pkey_free, as it is implemented today, is very safe due to 
key reuse by a subsequent pkey_alloc.  I see two problems:

(A) pkey_free allows reuse for they key while there are still mappings 
that use it.

(B) If a key is reused, existing threads retain their access rights, 
while there is an expectation that pkey_alloc denies access for the 
threads except the current one.

Issue (A) could be fixed by having pkey_free to mark the key for reuse, 
and only actually reuse it if all those mappings are gone.  This could 
have a significant performance cost, but pkey_free is supposed to be rare.

Issue (B) is much harder to fix.  There is no atomic way to change 
access for a single key, so there is always a race condition due to the 
read-modify-write cycle for the PKRU update in user space.  This means 
that even if the kernel iterated over all threads to revoke access on 
pkey_free, there is a chance that the race reinstantiates the old access 
rights.

One way to deal with this is to give up and just remove pkey_free from 
the API (i.e., we wouldn't provide it in glibc).  A slightly less 
drastic way could add two pkey_alloc flags, a flag to disable pkey_free 
for the new key (which would mainly serve as a documentation of intent), 
and another flag which requests a pristine key which has never been used 
before.  With the second flag, and assuming correct key management, 
libraries would have some confidence that other threads in the process 
would not implicitly gain access to the new key (although there is the 
init_pkru= boot flag, which overrides the thread default, so it doesn't 
look like the assumption is actually valid).

All this is of course a bit on thin ice anyway because code could just 
clear the PKRU register at any time.

I'm attaching my glibc patch for reference.  The interesting bits is 
probably the test case (and how it creates and joins threads) and the 
pkey_set/pkey_get functions.  The support/ subdirectory is just our 
testing framework which is still very younga??I needed a few more 
functions for debugging, which is why they are in this patch.

Key reuse is not the only problem, we also have an issue with siglongjmp:

   https://sourceware.org/bugzilla/show_bug.cgi?id=22396

I've started wondering whether it even makes sense to expose this 
interface for general use.  I don't think any other architecture will 
implement something like this in the same way (with a PKRU register 
which can simply be cleared, and keys which are easily guessed and 
reused).  I suspect the only use for this functionality is in-memory 
databases which use DAX mappings for persistence, and want to reduce 
risk of persistent data corruption due to random pointer writes.  (And 
maybe execute-only memory, but that's not really benefiting anyone anyway.)

Thanks,
Florian

PS: The manpages need fixing.  Right now, they are misleading.