Question regarding page fault handlers in kernel mappings

Question regarding page fault handlers in kernel mappings

[Andrey Grodzovsky]

Hello, I am AMD developer and I am trying to implement support for on the fly 
graceful graphic card extraction. One issue I am facing is how to avoid accesses 
to physical addresses both in RAM and MMIO from user mode and kernel after 
device is gone. For user accesses (mmap) I use the page fault handler to route 
all RW accesses to dummy zero page. I would like to do the same for kernel side 
mappings both form RAM (kmap) and device IO
(ioremap) but it looks like there is no same mechanism of page fault handlers 
for kernel side mappings.

Am I correct in my understanding ?

Thanks,
Andrey

Re: Question regarding page fault handlers in kernel mappings

[Matthew Wilcox]

On Mon, Jan 04, 2021 at 11:38:38AM -0500, Andrey Grodzovsky wrote:
> Hello, I am AMD developer and I am trying to implement support for on the
> fly graceful graphic card extraction.

Are you talking about surprise removal (eg card on the other end of
a Thunderbolt connector where there is no possibility for software
locking), or are you talking about an orderly removal (where the user
requests removal and there is time to tear everything down gracefully)?

> One issue I am facing is how to avoid
> accesses to physical addresses both in RAM and MMIO from user mode and
> kernel after device is gone. For user accesses (mmap) I use the page fault
> handler to route all RW accesses to dummy zero page. I would like to do the
> same for kernel side mappings both form RAM (kmap) and device IO
> (ioremap) but it looks like there is no same mechanism of page fault
> handlers for kernel side mappings.

ioremap() is done through the vmalloc space.  It would, in theory, be
possible to reprogram the page tables used for vmalloc to point to your
magic page.  I don't think we have such a mechanism today, and there are
lots of problems with things like TLB flushes.  It's probably going to
be harder than you think.

I'm adding the linux-pci mailing list so you can be helped with the
logistics of device hot-remove.

Re: Question regarding page fault handlers in kernel mappings

[Andrey Grodzovsky]

On 1/4/21 11:56 AM, Matthew Wilcox wrote:
> On Mon, Jan 04, 2021 at 11:38:38AM -0500, Andrey Grodzovsky wrote:
>> Hello, I am AMD developer and I am trying to implement support for on the
>> fly graceful graphic card extraction.
> Are you talking about surprise removal (eg card on the other end of
> a Thunderbolt connector where there is no possibility for software
> locking), or are you talking about an orderly removal (where the user
> requests removal and there is time to tear everything down gracefully)?


Surprise removal


>
>> One issue I am facing is how to avoid
>> accesses to physical addresses both in RAM and MMIO from user mode and
>> kernel after device is gone. For user accesses (mmap) I use the page fault
>> handler to route all RW accesses to dummy zero page. I would like to do the
>> same for kernel side mappings both form RAM (kmap) and device IO
>> (ioremap) but it looks like there is no same mechanism of page fault
>> handlers for kernel side mappings.
> ioremap() is done through the vmalloc space.  It would, in theory, be
> possible to reprogram the page tables used for vmalloc to point to your
> magic page.  I don't think we have such a mechanism today, and there are
> lots of problems with things like TLB flushes.  It's probably going to
> be harder than you think.
>
> I'm adding the linux-pci mailing list so you can be helped with the
> logistics of device hot-remove.


Thanks, that makes sense as I couldn't find any clear documentation on how to handle
page faults for kernel page table while there is a clear mechanism and 
documenting on how
it's done for user processes  page tables (implementing the 
vm_operations_struct.fault callback)

It indeed the would be useful if some one (maybe on PCI side) could give me an 
advise on how
best to avoid accessing MMIO mappings made through ioremap  once I identify the 
device is gone.
The best we came up with until now is to explicitly test for device being preset 
before
doing any MMIO r/w access.

Andrey