Is it possible to corrupt disk when writeback page with undetected UE?

Is it possible to corrupt disk when writeback page with undetected UE?

[Jane Chu]

Hi,

Suppose there is a UE in a DRAM page that is backed by a disk file.
The UE hasn't been reported to the kernel, but low level firmware 
initiated scrubbing has already logged the UE.

The page is then dirtied by a write, although the write clearly failed,
it didn't trigger an MCE.

And without a subsequent read from the page, at some point, the page is 
written back to the disk, leaving a PAGE_SIZE of zeros in the targeted 
disk blocks.

Is this mode of disk corruption possible?

Thanks a lot!
-jane

RE: Is it possible to corrupt disk when writeback page with undetected UE?

[Luck, Tony]

> Suppose there is a UE in a DRAM page that is backed by a disk file.
> The UE hasn't been reported to the kernel, but low level firmware
> initiated scrubbing has already logged the UE.
>
> The page is then dirtied by a write, although the write clearly failed,
> it didn't trigger an MCE.
>
> And without a subsequent read from the page, at some point, the page is
> written back to the disk, leaving a PAGE_SIZE of zeros in the targeted
> disk blocks.
>
> Is this mode of disk corruption possible?

I didn't look at what was written to disk, but I have seen this. My test sequence
was to compile and then immediately run an error injection test program that
injected a memory UC error to an instruction.

Because the program was freshly compiled, the executable file was in the
page cache with all pages marked as modified. Later a sync (or memory
pressure) wrote the dirty page with poison to filesystem.

I did see an error reported by the disk controller.

-Tony

Re: Is it possible to corrupt disk when writeback page with undetected UE?

[Jane Chu]

On 9/15/2022 3:50 PM, Luck, Tony wrote:
>> Suppose there is a UE in a DRAM page that is backed by a disk file.
>> The UE hasn't been reported to the kernel, but low level firmware
>> initiated scrubbing has already logged the UE.
>>
>> The page is then dirtied by a write, although the write clearly failed,
>> it didn't trigger an MCE.
>>
>> And without a subsequent read from the page, at some point, the page is
>> written back to the disk, leaving a PAGE_SIZE of zeros in the targeted
>> disk blocks.
>>
>> Is this mode of disk corruption possible?
> 
> I didn't look at what was written to disk, but I have seen this. My test sequence
> was to compile and then immediately run an error injection test program that
> injected a memory UC error to an instruction.
> 
> Because the program was freshly compiled, the executable file was in the
> page cache with all pages marked as modified. Later a sync (or memory
> pressure) wrote the dirty page with poison to filesystem.
> 
> I did see an error reported by the disk controller.

Thanks a lot for this information!

Were you using madvise to inject an error to a mmap'ed address?
or a different tool?  Do you still have the test documented
somewhere?

And, aside from verifying every write with a read prior to sync,
any suggestion to minimize the window of such corruption?

thanks!
-jane

> 
> -Tony

Re: Is it possible to corrupt disk when writeback page with undetected UE?

[Yang Shi]

On Thu, Sep 15, 2022 at 5:27 PM Jane Chu <jane.chu@oracle.com> wrote:
>
> On 9/15/2022 3:50 PM, Luck, Tony wrote:
> >> Suppose there is a UE in a DRAM page that is backed by a disk file.
> >> The UE hasn't been reported to the kernel, but low level firmware
> >> initiated scrubbing has already logged the UE.
> >>
> >> The page is then dirtied by a write, although the write clearly failed,
> >> it didn't trigger an MCE.
> >>
> >> And without a subsequent read from the page, at some point, the page is
> >> written back to the disk, leaving a PAGE_SIZE of zeros in the targeted
> >> disk blocks.
> >>
> >> Is this mode of disk corruption possible?
> >
> > I didn't look at what was written to disk, but I have seen this. My test sequence
> > was to compile and then immediately run an error injection test program that
> > injected a memory UC error to an instruction.
> >
> > Because the program was freshly compiled, the executable file was in the
> > page cache with all pages marked as modified. Later a sync (or memory
> > pressure) wrote the dirty page with poison to filesystem.
> >
> > I did see an error reported by the disk controller.
>
> Thanks a lot for this information!
>
> Were you using madvise to inject an error to a mmap'ed address?
> or a different tool?  Do you still have the test documented
> somewhere?
>
> And, aside from verifying every write with a read prior to sync,
> any suggestion to minimize the window of such corruption?

We discussed the topic at this year's LSFMM summit. Please refer to
https://lwn.net/Articles/893565/

>
> thanks!
> -jane
>
> >
> > -Tony
>

Re: Is it possible to corrupt disk when writeback page with undetected UE?

[Jane Chu]

On 9/15/2022 5:30 PM, Yang Shi wrote:
> On Thu, Sep 15, 2022 at 5:27 PM Jane Chu <jane.chu@oracle.com> wrote:
>>
>> On 9/15/2022 3:50 PM, Luck, Tony wrote:
>>>> Suppose there is a UE in a DRAM page that is backed by a disk file.
>>>> The UE hasn't been reported to the kernel, but low level firmware
>>>> initiated scrubbing has already logged the UE.
>>>>
>>>> The page is then dirtied by a write, although the write clearly failed,
>>>> it didn't trigger an MCE.
>>>>
>>>> And without a subsequent read from the page, at some point, the page is
>>>> written back to the disk, leaving a PAGE_SIZE of zeros in the targeted
>>>> disk blocks.
>>>>
>>>> Is this mode of disk corruption possible?
>>>
>>> I didn't look at what was written to disk, but I have seen this. My test sequence
>>> was to compile and then immediately run an error injection test program that
>>> injected a memory UC error to an instruction.
>>>
>>> Because the program was freshly compiled, the executable file was in the
>>> page cache with all pages marked as modified. Later a sync (or memory
>>> pressure) wrote the dirty page with poison to filesystem.
>>>
>>> I did see an error reported by the disk controller.
>>
>> Thanks a lot for this information!
>>
>> Were you using madvise to inject an error to a mmap'ed address?
>> or a different tool?  Do you still have the test documented
>> somewhere?
>>
>> And, aside from verifying every write with a read prior to sync,
>> any suggestion to minimize the window of such corruption?
> 
> We discussed the topic at this year's LSFMM summit. Please refer to
> https://lwn.net/Articles/893565/

Thanks!  I'll take a look.

-jane

> 
>>
>> thanks!
>> -jane
>>
>>>
>>> -Tony
>>

RE: Is it possible to corrupt disk when writeback page with undetected UE?

[Luck, Tony]

> Were you using madvise to inject an error to a mmap'ed address?
> or a different tool?  Do you still have the test documented
> somewhere?

I was injecting with ACPI/EINJ (so tweaking some ECC bits in memory to create
a real uncorrectable error). This was a long time back when I was just trying to
get basic recovery from usermode access to poison working reliably. So I just
noted the workaround ("make; sync; run_test") to keep making progress.

Handling poison in the page cache has been on my TODO list for a long time.
Someday it will make it to the top.

> And, aside from verifying every write with a read prior to sync,
> any suggestion to minimize the window of such corruption?

There's no cheap solution. As you point out the best that can be done
is to reduce the window (since bits may get flipped after you perform
your check but before DMS to storage).

-Tony

Re: Is it possible to corrupt disk when writeback page with undetected UE?

[Jane Chu]

On 9/16/2022 9:17 AM, Luck, Tony wrote:
>> Were you using madvise to inject an error to a mmap'ed address?
>> or a different tool?  Do you still have the test documented
>> somewhere?
> 
> I was injecting with ACPI/EINJ (so tweaking some ECC bits in memory to create
> a real uncorrectable error). This was a long time back when I was just trying to
> get basic recovery from usermode access to poison working reliably. So I just
> noted the workaround ("make; sync; run_test") to keep making progress.
> 
> Handling poison in the page cache has been on my TODO list for a long time.
> Someday it will make it to the top.

I see, looking forward to your patches.

> 
>> And, aside from verifying every write with a read prior to sync,
>> any suggestion to minimize the window of such corruption?
> 
> There's no cheap solution. As you point out the best that can be done
> is to reduce the window (since bits may get flipped after you perform
> your check but before DMS to storage).

Sounds like the disk controller is the last in the chain in terms
of detecting a late UE, so if the disk controller detection could
trickle up to a filesystem level action marking 'file:<offset,len>'
being bad and relate the information to user for repair, that might be 
reasonable...

thanks,
-jane

> 
> -Tony
>