Re: [LSF/MM/BPF TOPIC] Cloud storage optimizations

[Damien Le Moal <damien.lemoal@opensource.wdc.com>]

On 3/5/23 02:54, Matthew Wilcox wrote:
> On Sat, Mar 04, 2023 at 06:17:35PM +0100, Hannes Reinecke wrote:
>> On 3/4/23 17:47, Matthew Wilcox wrote:
>>> On Sat, Mar 04, 2023 at 12:08:36PM +0100, Hannes Reinecke wrote:
>>>> We could implement a (virtual) zoned device, and expose each zone as a
>>>> block. That gives us the required large block characteristics, and with
>>>> a bit of luck we might be able to dial up to really large block sizes
>>>> like the 256M sizes on current SMR drives.
>>>> ublk might be a good starting point.
>>>
>>> Ummmm.  Is supporting 256MB block sizes really a desired goal?  I suggest
>>> that is far past the knee of the curve; if we can only write 256MB chunks
>>> as a single entity, we're looking more at a filesystem redesign than we
>>> are at making filesystems and the MM support 256MB size blocks.
>>>
>> Naa, not really. It _would_ be cool as we could get rid of all the cludges
>> which have nowadays re sequential writes.
>> And, remember, 256M is just a number someone thought to be a good
>> compromise. If we end up with a lower number (16M?) we might be able
>> to convince the powers that be to change their zone size.
>> Heck, with 16M block size there wouldn't be a _need_ for zones in
>> the first place.
>>
>> But yeah, 256M is excessive. Initially I would shoot for something
>> like 2M.
> 
> I think we're talking about different things (probably different storage
> vendors want different things, or even different people at the same
> storage vendor want different things).
> 
> Luis and I are talking about larger LBA sizes.  That is, the minimum
> read/write size from the block device is 16kB or 64kB or whatever.
> In this scenario, the minimum amount of space occupied by a file goes
> up from 512 bytes or 4kB to 64kB.  That's doable, even if somewhat
> suboptimal.

FYI, that is already out there, even though hidden from the host for backward
compatibility reasons. Example: WD SMR drives use 64K distributed sectors, which
is essentially 16 4KB sectors stripped together to achieve stronger ECC).

C.f. Distributed sector format (DSEC):
https://documents.westerndigital.com/content/dam/doc-library/en_us/assets/public/western-digital/collateral/tech-brief/tech-brief-ultrasmr-technology.pdf

This is hidden to the host though, and the LBA remains 512B or 4KB. This however
does result in measurable impact on IOPS with small reads as a sub-64K read
needs to be internally processed as a 64KB read to get the entire DSEC. The drop
in performance is not dramatic: about 5% lower IOPS compared to an equivalent
drive without DSEC. Still, that matters considering HDD IO density issues
(IOPS/TB) but in the case of SMR, that is part of the increased capacity trade-off.

So exposing the DSEC directly as the LBA size is not a stretch for the HDD FW,
as long as the host supports that. There are no plans to do so though, but we
could try experimenting.

For host side experimentation, something like qemu/nvme device emulation or
tcmu-runner for scsi devices, should be able to allow emulating large block size
fairly easily.

> 
> Your concern seems to be more around shingled devices (or their equivalent
> in SSD terms) where there are large zones which are append-only, but
> you can still random-read 512 byte LBAs.  I think there are different
> solutions to these problems, and people are working on both of these
> problems.

The above example does show that the device can generally implement emulation of
smaller LBA even with an internally larger read/write size unit. Having that
larger size unit advertised as the optimal IO size alignment (as it should) and
being more diligent in having FSes & mm use that may be a good approach too.

> 
> But if storage vendors are really pushing for 256MB LBAs, then that's
> going to need a third kind of solution, and I'm not aware of anyone
> working on that.

No we are not pushing for such crazy numbers :)
And for SMR case, smaller zone sizes are not desired as small zone size leads to
more real estate waste on the HDD platters, so lower total capacity (not desired
given that SMR is all about getting higher capacity "for free").


-- 
Damien Le Moal
Western Digital Research