[9fans] Throwing in the Towel

[9fans] Throwing in the Towel

[Steven Stallion]

All,

It has taken almost 15 years, but fossil has finally managed to chew
through a second set of SSDs and I just don't have the heart to feed
it a third. As of last night, I backed up my venti store, dutifully
recorded vacs, and deracked the equipment.

I first encountered Plan 9 while working for Coraid in 2010. I can't
express how grateful I feel for the opportunity to have worked with
Coraid, the remaining few folks at Bell Labs, and even squeezing in a
few patches before everything was shut down. In the intervening years
my D525's have hummed along happily, long after I moved on to other
pursuits.

I've never had much of a presence on 9fans, but I've always been one
to keep an eye on the list and I wish everyone success. If 9p.io ever
comes back online, I've stuffed everything I could into my contrib
directory.

Take care,
Steve

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T2ca67486c7a13a77-Mbd55ad67ccc8316577d599bc
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Plan9 wept! - Throwing in the Towel

[Alexandr Babic]

plan9 is one of the silmarills and you just throw it into throat of ungoliant :-(


/n/fingolfinFS

 
----- Původní zpráva -----
Odesilatel: Steven Stallion (sstallion@gmail.com)
Datum: 05/04/24 17:15
Příjemce: Fans of the OS Plan 9 from Bell Labs (9fans@9fans.net)
Předmět: [9fans] Throwing in the Towel

All,

It has taken almost 15 years, but fossil has finally managed to chew
through a second set of SSDs and I just don't have the heart to feed
it a third. As of last night, I backed up my venti store, dutifully
recorded vacs, and deracked the equipment.

I first encountered Plan 9 while working for Coraid in 2010. I can't
express how grateful I feel for the opportunity to have worked with
Coraid, the remaining few folks at Bell Labs, and even squeezing in a
few patches before everything was shut down. In the intervening years
my D525's have hummed along happily, long after I moved on to other
pursuits.

I've never had much of a presence on 9fans, but I've always been one
to keep an eye on the list and I wish everyone success. If 9p.io ever
comes back online, I've stuffed everything I could into my contrib
directory.

Take care,
Steve


------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T274be0b8bc52ca8b-Mb6b83959b73bc199f9c4f736
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Throwing in the Towel

[Shawn Rutledge]

> On May 4, 2024, at 8:13 AM, Steven Stallion <sstallion@gmail.com> wrote:
> 
> All,
> 
> It has taken almost 15 years, but fossil has finally managed to chew
> through a second set of SSDs

What does that mean?  Does fossil do too many writes, or did they fail for some other reason?  Did you have a heavy write load in the first place?  Are you sure the SSDs would have lasted longer than 15 years with a different fs?  It seems to me that SSDs were less common and lower-quality 15 years ago; there was a big difference between brands back then.

Are there other filesystems that work with venti?  It seems to me that there should be, eventually.


------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T2ca67486c7a13a77-M56ac4e7c9103c284061b3f98
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Throwing in the Towel

[wb.kloke]

[-- Attachment #1: Type: text/plain, Size: 509 bytes --]

Imho, we would like to get some more info, what happened.

1) Was it the fossil write buffer?
2) Was it the venti index ?

We can safely exclude the venti arenas, or ?
As you mentioned a 2nd set of SSDs, how long did the last one hold?
Why is the whole set affected? Raid-x?
------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T2ca67486c7a13a77-M471128607f61f2803a13f15c
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 1102 bytes --]

Re: [9fans] Throwing in the Towel

[wb.kloke]

[-- Attachment #1: Type: text/plain, Size: 672 bytes --]

I would like to refresh my questions from may 4th.

Can it be the case that the venti index file exhibits a desastrous write pattern for SSDs?

I presume that each new block written to venti  causes a random block to be rewritten in the index file, until the bucket is full (after 215 writes). Given, that the erasure blocks in the SSD may be of size 1MB or even 128MB, each such write may be a big strain on the device. 

Any opinions?
------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T2ca67486c7a13a77-M45995ae775e5ed31efd6cdb6
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 1258 bytes --]

Re: [9fans] Throwing in the Towel

[Riddler]

[-- Attachment #1: Type: text/plain, Size: 2345 bytes --]

Back of the napkin:

venti(8) has an index entry as 40 bytes. That would fill a 1MB erasure
block in 25k writes. A SLC should handle at least twice that. A MLC is
probably less than half that from what I understand. But we haven't
discussed wear levelling yet. With the index recommended to be 5% of the
data size there should be plenty of data blocks with write cycles left for
the SSD to wear level with. I wouldn't worry about it.

The 128MB case however is pretty bad. It'll take 3,200k writes. Given the
data blocks, following the 4k assumption from the manual, will fill a 128MB
block in 32k writes that could kill a  cheap low write capacity SSD just
with the data blocks.

My cursory conclusion; back of the napkin, index behaviour is not ideal but
shouldn't cause any issues for regular wear levelling SSD (which is all of
them now right?) with a few MB erasure blocks. But going beyond that size
and yeah... Ouch.

But that's just small index updates and large erasure block SSDs for you.
Transactional databases doing lots of small updates are also known for
being able to kill SSDs. As I understand it common practice is to over
provision a transactional database SSD to make up for it.


On Sun, 26 May 2024, 20:40 , <wb.kloke@gmail.com> wrote:

> I would like to refresh my questions from may 4th.
>
> Can it be the case that the venti index file exhibits a desastrous write
> pattern for SSDs?
>
> I presume that each new block written to venti  causes a random block to
> be rewritten in the index file, until the bucket is full (after 215
> writes). Given, that the erasure blocks in the SSD may be of size 1MB or
> even 128MB, each such write may be a big strain on the device.
>
> Any opinions?
> *9fans <https://9fans.topicbox.com/latest>* / 9fans / see discussions
> <https://9fans.topicbox.com/groups/9fans> + participants
> <https://9fans.topicbox.com/groups/9fans/members> + delivery options
> <https://9fans.topicbox.com/groups/9fans/subscription> Permalink
> <https://9fans.topicbox.com/groups/9fans/T2ca67486c7a13a77-M45995ae775e5ed31efd6cdb6>
>

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T2ca67486c7a13a77-Mccab748b1a801a2849a72e6e
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 3084 bytes --]

Re: [9fans] Throwing in the Towel

[Dave Eckhardt]

> Can it be the case that the venti index file exhibits a desastrous
> write pattern for SSDs?

Maybe... but it would need to be worse than the load provided by
other file systems, including FAT and ext2/3/4.  Some of those
file systems rewrite some blocks like mad (e.g., superblock, journal
blocks).

Just because a flash translation layer might need to erase an entire
megabyte, or multiple megabytes, does not mean that it needs to put
the new value of a logical block in the same area of the flash that
the old value of that logical block used to be in (they do the opposite).

It would be interesting to get a dump of the SMART data for the drive(s)
that went bad, and also drive(s) that didn't go bad.

Dave Eckhardt

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T2ca67486c7a13a77-Mda9b0251a0482ce40f14c606
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Throwing in the Towel

[wb.kloke]

[-- Attachment #1: Type: text/plain, Size: 982 bytes --]

For the napkin calculation: On disk, the IEntry is 38Bytes. Alas, writes occur always in (the ssd internal) blocksize. So, essentially (assuming 4096 byte blocksize, which is quite optimistic), we have a write efficiency of less than 1 percent.

A good firmware in the ssd could avoid needing a new block for the write, if all bits are changed in teh same direction by the new data. This is defeated by the binary search in the index buckets. The entries must be sorted, and the number of used entries is tob updated. So, for each new entry the whole (internal) block has to be remapped. Therefore an erasure block must be erased after about  1m/4k index writes.

It seems, venti in its current form is a ssd killer, if they are used for the isects.
------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T2ca67486c7a13a77-M510201f156a99eb26c5c7268
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 1534 bytes --]

Re: [9fans] Throwing in the Towel

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 1581 bytes --]

i'm curious what straightforward storage structure wouldn't be. trying to
second-guess ssd firmware seems a tricky design criterion.

On Tue, 28 May 2024 at 20:34, <wb.kloke@gmail.com> wrote:

> For the napkin calculation: On disk, the IEntry is 38Bytes. Alas, writes
> occur always in (the ssd internal) blocksize. So, essentially (assuming
> 4096 byte blocksize, which is quite optimistic), we have a write efficiency
> of less than 1 percent.
>
> A good firmware in the ssd could avoid needing a new block for the write,
> if all bits are changed in teh same direction by the new data. This is
> defeated by the binary search in the index buckets. The entries must be
> sorted, and the number of used entries is tob updated. So, for each new
> entry the whole (internal) block has to be remapped. Therefore an erasure
> block must be erased after about  1m/4k index writes.
>
> It seems, venti in its current form is a ssd killer, if they are used for
> the isects.
> *9fans <https://9fans.topicbox.com/latest>* / 9fans / see discussions
> <https://9fans.topicbox.com/groups/9fans> + participants
> <https://9fans.topicbox.com/groups/9fans/members> + delivery options
> <https://9fans.topicbox.com/groups/9fans/subscription> Permalink
> <https://9fans.topicbox.com/groups/9fans/T2ca67486c7a13a77-M510201f156a99eb26c5c7268>
>

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T2ca67486c7a13a77-Me67a4d43e72cda0aa878fc21
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 1995 bytes --]

Re: [9fans] Throwing in the Towel

[Dave Eckhardt]

> For the napkin calculation: On disk, the IEntry is 38Bytes.  Alas,
> writes occur always in (the ssd internal) blocksize.  So, essentially
> (assuming 4096 byte blocksize, which is quite optimistic), we have
> a write efficiency of less than 1 percent.

While I see how such a model can predict disaster, I don't think that
model matches how FTLs work, because it can't.

Many file systems (FAT, ext2/3/4) write the same logical block over
and over and over and over and over.  I think the default interval
for ext4 to synch the superblock and the journal is five seconds,
which if true is more than 15,000 times every *day* for a busy
file system (and I think lots of Linux systems are busy in that
sense).

> A good firmware in the ssd could avoid needing a new block for the
> write, if all bits are changed in teh same direction by the new
> data.

Again, I believe this model predicts that no regular Linux file
system can be used on any SSD, thus I believe this model is not
accurate.

To quote Wikipedia:

  https://en.wikipedia.org/wiki/Flash_memory_controller

> The mapping units of an FTL can differ so that LBAs are mapped
> block-, page- or even sub-page-based.  Depending on the usage
> pattern, a finer mapping granularity can significantly reduce
> the flash wear out and maximize the endurance of a flash based
> storage media.

Also, I feel as if this point is several assumption layers deep.
I think one user reported an unknown number of failures in two
sets of SSDs of unknown brand and model.  I don't think we know
that it was venti SSDs that went bad as opposed to fossil SSDs,
let alone knowing it was index SSDs for venti.

> It seems, venti in its current form is a ssd killer, if they
> are used for the isects.

I don't think this claim is yet supported well.

Dave Eckhardt

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T2ca67486c7a13a77-Mfafe3b0a74f75cd266eb0a73
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Throwing in the Towel

[ori]

Finally,. SSDs just die over time. Especially if they are
not powered on and refreshing. JEDEC specs say that they
should retain data for 1 year unplugged when stored at 30
degrees celsius, assuming the internet isn't lying to me.

Keep backups.

Quoth Dave Eckhardt <davide+p9@cs.cmu.edu>:
> > For the napkin calculation: On disk, the IEntry is 38Bytes.  Alas,
> > writes occur always in (the ssd internal) blocksize.  So, essentially
> > (assuming 4096 byte blocksize, which is quite optimistic), we have
> > a write efficiency of less than 1 percent.
> 
> While I see how such a model can predict disaster, I don't think that
> model matches how FTLs work, because it can't.
> 
> Many file systems (FAT, ext2/3/4) write the same logical block over
> and over and over and over and over.  I think the default interval
> for ext4 to synch the superblock and the journal is five seconds,
> which if true is more than 15,000 times every *day* for a busy
> file system (and I think lots of Linux systems are busy in that
> sense).
> 
> > A good firmware in the ssd could avoid needing a new block for the
> > write, if all bits are changed in teh same direction by the new
> > data.
> 
> Again, I believe this model predicts that no regular Linux file
> system can be used on any SSD, thus I believe this model is not
> accurate.
> 
> To quote Wikipedia:
> 
>   https://en.wikipedia.org/wiki/Flash_memory_controller
> 
> > The mapping units of an FTL can differ so that LBAs are mapped
> > block-, page- or even sub-page-based.  Depending on the usage
> > pattern, a finer mapping granularity can significantly reduce
> > the flash wear out and maximize the endurance of a flash based
> > storage media.
> 
> Also, I feel as if this point is several assumption layers deep.
> I think one user reported an unknown number of failures in two
> sets of SSDs of unknown brand and model.  I don't think we know
> that it was venti SSDs that went bad as opposed to fossil SSDs,
> let alone knowing it was index SSDs for venti.
> 
> > It seems, venti in its current form is a ssd killer, if they
> > are used for the isects.
> 
> I don't think this claim is yet supported well.
> 
> Dave Eckhardt

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T2ca67486c7a13a77-M076297816c8a30d15e67368f
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Throwing in the Towel

[Steve Simon]

i  can only speak from experience, but i have had fossil and venti running on a single ssd (on a radpberry pi) for 5 years now - no rotating discs left at home.

i have mtime changes and ephemeral snapshots turned off to reduce the update rate. i chose a sandisk card, and take backups just in case…

so far so good.

-Steve

> On 29 May 2024, at 4:39 am, ori@eigenstate.org wrote:
> 
> Finally,. SSDs just die over time. Especially if they are
> not powered on and refreshing. JEDEC specs say that they
> should retain data for 1 year unplugged when stored at 30
> degrees celsius, assuming the internet isn't lying to me.
> 
> Keep backups.
> 
> Quoth Dave Eckhardt <davide+p9@cs.cmu.edu>:
>>> For the napkin calculation: On disk, the IEntry is 38Bytes.  Alas,
>>> writes occur always in (the ssd internal) blocksize.  So, essentially
>>> (assuming 4096 byte blocksize, which is quite optimistic), we have
>>> a write efficiency of less than 1 percent.
>> 
>> While I see how such a model can predict disaster, I don't think that
>> model matches how FTLs work, because it can't.
>> 
>> Many file systems (FAT, ext2/3/4) write the same logical block over
>> and over and over and over and over.  I think the default interval
>> for ext4 to synch the superblock and the journal is five seconds,
>> which if true is more than 15,000 times every *day* for a busy
>> file system (and I think lots of Linux systems are busy in that
>> sense).
>> 
>>> A good firmware in the ssd could avoid needing a new block for the
>>> write, if all bits are changed in teh same direction by the new
>>> data.
>> 
>> Again, I believe this model predicts that no regular Linux file
>> system can be used on any SSD, thus I believe this model is not
>> accurate.
>> 
>> To quote Wikipedia:
>> 
>>  https://en.wikipedia.org/wiki/Flash_memory_controller
>> 
>>> The mapping units of an FTL can differ so that LBAs are mapped
>>> block-, page- or even sub-page-based.  Depending on the usage
>>> pattern, a finer mapping granularity can significantly reduce
>>> the flash wear out and maximize the endurance of a flash based
>>> storage media.
>> 
>> Also, I feel as if this point is several assumption layers deep.
>> I think one user reported an unknown number of failures in two
>> sets of SSDs of unknown brand and model.  I don't think we know
>> that it was venti SSDs that went bad as opposed to fossil SSDs,
>> let alone knowing it was index SSDs for venti.
>> 
>>> It seems, venti in its current form is a ssd killer, if they
>>> are used for the isects.
>> 
>> I don't think this claim is yet supported well.
>> 
>> Dave Eckhardt

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T2ca67486c7a13a77-M8bfe23b05de7c8450157011c
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

Re: [9fans] Throwing in the Towel

[wb.kloke]

[-- Attachment #1: Type: text/plain, Size: 1035 bytes --]

> i'm curious what straightforward storage structure wouldn't be. trying to second-guess ssd firmware seems a tricky design criterion.
> 
Designing for minimal disk stress: Never rewrite data already written to disk. Now we have big and quite cheap main memory. 

I don't critisize the historical design choices. But times have changed.

If I had to design a venti now, I would drop the variable length blocks for 8k blocks (or whatever blocksize a modern fossil would use).
Further, I would store the data directly into disk blocks.. In a parallel log I would store the scores, so that the scores and blocks just can be referenced by the block number. 
At start up, read the score file and construct a memory search tree for a suitably shorted part of the score. (See Mechiel Lukkien's memventi).
------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T2ca67486c7a13a77-M26ca440629fc61a3eeb0bab2
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription

[-- Attachment #2: Type: text/html, Size: 1700 bytes --]

Re: [9fans] Throwing in the Towel

[hiro]

> Finally,. SSDs just die over time.

i can confirm.

> Keep backups.

i just hope i will be able to restore them, too. i'm too lazy to check
every year if stuff is still readable, and i fear it will wear out the
heads anyways ;)

most recent interesting read:
https://www.lto.org/wp-content/uploads/2024/05/LTOUltrium_2023_MediaShipmentReport.pdf

------------------------------------------
9fans: 9fans
Permalink: https://9fans.topicbox.com/groups/9fans/T2ca67486c7a13a77-Mae84dcb8843a0e9be3537403
Delivery options: https://9fans.topicbox.com/groups/9fans/subscription