Re: [9fans] fossil pb: FOUND!

["Aram Hăvărneanu" <aram.h@mgk.ro>]

erik quanstrom wrote:
> i think it would be fair to argue that source and executables are
> negligeable users of storage.  media files, which are already compressed,
> tend to dominate.

What about virtual machine images?

> the tradeoff for this compression is a large amount of memory,
> fragmentation, and cpu usage.  that is to say, storage latency.

I have 24GB RAM. My primary laptops have 8GB RAM. I have all this RAM
not because of dedup but because I do memory intensive tasks, like
running virtual machines. I believe this is true for many users.

I'm of a completely different opinion regarding fragmentation. On
SSDs, it's a non issue. Historically, one of the hardest things to do
right in a filesystem was minimizing fragmentation. Today you don't
have to do it so there's less complexity to manage in the file system.
Even if you still have rotating rust to store the bulk of the data, a
small SSD cache in front of it renders fragmentation irrelevant.

My CPU can SHA-1 hash orders of magnitude faster than it can read from
disk, and that's using only generic instructions, plus, it's sitting
idle anyway.

> so i wonder if we're not spending all our resources trying to optimize
> only a few percent of our storage needs.

Dedup is certainly not a panacea, but it's certainly useful for many workloads.

-- 
Aram Hăvărneanu