From: Pete Wright <pete@nomadlogic.org>
To: david@kdbarto.org, The Eunuchs Hysterical Society <tuhs@tuhs.org>
Subject: Re: [TUHS] Quotas - did anyone ever use them?
Date: Fri, 31 May 2019 12:07:22 -0700 [thread overview]
Message-ID: <78485efd-2cd3-a290-8142-48672bb847c5@nomadlogic.org> (raw)
In-Reply-To: <975B93B6-AD7C-41B5-A14D-2DE4FEFAD3A6@kdbarto.org>
On 5/30/19 6:49 AM, David wrote:
> I think it was BSD 4.1 that added quotas to the disk system, and I was just wondering if anyone ever used them, in academia or industry. As a user and an admin I never used this and, while I thought it was interesting, just figured that the users would sort it out amongst themselves. Which they mostly did.
>
> So, anyone ever use this feature?
Lots of interesting insights/stories on this thread so figured i'd throw
my hat in the ring and share a business anecdote...
For quite a while i worked in the special effects/animation industry
where fortunately (for me) unix has a long and interesting history. One
secret abut the VFX world is that it's tremendously expensive with
relatively little financial upside. in my experience it is the studios
who get most of the residual income from a blockbuster feature. also a
crew for a AAA feature requires lots of human power, computers and
storage. my shop frequently had 3-5 features in full production mode at
the same time so we were redlining all of our systems 24/7.
So aside from the cost of maintaining a large renderfarm, unix/linux 3d
workstations, editing bays etc we also had an enormous NetApp footprint
to support all these systems. Now artists love creating lots and lots
of high resolution images, and if they had their way there were be
unlimited storage so they'd never have to archive a shot to tape in the
event they need to reference it later. But obviously that's not
reasonable from a financial perspective.
Our solution was to make heavy use of storage quotas in our environment,
and then leverage quotas to provide per-department billing. An
individual user was given something like 1GB by default (here they had
their mailbox, source-code, scripts etc), then the show they were booked
on was then given an allocation of say 1TB of storage. The show would
then carve out this allocation in a per-shot basis. This allowed us as
an organization to actually keep pretty detailed records on our costs
and unfortunately isn't something I've seen replicated well at lots of
startups flush with cash these days.
I was briefly on the team responsible for managing these quotas for a
show and it was seriously an around the clock operation to keep our
disks from filling up. One of the tricks was to figure out how much
space a rendered sequence of images would consume, factor in the
time-to-render a frame and attempt to line up your backup jobs to free
up enough space so the render nodes could write out the images to NFS.
-pete
--
Pete Wright
pete@nomadlogic.org
@nomadlogicLA
next prev parent reply other threads:[~2019-05-31 19:14 UTC|newest]
Thread overview: 32+ messages / expand[flat|nested] mbox.gz Atom feed top
2019-05-30 13:49 David
2019-05-30 14:23 ` Diomidis Spinellis
2019-05-30 14:26 ` Dan Cross
2019-05-30 14:27 ` Rico Pajarola
2019-05-31 7:16 ` George Ross
2019-05-30 14:29 ` Robert Brockway
2019-05-30 14:34 ` Theodore Ts'o
2019-05-30 14:48 ` John P. Linderman
2019-05-30 14:57 ` Jim Geist
2019-05-30 14:55 ` Andreas Kusalananda Kähäri
2019-05-30 15:00 ` KatolaZ
2019-05-30 17:28 ` Grant Taylor via TUHS
2019-05-30 19:19 ` Michael Kjörling
2019-05-30 20:42 ` Warner Losh
2019-05-30 22:23 ` George Michaelson
2019-05-31 1:36 ` alan
2019-05-31 19:07 ` Pete Wright [this message]
2019-05-31 20:43 ` Grant Taylor via TUHS
2019-05-31 20:59 ` Pete Wright
2019-06-01 0:30 ` reed
2019-06-04 13:50 ` Tony Finch
2019-05-30 16:04 Noel Chiappa
2019-05-30 16:48 ` KatolaZ
2019-05-30 17:42 ` ron
2019-05-31 0:21 Nelson H. F. Beebe
2019-05-31 0:37 ` Larry McVoy
2019-05-31 0:42 ` Arthur Krewat
2019-05-31 15:05 ` Nelson H. F. Beebe
2019-05-31 16:06 ` Michael Kjörling
2019-05-31 16:15 ` Grant Taylor via TUHS
2019-05-31 16:38 ` Michael Kjörling
2019-05-31 15:55 ` Rico Pajarola
Reply instructions:
You may reply publicly to this message via plain-text email
using any one of the following methods:
* Save the following mbox file, import it into your mail client,
and reply-to-all from there: mbox
Avoid top-posting and favor interleaved quoting:
https://en.wikipedia.org/wiki/Posting_style#Interleaved_style
* Reply using the --to, --cc, and --in-reply-to
switches of git-send-email(1):
git send-email \
--in-reply-to=78485efd-2cd3-a290-8142-48672bb847c5@nomadlogic.org \
--to=pete@nomadlogic.org \
--cc=david@kdbarto.org \
--cc=tuhs@tuhs.org \
/path/to/YOUR_REPLY
https://kernel.org/pub/software/scm/git/docs/git-send-email.html
* If your mail client supports setting the In-Reply-To header
via mailto: links, try the mailto: link
Be sure your reply has a Subject: header at the top and a blank line
before the message body.
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).