The Unix Heritage Society mailing list
 help / Atom feed
* [TUHS] A question about ls(1)
@ 2019-04-27 14:16 Anthony Martin
  2019-04-27 15:38 ` Warner Losh
  2019-04-28 11:47 ` Dan Cross
  0 siblings, 2 replies; 23+ messages in thread
From: Anthony Martin @ 2019-04-27 14:16 UTC (permalink / raw)
  To: tuhs

From at least V2 to V6, the ls(1) command would not
show directory entries whose names began with a '.'
unless the -a flag was supplied.

This was changed in V7: only the directory entries
for "." and ".." would be skipped by default.

All further versions of Research Unix retain the
convention of V7 and Plan 9 ultimately made it
unnecessary. However, BSD and its descendants did
not follow suit. Instead, they continued behaving
like V6 with an additional -A flag to emulate V7.

Was the initial behavior intentional or just a
matter of expediency?

Who made the change and what was their motivation?
Was it a reaction to the intentional hiding of what
came to be known as "dot files"?

Thanks,
  Anthony

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [TUHS] A question about ls(1)
  2019-04-27 14:16 [TUHS] A question about ls(1) Anthony Martin
@ 2019-04-27 15:38 ` Warner Losh
  2019-04-27 15:42   ` Larry McVoy
  2019-04-28 23:59   ` Warner Losh
  2019-04-28 11:47 ` Dan Cross
  1 sibling, 2 replies; 23+ messages in thread
From: Warner Losh @ 2019-04-27 15:38 UTC (permalink / raw)
  To: Anthony Martin; +Cc: The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 1698 bytes --]

On Sat, Apr 27, 2019 at 8:26 AM Anthony Martin <ality@pbrane.org> wrote:

> From at least V2 to V6, the ls(1) command would not
> show directory entries whose names began with a '.'
> unless the -a flag was supplied.
>
> This was changed in V7: only the directory entries
> for "." and ".." would be skipped by default.
>

gnu ls does not follow this convention. The system V sources one can find
on the internet have the curious code:

#define DOTSUP  1
...
                        if (aflg==0 && dentry->d_name[0]=='.'
# ifndef DOTSUP
                                && (dentry->d_name[1]=='\0' ||
dentry->d_name[1]=='.'
                                && dentry->d_name[2]=='\0')
# endif
                                )  /* check for directory items '.', '..',
                                   *  and items without valid inode-number;
                                   */
                                continue;

which is the V7 behavior ifdef'd out.


> All further versions of Research Unix retain the
> convention of V7 and Plan 9 ultimately made it
> unnecessary. However, BSD and its descendants did
> not follow suit. Instead, they continued behaving
> like V6 with an additional -A flag to emulate V7.
>

This has been the BSD convention since at least 4BSD :)

But I find this interesting, since the 8th edition was based on BSD 4.1c I
thought....

And system III and later all have the above code.


> Was the initial behavior intentional or just a
> matter of expediency?
>
> Who made the change and what was their motivation?
> Was it a reaction to the intentional hiding of what
> came to be known as "dot files"?


It looks like, from the source, to be a blip in V7.

Warner

[-- Attachment #2: Type: text/html, Size: 2971 bytes --]

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [TUHS] A question about ls(1)
  2019-04-27 15:38 ` Warner Losh
@ 2019-04-27 15:42   ` Larry McVoy
  2019-04-28 23:59   ` Warner Losh
  1 sibling, 0 replies; 23+ messages in thread
From: Larry McVoy @ 2019-04-27 15:42 UTC (permalink / raw)
  To: Warner Losh; +Cc: The Eunuchs Hysterical Society

On Sat, Apr 27, 2019 at 09:38:27AM -0600, Warner Losh wrote:
> But I find this interesting, since the 8th edition was based on BSD 4.1c I
> thought....

Is that true?  I sort of lost track of V8 and V9, other than Dennis'
streams (not STREAMS) work.

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [TUHS] A question about ls(1)
@ 2019-04-27 20:11 norman
  0 siblings, 0 replies; 23+ messages in thread
From: norman @ 2019-04-27 20:11 UTC (permalink / raw)
  To: tuhs

On Sat, Apr 27, 2019 at 09:38:27AM -0600, Warner Losh wrote:
  But I find this interesting, since the 8th edition was based on BSD 4.1c I
  thought....

`Based on' is an overstatement, except in the kernel.

The system described in the 8th Edition manual (as noted in the
past, there was only sort of a real V8 release) had a kernel
that started as 4.1x BSD.  I'm not sure of the value of x.
It had the Joy/Babaolgu paging code and the complicated changes
to signals, and a lot of the gratuitous asms, but not a trace
of the BSD networking API.  Neither was the BSD FFS present.

Local additions included Dennis's original stream implementation,
which completely replaced the old tty code and rewrote the drivers
for serial-port devices.  The tty driver (responsible for cooked
mode, interrupt and quit signals, and the like) was a stream
module.  The BSD-style `new tty line discipline' was a separate
module, for the few people who couldn't live without csh.

Tom Killian's original version of /proc and Peter Weinberger's original
network file system (neta) client were there.  These were accessed
through a file system switch, also Peter's work.

Instead of FFS, Peter made a simple speedup to the V7 file system:
adding 64 to the minor device number meant the file system used 4KiB
blocks.  The unused space at the end of the now-4KiB superblock was a
bitmap of free blocks, allowing somewhat-smarter block allocation.

There was a hacky implementation of TCP/IP which we didn't really use:
4.Y BSD (I don't know the value of Y) protocol code, wrapped up to
work as stream modules* and shoehorned in, with a custom API quite
different from the BSD one.  The work was done by a summer student,
Robert T. Morris, who later became rather famous for a smaller but
rather more troublesome bit of network code.  Our production network
was Datakit, which was also implemented as stream devices and modules
(it was the network whose use inspired the stream design, in fact).

[* Quite a while ago, someone asked how multiplexing was handled
in the stream world.  I meant to write a reply but never did.  In
a sentence, by a paired device driver and stream module.  If someone
wants more details I'll be glad to write more about that.]

That's just the kernel, though.  The user-mode world was largely
descended from V7 rather than from BSD.  Most people used sh, which
had been augmented a bit by Rob Pike (perhaps et al) to add functions
and a simple external history mechanism (Tom Duff's idea, I think).
wc had no uucp-dependent flags, and cat had no flags at all.  ls did
sniff at whether standard output was a tty and put things in columns.

Things mutated further as time went on, further diverging from and
discarding aspects of the BSD origin.  (I can take credit or blame
for a lot of that, though not all.)  But that happened later, and
is reflected in the 9th and especially 10th editions of the manual.

Norman Wilson
Toronto ON

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [TUHS] A question about ls(1)
@ 2019-04-27 23:03 jnc
  0 siblings, 0 replies; 23+ messages in thread
From: jnc @ 2019-04-27 23:03 UTC (permalink / raw)
  To: tuhs; +Cc: jnc

    > From: Norman Wilson

    > Quite a while ago, someone asked how multiplexing was handled in the
    > stream world.  I meant to write a reply but never did.  In a sentence,
    > by a paired device driver and stream module.  If someone wants more
    > details I'll be glad to write more about that.

Please. Thanks!

	Noel

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [TUHS] A question about ls(1)
  2019-04-27 14:16 [TUHS] A question about ls(1) Anthony Martin
  2019-04-27 15:38 ` Warner Losh
@ 2019-04-28 11:47 ` Dan Cross
  2019-04-28 12:00   ` arnold
                     ` (2 more replies)
  1 sibling, 3 replies; 23+ messages in thread
From: Dan Cross @ 2019-04-28 11:47 UTC (permalink / raw)
  To: Anthony Martin; +Cc: The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 1536 bytes --]

On Sat, Apr 27, 2019 at 10:26 AM Anthony Martin <ality@pbrane.org> wrote:

> From at least V2 to V6, the ls(1) command would not
> show directory entries whose names began with a '.'
> unless the -a flag was supplied.
>
> This was changed in V7: only the directory entries
> for "." and ".." would be skipped by default.
>
> All further versions of Research Unix retain the
> convention of V7 and Plan 9 ultimately made it
> unnecessary. However, BSD and its descendants did
> not follow suit. Instead, they continued behaving
> like V6 with an additional -A flag to emulate V7.
>
> Was the initial behavior intentional or just a
> matter of expediency?
>

I believe it's been publicly stated that it was a mistake in early Unix.
Apparently, Rob Pike had a Google+ post to this effect back in 2012 (or
earlier): I see a reference to it from another mailing list around that
time. Unfortunately, the Google+ content is now lost. Rob, do you have a
copy?

Who made the change and what was their motivation?
> Was it a reaction to the intentional hiding of what
> came to be known as "dot files"?


Speaking from memory, I think the intent was to avoid showing '.' and '..',
and that a programming error accidentally hid all dotfiles; probably this
went unnoticed because there just weren't many of them at the time. This
was corrected in 7th Edition, but the existing behavior had already escaped
into the world via Berkeley.

I suspect the "dot file" convention came later, as a side-effect of
observed behavior of ls.

        - Dan C.

[-- Attachment #2: Type: text/html, Size: 2176 bytes --]

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [TUHS] A question about ls(1)
  2019-04-28 11:47 ` Dan Cross
@ 2019-04-28 12:00   ` arnold
  2019-04-28 14:44   ` jcs
  2019-04-28 16:15   ` Bakul Shah
  2 siblings, 0 replies; 23+ messages in thread
From: arnold @ 2019-04-28 12:00 UTC (permalink / raw)
  To: crossd, ality; +Cc: tuhs

Dan Cross <crossd@gmail.com> wrote:

> I suspect the "dot file" convention came later, as a side-effect of
> observed behavior of ls.

At some point the shell started colluding; echo *  didn't show
dot files either... That dates back at least as far as V7 also.

Arnold

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [TUHS] A question about ls(1)
  2019-04-28 11:47 ` Dan Cross
  2019-04-28 12:00   ` arnold
@ 2019-04-28 14:44   ` jcs
  2019-04-28 16:15   ` Bakul Shah
  2 siblings, 0 replies; 23+ messages in thread
From: jcs @ 2019-04-28 14:44 UTC (permalink / raw)
  To: tuhs


Dan Cross <crossd@gmail.com> writes:

> On Sat, Apr 27, 2019 at 10:26 AM Anthony Martin 
> <ality@pbrane.org> wrote:
>
>> From at least V2 to V6, the ls(1) command would not
>> show directory entries whose names began with a '.'
>> unless the -a flag was supplied.

[snip]

>> Was the initial behavior intentional or just a
>> matter of expediency?
>>
>
> I believe it's been publicly stated that it was a mistake in 
> early Unix.
> Apparently, Rob Pike had a Google+ post to this effect back in 
> 2012 (or
> earlier): I see a reference to it from another mailing list 
> around that
> time. Unfortunately, the Google+ content is now lost. Rob, do 
> you have a
> copy?

Here's a tweet that shows the post: 
https://twitter.com/rauchg/status/698689290667053060/photo/1?ref_src=twsrc%5Etfw%7Ctwcamp%5Etweetembed%7Ctwterm%5E698689290667053060

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [TUHS] A question about ls(1)
  2019-04-28 11:47 ` Dan Cross
  2019-04-28 12:00   ` arnold
  2019-04-28 14:44   ` jcs
@ 2019-04-28 16:15   ` Bakul Shah
  2 siblings, 0 replies; 23+ messages in thread
From: Bakul Shah @ 2019-04-28 16:15 UTC (permalink / raw)
  To: Dan Cross; +Cc: The Eunuchs Hysterical Society

On Apr 28, 2019, at 4:47 AM, Dan Cross <crossd@gmail.com> wrote:
> 
> I suspect the "dot file" convention came later, as a side-effect of observed behavior of ls.

This could’ve been avoided if there was a convention about
where to store per user per app settings & possibly state. On
one of my Unix machines I have over 200 dotfiles. 

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [TUHS] A question about ls(1)
@ 2019-04-28 16:54 norman
  2019-04-28 19:45 ` Jon Forrest
  0 siblings, 1 reply; 23+ messages in thread
From: norman @ 2019-04-28 16:54 UTC (permalink / raw)
  To: tuhs

Bakul Shah:

  This could've been avoided if there was a convention about
  where to store per user per app settings & possibly state. On
  one of my Unix machines I have over 200 dotfiles.

====

Some, I think including Ken and Dennis, might have argued
that real UNIX programs aren't complex enough to need
lots of configuration files.

Agree with it or not, that likely explains why the Research
stream never supplied a better convention about where to
store such files.  I do remember some general debate in the
community (e.g. on netnews) about the matter back in the
early 1980s.  One suggestion I recall was to move all the
files to subdirectory `$HOME/...'.  Personally I think
$HOME/conf would have been better (though I lean toward
the view that very few programs should need such files
anyway).

But by then BSD had spread the convention of leaving
`hidden' files in $HOME had spread too far to call
back.  It wouldn't surprise me if some at Berkeley
would rather have moved to a cleaner convention, just
as the silly uucp-baud-rate flags were removed from
wc, but the cat was already out of the bag and too
hard to stuff back in.

On the Ubuntu Linux systems I help run these days, there
is a directory $HOME/.config.  The tree within has 192
directories and 187 regular files.  I have no idea what
all those files are for, but from the names, most are
from programs I may have run once years ago to test
something, or from programs I run occasionally but
have no context I care about saving.  The whole tree
occupies almost six megabytes, which seems small
by current standards, but (as those on this list
certainly know) in the early 1980s it was possible
to run a complete multi-user UNIX system comfortably
from a single 2.5MB RK05 disk.

Norman Wilson
Toronto ON

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [TUHS] A question about ls(1)
  2019-04-28 16:54 norman
@ 2019-04-28 19:45 ` Jon Forrest
  2019-04-28 20:00   ` Bakul Shah
  2019-04-28 22:16   ` Thoma Paulsen
  0 siblings, 2 replies; 23+ messages in thread
From: Jon Forrest @ 2019-04-28 19:45 UTC (permalink / raw)
  To: tuhs



On 4/28/2019 9:54 AM, Norman Wilson wrote:
> Bakul Shah:
> 
>    This could've been avoided if there was a convention about
>    where to store per user per app settings & possibly state. On
>    one of my Unix machines I have over 200 dotfiles.

One of the things I think Windows did right was to have one
place where all configuration information is kept. Sure,
you can argue about the implementation details, but Windows
doesn't have configuration files all over the place.

Jon


^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [TUHS] A question about ls(1)
  2019-04-28 19:45 ` Jon Forrest
@ 2019-04-28 20:00   ` Bakul Shah
  2019-04-29 18:05     ` Michael Kjörling
  2019-04-28 22:16   ` Thoma Paulsen
  1 sibling, 1 reply; 23+ messages in thread
From: Bakul Shah @ 2019-04-28 20:00 UTC (permalink / raw)
  To: TUHS main list


On Apr 28, 2019, at 12:45 PM, Jon Forrest <nobozo@gmail.com> wrote:
> 
> On 4/28/2019 9:54 AM, Norman Wilson wrote:
>> Bakul Shah:
>>   This could've been avoided if there was a convention about
>>   where to store per user per app settings & possibly state. On
>>   one of my Unix machines I have over 200 dotfiles.
> 
> One of the things I think Windows did right was to have one
> place where all configuration information is kept. Sure,
> you can argue about the implementation details, but Windows
> doesn't have configuration files all over the place.

<Rant due to Windows PTSD elided>

IMHO separate files are fine but it would've been nice to
a) have a place other than $HOME to store these files and
b) a standardized plain text format

May still be worth doing.


^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [TUHS] A question about ls(1)
  2019-04-28 19:45 ` Jon Forrest
  2019-04-28 20:00   ` Bakul Shah
@ 2019-04-28 22:16   ` Thoma Paulsen
  2019-04-29  0:53     ` John P. Linderman
  1 sibling, 1 reply; 23+ messages in thread
From: Thoma Paulsen @ 2019-04-28 22:16 UTC (permalink / raw)
  To: Jon Forrest; +Cc: tuhs

Von: Jon Forrest <nobozo@gmail.com>

> One of the things I think Windows did right was to have one
> place where all configuration information is kept. Sure,
> you can argue about the implementation details, but Windows
> doesn't have configuration files all over the place.
>
in theories - yes, in real there are a lot of configuration files all over the place.



^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [TUHS] A question about ls(1)
  2019-04-27 15:38 ` Warner Losh
  2019-04-27 15:42   ` Larry McVoy
@ 2019-04-28 23:59   ` Warner Losh
  1 sibling, 0 replies; 23+ messages in thread
From: Warner Losh @ 2019-04-28 23:59 UTC (permalink / raw)
  To: Anthony Martin; +Cc: The Eunuchs Hysterical Society

[-- Attachment #1: Type: text/plain, Size: 2269 bytes --]

On Sat, Apr 27, 2019 at 9:38 AM Warner Losh <imp@bsdimp.com> wrote:

>
>
> On Sat, Apr 27, 2019 at 8:26 AM Anthony Martin <ality@pbrane.org> wrote:
>
>> From at least V2 to V6, the ls(1) command would not
>> show directory entries whose names began with a '.'
>> unless the -a flag was supplied.
>>
>> This was changed in V7: only the directory entries
>> for "." and ".." would be skipped by default.
>>
>
> gnu ls does not follow this convention. The system V sources one can find
> on the internet have the curious code:
>
> #define DOTSUP  1
> ...
>                         if (aflg==0 && dentry->d_name[0]=='.'
> # ifndef DOTSUP
>                                 && (dentry->d_name[1]=='\0' ||
> dentry->d_name[1]=='.'
>                                 && dentry->d_name[2]=='\0')
> # endif
>                                 )  /* check for directory items '.', '..',
>                                    *  and items without valid inode-number;
>                                    */
>                                 continue;
>
> which is the V7 behavior ifdef'd out.
>

I've confirmed that all the ls.c's that I have from AT&T, apart from
research, do the . is a hidden file thing in ls. All the research things
inherit the V7 behavior (though the V8 sources I found have an
off-by-default ifdef for the BSD behavior). All descendants of SysV that I
could find source to have the V6 behavior, not V7. Both system III and all
revs of System V I could find have an #ifdef for this, and a 1 line change
restore the V7 behavior. Illumos has the BSD semantics and has lost even
the ifdef.

As pointed out later in the thread I was incorrect about 'base on 4.1BSD. I
took a closer look at the sources we have. The kernel has various bits of
BSD included in it starting in V8, but as noted the networking bits seem
odd to my eye. I didn't do a detailed analysis beyond spot checking a few
files. The userland looks more like evolved V7 code with some BSDism
imported rather than a wholesale switch to the BSD versions.

I like the dot is hidden thing. It's simple, elegant, and transports to
other systems well, so long as they are unix or have filesystems that don't
get hung up on 'extension'. That latter bit is likely why many find it...
distasteful.

Warner

>

[-- Attachment #2: Type: text/html, Size: 3620 bytes --]

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [TUHS] A question about ls(1)
  2019-04-28 22:16   ` Thoma Paulsen
@ 2019-04-29  0:53     ` John P. Linderman
  0 siblings, 0 replies; 23+ messages in thread
From: John P. Linderman @ 2019-04-29  0:53 UTC (permalink / raw)
  To: Thoma Paulsen; +Cc: The Unix Heritage Society

[-- Attachment #1: Type: text/plain, Size: 742 bytes --]

I have a deep-seated distrust of global stuff. In C, obviously. But also
for things like git configs, which may differ substantially for different
directories (one for each project). I cannot imagine trying to control such
things in a single place. Seems intensely ugly.


On Sun, Apr 28, 2019 at 6:47 PM Thoma Paulsen <thomas.paulsen@firemail.de>
wrote:

> Von: Jon Forrest <nobozo@gmail.com>
>
> > One of the things I think Windows did right was to have one
> > place where all configuration information is kept. Sure,
> > you can argue about the implementation details, but Windows
> > doesn't have configuration files all over the place.
> >
> in theories - yes, in real there are a lot of configuration files all over
> the place.
>
>
>

[-- Attachment #2: Type: text/html, Size: 1253 bytes --]

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [TUHS] A question about ls(1)
  2019-04-28 20:00   ` Bakul Shah
@ 2019-04-29 18:05     ` Michael Kjörling
  2019-04-29 20:37       ` Warner Losh
                         ` (2 more replies)
  0 siblings, 3 replies; 23+ messages in thread
From: Michael Kjörling @ 2019-04-29 18:05 UTC (permalink / raw)
  To: tuhs

On 28 Apr 2019 13:00 -0700, from bakul@bitblocks.com (Bakul Shah):
> IMHO separate files are fine but it would've been nice to
> a) have a place other than $HOME to store these files and

XDG already does that. At least Norman already mentioned ~/.config in
this thread.

https://www.freedesktop.org/wiki/Software/xdg-user-dirs/

Not sure how common that is on non-Linux systems, but it seems pretty
common on modern Linux distributions.

My workstation Debian system has a staggering 3467 files in that
directory, spread around 444 directories (75 directories directly
under ~/.config). Plus another 142 dot-directories and 66 dotfiles in
~/. Now, ~/.config typically uses multiple files per application, and
at a glance there's some stuff there that could definitely go, but I
still shudder to think of having all of those directly under ~/, so
it's clearly doing _some_ good in that regard.

And that's to not even begin to talk about all the stuff you'll find
in /etc on a modern Linux system.

> b) a standardized plain text format

I'm not sure about that; different applications have very different
needs, and trying to shoehorn one into another would be ugly; quite
possibly even more ugly than just having different formats. Imagine
trying to write mail sorting recipies (think procmail) in a file with
the same format as that of one holding word processor settings or an
image metadata store. I guess that's half-way tolerable on Windows
because next to nobody edits the settings directly anyway, but on a
system where many such files are sometimes, or often, edited directly
by the user, it might well hinder more than it helps. I guess you
_could_ go with something like XML or JSON, but that's a bit like
saying "all cars should have an engine and a refillable fuel store",
in that it doesn't actually standardize anything _meaningful_ (in both
of those cases, the magic is in the schema, not the format). Lists of
examples not intended to be exhaustive.

-- 
Michael Kjörling • https://michael.kjorling.se • michael@kjorling.se
  “The most dangerous thought that you can have as a creative person
              is to think you know what you’re doing.” (Bret Victor)

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [TUHS] A question about ls(1)
  2019-04-29 18:05     ` Michael Kjörling
@ 2019-04-29 20:37       ` Warner Losh
  2019-04-29 20:44       ` Christopher Browne
  2019-04-29 22:32       ` Bakul Shah
  2 siblings, 0 replies; 23+ messages in thread
From: Warner Losh @ 2019-04-29 20:37 UTC (permalink / raw)
  To: Michael Kjörling; +Cc: TUHS main list

[-- Attachment #1: Type: text/plain, Size: 1935 bytes --]

On Mon, Apr 29, 2019 at 12:14 PM Michael Kjörling <michael@kjorling.se>
wrote:

> On 28 Apr 2019 13:00 -0700, from bakul@bitblocks.com (Bakul Shah):
> > b) a standardized plain text format
>
> I'm not sure about that; different applications have very different
> needs, and trying to shoehorn one into another would be ugly; quite
> possibly even more ugly than just having different formats. Imagine
> trying to write mail sorting recipies (think procmail) in a file with
> the same format as that of one holding word processor settings or an
> image metadata store. I guess that's half-way tolerable on Windows
> because next to nobody edits the settings directly anyway, but on a
> system where many such files are sometimes, or often, edited directly
> by the user, it might well hinder more than it helps. I guess you
> _could_ go with something like XML or JSON, but that's a bit like
> saying "all cars should have an engine and a refillable fuel store",
> in that it doesn't actually standardize anything _meaningful_ (in both
> of those cases, the magic is in the schema, not the format). Lists of
> examples not intended to be exhaustive.
>

The only thing that .profile and .Xdefaults share is a leading '.'. While
the latter could be XML or JSON (almost, neither of those formats has
conditional expressions and .Xdefaults is run through cpp), the former
never could be XML or JSON in any sane universe.

So while some config stuff is stored in the dot files, its nature is
somewhat different.

Also, to use git as an example. my repo has a .git/config in it. For work
repos I put my work email and preferred spelling of my name. those go in
repo/.git/config. But for everything else, I have some global settings in
$HOME/.git/config. There are from time to time other reasons to tweak the
settings of a repo and have it be local to that repo only.

It's a tricky problem...

Warner

[-- Attachment #2: Type: text/html, Size: 2470 bytes --]

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [TUHS] A question about ls(1)
  2019-04-29 18:05     ` Michael Kjörling
  2019-04-29 20:37       ` Warner Losh
@ 2019-04-29 20:44       ` Christopher Browne
  2019-04-30  6:44         ` Michael Kjörling
  2019-04-29 22:32       ` Bakul Shah
  2 siblings, 1 reply; 23+ messages in thread
From: Christopher Browne @ 2019-04-29 20:44 UTC (permalink / raw)
  To: Michael Kjörling; +Cc: tuhs

[-- Attachment #1: Type: text/plain, Size: 3554 bytes --]

On Mon, 29 Apr 2019 at 14:15, Michael Kjörling <michael@kjorling.se> wrote:

> On 28 Apr 2019 13:00 -0700, from bakul@bitblocks.com (Bakul Shah):
> > IMHO separate files are fine but it would've been nice to
> > a) have a place other than $HOME to store these files and
>
> XDG already does that. At least Norman already mentioned ~/.config in
> this thread.
>
> https://www.freedesktop.org/wiki/Software/xdg-user-dirs/
>
> Not sure how common that is on non-Linux systems, but it seems pretty
> common on modern Linux distributions.
>

That's application-specific, not distribution or OS specific.

I have 58 subdirectories of ~/.config on my workstation at work; that's not
deviating much from your case; 58 directories (indicating about 58 apps),
2047 subdirectories, 7422 files, 481MB of data, the largest bits being
cache data for Google Chrome web browser.


> My workstation Debian system has a staggering 3467 files in that
> directory, spread around 444 directories (75 directories directly
> under ~/.config). Plus another 142 dot-directories and 66 dotfiles in
> ~/. Now, ~/.config typically uses multiple files per application, and
> at a glance there's some stuff there that could definitely go, but I
> still shudder to think of having all of those directly under ~/, so
> it's clearly doing _some_ good in that regard.
>

I have had little reason to worry about this; I tend to concur.


> > b) a standardized plain text format
>
> I'm not sure about that; different applications have very different
> needs, and trying to shoehorn one into another would be ugly; quite
> possibly even more ugly than just having different formats.


Yeah, the time they tried doing that, we got "XML everywhere," and
that wasn't a notable improvement.

What you see on mobile devices these days is that the ".config/" file
is a SQLite database, which is where both configuration and application
data are captured.

As someone that spends a great deal of his time writing SQL code,
I find that not too heinous; others might disagree.  I think having a
SQLite file is better than getting some heinous XML file.
(Especially if the XML file is some thinly disguised serialization of
a hierarchy of COM objects, where modifying element order would be
liable to make your Windows application blow up.)

The "mobile platforms" have gotten quite a lot of milage out of using
SQLite databases (true both on Android and on iOS); there could
be worse things.

The SQLite web site does have a page somewhat proselytizing
this:  https://sqlite.org/appfileformat.html

I heard D Richard Hipp (author of SQLite, now head of the dev team)
explain this at PGCon 5 years ago <
https://www.pgcon.org/2014/schedule/events/736.en.html>

He also contended that the world might be a better place if LibreOffice
documents were captured as SQLite databases rather than being bundles of
XML stored in a .zip archive.  That's a nice lively argument to have.
Actually poke at the slides at PGCon 2014; he makes similar arguments
about git repos (what if metadata were in a database?) and ePub book
files.

If we were going to forcibly shoehorn everything into one thing, I think I'd
rather that than a lot of the alternatives.  But I'm admittedly excessively
comfortable with SQL ;-)

I can live with us having a number of data formats, particularly if
they remain simple.
-- 
When confronted by a difficult problem, solve it by reducing it to the
question, "How would the Lone Ranger handle this?"

[-- Attachment #2: Type: text/html, Size: 5102 bytes --]

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [TUHS] A question about ls(1)
  2019-04-29 18:05     ` Michael Kjörling
  2019-04-29 20:37       ` Warner Losh
  2019-04-29 20:44       ` Christopher Browne
@ 2019-04-29 22:32       ` Bakul Shah
  2019-04-30 11:52         ` Theodore Ts'o
  2 siblings, 1 reply; 23+ messages in thread
From: Bakul Shah @ 2019-04-29 22:32 UTC (permalink / raw)
  To: tuhs

On Mon, 29 Apr 2019 18:05:12 -0000 Michael =?utf-8?B?S2rDtnJsaW5n?= <michael@kjorling.se> wrote:
Michael Kj?rling writes:
> On 28 Apr 2019 13:00 -0700, from bakul@bitblocks.com (Bakul Shah):
> > IMHO separate files are fine but it would've been nice to
> > a) have a place other than $HOME to store these files and
>
> XDG already does that. At least Norman already mentioned ~/.config in
> this thread.
>
> https://www.freedesktop.org/wiki/Software/xdg-user-dirs/
>
> Not sure how common that is on non-Linux systems, but it seems pretty
> common on modern Linux distributions.

I meant to suggest that a unix wide convention, with an API to
access config data from programs, may still be of some use.
This should contain user specific configuration in a few lines.

mh for example has a single per user config file: .mh_profile.
It allows you to specify your local maildir's path (defaulting
to ~/Mail). All mail messages and additional state/data is then
stored under ~/Mail or its subdirectories.

So something like "path: foo" can be used store application
specific stuff that doesn't fit in the minimal config model.
The goal would be to cover the majority of programs and
provide some guidelines for more complex applications.

> My workstation Debian system has a staggering 3467 files in that
> directory, spread around 444 directories (75 directories directly
> under ~/.config). Plus another 142 dot-directories and 66 dotfiles in
> ~/. Now, ~/.config typically uses multiple files per application, and
> at a glance there's some stuff there that could definitely go, but I
> still shudder to think of having all of those directly under ~/, so
> it's clearly doing _some_ good in that regard.

I suspect most of these files contain some state and cached
application data or content as opposed to configuration.  Try
the following from your home dir:

du -a .??*|cat -n|grep -v / |\
awk 'BEGIN{x=1;} {print $1-x, $3; x=$1;}'|sort -nr|head

This prints out top 10 of total files/dirs under each
top-level dot directory. You can try similar from .config. For
me, most in the top 10 are programs I haven't used in years!

>
> And that's to not even begin to talk about all the stuff you'll find
> in /etc on a modern Linux system.
>
> > b) a standardized plain text format
>
> I'm not sure about that; different applications have very different
> needs, and trying to shoehorn one into another would be ugly; quite
> possibly even more ugly than just having different formats.

Not shoehorn in everything but support a core set. 

> trying to write mail sorting recipies (think procmail) in a file with
> the same format as that of one holding word processor settings or an
> image metadata store.

By "one place" I meant something like a ~/etc or ~/rc or
~/config directory but not a single file. A separate config
file for each program (and library) that needs it.

Ideally I'd separate config, state, content and cache. I find
<program>/{config,state,<content>,cache} to be more modular
than {config,state,<content>,cache}/<program> -- what Apple
forces on Macs. Ex: ~/Library/Preferences/org.tug.TexWorks.plist

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [TUHS] A question about ls(1)
  2019-04-29 20:44       ` Christopher Browne
@ 2019-04-30  6:44         ` Michael Kjörling
  2019-04-30  7:24           ` Kurt H Maier
  0 siblings, 1 reply; 23+ messages in thread
From: Michael Kjörling @ 2019-04-30  6:44 UTC (permalink / raw)
  To: tuhs

On 29 Apr 2019 16:44 -0400, from cbbrowne@gmail.com (Christopher Browne):
> He also contended that the world might be a better place if LibreOffice
> documents were captured as SQLite databases rather than being bundles of
> XML stored in a .zip archive.  That's a nice lively argument to have.
> Actually poke at the slides at PGCon 2014; he makes similar arguments
> about git repos (what if metadata were in a database?) and ePub book
> files.

At least OpenDocument is an ISO standard; and while it looks at a
glance like the Zip file format and compression themselves aren't
specified in it (at eight pages, there isn't a lot of room for
detailed technical descriptions), ISO/IEC 21320-1:2015 "normatively
references the Zip File Format Specification version 6.3.3 of PKWARE®
Inc", stating that "[d]ocument container files are conforming Zip
files as specified by that document". (Quoted from the summary page.)

https://www.iso.org/standard/60101.html

https://standards.iso.org/ittf/PubliclyAvailableStandards/c060101_ISO_IEC_21320-1_2015.zip

I could be wrong, but I don't _think_ that SQLite has reached quite
that level of adoption.

Also, relational databases have their advantages (I work with them
myself), but lots of office-type documents (word processing documents,
spreadsheets, presentations, etc.) inherently have a somewhat
hierarchical or run-on data structure, lending themselves well to a
hierarchical format. Whether we like it or not, XML also has the
advantage of being a well-established standard for data serialization,
and _with a schema_, can be readily validated.

And if you don't like the outer Zip file container, at least
OpenDocument also allows for single flat XML files. (Typical file name
extension .fo[dt][gpst], as opposed to .o[dt][gpst] for the Zip
container counterpart.) I'm pretty sure LibreOffice can be set up to
save as such files even by default, if that's your cup of tea. Those
get awfully big the moment you start including any non-trivial
content, though.

-- 
Michael Kjörling • https://michael.kjorling.se • michael@kjorling.se
  “The most dangerous thought that you can have as a creative person
              is to think you know what you’re doing.” (Bret Victor)

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [TUHS] A question about ls(1)
  2019-04-30  6:44         ` Michael Kjörling
@ 2019-04-30  7:24           ` Kurt H Maier
  2019-04-30 10:35             ` Wesley Parish
  0 siblings, 1 reply; 23+ messages in thread
From: Kurt H Maier @ 2019-04-30  7:24 UTC (permalink / raw)
  To: Michael Kjörling; +Cc: tuhs

On Tue, Apr 30, 2019 at 06:44:08AM +0000, Michael Kjörling wrote:
>
> At least OpenDocument is an ISO standard; 
        
So is "Office Open XML"; a sufficiently large campaign fund is 
indistinguishable from a standards committee.
        
> I could be wrong, but I don't _think_ that SQLite has reached quite
> that level of adoption.
          
Every Android device, iPhone/iPad, Chrome or Firefox installation, 
Windows 10 computer or Red Hat Enterprise Linux installation uses 
SQLite.  Also, the Library of Congress considers it as good as XML as a 
storage format.  It is fairly certain that SQLite is wider-adopted than
OpenDocument -- or at least wider-deployed.
          
I'd argue that SQLite files and XML documents both suffer from the same
problem, in that (during normal use) you have to load the whole thing in
order to use it effectively.  In other words, they don't play well with
pipes.
          
But for a document format, maybe the Unix Nature is less of a concern.
          
khm

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [TUHS] A question about ls(1)
  2019-04-30  7:24           ` Kurt H Maier
@ 2019-04-30 10:35             ` Wesley Parish
  0 siblings, 0 replies; 23+ messages in thread
From: Wesley Parish @ 2019-04-30 10:35 UTC (permalink / raw)
  To: Michael Kjörling, tuhs

I find it ironical; I put some thought into the problem of making a
very small Office Suite a while ago, and concluded that I could make
the text and spreadsheet and database apps dependent on the database
manager foundation, while making the display apps and the layout apps
(for things such as presentation and desktop publishing) dependent on
a web browser layer.

The native file format for such a miniature office suite would've been
the database file format, with options to export to other file
formats. I couldn't see how I could've escaped XML though for
reduction into various display options. It's big, it's not as precise
as TeX, but it's a lot more widely used.

FWVVLIW ...

Wesley Parish

On 4/30/19, Kurt H Maier <khm@sciops.net> wrote:
> On Tue, Apr 30, 2019 at 06:44:08AM +0000, Michael Kjörling wrote:
>>
>> At least OpenDocument is an ISO standard;
>
> So is "Office Open XML"; a sufficiently large campaign fund is
> indistinguishable from a standards committee.
>
>> I could be wrong, but I don't _think_ that SQLite has reached quite
>> that level of adoption.
>
> Every Android device, iPhone/iPad, Chrome or Firefox installation,
> Windows 10 computer or Red Hat Enterprise Linux installation uses
> SQLite.  Also, the Library of Congress considers it as good as XML as a
> storage format.  It is fairly certain that SQLite is wider-adopted than
> OpenDocument -- or at least wider-deployed.
>
> I'd argue that SQLite files and XML documents both suffer from the same
> problem, in that (during normal use) you have to load the whole thing in
> order to use it effectively.  In other words, they don't play well with
> pipes.
>
> But for a document format, maybe the Unix Nature is less of a concern.
>
> khm
>

^ permalink raw reply	[flat|nested] 23+ messages in thread

* Re: [TUHS] A question about ls(1)
  2019-04-29 22:32       ` Bakul Shah
@ 2019-04-30 11:52         ` Theodore Ts'o
  0 siblings, 0 replies; 23+ messages in thread
From: Theodore Ts'o @ 2019-04-30 11:52 UTC (permalink / raw)
  To: Bakul Shah; +Cc: tuhs

On Mon, Apr 29, 2019 at 03:32:19PM -0700, Bakul Shah wrote:
> > My workstation Debian system has a staggering 3467 files in that
> > directory, spread around 444 directories (75 directories directly
> > under ~/.config). Plus another 142 dot-directories and 66 dotfiles in
> > ~/. Now, ~/.config typically uses multiple files per application, and
> > at a glance there's some stuff there that could definitely go, but I
> > still shudder to think of having all of those directly under ~/, so
> > it's clearly doing _some_ good in that regard.
> 
> I suspect most of these files contain some state and cached
> application data or content as opposed to configuration.

Applications which follow the XDG specification (which is what
specified ~/.config are supposed to use ~/.cache for cache files (the
per-user analog of /var/cache) and ~/.local/share for data files (the
per-user analog of /usr/share).  These locations can be overridden by
the environment variables XDG_CACHE_HOME, XDG_DATA_HOME, and
XDG_CONFIG_HOME to override ~/.config.

While I would never under-estimate the ability for application writers
to Get Things Wrong[1], at least in theory there should *not* be state
or cache files stored in ~/.config.

[1] For example, GUI text editors updating precious files in place
using O_TRUNC, as opposed to writing to foo.new, reading the extended
attributes and Posix ACL's from file foo and writing them to foo.new,
calling fsync, and then renaming foo.new to foo --- because The Right
Way is too much trouble for an application author.  Sigh....

Cheers,

						- Ted

^ permalink raw reply	[flat|nested] 23+ messages in thread

end of thread, back to index

Thread overview: 23+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2019-04-27 14:16 [TUHS] A question about ls(1) Anthony Martin
2019-04-27 15:38 ` Warner Losh
2019-04-27 15:42   ` Larry McVoy
2019-04-28 23:59   ` Warner Losh
2019-04-28 11:47 ` Dan Cross
2019-04-28 12:00   ` arnold
2019-04-28 14:44   ` jcs
2019-04-28 16:15   ` Bakul Shah
2019-04-27 20:11 norman
2019-04-27 23:03 jnc
2019-04-28 16:54 norman
2019-04-28 19:45 ` Jon Forrest
2019-04-28 20:00   ` Bakul Shah
2019-04-29 18:05     ` Michael Kjörling
2019-04-29 20:37       ` Warner Losh
2019-04-29 20:44       ` Christopher Browne
2019-04-30  6:44         ` Michael Kjörling
2019-04-30  7:24           ` Kurt H Maier
2019-04-30 10:35             ` Wesley Parish
2019-04-29 22:32       ` Bakul Shah
2019-04-30 11:52         ` Theodore Ts'o
2019-04-28 22:16   ` Thoma Paulsen
2019-04-29  0:53     ` John P. Linderman

The Unix Heritage Society mailing list

Archives are clonable: git clone --mirror http://inbox.vuxu.org/tuhs

Newsgroup available over NNTP:
	nntp://inbox.vuxu.org/vuxu.archive.tuhs


AGPL code for this site: git clone https://public-inbox.org/ public-inbox