Re: [TUHS] mh/hm, mmh (was: fmt(1): history, POSIX, -t, -c)

Re: [TUHS] mh/hm, mmh (was: fmt(1): history, POSIX, -t, -c)

[Bakul Shah]

> Would you be so kind to explain a bit about the hm version of MH.

Not sure what Clem meant but I used "hm" by Jim Guyton (@Rand) on my
Fortune box until I retired it. It provided a 2D interface. Later I
tried xmh but didn't like it.

The early history of MH is covered in some detail in Willis Ware's "RAND
and the Information Evolution" book from page 128 onward. hm & Guyton
get a paragraph on page 136.

https://www.rand.org/content/dam/rand/pubs/corporate_pubs/2008/RAND_CP537.pdf

Re: [TUHS] mh/hm, mmh (was: fmt(1): history, POSIX, -t, -c)

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 425 bytes --]

On Tue, Jun 2, 2020 at 8:43 PM Bakul Shah <bakul@iitbombay.org> wrote:

> > Would you be so kind to explain a bit about the hm version of MH.
>
> Not sure what Clem meant but I used "hm" by Jim Guyton (@Rand) on my
> Fortune box until I retired it. It provided a 2D interface.
>
Yep, that would be the program ..   Jim did a great job with it.
It was built using a bunch of MH programs behind it, and was a curses front
end.

[-- Attachment #2: Type: text/html, Size: 1006 bytes --]

Re: [TUHS] mh/hm, mmh (was: fmt(1): history, POSIX, -t, -c)

[Jon Steinhart]

Ralph Corderoy writes:
> ...
>
> (I still use nmh and I'm using it for this reply.)

+1 on that.  I also still use nmh.  To me, the set of shell commands
exemplifies the UNIX philosophy as I can easily "show | wc -l" or
whatever that is cumbersome with monolithic packages.  My minor
contributions were adding the attachment support and the hooks.  The
hooks allow other programs to be triggered behind the scenes when
nmh commands are run; I use it to keep an elasticsearch database of
my messages.  I have two additional command script to support thise:
gpick and gscan which are the equivalent of pick and scan except that
they search the elasticsearch database and pretty much return results
instantaneously.

Jon

Re: [TUHS] mh/hm, mmh (was: fmt(1): history, POSIX, -t, -c)

[Ralph Corderoy]

Hi Clem,

> I believe it was Bruce Borden and team developed the original Rand
> Message Handler or MH.  I do not know who was the primary author, we
> need to ask Bruce or one of the old Rand folks.

https://en.wikipedia.org/wiki/MH_Message_Handling_System#History says

    MH was proposed by R. Stockton Gaines and Norman Shapiro and
    developed by Bruce S. Borden and others at RAND Corporation.

and cites
https://www.rand.org/content/dam/rand/pubs/notes/2009/N3017.pdf

Norman Shapiro, or Norm, is still using nmh well into his eighties, nmh
is a descendant of MH, and pops up on the nmh-workers mailing list to
either fill in bits of history or ask for installation advice as the
Linux distro shifts under his feet.  He did visit Bell Labs, though in
'54.  :-)  Details at https://en.wikipedia.org/wiki/Norman_Shapiro

> The new format, used a line a control-As followed by a nl, before and
> after the message.

MH's mts.conf(5) had the ability to set mmdelim{1,2} to something other
than \ca\ca\ca\ca\n; no idea why.  And that magic marker would be
escaped if it occurred in the middle of a message, to \cb\ca\ca\ca\n.

> Note the part of the story I left out was when Rand MH spun out to one
> of the other Universities, UCI I think.

Wikipedia again:

    Subsequently development was taken over by Marshall T. Rose and John
    L. Romine working at University of California, Irvine.

    nmh... was forked [from that] by Richard Coleman while working at
    the Georgia Institute of Technology.  It incorporates the "LBL
    changes" made to MH in the late 80s by Van Jacobson, Mike Karels and
    Craig Leres.

MH also had support for UCI's bulletin boards called ‘BBoards’,
including on ZOTnet.  They were optionally available over POP3 and NNTP,
but only one or the other.  :-)  There was an MH shell of some kind
called msh, and a visual mh, vmh.

ZOTnet was a LAN for mail by Marshall Rose at UCI with gateways for
CSnet and ARPAnet.

    https://escholarship.org/content/qt4bm4k2vh/qt4bm4k2vh.pdf
    The ZOTnet: a local area mailing network, 1983

(I still use nmh and I'm using it for this reply.)

-- 
Cheers, Ralph.

Re: [TUHS] mh/hm, mmh (was: fmt(1): history, POSIX, -t, -c)

[Larry McVoy]

On Sun, May 31, 2020 at 12:25:47PM -0400, Richard Salz wrote:
> Originally ihnp4!mirror!rs@seismo.arpa, then ihnp4!mirror!rs@seismo.css.gov
> and then rs%mirror.uucp@seismo.css.gov and then rs@mirror.tmc.com  Thanks
> to Mary Ann and the "UUCP Mapping Project" for making much of that possible
> and Peter Honeyman for Pathlias

My first real research paper was with Udi Manber and it was all about 
compressing the pathalias maps.  I had them on a 40MB disk on a Masscomp
that had 20 users.  The format was

seismo.css.gov	geowhiz!geophiz!uwisc!ihnp4!mirror

in other words

destination	path!to!get!there

I went to Udi and explained that all of our stuff started with
geowhiz!geophiz!uwisc
so that string was replicated over and over and it took too much
space.

Once Udi understood the problem he went "You have time best case,
space worst case, you can reverse them like so"

seismo	-> mirror
mirror	-> ihnp4
ihnp4	-> uwisc
uswisc	-> geophiz
geophiz	-> geowhiz

and you'll have time worst case and space best case.  I was like "oh,
yeah, you are right, cool, thanks" and got up to leave.  Not so fast,
says Udi, that's not "interesting".  What would be interesting is if
you could approximate time best case and space best case.

With that, I started down the only dynamic programming problem I have
ever solved.  Udi's idea was to break the graph at what he called
pivot points, where the pivot points as much as possible got rid of
the replicated strings.  He also wanted to limit lookups to just
2.

So I had to read in the maps and build a graph from them on my
creaky microvax with 4MB of ram.  Doing so lead me to the first
place I taught Udi about systems.  I was fgets() each line and 
mallocing a copy of it.  Then I sorted with qsort() and it took
overnight to get the sorted array.  Udi asked what sort?  Qsort.
Oh, that's garbage, use a radix sort, that will be better.  WTF
is a radix sort, look that up, implemented it, it was worse.

So I start poking around and somehow realized we were thrashing
VM, we didn't fit, and I have no idea how I figured out that
malloc was fragmenting memory but I did.  Probably figured out
that most of my program was backed by swap, not files, and that's
usually malloc or brk.

So I wrote my own malloc that grabbed memory in 300KB chunks (why
that size, I dunno, seemed right).  Then I allocated memory out
of those chunks.  Got rid of the fragmentation and we fit in
4MB.  qsort worked fine, took about 20 minutes (we didn't really
fit but we were way closer).

Udi was watching as I was doing all this, I think I was working
in his office, it was his microvax.  The fragmentation thing 
taught him that when Unix says it has virtual memory and you
can just use it, it only works well when you fit.

So the programming problem turned out that you calculated the 
space it work take for every possible pivot point and then at
the end, you took the set that gave you the minimum space.  We
wrote, heh, who am I kidding, Udi wrote, a paper about it.  I
was still green on writing papers, they were hard for me.  I told
Udi that and he stared at me blankly and said "Writing papers is
easy if you know the content.  You just make a good outline and
fill it in."  He's right, that's how you do it, and it is easy
if you know and can make the outline.

I went on to do a lot of work with/for him.  I wrote a user level
thread library that made me write swtch() in assembler for the VAX
(I believe I did a 68K version too but the source I have is VAX
only).  I "cheated" and did 99% of the work in C and popped down
the rabbit hole to assembler, did the nasty, and popped back out
as a different thread.  112 lines:

http://www.mcvoy.com/lm/T/src/Tasm.S

Funny Udi story, I was young and wild and drove too fast and got 
tickets.  So I got a radar detector that beeped when it saw the
radar gun.  Udi upgraded the microvax to some newer release of
X10 or X11 and the new xclock beeped on the hour.  Every time
it beeped, I jumped because it sounded just like the radar 
detector.  Udi asked about it, I explained, he laughed and said
"Larry, you're hacking too fast" each time I jumped :)

Good times.  I almost did a PhD under him but I didn't care for
the CS department at Tucson where he went and I liked industry
too much.

Re: [TUHS] mh/hm, mmh (was: fmt(1): history, POSIX, -t, -c)

[Richard Salz]

[-- Attachment #1: Type: text/plain, Size: 1538 bytes --]

"I'll try"  Wow, that was quite some "try"  Kudo's!

> This is important because the original ARPAnet NCP used FTP to do mail
transfer.

If you look at RFC 765 (dated June 1980) , there are nearly a dozen
mail-related FTP commands. By RFC 959, it's successor five years later, all
of them were gone.

Another eary MTA was the MMDF, the Multichannel Memorandum Distribution
Facility, from U Delware.  BBN ran it.  One key point is that somehow when
used with MH you could get real-time address verification before sending,
"user rs@bbn.com doesn't exist." BBN ran MMDF for a long time because a key
exec liked/needed that feature, long after 4.2 and sendmail. (My group was
one of the first to run sendmail, which I liked because of the "R$" lines
in its CF files.  (Not really :) My Usenet/email gateway code had to
support MMDF but it was only ever used on bbn.com)

>  could recognize ArpaNET address postfix and had a hack in it, that
allowed the 'user' part of the address to include UUCP addresses

Originally ihnp4!mirror!rs@seismo.arpa, then ihnp4!mirror!rs@seismo.css.gov
and then rs%mirror.uucp@seismo.css.gov and then rs@mirror.tmc.com  Thanks
to Mary Ann and the "UUCP Mapping Project" for making much of that possible
and Peter Honeyman for Pathlias, and MX records for the last part. The
pre-MX styles received much scorn from Research, and if you search for
"that hideous name" you can find a paper on it. There's also a pathlias
paper, which Honey later said "was too good for Usenix" but he had nowhere
else to submit it.

[-- Attachment #2: Type: text/html, Size: 2834 bytes --]

Re: [TUHS] mh/hm, mmh (was: fmt(1): history, POSIX, -t, -c)

[Clem Cole]

[-- Attachment #1: Type: text/plain, Size: 7241 bytes --]

On Sun, May 31, 2020 at 9:02 AM markus schnalke <meillo@marmaro.de> wrote:

> Would you be so kind to explain a bit about the hm version of MH.
>
I can try ...

At the time V6 (and later V7) had a program called /bin/mail.  The
important thing about this program is that it was both a Mail Transport
(and delivery) Agent as well as a Mail User Interface.   The original
version only delivered mail locally.  When networking, like UUCP, was added
to UNIX, it was easy to build a scheme that could send a transfer a message
using UUCP and then fork /bin/mail on the proper system to deliver it.
IIRC it was in Seventh Edition that /bin/mail was extended to recognize
email addresses in the prefix form of:  host!user, strip the first host!
part and pass it to uucp.

I never used it on anything like Spider, so I have not idea what it did
internally and in fact, the prefix address form could have predated UUCP.
We would need someone like Doug, Steve or Ken to tell us how email worked
host to host, pre-UUCP.

I believe it was Bruce Borden and team developed the original Rand Message
Handler or MH.  I do not know who was the primary author, we need to ask
Bruce or one of the old Rand folks.  MH ran on 6th edition when it was
first released to rest of the USENIX community (at least I never saw it on
V5, but it's possible it went back further).   I just remember talking to
Bruce about it at any early USENIX.

MH used a new Mailbox (on-disk) scheme and formatted all messages in RFC733
form with addresses being flat an in the form: user@host. The new format,
used a line a control-As followed by a nl, before and after the message.
 The headers of course where RFC733 (type: value with a trailing nl) and
were separated from the body of the message by a single nl.  MH assumed a
traditional UNIX command line for the user interface and had a number of
programs that ran behind the scene for delivery.   This is important
because the original ARPAnet NCP used FTP to do mail transfer.  At least
one version could call the early ARPAnet subsystem to perform host to host
communications.   The key point is that MH separated the MTA and MUI.

A number of us ran MH at different places.  I don't remember if it was on a
USENIX tape or I had sent Bruce a tape @ Rand when I got a copy in the late
1970's (77-79 timeframe, I've forgotten).  I'm pretty sure Goble, as did
the Purdue crew ran it, as did Holmgrem, Greg Chesson at al, @ UofI and I
would not surprised if it was the mailer at Harvard, given the Harvard/Rand
connection in those days.

By the time of Seventh Edition of UNIX, the Mail Transport Agent (MTA) that
was part of the MH subsystem, could recognize ArpaNET address postfix and
had a hack in it, that allowed the 'user' part of the address to include
UUCP addresses and /bin/mail replacement knew to work like the AT&T mailer
and pass it off to UUCP.   So, at that point, life was good for those of us
in strickly ARPA and/or UUCP land.

BTW: At some point, the BBN TCP code was releases and a separate SMTPD was
included in the release.   I don't remember if it was the Rand folks or
someone else, but at some point, the MH delivery agent was updated to call
it or be called by it.  Similarly, by the late 1970s, when Bruce, Greg
Shaw, and Bob Metcalfe formed 3Com; Bruce and Greg brought MH with them and
added an SMTPD that they wrote for their commercial product, UNET.   This
was the smtpd, I ran on the Teklabs machine before I went off to UCB.

Meanwhile, as Mary Ann and I noted in earlier messages, Kurt Shoen's wrote
a different MTA called Mail(1ucb).   Unlike MH, Kurt continues to support
the original UNIX mailbox format (later named 'mbox' format).   The header
lines were in a specific UNIX style prose starting with the ACSII 'From'
and there are no special characters to demark the messages in the format,
so recovery can be fraught with error, there are the famous From-line
munging issues etc. (lots of details in other places start with
https://en.wikipedia.org/wiki/Mbox).

At some point in time, UCB built the 'Berk-net' (whose original code was
written by ABC/Google's Eric Schmidt).  BTW: Eric would have seen the
Spider Network in his summers at BTL.  The key thing with Berknet was
cheap.  It ran over 3 wire RS-232C between hosts at 9600 baud.   Like UUCP
was used to transfer files and email.    Like UUCP it used a
pre-fix addressing form: host:user ; but like RFC733 and unlike UUCP was
flat.

Where Mary Ann and I differ in our memories is who wrote the original
version of UCB's delivermail (8ucb) program.   We both agree that it is
possible it was Eric Schmidt, as the switch to using delivermail(8ucb) was
were Berknet was spliced into the email namespace.  I had thought Kurt
wrote it, Mary Ann thought it was Eric Allman.   We agree Eric Allman was
hacking on it for the Ing70.  For this response, it doesn't really matter
other than to try to get the history right, because it does not matter for
the Rand Mail subsystem.

Around, now I arrive at UCB.   I was not the only person that had used MH,
but I had the advantage of having 3 Vaxen 780 in the CAD lab.   The key
change we made at that point was to stop using the Rand MTA and make it use
delivermail.   There was hackery to allow the shared mbox to say, but in
each user directory it stored the messages separately, MH style so the MH
'ui' suite of tools 'just worked.'   [There were a number of arguments at
the time.  I remember having one with Sam.  He hated MH because 'of all the
small files and it chewed up inodes].

Anyway around this time, the curses library was created by Ken Arnold
(originally to support Rogue) by pulling the screen code out of vi and
using Mary Ann's termcap stuff.   A couple of us in the USENIX community
started playing with screen-based front ends to MH, including the folks at
Rand.  Numerous messages were exchanged, and a collaboration started (Rand
team should get 99% percent of the work, I knew how to make curses go).
 The comment in the main code was, 'Well it sure isn't MH' -- so it was
called HM.   I later brought it Masscomp, and it was the start of their
mailer.

The key with HM, is that it uses the MH backend for all the real work.   It
would fork scan, but save the output in a text file that could be
manipulated with the arrow keys.   The user interface is not unlike Gmail.
I only stopped running HM when I finally switched to Gmail a few years ago.

In fact, when I was a DEC, the precursor to Gmail, was called
Pachyderm which used the Altavista search.  I glued MH to Pachyderm for a
short period.  I even ported them both to FreeBSD to run at home in the
mid-1990s.  Then   I left DEC, the authors of Pachyderm headed for Google,
*etc*...  I got a chance to be an early Gmail/Google Apps tester for ccc.com
and I have not switched back since.

FWIW: It's been long enough, that the sources are no longer easy to find.

It's all very incestuous.    We all have seen other ideas and good ideas
seem to have a way of reappearing in different places.

Clem

Note the part of the story I left out was when Rand MH spun out to one of
the other Universities, UCI I think.  I was never much involved with that
team, so I can not tell you much about it and their code base.

[-- Attachment #2: Type: text/html, Size: 12458 bytes --]

Re: [TUHS] mh/hm, mmh (was: fmt(1): history, POSIX, -t, -c)

[markus schnalke]

Hoi.

[2020-05-28 09:46] Clem Cole <clemc@ccc.com>
> 
> [...] until
> I finally switched from mh (actually the hm version) to the Gmail interface as
> my MUI client. 

Would you be so kind to explain a bit about the hm version of MH.


Ten years ago I wanted to improve nmh, because I found it bad that it took
me months to configure it in so many ways to get it usable for modern
emailing. Even at that point I hadn't found some of its cool features, which
all were deactivated by default. I argued but couldn't convince the nmh
community. Later I used my master's thesis as the opportunity to create an
experimental version of nmh, to convince by demonstration.

Have a look at my master's thesis, if you like:
	http://marmaro.de/docs/master/

Actually it became a fork, now named mmh. The project's still active.
Especially Philipp Takacs has done a lot, among that replacing m_getfld(),
a highly optimized mail reading function. See the pre-mmh version of it
for an entertaining read:
	http://git.marmaro.de/?p=mmh;a=blob;f=docs/m_getfld.c.humor;h=46449095d

This is our replacement:
	http://git.marmaro.de/?p=mmh;a=blob;f=sbr/m_getfld2.c;h=b9a618d16



I'm much interested in any MH background. Shockingly I cannot recall having
read about hm before ...


meillo