man.cgi in "online" mode

man.cgi in "online" mode

[Kristaps Dzonsons]

Hi,

If you're looking at commits and tech@ messages, you'll notice a lot of 
work in apropos(1) and mandocdb(8).  What's not mentioned much is 
man.cgi, which is similar to existing man.cgi scripts but with the power 
of our apropos(1) and mandoc(1) as a renderer.

To date, I've designed man.cgi to run in "secure" and "insecure" mode. 
"Insecure" mode is for non-jailed CGI processes, scanning the system's 
mandocdb(8) (like apropos(1)) and formatting output with mandoc(1).  The 
"secure" mode uses mandocdb(8) databases and a cache of pre-formatted 
manpages, both kept fresh with the new-born manup(8) utility.  One 
triggers insecure mode by passing the $INSECURE variable to the CGI script.

The question is as follows: is an "insecure" mode reasonable?  It's a 
security risk to allow a CGI to fork()/exec() at all, which makes it 
unlikely for public-facing servers.  But it's a hassle to set up and 
maintain a server just for internal manpages.  And do manuals really 
change so much that on-line rendering is necessary?  The "secure" mode 
cache is freshened with a single command invocation, only updating 
what's out of date (quickly, at that).

My thought is to write a Firefox add-in to do this instead, but it seems 
that somebody in discuss@-land may have a better idea.  Or better yet, 
might think this idea awesome enough to do on their own -- all the code 
to interact with mandocdb(8) is already in man.cgi (cgi.c).  The 
benefits are that it kicks some complexity out of man.cgi and lets me 
focus more on a single purpose, instead of spreading out over two.

Of course, this limits to Firefox.  I happen to despise Firefox, but am 
not sure if other solutions beyond a standalone daemon like cupsd.

Thoughts?

Take care,

Kristaps
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv

Re: man.cgi in "online" mode

[Paul Onyschuk]

On Thu, 24 Nov 2011 22:34:02 +0100
Kristaps Dzonsons wrote:

> 
> If you're looking at commits and tech@ messages, you'll notice a lot
> of work in apropos(1) and mandocdb(8).
>

Small offtopic, but web archives linked (e.g. discuss list [1]) at main
page seems to be down for some time (not updated in a while).  Will be
that fixed or there is another place where I can find archives (Gmane
or something like that).

[1] http://mdocml.bsd.lv/archives/discuss/ 
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv

Re: man.cgi in "online" mode

[Kristaps Dzonsons]

On 24/11/2011 23:02, Paul Onyschuk wrote:
> On Thu, 24 Nov 2011 22:34:02 +0100
> Kristaps Dzonsons wrote:
>
>>
>> If you're looking at commits and tech@ messages, you'll notice a lot
>> of work in apropos(1) and mandocdb(8).
>>
>
> Small offtopic, but web archives linked (e.g. discuss list [1]) at main
> page seems to be down for some time (not updated in a while).  Will be
> that fixed or there is another place where I can find archives (Gmane
> or something like that).
>
> [1] http://mdocml.bsd.lv/archives/discuss/

Paul,

Thanks for the reminder.  Indeed, hypermail changed between OpenBSD 
updates and broke the update script.  The spare minutes to fix it 
haven't arisen.  (Does anybody here have any mlmmj-fu?)

Best,

Kristaps
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv

Re: man.cgi in "online" mode

[Kristaps Dzonsons]

On 24/11/2011 23:06, Kristaps Dzonsons wrote:
> On 24/11/2011 23:02, Paul Onyschuk wrote:
>> On Thu, 24 Nov 2011 22:34:02 +0100
>> Kristaps Dzonsons wrote:
>>
>>>
>>> If you're looking at commits and tech@ messages, you'll notice a lot
>>> of work in apropos(1) and mandocdb(8).
>>>
>>
>> Small offtopic, but web archives linked (e.g. discuss list [1]) at main
>> page seems to be down for some time (not updated in a while). Will be
>> that fixed or there is another place where I can find archives (Gmane
>> or something like that).
>>
>> [1] http://mdocml.bsd.lv/archives/discuss/
>
> Paul,
>
> Thanks for the reminder. Indeed, hypermail changed between OpenBSD
> updates and broke the update script. The spare minutes to fix it haven't
> arisen. (Does anybody here have any mlmmj-fu?)

Hm, on second glance, gmane looks like a much much better idea, and 
relieves me of managing the hypermaily stuff myself.  Thanks for the 
suggestion!

Kristaps
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv

Re: man.cgi in "online" mode

[Ingo Schwarze]

Hi Kristaps,

Kristaps Dzonsons wrote on Thu, Nov 24, 2011 at 10:34:02PM +0100:

> If you're looking at commits and tech@ messages, you'll notice a lot
> of work in apropos(1) and mandocdb(8).

Yes, and we'll have to resync, i hope to do a bit of that this weekend,
let's see how far i can get.

> What's not mentioned much is man.cgi, which is similar to existing
> man.cgi scripts but with the power of our apropos(1) and mandoc(1)
> as a renderer.

The biggest problem with that currently is that the interface
is incompatible.  Yes, the old interface is horrible (sektion=,
anyone?), but we can't break existing links.  I hope to work on
that at one point, but one thing at a time, i first have to get
mandocdb(8) and apropos(1) ready for production in base.
Probably, in some way or another, man.cgi will have to support
both the traditional man.cgi syntax and the new apropos(1) syntax.
I haven't thought about the details yet.

> To date, I've designed man.cgi to run in "secure" and "insecure"
> mode.

In general, secure and insecure mode tend to make very little
sense.  You want to have one mode that's secure *and* works,
not two (one securely non-working and one working insecurely).

> "Insecure" mode is for non-jailed CGI processes, scanning the
> system's mandocdb(8) (like apropos(1)) and formatting output with
> mandoc(1).  The "secure" mode uses mandocdb(8) databases and a cache
> of pre-formatted manpages, both kept fresh with the new-born
> manup(8) utility.

The so-called secure mode makes more sense to me, also regarding
efficiency.

If we need to serve from source code, which i'm not yet sure we
really have to, but maybe i'm missing something, then i'd probably
try to just link both the parsers and the -Thtml formatter right
into the man.cgi executable.  Not sure whether that would fly, but
it might be worth a try.
Of course, it's not urgent.  First, what matters is a system
that serves the manuals at all, with the right interface.

> One triggers insecure mode by passing the
> $INSECURE variable to the CGI script.
> 
> The question is as follows: is an "insecure" mode reasonable?  It's
> a security risk to allow a CGI to fork()/exec() at all, which makes
> it unlikely for public-facing servers.  But it's a hassle to set up
> and maintain a server just for internal manpages.  And do manuals
> really change so much that on-line rendering is necessary?  The
> "secure" mode cache is freshened with a single command invocation,
> only updating what's out of date (quickly, at that).

I tend to think that the so-called secure mode is sufficient
and running one command for setup seems acceptable (i don't like
the name "manup", but that's not a crucial point).  Ultimately,
i think giving a working, compatible system and setup instruction
to Bob Beck and seeing what happens will be the test to see whether
what we built is usable, and only when that has run for some time
in public and -current snapshots have been sucessfully installed
for a few times (without Bob complaining about having to do tedious
work) should we consider the design finished and proven good.
Until then, we should be ready to implement tweaks that practice
might teach us.

> My thought is to write a Firefox add-in to do this instead,

I think this has nothing to do with browsers at all, and
writing Firefox plugins is a complete waste of time no matter
what the purpose might be.

Yours,
  Ingo
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv

Re: man.cgi in "online" mode

[Kristaps Dzonsons]

> Yes, and we'll have to resync, i hope to do a bit of that this weekend,
> let's see how far i can get.

Ingo,

Keep me posted.  I've made some changes here and there, but 
directory-diff'ing between OpenBSD's and mdocml's doesn't cause too many 
issues.  Your nice ignore-certain-files bits are still unmerged (I'd 
noted on tech@) and I don't see much beyond that.

>> What's not mentioned much is man.cgi, which is similar to existing
>> man.cgi scripts but with the power of our apropos(1) and mandoc(1)
>> as a renderer.
>
> The biggest problem with that currently is that the interface
> is incompatible.  Yes, the old interface is horrible (sektion=,
> anyone?), but we can't break existing links.  I hope to work on
> that at one point, but one thing at a time, i first have to get
> mandocdb(8) and apropos(1) ready for production in base.
> Probably, in some way or another, man.cgi will have to support
> both the traditional man.cgi syntax and the new apropos(1) syntax.
> I haven't thought about the details yet.

Compat isn't difficult -- I've designed man.cgi with an eye for this 
support (ok, I didn't, but it naturally grew that way).  The only 
headache part is the old manuals, but let's talk about that later.

>> To date, I've designed man.cgi to run in "secure" and "insecure"
>> mode.
>
> In general, secure and insecure mode tend to make very little
> sense.  You want to have one mode that's secure *and* works,
> not two (one securely non-working and one working insecurely).
>
>> "Insecure" mode is for non-jailed CGI processes, scanning the
>> system's mandocdb(8) (like apropos(1)) and formatting output with
>> mandoc(1).  The "secure" mode uses mandocdb(8) databases and a cache
>> of pre-formatted manpages, both kept fresh with the new-born
>> manup(8) utility.
>
> The so-called secure mode makes more sense to me, also regarding
> efficiency.
>
> If we need to serve from source code, which i'm not yet sure we
> really have to, but maybe i'm missing something, then i'd probably
> try to just link both the parsers and the -Thtml formatter right
> into the man.cgi executable.  Not sure whether that would fly, but
> it might be worth a try.
> Of course, it's not urgent.  First, what matters is a system
> that serves the manuals at all, with the right interface.

This (linking man.cgi with the mandoc(1) frontends) was my first 
thought, and in thinking about it twice, may prove the winner in the 
end.  I opted for fork/exec because it took fewer keystrokes at the 
time.  But manup(8) (or whatever the name) can be trivially fitted to 
copy in manual sources, at which point both "modes" would do the same 
thing.  Yes, I like it.

In fact, I think I'll slather some elbow-grease over this idea and see 
what comes of it.

> I tend to think that the so-called secure mode is sufficient
> and running one command for setup seems acceptable (i don't like
> the name "manup", but that's not a crucial point).  Ultimately,
> i think giving a working, compatible system and setup instruction
> to Bob Beck and seeing what happens will be the test to see whether
> what we built is usable, and only when that has run for some time
> in public and -current snapshots have been sucessfully installed
> for a few times (without Bob complaining about having to do tedious
> work) should we consider the design finished and proven good.
> Until then, we should be ready to implement tweaks that practice
> might teach us.

I agree.  Again, the "old versions" bothers me, but I don't really have 
any idea how that's maintained even now.  I'm guessing the man.cgi 
server has a forest of manpage trees in there somewhere.  Ugh.

>> My thought is to write a Firefox add-in to do this instead,
>
> I think this has nothing to do with browsers at all, and
> writing Firefox plugins is a complete waste of time no matter
> what the purpose might be.

Perhaps, although I really like the idea of a no-server, browser-based 
interface.  But just using Firefox is painful enough: writing a plugin 
for it might damage me forever!  Think of the wee manuals!

Take care,

Kristaps
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv

Re: man.cgi in "online" mode

[Ingo Schwarze]

Hi Kristaps,

agreeing with and snipping most of what you said,
let me just try to alleviate one headache:

Kristaps Dzonsons wrote on Fri, Nov 25, 2011 at 12:15:00AM +0100:

> The only headache part is the old manuals, but let's talk about
> that later.

Oh, the old manuals are not compulsory.
The idea is to serve 5.0 and higher, including -current,
with the new man.cgi and continue to serve the history archive
with the old script, at least for now.
In case we ever get bored, we might rethink that later,
but I agreed on this approch with Bob and Theo in Ljubljana.

> Again, the "old versions" bothers me, but I don't really have any
> idea how that's maintained even now.

Poorly.
By letting it rot.  ;-/

> I'm guessing the man.cgi server has a forest of manpage trees in
> there somewhere.

Yes, that is my understanding as well, and that is not going to
change: The new man.cgi will continue to need one tree per
release, and one additional -current tree that is replaced
by a new snapshot now and then.

Yours,
  Ingo
--
 To unsubscribe send an email to discuss+unsubscribe@mdocml.bsd.lv