My thoughts on the Gnus source code

My thoughts on the Gnus source code

[Nikolaus Rath]

Hello,

A little while ago I did some first-time hacking on Gnus, and then felt
the urgent need to tell the world about my thoughts. I've now finally
found the time to write things down nicely. In case you are curious,
please take a look at:

http://www.rath.org/whats-wrong-with-gnus.html

Feedback is very welcome. While I did not particularly like the
codebase, I do not want to state anything that's untrue.


Best,
-Nikolaus

(No Cc on replies please, I'm reading the list)
-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: My thoughts on the Gnus source code

[Eric Abrahamsen]

Nikolaus Rath <Nikolaus@rath.org> writes:

> Hello,
>
> A little while ago I did some first-time hacking on Gnus, and then felt
> the urgent need to tell the world about my thoughts. I've now finally
> found the time to write things down nicely. In case you are curious,
> please take a look at:
>
> http://www.rath.org/whats-wrong-with-gnus.html
>
> Feedback is very welcome. While I did not particularly like the
> codebase, I do not want to state anything that's untrue.

Hi Nikolaus,

Interesting read! And one that I'm mostly sympathetic with. But a few
responses:

1. I don't think there's anything inherently unsuitable about Elisp for
   writing a MUA (or any other type of software). It's a general
   programming language, a little different from Python or other more
   popular languages, but it comes with a full complement of modern
   programming features.
2. Gnus doesn't use many of those programming features. I think in part
   because it's very, very old (in computer-program years), and possibly
   because it was written in a fairly ad-hoc fashion, and then tweaked.
   This is definitely a problem.
3. I don't think the buffer issue is a real problem. You're getting
   responses from a server as a string, and there isn't really any
   difference between a buffer and a string: you'll need to chomp it, no
   matter what. I understand there are such things out there as Real
   Parsers, and that they're difficult to write, and that maybe we could
   benefit by having one. But I don't think the buffer itself is the
   problem.
4. Type checking and error handling is a real issue. I feel like half
   the errors I get from Gnus are (wrong-type-argument stringp nil).
   Then you look at the backtrace and the argument was already nil like
   six function calls up. But here again, the problem isn't that Elisp
   doesn't provide the tools, but that Gnus doesn't use them. We can and
   should be defining our own error types, and handling them.

Type checking is more interesting. Elisp doesn't do a lot of this by
default, but some of the newer tools can serve the purpose. I'll bet we
could make some interesting use of pattern-matching with pcase.

But I still think (I've been obsessed with this for a bit now) that
shifting to proper classes with EIEIO is the way to go. We could do away
with the awe and horror that is nnoo.el, and prevent servers from
"infecting" each other's variables and method calls. Generic functions
are another way of doing type checking: if a method's specialization
calls for arguments of a server, a string, and a symbol, you're not
getting anywhere unless that's what they are.

Now that Lars is doing some spring cleaning, I do hope we can start
making use of some of the more modern tools.

On a side note, I'd also hope we can start treating code like the below
as a bug. It may do what it's supposed to do, but...

--8<---------------cut here---------------start------------->8---
(while (and (cdr range)
		    (>= (car active)
			(or (and (atom (cadr range)) (cadr range))
			    (caadr range))))
	  (if (numberp (car range))
	      (setcar range
		      (cons (car range)
			    (or (and (numberp (cadr range))
				     (cadr range))
				(cdadr range))))
	    (setcdr (car range)
		    (or (and (numberp (nth 1 range)) (nth 1 range))
			(cdadr range))))
	  (setcdr range (cddr range)))
--8<---------------cut here---------------end--------------->8---

Anyway, thanks for the blog post! I hope you stick with Gnus, and send
more patches :)

E

Re: My thoughts on the Gnus source code

[Random832]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:
> 1. I don't think there's anything inherently unsuitable about Elisp for
>    writing a MUA (or any other type of software). It's a general
>    programming language, a little different from Python or other more
>    popular languages, but it comes with a full complement of modern
>    programming features.

The real problem isn't Lisp as a language, it's the lack of
multithreading or asynchronous constructs (I don't know if Emacs has the
latter, but if it does, Gnus doesn't seem to use it - it certainly
doesn't have the former.)

As a *user*, the biggest usability problem I've had with Gnus is the
inability to easily recover from dropped connections. The fact that it
tries to hold the connections open long-term in the first place is
another possible cause of this, and someone gave me a setting to try to
make it do this less, but I didn't see any improvement.

> 2. Gnus doesn't use many of those programming features. I think in part
>    because it's very, very old (in computer-program years), and possibly
>    because it was written in a fairly ad-hoc fashion, and then tweaked.
>    This is definitely a problem.
> 3. I don't think the buffer issue is a real problem. You're getting
>    responses from a server as a string, and there isn't really any
>    difference between a buffer and a string: you'll need to chomp it, no
>    matter what. I understand there are such things out there as Real
>    Parsers, and that they're difficult to write, and that maybe we could
>    benefit by having one. But I don't think the buffer itself is the
>    problem.

I do get occasional "selecting deleted buffer" errors (generally a
symptom of having hit C-g at the wrong moment)

> On a side note, I'd also hope we can start treating code like the below
> as a bug. It may do what it's supposed to do, but...

I think that A: any such code should be heavily commented, and B: it
should use let variables. I'm not very experienced with Elisp - is there
an "unpack/let" construct? Is that what pcase is supposed to be for?

(You know what might be nice? nth0, nth1, nth2, etc, as aliases for car,
cadr, etc. Or maybe 0th/1st/2nd...)

Re: My thoughts on the Gnus source code

[Sebastian Christ]

>>>>> On Sun, 14 Feb 2016 11:33:50 -0500, Random832 <random832@fastmail.com> said:
  > (You know what might be nice? nth0, nth1, nth2, etc, as aliases for car,
  > cadr, etc. Or maybe 0th/1st/2nd...)

You can use cl.el's first, second, etc. (or better cl-lib's cl-first,
cl-second, etc.)

Regards,
Sebastian

-- 
Sebastian (Rudolfo) Christ
http://rudolfochrist.github.io
GPG Fingerprint: 306D 8FD3 DFB6 4E44 5061
                 CE71 6407 D6F8 2AC5 55DD

Re: My thoughts on the Gnus source code

[Eric Abrahamsen]

Random832 <random832@fastmail.com> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>> 1. I don't think there's anything inherently unsuitable about Elisp for
>>    writing a MUA (or any other type of software). It's a general
>>    programming language, a little different from Python or other more
>>    popular languages, but it comes with a full complement of modern
>>    programming features.
>
> The real problem isn't Lisp as a language, it's the lack of
> multithreading or asynchronous constructs (I don't know if Emacs has the
> latter, but if it does, Gnus doesn't seem to use it - it certainly
> doesn't have the former.)
>
> As a *user*, the biggest usability problem I've had with Gnus is the
> inability to easily recover from dropped connections. The fact that it
> tries to hold the connections open long-term in the first place is
> another possible cause of this, and someone gave me a setting to try to
> make it do this less, but I didn't see any improvement.

That's definitely true! I'm in China, and the Great Firewall froths at
the mouth about long-running TLS connections. Recovering from bum
connections would be a great area for improvement, but unfortunately one
I'm not capable of coding or even really commenting about. I don't know
much about network programming.

[...]

>> On a side note, I'd also hope we can start treating code like the below
>> as a bug. It may do what it's supposed to do, but...
>
> I think that A: any such code should be heavily commented, and B: it
> should use let variables. I'm not very experienced with Elisp - is there
> an "unpack/let" construct? Is that what pcase is supposed to be for?

Yes, I think it would just be a matter of helper macros, lets, etc. This
isn't quite what pcase is for -- that'as more for matching input (sort
of a cond on steroids), whereas what we need are functions for
manipulating complicated/nested list structures.

E

Re: My thoughts on the Gnus source code

[Lars Ingebrigtsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> Yes, I think it would just be a matter of helper macros, lets, etc. This
> isn't quite what pcase is for -- that'as more for matching input (sort
> of a cond on steroids), whereas what we need are functions for
> manipulating complicated/nested list structures.

That was a function for manipulating complicated/nested list
structures.  Just avert your eyes and use the resulting function.  Some
things are not meant to be beheld by innocent eyes.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: My thoughts on the Gnus source code

[Eric Abrahamsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> Yes, I think it would just be a matter of helper macros, lets, etc. This
>> isn't quite what pcase is for -- that'as more for matching input (sort
>> of a cond on steroids), whereas what we need are functions for
>> manipulating complicated/nested list structures.
>
> That was a function for manipulating complicated/nested list
> structures.

I could argue it was a function for getting unread articles in a group.
:)

> Just avert your eyes and use the resulting function. Some things are
> not meant to be beheld by innocent eyes.

It just comes down to how many people are going to be able to pitch in
with bugfixes and enhancements. We mere mortals may need a lower barrier
to entry!

Anyway, something for later...

Re: My thoughts on the Gnus source code

[Lars Ingebrigtsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> I could argue it was a function for getting unread articles in a group.
> :)

Oh, I though the code fragment was from the gnus-range.el library...

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: My thoughts on the Gnus source code

[Eric Abrahamsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> I could argue it was a function for getting unread articles in a group.
>> :)
>
> Oh, I though the code fragment was from the gnus-range.el library...

Nope, from gnus-get-unread-articles-in-group, one of the first places
people will look when their unread counts go wonky. If there were
utility functions in gnus-range.el that could be used there, that would
at least give the rest of us a fighting chance to figure out what's
going on, that would be great...

Re: My thoughts on the Gnus source code

[Nikolaus Rath]

On Feb 13 2016, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
> Nikolaus Rath <Nikolaus@rath.org> writes:
>
>> Hello,
>>
>> A little while ago I did some first-time hacking on Gnus, and then felt
>> the urgent need to tell the world about my thoughts. I've now finally
>> found the time to write things down nicely. In case you are curious,
>> please take a look at:
>>
>> http://www.rath.org/whats-wrong-with-gnus.html
>>
>> Feedback is very welcome. While I did not particularly like the
>> codebase, I do not want to state anything that's untrue.
>
> Hi Nikolaus,
>
> Interesting read! And one that I'm mostly sympathetic with. But a few
> responses:
>
> 1. I don't think there's anything inherently unsuitable about Elisp for
>    writing a MUA (or any other type of software). It's a general
>    programming language, a little different from Python or other more
>    popular languages, but it comes with a full complement of modern
>    programming features.

You may be right. But I think in that case the relevant modern
programming features have a bit of a visibility problem. They are rarely
found in existing code (probably because most of it predates them), and
there also don't feature prominently in the "Emacs Lisp Reference
Manual".

> 3. I don't think the buffer issue is a real problem. You're getting
>    responses from a server as a string, and there isn't really any
>    difference between a buffer and a string: you'll need to chomp it, no
>    matter what.

I don't object to using buffers at all, I object to using them to pass
data around between functions. For example, I think a much better format
to pass around message headers would be a list of alists. That way, if a
function wants to access a specific header it doesn't have to
re-implement RFC822 parsing but it simply uses assq et al. And even
better, if I messed up and passed a buffer containing a full message
instead of a sequence of headers, things are going to fail loudly
instead of weird things happening silently.

>    I understand there are such things out there as Real
>    Parsers, and that they're difficult to write, and that maybe we could
>    benefit by having one

That's why you don't write the parser, but use a parser generator
:-). 

> 4. Type checking and error handling is a real issue. I feel like half
>    the errors I get from Gnus are (wrong-type-argument stringp nil).
>    Then you look at the backtrace and the argument was already nil like
>    six function calls up. But here again, the problem isn't that Elisp
>    doesn't provide the tools, but that Gnus doesn't use them. We can and
>    should be defining our own error types, and handling them.
>
> Type checking is more interesting. Elisp doesn't do a lot of this by
> default, but some of the newer tools can serve the purpose. I'll bet we
> could make some interesting use of pattern-matching with pcase.
>
> But I still think (I've been obsessed with this for a bit now) that
> shifting to proper classes with EIEIO is the way to go. We could do away
> with the awe and horror that is nnoo.el, and prevent servers from
> "infecting" each other's variables and method calls. Generic functions
> are another way of doing type checking: if a method's specialization
> calls for arguments of a server, a string, and a symbol, you're not
> getting anywhere unless that's what they are.

Very much in agreement with all of that.


Best,
-Nikolaus

(No Cc on replies please, I'm reading the list)
-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: My thoughts on the Gnus source code

[Eric Abrahamsen]

Nikolaus Rath <Nikolaus@rath.org> writes:

> On Feb 13 2016, Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>> Nikolaus Rath <Nikolaus@rath.org> writes:
>>

[...]

>> 1. I don't think there's anything inherently unsuitable about Elisp for
>>    writing a MUA (or any other type of software). It's a general
>>    programming language, a little different from Python or other more
>>    popular languages, but it comes with a full complement of modern
>>    programming features.
>
> You may be right. But I think in that case the relevant modern
> programming features have a bit of a visibility problem. They are rarely
> found in existing code (probably because most of it predates them), and
> there also don't feature prominently in the "Emacs Lisp Reference
> Manual".

That's definitely fair to say. And I think Emacs development energy gets
split between "making Elisp a better language" and "making Emacs a
better editor". But the tools are there.

[...]

> I don't object to using buffers at all, I object to using them to pass
> data around between functions. For example, I think a much better format
> to pass around message headers would be a list of alists. That way, if a
> function wants to access a specific header it doesn't have to
> re-implement RFC822 parsing but it simply uses assq et al. And even
> better, if I messed up and passed a buffer containing a full message
> instead of a sequence of headers, things are going to fail loudly
> instead of weird things happening silently.
>
>>    I understand there are such things out there as Real
>>    Parsers, and that they're difficult to write, and that maybe we could
>>    benefit by having one
>
> That's why you don't write the parser, but use a parser generator
> :-). 

Apparently we have not one but two parser generators! Have you looked at
the manuals for Bovine and Wisent? Bovine claims to be the simpler of
the two. The manuals seem quite geared towards parsing source code
buffers, but maybe they would do just as well for parsing IMAP server
output? It's not something I know much about (it's also not my
particular itch!).

external editor (was: My thoughts on the Gnus source code)

[Uwe Brauer]

>>> "Nikolaus" == Nikolaus Rath <Nikolaus@rath.org> writes:

   > Hello,
   > A little while ago I did some first-time hacking on Gnus, and then felt
   > the urgent need to tell the world about my thoughts. I've now finally
   > found the time to write things down nicely. In case you are curious,
   > please take a look at:

   > http://www.rath.org/whats-wrong-with-gnus.html

   > Feedback is very welcome. While I did not particularly like the
   > codebase, I do not want to state anything that's untrue.

Since you mentioned thunderbird and its awful editor[1]: you are
aware of the fact, that there is a TB extension, which allows you to use
gnuclient as an external editor. I have used it sometimes, but usually
stick with gnus.

Footnotes: 
[1]  which is true, I never understood why MUA can't have a decent editor.

Re: external editor

[Nikolaus Rath]

On Mar 26 2016, Uwe Brauer <oub@mat.ucm.es> wrote:
>>>> "Nikolaus" == Nikolaus Rath <Nikolaus@rath.org> writes:
>
>    > Hello,
>    > A little while ago I did some first-time hacking on Gnus, and then felt
>    > the urgent need to tell the world about my thoughts. I've now finally
>    > found the time to write things down nicely. In case you are curious,
>    > please take a look at:
>
>    > http://www.rath.org/whats-wrong-with-gnus.html
>
>    > Feedback is very welcome. While I did not particularly like the
>    > codebase, I do not want to state anything that's untrue.
>
> Since you mentioned thunderbird and its awful editor[1]: you are
> aware of the fact, that there is a TB extension, which allows you to use
> gnuclient as an external editor. I have used it sometimes, but usually
> stick with gnus.

I was expecting that there's something like that, but I also expected
the integration not to work properly so I never tried it.


However, I just tried searching the addons for "gnuclient" and "external
editor" and didn't get any matches at all. Do you happen to have a link?


Best,
-Nikolaus

-- 
GPG encrypted emails preferred. Key id: 0xD113FCAC3C4E599F
Fingerprint: ED31 791B 2C5C 1613 AF38 8B8A D113 FCAC 3C4E 599F

             »Time flies like an arrow, fruit flies like a Banana.«

Re: external editor

[Uwe Brauer]

    > On Mar 26 2016, Uwe Brauer <oub@mat.ucm.es> wrote:

    > I was expecting that there's something like that, but I also expected
    > the integration not to work properly so I never tried it.

That is not my experience, it worked ok.

    > However, I just tried searching the addons for "gnuclient" and "external
    > editor" and didn't get any matches at all. Do you happen to have a link?

Yes

http://globs.org/

I am using it with seamonkey not thunderbird. I had difficulties in
installing it. The program complained about a wrong version number, so I
had to modify the file install.rdf. May be you have to do something
similar.

regards

Uwe