Gnus work

Gnus work

[Eric Abrahamsen]

[-- Attachment #1: Type: text/plain, Size: 1976 bytes --]

Hi all,

I've been very slowly reading the Gnus code base, and looking at how to
go about doing some refactoring, cleanups, and a few new features. I
wrote a roadmap which I'm attaching here -- it's not so much an actual
roadmap as a collection of possible changes, as a starting point for
argument. It's also not meant to be a mini bug tracker in Org -- my idea
was that if any of the ideas looked like they were going to be
acceptable, the todo items would turn into real bug reports on debbugs.

In general, I've been working from a few principles:

1. Replace Gnus' homemade mechanisms with equivalent mechanisms that
   have since appeared in core (in many cases Gnus had them first).
2. Add docstrings and comments to minimize bug-hunter bewilderment.
3. Look at unpicking functions so that there's a clearer demarcation
   between code that does server logic (marks and whatnot) and code that
   does UX/presentation stuff.
4. #3 would ideally allow more of the code to be testable.
5. Reduce number of dynamic variables, to cut back on "spooky action at
   a distance". To be honest I don't even know where to start here.

The roadmap is also not meant to address all of Gnus -- for the most
part I'm sticking to groups, servers, and marks. I don't feel
comfortable or competent to mess with other areas like
MIME/threading/washing/network/definitely not security. Some of those
areas already have "shepherds", anyway.

I'd like to make this document publicly accessible and editable (a
branch? Emacswiki?), and see if anyone has anything to contribute, or
otherwise wants to jump in. I think there's a lot of love for Gnus out
there, and my hope is that the roadmap might spur people to pop
up and share their ideas, and maybe offer to pitch in.

Where could I put it?

Eric

PS: No, I don't have any code yet. The one patch I do have touches
nothing but docstrings, code comments, and "Commentary" sections --
stuff I've added over the course of reading the code.



[-- Attachment #2: roadmap.org --]
[-- Type: application/vnd.lotus-organizer, Size: 10633 bytes --]

Re: Gnus work

[Emanuel Berg]

Eric Abrahamsen wrote:

> I've been very slowly reading the Gnus code
> base, and looking at how to go about doing
> some refactoring, cleanups, and a few
> new features.

What strikes me immediately when I read the
Gnus source is that the defuns are often very,
very long. But I'm not suggesting anyone do
something about it for that reason alone.
However if anyone does work on a defun and has
thus acquired a close to complete understanding
of it, it could be considered breaking it up,
possibly repeatedly, as the last step of work,
and then along lines that are as natural
as possible, of course.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Gnus work

[Eric Abrahamsen]

Emanuel Berg <moasen@zoho.com> writes:

> Eric Abrahamsen wrote:
>
>> I've been very slowly reading the Gnus code
>> base, and looking at how to go about doing
>> some refactoring, cleanups, and a few
>> new features.
>
> What strikes me immediately when I read the
> Gnus source is that the defuns are often very,
> very long. But I'm not suggesting anyone do
> something about it for that reason alone.
> However if anyone does work on a defun and has
> thus acquired a close to complete understanding
> of it, it could be considered breaking it up,
> possibly repeatedly, as the last step of work,
> and then along lines that are as natural
> as possible, of course.

I think the main problem is not so much that the defuns are long, but
that they do too much, there's overlap, and there's lots of
work-by-side-effect. A long defun wouldn't be so bad if you could scan
it and feel confident that you knew what it was doing.

What I *do* think is a problem is that variables and functions are named
inconsistently, and often confusingly. It just makes debugging that much
harder. But on the other hand, renaming just for renaming's sake...

Re: Gnus work

[Emanuel Berg]

Eric Abrahamsen wrote:

> I think the main problem is not so much that
> the defuns are long, but that they do too
> much, there's overlap, and there's lots of
> work-by-side-effect. A long defun wouldn't be
> so bad if you could scan it and feel
> confident that you knew what it was doing.

More or less. The form always has an impact on
the contents. I think what you describe is in
part a function of the habit of writing too
long functions.

> What I *do* think is a problem is that
> variables and functions are named
> inconsistently, and often confusingly.
> It just makes debugging that much harder.
> But on the other hand, renaming just for
> renaming's sake...

Indeed, small steps while at it.

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: Gnus work

[Tim Landscheidt]

Eric Abrahamsen <eric@ericabrahamsen.net> wrote:

> I've been very slowly reading the Gnus code base, and looking at how to
> go about doing some refactoring, cleanups, and a few new features. I
> wrote a roadmap which I'm attaching here -- it's not so much an actual
> roadmap as a collection of possible changes, as a starting point for
> argument. It's also not meant to be a mini bug tracker in Org -- my idea
> was that if any of the ideas looked like they were going to be
> acceptable, the todo items would turn into real bug reports on debbugs.

> […]

You, Sir, are my hero :-).  I'm always torn by Gnus: On the
one hand, it offers ample opportunity to do whatever you
want.  On the other hand, if what you want is not described
in the manual or there is not already a suitable code snip-
let out there, reading through the source is a nightmare.  I
do not even have a mental picture where what data is stored
at which point of time.

So I can only wholeheartedly support any effort to change
this :-).  One thing I would prefer, though, is documenting
(and refactoring) before any user-visible changes (and given
how users typically interact with Gnus's internals that can
be even small changes).  It's much easier to review and test
that a change does not break something if it does not
(should not) affect Gnus's outwards behaviour.  For example,
while I agree some parts of Gnus's configuration & Co. are
way too byzantine, changing it in an incompatible way would
effectively say to Gnus users: "We broke your setup, you're
welcome."  So any such changes should come at the end of the
"road".

Tim

Re: Gnus work

[Eric Abrahamsen]

Tim Landscheidt <tim@tim-landscheidt.de> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> wrote:
>
>> I've been very slowly reading the Gnus code base, and looking at how to
>> go about doing some refactoring, cleanups, and a few new features. I
>> wrote a roadmap which I'm attaching here -- it's not so much an actual
>> roadmap as a collection of possible changes, as a starting point for
>> argument. It's also not meant to be a mini bug tracker in Org -- my idea
>> was that if any of the ideas looked like they were going to be
>> acceptable, the todo items would turn into real bug reports on debbugs.
>
>> […]
>
> You, Sir, are my hero :-).  I'm always torn by Gnus: On the
> one hand, it offers ample opportunity to do whatever you
> want.  On the other hand, if what you want is not described
> in the manual or there is not already a suitable code snip-
> let out there, reading through the source is a nightmare.  I
> do not even have a mental picture where what data is stored
> at which point of time.
>
> So I can only wholeheartedly support any effort to change
> this :-).  One thing I would prefer, though, is documenting
> (and refactoring) before any user-visible changes (and given
> how users typically interact with Gnus's internals that can
> be even small changes).  It's much easier to review and test
> that a change does not break something if it does not
> (should not) affect Gnus's outwards behaviour.  For example,
> while I agree some parts of Gnus's configuration & Co. are
> way too byzantine, changing it in an incompatible way would
> effectively say to Gnus users: "We broke your setup, you're
> welcome."  So any such changes should come at the end of the
> "road".

All excellent points. I do think at some point we should be making
backwards-incompatible changes, but you're quite right that it would be
infuriating to have those changes come out piecemeal, without adequate
warning. Ideally they would be made all at once, with some amount of
fanfare, and an upgrade guide to what we might call Gnus version 6.

But that's getting ahead of things :)

I agree it's best to stick to "internal" changes only in the beginning,
accompanied with tests.

The current Gnus manual explicitly says that it does not aim to provide
a tutorial. Users, however, are obviously in desperate need of a
tutorial. Most recently, I think Adam Sjøgren (was it Adam?) posted an
online version of his work-in-progress tutorial, and I know there are a
few others -- why not incorporate these into the manual? A "getting
started" tutorial, and a "moving your existing mail into Gnus" tutorial.
I had a hell of a time migrating to Gnus whenever that was, lo these
many years ago, though of course I took no notes and immediately after
couldn't remember what was so difficult about it. :(

Anyway: start with docs and comments, move to tests and refactoring, and
only at the end introduce user-visible changes.

Eric

Re: Gnus work

[Adam Sjøgren]

Eric writes:

> The current Gnus manual explicitly says that it does not aim to provide
> a tutorial. Users, however, are obviously in desperate need of a
> tutorial. Most recently, I think Adam Sjøgren (was it Adam?) posted an
> online version of his work-in-progress tutorial, and I know there are a
> few others -- why not incorporate these into the manual?

I guess it depends on the volatility of the information in the tutorial.
I wouldn't be comfortable with putting, say, Gmail specific instructions
in the manual proper (who knows when service-specific things will
change), where one some silly website, it's fine.

(Yes, I have the beginning of an unfinished tutorial sitting around¹ :-))

> A "getting started" tutorial, and a "moving your existing mail into
> Gnus" tutorial. I had a hell of a time migrating to Gnus whenever that
> was, lo these many years ago, though of course I took no notes and
> immediately after couldn't remember what was so difficult about it. :(

Often getting the idea of the direction to take is the problem, I
think/guess?

Importing email (these days) is as "easy" as connecting Gnus to the
IMAP-server containing your old email and copying it.


  Best regards,

    Adam


¹ https://koldfront.dk/text/gnus-email-tutorial.html

-- 
 "I pragmatically turn my whims into principles!"             Adam Sjøgren
                                                         asjo@koldfront.dk

Re: Gnus work

[Eric Abrahamsen]

asjo@koldfront.dk (Adam Sjøgren) writes:

> Eric writes:
>
>> The current Gnus manual explicitly says that it does not aim to provide
>> a tutorial. Users, however, are obviously in desperate need of a
>> tutorial. Most recently, I think Adam Sjøgren (was it Adam?) posted an
>> online version of his work-in-progress tutorial, and I know there are a
>> few others -- why not incorporate these into the manual?
>
> I guess it depends on the volatility of the information in the tutorial.
> I wouldn't be comfortable with putting, say, Gmail specific instructions
> in the manual proper (who knows when service-specific things will
> change), where one some silly website, it's fine.
>
> (Yes, I have the beginning of an unfinished tutorial sitting around¹ :-))

I suppose I see what you mean about the Gmail-specific stuff, but it
doesn't take long to edit the manual! If Google really goes under, I
think we'll see it coming :)

But in principle, would you be okay with putting the tutorial in the
manual, once you're happy with it?

>> A "getting started" tutorial, and a "moving your existing mail into
>> Gnus" tutorial. I had a hell of a time migrating to Gnus whenever that
>> was, lo these many years ago, though of course I took no notes and
>> immediately after couldn't remember what was so difficult about it. :(
>
> Often getting the idea of the direction to take is the problem, I
> think/guess?
>
> Importing email (these days) is as "easy" as connecting Gnus to the
> IMAP-server containing your old email and copying it.

In my case, anyway, this was before I was using IMAP -- I just had a
pile of old email in one-file-per-message format, and I was trying to
respool through mail-sources to an nnml backend, and had to keep
compacting groups, and it was generally horrible. Maybe I can also dig
up the how-to that Sivaram mentioned.

Eric

Re: Gnus work

[Adam Sjøgren]

Eric writes:

> I suppose I see what you mean about the Gmail-specific stuff, but it
> doesn't take long to edit the manual! If Google really goes under, I
> think we'll see it coming :)

Yeah, but it usually takes a while for a release of GNU Emacs to be
replaced completely by a new version (including the updated volatile
information).

> But in principle, would you be okay with putting the tutorial in the
> manual, once you're happy with it?

I'd be happy if anyone can use it for something in whatever form, but I
don't think I will be finishing it, I kind of lost momentum after a
couple of months.

> In my case, anyway, this was before I was using IMAP -- I just had a
> pile of old email in one-file-per-message format, and I was trying to
> respool through mail-sources to an nnml backend, and had to keep
> compacting groups, and it was generally horrible.

Yeah, these days it's often simpler.

But also people use/want to use email in very different ways...

I have stopped suggesting to people how handle their email, because I've
discovered that what works for me doesn't work for most other people.
And then they get mad at me :-)


  Best regards,

    Adam

-- 
 "Plasmids are circular bits of DNA that bacteria             Adam Sjøgren
  treat like Pokemon - trading with their friends        asjo@koldfront.dk
  collecting all the cool ones."

Re: Gnus work

[Eric Abrahamsen]

asjo@koldfront.dk (Adam Sjøgren) writes:

> Eric writes:
>
>> I suppose I see what you mean about the Gmail-specific stuff, but it
>> doesn't take long to edit the manual! If Google really goes under, I
>> think we'll see it coming :)
>
> Yeah, but it usually takes a while for a release of GNU Emacs to be
> replaced completely by a new version (including the updated volatile
> information).

That's a good point.

>> But in principle, would you be okay with putting the tutorial in the
>> manual, once you're happy with it?
>
> I'd be happy if anyone can use it for something in whatever form, but I
> don't think I will be finishing it, I kind of lost momentum after a
> couple of months.

Okay, no worries -- if I get around to stealing anything from it, I'll
check with you first.

>> In my case, anyway, this was before I was using IMAP -- I just had a
>> pile of old email in one-file-per-message format, and I was trying to
>> respool through mail-sources to an nnml backend, and had to keep
>> compacting groups, and it was generally horrible.
>
> Yeah, these days it's often simpler.
>
> But also people use/want to use email in very different ways...
>
> I have stopped suggesting to people how handle their email, because I've
> discovered that what works for me doesn't work for most other people.
> And then they get mad at me :-)

I also have no interest in being prescriptive! But helping people with
the very first steps seems relatively low-risk.

Eric

Re: Gnus work

[Sivaram Neelakantan]

On Wed, Oct 04 2017,Eric Abrahamsen wrote:


[snipped 45 lines]

>
> The current Gnus manual explicitly says that it does not aim to provide
> a tutorial. Users, however, are obviously in desperate need of a
> tutorial. Most recently, I think Adam Sjøgren (was it Adam?) posted an
> online version of his work-in-progress tutorial, and I know there are a
> few others -- why not incorporate these into the manual? A "getting
> started" tutorial, and a "moving your existing mail into Gnus" tutorial.
> I had a hell of a time migrating to Gnus whenever that was, lo these
> many years ago, though of course I took no notes and immediately after
> couldn't remember what was so difficult about it. :(

Same here.  A long time ago, someone posted that they didn't know
where to get started using gnus and I think Jason Rumney(sp?) posted
something along the lines of "do this, this and this and you're
good".Worked for me and promptly forgot what my fears were(to help
others and take more notes)

I can't find the post by Jason too now.  Pretty sure was either here
or help-Emacs-Windows list. or gnu.Emacs.help 

[snipped 8 lines]



sivaram
-- 

Re: Gnus work

[Julien Danjou]

[-- Attachment #1: Type: text/plain, Size: 1198 bytes --]

On Wed, Oct 04 2017, Eric Abrahamsen wrote:

Hi Eric,

> I've been very slowly reading the Gnus code base, and looking at how to
> go about doing some refactoring, cleanups, and a few new features. I
> wrote a roadmap which I'm attaching here -- it's not so much an actual
> roadmap as a collection of possible changes, as a starting point for
> argument. It's also not meant to be a mini bug tracker in Org -- my idea
> was that if any of the ideas looked like they were going to be
> acceptable, the todo items would turn into real bug reports on debbugs.

As a former regular Gnus contributor and as a current and long time Gnus
user, I want to give you my blessing. :-)

They are definitively ton of little code improvement you can make. All
of this technical debt reduction will lead to a better Gnus and a better
development future, so yes please!

I don't know how true it is nowadays, but a lot of what was blocking
improvement back then was support for old Emacs version and XEmacs
support. Now that Gnus is in Emacs source tree, I imagine that might not
applies anymore, so, lucky you! :)

-- 
Julien Danjou
# Free Software hacker
# https://julien.danjou.info

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 832 bytes --]

Re: Gnus work

[Lars Ingebrigtsen]

Eric Abrahamsen <eric@ericabrahamsen.net> writes:

> 1. Replace Gnus' homemade mechanisms with equivalent mechanisms that
>    have since appeared in core (in many cases Gnus had them first).

Sounds good.  The backend interface is particularly grungy and could to
with a complete rewrite.  Since there are probably extremely few
out-of-tree backends these days, I think you can massacre the interface
with impunity without thinking about compatibility.

> 2. Add docstrings and comments to minimize bug-hunter bewilderment.

Sounds good.

> 3. Look at unpicking functions so that there's a clearer demarcation
>    between code that does server logic (marks and whatnot) and code that
>    does UX/presentation stuff.

There's a lot of long functions in Gnus, but there's also a reason for
many of them being that way.  The first is that when setting up modes
and stuff, there's a lot of things that just have to be, er, set up.
Replacing a long litany of stuff like

(setq truncate-lines t
      mode-line-format foo)

with

(gnus-set-truncate-lines-to-t)
(gnus-set-mode-line-format foo)

isn't that much clearer.  :-)

(I'm just kidding, but that's kinda the Unclean Bob style of
refactoring).

The other consideration is that Emacs Lisp is really slow.  Splitting
central functions into many component functions will make Gnus even
slower than it already is.  And Gnus is slow.

> 5. Reduce number of dynamic variables, to cut back on "spooky action at
>    a distance". To be honest I don't even know where to start here.

Good luck, but Gnus has a gazillion settings that modify its workings in
many, many ways, and they're all done by variables.  Trimming down that
overload of settings would be nice, but every time I've tried, there's
always somebody that relied on just that single thing you'd think that
nobody would be weird enough to actually use.

-- 
(domestic pets only, the antidote for overdose, milk.)
   bloggy blog: http://lars.ingebrigtsen.no

Re: Gnus work

[Eric Abrahamsen]

Lars Ingebrigtsen <larsi@gnus.org> writes:

> Eric Abrahamsen <eric@ericabrahamsen.net> writes:
>
>> 1. Replace Gnus' homemade mechanisms with equivalent mechanisms that
>>    have since appeared in core (in many cases Gnus had them first).
>
> Sounds good.  The backend interface is particularly grungy and could to
> with a complete rewrite.  Since there are probably extremely few
> out-of-tree backends these days, I think you can massacre the interface
> with impunity without thinking about compatibility.

Great! This is definitely the most ambitious thing I've got in mind, it
will take a lot of work, but I think it would be a big gain.

[...]

> Replacing a long litany of stuff like
>
> (setq truncate-lines t
>       mode-line-format foo)
>
> with
>
> (gnus-set-truncate-lines-to-t)
> (gnus-set-mode-line-format foo)
>
> isn't that much clearer.  :-)
>
> (I'm just kidding, but that's kinda the Unclean Bob style of
> refactoring).

That's not what I was planning :) Mainly I'd like to try to separate out
concerns a bit: eg, instead of two functions that both do a little bit
of mark manipulation and a little bit of buffer setup, have one function
that only handles marks/headers/etc, and one function that only handles
the buffer creation. Probably no less code, but clearer and more easily
testable.

> The other consideration is that Emacs Lisp is really slow.  Splitting
> central functions into many component functions will make Gnus even
> slower than it already is.  And Gnus is slow.

I'll try not to make it worse. Speaking of which, do you have a general
sense of where the bottlenecks might be? I wasn't planning on attacking
the performance issue, but it would be good to have in mind.

>> 5. Reduce number of dynamic variables, to cut back on "spooky action at
>>    a distance". To be honest I don't even know where to start here.
>
> Good luck, but Gnus has a gazillion settings that modify its workings in
> many, many ways, and they're all done by variables.  Trimming down that
> overload of settings would be nice, but every time I've tried, there's
> always somebody that relied on just that single thing you'd think that
> nobody would be weird enough to actually use.

Yeah, I'm not too optimistic about this. We'll see.

I do hope you could take a moment and scan the roadmap. It would be
valuable even just to get a quick reaction from you -- you know way more
about what's feasible than I do.

Eric