strange memory usage for nntp groups

strange memory usage for nntp groups

[Eric S Fraga]

[-- Attachment #1: Type: text/plain, Size: 729 bytes --]

Hello,

I am using emacs 25.2 (debian) with built-in gnus on a very small
computer (ARM based OpenPandora [https://pyra-handheld.com/boards/]
running Debian aka pandian) with only 512 MB of RAM.  I can read all my
imap groups just fine.  However, as soon as I try to open one of the
(maybe large) nntp groups, e.g. gmane.emacs.help, the memory use by
Emacs jumps dramatically from 250 MB to 780 MB, making the system
unusable.

Is there some variable that I may be setting that affects memory use so
dramatically?  Or is there some variable I should set to avoid this
problem?  I have set gnus.thread.refer.limit to 0 but have done no other
tests.

Thanks,
eric

-- 
Eric S Fraga via Emacs 27.0.50, org 9.1.5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

Re: strange memory usage for nntp groups

[Emanuel Berg]

Eric S Fraga wrote:

> Is there some variable that I may be setting
> that affects memory use so dramatically?
> Or is there some variable I should set to
> avoid this problem? I have set
> gnus.thread.refer.limit to 0 but have done no
> other tests.

Could this be the same thing I have reported
a couple of times? I.e. the huge .overview
files? E.g., my gmane.emacs.help ditto isn't
that big at the moment - only 36M!

    $ find ./ -name .overview -exec ls -lh {} \;

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: strange memory usage for nntp groups

[Eric S Fraga]

[-- Attachment #1: Type: text/plain, Size: 842 bytes --]

On Thursday, 28 Dec 2017 at 00:24, Emanuel Berg wrote:
> Eric S Fraga wrote:
>
>> Is there some variable that I may be setting
>> that affects memory use so dramatically?
>> Or is there some variable I should set to
>> avoid this problem? I have set
>> gnus.thread.refer.limit to 0 but have done no
>> other tests.
>
> Could this be the same thing I have reported
> a couple of times? I.e. the huge .overview
> files? E.g., my gmane.emacs.help ditto isn't
> that big at the moment - only 36M!
>
>     $ find ./ -name .overview -exec ls -lh {} \;

Ah, interesting; it could be.  My gmane.linux.debian.user overview file,
for instance, is 162Mb!

Is there a solution?  What if I delete the overview file?  Will having
it regenerate lead to the same size?

Thanks,
eric

-- 
Eric S Fraga via Emacs 27.0.50, org 9.1.5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

Re: strange memory usage for nntp groups

[Emanuel Berg]

Eric S Fraga wrote:

> Ah, interesting; it could be.
> My gmane.linux.debian.user overview file, for
> instance, is 162Mb!
>
> Is there a solution? What if I delete the
> overview file? Will having it regenerate lead
> to the same size?

From what I can understand from the manual
entry, yanked last in this posts, the overview
files are generated by Gnus only for nnml; for
NNTP, the server makes them, so I suppose
additions are communicated and
appended locally.

If so, deleting a file will help but eventually
it will grow back to size.

Also, what functionality will be *lost* this
way? The ability to access old posts?

In general, the purpose of the overview files
isn't clear to me from the manual entry.
It seems to be some scheme with the purpose,
interestingly, to speed up populating the
summary buffer!


"NOV"
     NOV stands for News OverView, which is
     a type of news server header which provide
     datas containing the condensed header
     information of articles. They are produced
     by the server itself; in the ‘nntp’ back
     end Gnus uses the ones that the NNTP
     server makes, but Gnus makes them by
     itself for some backends (in particular,
     ‘nnml’).

     When Gnus enters a group, it asks the back
     end for the headers of all unread articles
     in the group. Most servers support the
     News OverView format, which is more
     compact and much faster to read and parse
     than the normal HEAD format.

     The NOV data consist of one or more text
     lines (*note Motion by Text Lines:
     (elisp)Text Lines.) where each line has
     the header information of one article.
     The header information is a tab-separated
     series of the header’s contents including
     an article number, a subject, an author,
     a date, a message-id, references, etc.

     Those data enable Gnus to generate summary
     lines quickly. However, if the server does
     not support NOV or you disable it
     purposely or for some reason, Gnus will
     try to generate the header information by
     parsing each article’s headers one by one.
     It will take time. Therefore, it is not
     usually a good idea to set nn*-nov-is-evil
     (*note Slow/Expensive Connection::) to
     a non-‘nil’ value unless you know that the
     server makes wrong NOV data. [1]

[1] line 23594, /usr/share/info/emacs-24/gnus.info.gz

-- 
underground experts united
http://user.it.uu.se/~embe8573

Re: strange memory usage for nntp groups

[Steinar Bang]

>>>>> Eric S Fraga <e.fraga@ucl.ac.uk>:


> I am using emacs 25.2 (debian) with built-in gnus on a very small
> computer (ARM based OpenPandora [https://pyra-handheld.com/boards/]
> running Debian aka pandian) with only 512 MB of RAM.

Off topic: Hey this looked really neat!  Maybe I've found a potential
replacement for my now-dead debian netbook...? :-)

Sadly they had stopped making netbooks when I went looking for a
replacement.

(The netbook-ish cases don't seem to be read yet, though...?)

Re: strange memory usage for nntp groups

[Eric S Fraga]

[-- Attachment #1: Type: text/plain, Size: 814 bytes --]

On Thursday, 28 Dec 2017 at 04:04, Emanuel Berg wrote:

[...]

> From what I can understand from the manual
> entry, yanked last in this posts, the overview
> files are generated by Gnus only for nnml; for
> NNTP, the server makes them, so I suppose
> additions are communicated and
> appended locally.
>
> If so, deleting a file will help but eventually
> it will grow back to size.

Well, I deleted the .overview file and I was able to enter the group and
read the posts in it.  Memory usage went up only marginally.  Simple
enough to make a cron script or equivalent that deletes .overview files
that get too large, I guess.

I did first try setting nntp-nov-is-evil to t but this seemed to make
little difference.

Thanks for your help.

-- 
Eric S Fraga via Emacs 27.0.50, org 9.1.5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

Re: strange memory usage for nntp groups

[Eric S Fraga]

[-- Attachment #1: Type: text/plain, Size: 1002 bytes --]

On Thursday, 28 Dec 2017 at 11:31, Steinar Bang wrote:
>>>>>> Eric S Fraga <e.fraga@ucl.ac.uk>:
>
>
>> I am using emacs 25.2 (debian) with built-in gnus on a very small
>> computer (ARM based OpenPandora [https://pyra-handheld.com/boards/]
>> running Debian aka pandian) with only 512 MB of RAM.
>
> Off topic: Hey this looked really neat!  Maybe I've found a potential
> replacement for my now-dead debian netbook...? :-)

My Pandora was indeed the replacement for my defunct Asus EEE-PC
netbook.  Battery lasts hours and hours and the sound system is very
good as well.  I am impatiently waiting for the replacement, the Pyra,
which is coming close to fruition and brings a larger screen in the same
form factor along with a relatively more modern processor and larger
memory. 

There are other palmtop/netbook systems out there such as the GPD
Win.  The nice thing about the Pandora & the Pyra is that they are
completely open.

-- 
Eric S Fraga via Emacs 27.0.50, org 9.1.5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

Re: strange memory usage for nntp groups

[Eric S Fraga]

[-- Attachment #1: Type: text/plain, Size: 1281 bytes --]

On Thursday, 28 Dec 2017 at 18:25, Eric S Fraga wrote:
> On Thursday, 28 Dec 2017 at 04:04, Emanuel Berg wrote:
>
> [...]
>
>> From what I can understand from the manual
>> entry, yanked last in this posts, the overview
>> files are generated by Gnus only for nnml; for
>> NNTP, the server makes them, so I suppose
>> additions are communicated and
>> appended locally.
>>
>> If so, deleting a file will help but eventually
>> it will grow back to size.
>
> Well, I deleted the .overview file and I was able to enter the group and
> read the posts in it.  Memory usage went up only marginally.  Simple
> enough to make a cron script or equivalent that deletes .overview files
> that get too large, I guess.

I take it back.  Deleting the .overview file is not a good idea.  It
makes it impossible to retrieve any old articles at all.  I've had to
recreate the group from the server to get at any old articles.

Can anybody suggest how I can read large newsgroups on systems with
small memory?  I would have thought that gnus should be able to do
this.  Does it need to keep the whole set of headers in memory?  I'm
happy to pay the price of network access in lieu of memory overload...

Thanks,
eric

-- 
Eric S Fraga via Emacs 27.0.50, org 9.1.5

[-- Attachment #2: signature.asc --]
[-- Type: application/pgp-signature, Size: 194 bytes --]

Re: strange memory usage for nntp groups

[Emanuel Berg]

Eric S Fraga wrote:

> Can anybody suggest how I can read large
> newsgroups on systems with small memory?
> I would have thought that gnus should be able
> to do this. Does it need to keep the whole
> set of headers in memory? I'm happy to pay
> the price of network access in lieu of memory
> overload...

Small system or big, having huge files is
almost never a good idea. Perhaps the file can
be split up over time, or when the get too big?

-- 
underground experts united
http://user.it.uu.se/~embe8573