* strange memory usage for nntp groups @ 2017-12-27 21:06 Eric S Fraga 2017-12-27 23:24 ` Emanuel Berg 2017-12-28 10:31 ` Steinar Bang 0 siblings, 2 replies; 9+ messages in thread From: Eric S Fraga @ 2017-12-27 21:06 UTC (permalink / raw) To: ding [-- Attachment #1: Type: text/plain, Size: 729 bytes --] Hello, I am using emacs 25.2 (debian) with built-in gnus on a very small computer (ARM based OpenPandora [https://pyra-handheld.com/boards/] running Debian aka pandian) with only 512 MB of RAM. I can read all my imap groups just fine. However, as soon as I try to open one of the (maybe large) nntp groups, e.g. gmane.emacs.help, the memory use by Emacs jumps dramatically from 250 MB to 780 MB, making the system unusable. Is there some variable that I may be setting that affects memory use so dramatically? Or is there some variable I should set to avoid this problem? I have set gnus.thread.refer.limit to 0 but have done no other tests. Thanks, eric -- Eric S Fraga via Emacs 27.0.50, org 9.1.5 [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 194 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: strange memory usage for nntp groups 2017-12-27 21:06 strange memory usage for nntp groups Eric S Fraga @ 2017-12-27 23:24 ` Emanuel Berg 2017-12-27 23:55 ` Eric S Fraga 2017-12-28 10:31 ` Steinar Bang 1 sibling, 1 reply; 9+ messages in thread From: Emanuel Berg @ 2017-12-27 23:24 UTC (permalink / raw) To: ding Eric S Fraga wrote: > Is there some variable that I may be setting > that affects memory use so dramatically? > Or is there some variable I should set to > avoid this problem? I have set > gnus.thread.refer.limit to 0 but have done no > other tests. Could this be the same thing I have reported a couple of times? I.e. the huge .overview files? E.g., my gmane.emacs.help ditto isn't that big at the moment - only 36M! $ find ./ -name .overview -exec ls -lh {} \; -- underground experts united http://user.it.uu.se/~embe8573 ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: strange memory usage for nntp groups 2017-12-27 23:24 ` Emanuel Berg @ 2017-12-27 23:55 ` Eric S Fraga 2017-12-28 3:04 ` Emanuel Berg 0 siblings, 1 reply; 9+ messages in thread From: Eric S Fraga @ 2017-12-27 23:55 UTC (permalink / raw) To: ding [-- Attachment #1: Type: text/plain, Size: 842 bytes --] On Thursday, 28 Dec 2017 at 00:24, Emanuel Berg wrote: > Eric S Fraga wrote: > >> Is there some variable that I may be setting >> that affects memory use so dramatically? >> Or is there some variable I should set to >> avoid this problem? I have set >> gnus.thread.refer.limit to 0 but have done no >> other tests. > > Could this be the same thing I have reported > a couple of times? I.e. the huge .overview > files? E.g., my gmane.emacs.help ditto isn't > that big at the moment - only 36M! > > $ find ./ -name .overview -exec ls -lh {} \; Ah, interesting; it could be. My gmane.linux.debian.user overview file, for instance, is 162Mb! Is there a solution? What if I delete the overview file? Will having it regenerate lead to the same size? Thanks, eric -- Eric S Fraga via Emacs 27.0.50, org 9.1.5 [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 194 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: strange memory usage for nntp groups 2017-12-27 23:55 ` Eric S Fraga @ 2017-12-28 3:04 ` Emanuel Berg 2017-12-28 18:25 ` Eric S Fraga 0 siblings, 1 reply; 9+ messages in thread From: Emanuel Berg @ 2017-12-28 3:04 UTC (permalink / raw) To: ding Eric S Fraga wrote: > Ah, interesting; it could be. > My gmane.linux.debian.user overview file, for > instance, is 162Mb! > > Is there a solution? What if I delete the > overview file? Will having it regenerate lead > to the same size? From what I can understand from the manual entry, yanked last in this posts, the overview files are generated by Gnus only for nnml; for NNTP, the server makes them, so I suppose additions are communicated and appended locally. If so, deleting a file will help but eventually it will grow back to size. Also, what functionality will be *lost* this way? The ability to access old posts? In general, the purpose of the overview files isn't clear to me from the manual entry. It seems to be some scheme with the purpose, interestingly, to speed up populating the summary buffer! "NOV" NOV stands for News OverView, which is a type of news server header which provide datas containing the condensed header information of articles. They are produced by the server itself; in the ‘nntp’ back end Gnus uses the ones that the NNTP server makes, but Gnus makes them by itself for some backends (in particular, ‘nnml’). When Gnus enters a group, it asks the back end for the headers of all unread articles in the group. Most servers support the News OverView format, which is more compact and much faster to read and parse than the normal HEAD format. The NOV data consist of one or more text lines (*note Motion by Text Lines: (elisp)Text Lines.) where each line has the header information of one article. The header information is a tab-separated series of the header’s contents including an article number, a subject, an author, a date, a message-id, references, etc. Those data enable Gnus to generate summary lines quickly. However, if the server does not support NOV or you disable it purposely or for some reason, Gnus will try to generate the header information by parsing each article’s headers one by one. It will take time. Therefore, it is not usually a good idea to set nn*-nov-is-evil (*note Slow/Expensive Connection::) to a non-‘nil’ value unless you know that the server makes wrong NOV data. [1] [1] line 23594, /usr/share/info/emacs-24/gnus.info.gz -- underground experts united http://user.it.uu.se/~embe8573 ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: strange memory usage for nntp groups 2017-12-28 3:04 ` Emanuel Berg @ 2017-12-28 18:25 ` Eric S Fraga 2017-12-28 18:51 ` Eric S Fraga 0 siblings, 1 reply; 9+ messages in thread From: Eric S Fraga @ 2017-12-28 18:25 UTC (permalink / raw) To: ding [-- Attachment #1: Type: text/plain, Size: 814 bytes --] On Thursday, 28 Dec 2017 at 04:04, Emanuel Berg wrote: [...] > From what I can understand from the manual > entry, yanked last in this posts, the overview > files are generated by Gnus only for nnml; for > NNTP, the server makes them, so I suppose > additions are communicated and > appended locally. > > If so, deleting a file will help but eventually > it will grow back to size. Well, I deleted the .overview file and I was able to enter the group and read the posts in it. Memory usage went up only marginally. Simple enough to make a cron script or equivalent that deletes .overview files that get too large, I guess. I did first try setting nntp-nov-is-evil to t but this seemed to make little difference. Thanks for your help. -- Eric S Fraga via Emacs 27.0.50, org 9.1.5 [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 194 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: strange memory usage for nntp groups 2017-12-28 18:25 ` Eric S Fraga @ 2017-12-28 18:51 ` Eric S Fraga 2017-12-28 19:03 ` Emanuel Berg 0 siblings, 1 reply; 9+ messages in thread From: Eric S Fraga @ 2017-12-28 18:51 UTC (permalink / raw) To: ding [-- Attachment #1: Type: text/plain, Size: 1281 bytes --] On Thursday, 28 Dec 2017 at 18:25, Eric S Fraga wrote: > On Thursday, 28 Dec 2017 at 04:04, Emanuel Berg wrote: > > [...] > >> From what I can understand from the manual >> entry, yanked last in this posts, the overview >> files are generated by Gnus only for nnml; for >> NNTP, the server makes them, so I suppose >> additions are communicated and >> appended locally. >> >> If so, deleting a file will help but eventually >> it will grow back to size. > > Well, I deleted the .overview file and I was able to enter the group and > read the posts in it. Memory usage went up only marginally. Simple > enough to make a cron script or equivalent that deletes .overview files > that get too large, I guess. I take it back. Deleting the .overview file is not a good idea. It makes it impossible to retrieve any old articles at all. I've had to recreate the group from the server to get at any old articles. Can anybody suggest how I can read large newsgroups on systems with small memory? I would have thought that gnus should be able to do this. Does it need to keep the whole set of headers in memory? I'm happy to pay the price of network access in lieu of memory overload... Thanks, eric -- Eric S Fraga via Emacs 27.0.50, org 9.1.5 [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 194 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: strange memory usage for nntp groups 2017-12-28 18:51 ` Eric S Fraga @ 2017-12-28 19:03 ` Emanuel Berg 0 siblings, 0 replies; 9+ messages in thread From: Emanuel Berg @ 2017-12-28 19:03 UTC (permalink / raw) To: ding Eric S Fraga wrote: > Can anybody suggest how I can read large > newsgroups on systems with small memory? > I would have thought that gnus should be able > to do this. Does it need to keep the whole > set of headers in memory? I'm happy to pay > the price of network access in lieu of memory > overload... Small system or big, having huge files is almost never a good idea. Perhaps the file can be split up over time, or when the get too big? -- underground experts united http://user.it.uu.se/~embe8573 ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: strange memory usage for nntp groups 2017-12-27 21:06 strange memory usage for nntp groups Eric S Fraga 2017-12-27 23:24 ` Emanuel Berg @ 2017-12-28 10:31 ` Steinar Bang 2017-12-28 18:30 ` Eric S Fraga 1 sibling, 1 reply; 9+ messages in thread From: Steinar Bang @ 2017-12-28 10:31 UTC (permalink / raw) To: ding >>>>> Eric S Fraga <e.fraga@ucl.ac.uk>: > I am using emacs 25.2 (debian) with built-in gnus on a very small > computer (ARM based OpenPandora [https://pyra-handheld.com/boards/] > running Debian aka pandian) with only 512 MB of RAM. Off topic: Hey this looked really neat! Maybe I've found a potential replacement for my now-dead debian netbook...? :-) Sadly they had stopped making netbooks when I went looking for a replacement. (The netbook-ish cases don't seem to be read yet, though...?) ^ permalink raw reply [flat|nested] 9+ messages in thread
* Re: strange memory usage for nntp groups 2017-12-28 10:31 ` Steinar Bang @ 2017-12-28 18:30 ` Eric S Fraga 0 siblings, 0 replies; 9+ messages in thread From: Eric S Fraga @ 2017-12-28 18:30 UTC (permalink / raw) To: ding [-- Attachment #1: Type: text/plain, Size: 1002 bytes --] On Thursday, 28 Dec 2017 at 11:31, Steinar Bang wrote: >>>>>> Eric S Fraga <e.fraga@ucl.ac.uk>: > > >> I am using emacs 25.2 (debian) with built-in gnus on a very small >> computer (ARM based OpenPandora [https://pyra-handheld.com/boards/] >> running Debian aka pandian) with only 512 MB of RAM. > > Off topic: Hey this looked really neat! Maybe I've found a potential > replacement for my now-dead debian netbook...? :-) My Pandora was indeed the replacement for my defunct Asus EEE-PC netbook. Battery lasts hours and hours and the sound system is very good as well. I am impatiently waiting for the replacement, the Pyra, which is coming close to fruition and brings a larger screen in the same form factor along with a relatively more modern processor and larger memory. There are other palmtop/netbook systems out there such as the GPD Win. The nice thing about the Pandora & the Pyra is that they are completely open. -- Eric S Fraga via Emacs 27.0.50, org 9.1.5 [-- Attachment #2: signature.asc --] [-- Type: application/pgp-signature, Size: 194 bytes --] ^ permalink raw reply [flat|nested] 9+ messages in thread
end of thread, other threads:[~2017-12-28 19:03 UTC | newest] Thread overview: 9+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2017-12-27 21:06 strange memory usage for nntp groups Eric S Fraga 2017-12-27 23:24 ` Emanuel Berg 2017-12-27 23:55 ` Eric S Fraga 2017-12-28 3:04 ` Emanuel Berg 2017-12-28 18:25 ` Eric S Fraga 2017-12-28 18:51 ` Eric S Fraga 2017-12-28 19:03 ` Emanuel Berg 2017-12-28 10:31 ` Steinar Bang 2017-12-28 18:30 ` Eric S Fraga
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).