Announcements and discussions for Gnus, the GNU Emacs Usenet newsreader
 help / color / mirror / Atom feed
* Re: Keeping IMAP connection alive when using it in mail-sources
@ 2023-01-08 18:10 Eric Abrahamsen
  2023-01-10 18:18 ` Adam Sjøgren
  0 siblings, 1 reply; 7+ messages in thread
From: Eric Abrahamsen @ 2023-01-08 18:10 UTC (permalink / raw)
  To: info-gnus-english

Adam Sjøgren <asjo@koldfront.dk> writes:

>   Hi,
>
>
> I was wondering if there is a way to use IMAP in mail-sources ("as
> POP"), but without closing the connection after every
> gnus-group-get-new-news?

Not with the code as it's written! There's a very definite
(imap-close buf) at the end of the mail source fetching, then the buffer
is deleted. I guess it wouldn't be a bad idea to add a keepalive option
to imap.el, but someone would have to do that!



^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Keeping IMAP connection alive when using it in mail-sources
  2023-01-08 18:10 Keeping IMAP connection alive when using it in mail-sources Eric Abrahamsen
@ 2023-01-10 18:18 ` Adam Sjøgren
  2023-01-10 19:24   ` Adam Sjøgren
  0 siblings, 1 reply; 7+ messages in thread
From: Adam Sjøgren @ 2023-01-10 18:18 UTC (permalink / raw)
  To: info-gnus-english

Eric writes:

> Not with the code as it's written! There's a very definite
> (imap-close buf) at the end of the mail source fetching, then the buffer
> is deleted. I guess it wouldn't be a bad idea to add a keepalive option
> to imap.el, but someone would have to do that!

I guess changing mail-source-fetch-imap to check if the buffer already
exists and if the connected process is still there, and then not closing
imap and not killing the buffer at the end, would be the bare minimum.

Interesting.

Thanks for the pointer!


  Best regards,

    Adam

-- 
 "Emacs [...] a [...] nuclear-powered Swiss Army            Adam Sjøgren
  knife favored by top-notch programmers and computer  asjo@koldfront.dk
  scientists."



^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Keeping IMAP connection alive when using it in mail-sources
  2023-01-10 18:18 ` Adam Sjøgren
@ 2023-01-10 19:24   ` Adam Sjøgren
  2023-01-10 20:08     ` Emanuel Berg
  2023-01-14 21:29     ` Adam Sjøgren
  0 siblings, 2 replies; 7+ messages in thread
From: Adam Sjøgren @ 2023-01-10 19:24 UTC (permalink / raw)
  To: info-gnus-english

Adam writes:

> Eric writes:
>
>> Not with the code as it's written! There's a very definite
>> (imap-close buf) at the end of the mail source fetching, then the buffer
>> is deleted. I guess it wouldn't be a bad idea to add a keepalive option
>> to imap.el, but someone would have to do that!
>
> I guess changing mail-source-fetch-imap to check if the buffer already
> exists and if the connected process is still there, and then not closing
> imap and not killing the buffer at the end, would be the bare minimum.

I've tried doing that now.

Fetching news and mail now takes me 1.3 second, down from 2.2 seconds.

Nice!

Here's the modified version of mail-source-fetch-imap I cobbled
together - the handling of deleting the buffer if there is no process is
not so pretty:

(defun mail-source-fetch-imap (source callback)
  "Fetcher for imap sources."
  (mail-source-bind (imap source)
    (mail-source-run-script
     prescript
     `((?p . ,password) (?t . ,mail-source-crash-box)
       (?s . ,server) (?P . ,port) (?u . ,user))
     prescript-delay)
    (let ((from (format "%s:%s:%s" server user port))
	  (found 0)
          (imap-shell-program (or (list program) imap-shell-program)))
      (let ((buf (or (and (or (get-buffer-process " *imap source*")
                              (and (get-buffer " *imap source*")
                                   (kill-buffer " *imap source*")))
                          (get-buffer " *imap source*"))
                     (let ((newbuf (generate-new-buffer " *imap source*")))
	               (if (and (imap-open server port stream authentication newbuf)
	                        (imap-authenticate
		                 user (or (cdr (assoc from mail-source-password-cache))
                                          password)
                                 newbuf))
                           newbuf
                         (progn
                           (imap-close newbuf)
	                   ;; We nix out the password in case the error
	                   ;; was because of a wrong password being given.
	                   (setq mail-source-password-cache
	                         (delq (assoc from mail-source-password-cache)
		                       mail-source-password-cache))
	                   (error "IMAP error: %s" (imap-error-text newbuf))))))))
        (let ((mailbox-list (if (listp mailbox) mailbox (list mailbox))))
          (dolist (mailbox mailbox-list)
            (when (imap-mailbox-select mailbox nil buf)
	      (let ((coding-system-for-write
                     mail-source-imap-file-coding-system)
	            (mail-source-string (format "imap:%s:%s" server mailbox))
	            str remove)
                (message "Fetching from %s..." mailbox)
	        (with-temp-file mail-source-crash-box
	          ;; Avoid converting 8-bit chars from inserted strings to
	          ;; multibyte.
	          (mm-disable-multibyte)
	          ;; remember password
	          (with-current-buffer buf
		    (when (and imap-password
			       (not (member (cons from imap-password)
                                            mail-source-password-cache)))
		      (push (cons from imap-password) mail-source-password-cache)))
	          ;; if predicate is nil, use all uids
	          (dolist (uid (imap-search (or predicate "1:*") buf))
		    (when (setq str
			        (if (imap-capability 'IMAP4rev1 buf)
				    (caddar (imap-fetch uid "BODY.PEEK[]"
						        'BODYDETAIL nil buf))
			          (imap-fetch uid "RFC822.PEEK" 'RFC822 nil buf)))
		      (push uid remove)
		      (insert "From imap " (current-time-string) "\n")
		      (save-excursion
		        (insert str "\n\n"))
		      (while (let ((case-fold-search nil))
			       (re-search-forward "^From " nil t))
		        (replace-match ">From "))
		      (goto-char (point-max))))
	          (nnheader-ms-strip-cr))
	        (cl-incf found (mail-source-callback callback server))
	        (mail-source-delete-crash-box)
	        (when (and remove fetchflag)
	          (setq remove (nreverse remove))
	          (imap-message-flags-add
	           (imap-range-to-message-set (gnus-compress-sequence remove))
	           fetchflag nil buf))
	        (if dontexpunge
		    (imap-mailbox-unselect buf)
                  (imap-mailbox-close nil buf))))))
        (mail-source-run-script
         postscript
         `((?p . ,password) (?t . ,mail-source-crash-box)
           (?s . ,server) (?P . ,port) (?u . ,user)))
        found))))


-- 
 "A cat has nine lives, but a bullfrog croaks every         Adam Sjøgren
  day."                                                asjo@koldfront.dk



^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Keeping IMAP connection alive when using it in mail-sources
  2023-01-10 19:24   ` Adam Sjøgren
@ 2023-01-10 20:08     ` Emanuel Berg
  2023-01-14 21:29     ` Adam Sjøgren
  1 sibling, 0 replies; 7+ messages in thread
From: Emanuel Berg @ 2023-01-10 20:08 UTC (permalink / raw)
  To: info-gnus-english

Adam Sjøgren wrote:

> Here's the modified version of mail-source-fetch-imap
> I cobbled together - the handling of deleting the buffer if
> there is no process is not so pretty:
>
> (defun mail-source-fetch-imap (source callback)
>   "Fetcher for imap sources."
>   (mail-source-bind (imap source)
>     (mail-source-run-script
>      prescript
>      `((?p . ,password) (?t . ,mail-source-crash-box)
>        (?s . ,server) (?P . ,port) (?u . ,user))
>      prescript-delay)
>     (let ((from (format "%s:%s:%s" server user port))
> 	  (found 0)
>           (imap-shell-program (or (list program) imap-shell-program)))
>       (let ((buf (or (and (or (get-buffer-process " *imap source*")
>                               (and (get-buffer " *imap source*")
>                                    (kill-buffer " *imap source*")))
>                           (get-buffer " *imap source*"))
>                      (let ((newbuf (generate-new-buffer " *imap source*")))
> 	               (if (and (imap-open server port stream authentication newbuf)
> 	                        (imap-authenticate
> 		                 user (or (cdr (assoc from mail-source-password-cache))
>                                           password)
>                                  newbuf))
>                            newbuf
>                          (progn
>                            (imap-close newbuf)
> 	                   ;; We nix out the password in case the error
> 	                   ;; was because of a wrong password being given.
> 	                   (setq mail-source-password-cache
> 	                         (delq (assoc from mail-source-password-cache)
> 		                       mail-source-password-cache))
> 	                   (error "IMAP error: %s" (imap-error-text newbuf))))))))
>         (let ((mailbox-list (if (listp mailbox) mailbox (list mailbox))))
>           (dolist (mailbox mailbox-list)
>             (when (imap-mailbox-select mailbox nil buf)
> 	      (let ((coding-system-for-write
>                      mail-source-imap-file-coding-system)
> 	            (mail-source-string (format "imap:%s:%s" server mailbox))
> 	            str remove)
>                 (message "Fetching from %s..." mailbox)
> 	        (with-temp-file mail-source-crash-box
> 	          ;; Avoid converting 8-bit chars from inserted strings to
> 	          ;; multibyte.
> 	          (mm-disable-multibyte)
> 	          ;; remember password
> 	          (with-current-buffer buf
> 		    (when (and imap-password
> 			       (not (member (cons from imap-password)
>                                             mail-source-password-cache)))
> 		      (push (cons from imap-password) mail-source-password-cache)))
> 	          ;; if predicate is nil, use all uids
> 	          (dolist (uid (imap-search (or predicate "1:*") buf))
> 		    (when (setq str
> 			        (if (imap-capability 'IMAP4rev1 buf)
> 				    (caddar (imap-fetch uid "BODY.PEEK[]"
> 						        'BODYDETAIL nil buf))
> 			          (imap-fetch uid "RFC822.PEEK" 'RFC822 nil buf)))
> 		      (push uid remove)
> 		      (insert "From imap " (current-time-string) "\n")
> 		      (save-excursion
> 		        (insert str "\n\n"))
> 		      (while (let ((case-fold-search nil))
> 			       (re-search-forward "^From " nil t))
> 		        (replace-match ">From "))
> 		      (goto-char (point-max))))
> 	          (nnheader-ms-strip-cr))
> 	        (cl-incf found (mail-source-callback callback server))
> 	        (mail-source-delete-crash-box)
> 	        (when (and remove fetchflag)
> 	          (setq remove (nreverse remove))
> 	          (imap-message-flags-add
> 	           (imap-range-to-message-set (gnus-compress-sequence remove))
> 	           fetchflag nil buf))
> 	        (if dontexpunge
> 		    (imap-mailbox-unselect buf)
>                   (imap-mailbox-close nil buf))))))
>         (mail-source-run-script
>          postscript
>          `((?p . ,password) (?t . ,mail-source-crash-box)
>            (?s . ,server) (?P . ,port) (?u . ,user)))
>         found))))

Good work but consider writing longer functions?

-- 
underground experts united
https://dataswamp.org/~incal



^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Keeping IMAP connection alive when using it in mail-sources
  2023-01-10 19:24   ` Adam Sjøgren
  2023-01-10 20:08     ` Emanuel Berg
@ 2023-01-14 21:29     ` Adam Sjøgren
  2023-01-15 19:55       ` Adam Sjøgren
  1 sibling, 1 reply; 7+ messages in thread
From: Adam Sjøgren @ 2023-01-14 21:29 UTC (permalink / raw)
  To: info-gnus-english

Adam writes:

> Here's the modified version of mail-source-fetch-imap I cobbled
> together [...]

I have been running this for the past 4 days without a single hiccup(!)

If anyone else is using IMAP as POP, it's worth trying.


  Best regards,

    Adam

-- 
 "This piece is about the inadequacy of traditional         Adam Sjøgren
  imagery and symbols to convey meaning in today's     asjo@koldfront.dk
  world."



^ permalink raw reply	[flat|nested] 7+ messages in thread

* Re: Keeping IMAP connection alive when using it in mail-sources
  2023-01-14 21:29     ` Adam Sjøgren
@ 2023-01-15 19:55       ` Adam Sjøgren
  0 siblings, 0 replies; 7+ messages in thread
From: Adam Sjøgren @ 2023-01-15 19:55 UTC (permalink / raw)
  To: info-gnus-english

Adam writes:

> Adam writes:
>
>> Here's the modified version of mail-source-fetch-imap I cobbled
>> together [...]
>
> I have been running this for the past 4 days without a single hiccup(!)

One thing to fix is to make the process buffer name unique to the
mail-sources entry, so if you have multiple imap accounts in your
mail-sources they get a process buffer each.

I think.


  Best regards,

    Adam

-- 
 "Du danser godt, men ude af takt"                          Adam Sjøgren
                                                       asjo@koldfront.dk



^ permalink raw reply	[flat|nested] 7+ messages in thread

* Keeping IMAP connection alive when using it in mail-sources
@ 2023-01-07 23:49 Adam Sjøgren
  0 siblings, 0 replies; 7+ messages in thread
From: Adam Sjøgren @ 2023-01-07 23:49 UTC (permalink / raw)
  To: info-gnus-english

  Hi,


I was wondering if there is a way to use IMAP in mail-sources ("as
POP"), but without closing the connection after every
gnus-group-get-new-news?

All my nntp-connections are kept alive and pressing "g" in the *Group*
buffer is pretty fast, except for the tls connection and login on the
IMAP-server that happens every time.

It would be nice if Gnus would just keep the connection open.


  Best regards,

    Adam

(I use IMAP "as POP" because I like having all my email locally, indexed
by notmuch, with local spam-classification by crm114 and so on and so
forth.)

-- 
 "I am not tropical. I am not a damn toucan."               Adam Sjøgren
                                                       asjo@koldfront.dk



^ permalink raw reply	[flat|nested] 7+ messages in thread

end of thread, other threads:[~2023-01-15 19:55 UTC | newest]

Thread overview: 7+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2023-01-08 18:10 Keeping IMAP connection alive when using it in mail-sources Eric Abrahamsen
2023-01-10 18:18 ` Adam Sjøgren
2023-01-10 19:24   ` Adam Sjøgren
2023-01-10 20:08     ` Emanuel Berg
2023-01-14 21:29     ` Adam Sjøgren
2023-01-15 19:55       ` Adam Sjøgren
  -- strict thread matches above, loose matches on Subject: below --
2023-01-07 23:49 Adam Sjøgren

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).