Getting gpg (or is it pgg?) to play nice with proxy

Getting gpg (or is it pgg?) to play nice with proxy

[Niklas Morberg]

Inspired by Jesper Harder's info on how to automatically
verify inline gpg signatures I decided to have a go. I got
pretty much everything set up, except I can't get gpg to
fetch keys from a keyserver through a proxy.

If I do `M-x shell-command' and execute:

gpg --recv-keys 0x80808080

I get the reply:

gpg: requesting key 80808080 from search.keyserver.net
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0

Which is fine since the key is invalid anyway. But if I try
to verify a signed message from within gnus the request
times out. I've tried to change the keyserver to a local one
inside the firewall and then it works. The local keyserver
only holds keys for local users, so it is not that useful
for me.

I've modified gpg.conf to use the http_proxy (and since it
works from a shell within emacs I guess it is properly set
up).

Some information that might be of help:

I'm running cygwin as my shell and unless I have:

(setq shell-command-switch "-ic")

the `M-x shell-command' also times out. It still works if I
run it in a shell in emacs (`M-x shell' and then the gpg
command.)

Niklas

Re: Getting gpg (or is it pgg?) to play nice with proxy

[Niklas Morberg]

Niklas Morberg <niklas@axis.com> writes:

> If I do `M-x shell-command' and execute:
>
> gpg --recv-keys 0x80808080

[snip] This works fine.

> But if I try to verify a signed message from within gnus
> the request times out. I've tried to change the keyserver
> to a local one inside the firewall and then it works.

[snip]

> I'm running cygwin as my shell and unless I have:
> 
> (setq shell-command-switch "-ic")
> 
> the `M-x shell-command' also times out. It still works if I
> run it in a shell in emacs (`M-x shell' and then the gpg
> command.)

Ok, I think I have found the problem. I would need some help
to find a solution.

In pgg-gpg.el, the function pgg-gpg-process-region uses
call-process-region. My guess is that call-process-region
does not use my proxy settings (defined in .emacs with
setenv).

Would it be possible to use shell-command-on-region instead?
I think that this would work better, but I don't quite know
how to change the function to test this as s-c-o-r takes
different arguments than call-process-region.

Niklas

Re: Getting gpg (or is it pgg?) to play nice with proxy

[Niklas Morberg]

Niklas Morberg <niklas@axis.com> writes:

> Ok, I think I have found the problem. I would need some help
> to find a solution.

No takers?

> In pgg-gpg.el, the function pgg-gpg-process-region uses
> call-process-region. My guess is that call-process-region
> does not use my proxy settings (defined in .emacs with
> setenv).
>
> Would it be possible to use shell-command-on-region instead?
> I think that this would work better, but I don't quite know
> how to change the function to test this as s-c-o-r takes
> different arguments than call-process-region.

Would using shell-command-on-region break something else?

Niklas

[PATCH] Use shell-command-on-region in pgg-gpg.el (was: Getting gpg (or is it pgg?) to play nice with proxy)

[Niklas Morberg]

[-- Attachment #1: Type: text/plain, Size: 724 bytes --]

Niklas Morberg <niklas@axis.com> writes:

>> In pgg-gpg.el, the function pgg-gpg-process-region uses
>> call-process-region. My guess is that call-process-region
>> does not use my proxy settings (defined in .emacs with
>> setenv).
>>
>> Would it be possible to use shell-command-on-region instead?
>> I think that this would work better, but I don't quite know
>> how to change the function to test this as s-c-o-r takes
>> different arguments than call-process-region.
>
> Would using shell-command-on-region break something else?

With the following patch it is possible to get keys from
servers through my proxy. The implementation probably sucks,
but it works on my system.

Does this break anything on other systems?


[-- Warning: decoded text below may be mangled, UTF-8 assumed --]
[-- Attachment #2: Use shell-command-on-region in pgg-gpg-process-region --]
[-- Type: text/x-patch, Size: 910 bytes --]

cd h:/pgnus/lisp/
diff -u -b "h:/pgnus/lisp/pgg-gpg.el.orig" "h:/pgnus/lisp/pgg-gpg.el"
--- h:/pgnus/lisp/pgg-gpg.el.orig	2003-05-09 11:31:24.000000000 +0200
+++ h:/pgnus/lisp/pgg-gpg.el	2003-05-09 11:31:45.000000000 +0200
@@ -73,8 +73,12 @@
                 (insert passphrase "\n"))
               (insert input)
               (setq exit-status
-                    (apply #'call-process-region (point-min) (point-max) program
-                           nil errors-buffer nil args))))
+                    (shell-command-on-region (point-min) (point-max)
+					     ;; call program with arguments
+					     (let ((value program))
+					       (dolist (element args value)
+						 (setq value (concat value " " element))))
+					     nil nil errors-buffer))))
 	  (with-current-buffer (get-buffer-create output-buffer)
 	    (buffer-disable-undo)
 	    (erase-buffer)

Diff finished at Fri May 09 11:31:56

[-- Attachment #3: Type: text/plain, Size: 221 bytes --]


I haven't been able to test the original implementation
(since call-process-region does not use my proxy settings)
and don't know if the output from the command is supposed to
pop up in a new frame. Now it does.

Niklas

Re: [PATCH] Use shell-command-on-region in pgg-gpg.el

[Lars Magne Ingebrigtsen]

Niklas Morberg <niklas.morberg@axis.com> writes:

>>> In pgg-gpg.el, the function pgg-gpg-process-region uses
>>> call-process-region. My guess is that call-process-region
>>> does not use my proxy settings (defined in .emacs with
>>> setenv).

I'm kinda surprised that that would make any difference.  If the
command uses environment variables, it should heed them in any case,
shouldn't it?

> I haven't been able to test the original implementation
> (since call-process-region does not use my proxy settings)
> and don't know if the output from the command is supposed to
> pop up in a new frame. Now it does.

It's not supposed to do that, which is why one doesn't use
`shell-command*' in programs.

-- 
(domestic pets only, the antidote for overdose, milk.)
   larsi@gnus.org * Lars Magne Ingebrigtsen

Re: [PATCH] Use shell-command-on-region in pgg-gpg.el

[Niklas Morberg]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> Niklas Morberg <niklas.morberg@axis.com> writes:
>
>>>> In pgg-gpg.el, the function pgg-gpg-process-region uses
>>>> call-process-region. My guess is that call-process-region
>>>> does not use my proxy settings (defined in .emacs with
>>>> setenv).
>
> I'm kinda surprised that that would make any difference.  If the
> command uses environment variables, it should heed them in any case,
> shouldn't it?

There certainly is a difference, at least on my system
(cygwin). I don't know why. The following small test case,
using search.keyserver.net which is only available through
my proxy, shows it:

Evaling the following in the *scratch* buffer: 

  (shell-command-on-region (point) (point) "gpg --recv-keys 0x80808080" t)

Returns immediately with the following result:

gpg: requesting key 80808080 from search.keyserver.net
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0

Evaling the following in the *scratch* buffer: 

  (call-process-region (point) (point) "gpg" nil t nil "--recv-keys" "0x80808080")

Hangs for a while and then returns:

gpg: requesting key 80808080 from search.keyserver.net
gpg: can't get key from keyserver: Connection timed out
gpg: Total number processed: 0

If I do the same but use a local keyserver (by modifying
~/.gnupg/gpg.conf) both functions return immediately with
the identical output:

gpg: requesting key 80808080 from keys.axis.se
gpg: no valid OpenPGP data found.
gpg: Total number processed: 0

Niklas

Re: [PATCH] Use shell-command-on-region in pgg-gpg.el

[Simon Josefsson]

Niklas Morberg <niklas.morberg@axis.com> writes:

> Evaling the following in the *scratch* buffer: 
>
>   (shell-command-on-region (point) (point) "gpg --recv-keys 0x80808080" t)
>
> Returns immediately with the following result:
>
> gpg: requesting key 80808080 from search.keyserver.net
> gpg: no valid OpenPGP data found.
> gpg: Total number processed: 0
>
> Evaling the following in the *scratch* buffer: 
>
>   (call-process-region (point) (point) "gpg" nil t nil "--recv-keys" "0x80808080")
>
> Hangs for a while and then returns:
>
> gpg: requesting key 80808080 from search.keyserver.net
> gpg: can't get key from keyserver: Connection timed out
> gpg: Total number processed: 0
>
> If I do the same but use a local keyserver (by modifying
> ~/.gnupg/gpg.conf) both functions return immediately with
> the identical output:
>
> gpg: requesting key 80808080 from keys.axis.se
> gpg: no valid OpenPGP data found.
> gpg: Total number processed: 0

Perhaps you could attach to the gpg process with a debugger to see
what it is doing, and why it isn't talking through your HTTP proxy?
Or if you have strace or similar, it is probably easier to replace
"gpg" above with "strace" (and make "gpg" a paramter).

Asking on a Windows-Emacs mailing list might help, this sounds like a
Windows-specific problem.

Re: [PATCH] Use shell-command-on-region in pgg-gpg.el

[Niklas Morberg]

Simon Josefsson <jas@extundo.com> writes:

> Perhaps you could attach to the gpg process with a debugger to see
> what it is doing, and why it isn't talking through your HTTP proxy?

Sorry, that's way out of my league (I haven't been working
with gdb for years).

> Or if you have strace or similar, it is probably easier to replace
> "gpg" above with "strace" (and make "gpg" a paramter).

It turns out I have strace. No idea what it is. I'll take a
look.

> Asking on a Windows-Emacs mailing list might help, this sounds like a
> Windows-specific problem.

I used `M-x report-emacs-bug' and sent a bug report. Are
there another address I should consider sending this
problem to?

Thanks for all the help, I won't be bothering the ding list
with this anymore :)

Niklas

Re: [PATCH] Use shell-command-on-region in pgg-gpg.el

[Simon Josefsson]

Niklas Morberg <niklas.morberg@axis.com> writes:

>> Or if you have strace or similar, it is probably easier to replace
>> "gpg" above with "strace" (and make "gpg" a paramter).
>
> It turns out I have strace. No idea what it is. I'll take a
> look.

Invoking "strace gpg ..." should print debug information to stderr,
you should be able to see, among other things, which IP it tries to
contact.  Not sure exactly how this helps, but at least it would pin
down the problem to gpg.  Perhaps it gives some clues on the cause of
the problem.

>> Asking on a Windows-Emacs mailing list might help, this sounds like a
>> Windows-specific problem.
>
> I used `M-x report-emacs-bug' and sent a bug report. Are
> there another address I should consider sending this
> problem to?

I don't know.

Re: [PATCH] Use shell-command-on-region in pgg-gpg.el

[Dave Love]

Lars Magne Ingebrigtsen <larsi@gnus.org> writes:

> I'm kinda surprised that that would make any difference.  If the
> command uses environment variables, it should heed them in any case,
> shouldn't it?

That presumably depends on details of what you've had to do under Doze
to use shell commands (e.g. Cygwin bash).
>
>> I haven't been able to test the original implementation
>> (since call-process-region does not use my proxy settings)
>> and don't know if the output from the command is supposed to
>> pop up in a new frame. Now it does.
>
> It's not supposed to do that, which is why one doesn't use
> `shell-command*' in programs.

But one does (in various places)!  See the OUTPUT-BUFFER arg to
redirect output.

Re: Getting gpg (or is it pgg?) to play nice with proxy

[Simon Josefsson]

Niklas Morberg <niklas.morberg@axis.com> writes:

> Ok, I think I have found the problem. I would need some help
> to find a solution.
>
> In pgg-gpg.el, the function pgg-gpg-process-region uses
> call-process-region. My guess is that call-process-region
> does not use my proxy settings (defined in .emacs with
> setenv).
>
> Would it be possible to use shell-command-on-region instead?

Yes, although call-process-region should work.  The setenv I define in
.emacs seem to be visible for me within sub-processes.  I think it
would be better to first understand exactly why call-process-region
doesn't work for you.

Try evaluate the following

(progn
  (setenv "FOO" "bar")
  (call-process-region (point-min) (point-max) shell-file-name
		       nil (current-buffer) nil shell-command-switch "set"))

I get:

BASH=/bin/bash
BASH_VERSINFO=([0]="2" [1]="05a" [2]="0" [3]="1" [4]="release" [5]="i386-pc-linux-gnu")
BASH_VERSION='2.05a.0(1)-release'
COLORTERM=gnome-terminal
CVS_RSH=ssh
DESKTOP_STARTUP_ID=
DIRSTACK=()
DISPLAY=:0.0
EUID=22541
FOO=bar
...

Is it possible that your shell somehow alters the http proxy variable?

Re: Getting gpg (or is it pgg?) to play nice with proxy

[Niklas Morberg]

Simon Josefsson <jas@extundo.com> writes:

> Niklas Morberg <niklas.morberg@axis.com> writes:
>
> I think it would be better to first understand exactly why
> call-process-region doesn't work for you.

Agreed.

> Try evaluate the following
>
> (progn
>   (setenv "FOO" "bar")
>   (call-process-region (point-min) (point-max) shell-file-name
> 		       nil (current-buffer) nil shell-command-switch "set"))

Not quite sure what's important here, but:

BASH=/usr/bin/bash
BASH_VERSINFO=([0]="2" [1]="05b" [2]="0" [3]="9" [4]="release" [5]="i686-pc-cygwin")
BASH_VERSION='2.05b.0(9)-release'
...
FOO=bar
...
HTTP_PROXY=[my-proxy-server]

The value for HTTP_PROXY is correct.

> Is it possible that your shell somehow alters the http proxy variable?

Certainly, but I don't know how to verify it. As I said in
my original mail: 
<URL:
http://article.gmane.org/gmane.emacs.gnus.general/50816> the
proxy works for both `M-x shell-command' and within the emacs
shell.

Niklas

Re: Getting gpg (or is it pgg?) to play nice with proxy

[Simon Josefsson]

Niklas Morberg <niklas.morberg@axis.com> writes:

> FOO=bar
> ...
> HTTP_PROXY=[my-proxy-server]
>
> The value for HTTP_PROXY is correct.

So it looks likes applications invoked via `call-process-region' see
the same environment variables as `shell-command-on-region', so there
must be something else that is different.  What does gpg output when
you run it in call-process-region vs shell-command-on-region via PGG?
I think there is a buffer *PGG output*, or something, that contains
the output.  It seems weird that one function works and the other
doesn't.  Perhaps gpg condition key fetching on something that depend
on how it is invoked?

Re: Getting gpg (or is it pgg?) to play nice with proxy

[Niklas Morberg]

Simon Josefsson <jas@extundo.com> writes:

> What does gpg output when you run it in call-process-region

Signature made Tue May 13 07:54:14 2003 WEDT using DSA key ID 34309C41
requesting key 34309C41 from search.keyserver.net
can't get key from keyserver: Connection timed out
Total number processed: 0
Can't check signature: public key not found

> vs shell-command-on-region via PGG? 

Signature made Tue May 13 07:54:14 2003 WEDT using DSA key ID 34309C41
requesting key 34309C41 from search.keyserver.net
key 34309C41: public key "Graham Murray <graham@gmurray.org.uk>" imported
Total number processed: 1
              imported: 1
Good signature from "Graham Murray <graham@gmurray.org.uk>"
                aka "Graham Murray <graham@barnowl.demon.co.uk>"
                aka "Graham Murray <gmurray@cix.compulink.co.uk>"
checking the trustdb
checking at depth 0 signed=0 ot(-/q/n/m/f/u)=0/0/0/0/0/1
WARNING: This key is not certified with a trusted signature!
         There is no indication that the signature belongs to the owner.

> I think there is a buffer *PGG output*, or something, that
> contains the output. It seems weird that one function
> works and the other doesn't. 

In the call-process-region version, emacs hangs for a while
until the request has timed out. The shell-command-on-region
works immediately.

> Perhaps gpg condition key fetching on something that
> depend on how it is invoked?

I have no idea what's going on here. I know that one works and
the other doesn't. The only difference between the two cases
above is the patch I included in <uy91go3o2.fsf_-_@axis.com>
(that Message-ID looks weird, doesn't it?).

In both cases I pressed the middle mouse button on the
button saying:

[[PGP Signed Part:Undecided]]

to verify the message.

I saw the problem in emacs 21.2 and I'm seeing it now in
emacs from CVS at the end of April/beginning of May.

Niklas