Supporting git access via smart HTTPS protocol for musl-libc

Supporting git access via smart HTTPS protocol for musl-libc

[vlse]

Hello,

Would musl-libc support git access via smart HTTPS protocol.
As git man page says as well as stackoverflow site that using git protocol
is fine for lan operations.
But for internet git access, either ssh or https smart protocol use
is necessary to prevent man in the middle attack.

Please consider giving secure git access. Also smart http/s protocol
is way better than dumb protocol. It avoids downloading too much data
again and also shows progress and stats.

Regards,
Veera

Re: Supporting git access via smart HTTPS protocol for musl-libc

[A. Wilcox]

[-- Attachment #1.1: Type: text/plain, Size: 1986 bytes --]

On 03/25/19 20:09, vlse wrote:
> Hello,
> 
> Would musl-libc support git access via smart HTTPS protocol.
> As git man page says as well as stackoverflow site that using git protocol
> is fine for lan operations.
> But for internet git access, either ssh or https smart protocol use
> is necessary to prevent man in the middle attack.


This is more an argument for signing commits so that they are
cryptographically provable.  HTTPS is trivial to MITM, especially for
the kind of actors that would care enough to MITM musl at all.

Threat models, people.


> Please consider giving secure git access. Also smart http/s protocol
> is way better than dumb protocol. It avoids downloading too much data
> again and also shows progress and stats.


There is absolutely no difference in transmitted data between the Git
protocol and the HTTP Git transport, other than the useless overhead of
HTTP messages, which actually skews favour towards the Git protocol.
Also, the Git protocol is in my experience much much faster.

The Git transport definitely can show progress and stats, the same as
the HTTP transport:

awilcox on gwyn [pts/18 Mon 25 20:13] ~: git clone
git://git.musl-libc.org/musl
Cloning into 'musl'...
remote: Counting objects: 31396, done.
remote: Compressing objects: 100% (12589/12589), done.
remote: Total 31396 (delta 22605), reused 25698 (delta 18440)
Receiving objects: 100% (31396/31396), 4.77 MiB | 3.17 MiB/s, done.
Resolving deltas: 100% (22605/22605), done.


(It did show the progress as it was downloading, but since I am on a
fairly fast link, I couldn't copy it.)

Personally I would be okay with musl offering an HTTP(S) transport as an
option, but please do not take away the Git transport.  It is much
faster in my experience.  Every second wasted on stupid HTTP traffic is
a second of my life I can't get back.

--arw

-- 
A. Wilcox (awilfox)
Project Lead, Adélie Linux
https://www.adelielinux.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Supporting git access via smart HTTPS protocol for musl-libc

[Rich Felker]

On Mon, Mar 25, 2019 at 08:17:26PM -0500, A. Wilcox wrote:
> On 03/25/19 20:09, vlse wrote:
> > Hello,
> > 
> > Would musl-libc support git access via smart HTTPS protocol.
> > As git man page says as well as stackoverflow site that using git protocol
> > is fine for lan operations.
> > But for internet git access, either ssh or https smart protocol use
> > is necessary to prevent man in the middle attack.
> 
> This is more an argument for signing commits so that they are
> cryptographically provable.  HTTPS is trivial to MITM, especially for
> the kind of actors that would care enough to MITM musl at all.
> 
> Threat models, people.

The request is reasonable. HTTPS is *not* "trivial to MITM", and
essentially impossible to do so without detection and a trail of
responsibility, especially now that CT logs are a thing. However,
until breaking sha1 (much worse than it's broken now) is practical,
you can also verify authenticity of a git repo via "git fsck" and a
known good source of the commit hash (e.g. cgit over https).

> > Please consider giving secure git access. Also smart http/s protocol
> > is way better than dumb protocol. It avoids downloading too much data
> > again and also shows progress and stats.
> 
> 
> There is absolutely no difference in transmitted data between the Git
> protocol and the HTTP Git transport, other than the useless overhead of
> HTTP messages, which actually skews favour towards the Git protocol.
> Also, the Git protocol is in my experience much much faster.
> 
> The Git transport definitely can show progress and stats, the same as
> the HTTP transport:
> 
> awilcox on gwyn [pts/18 Mon 25 20:13] ~: git clone
> git://git.musl-libc.org/musl
> Cloning into 'musl'...
> remote: Counting objects: 31396, done.
> remote: Compressing objects: 100% (12589/12589), done.
> remote: Total 31396 (delta 22605), reused 25698 (delta 18440)
> Receiving objects: 100% (31396/31396), 4.77 MiB | 3.17 MiB/s, done.
> Resolving deltas: 100% (22605/22605), done.
> 
> 
> (It did show the progress as it was downloading, but since I am on a
> fairly fast link, I couldn't copy it.)
> 
> Personally I would be okay with musl offering an HTTP(S) transport as an
> option, but please do not take away the Git transport.  It is much
> faster in my experience.  Every second wasted on stupid HTTP traffic is
> a second of my life I can't get back.

Of course the git transport won't be taken away. I'd like to add https
support, but I'm not sure how to do it without a nasty bloated httpd
that would increase server resource requirements by 1-2 orders of
magnitude. If anyone knows a way to hook up thttpd to it, I'll give it
a try.

Rich

Re: Supporting git access via smart HTTPS protocol for musl-libc

[vlse]

Hi,

On Mon, Mar 25, 2019 at 08:17:26PM -0500, A. Wilcox wrote:
> On 03/25/19 20:09, vlse wrote:
> > Hello,
> > 
> > Would musl-libc support git access via smart HTTPS protocol.
> > As git man page says as well as stackoverflow site that using git protocol
> > is fine for lan operations.
> > But for internet git access, either ssh or https smart protocol use
> > is necessary to prevent man in the middle attack.
> 
> This is more an argument for signing commits so that they are
> cryptographically provable.  HTTPS is trivial to MITM, especially for
> the kind of actors that would care enough to MITM musl at all.
> 

How HTTPS is trivial to MITM.
If we have secure certificates from musl site, then MITM is not possible.
And many important transactions in the world depend on https.

> 
> > Please consider giving secure git access. Also smart http/s protocol
> > is way better than dumb protocol. It avoids downloading too much data
> > again and also shows progress and stats.
> 
> There is absolutely no difference in transmitted data between the Git
> protocol and the HTTP Git transport, other than the useless overhead of
> HTTP messages, which actually skews favour towards the Git protocol.
> Also, the Git protocol is in my experience much much faster.
> 

Yeah I know git protocol is faster.
And http/s does have little more overhead.
For HTTP/S there are two protocols dumb and smart.
And here in it are the differences.

> 
> The Git transport definitely can show progress and stats, the same as
> the HTTP transport:
> 
> 
> awilcox on gwyn [pts/18 Mon 25 20:13] ~: git clone
> git://git.musl-libc.org/musl
> Cloning into 'musl'...
> remote: Counting objects: 31396, done.
> remote: Compressing objects: 100% (12589/12589), done.
> 

I was not comparing http/s with git protocol. But http/s smart with http/s dumb protocol.
Dumb protolcol does not shows progress and stats.

Also dumb protocol of http/s has more overhead for subsequent git pull or fetch.

> 
> Personally I would be okay with musl offering an HTTP(S) transport as an
> option, but please do not take away the Git transport.  It is much
> faster in my experience.  Every second wasted on stupid HTTP traffic is
> a second of my life I can't get back.
> 

I was not requesting to stop git protocol. All I wanted was addition of https
smart protocol for additional security and safety for git access in internet.

> --arw
> A. Wilcox (awilfox)
> Project Lead, Adélie Linux
> https://www.adelielinux.org
> 

Regards,
Veera

Re: Supporting git access via smart HTTPS protocol for musl-libc

[vlse]

Hi,

On Mon, Mar 25, 2019 at 09:37:06PM -0400, Rich Felker wrote:
> On Mon, Mar 25, 2019 at 08:17:26PM -0500, A. Wilcox wrote:
> > On 03/25/19 20:09, vlse wrote:
> > > Hello,
> > > 
> > > Would musl-libc support git access via smart HTTPS protocol.
> > > As git man page says as well as stackoverflow site that using git protocol
> > > is fine for lan operations.
> > > But for internet git access, either ssh or https smart protocol use
> > > is necessary to prevent man in the middle attack.
> > 
> > This is more an argument for signing commits so that they are
> > cryptographically provable.  HTTPS is trivial to MITM, especially for
> > the kind of actors that would care enough to MITM musl at all.
> > 
> > Threat models, people.
> 
> The request is reasonable. HTTPS is *not* "trivial to MITM", and
> essentially impossible to do so without detection and a trail of
> responsibility, especially now that CT logs are a thing. However,
> until breaking sha1 (much worse than it's broken now) is practical,
> you can also verify authenticity of a git repo via "git fsck" and a
> known good source of the commit hash (e.g. cgit over https).
> 

Yes. cgit over https. We need a secure start first.

> > > Please consider giving secure git access. Also smart http/s protocol
> > > is way better than dumb protocol. It avoids downloading too much data
> > > again and also shows progress and stats.
> > 
> > 
> > There is absolutely no difference in transmitted data between the Git
> > protocol and the HTTP Git transport, other than the useless overhead of
> > HTTP messages, which actually skews favour towards the Git protocol.
> > Also, the Git protocol is in my experience much much faster.
> > 
> > The Git transport definitely can show progress and stats, the same as
> > the HTTP transport:
> > 
> > awilcox on gwyn [pts/18 Mon 25 20:13] ~: git clone
> > git://git.musl-libc.org/musl
> > Cloning into 'musl'...
> > remote: Counting objects: 31396, done.
> > Receiving objects: 100% (31396/31396), 4.77 MiB | 3.17 MiB/s, done.
> > Resolving deltas: 100% (22605/22605), done.
> > 
> > Personally I would be okay with musl offering an HTTP(S) transport as an
> > option, but please do not take away the Git transport.  It is much
> > faster in my experience.  Every second wasted on stupid HTTP traffic is
> > a second of my life I can't get back.
> 
> Of course the git transport won't be taken away. I'd like to add https
> support, but I'm not sure how to do it without a nasty bloated httpd
> that would increase server resource requirements by 1-2 orders of
> magnitude. If anyone knows a way to hook up thttpd to it, I'll give it
> a try.

Nginx is bloat free I think. But perhaps not in comparison to thttpd.
I will look how to support cgit http/s with thttpd using a hook.

At skarnet.org, the author is using busybox httpd with cgi support and
cgit cgi hooks to give http/s git access.

> 
> Rich

Regards,
Veera

Re: Supporting git access via smart HTTPS protocol for musl-libc

[A. Wilcox]

[-- Attachment #1.1: Type: text/plain, Size: 3277 bytes --]

On 03/25/19 20:43, vlse wrote:
> Hi,
> 
> On Mon, Mar 25, 2019 at 08:17:26PM -0500, A. Wilcox wrote:
>> On 03/25/19 20:09, vlse wrote:
>>> Hello,
>>>
>>> Would musl-libc support git access via smart HTTPS protocol.
>>> As git man page says as well as stackoverflow site that using git protocol
>>> is fine for lan operations.
>>> But for internet git access, either ssh or https smart protocol use
>>> is necessary to prevent man in the middle attack.
>>
>> This is more an argument for signing commits so that they are
>> cryptographically provable.  HTTPS is trivial to MITM, especially for
>> the kind of actors that would care enough to MITM musl at all.
>>
> 
> How HTTPS is trivial to MITM.


I was under the (apparently false) impression that Git does not do host
name nor CA validation.  That is, as long as it can successfully connect
to port 443 with TLS, the connection will succeed.  This was an issue a
long time ago especially in Windows Git builds, but it looks like the
default for http.sslVerify is now 'true' instead of 'false' (at least on
Unix) so you may disregard that.  I apologise.


> If we have secure certificates from musl site, then MITM is not possible.
> And many important transactions in the world depend on https.
> 
>>
>>> Please consider giving secure git access. Also smart http/s protocol
>>> is way better than dumb protocol. It avoids downloading too much data
>>> again and also shows progress and stats.
>>
>> There is absolutely no difference in transmitted data between the Git
>> protocol and the HTTP Git transport, other than the useless overhead of
>> HTTP messages, which actually skews favour towards the Git protocol.
>> Also, the Git protocol is in my experience much much faster.
>>
> 
> Yeah I know git protocol is faster.
> And http/s does have little more overhead.
> For HTTP/S there are two protocols dumb and smart.
> And here in it are the differences.
> 
>>
>> The Git transport definitely can show progress and stats, the same as
>> the HTTP transport:
>>
>>
>> awilcox on gwyn [pts/18 Mon 25 20:13] ~: git clone
>> git://git.musl-libc.org/musl
>> Cloning into 'musl'...
>> remote: Counting objects: 31396, done.
>> remote: Compressing objects: 100% (12589/12589), done.
>>
> 
> I was not comparing http/s with git protocol. But http/s smart with http/s dumb protocol.
> Dumb protolcol does not shows progress and stats.


I'm sorry, I misunderstood.


> Also dumb protocol of http/s has more overhead for subsequent git pull or fetch.
> 
>>
>> Personally I would be okay with musl offering an HTTP(S) transport as an
>> option, but please do not take away the Git transport.  It is much
>> faster in my experience.  Every second wasted on stupid HTTP traffic is
>> a second of my life I can't get back.
>>
> 
> I was not requesting to stop git protocol. All I wanted was addition of https
> smart protocol for additional security and safety for git access in internet.


That sounds reasonable.

Again, I apologise for misunderstanding.  I had a rough weekend and
probably shouldn't have been writing emails.

Best to you and yours,
--arw

-- 
A. Wilcox (awilfox)
Project Lead, Adélie Linux
https://www.adelielinux.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Supporting git access via smart HTTPS protocol for musl-libc

[Rich Felker]

On Tue, Mar 26, 2019 at 07:24:35AM +0530, vlse wrote:
> Hi,
> 
> On Mon, Mar 25, 2019 at 09:37:06PM -0400, Rich Felker wrote:
> > On Mon, Mar 25, 2019 at 08:17:26PM -0500, A. Wilcox wrote:
> > > On 03/25/19 20:09, vlse wrote:
> > > > Hello,
> > > > 
> > > > Would musl-libc support git access via smart HTTPS protocol.
> > > > As git man page says as well as stackoverflow site that using git protocol
> > > > is fine for lan operations.
> > > > But for internet git access, either ssh or https smart protocol use
> > > > is necessary to prevent man in the middle attack.
> > > 
> > > This is more an argument for signing commits so that they are
> > > cryptographically provable.  HTTPS is trivial to MITM, especially for
> > > the kind of actors that would care enough to MITM musl at all.
> > > 
> > > Threat models, people.
> > 
> > The request is reasonable. HTTPS is *not* "trivial to MITM", and
> > essentially impossible to do so without detection and a trail of
> > responsibility, especially now that CT logs are a thing. However,
> > until breaking sha1 (much worse than it's broken now) is practical,
> > you can also verify authenticity of a git repo via "git fsck" and a
> > known good source of the commit hash (e.g. cgit over https).
> > 
> 
> Yes. cgit over https. We need a secure start first.
> 
> > > > Please consider giving secure git access. Also smart http/s protocol
> > > > is way better than dumb protocol. It avoids downloading too much data
> > > > again and also shows progress and stats.
> > > 
> > > 
> > > There is absolutely no difference in transmitted data between the Git
> > > protocol and the HTTP Git transport, other than the useless overhead of
> > > HTTP messages, which actually skews favour towards the Git protocol.
> > > Also, the Git protocol is in my experience much much faster.
> > > 
> > > The Git transport definitely can show progress and stats, the same as
> > > the HTTP transport:
> > > 
> > > awilcox on gwyn [pts/18 Mon 25 20:13] ~: git clone
> > > git://git.musl-libc.org/musl
> > > Cloning into 'musl'...
> > > remote: Counting objects: 31396, done.
> > > Receiving objects: 100% (31396/31396), 4.77 MiB | 3.17 MiB/s, done.
> > > Resolving deltas: 100% (22605/22605), done.
> > > 
> > > Personally I would be okay with musl offering an HTTP(S) transport as an
> > > option, but please do not take away the Git transport.  It is much
> > > faster in my experience.  Every second wasted on stupid HTTP traffic is
> > > a second of my life I can't get back.
> > 
> > Of course the git transport won't be taken away. I'd like to add https
> > support, but I'm not sure how to do it without a nasty bloated httpd
> > that would increase server resource requirements by 1-2 orders of
> > magnitude. If anyone knows a way to hook up thttpd to it, I'll give it
> > a try.
> 
> Nginx is bloat free I think. But perhaps not in comparison to thttpd.
> I will look how to support cgit http/s with thttpd using a hook.
> 
> At skarnet.org, the author is using busybox httpd with cgi support and
> cgit cgi hooks to give http/s git access.

OK, that sounds promising. If it can be done with cgi, it should be
easy to setup, assuming the git client is forgiving of thttpd's
slightly non-conforming cgi behavior regarding headers.

Rich

Re: Supporting git access via smart HTTPS protocol for musl-libc

[vlse]

On Mon, Mar 25, 2019 at 10:59:37PM -0400, Rich Felker wrote:
> On Tue, Mar 26, 2019 at 07:24:35AM +0530, vlse wrote:
> > Hi,
> > 
> > On Mon, Mar 25, 2019 at 09:37:06PM -0400, Rich Felker wrote:
> > > On Mon, Mar 25, 2019 at 08:17:26PM -0500, A. Wilcox wrote:
> > > > On 03/25/19 20:09, vlse wrote:
> > > > > Hello,
> > > > > 
> > > > > But for internet git access, either ssh or https smart protocol use
> > > > > is necessary to prevent man in the middle attack.
> > > > 
> > > 
> > > The request is reasonable. HTTPS is *not* "trivial to MITM", and
> > > essentially impossible to do so without detection and a trail of
> > > you can also verify authenticity of a git repo via "git fsck" and a
> > > known good source of the commit hash (e.g. cgit over https).
> > > 
> > 
> > Yes. cgit over https. We need a secure start first.
> > 
> > > > > Please consider giving secure git access. Also smart http/s protocol
> > > > > is way better than dumb protocol. It avoids downloading too much data
> > > > > again and also shows progress and stats.
> > > > 
> > > 
> > > Of course the git transport won't be taken away. I'd like to add https
> > > support, but I'm not sure how to do it without a nasty bloated httpd
> > > that would increase server resource requirements by 1-2 orders of
> > > magnitude. If anyone knows a way to hook up thttpd to it, I'll give it
> > > a try.
> > 
> > Nginx is bloat free I think. But perhaps not in comparison to thttpd.
> > I will look how to support cgit http/s with thttpd using a hook.
> > 
> > At skarnet.org, the author is using busybox httpd with cgi support and
> > cgit cgi hooks to give http/s git access.
> 
> OK, that sounds promising. If it can be done with cgi, it should be
> easy to setup, assuming the git client is forgiving of thttpd's
> slightly non-conforming cgi behavior regarding headers.
> 
> Rich

Hi,

On further enquiry I found that the latest cgit only supports dumb http protocol
for cloning or fetch. But it has option to disable the http/s cloning support,
so that another program can do it. Sorry, I was on the impression that skarnet was
supporting git http/s smart protocol by using cgit itself.

So we have to setup the git-http-backend provided with git-scm to work with web
server cgi setup. Again the examples given at upstream site is with apache.

So I am thinking of trying setting up thttpd with git-scm for support of http/s smart
protocol locally in my linux machine.

So I want to know the
Source of thttpd your are running on git.musl-libc.org
Version of thttpd

Latest upstream version: https://acme.com/software/thttpd/thttpd-2.29.tar.gz

Also I find you are providing https version of git.musl-libc.org site.
thttpd does not supports https. Are you using stunnel for it?

Let I may be of some help!

Regards,
Veera

Re: Supporting git access via smart HTTPS protocol for musl-libc

[Jens Gustedt]

[-- Attachment #1: Type: text/plain, Size: 1244 bytes --]

Hello,

On Mon, 25 Mar 2019 22:59:37 -0400 Rich Felker <dalias@libc.org> wrote:

> > Nginx is bloat free I think. But perhaps not in comparison to
> > thttpd. I will look how to support cgit http/s with thttpd using a
> > hook.
> > 
> > At skarnet.org, the author is using busybox httpd with cgi support
> > and cgit cgi hooks to give http/s git access.  
> 
> OK, that sounds promising. If it can be done with cgi, it should be
> easy to setup, assuming the git client is forgiving of thttpd's
> slightly non-conforming cgi behavior regarding headers.

One thing you'd have to have in mind is to chose a server for which it
is possible to update the certificate automatically. For some time I
used mini_httpd, which is really minimal footprint, but I had to
update the letsencrypt certificate every three month manually.

Now I switched to nginx and here automatic update of the certificate
works like a charm.

Jens

-- 
:: INRIA Nancy Grand Est ::: Camus ::::::: ICube/ICPS :::
:: ::::::::::::::: office Strasbourg : +33 368854536   ::
:: :::::::::::::::::::::: gsm France : +33 651400183   ::
:: ::::::::::::::: gsm international : +49 15737185122 ::
:: http://icube-icps.unistra.fr/index.php/Jens_Gustedt ::

[-- Attachment #2: Digitale Signatur von OpenPGP --]
[-- Type: application/pgp-signature, Size: 195 bytes --]

Re: Supporting git access via smart HTTPS protocol for musl-libc

[vlse]

On Tue, Mar 26, 2019 at 11:19:27AM +0100, Jens Gustedt wrote:
> Hello,
> 
> 
> On Mon, 25 Mar 2019 22:59:37 -0400 Rich Felker <dalias@libc.org> wrote:
> 
> 
> > > Nginx is bloat free I think. But perhaps not in comparison to
> > > thttpd. I will look how to support cgit http/s with thttpd using a
> > > hook.
> > > 
> > > At skarnet.org, the author is using busybox httpd with cgi support
> > > and cgit cgi hooks to give http/s git access.  
> > 
> > OK, that sounds promising. If it can be done with cgi, it should be
> > easy to setup, assuming the git client is forgiving of thttpd's
> > slightly non-conforming cgi behavior regarding headers.
> 
> 
> One thing you'd have to have in mind is to chose a server for which it
> is possible to update the certificate automatically. For some time I
> used mini_httpd, which is really minimal footprint, but I had to
> update the letsencrypt certificate every three month manually.
> 
> 
> Now I switched to nginx and here automatic update of the certificate
> works like a charm.
> 
The last time I enquired about Let's Encrypt certificates updating, I found
that it can be done with simple acme1 or acme2 clients including some based
on simple shell scripts and using automation like cron.

> 
> Jens
> 
> 
> -- 
> :: INRIA Nancy Grand Est ::: Camus ::::::: ICube/ICPS :::
> :: ::::::::::::::: office Strasbourg : +33 368854536   ::
> :: :::::::::::::::::::::: gsm France : +33 651400183   ::
> :: ::::::::::::::: gsm international : +49 15737185122 ::
> :: http://icube-icps.unistra.fr/index.php/Jens_Gustedt ::

Regards,
Veera

Re: Supporting git access via smart HTTPS protocol for musl-libc

[Laurent Bercot]

>On further enquiry I found that the latest cgit only supports dumb http protocol
>for cloning or fetch. But it has option to disable the http/s cloning support,
>so that another program can do it. Sorry, I was on the impression that skarnet was
>supporting git http/s smart protocol by using cgit itself.

No, this is much simpler than that: HTTPS on skarnet.org is supported
by having busybox httpd run under a TLS-capable superserver
(s6-tlsserver, from s6-networking, which can use BearSSL as its
crypto backend). It's literally HTTP in a TLS tunnel, and has nothing
to do with cgit or git, which are not TLS-aware at all.

Unfortunately, that solution isn't applicable to git.musl-libc.org,
because thttpd apparently insists on doing the socket listening
itself - it doesn't seem to support inetd-style, which is how
s6-tlsserver operates.

This is a direct illustration of the superior convenience of
inetd-style servers: they can be plugged with other tools in order
to achieve functionality the original author didn't plan for.

For thttpd, a different approach will be necessary, very likely
at the CGI level. Good luck, Jim.

--
Laurent

Re: Supporting git access via smart HTTPS protocol for musl-libc

[Rich Felker]

On Tue, Mar 26, 2019 at 11:19:27AM +0100, Jens Gustedt wrote:
> Hello,
> 
> On Mon, 25 Mar 2019 22:59:37 -0400 Rich Felker <dalias@libc.org> wrote:
> 
> > > Nginx is bloat free I think. But perhaps not in comparison to
> > > thttpd. I will look how to support cgit http/s with thttpd using a
> > > hook.
> > > 
> > > At skarnet.org, the author is using busybox httpd with cgi support
> > > and cgit cgi hooks to give http/s git access.  
> > 
> > OK, that sounds promising. If it can be done with cgi, it should be
> > easy to setup, assuming the git client is forgiving of thttpd's
> > slightly non-conforming cgi behavior regarding headers.
> 
> One thing you'd have to have in mind is to chose a server for which it
> is possible to update the certificate automatically. For some time I
> used mini_httpd, which is really minimal footprint, but I had to
> update the letsencrypt certificate every three month manually.
> 
> Now I switched to nginx and here automatic update of the certificate
> works like a charm.

Nothing special is needed for this; any httpd that serves from the
filesystem works fine. I already have LE on *.musl-libc.org and other
domains, using acme-tiny from cron:
https://github.com/diafygi/acme-tiny/

Rich

Re: Supporting git access via smart HTTPS protocol for musl-libc

[Rich Felker]

On Tue, Mar 26, 2019 at 03:32:46PM +0530, vlse wrote:
> On Mon, Mar 25, 2019 at 10:59:37PM -0400, Rich Felker wrote:
> > On Tue, Mar 26, 2019 at 07:24:35AM +0530, vlse wrote:
> > > Hi,
> > > 
> > > On Mon, Mar 25, 2019 at 09:37:06PM -0400, Rich Felker wrote:
> > > > On Mon, Mar 25, 2019 at 08:17:26PM -0500, A. Wilcox wrote:
> > > > > On 03/25/19 20:09, vlse wrote:
> > > > > > Hello,
> > > > > > 
> > > > > > But for internet git access, either ssh or https smart protocol use
> > > > > > is necessary to prevent man in the middle attack.
> > > > > 
> > > > 
> > > > The request is reasonable. HTTPS is *not* "trivial to MITM", and
> > > > essentially impossible to do so without detection and a trail of
> > > > you can also verify authenticity of a git repo via "git fsck" and a
> > > > known good source of the commit hash (e.g. cgit over https).
> > > > 
> > > 
> > > Yes. cgit over https. We need a secure start first.
> > > 
> > > > > > Please consider giving secure git access. Also smart http/s protocol
> > > > > > is way better than dumb protocol. It avoids downloading too much data
> > > > > > again and also shows progress and stats.
> > > > > 
> > > > 
> > > > Of course the git transport won't be taken away. I'd like to add https
> > > > support, but I'm not sure how to do it without a nasty bloated httpd
> > > > that would increase server resource requirements by 1-2 orders of
> > > > magnitude. If anyone knows a way to hook up thttpd to it, I'll give it
> > > > a try.
> > > 
> > > Nginx is bloat free I think. But perhaps not in comparison to thttpd.
> > > I will look how to support cgit http/s with thttpd using a hook.
> > > 
> > > At skarnet.org, the author is using busybox httpd with cgi support and
> > > cgit cgi hooks to give http/s git access.
> > 
> > OK, that sounds promising. If it can be done with cgi, it should be
> > easy to setup, assuming the git client is forgiving of thttpd's
> > slightly non-conforming cgi behavior regarding headers.
> > 
> > Rich
> 
> Hi,
> 
> On further enquiry I found that the latest cgit only supports dumb http protocol
> for cloning or fetch. But it has option to disable the http/s cloning support,
> so that another program can do it. Sorry, I was on the impression that skarnet was
> supporting git http/s smart protocol by using cgit itself.
> 
> So we have to setup the git-http-backend provided with git-scm to work with web
> server cgi setup. Again the examples given at upstream site is with apache.

As long as it's cgi and not sensitive to thttpd cgi conformance
issues, it should work.

> So I am thinking of trying setting up thttpd with git-scm for support of http/s smart
> protocol locally in my linux machine.
> 
> So I want to know the
> Source of thttpd your are running on git.musl-libc.org
> Version of thttpd
> 
> Latest upstream version: https://acme.com/software/thttpd/thttpd-2.29.tar.gz

It's the version in Alpine Linux which also has some significant bug
fixes on top of that.

> Also I find you are providing https version of git.musl-libc.org site.
> thttpd does not supports https. Are you using stunnel for it?

I'm presently using haproxy's TLS-layer (vs HTTPS-layer) proxying,
because stunnel suggers from a 2.5-decades-old wrong handling of TCP
connection closing that makes it unusable, and because haproxy is what
I knew at the time. I think openssl s_server could handle it too, but
might not support SNI (?). What I'd really prefer is a non-broken
stunnel workalike using BearSSL as the backend, since BearSSL is the
only non-awful TLS implementation. If anyone wants to work on
something like that I'd be happy to test and eventually dogfood it on
musl site.

Rich

Re: Supporting git access via smart HTTPS protocol for musl-libc

[Drew DeVault]

On 2019-03-26 11:04 AM, Rich Felker wrote:
> > Also I find you are providing https version of git.musl-libc.org site.
> > thttpd does not supports https. Are you using stunnel for it?
> 
> I'm presently using haproxy's TLS-layer (vs HTTPS-layer) proxying,
> because stunnel suggers from a 2.5-decades-old wrong handling of TCP
> connection closing that makes it unusable, and because haproxy is what
> I knew at the time. I think openssl s_server could handle it too, but
> might not support SNI (?). What I'd really prefer is a non-broken
> stunnel workalike using BearSSL as the backend, since BearSSL is the
> only non-awful TLS implementation. If anyone wants to work on
> something like that I'd be happy to test and eventually dogfood it on
> musl site.

If a working haproxy solution is already in place, why not rig it up for
cloning as well? What's the old phrase - perfect is the enemy of good,
or something like that.

Re: Supporting git access via smart HTTPS protocol for musl-libc

[Rich Felker]

On Tue, Mar 26, 2019 at 11:09:01AM -0400, Drew DeVault wrote:
> On 2019-03-26 11:04 AM, Rich Felker wrote:
> > > Also I find you are providing https version of git.musl-libc.org site.
> > > thttpd does not supports https. Are you using stunnel for it?
> > 
> > I'm presently using haproxy's TLS-layer (vs HTTPS-layer) proxying,
> > because stunnel suggers from a 2.5-decades-old wrong handling of TCP
> > connection closing that makes it unusable, and because haproxy is what
> > I knew at the time. I think openssl s_server could handle it too, but
> > might not support SNI (?). What I'd really prefer is a non-broken
> > stunnel workalike using BearSSL as the backend, since BearSSL is the
> > only non-awful TLS implementation. If anyone wants to work on
> > something like that I'd be happy to test and eventually dogfood it on
> > musl site.
> 
> If a working haproxy solution is already in place, why not rig it up for
> cloning as well? What's the old phrase - perfect is the enemy of good,
> or something like that.

The problem is that I don't know how to hook up the smart git http
backend via cgi. Maybe you're suggesting running it on a separate
httpd with haproxy doing the routing, but that's not compatible with
TLS-layer (rather than HTTP-layer) use of haproxy, and the latter does
not work with thttpd's cgi conformance issues, nor do I want to
introduce further dependency on haproxy, which is a big hammer. I'd
rather move in the opposite direction towards something like a
non-broken version of stunnel.

Rich

Re: Supporting git access via smart HTTPS protocol for musl-libc

[Drew DeVault]

On 2019-03-26 11:13 AM, Rich Felker wrote:
> The problem is that I don't know how to hook up the smart git http
> backend via cgi. Maybe you're suggesting running it on a separate
> httpd with haproxy doing the routing, but that's not compatible with
> TLS-layer (rather than HTTP-layer) use of haproxy, and the latter does
> not work with thttpd's cgi conformance issues, nor do I want to
> introduce further dependency on haproxy, which is a big hammer. I'd
> rather move in the opposite direction towards something like a
> non-broken version of stunnel.

After a more detailed study of thttpd, I understand why this is
difficult. Honestly, thttpd seems like a really antiquated and
inflexible httpd. I'd vote in favor of switching to nginx, in the
interest of having good before having perfect, with a plan to perhaps
port OpenBSD httpd or something similar.

However, the other suggestion of busybox httpd + git-http-backend seems
reasonable to me. It sounds like you were hoping to keep thttpd in play,
though, which seems folly to me.

Re: Supporting git access via smart HTTPS protocol for musl-libc

[Rich Felker]

On Tue, Mar 26, 2019 at 11:43:04AM -0400, Drew DeVault wrote:
> On 2019-03-26 11:13 AM, Rich Felker wrote:
> > The problem is that I don't know how to hook up the smart git http
> > backend via cgi. Maybe you're suggesting running it on a separate
> > httpd with haproxy doing the routing, but that's not compatible with
> > TLS-layer (rather than HTTP-layer) use of haproxy, and the latter does
> > not work with thttpd's cgi conformance issues, nor do I want to
> > introduce further dependency on haproxy, which is a big hammer. I'd
> > rather move in the opposite direction towards something like a
> > non-broken version of stunnel.
> 
> After a more detailed study of thttpd, I understand why this is
> difficult.

I don't see why thttpd is making it difficult. It makes routing with
haproxy difficult only because haproxy is more pedantic than any web
browser is about headers, but I don't want to use haproxy routing
anyway.

> Honestly, thttpd seems like a really antiquated and
> inflexible httpd.

I'd love to have a modern one with the same type of design.
Unfortunately all the modern ones are hideous.

> I'd vote in favor of switching to nginx, in the

nginx doesn't even support cgi. It just forwards to another server for
cgi. It's also horribly bloated and enterprise-ey. In 5-10 years it
will go exactly the same way Apache did. Watch for them to have their
own Tomcat but for whatever language displaces Java...

> interest of having good before having perfect, with a plan to perhaps
> port OpenBSD httpd or something similar.
> 
> However, the other suggestion of busybox httpd + git-http-backend seems
> reasonable to me. It sounds like you were hoping to keep thttpd in play,
> though, which seems folly to me.

If it works with busybox httpd it should work with thttpd.

Rich

Re: Supporting git access via smart HTTPS protocol for musl-libc

[Drew DeVault]

On 2019-03-26 11:47 AM, Rich Felker wrote:
> I don't see why thttpd is making it difficult. It makes routing with
> haproxy difficult only because haproxy is more pedantic than any web
> browser is about headers, but I don't want to use haproxy routing
> anyway.

Does thttpd even have routing at all? As far as I can tell it is not
capable of sourcing routes from anywhere other than the filesystem. This
makes conditionally dispatching requests to git difficult. Perhaps I
misunderstand how this kind of thing could be configured with thttpd?

> I'd love to have a modern one with the same type of design.
> Unfortunately all the modern ones are hideous.

The design is what's antiquted.

> > I'd vote in favor of switching to nginx, in the
> 
> nginx doesn't even support cgi. It just forwards to another server for
> cgi. It's also horribly bloated and enterprise-ey. In 5-10 years it
> will go exactly the same way Apache did. Watch for them to have their
> own Tomcat but for whatever language displaces Java...

I agree on all of these points, but I'd like to draw attention to the
fact that 5-10 years from now is 5-10 years from now. That's plenty of
time to come up with a better httpd, and in the meanwhile nginx has yet
to go the Apache way.

> > However, the other suggestion of busybox httpd + git-http-backend seems
> > reasonable to me. It sounds like you were hoping to keep thttpd in play,
> > though, which seems folly to me.
> 
> If it works with busybox httpd it should work with thttpd.

I admit that I know little about busybox httpd, but assumed that the
opinion expressed earlier in the thread about its viability was an
informed one. However, upon studying thttpd's documentation, I concluded
that it was not up to this task without egregious hacks.

A similar survey of busybox httpd agrees that serving git should be
possible, though somewhat convoluted - less so than thttpd would be.

Re: Supporting git access via smart HTTPS protocol for musl-libc

[Rich Felker]

On Tue, Mar 26, 2019 at 11:57:43AM -0400, Drew DeVault wrote:
> On 2019-03-26 11:47 AM, Rich Felker wrote:
> > I don't see why thttpd is making it difficult. It makes routing with
> > haproxy difficult only because haproxy is more pedantic than any web
> > browser is about headers, but I don't want to use haproxy routing
> > anyway.
> 
> Does thttpd even have routing at all? As far as I can tell it is not
> capable of sourcing routes from anywhere other than the filesystem. This
> makes conditionally dispatching requests to git difficult. Perhaps I
> misunderstand how this kind of thing could be configured with thttpd?

If configured to do so, when foo/bar does not exist but foo/ does, it
will exec foo/index.cgi with access to the full path requested. This
suffices for cgit paths so it should suffice for git http protocol
too.

> > I'd love to have a modern one with the same type of design.
> > Unfortunately all the modern ones are hideous.
> 
> The design is what's antiquted.

Well we disagree on this. I consider handling of all requests via the
filesystem, using a single-task poll() loop, and exec'ing a cgi child
process per dynamic request, to be the reasonable design from a
standpoint of both proper isolation and decent efficiency. The whole
"persistent task to handle dynamic content" thing was a mistake and is
a lot more error-prone *and* resource-costly unless you're using awful
language runtimes that are slow to startup.

> > > I'd vote in favor of switching to nginx, in the
> > 
> > nginx doesn't even support cgi. It just forwards to another server for
> > cgi. It's also horribly bloated and enterprise-ey. In 5-10 years it
> > will go exactly the same way Apache did. Watch for them to have their
> > own Tomcat but for whatever language displaces Java...
> 
> I agree on all of these points, but I'd like to draw attention to the
> fact that 5-10 years from now is 5-10 years from now. That's plenty of
> time to come up with a better httpd, and in the meanwhile nginx has yet
> to go the Apache way.

It's already bad, just not *as bad*. And it's much worse than what we
have now, so there's no reason to switch to it.

Rich

Re: Supporting git access via smart HTTPS protocol for musl-libc

[A. Wilcox]

[-- Attachment #1.1: Type: text/plain, Size: 514 bytes --]

On 03/26/19 12:57, Rich Felker wrote:
> It's already bad, just not *as bad*.

Apache's gotten better as nginx has gotten worse, IMO.  We're running
Apache for all of adelielinux.org infra.  Right now, the main Web server
(which handles bts, wiki, www, et al) is using about 10.7 MB RAM.

> And it's much worse than what we
> have now, so there's no reason to switch to it.

That, I will agree with.

Best,
--arw

-- 
A. Wilcox (awilfox)
Project Lead, Adélie Linux
https://www.adelielinux.org


[-- Attachment #2: OpenPGP digital signature --]
[-- Type: application/pgp-signature, Size: 833 bytes --]

Re: Supporting git access via smart HTTPS protocol for musl-libc

[Assaf Gordon]

Hello,

I might be able to suggest few pointers on setting up git/http CGI access.

The git package contains 'git-http-backend' (typically in /usr/lib/git-core)
which is a cgi backend meant for smart/dump git cloning.

On GNU Savannah we use NGINX with the following configuration:

  location = /r { return 302 $request_uri/; }
  location /r/ {
    autoindex on;
    alias /srv/git/;
    location ~ ^/r(/.*/(info/refs|git-upload-pack)$) {
      gzip off;
      include fastcgi_params;
      fastcgi_pass unix:/var/run/fcgiwrap.socket;
      fastcgi_param SCRIPT_FILENAME /usr/local/sbin/git-http-backend;
      fastcgi_param PATH_INFO $1;
      fastcgi_param GIT_HTTP_EXPORT_ALL true;
      fastcgi_param GIT_PROJECT_ROOT /srv/git;
      client_max_body_size 0;
    }
  }

(You made your opinion on nginx clear, but this is just for reference for
a working configuration).

-----

To run the backend manually, try variations of the following:

  $ REQUEST_METHOD=GET GIT_HTTP_EXPORT_ALL=true \
    GIT_PROJECT_ROOT=/home/gordon/projects/ PATH_INFO=/musl/.git/HEAD \
    /usr/lib/git-core/git-http-backend

  Content-Length: 23
  Content-Type: text/plain
  ref: refs/heads/master

(running 'man git-http-bckend' will give more details about GIT_PROJECT_ROOT
etc.).

----

To run under busybox's httpd, I used the following contrived setup:

    mkdir www
    mkdir www/cgi-bin
    echo "hello world" > www/index.html
    cat<<EOF>www/cgi-bin/test.sh
    #!/bin/sh
    echo "Content-type: text/html"
    echo ""
    echo "Hello CGI World"
    EOF
    chmod a+x ./www/cgi-bin/test.sh

    busybox httpd -v -f -p 9999 -h ./www

This will start the busybox httpd server, serving files from ./www folder.
Assuming busybox/httpd was compiled with CGI support, the script in the
'cgi-bin' directory should "just work". Test with:

    $ curl http://localhost:9999/
    hello world

    $ curl http://localhost:9999/cgi-bin/test.sh
    Hello CGI World

If the above worked, the CGI setup is fine and we can move on the git.

---

Create the following wrapper in ./www/cgi-bin/ (any file name would work,
but a file name without extension 'looks' better, e.g. 'view'):

    #!/bin/sh
    export GIT_HTTP_EXPORT_ALL=true
    export GIT_PROJECT_ROOT=/home/gordon/projects/
    export HTTP_CONTENT_ENCODING=gzip
    exec /usr/lib/git-core/git-http-backend

and make it executable with "chmod a+x ./www/cgi-bin/view".

This setup will serve ANY repository under the 'GIT_PROJECT_ROOT'.
You can of course adjust as needed.
In my case, I have '/home/gordon/projects/musl/',
which is tested below like so:

    $ curl -D /dev/stderr http://localhost:9999/cgi-bin/view/musl/HEAD
    HTTP/1.0 200 OK
    Content-Length: 23
    Content-Type: text/plain

    ref: refs/heads/master

The above curl command executed the 'view' script with PATH_INFO being
'/musl/HEAD' - which is a request git-http-backend knows how to handle.

If the above worked, cloning 'should work' as well:

    $ git clone http://localhost:9999/cgi-bin/view/musl
    Cloning into 'musl'...
    remote: Counting objects: 31250, done.
    remote: Compressing objects: 100% (9126/9126), done.
    remote: Total 31250 (delta 22523), reused 30465 (delta 21759)
    Receiving objects: 100% (31250/31250), 4.78 MiB | 0 bytes/s, done.
    Resolving deltas: 100% (22523/22523), done.

----

Others in this thread talked about URL re-routing/aliasing.
This would be useful to hide the "cgi-bin" part of the URL, but busybox's
httpd doesn't support it. Having it in the URL isn't the end of the world
if one insist on using a minimalistic web server.

----

I haven't used thttpd, but it should work very similarly.

Hope this helps,
regards,
 - assaf

Re: Supporting git access via smart HTTPS protocol for musl-libc

[Rich Felker]

On Tue, Mar 26, 2019 at 02:39:13PM -0600, Assaf Gordon wrote:
> Hello,
> 
> I might be able to suggest few pointers on setting up git/http CGI access.
> 
> The git package contains 'git-http-backend' (typically in /usr/lib/git-core)
> which is a cgi backend meant for smart/dump git cloning.
> 
> On GNU Savannah we use NGINX with the following configuration:
> 
>   location = /r { return 302 $request_uri/; }
>   location /r/ {
>     autoindex on;
>     alias /srv/git/;
>     location ~ ^/r(/.*/(info/refs|git-upload-pack)$) {
>       gzip off;
>       include fastcgi_params;
>       fastcgi_pass unix:/var/run/fcgiwrap.socket;
>       fastcgi_param SCRIPT_FILENAME /usr/local/sbin/git-http-backend;
>       fastcgi_param PATH_INFO $1;
>       fastcgi_param GIT_HTTP_EXPORT_ALL true;
>       fastcgi_param GIT_PROJECT_ROOT /srv/git;
>       client_max_body_size 0;
>     }
>   }
> 
> (You made your opinion on nginx clear, but this is just for reference for
> a working configuration).
> 
> -----
> 
> To run the backend manually, try variations of the following:
> 
>   $ REQUEST_METHOD=GET GIT_HTTP_EXPORT_ALL=true \
>     GIT_PROJECT_ROOT=/home/gordon/projects/ PATH_INFO=/musl/.git/HEAD \
>     /usr/lib/git-core/git-http-backend
> 
>   Content-Length: 23
>   Content-Type: text/plain
>   ref: refs/heads/master
> 
> (running 'man git-http-bckend' will give more details about GIT_PROJECT_ROOT
> etc.).
> 
> ----
> 
> To run under busybox's httpd, I used the following contrived setup:
> 
>     mkdir www
>     mkdir www/cgi-bin
>     echo "hello world" > www/index.html
>     cat<<EOF>www/cgi-bin/test.sh
>     #!/bin/sh
>     echo "Content-type: text/html"
>     echo ""
>     echo "Hello CGI World"
>     EOF
>     chmod a+x ./www/cgi-bin/test.sh
> 
>     busybox httpd -v -f -p 9999 -h ./www
> 
> This will start the busybox httpd server, serving files from ./www folder.
> Assuming busybox/httpd was compiled with CGI support, the script in the
> 'cgi-bin' directory should "just work". Test with:
> 
>     $ curl http://localhost:9999/
>     hello world
> 
>     $ curl http://localhost:9999/cgi-bin/test.sh
>     Hello CGI World
> 
> If the above worked, the CGI setup is fine and we can move on the git.
> 
> ---
> 
> Create the following wrapper in ./www/cgi-bin/ (any file name would work,
> but a file name without extension 'looks' better, e.g. 'view'):
> 
>     #!/bin/sh
>     export GIT_HTTP_EXPORT_ALL=true
>     export GIT_PROJECT_ROOT=/home/gordon/projects/
>     export HTTP_CONTENT_ENCODING=gzip
>     exec /usr/lib/git-core/git-http-backend
> 
> and make it executable with "chmod a+x ./www/cgi-bin/view".
> 
> This setup will serve ANY repository under the 'GIT_PROJECT_ROOT'.
> You can of course adjust as needed.
> In my case, I have '/home/gordon/projects/musl/',
> which is tested below like so:
> 
>     $ curl -D /dev/stderr http://localhost:9999/cgi-bin/view/musl/HEAD
>     HTTP/1.0 200 OK
>     Content-Length: 23
>     Content-Type: text/plain
> 
>     ref: refs/heads/master
> 
> The above curl command executed the 'view' script with PATH_INFO being
> '/musl/HEAD' - which is a request git-http-backend knows how to handle.
> 
> If the above worked, cloning 'should work' as well:
> 
>     $ git clone http://localhost:9999/cgi-bin/view/musl
>     Cloning into 'musl'...
>     remote: Counting objects: 31250, done.
>     remote: Compressing objects: 100% (9126/9126), done.
>     remote: Total 31250 (delta 22523), reused 30465 (delta 21759)
>     Receiving objects: 100% (31250/31250), 4.78 MiB | 0 bytes/s, done.
>     Resolving deltas: 100% (22523/22523), done.
> 
> ----
> 
> Others in this thread talked about URL re-routing/aliasing.
> This would be useful to hide the "cgi-bin" part of the URL, but busybox's
> httpd doesn't support it. Having it in the URL isn't the end of the world
> if one insist on using a minimalistic web server.
> 
> ----
> 
> I haven't used thttpd, but it should work very similarly.

Thanks for the info. I've been playing with it, but haven't been able
to get it to work yet. I suspect thttpd is doing something broken with
the POST request since the git clone breaks during that. Going to look
at it in more detail later.

Rich

Re: Supporting git access via smart HTTPS protocol for musl-libc

[Assaf Gordon]

>
> Thanks for the info. I've been playing with it, but haven't been able
> to get it to work yet. I suspect thttpd is doing something broken with
> the POST request since the git clone breaks during that. Going to look
> at it in more detail later.

The same happened to me with busybox, and was solved by forcing:

    export HTTP_CONTENT_ENCODING=gzip

Seems to me (from a cursory look) that "git clone" client always sends
gzipped (inflate) data, but either doesn't set the HTTP encoding header,
or the header gets lost somehow.

The git-http-backend uses that header to decide whether to deflate
the stream or not (
https://github.com/git/git/blob/master/http-backend.c#L460 ).

That's the first thing I'd try if the GET request worked but the
following "POST git-update-pack"
does not.

regards,
 - assaf

Re: Supporting git access via smart HTTPS protocol for musl-libc

[Rich Felker]

On Tue, Mar 26, 2019 at 04:32:32PM -0600, Assaf Gordon wrote:
> >
> > Thanks for the info. I've been playing with it, but haven't been able
> > to get it to work yet. I suspect thttpd is doing something broken with
> > the POST request since the git clone breaks during that. Going to look
> > at it in more detail later.
> 
> The same happened to me with busybox, and was solved by forcing:
> 
>     export HTTP_CONTENT_ENCODING=gzip
> 
> Seems to me (from a cursory look) that "git clone" client always sends
> gzipped (inflate) data, but either doesn't set the HTTP encoding header,
> or the header gets lost somehow.
> 
> The git-http-backend uses that header to decide whether to deflate
> the stream or not (
> https://github.com/git/git/blob/master/http-backend.c#L460 ).
> 
> That's the first thing I'd try if the GET request worked but the
> following "POST git-update-pack"
> does not.

Amazingly, this works, but only if I do it only for
REQUEST_METHOD=POST. Otherwise it breaks the GET request and it never
makes it to the POST. *sigh* There's got to be some better way to fix
this.

Rich

Re: Supporting git access via smart HTTPS protocol for musl-libc

[Rich Felker]

On Tue, Mar 26, 2019 at 07:58:35PM -0400, Rich Felker wrote:
> On Tue, Mar 26, 2019 at 04:32:32PM -0600, Assaf Gordon wrote:
> > >
> > > Thanks for the info. I've been playing with it, but haven't been able
> > > to get it to work yet. I suspect thttpd is doing something broken with
> > > the POST request since the git clone breaks during that. Going to look
> > > at it in more detail later.
> > 
> > The same happened to me with busybox, and was solved by forcing:
> > 
> >     export HTTP_CONTENT_ENCODING=gzip
> > 
> > Seems to me (from a cursory look) that "git clone" client always sends
> > gzipped (inflate) data, but either doesn't set the HTTP encoding header,
> > or the header gets lost somehow.
> > 
> > The git-http-backend uses that header to decide whether to deflate
> > the stream or not (
> > https://github.com/git/git/blob/master/http-backend.c#L460 ).
> > 
> > That's the first thing I'd try if the GET request worked but the
> > following "POST git-update-pack"
> > does not.
> 
> Amazingly, this works, but only if I do it only for
> REQUEST_METHOD=POST. Otherwise it breaks the GET request and it never
> makes it to the POST. *sigh* There's got to be some better way to fix
> this.

Anyway, for now this is vaguely working, but it's a hack and it's
experimental and might break. Users interested in testing can clone:

	https://git.musl-libc.org/git/musl

It might go up and down while I'm hacking on getting it right, but I
intend for the URL to be permanent in the long term.

Rich

Re: Supporting git access via smart HTTPS protocol for musl-libc

[vlse]

On Tue, Mar 26, 2019 at 08:15:42PM -0400, Rich Felker wrote:
> On Tue, Mar 26, 2019 at 07:58:35PM -0400, Rich Felker wrote:
> > On Tue, Mar 26, 2019 at 04:32:32PM -0600, Assaf Gordon wrote:
> > > >
> > > > Thanks for the info. I've been playing with it, but haven't been able
> > > > to get it to work yet. I suspect thttpd is doing something broken with
> > > > the POST request since the git clone breaks during that. Going to look
> > > > at it in more detail later.
> > > 
> > > The same happened to me with busybox, and was solved by forcing:
> > > 
> > >     export HTTP_CONTENT_ENCODING=gzip
> > > 
> > > Seems to me (from a cursory look) that "git clone" client always sends
> > > gzipped (inflate) data, but either doesn't set the HTTP encoding header,
> > > or the header gets lost somehow.
> > > 
> > > The git-http-backend uses that header to decide whether to deflate
> > > the stream or not (
> > > https://github.com/git/git/blob/master/http-backend.c#L460 ).
> > > 
> > > That's the first thing I'd try if the GET request worked but the
> > > following "POST git-update-pack"
> > > does not.
> > 
> > Amazingly, this works, but only if I do it only for
> > REQUEST_METHOD=POST. Otherwise it breaks the GET request and it never
> > makes it to the POST. *sigh* There's got to be some better way to fix
> > this.
> 
> Anyway, for now this is vaguely working, but it's a hack and it's
> experimental and might break. Users interested in testing can clone:
> 
> 	https://git.musl-libc.org/git/musl
> 
> It might go up and down while I'm hacking on getting it right, but I
> intend for the URL to be permanent in the long term.
> 
> Rich

Hi,

I am using git-2.0.2 version.

Following works:
git clone https://git.musl-libc.org/git/musl
cd musl
git pull
git fetch

git fsck --full
 shows no error

This does not works:
mkdir musl.git
cd musl.git
git init
git remote add -t master -m master origin https://git.musl-libc.org/git/musl

git fetch

git-fetch gives errors:
  fatal: protocol error: bad line length character: erro


Running 'GIT_TRACE=2 git fetch' outputs:

trace: built-in: git 'fetch'
trace: run_command: 'git-remote-https' 'origin' 'https://git.musl-libc.org/git/musl'
trace: run_command: 'rev-list' '--objects' '--stdin' '--not' '--all' '--quiet'
trace: run_command: 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--include-tag' '--thin' 'https://git.musl-libc.org/git/musl/'
trace: exec: 'git' 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--include-tag' '--thin' 'https://git.musl-libc.org/git/musl/'
trace: built-in: git 'fetch-pack' '--stateless-rpc' '--stdin' '--lock-pack' '--include-tag' '--thin' 'https://git.musl-libc.org/git/musl/'
fatal: protocol error: bad line length character: erro
fatal: The remote end hung up unexpectedly


Regards,
Veera

Re: Supporting git access via smart HTTPS protocol for musl-libc

[Assaf Gordon]

Hello,

On 2019-03-26 11:39 p.m., vlse wrote:
> On Tue, Mar 26, 2019 at 08:15:42PM -0400, Rich Felker wrote:
>> On Tue, Mar 26, 2019 at 07:58:35PM -0400, Rich Felker wrote:
>>> On Tue, Mar 26, 2019 at 04:32:32PM -0600, Assaf Gordon wrote:
>>>>>
>>>>> [...] I suspect thttpd is doing something broken with
>>>>> the POST request since the git clone breaks during that. 
>>>>
>>>> The same happened to me with busybox, and was solved by forcing:
>>>>
>>>>      export HTTP_CONTENT_ENCODING=gzip
>>>>
>>> Amazingly, this works, but only if I do it only for
>>> REQUEST_METHOD=POST. Otherwise it breaks the GET request and it never
>>> makes it to the POST.

> This does not works:
> mkdir musl.git
> cd musl.git
> git init
> git remote add -t master -m master origin https://git.musl-libc.org/git/musl
> 
> git-fetch gives errors:
>    fatal: protocol error: bad line length character: erro

Thanks for checking and for the detailed reproducer.

This error tells me there is (again) a binary/text conflict,

The (text) git protocol always starts with 4 hex digits
indicating length. If the input is gzip'd and wasn't
decompressed, the length indicator will be invalid.

Digging further, it seems that in this case the "git fetch"
command sent POST data which is *not* compressed.
Forcing "CONTENT_ENCODING=gzip" always or for all POST request
is not sufficient.

I now use the following shell wrapper, which works for
both "git clone" and your "git fetch" case.
I won't claim it's pretty, but it works (with busybox httpd).
The "echo>&2" will show up on webserver's error log.

----
#!/bin/sh

export GIT_HTTP_EXPORT_ALL=true
export GIT_PROJECT_ROOT=/home/gordon/projects/

echo START - $REQUEST_METHOD $PATH_INFO >&2

# Check if POST data is text or binary, add HTTP header if needed.
if test "$REQUEST_METHOD" = POST ; then
     t=$(mktemp -t git-http-backend-XXXXXX)
     # Store STDIN and examine it
     cat - > $t

     # Git (text) protocol starts with 4 hex digits indicating length.
     # If the first 4 bytes aren't hexdigits, assume STDIN is compressed.
     if head -c4 $t | grep -q '^[0-9a-f][0-9a-f][0-9a-f][0-9a-f]$' ; then
         echo "POST data is not gzipped" >&2
     else
         echo "POST data is gzipped" >&2
         export HTTP_CONTENT_ENCODING=gzip
     fi

     # restore STDIN
     exec < $t
     rm $t
fi
/usr/lib/git-core/git-http-backend
echo END - $REQUEST_METHOD $PATH_INFO >&2
echo >&2
echo >&2

----

As Rich said, there's got to be a better way...


Regards,
   -assaf

Re: Supporting git access via smart HTTPS protocol for musl-libc

[Assaf Gordon]

More details:

>>>>>> [...] I suspect thttpd is doing something broken with
>>>>>> the POST request since the git clone breaks during that. 
>>>>>
>>>>> The same happened to me with busybox, and was solved by forcing:
>>>>>
>>>>>      export HTTP_CONTENT_ENCODING=gzip
>>>>>
>>>> Amazingly, this works, but only if I do it only for
>>>> REQUEST_METHOD=POST. Otherwise it breaks the GET request and it never
>>>> makes it to the POST.

[...]

> Digging further, it seems that in this case the "git fetch"
> command sent POST data which is *not* compressed.
> Forcing "CONTENT_ENCODING=gzip" always or for all POST request
> is not sufficient.


The git program (at least version 2.11.0) does send the
CONTENT_ENCODING header if the POST data is gzipped.

In the case of "busybox httpd" webserver, it simply does
not pass this header on to the CGI program - hence
all these problems:
https://git.busybox.net/busybox/tree/networking/httpd.c#n1443

Possibly thttpd behaves the same (while nginx/apache do pass
all headers, and there's no problem with them).

regards,
  - assaf

Re: Supporting git access via smart HTTPS protocol for musl-libc

[vlse]

On Wed, Mar 27, 2019 at 11:26:22AM -0600, Assaf Gordon wrote:
> Hello,
> 
> On 2019-03-26 11:39 p.m., vlse wrote:
> >On Tue, Mar 26, 2019 at 08:15:42PM -0400, Rich Felker wrote:
> >>On Tue, Mar 26, 2019 at 07:58:35PM -0400, Rich Felker wrote:
> >>>On Tue, Mar 26, 2019 at 04:32:32PM -0600, Assaf Gordon wrote:
> >>>>>
> >>>>>[...] I suspect thttpd is doing something broken with
> >>>>>the POST request since the git clone breaks during that. 
> >>>>
> >>>>The same happened to me with busybox, and was solved by forcing:
> >>>>
> >>>>     export HTTP_CONTENT_ENCODING=gzip
> >>>>
> >>>Amazingly, this works, but only if I do it only for
> >>>REQUEST_METHOD=POST. Otherwise it breaks the GET request and it never
> >>>makes it to the POST.
> 
> >This does not works:
> >mkdir musl.git
> >cd musl.git
> >git init
> >git remote add -t master -m master origin 
> >https://git.musl-libc.org/git/musl
> >
> >git-fetch gives errors:
> >   fatal: protocol error: bad line length character: erro
> 
> Thanks for checking and for the detailed reproducer.
> 
> This error tells me there is (again) a binary/text conflict,
> 
> The (text) git protocol always starts with 4 hex digits
> indicating length. If the input is gzip'd and wasn't
> decompressed, the length indicator will be invalid.
> 
> Digging further, it seems that in this case the "git fetch"
> command sent POST data which is *not* compressed.
> Forcing "CONTENT_ENCODING=gzip" always or for all POST request
> is not sufficient.
> 
> I now use the following shell wrapper, which works for
> both "git clone" and your "git fetch" case.
> I won't claim it's pretty, but it works (with busybox httpd).
> The "echo>&2" will show up on webserver's error log.
> 
> ----
> #!/bin/sh
> 
> export GIT_HTTP_EXPORT_ALL=true
> export GIT_PROJECT_ROOT=/home/gordon/projects/
> 
> echo START - $REQUEST_METHOD $PATH_INFO >&2
> 
> # Check if POST data is text or binary, add HTTP header if needed.
> if test "$REQUEST_METHOD" = POST ; then
>     t=$(mktemp -t git-http-backend-XXXXXX)
>     # Store STDIN and examine it
>     cat - > $t
> 
>     # Git (text) protocol starts with 4 hex digits indicating length.
>     # If the first 4 bytes aren't hexdigits, assume STDIN is compressed.
>     if head -c4 $t | grep -q '^[0-9a-f][0-9a-f][0-9a-f][0-9a-f]$' ; then
>         echo "POST data is not gzipped" >&2
>     else
>         echo "POST data is gzipped" >&2
>         export HTTP_CONTENT_ENCODING=gzip
>     fi
> 
>     # restore STDIN
>     exec < $t
>     rm $t
> fi
> /usr/lib/git-core/git-http-backend
> echo END - $REQUEST_METHOD $PATH_INFO >&2
> echo >&2
> echo >&2
> 
> ----
> 
> As Rich said, there's got to be a better way...
> 
> 
> Regards,
>   -assaf

Hi,

I previously reported that doing git clone and then git pull was showing no errors.
git pull said "Already upto Date"

But when I did git pull again, I got the following errors:

 fatal: protocol error: bad line length character: erro

This was same error which I told when a fresh git fetch was producing.

The git pull actually invokes git fetch followed by git merge.
After a fresh clone and then doing git pull immediately does not invokes git fetch at all.

Regards,
Veera