From mboxrd@z Thu Jan 1 00:00:00 1970 X-Msuck: nntp://news.gmane.org/gmane.linux.lib.musl.general/14034 Path: news.gmane.org!.POSTED.blaine.gmane.org!not-for-mail From: Assaf Gordon Newsgroups: gmane.linux.lib.musl.general Subject: Re: Supporting git access via smart HTTPS protocol for musl-libc Date: Wed, 27 Mar 2019 11:26:22 -0600 Message-ID: References: <20190326151344.GB23599@brightrain.aerifal.cx> <20190326154304.GB2267@homura.localdomain> <20190326154700.GC23599@brightrain.aerifal.cx> <20190326155743.GC2267@homura.localdomain> <20190326175700.GD23599@brightrain.aerifal.cx> <20190326220225.GE23599@brightrain.aerifal.cx> <20190326235835.GF23599@brightrain.aerifal.cx> <20190327001542.GG23599@brightrain.aerifal.cx> <20190327053933.GA2518@localhost> Reply-To: musl@lists.openwall.com Mime-Version: 1.0 Content-Type: text/plain; charset=utf-8; format=flowed Content-Transfer-Encoding: 7bit Injection-Info: blaine.gmane.org; posting-host="blaine.gmane.org:195.159.176.226"; logging-data="148853"; mail-complaints-to="usenet@blaine.gmane.org" User-Agent: Mozilla/5.0 (X11; Linux x86_64; rv:60.0) Gecko/20100101 Thunderbird/60.5.1 Cc: dalias@libc.org To: musl@lists.openwall.com, vlse Original-X-From: musl-return-14050-gllmg-musl=m.gmane.org@lists.openwall.com Wed Mar 27 18:26:40 2019 Return-path: Envelope-to: gllmg-musl@m.gmane.org Original-Received: from mother.openwall.net ([195.42.179.200]) by blaine.gmane.org with smtp (Exim 4.89) (envelope-from ) id 1h9CK0-000ccs-Ap for gllmg-musl@m.gmane.org; Wed, 27 Mar 2019 18:26:40 +0100 Original-Received: (qmail 21988 invoked by uid 550); 27 Mar 2019 17:26:37 -0000 Mailing-List: contact musl-help@lists.openwall.com; run by ezmlm Precedence: bulk List-Post: List-Help: List-Unsubscribe: List-Subscribe: List-ID: Original-Received: (qmail 21966 invoked from network); 27 Mar 2019 17:26:37 -0000 DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20161025; h=subject:to:cc:references:from:message-id:date:user-agent :mime-version:in-reply-to:content-language:content-transfer-encoding; bh=/TtsWJXVn2jG+4jwkws39zTqJSkPgOHtE5SMGg97Ulw=; b=P3jrfh69hk14GKyHPVvjLBdU0lsRH6f8vWWuj5gE1iNTzWCRsxnYCgD5qsr0e1xaU+ h9KBJ9u1BhD6E4zZdQkdMHsN1PeNjVa3Xa6Kj+ciZ8KTeSj3x0Wt+oUq7HD3+szvZ598 /HJdv2eQ8sC08KG6TsL4G2TaivQyA31l9fJx+Bo90yXNIV05G0gOyTygTVPZ4FwRKLNw hdRE7tJhH3zAhp82z8ktf2KocHW+aIFF9Jzn1l/kY3qfY9CrTpKXn3ANTU4h1dL5DhpQ 4Ke4JE1Ohr9o6x4GmhO91DOHmXkTIFqgiAwGofas7kdZfPT4F/BxEA3NyxoifoHIqgvd Oj1A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20161025; h=x-gm-message-state:subject:to:cc:references:from:message-id:date :user-agent:mime-version:in-reply-to:content-language :content-transfer-encoding; bh=/TtsWJXVn2jG+4jwkws39zTqJSkPgOHtE5SMGg97Ulw=; b=df2WTSUrFcnHnf6CwcRTie1Nwv6+5Y3yv3szINMkxzvPg9aToQc0qcTd5k4cX/Xvl8 IIchPxa91tY6sw0K98HjQhap3raiVtx/+uK6NUn7CWaQ2IXI8hbtZDHXRz3xXRBPDefo f/rAPZaPWXWsZZF4+apSuvVXqNzbx0cOfwoCqOSjpUokQjB3inc1mKxEPk/3fQGV1rSM 9WtJiaWiucdVBQ6tJvsPgt+Y0wTgrtKnbdX0mARDTZsqftZdX4+cg9xR4CRwbWFrvRQj /mzqmz2NAoa1Zz1vf9dlLv0tjtz1Vy+ZIGtMGEZ1rP/2mr1Hi78rfw0dCwYy8u24Tam+ urcQ== X-Gm-Message-State: APjAAAVFfQQQxVYnaHXB3SdaSGE4VgAXntlxZk261jAcY3Zd8rZ1dksV YaU5kGcIFOauhmTlKvW5Zj3ynC6h X-Google-Smtp-Source: APXvYqz/RQg1M2ea+nzrDKWzC5LBULvgyxODJsS4i4t5U5UWOEjD5xDTcLdJTgyNhqdMc33jV1gZnQ== X-Received: by 2002:a62:5543:: with SMTP id j64mr36369851pfb.105.1553707585145; Wed, 27 Mar 2019 10:26:25 -0700 (PDT) In-Reply-To: <20190327053933.GA2518@localhost> Content-Language: en-US Xref: news.gmane.org gmane.linux.lib.musl.general:14034 Archived-At: Hello, On 2019-03-26 11:39 p.m., vlse wrote: > On Tue, Mar 26, 2019 at 08:15:42PM -0400, Rich Felker wrote: >> On Tue, Mar 26, 2019 at 07:58:35PM -0400, Rich Felker wrote: >>> On Tue, Mar 26, 2019 at 04:32:32PM -0600, Assaf Gordon wrote: >>>>> >>>>> [...] I suspect thttpd is doing something broken with >>>>> the POST request since the git clone breaks during that. >>>> >>>> The same happened to me with busybox, and was solved by forcing: >>>> >>>> export HTTP_CONTENT_ENCODING=gzip >>>> >>> Amazingly, this works, but only if I do it only for >>> REQUEST_METHOD=POST. Otherwise it breaks the GET request and it never >>> makes it to the POST. > This does not works: > mkdir musl.git > cd musl.git > git init > git remote add -t master -m master origin https://git.musl-libc.org/git/musl > > git-fetch gives errors: > fatal: protocol error: bad line length character: erro Thanks for checking and for the detailed reproducer. This error tells me there is (again) a binary/text conflict, The (text) git protocol always starts with 4 hex digits indicating length. If the input is gzip'd and wasn't decompressed, the length indicator will be invalid. Digging further, it seems that in this case the "git fetch" command sent POST data which is *not* compressed. Forcing "CONTENT_ENCODING=gzip" always or for all POST request is not sufficient. I now use the following shell wrapper, which works for both "git clone" and your "git fetch" case. I won't claim it's pretty, but it works (with busybox httpd). The "echo>&2" will show up on webserver's error log. ---- #!/bin/sh export GIT_HTTP_EXPORT_ALL=true export GIT_PROJECT_ROOT=/home/gordon/projects/ echo START - $REQUEST_METHOD $PATH_INFO >&2 # Check if POST data is text or binary, add HTTP header if needed. if test "$REQUEST_METHOD" = POST ; then t=$(mktemp -t git-http-backend-XXXXXX) # Store STDIN and examine it cat - > $t # Git (text) protocol starts with 4 hex digits indicating length. # If the first 4 bytes aren't hexdigits, assume STDIN is compressed. if head -c4 $t | grep -q '^[0-9a-f][0-9a-f][0-9a-f][0-9a-f]$' ; then echo "POST data is not gzipped" >&2 else echo "POST data is gzipped" >&2 export HTTP_CONTENT_ENCODING=gzip fi # restore STDIN exec < $t rm $t fi /usr/lib/git-core/git-http-backend echo END - $REQUEST_METHOD $PATH_INFO >&2 echo >&2 echo >&2 ---- As Rich said, there's got to be a better way... Regards, -assaf