Downloading objects hangs up around 65kB

Downloading objects hangs up around 65kB

[Valdis Vītoliņš]

Probably it is issue on 64-bit ARM architecture, because I tested that 
it works on x86_64 virtual machine.

Some time ago (because I haven't used HTTPS protocol for repositories 
recently) downloading plain objects/blobs hangs up around 65kB of data.

You can test it with:

wget -O "01_About.odp" -d 
"https://odo.lv/git/JTM/plain/doc/presentations/01_About.odp"

It shows:
...
Saving to: ‘01_About.odp’

01_About.odp                        8%[====>                           ] 
  63,64K  13,0KB/s    in 4,9s

2022-09-16 22:32:33 (13,0 KB/s) - Connection closed at byte 65169. Retrying.
...
etc.

Server architecture is 64-bit ARM:
uname -a
Linux odo 5.15.0-1019-aws #23~20.04.1-Ubuntu SMP Thu Aug 18 03:23:03 UTC 
2022 aarch64 aarch64 aarch64 GNU/Linux

Cgit package version:
cgit    1.2.3+git2.25.1-1

Apache version:
apache2 -v
Server version: Apache/2.4.41 (Ubuntu)
Server built:   2022-06-14T13:30:55

Sincerely,
Valdis

Re: Downloading objects hangs up around 65kB

[John Keeping]

On Fri, Sep 16, 2022 at 10:55:39PM +0300, Valdis Vītoliņš wrote:
> Probably it is issue on 64-bit ARM architecture, because I tested that it
> works on x86_64 virtual machine.
> 
> Some time ago (because I haven't used HTTPS protocol for repositories
> recently) downloading plain objects/blobs hangs up around 65kB of data.
> 
> You can test it with:
> 
> wget -O "01_About.odp" -d
> "https://odo.lv/git/JTM/plain/doc/presentations/01_About.odp"
> 
> It shows:
> ...
> Saving to: ‘01_About.odp’
> 
> 01_About.odp                        8%[====>                           ]
> 63,64K  13,0KB/s    in 4,9s
> 
> 2022-09-16 22:32:33 (13,0 KB/s) - Connection closed at byte 65169. Retrying.
> ...
> etc.

This seems very strange (it also happens with the .../tree/... pages
showing the hexdump file content).

My guess is that this is caused by the Apache configuration rather than
anything CGit is doing itself.

Have you tested running cgit outside Apache using something like:

	CGIT_CONFIG=/path/to/cgitrc \
	QUERY_STRING=url=JTM/plain/doc/presentations/01_About.odb \
	cgit

That should give some idea whether CGit itself is returning short data
or if the response is being truncated downstream.


Regards,
John

Re: Downloading objects hangs up around 65kB

[Valdis Vītoliņš]

Thanks, John, for hints!

That allowed me to find this page:

https://blog.cryptomilk.org/2011/08/04/debugging-cgit/

And I got content returned with these settings (and had to run as root 
user):

CGIT_CONFIG=/etc/cgitrc \
QUERY_STRING="url=JTM/plain/doc/presentations/01_About.odp" \
/usr/lib/cgit/cgit.cgi 1>cgit.odt 2>cgit.log

cgit.log is empty and
cgit.odt is 713K file which starts with content:

X-Content-Type-Options: nosniff
Content-Security-Policy: default-src 'none'
Content-Type: application/octet-stream
Content-Length: 729281
Content-Disposition: inline; filename="01_About.odp"
Last-Modified: Sat, 17 Sep 2022 19:48:19 GMT
Expires: Sat, 17 Sep 2022 19:53:19 GMT
ETag: "7048b4feba4cd80d4d56dcb3ea4b063c93b620bd"

P�CqR3&��/mimetypeapplication/vnd.oasis.opendocument.presentationP�CqRConfigurations2/toolbar/P�CqRConfigurations2/floater/P�CqRConfigurations2/menubar/P�CqR�Configurations2/popupmenu/P�CqRConfigurations2/progressbar/P�CqR�Configurations2/statusbar�CqR'Configurations2/accelerator/current.xmlPP�CqRConfigurations2/images/Bitmaps/P�CqR�Configurations2/toolpanel�CqR
...

So, it may be problem with apache. I have quite many of modules loaded 
for reverse proxy and and url and html link rewrites to shorten URL 
coming from XWiki content management system.

apachectl -M
Loaded Modules:
  core_module (static)
  so_module (static)
  watchdog_module (static)
  http_module (static)
  log_config_module (static)
  logio_module (static)
  version_module (static)
  unixd_module (static)
  access_compat_module (shared)
  alias_module (shared)
  auth_basic_module (shared)
  authn_core_module (shared)
  authn_file_module (shared)
  authz_core_module (shared)
  authz_host_module (shared)
  authz_user_module (shared)
  autoindex_module (shared)
  cache_module (shared)
  cache_disk_module (shared)
  cgi_module (shared)
  deflate_module (shared)
  dir_module (shared)
  env_module (shared)
  filter_module (shared)
  headers_module (shared)
  mime_module (shared)
  mpm_prefork_module (shared)
  negotiation_module (shared)
  php7_module (shared)
  proxy_module (shared)
  proxy_connect_module (shared)
  proxy_html_module (shared)
  proxy_http_module (shared)
  reqtimeout_module (shared)
  rewrite_module (shared)
  setenvif_module (shared)
  socache_shmcb_module (shared)
  ssl_module (shared)
  status_module (shared)
  xml2enc_module (shared)

Question is how I can debug it, knowing that this is production site and 
that I don't have much of experience debugging apache?

Valdis

> On Fri, Sep 16, 2022 at 10:55:39PM +0300, Valdis Vītoliņš wrote:
>> Probably it is issue on 64-bit ARM architecture, because I tested that it
>> works on x86_64 virtual machine.
>>
>> Some time ago (because I haven't used HTTPS protocol for repositories
>> recently) downloading plain objects/blobs hangs up around 65kB of data.
>>
>> You can test it with:
>>
>> wget -O "01_About.odp" -d
>> "https://odo.lv/git/JTM/plain/doc/presentations/01_About.odp"
>>
>> It shows:
>> ...
>> Saving to: ‘01_About.odp’
>>
>> 01_About.odp                        8%[====>                           ]
>> 63,64K  13,0KB/s    in 4,9s
>>
>> 2022-09-16 22:32:33 (13,0 KB/s) - Connection closed at byte 65169. Retrying.
>> ...
>> etc.
> 
> This seems very strange (it also happens with the .../tree/... pages
> showing the hexdump file content).
> 
> My guess is that this is caused by the Apache configuration rather than
> anything CGit is doing itself.
> 
> Have you tested running cgit outside Apache using something like:
> 
> 	CGIT_CONFIG=/path/to/cgitrc \
> 	QUERY_STRING=url=JTM/plain/doc/presentations/01_About.odb \
> 	cgit
> 
> That should give some idea whether CGit itself is returning short data
> or if the response is being truncated downstream.
> 
> 
> Regards,
> John