Cgit cache and slownesses

Cgit cache and slownesses

[esajine]

On 8/23/2013 8:17 AM, Jason A. Donenfeld wrote:
> The list is now opened up. Would you pass this message along to it?
> Sorry for the delay in this. Back from vacation now, getting things
> organized finally.
>

Hi!

Unfortunately, my message didn't seem to get any traction, so i have to 
repeat it:

http://lists.zx2c4.com/pipermail/cgit/2013-August/001526.html

I have 1600+ git repos that cgit is working with and this amount is
growing consistently.
Cgit experiences some slowness at times and I'm trying to find some
ways to improve the performance.

1. I have specified the cache amount as 2000 repos. sometimes it takes
up to a minute or more to get to the page while drilling down to the
tree or snapshots. I'm curious if I should try to increase ttl for the
repos (currently using default values) Is there anything else that could
make Cgit faster? Increasing TTL is one way probably but it will show
some outdated info...

Considering this I thought about the following solution (analog with
Jenkins):
We are serving repositories via Git protocol. Git protocol since some
1.8.* version supports --access-hook. Access hook script that we have can get the name of
the git service used and if it is receive-pack it calls Jenkins URL with
the url of the repository affected using curl and schedule Jenkins poll
immediately. This poll effectively gets the most recent changes and
builds the code.

Now It would probably increase the performance a lot if instead of
expiring Cgit cache every 5 min we could expire it upon a push.
But when i checked how many pushes per day we have we
averaged in about 200. That means that if we would use that approach we
would rescan even more often sometimes then with hardcoded value.
So, may be the solution would be either in
a) rescanning only the repo that was touched (that's preferred of
course) and then combine it with full rescan every ttl interval,
but in this case it would be like daily or
b) introduce a logic that would only rescan upon push but not more often
then once in ttl period I.e. the ttl time would be the least interval
for cache expiration. (if last_push_time - last_scan_time > ttl ; then
scan, else wait for ttl expiration)

So eventually i'm looking for Cgit to have a URL like that:
http://server/cgi-bin/cgit.cgi/myrepo?do_scan  that would update this
particular repo info in cgit cache.

2. It seems that the caching mechanism doesn't work properly for me
because I have changed the cache-root and i don't see any files being
saved in this new location - the folder is always empty.

Thanks,
Eugene

Cgit cache and slownesses

[esajine]

On 3/19/2014 10:30 AM, Eugene Sajine wrote:
> On 8/23/2013 8:17 AM, Jason A. Donenfeld wrote:
>> The list is now opened up. Would you pass this message along to it?
>> Sorry for the delay in this. Back from vacation now, getting things
>> organized finally.
>>
>
> Hi!
>
> Unfortunately, my message didn't seem to get any traction, so i have 
> to repeat it:
>
> http://lists.zx2c4.com/pipermail/cgit/2013-August/001526.html
>
> I have 1600+ git repos that cgit is working with and this amount is
> growing consistently.
> Cgit experiences some slowness at times and I'm trying to find some
> ways to improve the performance.
>
> 1. I have specified the cache amount as 2000 repos. sometimes it takes
> up to a minute or more to get to the page while drilling down to the
> tree or snapshots. I'm curious if I should try to increase ttl for the
> repos (currently using default values) Is there anything else that could
> make Cgit faster? Increasing TTL is one way probably but it will show
> some outdated info...
>
> Considering this I thought about the following solution (analog with
> Jenkins):
> We are serving repositories via Git protocol. Git protocol since some
> 1.8.* version supports --access-hook. Access hook script that we have 
> can get the name of
> the git service used and if it is receive-pack it calls Jenkins URL with
> the url of the repository affected using curl and schedule Jenkins poll
> immediately. This poll effectively gets the most recent changes and
> builds the code.
>
> Now It would probably increase the performance a lot if instead of
> expiring Cgit cache every 5 min we could expire it upon a push.
> But when i checked how many pushes per day we have we
> averaged in about 200. That means that if we would use that approach we
> would rescan even more often sometimes then with hardcoded value.
> So, may be the solution would be either in
> a) rescanning only the repo that was touched (that's preferred of
> course) and then combine it with full rescan every ttl interval,
> but in this case it would be like daily or
> b) introduce a logic that would only rescan upon push but not more often
> then once in ttl period I.e. the ttl time would be the least interval
> for cache expiration. (if last_push_time - last_scan_time > ttl ; then
> scan, else wait for ttl expiration)
>
> So eventually i'm looking for Cgit to have a URL like that:
> http://server/cgi-bin/cgit.cgi/myrepo?do_scan  that would update this
> particular repo info in cgit cache.
>
> 2. It seems that the caching mechanism doesn't work properly for me
> because I have changed the cache-root and i don't see any files being
> saved in this new location - the folder is always empty.
>
> Thanks,
> Eugene
>
>
>
I didn't see that that the new version has some chaching fixes - I'll 
try to use the newest one and come back with the results


Thanks,
Eugene