Remove a few loops in cgit for a large number of repos.

Remove a few loops in cgit for a large number of repos.

[mlhess]

Hi Everyone,


I am looking into using cgit for Drupal.org's repos, and wondering if we
could remove code like this:
http://git.zx2c4.com/cgit/tree/shared.c#n79


We have almost 10,000 repos and are worried about the load from loops, and
building the index it loops over. All of the repos are under a directory
(in 2 different sub directories), so I was hoping we could just validate
the directory path (making sure someone is not trying to do a ../../,etc)
and allow it?

Could that be done?  Please let me know your thoughts.

Thanks
Michael
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.zx2c4.com/pipermail/cgit/attachments/20131010/06a3d09d/attachment.html>

Remove a few loops in cgit for a large number of repos.

[john]

On Thu, Oct 10, 2013 at 10:44:55AM -0400, Michael Hess wrote:
> I am looking into using cgit for Drupal.org's repos, and wondering if we
> could remove code like this:
> http://git.zx2c4.com/cgit/tree/shared.c#n79
> 
> 
> We have almost 10,000 repos and are worried about the load from loops, and
> building the index it loops over. All of the repos are under a directory
> (in 2 different sub directories), so I was hoping we could just validate
> the directory path (making sure someone is not trying to do a ../../,etc)
> and allow it?
> 
> Could that be done?  Please let me know your thoughts.

Have you actually seen this causing excessive load, or is it only a
theoretical issue?

I expect it would be possible, in the case of scan-path, to load repos
from disk lazily, but that will probably add quite a lot of complexity
and I'm not convinced it's worthwhile.

That particular loop will only be executed once and I suspect it is
dwarfed by the time spent loading and parsing the config (cached project
list if you're using scan-path).

Remove a few loops in cgit for a large number of repos.

[mlhess]

I have not seen the load yet, we will do some load testing and let you
know.

Thanks,
Michael


On Thu, Oct 10, 2013 at 11:48 AM, John Keeping <john at keeping.me.uk> wrote:

> On Thu, Oct 10, 2013 at 10:44:55AM -0400, Michael Hess wrote:
> > I am looking into using cgit for Drupal.org's repos, and wondering if we
> > could remove code like this:
> > http://git.zx2c4.com/cgit/tree/shared.c#n79
> >
> >
> > We have almost 10,000 repos and are worried about the load from loops,
> and
> > building the index it loops over. All of the repos are under a directory
> > (in 2 different sub directories), so I was hoping we could just validate
> > the directory path (making sure someone is not trying to do a ../../,etc)
> > and allow it?
> >
> > Could that be done?  Please let me know your thoughts.
>
> Have you actually seen this causing excessive load, or is it only a
> theoretical issue?
>
> I expect it would be possible, in the case of scan-path, to load repos
> from disk lazily, but that will probably add quite a lot of complexity
> and I'm not convinced it's worthwhile.
>
> That particular loop will only be executed once and I suspect it is
> dwarfed by the time spent loading and parsing the config (cached project
> list if you're using scan-path).
>
-------------- next part --------------
An HTML attachment was scrubbed...
URL: <http://lists.zx2c4.com/pipermail/cgit/attachments/20131010/d2586b69/attachment.html>