The road to v0.10.1 or v0.11

The road to v0.10.1 or v0.11

[Jason]

Hi guys,

Here's what I'm thinking about for the next release (or releases?):

+ FastCGI support
+ More malloc()/free() cleanups
+ git-blame support
+ git-grep support
+ HTML5 compliance
+ More filters everywhere
+ Expanded test suite
+ Line number anchors highlighting the current line
+ sendfile() support

This is in no particular order. Any opinions about priorities? Or
anything else to add?

Jason

The road to v0.10.1 or v0.11

[john]

On Fri, Jan 17, 2014 at 05:12:26PM +0100, Jason A. Donenfeld wrote:
> Here's what I'm thinking about for the next release (or releases?):
> 
> + FastCGI support

I really can't see this being sensible without moving to libgit2.  As
long as we stick with libgit.a then we need to fork for each request so
I'm not sure there's much benefit to supporting FastCGI without moving
to something that lets us free resources when we're done processing a
request.

> + More malloc()/free() cleanups
> + git-blame support
> + git-grep support
> + HTML5 compliance
> + More filters everywhere
> + Expanded test suite
> + Line number anchors highlighting the current line
> + sendfile() support
> 
> This is in no particular order. Any opinions about priorities? Or
> anything else to add?

I'd like to get "new graph implementation" into this list - it's come up
on the list twice in the last 24 hours!  That doesn't mean I'm claiming
the task though ;-)

The road to v0.10.1 or v0.11

[Jason]

On Fri, Jan 17, 2014 at 5:28 PM, John Keeping <john at keeping.me.uk> wrote:
> I really can't see this being sensible without moving to libgit2.  As
> long as we stick with libgit.a then we need to fork for each request so
> I'm not sure there's much benefit to supporting FastCGI without moving
> to something that lets us free resources when we're done processing a
> request.

The advantage would be not having to reparse the config and scan for
repos on every.single.solitary.request.

> I'd like to get "new graph implementation" into this list - it's come up
> on the list twice in the last 24 hours!  That doesn't mean I'm claiming
> the task though ;-)

That's what I forgot!

+ new graph implementation

The road to v0.10.1 or v0.11

[john]

On Fri, Jan 17, 2014 at 05:38:29PM +0100, Jason A. Donenfeld wrote:
> On Fri, Jan 17, 2014 at 5:28 PM, John Keeping <john at keeping.me.uk> wrote:
> > I really can't see this being sensible without moving to libgit2.  As
> > long as we stick with libgit.a then we need to fork for each request so
> > I'm not sure there's much benefit to supporting FastCGI without moving
> > to something that lets us free resources when we're done processing a
> > request.
> 
> The advantage would be not having to reparse the config and scan for
> repos on every.single.solitary.request.

But scan for repos is caught by the cache most of the time, and
presumably even if we run persistently we still need to do that
periodically (or use inotify); or do we just rely on the process being
replaced when the set of repositories changes?

The road to v0.10.1 or v0.11

[Jason]

On Fri, Jan 17, 2014 at 5:53 PM, John Keeping <john at keeping.me.uk> wrote:
> But scan for repos is caught by the cache most of the time, and
> presumably even if we run persistently we still need to do that
> periodically (or use inotify); or do we just rely on the process being
> replaced when the set of repositories changes?

Generally the idea is you restart the fcgi process when things change,
or at least send it a SIGUSR1. But we could be fancy and support
inotify/kqueue...

The road to v0.10.1 or v0.11

[mricon]

On 17/01/14 01:22 PM, Jason A. Donenfeld wrote:
>> But scan for repos is caught by the cache most of the time, and
>> > presumably even if we run persistently we still need to do that
>> > periodically (or use inotify); or do we just rely on the process being
>> > replaced when the set of repositories changes?
> Generally the idea is you restart the fcgi process when things change,
> or at least send it a SIGUSR1. But we could be fancy and support
> inotify/kqueue...

The process that updates the repositories may not have permissions to
send SIGUSR1 to the fcgid process -- either because they are running as
different users or because there are SELinux policies preventing it.

It's really best if cgit recognizes when things like projects.list have
changed.

Best,
-- 
Konstantin Ryabitsev
Senior Systems Administrator
Linux Foundation Collab Projects
Montr?al, Qu?bec

-------------- next part --------------
A non-text attachment was scrubbed...
Name: signature.asc
Type: application/pgp-signature
Size: 713 bytes
Desc: OpenPGP digital signature
URL: <http://lists.zx2c4.com/pipermail/cgit/attachments/20140117/49ddb4b9/attachment-0001.asc>

The road to v0.10.1 or v0.11

[john]

On Fri, Jan 17, 2014 at 02:29:19PM -0500, Konstantin Ryabitsev wrote:
> On 17/01/14 01:22 PM, Jason A. Donenfeld wrote:
> >> But scan for repos is caught by the cache most of the time, and
> >> > presumably even if we run persistently we still need to do that
> >> > periodically (or use inotify); or do we just rely on the process being
> >> > replaced when the set of repositories changes?
> > Generally the idea is you restart the fcgi process when things change,
> > or at least send it a SIGUSR1. But we could be fancy and support
> > inotify/kqueue...
> 
> The process that updates the repositories may not have permissions to
> send SIGUSR1 to the fcgid process -- either because they are running as
> different users or because there are SELinux policies preventing it.
> 
> It's really best if cgit recognizes when things like projects.list have
> changed.

Presumably you are OK with this having the same latency as the existing
cache mechanism.  The simplest implementation will probably be to keep
the existing "cache valid?" check and re-scan repositories as we
currently do.

The road to v0.10.1 or v0.11

[Jason]

On Fri, Jan 17, 2014 at 8:32 PM, John Keeping <john at keeping.me.uk> wrote:
> Presumably you are OK with this having the same latency as the existing
> cache mechanism.  The simplest implementation will probably be to keep
> the existing "cache valid?" check and re-scan repositories as we
> currently do.

Or even just stat() the config / directory, if inotify is too cumbersome.

The road to v0.10.1 or v0.11

[Jason]

On Fri, Jan 17, 2014 at 8:29 PM, Konstantin Ryabitsev <mricon at kernel.org> wrote:
>
> The process that updates the repositories may not have permissions to
> send SIGUSR1 to the fcgid process -- either because they are running as
> different users or because there are SELinux policies preventing it.
>
> It's really best if cgit recognizes when things like projects.list have
> changed.

I had considered this as well. Indeed it is best if cgit notices the
change, either by inotify, or just by a simple stat and timestamp
change.

The road to v0.10.1 or v0.11

[Jason]

On Fri, Jan 17, 2014 at 5:38 PM, Jason A. Donenfeld <Jason at zx2c4.com> wrote:
> On Fri, Jan 17, 2014 at 5:28 PM, John Keeping <john at keeping.me.uk> wrote:
>> I really can't see this being sensible without moving to libgit2.  As
>> long as we stick with libgit.a then we need to fork for each request so
>> I'm not sure there's much benefit to supporting FastCGI without moving
>> to something that lets us free resources when we're done processing a
>> request.
>
> The advantage would be not having to reparse the config and scan for
> repos on every.single.solitary.request.

Oh, and we could avoid a fork() for cached pages...

The road to v0.10.1 or v0.11

[john]

On Fri, Jan 17, 2014 at 06:09:15PM +0100, Jason A. Donenfeld wrote:
> On Fri, Jan 17, 2014 at 5:38 PM, Jason A. Donenfeld <Jason at zx2c4.com> wrote:
> > On Fri, Jan 17, 2014 at 5:28 PM, John Keeping <john at keeping.me.uk> wrote:
> >> I really can't see this being sensible without moving to libgit2.  As
> >> long as we stick with libgit.a then we need to fork for each request so
> >> I'm not sure there's much benefit to supporting FastCGI without moving
> >> to something that lets us free resources when we're done processing a
> >> request.
> >
> > The advantage would be not having to reparse the config and scan for
> > repos on every.single.solitary.request.
> 
> Oh, and we could avoid a fork() for cached pages...

Good point.  I think that probably does make it worthwhile.

It may even make sense for a FastCGI process to do:

    while read_request
        if not in cache:
            process and exit
        return_cache

and just rely on the web server restarting us.