Git mirror at SF + link to "repos" script

Git mirror at SF + link to "repos" script

[Wayne Davison]

I've finished the setup of the git mirror at SF.  It is updating from
the CVS version every 10 minutes.  This lets me do all my zsh work using
git now.  I set it up this way:

I did a read-only git clone (from git://zsh.git.sf.net/gitroot/zsh) and
then put a read/write CVS checkout named "cvs" inside the .git subdir.
I have a push script that looks for that dir and chooses to run the git
cvsexportcommit command, which will save all my recent git commits to
the CVS repo.

If anyone is interested in seeing what I did to make pushing from git
easy, check out my multi-repo helper script, "repos" (look at the git_pu
and git_cvsexportcommit functions):

    http://opencoder.net/repos

That "repos" script expects to have been called via a 2-letter symlink
of the actual operation to perform:  "di", "st", "up", "lo", "ci", or
"pu".  It figures out what kind of repo you're in, and calls the right
command for that action.

Some of the "repo" script's features:

- "st" (status) for CVS implements a local-only status of changed files
  (though it doesn't list unknown files).  Use "st -u" for a combined
  remote + local status that also includes unknown files.  If the local
  status shows too many changes (e.g. if you touched a file without
  changing it), running "di" or "up" on the file will make "st" accurate
  again.

- "di" (diff) and "lo" (log) automatically pipe the output through a
  pager if STDOUT is a terminal (emulating git).  For svn it turns off
  the displaying of deleted files (i.e. --no-diff-deleted).

- "up" (update) for git does a rebase update instead of a merge update.
  For CVS it supports the -n option to disable the update.

- "pu" (push) in a git repo knows now to deal with a git-svn repo and
  (with an embedded "cvs" checkout), a cvsexportcommit repo (the cvs
  code has been lightly tested so far, but I set it up to be pedantic,
  so it should be safe).  I also made "pu" for git understand that if
  there is a remote named "pu" in the config, it should use that for the
  push (this lets you have one URL for pushing (e.g. ssh://...), and one
  for fetching (e.g. git://...), should you so desire).

- "lo" (log) for git has a -v option that makes it easier to get the
  --name-status output.  It also has an -s option that lets you see the
  git-svn log output when in a git-svn repo.

- "ci" (checkin) for git makes the default to commit everything unless
  you specify other (non-"-m") options or files.  I also made a trailing
  -i mean to commit just the changes that are currently added (which is
  the normal git default).  This makes "ci" for git behave more like
  other SCMs I'm used to.

Though the script has support for hg and bzr, I don't do much committing
to those repos, so those SCMs could probably use some more work.

Examples:

ci -m 'The commit message'
di foobar.c
lo -p        (-p is git only, but is so useful that I thought I'd mention it)

..wayne..

Re: Git mirror at SF + link to "repos" script

[Frank Terbeck]

Wayne Davison <wayned@users.sourceforge.net>:
> I've finished the setup of the git mirror at SF.  It is updating from
> the CVS version every 10 minutes.  This lets me do all my zsh work using
> git now.[...]

Awesome news.

For the record, the two URIs Wayne posted in an earlier thread (I
think he only mentioned the read-only one in the message I am replying
to) are:

    git://zsh.git.sf.net/gitroot/zsh            (read-only)
    ssh://USERNAME@zsh.git.sf.net/gitroot/zsh   (read/write)

Finally, I can put my local cvs-to-git conversion to rest.
Thanks Wayne! :-)

Regards, Frank

-- 
In protocol design, perfection has been reached not when there is
nothing left to add, but when there is nothing left to take away.
                                                  -- RFC 1925

Re: Git mirror at SF + link to "repos" script

[Ingmar Vanhassel]

On Sat, Mar 14, 2009 at 12:23:59PM -0700, Wayne Davison wrote:
> I've finished the setup of the git mirror at SF.  It is updating from
> the CVS version every 10 minutes.  This lets me do all my zsh work using
> git now:

Awesome, thanks a lot! :)

-- 
Exherbo KDE, X.org maintainer

Re: Git mirror at SF + link to "repos" script

[Wayne Davison]

On Sat, Mar 14, 2009 at 10:51:23PM +0100, Frank Terbeck wrote:
> For the record, the two URIs Wayne posted in an earlier thread (I
> think he only mentioned the read-only one in the message I am replying
> to) are:

Yes, I only mentioned the read-only one because (1) nobody else should
be writing to the git repo but the CVS importer, and (2) git: access is
faster than ssh: access (and avoids any accidental pushes).

..wayne..

Re: Git mirror at SF + link to "repos" script

[Wayne Davison]

On Sat, Mar 14, 2009 at 12:23:59PM -0700, Wayne Davison wrote:
> If anyone is interested in seeing what I did to make pushing from git
> easy, check out my multi-repo helper script, "repos" (look at the git_pu
> and git_cvsexportcommit functions):
> 
>     http://opencoder.net/repos

Note that I just uploaded an improved version that makes sure that it
stops scanning for new commits at the origin commit.  Without that, the
pulled version of a commit could be considered a candidate for pushing.

..wayne..

Re: Git mirror at SF + link to "repos" script

[Clint Adams]

On Sat, Mar 14, 2009 at 10:51:23PM +0100, Frank Terbeck wrote:
>     git://zsh.git.sf.net/gitroot/zsh            (read-only)
>     ssh://USERNAME@zsh.git.sf.net/gitroot/zsh   (read/write)
> 
> Finally, I can put my local cvs-to-git conversion to rest.
> Thanks Wayne! :-)

I'm excited about doing this too, but the git repo doesn't seem to
quite match up with CVS HEAD:


diff -ur -x .git zsh-git/Completion/Unix/Command/_mount zsh-cvs/Completion/Unix/Command/_mount
--- zsh-git/Completion/Unix/Command/_mount	2009-03-15 01:48:03.000000000 -0400
+++ zsh-cvs/Completion/Unix/Command/_mount	2009-03-01 21:54:16.000000000 -0500
@@ -867,6 +867,7 @@
     ;;
   esac
 
+  local MATCH MBEGIN MEND
   # "Mummy, why is mount point matching full of squiggles?"
   #
   # "Well, dear, the clever people who wrote Linux decided that some
diff -ur -x .git zsh-git/Test/V04features.ztst zsh-cvs/Test/V04features.ztst
--- zsh-git/Test/V04features.ztst	2009-03-15 01:48:03.000000000 -0400
+++ zsh-cvs/Test/V04features.ztst	2009-03-01 21:53:53.000000000 -0500
@@ -101,7 +101,6 @@
     zmodload zsh/datetime
   }
   fn
-# status is zero because load succeded although features not all enabled
 2:Failed to add parameter if local parameter present
 ?fn:2: Can't add module parameter `EPOCHSECONDS': local parameter exists
 ?fn:zsh/datetime:2: error when adding parameter `EPOCHSECONDS'

Re: Git mirror at SF + link to "repos" script

[Wayne Davison]

On Sun, Mar 15, 2009 at 05:59:16AM +0000, Clint Adams wrote:
> I'm excited about doing this too, but the git repo doesn't seem to
> quite match up with CVS HEAD:

The two errors you cited are both cvsps errors triggered by the same
bug:  in each case, the commits went like this:  (1) a ChangeLog commit
with a particular log message, (2) a commit to another file with the
same log message, (3) a second commit to the same extra file (within the
default fuzz factor of 5 minutes) that has the same log message as the
other 2 commits.  This caused the reverse-scan of the cvs log data to
associate the newer revision with the ChangeLog commit (giving it the
patchset time of the ChangeLog change), and then create a new patchset
entry for the 2nd, earlier entry, but one that is later in time than
the patchset of the first revision.  Ouch.

The easiest "fix" is to specify a smaller fuzz factor size, but I'm also
looking at fixing the source code to notice that is is miss-ordering the
entries in the file, since that is never the right thing to do.

When I get done, I'll create a new git mirror from the cvs repo.

..wayne..

Re: Git mirror at SF + link to "repos" script

[Clint Adams]

On Sun, Mar 15, 2009 at 09:54:37AM -0700, Wayne Davison wrote:
> The easiest "fix" is to specify a smaller fuzz factor size, but I'm also
> looking at fixing the source code to notice that is is miss-ordering the
> entries in the file, since that is never the right thing to do.

You might want to take a look at what parsecvs is doing, since I vaguely
remember that one being the only thing handling such cases properly.

Re: Git mirror at SF + link to "repos" script

[Wayne Davison]

On Sun, Mar 15, 2009 at 09:54:37AM -0700, Wayne Davison wrote:
> When I get done, I'll create a new git mirror from the cvs repo.

I fixed the cvsps that I'm running, sent off a patch to the maintainer,
and used it to create a better zsh repo.  The revised code fixes a
handful of affected files (some of which we did not notice due to the
affected revisions being back in the history of the file).

For anyone who cloned the git repository:  since the repository has been
regenerated, you'll probably want to clone it anew.

..wayne..

Re: Git mirror at SF

[Wayne Davison]

On Sun, Mar 15, 2009 at 02:20:17PM -0700, Wayne Davison wrote:
> I fixed the cvsps that I'm running, sent off a patch to the maintainer,
> and used it to create a better zsh repo.  The revised code fixes a
> handful of affected files (some of which we did not notice due to the
> affected revisions being back in the history of the file).

Unfortunately, my patched version had a bug where incremental updates
would often leave files ungrouped into a proper patch set.  I had been
working around this problem by telling cvsps to rebuild its cache file
from scratch each time there were some changes that needed to be added.

Today I implemented a better fix for the cvsps code, and am testing that
to see how it will handle the new commits that show up.  While that is
going on you may see a delay in new cvs commits showing up in the git
repo.

When I'm done, I'm going to push a brand new git repository because my
new cvsps code managed to join together a split commit into a proper
changeset, and I want to update the repository with that change.

For anyone who grabs changes via "git pull --rebase", git should sync up
just fine when the new repository goes out (I tested it, and it uses a
3-way merge for the spot where the 2 commits become one, and the end
result is identical to a fresh pull).

..wayne..

Re: Git mirror at SF

[Wayne Davison]

If anyone is interested in my patch to cvsps, I have a version for 2.1
and one for 2.2b1 (which are nearly identical):

    http://opencoder.net/cvsps-2.1.patch
    http://opencoder.net/cvsps-2.2b1.patch

..wayne..

Re: Git mirror at SF

[Frank Terbeck]

Wayne Davison <wayned@users.sourceforge.net>:
[...]
> Today I implemented a better fix for the cvsps code, and am testing that
> to see how it will handle the new commits that show up.  While that is
> going on you may see a delay in new cvs commits showing up in the git
> repo.
> 
> When I'm done, I'm going to push a brand new git repository because my
> new cvsps code managed to join together a split commit into a proper
> changeset, and I want to update the repository with that change.
> 
> For anyone who grabs changes via "git pull --rebase", git should sync up
> just fine when the new repository goes out (I tested it, and it uses a
> 3-way merge for the spot where the 2 commits become one, and the end
> result is identical to a fresh pull).

Hi Wayne,

Currently, the git repository's history seems a little off (although
the repo's contents appear to be correct).

a) I don't see any new tags (like one for 4.3.10-dev-3).

b) I think the history of the git repository should be linear,
   shouldn't it? But year it seems to be broken up in one or two
   places, see: http://ft.bewatermyfriend.org/tmp/zshgithist.png
   Note that commits (alongside their tags) from November 2008 precede
   commits from January 2008 in that history.
   I am sure, this was not the case in earlier git imports, since I
   sometimes used the git history to check when a certain change was
   made in zsh.

This is a newly cloned repository, to avoid possible problems when
using --rebase when pulling.

My current git version is: 1.6.2.2.446.gfbdc0

Any ideas? Or am I misjudging something here?

Regards, Frank

-- 
In protocol design, perfection has been reached not when there is
nothing left to add, but when there is nothing left to take away.
                                                  -- RFC 1925

Re: Git mirror at SF

[Wayne Davison]

On Sun, Apr 26, 2009 at 12:29:37PM +0200, Frank Terbeck wrote:
> a) I don't see any new tags (like one for 4.3.10-dev-3).

Tags weren't being pushed, so the newest one (I assume you meant
4.3.9-dev-3) wasn't in the public repo, and older tags were pointing
to older hash values.  See if my tag push helps your other issue too
(since I'm not sure what visualization tool you were using).  The
various http://zsh.git.sf.net/ shortlog pages show the tags in their
right places now.

..wayne..

Re: Git mirror at SF

[Frank Terbeck]

Wayne Davison <wayned@users.sourceforge.net>:
> On Sun, Apr 26, 2009 at 12:29:37PM +0200, Frank Terbeck wrote:
> > a) I don't see any new tags (like one for 4.3.10-dev-3).
> 
> Tags weren't being pushed, so the newest one (I assume you meant
> 4.3.9-dev-3) wasn't in the public repo, and older tags were pointing

That's what I meant, yeah.

> to older hash values.  See if my tag push helps your other issue too
> (since I'm not sure what visualization tool you were using).  The

It was gitk.

> various http://zsh.git.sf.net/ shortlog pages show the tags in their
> right places now.

Well, zsh-4.3.9-dev-3 is there now. 4.3.9-dev-2 and -1 are not, but it
seems the cvs repo is lacking those symbolic names, too (at least 'cvs
log' does not mention them) - so I guess that's fine.

The non-linearity of the repository didn't vanish after I pulled. I
did a re-clone after that and now I cannot find it anymore... So
I guess my previous repo was broken for some reason. So, sorry for the
noise.

Regards, Frank

-- 
In protocol design, perfection has been reached not when there is
nothing left to add, but when there is nothing left to take away.
                                                  -- RFC 1925

Re: Git mirror at SF

[Wayne Davison]

On Sun, Apr 26, 2009 at 07:44:44PM +0200, Frank Terbeck wrote:
> The non-linearity of the repository didn't vanish after I pulled. I
> did a re-clone after that and now I cannot find it anymore... So
> I guess my previous repo was broken for some reason.

The old tags were broken (outdated) in git due to my last re-gen of the
git version from cvs where I (alas) didn't re-push the tags.  For anyone
who wants the newest, fixed tags, they'll need to also do a re-clone,
since git does not update tags that it already knows about.

Glad to hear that things are all fixed up.

..wayne..

Re: Git mirror at SF + link to "repos" script

[Wayne Davison]

[-- Attachment #1: Type: text/plain, Size: 302 bytes --]

On Thu, Dec 3, 2009 at 8:50 PM, Clint Adams <clint@zsh.org> wrote:

> Did [the git mirroring] choke?
>

Yeah.  I hadn't noticed that the ssh update of the changed git archive was
failing.  It should be fine now, and I've got things setup where I should
see any future problems more readily.

..wayne..