Repository/mirrors question

Repository/mirrors question

[Jim]

[-- Attachment #1: Type: text/plain, Size: 749 bytes --]

Hi All,

Not a git expert, so I'm not sure what I'm seeing.  Maybe nothing, if so
sorry for the
bother.

Building a new system and cloned the git repository and(for whatever
reason) one of
the mirrors(github).  I noticed  the size of the two was over 15M
difference.  So I cloned
the other mirror. It is only ~1.5M larger.  So after some googling found
and executed
"git count-objects -v -H' which reported a difference in the number of
objects in one
of the mirrors(gitlab).

As I said, not a git expert so don't know if this means anything. Hopefully
someone
with git expertise can tell me if there is or isn't something amiss or if
mirror(s) need an
update/maintenance/whatever.

Sorry if this is just noise, or my lack of knowledge of git,

Jim

[-- Attachment #2: Type: text/html, Size: 1010 bytes --]

Re: Repository/mirrors question

[Bart Schaefer]

On Sat, Jan 22, 2022 at 11:16 AM Jim <linux.tech.guy@gmail.com> wrote:
>
> Building a new system and cloned the git repository and(for whatever reason) one of
> the mirrors(github).  I noticed  the size of the two was over 15M difference.

I think the difference is in the number of forks.  There are 341 forks
of the mirror on github and only 4 on the SourceForge original.

Re: Repository/mirrors question

[Jim]

[-- Attachment #1.1: Type: text/plain, Size: 1694 bytes --]

On Sat, Jan 22, 2022 at 1:24 PM Bart Schaefer <schaefer@brasslantern.com>
wrote:

> On Sat, Jan 22, 2022 at 11:16 AM Jim <linux.tech.guy@gmail.com> wrote:
> >
> > Building a new system and cloned the git repository and(for whatever
> reason) one of
> > the mirrors(github).  I noticed  the size of the two was over 15M
> difference.
>
> I think the difference is in the number of forks.  There are 341 forks
> of the mirror on github and only 4 on the SourceForge original.
>

Aren't forks just copies of the git repository, once forked then the fork
does
their own thing. That wouldn't change the repository that the fork was taken
from, would it?  'man git' hasn't any references to forks. But it wouldn't
be
the first time I was wrong. What would concern me more is the object count
on gitlab's mirror.

git count-objects -v -H  ( Take prior to Sunday's commits )
sourceforge: in-pack: 98608  size-pack: 19.53 MiB
github: in-pack: 98608   size-pack: 36.25 MiB
gitlab:  in-pack: 98494  size-pack: 21.13 MiB

After Sunday's commits, I did an object comparison between sourceforce's
repository and the two mirrors.  Ignoring the size difference, the objects
in
the github mirror match those in sourceforge.  But there are differences
between the gitlab mirror and sourceforge.  There are 124 objects in
sourceforge that are not in the gitlab mirror. But more surprising there are
10 objects in the gitlab mirror that are not in sourceforge.

Find differences in attachments.

Again, I don't know if this is an integrity issue or not.  I'll leave that
for someone else to determine. I do believe it should be brought to
someone's attention in case it is a problem.

I appreciate your time,

Jim

[-- Attachment #1.2: Type: text/html, Size: 2500 bytes --]

[-- Attachment #2: diff_gl_sf --]
[-- Type: application/octet-stream, Size: 390 bytes --]

231d7cb000b1688b66641bc81d30cbb59a457e
5623335e829cbdfde09ad0beb6a4fa9e4b3861
57a7c5b803794f938b5184abbef6a5fb6dacbf
7f517c62f0e648fd3815edb3072ebb8335809f
94d1af85cab974594fa3c1a758e1a1c53403e0
cd0499762229e6568728c281538017013c49d1
e4f9e81cef4908f159a8e826b1f6a8aba49fd0
eb75bbc67ef0571f832ca97a951e8936ebadff
ed333d550ebc71bae245b279821806b5aaf072
f8537764c08f069482bb0731f9f26e90b8ef77

[-- Attachment #3: diff_sf_gl --]
[-- Type: application/octet-stream, Size: 4836 bytes --]

00b0261ac12764902bb1cc1077f108426cf9cd
01138d0f2fd80ab8b48c4b85cea4fc4464b9d6
0407ad0e1148df3d38e0ada4312fe6593c710e
047d9e0f2f43d8905402eee6a33957ad4ab0c5
054a32b906c475bf5ff9df16c58220b218499b
0c02283983a61287a013dcb548d0250fc63515
1058f0455e46dcd3519cb681d4e178ab57a2c8
11abe01c1f2247fc078230eec205ba7e6b08d2
11b330b6f101e3a22aaf46d6b43e60ad7f39ef
16cc35596d0f007550d0f3c605562a14b689b8
17342216d8db4c97397ca18c5a3ef43a8ec8f4
1ace54cd1b4e4deaa1e5fe79771c45a940209d
1e8c1f657e7f0c06eda80d6b3fab2218b25ed7
1f4c1e1b3c515dd4cc2e58194bc4321f25a46b
1f82b072f67fdd9c80520ee8c4991879a57d46
20497ffe7a2a4bf836bc6ce4ecb0195b4fe719
21fb7ab8ba9a7d4992a45b4b1033e7ef897154
2533ab86b61bc249fd17ba77a030caf3d7a081
25c543c118c5ade169dfa24a7c8d3076bd3390
2ad7de15aeefb5a90b003f09df68e2b60e5333
2bee09c1f85fc7bdb08b6bdb142a0047ed7ed4
2d6589ba5e088b959657e35fee867d3d72e55b
2f4562c48b915b1ca189c88fac2eadb52cbb1f
3738b53372c1962fbb4e221980026f15011aa1
3866411235405300903fdad4557bf9834d6625
394986d460eaeab6d960bff5f9bc3ba4ad54fa
3c166115841e42c39a5a11fed0a3e79b12d7d1
3c16b9f1a2bfbcc7ce1335eecaea152a3c413f
3dfec62c39d1b1fe6d829445ad3a4d8fe0695e
3f8259d07ca79cf22b3b75e333a64f9dfd0f4d
404daf872d649d6c02af718943d403e003d0fb
4093659ba316e8fb1ba0559457c72ba450ca99
413980a51925aa01ac3091efb3535835d1f32a
46060cb93ceec140de2ae8bc207a5fbebf53fa
47ed968b2c4ad19ed7ec181bc60e85dfca9bd2
4851af43c680d0b72e495fc89fc83fbb210d10
48b9d9608c3d2d45fc6ea524eb242927ac44c5
49bf78dc3096d6ca0c705257a928b1acce3b82
4a6cb2f94c1775cae6b0396d2a0a6c9b583965
4c33b7489f4f814b34b79760d62cfd670949d9
4de57a7e3bdee56c7c3b7f4e0ded7a7d3d7974
4f418c1edb35b7b0d124b9653117abf0919962
50a6161176739e6cc892a47a87a1ac4ff732aa
56faa18c296d972bb05858a2a15a2a1d690160
5aca42a1f058c85206439b6d417d668a4f6795
5cb4504561df0b486e3e140d4c0434516a0f7f
5d543a72260ac9ac6774dd5b2d7505475da4a4
5f59bf710099dcaa8f22729591c2580c052db9
613e4895f7b059558c4ef211189b516dbf903d
6485ece60df85519abe3e9628cf30b82cdfd0d
68e8294a0ed0a24f72d5f6cabb7e24b2960fc1
6ad44859e5550cf006f581a12447295e7b9bc9
6d23e7ce3c8cce5e39df638f5cc01389d89d20
6e1c7ff63c0b59573f5f993e50b63791b7f651
6e64f79ea0942c55152fb0427554208479e454
6eddebad00b1c2603d5b7da9746a6f83d1790e
6ffdfdb2a8d0c9c63de34d537a9cfefcb744dc
72325d2e9e2b44d817161a6a1741434ad8f394
769f3aa3484b6f0df05ca86a024e030a72142a
7afaafa9a0f2d618746c371ee79662ac31061f
7ccb60409c5508410b163e67c1164955724801
7ccd7f1efae7696dc17f6e3e720cd994e90155
8035c93c5bc149e9c3babd48c8c145609e29e0
81e64c002e1b2c1b389e4c2a45684524bdd3e7
8323dadd9dc2dd2246ba01f7d595939d80889f
8374fd003d43d02ca3de8006a832f1b7b5ac4c
86eb982ba89eaf5ae323cff206ae84ad279b9e
87145a4655767cc29dc053be082ede0dfef487
886b5bac75cda19506ba357136200ccf2fe2d4
89e7cefa4d3806f3f469d5ad4b496d130305a6
8d620a97dd7130e40cb6067d0063ff664499ca
8f3e32819d95301a2103fbfa190688879b04d2
8ff68e0e025111088b6ba3977b26a1794a4045
9039499c06d2908572f0e4934ae799aa818bee
915f9d1b95853333e76410d7cd30562385e733
935ae187fef33477a8dfa900bf7a8c59588477
93bcc363d98b74f3ded52fad4ecede930d245a
93ce4e61290d15ee7e53c15a2b8425e20c806c
99e79d25cd1cb1f718a060eadb4bb55080d816
9a1f5e80d018e28cf6760cda7ffe18cc5a2df8
9a3eccf74519effbf494e107744a4d7ebf57bc
9af2bf3c7eba33c1425b2dcd2071a2c5bb1179
9ca4f6e2205e86df0731b46ba2dbffc50c6225
a25afe323abe0183c2a530c5324e7e93426b64
a4945fde48a99974192a9670eae88c5044b983
a6c7ca58c8d7a09126bb5af05b87ddaadf16f3
a7cde2efe7e7d3dfe65661039bc74c7b024496
ae7720ba5e9d659776a3bc03da8f4f4db052d2
b21f218b7f72b0746c9eb5096bc979956d004b
b2a40c5b7d7773c7587bfef36b13d2fedafbb5
b661623c9838521ba90d764567532651ed7108
b9289d1277bf71b1b9dea2addbddd36b61f276
ba4cbc7da05ff77a7308d35d10a5d4df4e6678
bd4a197c0df766d84dcaadbd76bc7a8e94a6ce
bd99b67f1a2e09dff651f102249c441529d530
be1f7883e4786d506d95d0e80a5188d2c7286f
c0078fe2a08b057c4cb635dc8cbb4889a718dd
c1125316b07ddc9d51fa032fffc5b09b98f2ac
c49011944484608ecbc779c0b20a02274f2db0
c52f0dfb787f2351ce81ec33d3a0db11fc53b7
c534ce950063cb8e606428a7b9b167e366becc
c6989a564cfb079b73a1f13a3d2a89ff881bb9
d009a11ac4ae3bf81b1e300d5721f7b48eb883
d0cb2e035f4fa93a2f4851bedb612067ff7f1a
d3a85b48fb47034db266728f453eb629b11e1d
d4439a9f3a2212d753ece6ef644aed2ec4d333
d4cad44b06062092bf65bd800bf9c75aa4e2d3
d4eb7f7b2e0a3097e01b207131d15fd4b418ce
d7e8bf2b2e7f809d4a1c0587efac038ce7a73f
dae7f05582c6c8e91501dbea6106636cd76f6c
dec73420804cbe74a6cab7aab4ba40fb260757
e394ed33f4af82ee92295da119cd9a5adeadd1
e5c76c0c429f189a1985ff14caa092459f6ab5
ebead7ec64e9c44a679394cadc229732114cee
ecce55a3c68b7df2fc4017b65485fe6f07d2fd
efb784ff7c14118ef00f605be9ddb284a684c6
efc438c913466747e219618ca768e0d6caba28
f0b641f04f6439e82d768cf916aa68c39ca68e
f2a5d0c868e2a2263bbfaf98713f43477ba896
f727a628ea701d9123cf8da0420f3c3fa64972
f74cb2303532c85fc43b098dc85a4965211cd8
f85796ac15c871cf760f6415424f4f96c0a460
fa4c36896e0d89bf4b32388db08fc1a8523dd4
fd3854b4abea8b46a2ad568762d3a710af77d9

Re: Repository/mirrors question

[Phil Pennock]

On 2022-01-23 at 15:42 -0600, Jim wrote:
> Aren't forks just copies of the git repository, once forked then the fork
> does
> their own thing. That wouldn't change the repository that the fork was taken
> from, would it?  'man git' hasn't any references to forks. But it wouldn't
> be
> the first time I was wrong. What would concern me more is the object count
> on gitlab's mirror.

I'm going to answer the question and explain a few ways this can happen,
but the last section below has the actual explanation of what's
happening here: GitLab has one dev branch behind the others.  Whoever
owns that copy should take a look at why that branch wedged.

---

So a fork is not integral to git, it's a feature of the various git
forges.  There have been a number of interesting stories over the years
about the fact that many of the forges did not isolate the
content-addressed namespaces (the bits where you go in by object ID),
leading to people faking up things to "appear" to be in the original
repository.

Thus if B forks from A and then adds commits and objects, going via any
tree reference or commit in the hosted A won't show those items, and
cloning _shouldn't_, normally, but you have been able to use the direct
SHA1 of the objects to still reach them via the hosted A.  I _think_
GitHub put in some mitigations because they got tired of the games
people were playing.

That said, often a pull-request, and its objects, _are_ available in the
original repository, but not in the normal heads.  So a regular `git
clone` wont see them, but if you do a `git clone --mirror` then you
might see more.  Eg, given:

[remote "origin"]
   url = https://github.com/a/b
   fetch = +refs/heads/*:refs/remotes/origin/*
   fetch = +refs/pull/*/head:refs/remotes/pr/*

then GitHub's extra "pull" area becomes visible under a pseudo-remote of
"pr", letting you diff locally and do fancier stuff.

Since `--mirror` will set up `fetch = +refs/*:refs/*` you would then see
all of the pull-requests from all of the forks show up within the
repository.

---

The next major difference is in history and what happens as branches get
abandoned.  `git prune` honors presence in the "reflog" for a certain
amount of time, so that you can go back to recently-deleted items.

---

And then there's the fact that git transfers "packs" and then the loose
objects, and if the pack contains extra items since deleted, then you'll
end up with a bunch of now-garbage items in the pack.  "git help repack"
might be your friend for your own repos, but doesn't help with
controlling what the forges are doing.

---

In some fresh clones,

         ReachableObjects  CommitGraphCount
GitHub              98615             14158
GitLab              98501             14132
SF                  98615             14158

So GitLab sees 26 fewer commits in the entire graph than the other two.
But is up-to-date.  So, we check which branches exist and what they
point at, and find one branch discrepancy.

The `declarednull` branch is zsh-5.8-401-gf85cb4504 on GH and SF, but
zsh-5.8-270-g6bcd04997 on GitLab.

So that branch is 131 commits behind, but given the difference in graph
commits, I suspect that there has been some merging along the way.

You can get a robust API and see where the commits are for each remote
using `git ls-remote --heads` for any given remote; that output will be
diffable.  I cheated and used a porcelain command, which is not
guaranteed stable for scripting:

  git branch -r | grep -v /HEAD | while read B; do
    printf '%30s\t' $B
    git describe --tags $B
  done

which had the advantage of letting me see where the commits were
relative to tags, so we can see the "401 commits since 5.8" vs the
"270 commits since 5.8".

The last commit on branch origin/declarednull in the gitlab repo is from
2020-11-28, the last on the other two is from 2021-04-13.

-Phil

Re: Repository/mirrors question

[Bart Schaefer]

On Sun, Jan 23, 2022 at 4:15 PM Phil Pennock
<zsh-workers+phil.pennock@spodhuis.org> wrote:
>
> So GitLab sees 26 fewer commits in the entire graph than the other two.
> But is up-to-date.  So, we check which branches exist and what they
> point at, and find one branch discrepancy.
>
> The `declarednull` branch is zsh-5.8-401-gf85cb4504 on GH and SF, but
> zsh-5.8-270-g6bcd04997 on GitLab.
>
> So that branch is 131 commits behind, but given the difference in graph
> commits, I suspect that there has been some merging along the way.

Hmm.  I created and manipulated that branch on the SF repository, but
I do not have accounts on either github or GitLab, so there's nothing
I can do about it there.

Could this be a side-effect of a forced push?

That branch has been abandoned and the relevant changes from it
applied to the master branch, so it can probably just be discarded, if
that's possible.

Re: Repository/mirrors question

[Daniel Shahaf]

Phil Pennock wrote on Sun, Jan 23, 2022 at 19:14:24 -0500:
> On 2022-01-23 at 15:42 -0600, Jim wrote:
> > Aren't forks just copies of the git repository, once forked then the fork
> > does
> > their own thing. That wouldn't change the repository that the fork was taken
> > from, would it?  'man git' hasn't any references to forks. But it wouldn't
> > be
> > the first time I was wrong. What would concern me more is the object count
> > on gitlab's mirror.
> 
> I'm going to answer the question and explain a few ways this can happen,
> but the last section below has the actual explanation of what's
> happening here: GitLab has one dev branch behind the others.  Whoever
> owns that copy should take a look at why that branch wedged.

That'd be Oliver and I.

When I go to [1] it has a "diverged from upstream" warning label, in red
with a ⚠ next to it, and the following mouseover text:

    "The branch could not be updated automatically because it has
    diverged from its upstream counterpart.<br>To discard the
    local changes and overwrite the branch with the upstream version,
    delete it here and choose 'Update Now' above."

That does sound like the fallout of a forced push.

Reviewing the branch, everything on it is already on the sf.net copy of
the branch.

I've done what the mouseover text recommended.  The tip of that branch
on gitlab is now identical to its tip on sf.

Cheers,

Daniel

[1] https://gitlab.com/zsh-org/zsh/-/branches?state=all&sort=updated_desc&search=declarednull