From mboxrd@z Thu Jan 1 00:00:00 1970 X-Spam-Checker-Version: SpamAssassin 3.4.4 (2020-01-24) on inbox.vuxu.org X-Spam-Level: X-Spam-Status: No, score=-3.4 required=5.0 tests=DKIM_SIGNED,DKIM_VALID, DKIM_VALID_AU,MAILING_LIST_MULTI,RCVD_IN_DNSWL_MED,UNPARSEABLE_RELAY autolearn=ham autolearn_force=no version=3.4.4 Received: (qmail 11326 invoked from network); 24 Jan 2022 00:15:09 -0000 Received: from zero.zsh.org (2a02:898:31:0:48:4558:7a:7368) by inbox.vuxu.org with ESMTPUTF8; 24 Jan 2022 00:15:09 -0000 ARC-Seal: i=1; cv=none; a=rsa-sha256; d=zsh.org; s=rsa-20210803; t=1642983309; b=D3sUlc5Iug3CbEPpAkwBSQOYoUEfmhO0beFOpp+uyL6jK/TnshYZD2XOVUaMus4JMEhkfeLw/X OddIXyYNg53rjr3bA7+ceeuWtzTLvWyP0tm9Yn/MQcKfhU4IXRsW75u+ktK1GBeXriKi9cAkxf 5d87OJ/JsHUmnj/6vwtxd4ngPbWk8KvR1GV2f0/4LIZIgnb3UIlCQpDTlBxYZqQq2QBNmUuSQ5 i3AADx2W3F86O9//S2ExeGIUGloFu5aAs67TrJubnCtVF3kxW6v2cLwh7NTjYCRHd1QcNBMr2e 6rSrrg5XzXUtNQm367Nr8ccaksdK5+EpBX+OHz4mCy9K9A==; ARC-Authentication-Results: i=1; zsh.org; iprev=pass (mx.spodhuis.org) smtp.remote-ip=94.142.241.89; dkim=pass header.d=spodhuis.org header.s=d202111e2 header.a=ed25519-sha256; dkim=pass header.d=spodhuis.org header.s=d202111 header.a=rsa-sha256; dmarc=pass header.from=spodhuis.org; arc=none ARC-Message-Signature: i=1; a=rsa-sha256; c=relaxed; d=zsh.org; s=rsa-20210803; t=1642983309; bh=uBuWjdADSVTT6F4qsAYiSJGywBkcyzQWmp/C5U9UK70=; h=List-Archive:List-Owner:List-Post:List-Unsubscribe:List-Subscribe:List-Help: List-Id:Sender:In-Reply-To:Content-Type:MIME-Version:References:Message-ID: Subject:Cc:To:From:Date:DKIM-Signature:DKIM-Signature:DKIM-Signature; b=hvLKX5+0UOcv2JjZ2bpvInvM2oowzk0fRjm79i5EoL8Ec1zjjHn1vqOj+Iccd8hOQCL24e42Lr uK0ZgHvj/ML+I2ik33Udr5lmW4gBndMpET6158qiRpvqF6+krcrX1kdCcGpAkZHI3ErE5jdFPq oUwOQj9Bnv7UjCejzSc8rPia/tOE+UY0g1SRlWMgFSzqtjmhJ2LVKFr3/uddHv1xb6bcKiE4o2 g/hDxDVKfgAakPThpun7tl7UgwrNZxw2eBaNsKV2zMW7n+DgFxPCaLPhN3bNFFWbMY0z3eqX7R R0xIuF/Nuc5tMgPNIvqgEIay7adgv3MMb/qf9OKjxaYvLQ==; DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=zsh.org; s=rsa-20210803; h=List-Archive:List-Owner:List-Post:List-Unsubscribe: List-Subscribe:List-Help:List-Id:Sender:In-Reply-To:Content-Type:MIME-Version :References:Message-ID:Subject:Cc:To:From:Date:Reply-To: Content-Transfer-Encoding:Content-ID:Content-Description:Resent-Date: Resent-From:Resent-Sender:Resent-To:Resent-Cc:Resent-Message-ID; bh=FGMt1sfDw1WaTiVKF3AsEA8nBnUUbl+PzGS/nCcTtAs=; b=c/dadEX4Xmh3600DI7W+h9xEEC VuHJeWTQ9Uhpu5GYKU/9uxeuEI0iJ+KMeIHngSmDYvUDW4C+FpkFZu17V8RxT2aHG+PslzsGb9G7c bMpxN4jvtqXF90fxTXc/jMaZPkNbNGO65FP0GWaUo8s8v9Qt3MwPiDEljx89l5I0qmAGI7BDvJZ72 RI0e0ds+gS8hUGsQiPGevyo6xANBAoMeGZOexhsabd/QU9DHivYi8hUqJyHrCPD8JtPcTuIzPXkrD +yp4Fl7fao5Rg6hoIlwGUKDObNAkHVDW4s7kNIoRKxW4bNbL2SOYu4ewN67VRU8lLm5QIrFPRZvnt THkCrO2w==; Received: from authenticated user by zero.zsh.org with local id 1nBn0k-00027D-0K; Mon, 24 Jan 2022 00:15:06 +0000 Authentication-Results: zsh.org; iprev=pass (mx.spodhuis.org) smtp.remote-ip=94.142.241.89; dkim=pass header.d=spodhuis.org header.s=d202111e2 header.a=ed25519-sha256; dkim=pass header.d=spodhuis.org header.s=d202111 header.a=rsa-sha256; dmarc=pass header.from=spodhuis.org; arc=none Received: from mx.spodhuis.org ([94.142.241.89]:39007) (DNSSEC AD) by zero.zsh.org with esmtps (TLS1.3:TLS_AES_256_GCM_SHA384:256) id 1nBn08-0001mp-HK; Mon, 24 Jan 2022 00:14:28 +0000 DKIM-Signature: v=1; a=rsa-sha256; q=dns/txt; c=relaxed/relaxed; d=spodhuis.org; s=d202111; h=OpenPGP:In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:From:Reply-To:Subject:Date:To: Cc:Content-Transfer-Encoding:Content-ID:Content-Description:OpenPGP: Organization:Auto-Submitted; bh=FGMt1sfDw1WaTiVKF3AsEA8nBnUUbl+PzGS/nCcTtAs=; t=1642983268; x=1644192868; b=ixlOLmMZBmXC/XujDpJJR6BJ3feSs9X8ev9b8oCcx+2X0qv Z9Jhp/fudRDcFAE5CdmQ89v6kWZIgnKyQZR50k2ZR1BTk46K0DBVQDpNWsK1Ai7mFk0sFCXi1Zy9F VZaWG+usX5thGQ9mRyfWQDFSp3Bx6W5nxBNppjlYQOvTgNQHyI7vwm8YOBD6+lxILc4xXBCyoLoGK jBRBKK4RcGlbMJARJiEEW98ycf6OwbeTM4HddgDlTYXeZYLOP6fA81BzpZV+0hSDpdzC46Ma1XxhG QFoRZGrYgzYRdo+wOwsB5gDuMIEhED9r9HAIlJyXMXzkqvjuLlPM3KmU63AZOWqg==; DKIM-Signature: v=1; a=ed25519-sha256; q=dns/txt; c=relaxed/relaxed; d=spodhuis.org; s=d202111e2; h=OpenPGP:In-Reply-To:Content-Type:MIME-Version: References:Message-ID:Subject:Cc:To:From:Date:From:Reply-To:Subject:Date:To: Cc:Content-Transfer-Encoding:Content-ID:Content-Description:OpenPGP: Organization:Auto-Submitted; bh=FGMt1sfDw1WaTiVKF3AsEA8nBnUUbl+PzGS/nCcTtAs=; t=1642983268; x=1644192868; b=jSSxBRhGe4HE2lWHl2cVe+V08U6KeSqYptXY6PtQq4+YOzu 3YBM6Rf/dlP1sQYw6Kl5bPSegDVWaKRRgtaBDAA==; Received: from authenticated user by smtp.spodhuis.org with esmtpsa (TLS1.3:TLS_AES_256_GCM_SHA384:256) id 1nBn07-0001mf-7v; Mon, 24 Jan 2022 00:14:27 +0000 Date: Sun, 23 Jan 2022 19:14:24 -0500 From: Phil Pennock To: linuxtechguy@gmail.com Cc: devs Subject: Re: Repository/mirrors question Message-ID: Mail-Followup-To: linuxtechguy@gmail.com, devs References: MIME-Version: 1.0 Content-Type: text/plain; charset=us-ascii Content-Disposition: inline In-Reply-To: OpenPGP: url=https://www.security.spodhuis.org/PGP/keys/keys-2013rsa-2020cv25519.asc X-Seq: 49706 Archived-At: X-Loop: zsh-workers@zsh.org Errors-To: zsh-workers-owner@zsh.org Precedence: list Precedence: bulk Sender: zsh-workers-request@zsh.org X-no-archive: yes List-Id: List-Help: List-Subscribe: List-Unsubscribe: List-Post: List-Owner: List-Archive: On 2022-01-23 at 15:42 -0600, Jim wrote: > Aren't forks just copies of the git repository, once forked then the fork > does > their own thing. That wouldn't change the repository that the fork was taken > from, would it? 'man git' hasn't any references to forks. But it wouldn't > be > the first time I was wrong. What would concern me more is the object count > on gitlab's mirror. I'm going to answer the question and explain a few ways this can happen, but the last section below has the actual explanation of what's happening here: GitLab has one dev branch behind the others. Whoever owns that copy should take a look at why that branch wedged. --- So a fork is not integral to git, it's a feature of the various git forges. There have been a number of interesting stories over the years about the fact that many of the forges did not isolate the content-addressed namespaces (the bits where you go in by object ID), leading to people faking up things to "appear" to be in the original repository. Thus if B forks from A and then adds commits and objects, going via any tree reference or commit in the hosted A won't show those items, and cloning _shouldn't_, normally, but you have been able to use the direct SHA1 of the objects to still reach them via the hosted A. I _think_ GitHub put in some mitigations because they got tired of the games people were playing. That said, often a pull-request, and its objects, _are_ available in the original repository, but not in the normal heads. So a regular `git clone` wont see them, but if you do a `git clone --mirror` then you might see more. Eg, given: [remote "origin"] url = https://github.com/a/b fetch = +refs/heads/*:refs/remotes/origin/* fetch = +refs/pull/*/head:refs/remotes/pr/* then GitHub's extra "pull" area becomes visible under a pseudo-remote of "pr", letting you diff locally and do fancier stuff. Since `--mirror` will set up `fetch = +refs/*:refs/*` you would then see all of the pull-requests from all of the forks show up within the repository. --- The next major difference is in history and what happens as branches get abandoned. `git prune` honors presence in the "reflog" for a certain amount of time, so that you can go back to recently-deleted items. --- And then there's the fact that git transfers "packs" and then the loose objects, and if the pack contains extra items since deleted, then you'll end up with a bunch of now-garbage items in the pack. "git help repack" might be your friend for your own repos, but doesn't help with controlling what the forges are doing. --- In some fresh clones, ReachableObjects CommitGraphCount GitHub 98615 14158 GitLab 98501 14132 SF 98615 14158 So GitLab sees 26 fewer commits in the entire graph than the other two. But is up-to-date. So, we check which branches exist and what they point at, and find one branch discrepancy. The `declarednull` branch is zsh-5.8-401-gf85cb4504 on GH and SF, but zsh-5.8-270-g6bcd04997 on GitLab. So that branch is 131 commits behind, but given the difference in graph commits, I suspect that there has been some merging along the way. You can get a robust API and see where the commits are for each remote using `git ls-remote --heads` for any given remote; that output will be diffable. I cheated and used a porcelain command, which is not guaranteed stable for scripting: git branch -r | grep -v /HEAD | while read B; do printf '%30s\t' $B git describe --tags $B done which had the advantage of letting me see where the commits were relative to tags, so we can see the "401 commits since 5.8" vs the "270 commits since 5.8". The last commit on branch origin/declarednull in the gitlab repo is from 2020-11-28, the last on the other two is from 2021-04-13. -Phil