[9fans] Software preservation in the post-hg era

[9fans] Software preservation in the post-hg era

[Sean Hinchee]

In the wake of Bitbucket removing hg (Mercurial) support [1], I feel
it's topical to bring up software preservation for the plan9
community.

A lot of community contributed software has been put up on Bitbucket
or other hg hosts over time (RIP Google Code), but no consolidated
effort, to my knowledge, seems to have been made to index, let alone
mirror, this software.

For now, as a stop-gap, I've made a GitHub organization in which I've
consolidated most of what I had indexed from Bitbucket and a few other
places.

Thanks to people like Ori Bernstein, we have a native git client for
plan9 [3]; without a native client, this kind of transition wouldn't
be nearly as simple, thank you.

I'm more than happy to add anyone interested in the curation of this
archive to the GitHub organization. It would be nice to have spare
hands around to add README's, mkfiles, and attributions where they
have been missed or never existed.

In the long term, it would be nice to have a federated or otherwise
decentralized solution to pooling community contributed software,
especially keeping in mind ease of mirroring and picking up old
projects as contributors come and go.

The contrib/ directory on sources and 9front are fine and good, but
they are centralized. I don't have a proposed solution to this
problem, but it would be nice to have ideas or insight posted ☺.

I recognize that GitHub is also centralized and doesn't solve the
centralization problem, but at least git is really straightforward to
mirror with multiple remotes, etc. and having an index/archive is
valuable at least to me.

If anyone has further thoughts, anything they want added, or any lists
or indices of works they want archived/mirrored, I would love to see
these posted.

If anyone wants to mirror the archive, that would be wonderful. I was
considering mirroring everything to a remote in sr.ht in the future,
but haven't gotten around to it.

As a footnote, there's a decent git client written in Go that works
alright on plan9 [4], but it's slow and memory intensive at the
moment.

Cheers,
Sean

[1] https://twitter.com/traverser/status/1244398479591563265
[2] https://github.com/Plan9-Archive
[3] https://github.com/oridb/git9
[4] https://github.com/driusan/dgit

Re: [9fans] Software preservation in the post-hg era

[Fazlul Shahriar]

[-- Attachment #1: Type: text/plain, Size: 258 bytes --]

>
> As a footnote, there's a decent git client written in Go that works
> alright on plan9 [4], but it's slow and memory intensive at the
> moment.


 go-git also works well. I recently wrote a wrapper around it for plan9:
https://github.com/fhs/gig

[-- Attachment #2: Type: text/html, Size: 528 bytes --]

Re: [9fans] Software preservation in the post-hg era

[ori]

> If anyone has further thoughts, anything they want added, or any lists
> or indices of works they want archived/mirrored, I would love to see
> these posted.

I think the lede got buried here, and people went of discussing git
clients.

Given that the aptly-named bitbucket is deleting mercurial repositories,
is there anything of interest that should be retrieved before they empty
the bitbucket onto the curb?

Re: [9fans] Software preservation in the post-hg era

[Lucio De Re]

On 3/31/20, ori@eigenstate.org <ori@eigenstate.org> wrote:
>
> I think the lede got buried here, and people went of discussing git
> clients.
>
Entirely your fault: that's the thanks you get for doing such a good job.

Lucio.

Re: [9fans] Software preservation in the post-hg era

[Dave MacFarlane]

On Tue, Mar 31, 2020 at 10:46 AM <ori@eigenstate.org> wrote:
>
> > If anyone has further thoughts, anything they want added, or any lists
> > or indices of works they want archived/mirrored, I would love to see
> > these posted.
>
> I think the lede got buried here, and people went of discussing git
> clients.
>
> Given that the aptly-named bitbucket is deleting mercurial repositories,
> is there anything of interest that should be retrieved before they empty
> the bitbucket onto the curb?
>

Is Inferno already mirrored?

Re: [9fans] Software preservation in the post-hg era

[Xiao-Yong Jin]

Thanks for doing this.
Please add me (jxy) to the GitHub org.
I plan to maintain drawterm-metal so long as I'm using it.

Re: [9fans] Software preservation in the post-hg era

[Sigrid Solveig Haflínudóttir]

On Tue, Mar 31, 2020 at 8:38 PM Dave MacFarlane <driusan@gmail.com> wrote:
>
> On Tue, Mar 31, 2020 at 10:46 AM <ori@eigenstate.org> wrote:
> > > If anyone has further thoughts, anything they want added, or any lists
> > > or indices of works they want archived/mirrored, I would love to see
> > > these posted.
> >
> > I think the lede got buried here, and people went of discussing git
> > clients.
> >
> > Given that the aptly-named bitbucket is deleting mercurial repositories,
> > is there anything of interest that should be retrieved before they empty
> > the bitbucket onto the curb?
> >
>
> Is Inferno already mirrored?

If https://github.com/Plan9-Archive/inferno-os is the one then yes.
There are multiple other forks of Inferno there.

Re: [9fans] Software preservation in the post-hg era

[Fazlul Shahriar]

[-- Attachment #1: Type: text/plain, Size: 3250 bytes --]

It's worth highlighting the fact that Bitbucket will be deleting *all* hg
repositories on May 30, 2020. Thankfully, the Software Heritage (
https://www.softwareheritage.org/) seems to be doing a great job of
archiving all open source work. For example:
https://archive.softwareheritage.org/browse/origin/https://bitbucket.org/inferno-os/inferno-os/directory/
I don't know how much they have archived or how up-to-date it is. It's
worth converting repos to git anyway, especially if we want to continue
maintaining them.

On Mon, Mar 30, 2020 at 9:11 PM Sean Hinchee <henesy.dev@gmail.com> wrote:

> In the wake of Bitbucket removing hg (Mercurial) support [1], I feel
> it's topical to bring up software preservation for the plan9
> community.
>
> A lot of community contributed software has been put up on Bitbucket
> or other hg hosts over time (RIP Google Code), but no consolidated
> effort, to my knowledge, seems to have been made to index, let alone
> mirror, this software.
>
> For now, as a stop-gap, I've made a GitHub organization in which I've
> consolidated most of what I had indexed from Bitbucket and a few other
> places.
>
> Thanks to people like Ori Bernstein, we have a native git client for
> plan9 [3]; without a native client, this kind of transition wouldn't
> be nearly as simple, thank you.
>
> I'm more than happy to add anyone interested in the curation of this
> archive to the GitHub organization. It would be nice to have spare
> hands around to add README's, mkfiles, and attributions where they
> have been missed or never existed.
>
> In the long term, it would be nice to have a federated or otherwise
> decentralized solution to pooling community contributed software,
> especially keeping in mind ease of mirroring and picking up old
> projects as contributors come and go.
>
> The contrib/ directory on sources and 9front are fine and good, but
> they are centralized. I don't have a proposed solution to this
> problem, but it would be nice to have ideas or insight posted ☺.
>
> I recognize that GitHub is also centralized and doesn't solve the
> centralization problem, but at least git is really straightforward to
> mirror with multiple remotes, etc. and having an index/archive is
> valuable at least to me.
>
> If anyone has further thoughts, anything they want added, or any lists
> or indices of works they want archived/mirrored, I would love to see
> these posted.
>
> If anyone wants to mirror the archive, that would be wonderful. I was
> considering mirroring everything to a remote in sr.ht in the future,
> but haven't gotten around to it.
>
> As a footnote, there's a decent git client written in Go that works
> alright on plan9 [4], but it's slow and memory intensive at the
> moment.
>
> Cheers,
> Sean
>
> [1] https://twitter.com/traverser/status/1244398479591563265
> [2] https://github.com/Plan9-Archive
> [3] https://github.com/oridb/git9
> [4] https://github.com/driusan/dgit
>
> ------------------------------------------
> 9fans: 9fans
> Permalink:
> https://9fans.topicbox.com/groups/9fans/T303744e1ec6d2108-Ma1c49d00e7042a1a8f6713d2
> Delivery options: https://9fans.topicbox.com/groups/9fans/subscription
>

[-- Attachment #2: Type: text/html, Size: 4523 bytes --]

Re: [9fans] Software preservation in the post-hg era

[Charles Forsyth]

[-- Attachment #1: Type: text/plain, Size: 942 bytes --]

I moved all my hg repositories to git some time ago

On Tue, Mar 31, 2020 at 7:39 PM Dave MacFarlane <driusan@gmail.com> wrote:

> On Tue, Mar 31, 2020 at 10:46 AM <ori@eigenstate.org> wrote:
> > > If anyone has further thoughts, anything they want added, or any lists
> > > or indices of works they want archived/mirrored, I would love to see
> > > these posted.
> >
> > I think the lede got buried here, and people went of discussing git
> > clients.
> >
> > Given that the aptly-named bitbucket is deleting mercurial repositories,
> > is there anything of interest that should be retrieved before they empty
> > the bitbucket onto the curb?
> >
>
> Is Inferno already mirrored?
>
> ------------------------------------------
> 9fans: 9fans
> Permalink:
> https://9fans.topicbox.com/groups/9fans/T303744e1ec6d2108-M8ffeabfca5531d4d9caffcb1
> Delivery options: https://9fans.topicbox.com/groups/9fans/subscription
>

[-- Attachment #2: Type: text/html, Size: 1584 bytes --]

Re: [9fans] Software preservation in the post-hg era

[Dave MacFarlane]

[-- Attachment #1: Type: text/plain, Size: 806 bytes --]

On Mon, Mar 30, 2020 at 9:12 PM Sean Hinchee <henesy.dev@gmail.com> wrote:

> As a footnote, there's a decent git client written in Go that works
> alright on plan9 [4], but it's slow and memory intensive at the
> moment.
>
>
[...]

[4] https://github.com/driusan/dgit
>

This (and the fact that the speed of Go on Plan9/amd64 seems to be finally
be useable enough to do development again as of 1.14..) finally gave me the
kick I needed to fix some of the hacks that were causing performance
problems on clone. The self-clone time went from ~160s to ~13s on my
machine (compared to ~8s with "real" git) If there's other parts that you
were referring to as being slow and memory intensive let me know (or if you
still find it's memory intensive, I didn't benchmark that part..)

- Dave

[-- Attachment #2: Type: text/html, Size: 1425 bytes --]

Re: [9fans] Software preservation in the post-hg era

[Lucio De Re]

Kudos to you, Dave, in spades!

Could you please contact me off-list with a very brief summary of the
state of shiny for Plan 9 from your advantage point? It deserves a
full conference, but that isn't going to happen, is it?

Lucio.

Re: [9fans] Software preservation in the post-hg era

[Jens Staal]

[-- Attachment #1: Type: text/plain, Size: 1369 bytes --]

Den tors 7 maj 2020 16:17Dave MacFarlane <driusan@gmail.com> skrev:

>
> On Mon, Mar 30, 2020 at 9:12 PM Sean Hinchee <henesy.dev@gmail.com> wrote:
>
>> As a footnote, there's a decent git client written in Go that works
>> alright on plan9 [4], but it's slow and memory intensive at the
>> moment.
>>
>>
> [...]
>
> [4] https://github.com/driusan/dgit
>>
>
> This (and the fact that the speed of Go on Plan9/amd64 seems to be finally
> be useable enough to do development again as of 1.14..) finally gave me the
> kick I needed to fix some of the hacks that were causing performance
> problems on clone. The self-clone time went from ~160s to ~13s on my
> machine (compared to ~8s with "real" git) If there's other parts that you
> were referring to as being slow and memory intensive let me know (or if you
> still find it's memory intensive, I didn't benchmark that part..)
>
> - Dave
>


How does it compare performance wise with git9 ?

https://github.com/oridb/git9


*9fans <https://9fans.topicbox.com/latest>* / 9fans / see discussions
> <https://9fans.topicbox.com/groups/9fans> + participants
> <https://9fans.topicbox.com/groups/9fans/members> + delivery options
> <https://9fans.topicbox.com/groups/9fans/subscription> Permalink
> <https://9fans.topicbox.com/groups/9fans/T303744e1ec6d2108-Mbc073232aabcf555586a9843>
>

[-- Attachment #2: Type: text/html, Size: 3069 bytes --]

Re: [9fans] Software preservation in the post-hg era

[hiro]

> state of shiny for Plan 9 from your advantage point? It deserves a

since you posted this question to the list:
what is shiny?

Re: [9fans] Software preservation in the post-hg era

[Jim Manley]

[-- Attachment #1: Type: text/plain, Size: 3473 bytes --]

I don't know if he's on here, but Al Kossow (aek at bitsavers dot org), the
Software Curator at the Computer History Museum in Mountain View, CA
(SillyCon Valley, just down the road from the Googleplex) might be
interested in this, if he isn't already aware of what's going on.  He's an
expert on The Good Fight Against Bit-Rot and has tools for preserving
pretty much everything in Original Flavor media formats, as well as on-line.

In addition to the software library collection, the museum also has a
separate library dedicated to preservation of the all-important
documentation which, if lost, often results in the loss of much of the
ability to comprehend the functionality of, let alone maintain, executables
and associated data.


On Mon, Mar 30, 2020 at 7:12 PM Sean Hinchee <henesy.dev@gmail.com> wrote:

> In the wake of Bitbucket removing hg (Mercurial) support [1], I feel
> it's topical to bring up software preservation for the plan9
> community.
>
> A lot of community contributed software has been put up on Bitbucket
> or other hg hosts over time (RIP Google Code), but no consolidated
> effort, to my knowledge, seems to have been made to index, let alone
> mirror, this software.
>
> For now, as a stop-gap, I've made a GitHub organization in which I've
> consolidated most of what I had indexed from Bitbucket and a few other
> places.
>
> Thanks to people like Ori Bernstein, we have a native git client for
> plan9 [3]; without a native client, this kind of transition wouldn't
> be nearly as simple, thank you.
>
> I'm more than happy to add anyone interested in the curation of this
> archive to the GitHub organization. It would be nice to have spare
> hands around to add README's, mkfiles, and attributions where they
> have been missed or never existed.
>
> In the long term, it would be nice to have a federated or otherwise
> decentralized solution to pooling community contributed software,
> especially keeping in mind ease of mirroring and picking up old
> projects as contributors come and go.
>
> The contrib/ directory on sources and 9front are fine and good, but
> they are centralized. I don't have a proposed solution to this
> problem, but it would be nice to have ideas or insight posted ☺.
>
> I recognize that GitHub is also centralized and doesn't solve the
> centralization problem, but at least git is really straightforward to
> mirror with multiple remotes, etc. and having an index/archive is
> valuable at least to me.
>
> If anyone has further thoughts, anything they want added, or any lists
> or indices of works they want archived/mirrored, I would love to see
> these posted.
>
> If anyone wants to mirror the archive, that would be wonderful. I was
> considering mirroring everything to a remote in sr.ht in the future,
> but haven't gotten around to it.
>
> As a footnote, there's a decent git client written in Go that works
> alright on plan9 [4], but it's slow and memory intensive at the
> moment.
>
> Cheers,
> Sean
>
> [1] https://twitter.com/traverser/status/1244398479591563265
> [2] https://github.com/Plan9-Archive
> [3] https://github.com/oridb/git9
> [4] https://github.com/driusan/dgit
>
> ------------------------------------------
> 9fans: 9fans
> Permalink:
> https://9fans.topicbox.com/groups/9fans/T303744e1ec6d2108-Ma1c49d00e7042a1a8f6713d2
> Delivery options: https://9fans.topicbox.com/groups/9fans/subscription
>

[-- Attachment #2: Type: text/html, Size: 4609 bytes --]

Re: [9fans] Software preservation in the post-hg era

[ori]

> Den tors 7 maj 2020 16:17Dave MacFarlane <driusan@gmail.com> skrev:
> 
>>
>> On Mon, Mar 30, 2020 at 9:12 PM Sean Hinchee <henesy.dev@gmail.com> wrote:
>>
>>> As a footnote, there's a decent git client written in Go that works
>>> alright on plan9 [4], but it's slow and memory intensive at the
>>> moment.
>>>
>>>
>> [...]
>>
>> [4] https://github.com/driusan/dgit
>>
>> This (and the fact that the speed of Go on Plan9/amd64 seems to be finally
>> be useable enough to do development again as of 1.14..) finally gave me the
>> kick I needed to fix some of the hacks that were causing performance
>> problems on clone. The self-clone time went from ~160s to ~13s on my
>> machine (compared to ~8s with "real" git) If there's other parts that you
>> were referring to as being slow and memory intensive let me know (or if you
>> still find it's memory intensive, I didn't benchmark that part..)
>>
>> - Dave
>>
> 
> 
> How does it compare performance wise with git9 ?
> 
> https://github.com/oridb/git9

I'll be honest, I'm using git9 because of the improved
interactions, rather than performance -- it's fast enough
for most of my usage. Still, this got curious enough to
test a bit. Here are the results:

It's close for cloning dgit -- I'm seeing about 3 seconds
for dgit with git/clone, 4.5 using dgit to clone itself.

	% time git/clone https://github.com/driusan/dgit
	0.81u 1.08s 2.70r 

(Looking closer, about 1.5 seconds of that comes
from the dircp to pull data out of /mnt/git/ and
into the working directory.)

When testing dgit, I redirected output to /dev/null,
since it printed enough that it affected the time.
It's *really* chatty -- for the larger test, it
produced more than 50 megabytes of status text.

	cpu% time rc -c 'dgit clone https://github.com/driusan/dgit >[2]/dev/null'
	0.47u 0.55s 4.32r

It seems like there's something accidentally
quadratic, though. Cloning a larger repository
-- in this case, perl5 -- takes 160s on git9,
and 1200 seconds on dgit. For comparison, git
on OpneBSD with different (but comparable)
hardware takes about 90 seconds.

	cpu% time git/clone https://github.com/Perl/perl5.git
	94.40u 14.16s 159.30r

	cpu% time ./dgit clone https://github.com/Perl/perl5.git >[2]/tmp/dgit.log
	121.93u 22.16s 1211.30r

I only skimmed the dgit code quickly, and didn't see
an obvious answer: do you cache objects that you've
decompressed, or do you iterate over full delta chains
every time?

One other bug report -- it seems that dgit hard-codes the
default branch as origin/master, but perl uses 'origin/blead',
so the checkout fails with 'Could not find origin/master'

There are still places where git9 is very slow. Sending lots
of commits at once in big repositories stands out.

Two reasons for this: we don't deltify, and we walk too much
data deciding what should go into the pack. There's also a
bug that causes certain kinds of merge to push the whole
history spuriously, which is.. only wasteful rather than
incorrect -- but wasteful isn't good.

Pushing all perl commits to an empty repository, for example:

	# this is the size of the packfile git gives us
	cpu% du -sh .git
	297.043M	.git

	# pushing to git is slow
	cpu% git/push -u git+ssh://192.168.1.10/tmp/p5.git
	1783.08u 444.15s 2835.86r

	# and our undeltified packfiles are 10x the size
	# that they should be
	$ du -sh p5.git; 
	4.2Gp5.git

I can't compare with dgit, since dgit doesn't support ssh
pushes, and I'm not going to set up http pushes right now.

Re: [9fans] Software preservation in the post-hg era

[Dave MacFarlane]

[-- Attachment #1: Type: text/plain, Size: 432 bytes --]

>
>
>
> How does it compare performance wise with git9 ?
>
> https://github.com/oridb/git9
>

Based on the unscientific testing I just did of cloning the same repo
(dgit's) a few times,  they're in the same ballpark (~13s on 9front amd64.)
git9 had a couple faster runs and dgit a couple slower ones,  but nothing
outside of what could be explained by my wifi connection since the network
is the bottleneck in cloning.

[-- Attachment #2: Type: text/html, Size: 1289 bytes --]