[ISSUE] [RFC] sorted common/shlibs to avoid merge/rebase conflicts

[ISSUE] [RFC] sorted common/shlibs to avoid merge/rebase conflicts

[gt7-void]

[-- Attachment #1: Type: text/plain, Size: 1478 bytes --]

New issue by gt7-void on void-packages repository

https://github.com/void-linux/void-packages/issues/29965

Description:
The way new entries are added to `common/shlibs` (generally at the end) often results in merge conflicts (e.g. all PR for new packages that are open at a given time are incompatible since they all append at the same position).

It occurs to me that a way to minimize those conflicts would be to have `common/shlibs` be sorted by soname. In this way it becomes very unlikely that two commits conflict (they will only if they are lexicographically next to each other).

In addition, this would help to be explicit about repeated sonames and remove unnecessary duplicates.

This may be a little bit painful at the time the list is sorted, but then it's just a matter to keep it sorted (maybe some hook can check and complain about unsorted entries?).

The same applies to `common/options.description`, although that file is not updated so often, OTOH it's almost sorted (only 6 entries out of order).

A possible sort order would be to use:
```
$ sort common/shlibs -sk1,1
```
and
```
$ sort common/options.description -sk1,1 -t=
```

To see the kinds of soname repetitions:
```
$ sort common/shlibs -sk1,1 | awk '$1=="#" { next } $1==k&&first{print first; first="";d++} $1==k{print;x++} $1!=k{k=$1;first=$0;t++} END{print t, d, x}'
...
3866 61 92
```
I.e. out of 3866 different sonames, there are 61 repeated (61+92 times in total).

Re: [RFC] sorted common/shlibs to avoid merge/rebase conflicts

[dkwo]

[-- Attachment #1: Type: text/plain, Size: 158 bytes --]

New comment by dkwo on void-packages repository

https://github.com/void-linux/void-packages/issues/29965#issuecomment-813071261

Comment:
Makes sense to me.

Re: [RFC] sorted common/shlibs to avoid merge/rebase conflicts

[ericonr]

[-- Attachment #1: Type: text/plain, Size: 306 bytes --]

New comment by ericonr on void-packages repository

https://github.com/void-linux/void-packages/issues/29965#issuecomment-813072260

Comment:
> In addition, this would help to be explicit about repeated sonames and remove unnecessary duplicates.

This is a good argument for it that I hadn't thought of.

Re: [RFC] sorted common/shlibs to avoid merge/rebase conflicts

[Duncaen]

[-- Attachment #1: Type: text/plain, Size: 389 bytes --]

New comment by Duncaen on void-packages repository

https://github.com/void-linux/void-packages/issues/29965#issuecomment-813086622

Comment:
> > In addition, this would help to be explicit about repeated sonames and remove unnecessary duplicates.
> 
> This is a good argument for it that I hadn't thought of.

This is a better lint step than reordering the list if someone forgot it.

Re: [RFC] sorted common/shlibs to avoid merge/rebase conflicts

[Duncaen]

[-- Attachment #1: Type: text/plain, Size: 450 bytes --]

New comment by Duncaen on void-packages repository

https://github.com/void-linux/void-packages/issues/29965#issuecomment-813086622

Comment:
> > In addition, this would help to be explicit about repeated sonames and remove unnecessary duplicates.
> 
> This is a good argument for it that I hadn't thought of.

This is a better lint step than complaining about an unsorted list and a random commit having to reorder it because someone forgot it.

Re: [RFC] sorted common/shlibs to avoid merge/rebase conflicts

[Duncaen]

[-- Attachment #1: Type: text/plain, Size: 595 bytes --]

New comment by Duncaen on void-packages repository

https://github.com/void-linux/void-packages/issues/29965#issuecomment-813086622

Comment:
> > In addition, this would help to be explicit about repeated sonames and remove unnecessary duplicates.
> 
> This is a good argument for it that I hadn't thought of.

This is a better lint step than failing on an unsorted list. If someone forgets to sort it then we have 10 PRs all trying to reorder it, affecting more lines than just adding a new soname in a random position instead of the end to avoid merge conflicts for newly added libraries.

Re: [RFC] sorted common/shlibs to avoid merge/rebase conflicts

[Duncaen]

[-- Attachment #1: Type: text/plain, Size: 620 bytes --]

New comment by Duncaen on void-packages repository

https://github.com/void-linux/void-packages/issues/29965#issuecomment-813086622

Comment:
> > In addition, this would help to be explicit about repeated sonames and remove unnecessary duplicates.
> 
> This is a good argument for it that I hadn't thought of.

This is a better lint step (looking for duplicates) than failing on an unsorted list. If someone forgets to sort it then we have 10 PRs all trying to reorder it, affecting more lines than just adding a new soname in a random position instead of the end to avoid merge conflicts for newly added libraries.

Re: [RFC] sorted common/shlibs to avoid merge/rebase conflicts

[gt7-void]

[-- Attachment #1: Type: text/plain, Size: 1143 bytes --]

New comment by gt7-void on void-packages repository

https://github.com/void-linux/void-packages/issues/29965#issuecomment-813114105

Comment:
> This is a better lint step (looking for duplicates) than failing on an unsorted list.

Is it easy to decide when a duplicate soname is ok or not?

> If someone forgets to sort it then we have 10 PRs all trying to reorder it,

Could we have a lint step checking that the file is sorted?

> affecting more lines than just adding a new soname in a random position instead of the end to avoid merge conflicts for newly added libraries.

Is there a policy about that? I understood sonames had to be placed at (near) the end of the file.

I don't care so much about having the file sorted. The fundamental itch I have is constantly having to amend patches.

The real pain for me was the experience of working with the branch in #29997, which has 7 commits for different packages which add sonames, and any rebase/editing of previous patches would cause a lot of pain. Of course, all this pain would mostly go away if I were to place each soname in a different random place in the file...

Re: [RFC] sorted common/shlibs to avoid merge/rebase conflicts

[gt7-void]

[-- Attachment #1: Type: text/plain, Size: 1340 bytes --]

New comment by gt7-void on void-packages repository

https://github.com/void-linux/void-packages/issues/29965#issuecomment-813114661

Comment:
Regarding duplicates, there are many different situations:
- exact duplicate lines are obvious to remove, but harmless
- Something like this:
```
ld.so.1 glibc-2.32_1 mips
ld.so.1 glibc-2.32_1 ppc
```
May be ok, although the documentation for the third field is unclear.
- what about this:
```
libClp.so.1 CoinMP-1.8.3_1
libClp.so.1 libClp-1.16.11_1
```
using `xlocate` shows that both packages ship `/usr/lib/libClp.so.1`, but they don't conflict to each other.
- what about this:
```
libGLESv2.so.2 libGLES-1.0_1
libGLESv2.so rpi-userland-0.0.0.0.20150907_1
libGLESv2.so atom-1.41.0_1
libGLESv2.so opera-55.0.2994.37_2
libGLESv2.so discord-0.0.7_1
libGLESv2.so keybase-desktop-3.1.2_1
libGLESv2.so wire-desktop-3.6.2885_1
libGLESv2.so Signal-Desktop-1.23.2_1
libGLESv2.so slack-desktop-3.4.2_1
```
in this case, only `libglvnd` actually ships `/usr/lib/libGLESv2.so`, the ones listed there only have this file in some non-system location, and it doesn't really look like a soname. Is this included here just to keep the linter quiet?
- this one looks ok (packages conflict)
```
libboinc_api.so.7 boinc-7.16.16_2
libboinc_api.so.7 boinc-nox-7.16.16_2
```
- etc.

Re: [RFC] sorted common/shlibs to avoid merge/rebase conflicts

[Duncaen]

[-- Attachment #1: Type: text/plain, Size: 1795 bytes --]

New comment by Duncaen on void-packages repository

https://github.com/void-linux/void-packages/issues/29965#issuecomment-813124670

Comment:
> > This is a better lint step (looking for duplicates) than failing on an unsorted list.
> 
> Is it easy to decide when a duplicate soname is ok or not?

If the package name is the same and it is the same as one of the edited templates could be a good first heuristic.
I don't think PRs should lint for things that are not related to the current package.

> > If someone forgets to sort it then we have 10 PRs all trying to reorder it,
> 
> Could we have a lint step checking that the file is sorted?

The problem is the lint step, forcing more things into the lint, especially things that don't belong to the PR in question just makes the changes larger and does not really help anyone.

> > affecting more lines than just adding a new soname in a random position instead of the end to avoid merge conflicts for newly added libraries.
> 
> Is there a policy about that? I understood sonames had to be placed at (near) the end of the file.

No, could be a "tip" instead of a policy, as it "might" reduce the work of the contributor, but it is not required. 

> The real pain for me was the experience of working with the branch in #29997, which has 7 commits for different packages which add sonames, and any rebase/editing of previous patches would cause a lot of pain. Of course, all this pain would mostly go away if I were to place each soname in a different random place in the file...

Right, its a good tip to mention, and people are doing it. Requiring a sorted file through a lint step is one work around with the mentioned downsides, randomly placing the additions is a simple workaround without any real downsides IMHO.

Re: [RFC] sorted common/shlibs to avoid merge/rebase conflicts

[Duncaen]

[-- Attachment #1: Type: text/plain, Size: 1815 bytes --]

New comment by Duncaen on void-packages repository

https://github.com/void-linux/void-packages/issues/29965#issuecomment-813124670

Comment:
> > This is a better lint step (looking for duplicates) than failing on an unsorted list.
> 
> Is it easy to decide when a duplicate soname is ok or not?

If the package name is the same and it is the same as one of the edited templates or its sub packages could be a good first heuristic.
I don't think PRs should lint for things that are not related to the current package.

> > If someone forgets to sort it then we have 10 PRs all trying to reorder it,
> 
> Could we have a lint step checking that the file is sorted?

The problem is the lint step, forcing more things into the lint, especially things that don't belong to the PR in question just makes the changes larger and does not really help anyone.

> > affecting more lines than just adding a new soname in a random position instead of the end to avoid merge conflicts for newly added libraries.
> 
> Is there a policy about that? I understood sonames had to be placed at (near) the end of the file.

No, could be a "tip" instead of a policy, as it "might" reduce the work of the contributor, but it is not required. 

> The real pain for me was the experience of working with the branch in #29997, which has 7 commits for different packages which add sonames, and any rebase/editing of previous patches would cause a lot of pain. Of course, all this pain would mostly go away if I were to place each soname in a different random place in the file...

Right, its a good tip to mention, and people are doing it. Requiring a sorted file through a lint step is one work around with the mentioned downsides, randomly placing the additions is a simple workaround without any real downsides IMHO.

Re: [RFC] sorted common/shlibs to avoid merge/rebase conflicts

[Duncaen]

[-- Attachment #1: Type: text/plain, Size: 1895 bytes --]

New comment by Duncaen on void-packages repository

https://github.com/void-linux/void-packages/issues/29965#issuecomment-813124670

Comment:
> > This is a better lint step (looking for duplicates) than failing on an unsorted list.
> 
> Is it easy to decide when a duplicate soname is ok or not?

If the package name is the same and it is the same as one of the edited templates or its sub packages could be a good first heuristic.
I don't think PRs should lint for things that are not related to the current package.

> > If someone forgets to sort it then we have 10 PRs all trying to reorder it,
> 
> Could we have a lint step checking that the file is sorted?

The problem is the lint step, forcing more things into the lint, especially things that don't belong to the PR in question just makes the changes larger and does not really help anyone and would probably lead to people duplicating those "lint" fixes into their PRs.

> > affecting more lines than just adding a new soname in a random position instead of the end to avoid merge conflicts for newly added libraries.
> 
> Is there a policy about that? I understood sonames had to be placed at (near) the end of the file.

No, could be a "tip" instead of a policy, as it "might" reduce the work of the contributor, but it is not required. 

> The real pain for me was the experience of working with the branch in #29997, which has 7 commits for different packages which add sonames, and any rebase/editing of previous patches would cause a lot of pain. Of course, all this pain would mostly go away if I were to place each soname in a different random place in the file...

Right, its a good tip to mention, and people are doing it. Requiring a sorted file through a lint step is one work around with the mentioned downsides, randomly placing the additions is a simple workaround without any real downsides IMHO.

Re: [RFC] sorted common/shlibs to avoid merge/rebase conflicts

[github-actions]

[-- Attachment #1: Type: text/plain, Size: 293 bytes --]

New comment by github-actions[bot] on void-packages repository

https://github.com/void-linux/void-packages/issues/29965#issuecomment-1128327603

Comment:
Issues become stale 90 days after last activity and are closed 14 days after that.  If this issue is still relevant bump it or assign it.

Re: [ISSUE] [CLOSED] [RFC] sorted common/shlibs to avoid merge/rebase conflicts

[github-actions]

[-- Attachment #1: Type: text/plain, Size: 1481 bytes --]

Closed issue by gt7-void on void-packages repository

https://github.com/void-linux/void-packages/issues/29965

Description:
The way new entries are added to `common/shlibs` (generally at the end) often results in merge conflicts (e.g. all PR for new packages that are open at a given time are incompatible since they all append at the same position).

It occurs to me that a way to minimize those conflicts would be to have `common/shlibs` be sorted by soname. In this way it becomes very unlikely that two commits conflict (they will only if they are lexicographically next to each other).

In addition, this would help to be explicit about repeated sonames and remove unnecessary duplicates.

This may be a little bit painful at the time the list is sorted, but then it's just a matter to keep it sorted (maybe some hook can check and complain about unsorted entries?).

The same applies to `common/options.description`, although that file is not updated so often, OTOH it's almost sorted (only 6 entries out of order).

A possible sort order would be to use:
```
$ sort common/shlibs -sk1,1
```
and
```
$ sort common/options.description -sk1,1 -t=
```

To see the kinds of soname repetitions:
```
$ sort common/shlibs -sk1,1 | awk '$1=="#" { next } $1==k&&first{print first; first="";d++} $1==k{print;x++} $1!=k{k=$1;first=$0;t++} END{print t, d, x}'
...
3866 61 92
```
I.e. out of 3866 different sonames, there are 61 repeated (61+92 times in total).