AArch64 merge back

AArch64 merge back

[Weiming Zhao]

[-- Attachment #1: Type: text/plain, Size: 299 bytes --]

Hi,

 

I'm wondering if there is any plan to merge back the changes for AArch64
from https://github.com/crxz0193/musl-aarch64 ?

So we can stay on the main repository.

 

Thanks,

Weiming

 

Qualcomm Innovation Center, Inc. is a member of Code Aurora Forum, hosted by
The Linux Foundation

 

 


[-- Attachment #2: Type: text/html, Size: 2449 bytes --]

Re: AArch64 merge back

[Isaac Dunham]

On Mon, Jun 30, 2014 at 03:56:45PM -0700, Weiming Zhao wrote:
> Hi,
> I'm wondering if there is any plan to merge back the changes for AArch64
> from https://github.com/crxz0193/musl-aarch64 ?
> 
> So we can stay on the main repository.

What's the status of that? I see commits related to a few syscalls from
the end of March/April 1, and nothing newer.
The last news I heard, some of the work that would be needed was done,
but I did not get the impression that it was possible to build working
binaries, even static ones.
(If this is incorrect, I'd like to hear the current status.)

Some ports have been merged before all functionality worked 
(including the dynamic linker), but I don't recall seeing any ports
get merged before it was possible to produce a working executable.

Or are you referring to just the non-arch-specific changes?

Thanks,
Isaac Dunham

RE: AArch64 merge back

[Weiming Zhao]

Hi Isaac,

Do you mean those arch independent functions (e.g. abs, sprint, memcpy) have
already been merged? 

Thanks,
Weiming

-----Original Message-----
From: Isaac Dunham [mailto:ibid.ag@gmail.com] 
Sent: Monday, June 30, 2014 5:07 PM
To: musl@lists.openwall.com
Subject: Re: [musl] AArch64 merge back

On Mon, Jun 30, 2014 at 03:56:45PM -0700, Weiming Zhao wrote:
> Hi,
> I'm wondering if there is any plan to merge back the changes for 
> AArch64 from https://github.com/crxz0193/musl-aarch64 ?
> 
> So we can stay on the main repository.

What's the status of that? I see commits related to a few syscalls from the
end of March/April 1, and nothing newer.
The last news I heard, some of the work that would be needed was done, but I
did not get the impression that it was possible to build working binaries,
even static ones.
(If this is incorrect, I'd like to hear the current status.)

Some ports have been merged before all functionality worked (including the
dynamic linker), but I don't recall seeing any ports get merged before it
was possible to produce a working executable.

Or are you referring to just the non-arch-specific changes?

Thanks,
Isaac Dunham

Re: AArch64 merge back

[Isaac Dunham]

On Mon, Jun 30, 2014 at 06:14:36PM -0700, Weiming Zhao wrote:
> Hi Isaac,
> 
> Do you mean those arch independent functions (e.g. abs, sprint, memcpy) have
> already been merged? 
 
No, I was asking if you meant the whole port-in-progress or if you were only
asking about said functions; at that point I had not checked.

At present, I can tell you what the git log says:
-memcpy.c was last touched in August last year implementing a portable
optimized memcpy.
-abs.c has not changed in the 3 years of git history
-sprintf.c (I assume that's what you mean by 'sprint'?) has not been 
touched since 2012.

...all of which are before AArch64 porting.

Is the relevant code in 'rebase-1.0'?

Thanks,
Isaac Dunham
 
> -----Original Message-----
> From: Isaac Dunham [mailto:ibid.ag@gmail.com] 
> Sent: Monday, June 30, 2014 5:07 PM
> To: musl@lists.openwall.com
> Subject: Re: [musl] AArch64 merge back
> 
> On Mon, Jun 30, 2014 at 03:56:45PM -0700, Weiming Zhao wrote:
> > Hi,
> > I'm wondering if there is any plan to merge back the changes for 
> > AArch64 from https://github.com/crxz0193/musl-aarch64 ?
> > 
> > So we can stay on the main repository.
> 
> What's the status of that? I see commits related to a few syscalls from the
> end of March/April 1, and nothing newer.
> The last news I heard, some of the work that would be needed was done, but I
> did not get the impression that it was possible to build working binaries,
> even static ones.
> (If this is incorrect, I'd like to hear the current status.)
> 
> Some ports have been merged before all functionality worked (including the
> dynamic linker), but I don't recall seeing any ports get merged before it
> was possible to produce a working executable.
> 
> Or are you referring to just the non-arch-specific changes?
> 
> Thanks,
> Isaac Dunham
> 
> 

RE: AArch64 merge back

[Weiming Zhao]

Hi Isaac,

So if I just want to use some arch-independent functions, then I just need
to build the main musl repo with aarch64 compiler. Is my understanding
correct?

Thanks,
Weiming

-----Original Message-----
From: Isaac Dunham [mailto:ibid.ag@gmail.com] 
Sent: Monday, June 30, 2014 6:45 PM
To: musl@lists.openwall.com
Subject: Re: [musl] AArch64 merge back

On Mon, Jun 30, 2014 at 06:14:36PM -0700, Weiming Zhao wrote:
> Hi Isaac,
> 
> Do you mean those arch independent functions (e.g. abs, sprint, 
> memcpy) have already been merged?
 
No, I was asking if you meant the whole port-in-progress or if you were only
asking about said functions; at that point I had not checked.

At present, I can tell you what the git log says:
-memcpy.c was last touched in August last year implementing a portable
optimized memcpy.
-abs.c has not changed in the 3 years of git history -sprintf.c (I assume
that's what you mean by 'sprint'?) has not been touched since 2012.

...all of which are before AArch64 porting.

Is the relevant code in 'rebase-1.0'?

Thanks,
Isaac Dunham
 
> -----Original Message-----
> From: Isaac Dunham [mailto:ibid.ag@gmail.com]
> Sent: Monday, June 30, 2014 5:07 PM
> To: musl@lists.openwall.com
> Subject: Re: [musl] AArch64 merge back
> 
> On Mon, Jun 30, 2014 at 03:56:45PM -0700, Weiming Zhao wrote:
> > Hi,
> > I'm wondering if there is any plan to merge back the changes for
> > AArch64 from https://github.com/crxz0193/musl-aarch64 ?
> > 
> > So we can stay on the main repository.
> 
> What's the status of that? I see commits related to a few syscalls 
> from the end of March/April 1, and nothing newer.
> The last news I heard, some of the work that would be needed was done, 
> but I did not get the impression that it was possible to build working 
> binaries, even static ones.
> (If this is incorrect, I'd like to hear the current status.)
> 
> Some ports have been merged before all functionality worked (including 
> the dynamic linker), but I don't recall seeing any ports get merged 
> before it was possible to produce a working executable.
> 
> Or are you referring to just the non-arch-specific changes?
> 
> Thanks,
> Isaac Dunham
> 
> 

Re: AArch64 merge back

[Rich Felker]

On Tue, Jul 01, 2014 at 10:41:10AM -0700, Weiming Zhao wrote:
> Hi Isaac,
> 
> So if I just want to use some arch-independent functions, then I just need
> to build the main musl repo with aarch64 compiler. Is my understanding
> correct?

No. I didn't really understand what you were asking before, but it
might be more clear if you read the porting documents on the wiki.
Rather than trying to answer point by point I'll try to explain a few
things:

It's not possible to use musl on a new arch simply by compiling the
arch-independent C. This is partly because there are a number of
components of libc that fundamentally cannot be expressed in C, and
partly because the kernel (gratuitously) has a different set of
constants, struct definitions, etc. for each architecture.

Generally ports are not merged (in the "git merge" sense) because most
of the early work on them is a mess of incompleteness,
trial-and-error, etc. Also merging would be a lot of work since we
normally rebase all merges, whereas master has often diverged quite a
bit before a port is ready to merge. Instead, once the port is
working, we usually just add it as a single commit, followed by any
fixes for issues that weren't found before commit.

I'm not clear on the status of the aarch64 port right now. It was
stalled for a while because of changes needed in many of the
arch-independent files to accomodate the way the kernel does things on
newer archs (omitting lots of simple syscalls that can be emulated
using more complex ones). That work is done in mainline musl now
though, so it's no longer blocking ports. Further progress is up to
the people working on those ports (or anyone else who wants to build
on their work). I'm really hoping it will be finished during this
release cycle so we can have aarch64 support in 1.1.4.

If you or anyone else wants to play around with trying to get it to
work based on the in-progress ports, the best approach would be to
simply copy over the new files added in the aarch64 musl git tree into
a more recent musl (1.1.2 or later).

Rich

RE: AArch64 merge back

[Weiming Zhao]

Hi Rich,

Thanks a lot for your response.

We're trying static linking of AArch64 with some simple tests.
Some simple tests are working, but we do see some syscall issue as you
mentioned.
We suspect it's related with arch/aarch64/bits/syscall.h.
Is the file written from scratch or based on some existing open source
projects?

Thanks,
Weiming

-----Original Message-----
From: Rich Felker [mailto:dalias@aerifal.cx] On Behalf Of Rich Felker
Sent: Tuesday, July 01, 2014 11:25 AM
To: musl@lists.openwall.com
Subject: Re: [musl] AArch64 merge back

On Tue, Jul 01, 2014 at 10:41:10AM -0700, Weiming Zhao wrote:
> Hi Isaac,
> 
> So if I just want to use some arch-independent functions, then I just 
> need to build the main musl repo with aarch64 compiler. Is my 
> understanding correct?

No. I didn't really understand what you were asking before, but it might be
more clear if you read the porting documents on the wiki.
Rather than trying to answer point by point I'll try to explain a few
things:

It's not possible to use musl on a new arch simply by compiling the
arch-independent C. This is partly because there are a number of components
of libc that fundamentally cannot be expressed in C, and partly because the
kernel (gratuitously) has a different set of constants, struct definitions,
etc. for each architecture.

Generally ports are not merged (in the "git merge" sense) because most of
the early work on them is a mess of incompleteness, trial-and-error, etc.
Also merging would be a lot of work since we normally rebase all merges,
whereas master has often diverged quite a bit before a port is ready to
merge. Instead, once the port is working, we usually just add it as a single
commit, followed by any fixes for issues that weren't found before commit.

I'm not clear on the status of the aarch64 port right now. It was stalled
for a while because of changes needed in many of the arch-independent files
to accomodate the way the kernel does things on newer archs (omitting lots
of simple syscalls that can be emulated using more complex ones). That work
is done in mainline musl now though, so it's no longer blocking ports.
Further progress is up to the people working on those ports (or anyone else
who wants to build on their work). I'm really hoping it will be finished
during this release cycle so we can have aarch64 support in 1.1.4.

If you or anyone else wants to play around with trying to get it to work
based on the in-progress ports, the best approach would be to simply copy
over the new files added in the aarch64 musl git tree into a more recent
musl (1.1.2 or later).

Rich

Re: AArch64 merge back

[Rich Felker]

On Tue, Jul 15, 2014 at 11:58:59AM -0700, Weiming Zhao wrote:
> Hi Rich,
> 
> Thanks a lot for your response.
> 
> We're trying static linking of AArch64 with some simple tests.
> Some simple tests are working, but we do see some syscall issue as you
> mentioned.
> We suspect it's related with arch/aarch64/bits/syscall.h.
> Is the file written from scratch or based on some existing open source
> projects?

It should match the definitions from the kernel, but definitions for
any syscalls that do not actually exist on aarch64 need to be removed.
If the kernel is using the asm-generic values for aarch64, you can get
a matching bits/syscall.h from the or1k (openrisc) port that's
in-progress, at:

https://github.com/skristiansson/musl-or1k

Rich

RE: AArch64 merge back

[Weiming Zhao]

Hi Rich,

Thanks.
Yes, those __NR_xxx values should match with kernel defs. 
But do you know where the current syscall.h originally come from? 

In the beginning of aarch64/bits/syscall.h, we can see comments like:
/*
 * This file contains the system call numbers, based on the
 * layout of the x86-64 architecture, which embeds the
 * pointer to the syscall in the table.
 *
 * As a basic principle, no duplication of functionality
 * should be added, e.g. we don't use lseek when llseek
 * is present. New architectures should use this file
 * and implement the less feature-full calls in user space.
 */

It looks like it's coming from some other projects. 
Could you help to confirm its source? The reason is we plan to use MUSL for
some internal projects but we need to make sure there is no legal/license
issues.  
 
For musl-or1k, there is no aarch64 version of syscall. Are you suggesting
that we can refer to its ARM version of syscall.h?

Thanks a lot,
Weiming

-----Original Message-----
From: Rich Felker [mailto:dalias@aerifal.cx] On Behalf Of Rich Felker
Sent: Tuesday, July 15, 2014 12:07 PM
To: Weiming Zhao
Cc: musl@lists.openwall.com
Subject: Re: [musl] AArch64 merge back

On Tue, Jul 15, 2014 at 11:58:59AM -0700, Weiming Zhao wrote:
> Hi Rich,
> 
> Thanks a lot for your response.
> 
> We're trying static linking of AArch64 with some simple tests.
> Some simple tests are working, but we do see some syscall issue as you 
> mentioned.
> We suspect it's related with arch/aarch64/bits/syscall.h.
> Is the file written from scratch or based on some existing open source 
> projects?

It should match the definitions from the kernel, but definitions for any
syscalls that do not actually exist on aarch64 need to be removed.
If the kernel is using the asm-generic values for aarch64, you can get a
matching bits/syscall.h from the or1k (openrisc) port that's in-progress,
at:

https://github.com/skristiansson/musl-or1k

Rich

Re: AArch64 merge back

[Szabolcs Nagy]

* Weiming Zhao <weimingz@codeaurora.org> [2014-07-15 13:42:11 -0700]:
> In the beginning of aarch64/bits/syscall.h, we can see comments like:
> 
> It looks like it's coming from some other projects. 
> Could you help to confirm its source? The reason is we plan to use MUSL for
> some internal projects but we need to make sure there is no legal/license
> issues.  

this is just include/uapi/asm-generic/unistd.h from the linux kernel

and it should be cleaned up because it's full of noise
(musl only needs the __NR_ and SYS_ macros)

> For musl-or1k, there is no aarch64 version of syscall. Are you suggesting
> that we can refer to its ARM version of syscall.h?

all the new archs (like aarch64 and or1k) use the same asm-generic/unistd.h
for the syscall numbers so they should have identical syscall.h in musl

Re: AArch64 merge back

[Szabolcs Nagy]

* Weiming Zhao <weimingz@codeaurora.org> [2014-06-30 15:56:45 -0700]:
> I'm wondering if there is any plan to merge back the changes for AArch64
> from https://github.com/crxz0193/musl-aarch64 ?
> 
> So we can stay on the main repository.

do you plan to use musl on aarch64?

the porting was stopped because aarch64 uses the
new syscall set of linux which needed significant
changes in musl (and without working essential
syscalls it was not possible to test the port)

now the new syscalls are supported so porting can
continue, so somebody with an emulator or hardware
can pick it up

that repo probably wont be merged directly but a
squashed version of the changes will be committed
once most things work reliably

this is on the roadmap, but somebody has to do the
work (should not be hard with an emulator and docs)
http://wiki.musl-libc.org/wiki/Roadmap

there were plans to refactor arch/*/bits to reduce
the redundancy there, but it was postponed because
of build system issues.  if or1k and aarch64 gets
merged then maybe this item should be picked up
again..