Re: [9fans] mmap and shared libraries

[Eris Discordia <eris.discordia@gmail.com>]

> Sadly, the picture changes at run time: clock on plan 9 is 128k in
> memory, xclock is 4.2M RSS and 10M VSZ.
> Sic transit gloria .so. Of course, then we hear that "well, all that
> is shared". Hmm. Prove it.

I know one thing: shared libraries are employed on every major operating
system I have ever heard of. Can all these people be wrong? I don't think
so.

> FWIW, the whole issue of goodness/badness of shared libraries has
> never been systematically measured as far as I know -- in terms of
> performance cost, overhead, throughput, the usual suspects -- or at
> least
> I've never seen it done. It would be a lot of work to do it correctly.
> Might be interesting.

Title: Dynamic Linking of Software Components
Author: Michael Franz
YoP: 1997

> ABSTRACT
>
> The concepts underlying dynamic linking reached maturity through modular
> operating systems and are being reintroduced in some of the most popular
> workstation and PC operating systems. [...]

<http://portal.acm.org/citation.cfm?id=619016.620662&coll=GUIDE&dl=GUIDE&CFID=8901601&CFTOKEN=68907171>

The abstract says it is an overall survey of dynamic linking. I don't have
access to the full text and even if I had I wouldn't understand a word of
it. You are the CS/CE person. Be the judge, too.

> So the compressed version is only 50% larger than acme. The point being?

That there's a lot of redundancy in the executable principally imposed by
the binary format.

In case of the FreeBSD vim executable:

$ readelf -e /usr/local/bin/vim

Reveals 171,365 bytes of redundant data only in .data, .rodata, and .bss
sections.

$ objdump -s /usr/local/bin/vim

Will display tons of zeroes in all other sections except .text. Even RLE
could significantly reduce the image size.

> note that filesize is < memory size. I'll let you figure out how it works.

I'm not that intelligent but didn't I say it was _initialized_ data that
had to be written to executable image?

Example from a simple program I wrote to be assembled by Flat Assembler for
Win32:

--------------------------------------------------------------------------------
section '.data' data readable writeable
.
.
.
  buffer_ptr	dd 00h           <----- pointer reserved, and initialized
.
.
.
  buffer	db 00FFFFh dup (?)     <----- buffer reserved, not initialized
.
.
.
--------------------------------------------------------------------------------

File size can be less than memory size when you have data reserved but not
initialized. That happens in many cases, e.g. when you reserve a buffer.

As for ELF:

> Defining data elements in the bss section is somewhat different from
> defining them in the data section. Instead of declaring specific data
> types, you just declare raw segments of memory that  are reserved for
> whatever purpose you need them for.
>
> [...]
>
> One benefit to declaring data in the bss section is that the data is not
> included in the executable program. When data is defined in the data
> section, it must be included in the executable program, since it must be
> initialized with a specific value. Because the data areas declared in the
> bss section are not initialized with program data, the memory areas are
> reserved at runtime, and do not have to be included in the final program.

-- Richard Blum, Professional Assembly Language, in the context of using
(g)as and ELF format

I don't want to sound authoritative but when something is right it is right
and there's nothing you or I can do about it.

> Does the redundant litter cross the network when I have it mounted via
> a share and execute the program or not? That was the original
> discussion.

I don't know. The size of Emacs executable has as much connection to that
question as I do, but it was _you_ who made the bad Emacs joke.

> Ah! Now we're into feature comparisons! I'm game. How did you get to 1
> in 5 and not 1 in 4.8, or 1 in 6?

You know very well that the ratio is just an inaccurate measure as the
context suggests. 1 to 5 means like your pinkie compared to your entire
hand. Trying to "context-switch" and move from an inaccurate measure to an
exact number is just a diversionary technique you have employed for
countering an easily demonstrable fact. Namely that Acme is a minimal text
editor claiming to be an IDE while Emacs is a behemoth with more features
than you could count through a day, and the following night--not that I
believe Emacs is any better than Acme.

--On Wednesday, November 05, 2008 2:13 PM -0800 ron minnich
<rminnich@gmail.com> wrote:

> On Wed, Nov 5, 2008 at 12:54 PM, Eris Discordia
> <eris.discordia@gmail.com> wrote:
>>> yes, I agree, I was being terribly unfair to plan 9. Acme on plan 9 is
>>> about 1/2 M. Vim on DOS is 3x larger? impressive.
>>
>> My intent was, of course, to show your comparison is baseless. It seems
>> you still haven't realized that. You think Plan 9 is great?
>
> Actually, to keep it simple, I was trying to make a simple point, with
> a bit of humor intended, and not actually even directed at you,: that
> code built with shared libraries is not
> necessarily, or always, smaller than code built otherwise, for
> programs of similar capability. I don't have an emacs
> linked statically, however, so I took the nearest materials at hand.
> It depends on a lot of circumstances.
>
> There is the example of xclock which is small with X11 .so, and quite
> large otherwise. And there, for the case of a "clock", shared
> libraries clearly reduce disk footprint:
> xclock binary is about 1/10 the size of plan 9 clock (19K vs. 158K)..
>
> Sadly, the picture changes at run time: clock on plan 9 is 128k in
> memory, xclock is 4.2M RSS and 10M VSZ.
> Sic transit gloria .so. Of course, then we hear that "well, all that
> is shared". Hmm. Prove it.
>
> FWIW, the whole issue of goodness/badness of shared libraries has
> never been systematically measured as far as I know -- in terms of
> performance cost, overhead, throughput, the usual suspects -- or at
> least
> I've never seen it done. It would be a lot of work to do it correctly.
> Might be interesting.
>
>> The GUI version of Vim on Windows (gvim.exe) compresses to 734,713 bytes
>
> So the compressed version is only 50% larger than acme. The point being?
>
>> because the bigger part of the uncompressed 1,585,152 bytes is redundant
>> filler required by the binary format specification.
>
> Does the redundant litter cross the network when I have it mounted via
> a share and execute the program or not? That was the original
> discussion.
>
>> The same happens for the
>> vim executable on FreeBSD (1,221,212 bzip2'ed to 616,236). The Windows PE
>> binary format and ELF both require the executable image to contain all
>> initialized but essentially redundant (i.e. zeroed) parts of the data
>> segement. Don't pretend you didn't know that.
>
> Wrong for ELF.
>
> Simple example from /bin/cat:
>   Type           Offset   VirtAddr   PhysAddr   FileSiz MemSiz  Flg Align
>   LOAD           0x0049e8 0x100149e8 0x100149e8 0x00284 0x003cc RW
> 0x10000
>
> note that filesize is < memory size. I'll let you figure out how it works.
>
> This is the problem with your posts. You sound very authoritative, and
> I'm sure you figure you know all these bits,
> but you're wrong so often. It's confusing. It would be unfortunate
> were people to believe you are as right as you think you are.
>
>>
>> Also, Acme in p9p or on Plan 9 performs less than 1 in 5 of the functions
>> vi/vim does. That ratio is even smaller when comparing Acme to Emacs. So
>> you have been unfair. No kidding.
>>
>
> Ah! Now we're into feature comparisons! I'm game. How did you get to 1
> in 5 and not 1 in 4.8, or 1 in 6?
> Of course, you have to take into account that acme's extensions live
> in /bin or your number is bogus.
>
> ron
>