Re: ldso: dlclose.

[musl <b.brezillon.musl@gmail.com>]

On 24/08/2012 14:27, Rich Felker wrote:
> On Fri, Aug 24, 2012 at 09:52:28AM +0200, musl wrote:
>> On 23/08/2012 20:01, Rich Felker wrote:
>>> On Fri, Aug 24, 2012 at 12:02:09AM +0800, orc wrote:
>>>> On Thu, 23 Aug 2012 08:48:16 -0400
>>>> Rich Felker <dalias@aerifal.cx> wrote:
>>>>
>>>>> Anyway, unless the issue is fixed in binutils so that the vast
>>>>> majority of libraries are marked non-unloadable, I don't see anything
>>>>> we can do in musl. "glibc does it that way too" is not an excuse for
>>>>> adding unsafe/non-robust behavior to musl.
>>>>>
>>>>> Rich
>>>> The whole dlopen/dlclose/dlsym functions family are 'harmful': even if
>>>> we want static linking, application will still rely on them and fail
>>>> invisibly, creating more headaches.
>>>> I think better leave dlclose() in it's current state now. It will always
>>>> 'success', nobody will care.
>>> In my view, there are only two downsides to the current behavior:
>>>
>>> 1. Some buggy plugin-based applications may expect dlclose(plugin) to
>>> call the destructors in the plugin. This is of course an invalid
>>> expectation per POSIX, but it may be the reality for some apps.
>> Indeed, many plugins implem rely on constructors/destructors to
>> allocate/free memory or intialize/cleanup context.
>> This may lead to memory leaks or other issues if the plugin is
>> loaded/unloaded multiple times.
> A plugin cannot be loaded more than once. Subsequent calls to dlopen
> use the existing loaded image. The only way it could be loaded again
> is if the file were replaced by a new version.
>
> I think maybe you're not realizing that the "leak" can only happen if
> a new version of the .so file is put in place of the old one...
I was talking about this specific case :
1) unloding a plugin
2) updating the plugin (new plugin.so)
3) reloading the plugin

During the whole sequence the application is up and running.

Here is how I should do it if dlclose is implemented per posix :
1) stop the application
2) update the plugin
3) restart the application

The application is not available during this sequence.

>
>>> 2. In an extremely long-lived app that loads and unloads plugins which
>>> may be upgraded multiple times during the application's lifetime, each
>>> new version of the plugin will consume additional virtual memory space
>>> and commit charge, i.e. you have a memory leak. In the real world the
>>> leak should be very slow, but it could become significant if the
>>> plugins are very large and get reinstalled many times, perhaps if
>>> someone is experimenting and running "make install" each time...
>> It might be worst for long-lived apps running in a memory
>> constrained environment (embedded systems).
> Yes, but in this kind of system, ANY use of dynamic memory allocation
> is frowned upon. Dynamic module loading even moreso. And of course I
> don't think you'll be constantly replacing .so files on such a system
> with new versions.
>
>>> In my view #2 is a very low-priority problem that's not worth caring
>>> about on its own, but #1 may be relevant. If does become an important
>>> issue that we can't get fixed at the application level, I think the
>>> solution would be to add unloading, but have it only take effect for
>>> the actual argument to dlopen/dlclose, never any libraries implicitly
>>> loaded as dependencies (and of course to honor the flag that prevents
>>> unloading).
>> Does this mean you want to call plugin destructors in dlclose
>> function and keep the plugin memory mapping ?
> No. Calling dtors and unloading always come in a pair. You cannot call
> dtors but keep and reuse the mapping because the static-storage
> objects would retain their old values from the prior load, but a new
> load would be visible to the code in the plugin.
>
> The potential design I'm talking about would have only the dlopen'd
> library itself ever unloaded/unmapped. For example, if myplugin.so
> depends on libfoo.so and libbar.so, libfoo.so and libbar.so, which
> were implicitly loaded when loading myplugin.sh, will never be
> unmappable. Only myplugin.so itself would be unmappable. On
> unloading/unmapping dtors would be called as usual, and then the
> reference would be removed entirely from the DSO chain, causing it to
> be searched-out and loaded new next time dlopen is called.
>
> I do not want to do this except as a last resort, since as I've
> already mentioned it's highly error-prone (see glibc) and fragile.
I understand your concern and I'll modify my code to get rid of the dlclose function.
I hope there's no other apps or libs relying on gnu dlclose specific implem.
It should not if they've read carrefully the dlclose man page :-).

BTW, thanks for taking the time to explain the dlclose implications.
>
> Rich