Possible memory leak?

Possible memory leak?

[Yue Wang]

Hi, Hans:

in font-afm.lua, you use fontforge to open and to_table a pfb, but
without closing it?

btw, when MKIV is running for the first time (i.e., no cache file). it
will use a lot of memory (parsing a 10M font will use 700-800 MB
memory), are there some ways to reduce that? Another problem is, the
memory usage seems incremental:
I watch the memory usage in "top" carefully, it does not reduce when
MKIV finish reading and caching an otf font. (well I read the
font-otf.lua, and you are actually closing that font. so maybe that's
because luatex is leaking memory?)

btw, I load arial 300 times using the following file,

%engine=luatex
\directlua0{
function load_font (filename)
local metrics = nil
    local font = fontforge.open(filename)
    if font then
        metrics = fontforge.to_table(font)
        fontforge.close(font)
    end
    return metrics
end
myfont = load_font('arial.ttf')
% 300 myfont = load_font('arial.ttf')
myfont = load_font('arial.ttf')
myfont = load_font('arial.ttf')
myfont = load_font('arial.ttf')
myfont = load_font('arial.ttf')
}
and I can see the memory usage may ebb and flow, but actually it is
leaking memory.

Yue Wang
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Possible memory leak?

[Hans Hagen]

Yue Wang wrote:
> Hi, Hans:
> 
> in font-afm.lua, you use fontforge to open and to_table a pfb, but
> without closing it?

ok, thanks for noticing, i added a

   fontforge.close(pfbblob)

> btw, when MKIV is running for the first time (i.e., no cache file). it
> will use a lot of memory (parsing a 10M font will use 700-800 MB

hm so much?

> memory), are there some ways to reduce that? Another problem is, the
> memory usage seems incremental:
> I watch the memory usage in "top" carefully, it does not reduce when
> MKIV finish reading and caching an otf font. (well I read the
> font-otf.lua, and you are actually closing that font. so maybe that's
> because luatex is leaking memory?)

it's hard to say ... it related to the lua garbage collector

i already discard the loaded table and reload from cache (which save 
mem) but one has to wait till the garbage collector kicks in which can 
later

also, mem consumption reported is a bit of a lie because lua seems to 
allocates double of what it needed with each increment but not always 
use it

otf fonts are already quite packed in mkiv (i'm considering adding an 
option to mtx-font to precache fonts but i'm not sure if it's worth the 
trouble

is mem consumption less a second run?

we use our mk document as test and there mem consumption is ok (given 
the number of loaded fonts)

Hans



-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
      tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Possible memory leak?

[Yue Wang]

> ok, thanks for noticing, i added a
>
>  fontforge.close(pfbblob)
>

good

>> btw, when MKIV is running for the first time (i.e., no cache file). it
>> will use a lot of memory (parsing a 10M font will use 700-800 MB
>
> hm so much?
>

Yes, when finish reading the following tex file without \bye using plain TeX,
LuaTeX will cost 719M memory on my machine. The font used is Adobe
song std light.

[yue@localhost ~]$ cat a.tex
\directlua0{
function load_font (filename)
local metrics = nil
   local font = fontforge.open(filename)
   if font then
       metrics = fontforge.to_table(font)
       fontforge.close(font)
   end
   return metrics
end
myfont = load_font('song.otf')
}
[yue@localhost ~]$

when using the above script loading four fonts (you can try this by
adding 3 myfont = load_font('song.otf')),
the memory consumption will be higher than 1G on my machine. Remember
I am using Plain TeX to do this test, MKIV will consume more memory.


> also, mem consumption reported is a bit of a lie because lua seems to
> allocates double of what it needed with each increment but not always use it
>
> otf fonts are already quite packed in mkiv (i'm considering adding an option
> to mtx-font to precache fonts but i'm not sure if it's worth the trouble
>
> is mem consumption less a second run?

Yes, dramatically less. with 4 chinese fonts loaded (I use zhfonts.tex
for testing), MKIV consumes 300-400MB.

Yue Wang
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Possible memory leak?

[Hans Hagen]

Yue Wang wrote:

>> also, mem consumption reported is a bit of a lie because lua seems to
>> allocates double of what it needed with each increment but not always use it
>>
>> otf fonts are already quite packed in mkiv (i'm considering adding an option
>> to mtx-font to precache fonts but i'm not sure if it's worth the trouble
>>
>> is mem consumption less a second run?
> 
> Yes, dramatically less. with 4 chinese fonts loaded (I use zhfonts.tex
> for testing), MKIV consumes 300-400MB.

on my machine running mk with a fresh cache gives the same lua mem usage 
as with filled cache, however a fresh run tops at 1 meg while a filled 
run tops ar 500 (windows vista); that document uses several cjk fonts 
(relatively large due to many glyphs) and script fonts (some arabic and 
zapfino as well as cambria math which are much larger fonts due to many 
features

mkiv lua stats : loaded fonts              - adobesongstd-light:otf 
arabtype:ttf c:/windows/fonts/cambria:ttc htfs:ttf htsong:ttf ko
eieletters:afm lmex10:tfm lmmi10:tfm lmmib10:tfm lmmono10-regular:otf 
lmmonoltcond10-regular:otf lmroman10-italic:otf lmroman10-regu
lar:otf lmroman12-regular:otf lmsans10-regular:otf lmsy10:tfm 
lmtypewriter10-regular:otf msam10:tfm msbm10:tfm palatinoltarabic-regu
lar:ttf rm-lmbx10:tfm rm-lmr10:tfm rm-lmr12:tfm rm-lmr8:tfm 
rm-lmtt10:tfm scheherazaderegot:ttf texnansi-lt55484:afm texnansi-lt5548
5:afm texnansi-lt55488:afm texnansi-lt55507:afm zapfinoextraltpro:otf 


mkiv lua stats : current memory usage      - 361 MB (ctx: 370 MB)

(keep in mind that at some point luatex has to load the glyph shapes and 
pass them to the pdf file so there will always be otf/pfb loading at the 
end of a run)

as taco mentioned, real memory usage for a fresh run is probably 
somewhat lower due to fragmentation

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
      tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com
                                              | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Possible memory leak?

[Yue Wang]

> on my machine running mk with a fresh cache gives the same lua mem usage as
> with filled cache, however a fresh run tops at 1 meg while a filled run tops
> ar 500 (windows vista); that document uses several cjk fonts (relatively
> large due to many glyphs) and script fonts (some arabic and zapfino as well
> as cambria math which are much larger fonts due to many features
>
> mkiv lua stats : loaded fonts              - adobesongstd-light:otf
> arabtype:ttf c:/windows/fonts/cambria:ttc htfs:ttf htsong:ttf ko
> eieletters:afm lmex10:tfm lmmi10:tfm lmmib10:tfm lmmono10-regular:otf
> lmmonoltcond10-regular:otf lmroman10-italic:otf lmroman10-regu
> lar:otf lmroman12-regular:otf lmsans10-regular:otf lmsy10:tfm
> lmtypewriter10-regular:otf msam10:tfm msbm10:tfm palatinoltarabic-regu
> lar:ttf rm-lmbx10:tfm rm-lmr10:tfm rm-lmr12:tfm rm-lmr8:tfm rm-lmtt10:tfm
> scheherazaderegot:ttf texnansi-lt55484:afm texnansi-lt5548
> 5:afm texnansi-lt55488:afm texnansi-lt55507:afm zapfinoextraltpro:otf
>

depends on how you use them.
\font\a=simsun at 10pt and \definetypeface blablabla [simsun] are different.

> mkiv lua stats : current memory usage      - 361 MB (ctx: 370 MB)

You cannot depend on the memory report by mtxrun.
The actual memory usage is *much* larger than the reported value.
Only the one reported by the system is believable.

>
> (keep in mind that at some point luatex has to load the glyph shapes and
> pass them to the pdf file so there will always be otf/pfb loading at the end
> of a run)
>
> as taco mentioned, real memory usage for a fresh run is probably somewhat
> lower due to fragmentation

The memory usage without taking fragmentation into account has nothing
to do with the user's experience.

With 1G memory on my machine, and the best memory allocation
implementation I can find (The FreeBSD operating system's memory
allocation algorithm is very efficient. Firefox 3 also uses its
implementation). In order to use ConTeXt with 4 chinese fonts, I
should
- First, quit Firefox, Gnome and X Window
- Run sudo shutdown now and log into the single user mode
- Run context foobar.tex, when finishing dumping the first Chinese
font's font cache, press control + C, and run that again for the
second font.

And that's my experience. If I don't do that, the swap will be caching
for several minutes, and finally the kernel stopped the luatex program
since it uses too much memory.


>
> -----------------------------------------------------------------
>                                          Hans Hagen | PRAGMA ADE
>              Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
>     tel: 038 477 53 69 | fax: 038 477 53 74 | www.pragma-ade.com
>                                             | www.pragma-pod.nl
> -----------------------------------------------------------------
> ___________________________________________________________________________________
> If your question is of interest to others as well, please add an entry to
> the Wiki!
>
> maillist : ntg-context@ntg.nl /
> http://www.ntg.nl/mailman/listinfo/ntg-context
> webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
> archive  : https://foundry.supelec.fr/projects/contextrev/
> wiki     : http://contextgarden.net
> ___________________________________________________________________________________
>
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Possible memory leak?

[Taco Hoekwater]

Yue Wang wrote:

> And that's my experience. If I don't do that, the swap will be caching
> for several minutes, and finally the kernel stopped the luatex program
> since it uses too much memory.

Can you send me that input file and the fonts it needs (if they are not
freely available)?

Best wishes,
Taco

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Possible memory leak?

[Taco Hoekwater]

Hi Yue Wang,

Yue Wang wrote:
> myfont = load_font('arial.ttf')
> myfont = load_font('arial.ttf')
> }
> and I can see the memory usage may ebb and flow, but actually it is
> leaking memory.

I also see a non-trivila leak (somewhere inside the fontforge library).
I will investigate further next week, it is not that massive. I get this
result for your input file, with 500 arial.ttfs (ArialMT  version 3.0,
367112 bytes):

==8819== LEAK SUMMARY:
==8819==    definitely lost: 16,965,191 bytes in 707,541 blocks.
==8819==      possibly lost: 1,128 bytes in 22 blocks.
==8819==    still reachable: 202,135,427 bytes in 81,127 blocks.
==8819==         suppressed: 0 bytes in 0 blocks.
==8819== Rerun with --leak-check=full to see details of leaked memory.

So some 32kb per font. Noticeable, but not enormous. The growth in
'top' is probably caused more by memory fragmentation than by this leak.

Best wishes,
Taco







___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://tex.aanhet.net
archive  : https://foundry.supelec.fr/projects/contextrev/
wiki     : http://contextgarden.net
___________________________________________________________________________________