Pandoc killed after eating all memory (large table, multiple links)

Pandoc killed after eating all memory (large table, multiple links)

[jn]

[-- Attachment #1.1: Type: text/plain, Size: 5982 bytes --]

When converting a database-generated markdown table with about 40.000 lines 
(companies_with_metadata.de.md , 7.8MB), Pandoc is killed (Command 
terminated by signal 9) after having claimed almost all of the available 32 
GB of memory. Elapsed time is more than 8 minutes - see timing and memory 
usage details below.

With a prior slightly smaller version of the same file, pandoc finished 
successfully after 2 minutes and having used 2 GB of memory. 
(companies_with_metadata.de.md vs. companies_with_metadata_smaller.de.md; 
main difference is two additional table columns with sparse links.) Smaller 
tables used comparatively much less memory and processing time.

I was not able to find anything wrong (e.g. improper nesting) in the file. 
It would be great if someone knowlegdeable could take a look - source files 
are available at https://pm20.zbw.eu/tmp/pandoc_memory/pandoc_memory.tgz.

The issue is not version-specific and occured already in Pandoc 1.7.x. 

Cheers, Joachim

/usr/bin/time -v /home/nbt/bin/pandoc-2.14.2 --standalone --variable is_de 
--variable lang:de --template pm20_default.html --css simple.css -o 
companies_with_metadata.de.html companies_with_metadata.de.md
Command terminated by signal 9
        Command being timed: "/home/nbt/bin/pandoc-2.14.2 --standalone 
--variable is_de --variable lang:de --template pm20_default.html --css 
simple.css -o companies_with_metadata.de.html companies_with_metadata.de.md"
        User time (seconds): 322.20
        System time (seconds): 50.70
        Percent of CPU this job got: 73%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 8:30.52
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 32189460
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 807658
        Minor (reclaiming a frame) page faults: 10704307
        Voluntary context switches: 860320
        Involuntary context switches: 837943
        Swaps: 0
        File system inputs: 60409704
        File system outputs: 8
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0


companies_with_metadata_smaller.de.md (7.2M)
        Command being timed: "/home/nbt/bin/pandoc-2.14.2 --standalone 
--variable is_de --variable lang:de --template pm20_default.html --css 
simple.css companies_with_metadata_smaller.de.md"
        User time (seconds): 113.59
        System time (seconds): 2.79
        Percent of CPU this job got: 92%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 2:05.45
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 2084384
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 91
        Minor (reclaiming a frame) page faults: 503767
        Voluntary context switches: 12754
        Involuntary context switches: 5527
        Swaps: 0
        File system inputs: 287160
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0


persons_with_metadata.de.md (3.2M)

        Command being timed: "/home/nbt/bin/pandoc-2.14.2 --standalone 
--variable is_de --variable lang:de --template pm20_default.html --css 
simple.css persons_with_metadata.de.md"
        User time (seconds): 19.25
        System time (seconds): 0.87
        Percent of CPU this job got: 87%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:23.04
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 874200
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 1
        Minor (reclaiming a frame) page faults: 201308
        Voluntary context switches: 2344
        Involuntary context switches: 833
        Swaps: 0
        File system inputs: 2280
        File system outputs: 0
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0


intern.k1_sh.de.md (1.6 M)

        Command being timed: "/home/nbt/bin/pandoc-2.14.2 --standalone 
--variable is_de --variable lang:de --template pm20_default.html --css 
simple.css intern.k1_sh.de.md"
        User time (seconds): 15.91
        System time (seconds): 1.31
        Percent of CPU this job got: 86%
        Elapsed (wall clock) time (h:mm:ss or m:ss): 0:19.89
        Average shared text size (kbytes): 0
        Average unshared data size (kbytes): 0
        Average stack size (kbytes): 0
        Average total size (kbytes): 0
        Maximum resident set size (kbytes): 620008
        Average resident set size (kbytes): 0
        Major (requiring I/O) page faults: 90
        Minor (reclaiming a frame) page faults: 137687
        Voluntary context switches: 2142
        Involuntary context switches: 737
        Swaps: 0
        File system inputs: 271072
        File system outputs: 8
        Socket messages sent: 0
        Socket messages received: 0
        Signals delivered: 0
        Page size (bytes): 4096
        Exit status: 0


-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/65dfddbb-a2d9-4ae4-91bf-ba3dec4b8da4n%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 9859 bytes --]