build customizability

build customizability

[brainchild]

[-- Attachment #1.1: Type: text/plain, Size: 855 bytes --]


If someone wished to ship a functional executable of Pandoc in the 
distribution of a larger application, what possibility, if any, currently 
exists for building such an executable that only provides selected features 
of the full application but is significantly smaller in distribution 
footprint? In other words, what possibility is available, separate from any 
generic compiler settings, to customize a build for reduced executable size?

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/43e8ce91-0738-4477-bcf5-e826219d9b1d%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 1191 bytes --]

Re: build customizability

[John MacFarlane]

If you want just some of pandoc's features you'd have to do
surgery on the source code.  Not recommended unless you know
your way around with Haskell.  Of course the GPL permits
this kind of alteration, and distribution of the result,
provided the terms of the license are followed.

brainchild <ericlevy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> writes:

> If someone wished to ship a functional executable of Pandoc in the 
> distribution of a larger application, what possibility, if any, currently 
> exists for building such an executable that only provides selected features 
> of the full application but is significantly smaller in distribution 
> footprint? In other words, what possibility is available, separate from any 
> generic compiler settings, to customize a build for reduced executable size?
>
> -- 
> You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/43e8ce91-0738-4477-bcf5-e826219d9b1d%40googlegroups.com.

Re: build customizability

[brainchild]

[-- Attachment #1.1: Type: text/plain, Size: 2204 bytes --]

I understand the answer.

The reason that I ask is that many user applications, such as editors and 
other writing tools, are using Pandoc for writing printable or 
distribution-ready output documents. Some simply invoke the command through 
the system, yet others are including a copy of Pandoc in their 
distribution. The latter approach circumvents the uncertainty of dependence 
on a system installation, but at the cost of bloating distribution size. 
And yet, of the many capabilities of Pandoc, it is unlikely more than a 
select few are needed by any particular application of the above kind.

This observation leads me to wonder whether it is possible, at least in 
principle, to achieve a substantially smaller executable size that offers a 
specific selection of the full application's capabilities.

I understand that the current design would not make such a possibility 
immediately feasible.

Still, I think the question is useful to consider.

Because of library dependencies, and the question of which libraries may be 
unnecessary in specific cases, it very difficult for me, at the moment, to 
address quantitatively how much smaller the footprint might be for some 
particular case, if internal and external and components could be excluded 
that are not required by a given set of needs.

I do notice that the readers and writers collectively consume about 80% of 
the core application source code.



On Friday, May 1, 2020 at 11:51:20 AM UTC-4, John MacFarlane wrote:
>
>
> If you want just some of pandoc's features you'd have to do 
> surgery on the source code.  Not recommended unless you know 
> your way around with Haskell.  Of course the GPL permits 
> this kind of alteration, and distribution of the result, 
> provided the terms of the license are followed. 
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/a9b39338-0830-45dc-bf30-59d8de61278b%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 3013 bytes --]

Re: build customizability

[Albert Krewinkel]

brainchild writes:

> This observation leads me to wonder whether it is possible, at least in
> principle, to achieve a substantially smaller executable size that offers a
> specific selection of the full application's capabilities.

I had a look into this some time ago. My conclusion was: while it is
possible to arrive at a smaller executable by manually disabling all
unwanted code, it is not worth the effort.

The question usually arises with the idea of using pandoc for Markdown
parsing, outputting various other formats. Many large parsers, like
those for LaTeX and HTML, are dependencies of the Markdown reader, so
they cannot be removed. The most commonly used writers are also some of
the most complex, so there is not that much to gain there either.

I seem to remember that, even after removing a significant amount of
functionality, I still achieved only a meager 20% reduction in binary
size. More aggressive pruning should be able to squeeze out some more
percentage points, but the binary will never really be "small".

--
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

Re: build customizability

[brainchild]

[-- Attachment #1.1: Type: text/plain, Size: 1348 bytes --]

Albert, 

The truth may be as you say, but my intuitive sense of the number of 
separate readers, writers, and dependencies leaves me to feel that some 
suitable amount of engineering might produce results more optimal than the 
ones you suggest. I have not done any tests, of course, myself.

A few questions on your response appear below.

parsing, outputting various other formats. Many large parsers, like  
>
those for LaTeX and HTML, are dependencies of the Markdown reader, so  


This observations strikes me as quite peculiar. Would you elaborate the 
reasons for the dependencies?
 

> I seem to remember that, even after removing a significant amount of 
> functionality, I still achieved only a meager 20% reduction in binary 
>

Did your tests include the comprehensive elimination of unneeded library 
dependencies? Doing so correctly might be nontrivial as it requires having 
an accurate graph for dependency resolution.


-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/459f63cf-31b3-48a4-adfe-21d5f23f9d22%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 2214 bytes --]

Re: build customizability

[Ivan Lazar Miljenovic]

[-- Attachment #1: Type: text/plain, Size: 1238 bytes --]

On Sun, 3 May 2020 at 20:55, brainchild <ericlevy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:

>
> parsing, outputting various other formats. Many large parsers, like
>>
> those for LaTeX and HTML, are dependencies of the Markdown reader, so
>
>
> This observations strikes me as quite peculiar. Would you elaborate the
> reasons for the dependencies?
>

Because the Markdown parser can parse HTML and LaTeX fragments in Markdown
documents.

Another factor to take into account in the binary size is the GHC Run Time
System, etc.  This link is a bit dated but might give you some ideas:
https://stackoverflow.com/questions/6115459/small-haskell-program-compiled-with-ghc-into-huge-binary

-- 
Ivan Lazar Miljenovic
Ivan.Miljenovic-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
http://IvanMiljenovic.wordpress.com

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CA%2Bu6gbwbuU9QhDFygzHh259Pa99qCTNBAq6-Jym6z24hhCF%3DPQ%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 2524 bytes --]

Re: build customizability

[BPJ]

[-- Attachment #1: Type: text/plain, Size: 2643 bytes --]

Also binary size as opposed to memory usage and CPU usage is relatively
unimportant. I still remember when you had your system on one 3.5" disk and
your program *and* files on another. Now that was size restriction! I felt
like I had unlimited storage when I got an 80 MB HDD! If you have
experienced that it is hard to get worked up about saving a few tenths of
MBs in a binary which can be compressed to less than 25 MB in an age where
a somewhat dated phone has more than 60 GB for storage only. I'd be
interested to know where you are under that kind of size strictures in the
year 2020!

--
Better --help|less than helpless

Den sön 3 maj 2020 15:40Ivan Lazar Miljenovic <ivan.miljenovic@gmail.com>
skrev:

> On Sun, 3 May 2020 at 20:55, brainchild <ericlevy-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org> wrote:
>
>>
>> parsing, outputting various other formats. Many large parsers, like
>>>
>> those for LaTeX and HTML, are dependencies of the Markdown reader, so
>>
>>
>> This observations strikes me as quite peculiar. Would you elaborate the
>> reasons for the dependencies?
>>
>
> Because the Markdown parser can parse HTML and LaTeX fragments in Markdown
> documents.
>
> Another factor to take into account in the binary size is the GHC Run Time
> System, etc.  This link is a bit dated but might give you some ideas:
> https://stackoverflow.com/questions/6115459/small-haskell-program-compiled-with-ghc-into-huge-binary
>
> --
> Ivan Lazar Miljenovic
> Ivan.Miljenovic-Re5JQEeQqe8AvxtiuMwx3w@public.gmane.org
> http://IvanMiljenovic.wordpress.com
>
> --
> You received this message because you are subscribed to the Google Groups
> "pandoc-discuss" group.
> To unsubscribe from this group and stop receiving emails from it, send an
> email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
> To view this discussion on the web visit
> https://groups.google.com/d/msgid/pandoc-discuss/CA%2Bu6gbwbuU9QhDFygzHh259Pa99qCTNBAq6-Jym6z24hhCF%3DPQ%40mail.gmail.com
> <https://groups.google.com/d/msgid/pandoc-discuss/CA%2Bu6gbwbuU9QhDFygzHh259Pa99qCTNBAq6-Jym6z24hhCF%3DPQ%40mail.gmail.com?utm_medium=email&utm_source=footer>
> .
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/CADAJKhDW7y4JF3pDCMD4_6RoU36BwETeuPThmzKm%2BSG76rNVZw%40mail.gmail.com.

[-- Attachment #2: Type: text/html, Size: 4593 bytes --]

Re: build customizability

[Albert Krewinkel]

brainchild writes:
>> Many large parsers, like those for LaTeX and HTML, are dependencies
>> of the Markdown reader
>
> This observations strikes me as quite peculiar. Would you elaborate the
> reasons for the dependencies?

HTML can always be included in Markdown, and extensions like
`markdown_in_html_blocks` make it necessary to parse it.
IIRC, LaTeX is required for math support as well as the `+raw_tex`
extension. Removing those would mean rewriting large parts of the
parser.

>> I seem to remember that, even after removing a significant amount of
>> functionality, I still achieved only a meager 20% reduction in binary
>
> Did your tests include the comprehensive elimination of unneeded library
> dependencies? Doing so correctly might be nontrivial as it requires having
> an accurate graph for dependency resolution.

No, mostly for the reason you mention, but also because the number of
packages made redundant in my experiment were countable on one hand.
That's less than 10% of total dependencies.

--
Albert Krewinkel
GPG: 8eed e3e2 e8c5 6f18 81fe  e836 388d c0b2 1f63 1124

Re: build customizability

[brainchild]

[-- Attachment #1.1: Type: text/plain, Size: 3131 bytes --]

Thank you for the recent comments. Let me respond to all in one message, as 
they overlap in scope.

I address a few issues, in sequence, as follows:

   1. *Markdown reader dependencies: *I now understand the conflict. In 
   many potential deployment scenarios, for example applications that employ 
   Markdown source, the embedding of snippets in some target language is 
   unlikely to be needed, except perhaps as narrowly allowed for equations. 
   Thus, if this functionality were entirely enclosed in extensions, and if 
   those extensions could be disabled in the build, then in principle it may 
   be possible to realize the benefit of removing the dependencies on the 
   parsers. I understand that it would require restructuring the code.
   2. *Evaluating size of library dependencies: *If, as Albert says, only a 
   handful of packages may be eliminated from the build, even in the case that 
   only a few readers or writers were needed, and if the size of these 
   packages is not vastly greater than the size of others, then clearly the 
   material benefit of eliminating certain readers and writers from the build 
   is very limited. However, I make a few notes:
      1. Is the reason that most dependencies would remain even after 
      removing most of the readers and writers that these dependencies are needed 
      separately by all the readers and writers, or rather that they are needed 
      by the core application? In the latter case, perhaps some of the core 
      functionality not needed in basic uses, again at least in principle, can be 
      separated.
      2. My earlier question directed mostly at the difficulty of 
      determining confidently the complete set of dependencies to remove, without 
      invoking a formal solution such as graph traversal, compared to the 
      possibility of missing a set in an informal assessment.
      3. *Footprint of Runtime System:* No, I was not specifically aware of 
   the size of the Runtime System. I am having difficulty finding a clear 
   indication of how much size it adds to the final executable size, but 
   estimates that seem to recur are in the 10-15 megabyte range. If these 
   estimates are accurate, then it would be impossible to achieve vastly 
   smaller sizes, obviously, but this information alone is not sufficient to 
   conclude the impossibility of achieving significant savings through methods 
   already discussed. Also, I am seeing some hints that a GHC backend for LLVM 
   is available that makes possible much more optimal build results. 
   Unfortunately, I have not found any quantitative assessment of these 
   savings, nor can I find any active project maintaining this interface.
   

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/f9fd9cfe-e03f-45d3-bcb5-07267c28e565%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 3406 bytes --]

Re: build customizability

[brainchild]

[-- Attachment #1.1: Type: text/plain, Size: 1110 bytes --]

By the way, using a *hello world* example, I obtained the following results:

$ echo 'main = putStrLn "Hello, world!"' > hello.hs
$ ghc -o hello hello.hs
[1 of 1] Compiling Main             ( hello.hs, hello.o )
Linking hello ...
$ ./hello
Hello, world!
$ ls -sh ./hello
1008K ./hello

Perhaps GHC is using only a subset of the RTS, due to the limited scope of 
the program. 

(The build target does appear to have a few dependencies, but I believe all 
are independent of GHC: libgmp.so.10, libm.so.6, librt.so.1, libdl.so.2, 
libffi.so.6, libpthread.so.0, libc.so.6).

Someone with more experience than I on the topic might comment on what, if 
any, significance to associate with this result.





-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/4afcbd69-3e64-4edd-adae-5782c59ed2f6%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 4893 bytes --]

Re: build customizability

[Daniel Staal]

On 5/4/20 2:45 AM, brainchild wrote:
> *Markdown reader dependencies: *I now understand the conflict. In many 
> potential deployment scenarios, for example applications that employ 
> Markdown source, the embedding of snippets in some target language is 
> unlikely to be needed, except perhaps as narrowly allowed for equations. 
> Thus, if this functionality were entirely enclosed in extensions, and if 
> those extensions could be disabled in the build, then in principle it 
> may be possible to realize the benefit of removing the dependencies on 
> the parsers. I understand that it would require restructuring the code.

Just replying to this - in this case, you're getting far removed from 
Pandoc's processing, to the point where the input file for Pandoc is no 
longer compatible with your system.  What - at this point - is the 
benefit for using the Pandoc codebase?  If all you want is to parse 
Markdown or Latex or some other input format, there's lots of 
specialized parsers out there.

And further than that - what is the benefit to *Pandoc* of you using the 
Pandoc code base?  Since this would require a large restructure of the 
core code, what makes that restructure worthwhile to the Pandoc project? 
  Again, you're getting well off of Pandoc's goals and aim - and there 
are other products that do aim for that use.

Daniel T. Staal

-- 
---------------------------------------------------------------
This email copyright the author.  Unless otherwise noted, you
are expressly allowed to retransmit, quote, or otherwise use
the contents for non-commercial purposes.  This copyright will
expire 5 years after the author's death, or in 30 years,
whichever is longer, unless such a period is in excess of
local copyright law.
---------------------------------------------------------------

Re: build customizability

[brainchild]

[-- Attachment #1.1: Type: text/plain, Size: 1834 bytes --]



> Just replying to this - in this case, you're getting far removed from 
> Pandoc's processing, to the point where the input file for Pandoc is no 
> longer compatible with your system.  What - at this point - is the 
> benefit for using the Pandoc codebase?  If all you want is to parse 
> Markdown or Latex or some other input format, there's lots of 
> specialized parsers out there. 
>

I believe the idea I am expressing is very different from the effect you 
describe.
 

> And further than that - what is the benefit to *Pandoc* of you using the 
> Pandoc code base?  Since this would require a large restructure of the 
> core code, what makes that restructure worthwhile to the Pandoc project? 
>   Again, you're getting well off of Pandoc's goals and aim - and there 
> are other products that do aim for that use. 
>

I'm not sure what you're asking. Pandoc itself can experience no benefits, 
only its users and contributors, who participate in discussions such as 
this one. The amount of restructuring required, however much it is 
determined to be, as for any proposed enhancement, indeed might be a 
prohibitive factor. At the moment, a discussion is occurring such that the 
various difficulties may be discussed openly. 


Note further my earlier comments, which make note of a group already using 
Pandoc. 


What specific project goals do you feel the proposed idea might undermine 
or challenge?
  
 

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/a6771fd7-3253-43bc-82b3-caf659052052%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 2585 bytes --]

Colored text

[R. Wils]

[-- Attachment #1: Type: text/plain, Size: 144 bytes --]

Does anyone know how to color text in a PDF (converting from a pandoc.
markdown text file)?

I tried to use a span and div tag without success.

Re: Colored text

[brainchild]

[-- Attachment #1.1: Type: text/plain, Size: 685 bytes --]

Perhaps you would present this question in a new topic?


On Tuesday, May 5, 2020 at 4:39:31 AM UTC-4, R. Wils wrote:
>
> Does anyone know how to color text in a PDF (converting from a pandoc. 
> markdown text file)?
>
> I tried to use a span and div tag without success. 
>

-- 
You received this message because you are subscribed to the Google Groups "pandoc-discuss" group.
To unsubscribe from this group and stop receiving emails from it, send an email to pandoc-discuss+unsubscribe-/JYPxA39Uh5TLH3MbocFF+G/Ez6ZCGd0@public.gmane.org
To view this discussion on the web visit https://groups.google.com/d/msgid/pandoc-discuss/0b0ff08e-a73f-4893-86fe-cf23dfd2ad82%40googlegroups.com.

[-- Attachment #1.2: Type: text/html, Size: 1204 bytes --]