Re: Multimedia, PDF and ConTeXt

["Michal Vlasák via ntg-context" <ntg-context@ntg.nl>]

On Tue Jul 27, 2021 at 8:54 AM CEST, Hans Hagen wrote:
> On 7/27/2021 12:49 AM, Michal Vlasák via ntg-context wrote:
>
> > The viewers I tested were: Acrobat Reader DC, Foxit Reader, Sumatra PDF
> > on Windows and Evince, Okular, Xpdf, MuPDF, Firefox and Google Chrome on
> > Linux.
>
> also add the new edge on windows .. there are subtle differences in
> interfaces; although it doesn't fit into this topic, printing is also
> somethign to pay attention to; browser based pdf's can crash (mem
> consumption), the sometimes present doubleside directive can also
> confuse viewers

I thought that recent Edge versions used Chrome's PDFium, as you say
with some additions (like drawing). I would have to double check that.

From what I tried, from the tested viewers only Acrobat seemed to do
something with the double sided directive, if

    /ViewerPreferences <<
        /Duplex /DuplexFlipLongEdge
    >>

is what we are talking about.

> when it comes to checking you need to keep in mind that some readers
> start top-down (collect objects), others bottom up (xref), some use a
> mix, some apply heuristics (irr xpdf did something with t1 fonts that
> made cut/paste an issue), some are tolerant, others more strict,
> treatment of updates objects can differ, but not always consistent ...
> so, that also makes testing annotations a problem

Yeah, I noticed that some viewers have no problem with broken xref,
others do -- that is surely the symptom of top to bottom and bottom to
top as well as the tolerance.

I only tested with simplistic and mostly hand crafted documents with a
few objects. No updates. I was concerned with creating PDF with
annotations, not with adding them afterwards. (For some annotations both
make sense, not much for others.)

Although I noticed when I tried to add annotations that Acrobat didn't
even use the object generation numbers, but just added new objects.

> > 3) Multimedia ("Renditions")
> >
> >     [...]
> >
> >     This mechanism allows multimedia player controls, as well as PDF
> >     actions. The PDF action can be either one of the predefined ones or
> >     entirely specified in JavaScript (extra API is available for this).
>
> brings us to javascript ... it would be interesting to have the 'minimal
> useful' set of support needed; mupdf has javascript but nto realaly for
> annotations (for instance the ability to control layers and widget
> states would be nice ... it's trivial to program in viewers i bet)

I also looked into JavaScript. Exactly as you say MuPDF has JavaScript,
but only for forms. Firefox and Chrome also have JavaScript support, but
again mostly for forms.

None of the three support normal (not forms) JavaScript PDF actions.
Adding support presumably wouldn't be hard -- they already support some
of the API, they just don't allow it from these actions (but they do
allow Document Level JavaScript).

As you know, the JavaScript engine used in MuPDF has very Lua like C
API, so I too think that adding JavaScript API for something already
implemented in the viewer should be easy.

> > 4) 3D art
> > 
> >   - First appeared in PDF 1.6 (2004).
> > 
> >   - Only 3D files supported. This means U3D and later PRC files. The 3D
> >     objects described in the files are shown in a scene whose parameters
> >     (like camera position, angle, background color, etc.) can be
> >     configured.
> > 
> >   - The source is not a file, but a "PDF stream" (which is essentially
> >     embedded file with different metadata, but allows also "external
> >     files" to contain the stream data).
> > 
> >   - The 3D functionality is nice. It allows great amount of interactivity
> >     (playing with the camera, selectively disabling 3D objects, etc.) and
> >     also scriptability (switching between predefined "views" with PDF
> >     actions and a _lot_ of possibilities with JavaScript scripts).
> > 
> >   - This is the mechanism used for u3d and prc files in the ConTeXt
> >     "figure" mechanism (\externalfigure).
> > 
> >   - Apart from the external streams (see above) everything works in Adobe
> >     Acrobat. Foxit Reader also has support, but it is limited (no
> >     support for JavaScript and printing).
>
> Wasn't that driven by apps like autocad? Is U3D kind of a standard that
> will stay?

My understanding is that U3D was the first thing they designed. It is
standardised by ECMA, so freely available. Then they came up with PRC,
which should be more space efficient IIRC. It is standardized by ISO, so
it is open, but not free.

Both formats seemed to worked fine in Acrobat and Foxit and I don't see
that changing soon.

Both have some writing support in open source tools - meshlab supports
U3D export with https://github.com/ningfei/u3d, and Asymptote supports
PRC export. Although I didn't get either to quite work and used export
from Autodesk Inventor instead for my tests.

> > 5) Rich Media
> > 
> >    [...]
> >
> >     The old idea was that the PDF viewer would support Flash (and playing
> >     its video as well as mp4), but the audio/video wouldn't be played
> >     directly by the PDF viewer, but by a Flash application (embedded in
> >     the PDF along with the media file). This means that the mechanism has
> >     inherent complexity that is not justified nowadays (essentially four
> >     levels of indirection for a plain audio / video file).
>
> yes, flash is gone (one can say: of course, why else would adobe has
> bought them) .. i think i already moved the files to the attic .. one
> had to embed a 'flash player' with gui stuff

The support is still there (although not in the format, it has to be
explicitly loaded). java-imp-vplayer.mkiv, java-imp-videoplayer.mkiv, as
well as lpdf-swf.lua, grph-swf.lua. Lastly back-swf.mkiv serves as an
example of usage.

(One has to download the .swf player from other sourcese manually.)

> >     While the same thing should have been true for 3D files I couldn't
> >     find any real usage like that. Instead it seems that 3D files with
> >     Rich Media have always been used like with the "3D art" mechanism
> >     (but with different wrappers).
> > 
> >     There is essentially no scriptability for audio and video. (Note that
> >     in this regard 3D files work just like with the "3D art" mechanism).
> >     There also isn't an easy way to display multimedia player controls (a
> >     hack works for Acrobat). One thing that it allows is playing the
> >     media in a customizable window, even full screen (not only in a part
> >     of a page like the previous mechanisms allow).
>
> one could / can start/stop etc from, javascript

That was tied to the Flash player and is not possible today.

(Using JavaScript you would call ActionScript actions on the .swf media
player.)

> >   - ConTeXt uses this mechanism for Flash (SWF) files in the figure
> >     mechanism. This is also allows audio/video (Flash media player, like
> >     "vplayer" is inserted and the media file is its parameter), see for
> >     example "java-imp-vplayer.mkiv".
> > 
> >   - Both Flash and "plain" Rich media are supported by Acrobat Reader.
> >     Okular only supports Flash Rich Media. How is this possible,
> >     considering that Flash player is dead? Well, both viewers have a
> >     compatibility layer that detects embedded Flash media player file and
> >     doesn't use it to play the video, but instead plays the video
> >     natively. This is good, because there are a lot of documents out
> >     there which use Flash based Rich Media. But there is absolutely no
> >     need to create new documents with embedded Flash player applications,
> >     it only takes space and isn't even used.
>
> hm, i didn't know that ... is that kind of official? this 'replacement'
> trick?

Not fully official, but it is only logical that they need a compatibility
layer for old documents. But, Adobe employee mentionts the use of OS Player
for these Flash using documents here:

https://community.adobe.com/t5/acrobat-reader/playing-embedded-video-without-flash/td-p/11266205/page/2

(IIRC The check box for not using the Flash player but OS player later
disappeared and became the default.)

Not that there is a reason to depend on the trick. New document should
be created without Flash media players embedded.

> > All in all, of the five mechanisms 2 are deprecated and unfortunately no
> > longer supported in the most used PDF viewer and other 2 mechanisms are
> > needlessly complex and in reality limited. For example, while the
> > multimedia mechanism supports JavaScript, (AFAIK) only Acrobat Reader
> > supports that, this further limits the viewer support or available
> > features, choose one.
>
> a more philosophical: do "standards" or "formats" guarantee fuiture
> usage .. is investing time in some formats worth the effort

The situation is difficult. I wouldn't risk creating a document with
audio / video today and hoping that it would work in a few years. But if
it is something that is needed now, then I can be pretty sure of the
support.

> > After evaluating these mechanisms I came to conclusion, that a PDF
> > writer today is best at:
> > 
> >   - Embedding video and audio using the "multimedia" ("renditions")
> >     mechanism. It is supported in proprietary and open source viewers
> >     alike. Customization and scripting / PDF actions is out of the
> >     question, though.
> > 
> >   - Embedding 3D files using the "Rich Media" mechanism. While it is
> >     essentially just a few differences in wrappers, it has real
> >     advantages (data sources are files not streams, and multiple
> >     JavaScript script files are supported), that I find nice enough for
> >     the implementation and users alike.
>
> maybe we need some javascript supported actions in viewers table

Do you mean table summarizing what parts of the Acrobat JavaScript API
is supported in individual viewers?

There is a separate API for normal things:

https://opensource.adobe.com/dc-acrobat-sdk-docs/acrobatsdk/pdfs/acrobatsdk_jsapiref.pdf

And for 3D:

(seems to have disappeared from the internet, here is the TOC
https://opensource.adobe.com/dc-acrobat-sdk-docs/acrobatsdk/pdfs/acrobatsdk_3djs.pdf,
I have the full document saved from earlier, though).


> > Some sources for this topic are also the LaTeX centric [4] and [5]. I go
> > into more details in the former. In the latter the "plain" Rich Media
> > and "multimedia" ("renditions") mechanisms are suggested as solutions
> > for the Flash media player approach.
>
> but we have to stick to what we think will stay (kind of generic
> features of such things)

Renditions is what works best today. But I think that for future
bringing Evince and Okular up to par with Acrobat in supporting plain
(Flashless) Rich Media is the way to go.

I would have to peek into the latest standard to know if there is
anything new with regards to showing media player controls or control
using actions / JavaScript. But other than that it is the most modern
mechanism and not lot required for supporting it in viewers that already
can play some video / audio.

> > And now for the future. What should ConTeXt do? On one hand all
> > available mechanisms are flawed in one way or the other. On the other
> > hand some users may still find the functionality useful. My suggestions
> > is to either delete all the support for audio/video or:
> > 
> >   1) Delete the "Movie objects" implementation of figures. It is not
> >      supported in viewers, where users expect it to [6].
>
> it's actually an abstraction but we can map it onto somethign else

Yes, I mentioned it below that it could switch to another backing
mechanism.

> >   2) Delete all mentions of Flash. There is no reason to create new
> >      documents with embedded Flash files, even though they may work in
> >      some viewers. Plain Rich Media can be used instead, with hopefully
> >      soon equal support [2].
>
> indeed
>
> >   3) The "externalrendering" mechanism (multimedia/renditions) can stay.
> >      If the insertion of audio/video as "figures" is to stay, then I
> >      suggest to use multimedia/renditions for it (in simplified form).
> > 
> > Note that the 3D support in ConTeXt is completely fine and works in
> > Acrobat and Foxit.
>
> do we actually have some simple test files for that? do we need a small
> set of files for media?

Do you mean existing PDFs that work in viewers or ConTeXt source files
that use these features? I can provide both.

I am not sure what you mean by "small set of files for media".

> > The "externalrendering" part currently has three "bugs". Previous
> > discussion at this list provides some context [7]. The following is
> > currently "wrong":
> > 
> >   - Currently ConTeXt wraps a PDF file specification for embedded file
> >     inside another file specification (i.e. embedded files don't work).
> >   - As a result of "externalrendering" inheriting from \framed, the
> >     PDF annotation late_lua whatsit is centered inside the frame and so
> >     the annotation itself is offset by half its width to the right.
> >   - ConTeXt doesn't explicitly allow the viewer to create temporary
> >     files, hence the playback fails in Acrobat Reader.
>
> hm, never seen that temp thing ... must be something recent (or maybe it
> is a preference)

It is possible that it is recent -- maybe the Windows Media Player
requires it, while the Quicktime did not.

> >      \starttext
> >      \setupinteraction[state=start]
> >      
> >      \useexternalrendering[myvideo][video/mp4][video.mp4][embed=yes]
> >      \useexternalrendering[myvideo2][video/mp4][https://gitlab.com/agrahn/media9/uploads/c7e2ae944fbd711df4ad7bd58000f83a/nightseq1.mp4]
> >      \useexternalrendering[myvideo3][video/mp4][video.mp4]
> >      
> >      \definerenderingwindow[myrenderingwindow][width=\textwidth, height=\textwidth]
> >      
> >      \noindent
> >      \placerenderingwindow[myrenderingwindow][myvideo]
> >      
> >      \goto{START}[StartRendering{myvideo}]
> >      \goto{STOP} [StopRendering{myvideo}]
> >      \goto{PAUSE}[PauseRendering{myvideo}]
> >      
> >      \vfil\break\noindent
> >      \placerenderingwindow[myrenderingwindow][myvideo2]
> >      
> >      \vfil\break\noindent
> >      \placerenderingwindow[myrenderingwindow][myvideo3]
> >      
> >      \stoptext
> > 
> > All three file source types are demonstrated. Any "video.mp4" in the
> > directory you compile in will do. (Works as expected in Okular on
> > Linux.)
>
> good. but not in mupdf viewers, right?

No, not in MuPDF viewers. Only Acrobat (first page only), Foxit, Okular
and Evince.

MuPDF doesn't even try to support video. They target multiple platforms
including mobile. The functionality would probably have to be added on
top of the core (i.e. in Sumatra).

> > If more documents / snippets / explanations are needed I hope I can
> > provide them.
> > 
> > Sadly, while working on this, I didn't have access to the PDF 2.0
> > standard. My information mostly comes from the PDF 1.7 standard and
> > publicly known information about PDF 2.0 - the Rich Media mechanism got
> > included in PDF 2.0, but I am not sure in what extent did the Flash part
> > get included. I also don't know if there really is anything new, but
> > nothing suggests it. Regardless, viewer support isn't complete for
> > something standardized over 20 years ago, I don't expect revolution in
> > the PDF viewers, considering the price of the standard(s).
>
> indeed. but wrt viewers, what puzzles me is that much of this rendering
> stuff is around in browsers so i guess it also has to do with lack of
> interest

For sure. All of the interaction / multimedia features are not well
supported. It must be circular -- not supported in viewers, users don't
use it, viewers don't implement it.

> What actuially was also supported is
>
> https://www.w3.org/TR/SMIL/
>
> but somehow that became unreliable / went away .. when it first showed
> up it worked perfect and i always though that it would stay

I found some references to it from the PDF standard (in the Renditions
section), but honestly I think that it is long dead.


Michal Vlasák
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________