Re: Multimedia, PDF and ConTeXt

[Hans Hagen via ntg-context <ntg-context@ntg.nl>]

On 7/27/2021 12:49 AM, Michal Vlasák via ntg-context wrote:

> The viewers I tested were: Acrobat Reader DC, Foxit Reader, Sumatra PDF
> on Windows and Evince, Okular, Xpdf, MuPDF, Firefox and Google Chrome on
> Linux.

also add the new edge on windows .. there are subtle differences in 
interfaces; although it doesn't fit into this topic, printing is also 
somethign to pay attention to; browser based pdf's can crash (mem 
consumption), the sometimes present doubleside directive can also 
confuse viewers

when it comes to checking you need to keep in mind that some readers 
start top-down (collect objects), others bottom up (xref), some use a 
mix, some apply heuristics (irr xpdf did something with t1 fonts that 
made cut/paste an issue), some are tolerant, others more strict, 
treatment of updates objects can differ, but not always consistent ... 
so, that also makes testing annotations a problem

the command line program qpdf is one of the best programs to check (when 
luigi sand i were checking luatex pdf issues we used qpdf and mupdf mostly)

> Now to the different mechanisms:
> 
> 1) Sound objects
> 
>   - First appeared in PDF 1.2 (1996), but had since been deprecated (PDF
>     1.5, 2003) and became unsupported (PDF 2.0, 2017).
> 
>   - Only audio is supported.
> 
>   - "Raw" and in practice uncompressed PCM audio can be embedded (i.e.
>     ".wav" format without the metadata).
>     Otherwise an external file may be used (this one has to be in a real
>     audio format - like ".wav" - i.e. with metadata).
> 
>   - Users usually don't have raw audio. So embedding requires
>     preprocessing. Some control using PDF actions is possible.
> 
>   - Not supported in ConTeXt.
>   - None of the viewers supports the external files. Only Acrobat Reader
>     supports the embedded raw audio.

amazing right? all that talk of accessibility and no proper easy audio 
support

> 2) Movie objects
> 
>   - First appeared in PDF 1.2 (1996), but had since been deprecated (PDF
>     1.5, 2003) and became unsupported (PDF 2.0, 2017).
> 
>   - Both video and audio is supported.
> 
>   - Any source (embedded, external and URL file).
> 
>   - In all regards superior to sound objects. Is still relatively simple
>     and allows some customization and control (media player controls, PDF
>     actions).
> 
>   - This is the backing mechanism for including video and audio in
>     ConTeXt (\externalfigure, \useexternalsoundtrack).
> 
>   - Supported only in Evince and Okular (with their usual quirks, see
>     below). Notably Acrobat Reader does no longer support this mechanism.

this all became an issue when they dropped the quicktime plugin 
(probably also relates to the usual big companies competing politics)

> 3) Multimedia ("Renditions")
> 
>   - First appeared in PDF 1.5 (2003). Adobe Acrobat considers them
>     "legacy".
> 
>   - Both video and audio supported (as well as other unspecified types of
>     multimedia, like images and Flash, but not really, see below).
> 
>   - In theory all source types should be possible.
> 
>   - This mechanism was supposed to replace sound and movie objects. Hence
>     their deprecation. The mechanism is complex (the spec is 10 times
>     longer than that for movie objects). It expects the PDF viewers to
>     work with plugins and introduces ways for determining if a media file
>     is really playable in some plugin. It is allowed to even include
>     more media files (to serve as fallback should the primary one be
>     unsupported by the viewer). Other complexity is that the concept of
>     the rectangle where the media will be played ("screen") is separated
>     from the media itself ("rendition"). In theory this allows mixing and
>     matching them, but in practice is a lot of unnecessary complexity, at
>     least in my opinion.

I fully agree. As with many of these annotation pdf things it's probably 
reversed engeneered from some application or plugin, not the result of 
clear design. It also evolves when there is no way to generate it (tex 
is often one of the first to do that and then we run into specs not 
fitting reality). Kind of making equipment that should work on mars and 
test it on earth (if at all).

>     This mechanism allows multimedia player controls, as well as PDF
>     actions. The PDF action can be either one of the predefined ones or
>     entirely specified in JavaScript (extra API is available for this).

brings us to javascript ... it would be interesting to have the 'minimal 
useful' set of support needed; mupdf has javascript but nto realaly for 
annotations (for instance the ability to control layers and widget 
states would be nice ... it's trivial to program in viewers i bet)

>   - This is the mechanism behind \useexternalrendering. This has been
>     used for Flash (.swf files) and "manual" audio + video insertion as
>     far as I can tell.
> 
>   - Evince and Okular support this (with usual quirks), but not for
>     external files (Evince segfaults). Acrobat and Foxit support this
>     mechanism as well, but Acrobat only allows embedded files. Okular by
>     mistake auto-plays all media [1].

hm, sounds bad

> 4) 3D art
> 
>   - First appeared in PDF 1.6 (2004).
> 
>   - Only 3D files supported. This means U3D and later PRC files. The 3D
>     objects described in the files are shown in a scene whose parameters
>     (like camera position, angle, background color, etc.) can be
>     configured.
> 
>   - The source is not a file, but a "PDF stream" (which is essentially
>     embedded file with different metadata, but allows also "external
>     files" to contain the stream data).
> 
>   - The 3D functionality is nice. It allows great amount of interactivity
>     (playing with the camera, selectively disabling 3D objects, etc.) and
>     also scriptability (switching between predefined "views" with PDF
>     actions and a _lot_ of possibilities with JavaScript scripts).
> 
>   - This is the mechanism used for u3d and prc files in the ConTeXt
>     "figure" mechanism (\externalfigure).
> 
>   - Apart from the external streams (see above) everything works in Adobe
>     Acrobat. Foxit Reader also has support, but it is limited (no
>     support for JavaScript and printing).

Wasn't that driven by apps like autocad? Is U3D kind of a standard that 
will stay?

> 
> 5) Rich Media
> 
>   - First appeared in Adobe extension level 3 to PDF 1.7 (2008). Later
>     included in PDF 2.0 (2017). It was meant to replace both multimedia
>     (renditions) and 3D art mechanisms, with unified mechanism based on
>     Flash, thus also supporting arbitrary Flash applications.
> 
>   - Supports video, audio and 3D.
> 
>   - Only embedded files are supported.
> 
>   - While the mechanism is heavily based on Flash (which is dead, since
>     December 2020) it allows also "plain" Rich Media without Flash.
> 
>     The old idea was that the PDF viewer would support Flash (and playing
>     its video as well as mp4), but the audio/video wouldn't be played
>     directly by the PDF viewer, but by a Flash application (embedded in
>     the PDF along with the media file). This means that the mechanism has
>     inherent complexity that is not justified nowadays (essentially four
>     levels of indirection for a plain audio / video file).

yes, flash is gone (one can say: of course, why else would adobe has 
bought them) .. i think i already moved the files to the attic .. one 
had to embed a 'flash player' with gui stuff

>     While the same thing should have been true for 3D files I couldn't
>     find any real usage like that. Instead it seems that 3D files with
>     Rich Media have always been used like with the "3D art" mechanism
>     (but with different wrappers).
> 
>     There is essentially no scriptability for audio and video. (Note that
>     in this regard 3D files work just like with the "3D art" mechanism).
>     There also isn't an easy way to display multimedia player controls (a
>     hack works for Acrobat). One thing that it allows is playing the
>     media in a customizable window, even full screen (not only in a part
>     of a page like the previous mechanisms allow).

one could / can start/stop etc from, javascript

>   - ConTeXt uses this mechanism for Flash (SWF) files in the figure
>     mechanism. This is also allows audio/video (Flash media player, like
>     "vplayer" is inserted and the media file is its parameter), see for
>     example "java-imp-vplayer.mkiv".
> 
>   - Both Flash and "plain" Rich media are supported by Acrobat Reader.
>     Okular only supports Flash Rich Media. How is this possible,
>     considering that Flash player is dead? Well, both viewers have a
>     compatibility layer that detects embedded Flash media player file and
>     doesn't use it to play the video, but instead plays the video
>     natively. This is good, because there are a lot of documents out
>     there which use Flash based Rich Media. But there is absolutely no
>     need to create new documents with embedded Flash player applications,
>     it only takes space and isn't even used.

hm, i didn't know that ... is that kind of official? this 'replacement' 
trick?

>     Okular notably doesn't support plain Rich Media. The support is easy
>     to add, but my proposed patch [2] depends on changes to poppler. The
>     poppler developers want to take this chance to improve the Rich Media
>     representation [3] but I haven't gotten to that, yet.

good

>     Support similar to Okular's should be relatively easy to add to
>     Evince as well.

good

>     The 3D support is the same as with 3D art for Acrobat Reader. Weirdly
>     Foxit Reader doesn't support 3D files wrapped in Rich Media, although
>     there doesn't seem any good reason for it. 
> 
> All in all, of the five mechanisms 2 are deprecated and unfortunately no
> longer supported in the most used PDF viewer and other 2 mechanisms are
> needlessly complex and in reality limited. For example, while the
> multimedia mechanism supports JavaScript, (AFAIK) only Acrobat Reader
> supports that, this further limits the viewer support or available
> features, choose one.

a more philosophical: do "standards" or "formats" guarantee fuiture 
usage .. is investing time in some formats worth the effort

> The support for video and audio in Okular and Evince is based on
> Gstreamer. Explaining Gstreamer is tricky, but essentially it allows the
> viewers to play any media type as long as the right plugin is installed.
> These plugins are distributed in bundless and three of them cover all
> reasonable formats and more. But while the media file format support is
> great, these viewers don't really support PDF actions or JavaScript for
> more control over the media playback.
> 
> Acrobat and Foxit both use Windows Media Player for playing the video.
> Both support controls, but behave differently -- Acrobat displays the
> controls outside of the multimedia annotation, Foxit within...
> 
> As if it wasn't enough there is other trouble with playing multimedia in
> Acrobat Reader and Foxit Reader. They nag you to allow the media
> playback every time. You can select to trust the file once or from now
> on, but if somebody opens a foreign PDF with video, they aren't going to
> get smooth experience.
> 
> Another thing (but I don't remember well) is that there is a check box
> in Acrobat Reader, that allows the "legacy" Multimedia mechanism. I
> don't remember its state in an unaltered installation.

ah, these properties dialogs ... with changing defaults per version, 
yes, easy to forget

> After evaluating these mechanisms I came to conclusion, that a PDF
> writer today is best at:
> 
>   - Embedding video and audio using the "multimedia" ("renditions")
>     mechanism. It is supported in proprietary and open source viewers
>     alike. Customization and scripting / PDF actions is out of the
>     question, though.
> 
>   - Embedding 3D files using the "Rich Media" mechanism. While it is
>     essentially just a few differences in wrappers, it has real
>     advantages (data sources are files not streams, and multiple
>     JavaScript script files are supported), that I find nice enough for
>     the implementation and users alike.

maybe we need some javascript supported actions in viewers table

> Some sources for this topic are also the LaTeX centric [4] and [5]. I go
> into more details in the former. In the latter the "plain" Rich Media
> and "multimedia" ("renditions") mechanisms are suggested as solutions
> for the Flash media player approach.

but we have to stick to what we think will stay (kind of generic 
features of such things)

> And now for the future. What should ConTeXt do? On one hand all
> available mechanisms are flawed in one way or the other. On the other
> hand some users may still find the functionality useful. My suggestions
> is to either delete all the support for audio/video or:
> 
>   1) Delete the "Movie objects" implementation of figures. It is not
>      supported in viewers, where users expect it to [6].

it's actually an abstraction but we can map it onto somethign else

>   2) Delete all mentions of Flash. There is no reason to create new
>      documents with embedded Flash files, even though they may work in
>      some viewers. Plain Rich Media can be used instead, with hopefully
>      soon equal support [2].

indeed

>   3) The "externalrendering" mechanism (multimedia/renditions) can stay.
>      If the insertion of audio/video as "figures" is to stay, then I
>      suggest to use multimedia/renditions for it (in simplified form).
> 
> Note that the 3D support in ConTeXt is completely fine and works in
> Acrobat and Foxit.

do we actually have some simple test files for that? do we need a small 
set of files for media?

> The "externalrendering" part currently has three "bugs". Previous
> discussion at this list provides some context [7]. The following is
> currently "wrong":
> 
>   - Currently ConTeXt wraps a PDF file specification for embedded file
>     inside another file specification (i.e. embedded files don't work).
>   - As a result of "externalrendering" inheriting from \framed, the
>     PDF annotation late_lua whatsit is centered inside the frame and so
>     the annotation itself is offset by half its width to the right.
>   - ConTeXt doesn't explicitly allow the viewer to create temporary
>     files, hence the playback fails in Acrobat Reader.

hm, never seen that temp thing ... must be something recent (or maybe it 
is a preference)

> Hopefully the patch included below fixes all three. But note that while
> I love ConTeXt I don't know it well and may be terribly wrong. I also
> was aiming at a minimal diff for inclusion in this e-mail. This is a
> test file for this:

sure

>      \starttext
>      \setupinteraction[state=start]
>      
>      \useexternalrendering[myvideo][video/mp4][video.mp4][embed=yes]
>      \useexternalrendering[myvideo2][video/mp4][https://gitlab.com/agrahn/media9/uploads/c7e2ae944fbd711df4ad7bd58000f83a/nightseq1.mp4]
>      \useexternalrendering[myvideo3][video/mp4][video.mp4]
>      
>      \definerenderingwindow[myrenderingwindow][width=\textwidth, height=\textwidth]
>      
>      \noindent
>      \placerenderingwindow[myrenderingwindow][myvideo]
>      
>      \goto{START}[StartRendering{myvideo}]
>      \goto{STOP} [StopRendering{myvideo}]
>      \goto{PAUSE}[PauseRendering{myvideo}]
>      
>      \vfil\break\noindent
>      \placerenderingwindow[myrenderingwindow][myvideo2]
>      
>      \vfil\break\noindent
>      \placerenderingwindow[myrenderingwindow][myvideo3]
>      
>      \stoptext
> 
> All three file source types are demonstrated. Any "video.mp4" in the
> directory you compile in will do. (Works as expected in Okular on
> Linux.)

good. but not in mupdf viewers, right?

> This was a dump of knowledge that I gained from writing my thesis.
> Sadly its in Czech, but part of it is PDF code snippets and tables
> summarizing viewer support, that I can translate and provide if there is
> interest. But a large part of what I deem practical today is implemented
> and documented here:
> http://mirrors.ctan.org/macros/luatex/optex/pdfextra/pdfextra-doc.pdf.
> The source is probably hard to read, because of the "_" and "." prefixes
> in the control sequences, but those can be ignored. I posted some "real"
> documents in [3] and [5].

thanks for paying attention

> If more documents / snippets / explanations are needed I hope I can
> provide them.
> 
> Sadly, while working on this, I didn't have access to the PDF 2.0
> standard. My information mostly comes from the PDF 1.7 standard and
> publicly known information about PDF 2.0 - the Rich Media mechanism got
> included in PDF 2.0, but I am not sure in what extent did the Flash part
> get included. I also don't know if there really is anything new, but
> nothing suggests it. Regardless, viewer support isn't complete for
> something standardized over 20 years ago, I don't expect revolution in
> the PDF viewers, considering the price of the standard(s).

indeed. but wrt viewers, what puzzles me is that much of this rendering 
stuff is around in browsers so i guess it also has to do with lack of 
interest

> [1]: https://bugs.kde.org/show_bug.cgi?id=436709
> [2]: https://invent.kde.org/graphics/okular/-/merge_requests/426
> [3]: https://gitlab.freedesktop.org/poppler/poppler/-/merge_requests/855
> [4]: https://tex.stackexchange.com/questions/516029/media9-is-becoming-obsolete-dec-2020-any-alternatives-for-embedding-video-audio/516102
> [5]: https://gitlab.com/agrahn/media9/-/issues/9
> [6]: https://wiki.contextgarden.net/Command/externalfigure
> [7]: https://www.mail-archive.com/ntg-context@ntg.nl/msg88639.html

What actuially was also supported is

https://www.w3.org/TR/SMIL/

but somehow that became unreliable / went away .. when it first showed 
up it worked perfect and i always though that it would stay

Hans


-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________