JavaScript in multimedia PDF

JavaScript in multimedia PDF

[Pablo Rodriguez via ntg-context]

Dear list,

I have the following sample:

  \setupinteraction
      [state=start]

  \startJSpreamble varia used now
  this.pageNum = 0 ; // start at 0

  function GoToFirstSlide(label) {
  this.pageNum = 0 ;
  var rendition = this.media.getRendition(label) ;
  var player = app.media.openPlayer({
      rendition: rendition,
    });
  }

  function GoToLastSlide(label) {
    this.pageNum = this.numPages ;
  }

  function GoToNextSlide(label) {
      ++this.pageNum ;
  }

  function GoToPreviousSlide(label) {
      --this.pageNum ;
  }

  function SwitchFS() {
    if (app.fs.isFullScreen == true)
      app.fs.isFullScreen = false ;
    else
      app.fs.isFullScreen = true ;
  }
  \stopJSpreamble


  \setuppapersize[A9, landscape]

  \definefontfamily
      [mainface]
      [rm]
      [Latin Modern Sans]

  \definefontfamily
      [mainface]
      [ss]
      [Hans]

  \setupbodyfont
      [mainface, 25pt]

  \setupalign[middle]

  \setupfooter
      [style={\ss}]

  \def\SlideNavigationButtons{%
      \goto{A}[JS(GoToFirstSlide{mainsound})]%
      \goto{K}[JS(GoToPreviousSlide{mainsound})]%
      \goto{L}[JS(GoToNextSlide{mainsound})]%
      \goto{B}[JS(GoToLastSlide{mainsound})]
      \goto{{\ssb Q}}[JS(SwitchFS{})]%
      }

  \setupfootertexts[\SlideNavigationButtons]

  \starttext

    \definerenderingwindow[soundplace]
      [width=0pt, height=0pt]

    \useexternalrendering[mainsound][audio/mp3][sound.mp3][embed=yes]

    \placerenderingwindow[soundplace][mainsound]
  \dorecurse{25}{\null\page}
  \stoptext

I don’t know why I get the following message:

    TypeError: a.doc is undefined

It seems to be caused by "app.media.openPlayer", but the code is copied
from the documentation Adobe released early this year.

Does anyone know what is wrong here or what am I missing?

Many thanks for your help in advance,

Pablo
--
http://www.ousia.tk
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: JavaScript in multimedia PDF

[Michal Vlasák via ntg-context]

On Sun Aug 8, 2021 at 8:22 PM CEST, Pablo Rodriguez via ntg-context wrote:
> I don’t know why I get the following message:
>
> TypeError: a.doc is undefined
>
> It seems to be caused by "app.media.openPlayer", but the code is copied
> from the documentation Adobe released early this year.

Is there a newer document than "JavaScript for Acrobat API Reference"
(May 2015)?

Anyways, I thought that the culprit is that you only specify the
rendition (what to play) and not screen (where to play):


     function GoToFirstSlide(label) {
     this.pageNum = 0 ;
     var rendition = this.media.getRendition(label) ;
    +var screen = this.media.getAnnots({ nPage: 0})[0];
     var player = app.media.openPlayer({
         rendition: rendition,
    +    annot: screen,
       });
     }

But as it turns out, while the screen query succesfully returns an
object, the rendition query returns null.

The PDF 1.7 standard specifies that the name (/N) of Rendition PDF
object should be Unicode, but that doesn't seem to make it work either,
as does deleting the name of the media clip (which was the same).

(But I now get "Invalid arguments" error for the "app.media.openPlayer"
call, which is kind of expected.)

Can you confirm, that "rendition" is null on your side as well?

    console.println(rendition);
    console.println(screen);
    console.show();


Michal Vlasák
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: JavaScript in multimedia PDF

[Pablo Rodriguez via ntg-context]

On 8/8/21 11:00 PM, Michal Vlasák via ntg-context wrote:
> On Sun Aug 8, 2021 at 8:22 PM CEST, Pablo Rodriguez via ntg-context wrote:
>> I don’t know why I get the following message:
>>
>> TypeError: a.doc is undefined
>>
>> It seems to be caused by "app.media.openPlayer", but the code is copied
>> from the documentation Adobe released early this year.
>
> Is there a newer document than "JavaScript for Acrobat API Reference"
> (May 2015)?

Well, the link you provided
(https://mailman.ntg.nl/pipermail/ntg-context/2021/103023.html) is from
February 2021:

https://opensource.adobe.com/dc-acrobat-sdk-docs/acrobatsdk/pdfs/acrobatsdk_jsapiref.pdf

Here is the guide to developers (from the same date):

https://opensource.adobe.com/dc-acrobat-sdk-docs/acrobatsdk/pdfs/acrobatsdk_jsdevguide.pdf

> Anyways, I thought that the culprit is that you only specify the
> rendition (what to play) and not screen (where to play):
>
>
>      function GoToFirstSlide(label) {
>      this.pageNum = 0 ;
>      var rendition = this.media.getRendition(label) ;
>     +var screen = this.media.getAnnots({ nPage: 0})[0];
>      var player = app.media.openPlayer({
>          rendition: rendition,
>     +    annot: screen,
>        });
>      }
>
> But as it turns out, while the screen query succesfully returns an
> object, the rendition query returns null.
>
> The PDF 1.7 standard specifies that the name (/N) of Rendition PDF
> object should be Unicode, but that doesn't seem to make it work either,
> as does deleting the name of the media clip (which was the same).
>
> (But I now get "Invalid arguments" error for the "app.media.openPlayer"
> call, which is kind of expected.)
>
> Can you confirm, that "rendition" is null on your side as well?
>
>     console.println(rendition);
>     console.println(screen);
>     console.show();

I get exactly the same results: screen object, but null rendition.

I have no idea what it is needed here.

Many thanks for your help,

Pablo
--
http://www.ousia.tk
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: JavaScript in multimedia PDF

[Michal Vlasák via ntg-context]

[-- Attachment #1.1: Type: text/plain, Size: 6752 bytes --]

On Mon Aug 9, 2021 at 8:32 AM CEST, Pablo Rodriguez via ntg-context wrote:
> On 8/8/21 11:00 PM, Michal Vlasák via ntg-context wrote:
> > Is there a newer document than "JavaScript for Acrobat API Reference"
> > (May 2015)?
>
> Well, the link you provided
> (https://mailman.ntg.nl/pipermail/ntg-context/2021/103023.html) is from
> February 2021:
>
> https://opensource.adobe.com/dc-acrobat-sdk-docs/acrobatsdk/pdfs/acrobatsdk_jsapiref.pdf
>
> Here is the guide to developers (from the same date):
>
> https://opensource.adobe.com/dc-acrobat-sdk-docs/acrobatsdk/pdfs/acrobatsdk_jsdevguide.pdf

Thank you, I didn't realize that they updated the document after I
saved it. The (public) update also must have been more recently than
in February (I think the 2015 one was still up in May).

I wasn't every of the guide. It seems nice.

> I get exactly the same results: screen object, but null rendition.
>
> I have no idea what it is needed here.

Sorry, in the last mail I forgot to mention one idea I had. But now I
had time to try it, and in the end came up with three different ways to
get it working (at least the multimedia part, didn't test the rest).

It turns out, that Acrobat Reader ignores the names assigned to the
Renditions -- that's why it can't be found by name (this is not
complaint with the spec). But there is another mapping of Rendition
names and objects -- the "Renditions" name tree, that maps Unicode
encoded names of renditions to the respective objects. After generating
the name tree, the method "getRendition" works correctly. This is the
possibility number one -- generate the name tree.

Another possibility uses the fact, that the "openPlayer" method needs to
know which Rendition (what to play) and Screen annotation (where to
play) to target. While in ordinary JavaScript actions one needs to
specify them manually (because there is no context), in _Rendition_ PDF
actions (that can also execute JavaScript) there is a context -- the
action specifies the annotation and rendition at the PDF dictionary
level. So possibility number two is to use JavaScript Rendition actions.

Lastly, possibility number three. If one can't play the multimedia with
JavaScript, there still is a plain (no JavaScript) Rendition action,
that can do this -- "/OP 0" (StartRendering in ConTeXt). Also one can
chain several PDF actions one after the other. So if we first play the
Rendition through this ordinary Rendition action, then we can chain
arbitrary JavaScript after it. The order is important here, because the
second action doesn't start until first one is finished (which you may
or not desire, so feel free to swap the order).

All three possibilities work in Acrobat Reader 2021.005.20060. Depending
on the length of your audio, you may be interested in the fact, that
Acrobat Reader starts playing the audio from the start, if it didn't
finish it before. And also supports only embedded files.

Foxit Reader 11.0.1.49938 supports only the second and third way. It
also starts playing the same audio multiple times if started multiple
times (the same track plays multiple times over itself). All three file
types are supported, though I only check embedded and external files
(not URL files).

There are probably functions in the JS API to achieve better control and
consequently the same behaviour in both viewers, but I didn't look.

The first two ways are not currently possible in ConTeXt. The last one
should work, because ConTeXt should support chaining actions with
/Next, although I would have to check what is the user interface
(implementation is in lpdf.action).

I however, have a proof of concept in OpTeX, that uses the package from
my thesis (but I had to extend it for the Unicode encoded Rendition name
tree). To compile, you would need OpTeX and the updated "pdfextra.opm"
from
https://raw.githubusercontent.com/vlasakm/pdfextra/master/pdfextra.opm.
But the compiled PDF from the below example is attached.

With the proof of concept, and real world PDF document available, we
should probably get the functionality into ConTeXt. First way should
work if the name tree is added (I will look into that), but also needs a
better way to refer to the screen annotations, than what I am using
below (which maybe means more book keeping, have to check). The second
way would need a new executer and its shortcut, maybe something used
like this:

    \goto{...}[JSRendition{<rendition>, <javascript>}]

But I presume that more book keeping will be necessary.

The third way should hopefully already work.

Also, beware that both in my example, and in ConTeXt, multimedia
insertion produces whatsits, which especially when inserted with zero
dimensions like here for audio, may cause surprising interactions when
around normal typsetting content. Putting the whatsit insertion in a
page background is a way to keep it away. "back-swf.mkiv" suggests:

    \setupbackgrounds[page][background=resources]
    \setlayer[resources]{\placerenderingwindow[foo][foo]}



Michal Vlasák



% compile twice with "optex" and use pdfextra.opm from github
\fontfam[lm]
\load[pdfextra]
\hyperlinks\Blue\Blue

% The demo sound is randomly found, from Marianne Gagnon
% https://soundbible.com/1682-Robot-Blip.html

% initialization javascript (runs on document open)
\dljavascript[whatever]{%
function play_rendition(label) {
    this.pageNum = 0;
    var rendition = this.media.getRendition(label);
    var screen = this.media.getAnnots({ nPage: 0 })[0];
    console.println(label);
    console.println(rendition);
    console.println(screen);
    console.show();
    var player = app.media.openPlayer({
        rendition: rendition,
        annot: screen,
    });
}
}

% Rendition + Screen annotation (beware of the whatsit node!)
\newbox \backgroundbox
\setbox\backgroundbox=\hbox{\render[sound.mp3]{}}
\pgbackground={\box\backgroundbox}

% First possibility
\hlink[js:play_rendition("sound.mp3");]
  {Play sound with ordinary JavaScript action
   (function \code{GoToFirstSlide})}


% Second possibility
\sdef{_pdfextra_renditionaction:jsplay}{/JS (
    % argsDWIM ("Do what I mean") is what openPlayer
    % calls internally on its argument
    console.println(app.media.argsDWIM({}));
    console.show();
    var player = app.media.openPlayer({});
)}
\hlink[rendition::jsplay]
  {Play sound with (JavaScript) Rendition action (\code{rendition}
   and \code{annot} inherited from the action)}


% Third possibility
\hlink
  [rendition::play,
   js:app.alert("javascript code");]
  {Play sound with (Play) Rendition action chained before
   JavaScript action}

\bye

[-- Attachment #2: audio_renderings.pdf --]
[-- Type: application/pdf, Size: 16527 bytes --]

[-- Attachment #3: Type: text/plain, Size: 493 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: JavaScript in multimedia PDF

[Pablo Rodriguez via ntg-context]

On 8/10/21 11:52 PM, Michal Vlasák via ntg-context wrote:
> On Mon Aug 9, 2021 at 8:32 AM CEST, Pablo Rodriguez via ntg-context wrote:
> [...]
>> I get exactly the same results: screen object, but null rendition.
>>
>> I have no idea what it is needed here.
>
> Sorry, in the last mail I forgot to mention one idea I had. But now I
> had time to try it, and in the end came up with three different ways to
> get it working (at least the multimedia part, didn't test the rest).

Many thanks for your in-depth reply and explanation, Michal.

Since there are some related issues, I think it is better to discuss it
in private.

Many thanks for your help again,

Pablo
--
http://www.ousia.tk
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: JavaScript in multimedia PDF

[Michal Vlasák via ntg-context]

More context is in previous messages, but here is the patch for ConTeXt
to make scripting multimedia possible:

    --- a/tex/context/base/mkxl/lpdf-wid.lmt
    +++ b/tex/context/base/mkxl/lpdf-wid.lmt
    @@ -649,6 +649,7 @@ local function insertrenderingwindow(specification)
             Subtype = pdfconstant("Screen"),
             P       = pdfreference(pdfpagereference(page)),
             A       = a, -- needed in order to make the annotation clickable (i.e. don't bark)
    +        T       = pdfunicode(label), -- title (for JS)
             Border  = bs,
             C       = bc,
             AA      = actions,
    @@ -693,7 +694,7 @@ local function insertrendering(specification)
          --     }
          -- }
             local parameters = pdfdictionary {
    -            Type = pdfconstant(MediaPermissions),
    +            Type = pdfconstant("MediaPermissions"),
                 TF   = pdfstring("TEMPALWAYS"), -- TEMPNEVER TEMPEXTRACT TEMPACCESS TEMPALWAYS / needed for acrobat/wmp
             }
             local descriptor = pdfdictionary {
    @@ -723,7 +724,7 @@ local function insertrendering(specification)
             local rendition = pdfdictionary {
                 Type = pdfconstant("Rendition"),
                 S    = pdfconstant("MR"),
    -            N    = label,
    +            N    = pdfunicode(label),
                 C    = pdfreference(pdfflushobject(clip)),
             }
             mf[label] = pdfreference(pdfflushobject(rendition))
    @@ -761,6 +762,20 @@ function codeinjections.processrendering(label)
         end
     end
    
    +-- needed mapping for access from JS
    +local function flushrenderings()
    +    if next(mf) then
    +        local r = pdfarray()
    +        for label, reference in sortedhash(mf) do
    +            r[#r+1] = pdfunicode(label)
    +            r[#r+1] = reference -- already a reference
    +        end
    +        lpdf.addtonames("Renditions",pdfreference(pdfflushobject(pdfdictionary{ Names = r })))
    +    end
    +end
    +
    +lpdf.registerdocumentfinalizer(flushrenderings,"renderings")
    +
     function codeinjections.insertrenderingwindow(specification)
         local label = specification.label
         codeinjections.processrendering(label)

The patch is also be available here
https://github.com/vlasakm/context-mirror/commit/99f81beae0d13f1aecc20be187ada9eb88a2aa68
until it is applied (for full file download, which is available under
the three dots next to the file name).

The created Screen annotation will be given the title, that corresponds
to the second \placerenderingwindow parameter. So here:

    \definerenderingwindow[soundplace]
      [width=0pt, height=0pt]
    
    \useexternalrendering[mainsound][audio/mp3][sound.mp3][embed=yes]
    
    \placerenderingwindow[soundplace][mainsound]

both the screen annotation and rendition are available under the name
"mainsound". The name "soundplace" can be anything (even "mainsound"),
it doesn't propagate to the PDF file in any way.

And here is the patch required to make Pablo's example work:

    --- a/pablo.tex
    +++ b/pablo.tex
    @@ -7,8 +7,10 @@
     function GoToFirstSlide(label) {
     this.pageNum = 0 ;
     var rendition = this.media.getRendition(label) ;
    +var screen = this.media.getAnnot({nPage: 0, cAnnotTitle: label}) ;
     var player = app.media.openPlayer({
         rendition: rendition,
    +    annot: screen,
       });
     }

The second possibility I proposed previously is not as general, and
implementing it is not worth, until it is really needed (which I presume
won't happen).

Michal Vlasák
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: JavaScript in multimedia PDF

[Hans Hagen via ntg-context]

Hi

>      -            Type = pdfconstant(MediaPermissions),
>      +            Type = pdfconstant("MediaPermissions"),
That's an interesting one ... easilly goes unnoticed ... thanks

Hans

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: JavaScript in multimedia PDF

[Michal Vlasák via ntg-context]

On Thu Aug 12, 2021 at 12:55 AM CEST, Hans Hagen via ntg-context wrote:
> Hi
>
> >      -            Type = pdfconstant(MediaPermissions),
> >      +            Type = pdfconstant("MediaPermissions"),
> That's an interesting one ... easilly goes unnoticed ... thanks

To put credit where its due, this was Pablo's find.

Indeed very subtle, I didn't notice at all, and PDF viewers also
probably mostly ignore these "/Type"s.

Michal
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: JavaScript in multimedia PDF

[Pablo Rodriguez via ntg-context]

On 8/12/21 3:02 AM, Michal Vlasák via ntg-context wrote:
> On Thu Aug 12, 2021 at 12:55 AM CEST, Hans Hagen via ntg-context wrote:
>> Hi
>>
>>>      -            Type = pdfconstant(MediaPermissions),
>>>      +            Type = pdfconstant("MediaPermissions"),
>> That's an interesting one ... easilly goes unnoticed ... thanks
>
> To put credit where its due, this was Pablo's find.

Well, I only noticed that in the generated PDF code read:

  /Type /none

It was Michal who found the bug in the Lua code.

Many thanks for the work improving multimedia in ConTeXt,

Pablo
--
http://www.ousia.tk
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________