Files on a server

Files on a server

[Hans van der Meer]

[-- Attachment #1.1: Type: text/plain, Size: 859 bytes --]

I tried to determine the existence of a file on the internet. See the following macro call:

        % Test if file exists.
        \edef\theurl{\linkprotocol://\urlbase\xmlatt{#1}{link}\thesuffix}
        \doiffileelse
        	{\theurl}
        	{\verbose{HVDM-PEV-TEST}{file exists: \theurl}}
        	{\verbose{HVDM-PEV-TEST}{file does not exist: \theurl}}

It seems that \doiffileelse always results in true except for local files where nonexistence is correctly handled.

For example the following url definitely does not exist but nevertheless \doiffileelse reports its existence:
HVDM-PEV-TEST   > file exists: http://hansvandermeer.myqnapcloud.com/archive/denhaag/hga-dtb-1869-6040.pdf

Is it possible that the \doiffile macros are extended for files on the internet or a correponding set \doifurl is developed?

Hans van der Meer


[-- Attachment #1.2: Type: text/html, Size: 2292 bytes --]

[-- Attachment #2: Type: text/plain, Size: 492 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Files on a server

[Henri Menke]

On 05/16/2018 09:31 PM, Hans van der Meer wrote:
> I tried to determine the existence of a file on the internet. See the
> following macro call:
> 
>         % Test if file exists.
>         \edef\theurl{\linkprotocol://\urlbase\xmlatt{#1}{link}\thesuffix}
>         \doiffileelse
>         {\theurl}
>         {\verbose{HVDM-PEV-TEST}{file exists: \theurl}}
>         {\verbose{HVDM-PEV-TEST}{file does not exist: \theurl}}
> 
> It seems that \doiffileelse always results in true except for local
> files where nonexistence is correctly handled.
> 
> For example the following url definitely does not exist but nevertheless
> \doiffileelse reports its existence:
> HVDM-PEV-TEST   > file exists:
> http://hansvandermeer.myqnapcloud.com/archive/denhaag/hga-dtb-1869-6040.pdf
> 
> Is it possible that the \doiffile macros are extended for files on the
> internet or a correponding set \doifurl is developed?

I'm not speaking for the devs here, but I don't think that this is going
to be included.  It would require that all webpages send proper 404
codes and timeout after reasonable delay.  Also how would you handle
redirects?  Should the ConTeXt run be stalled because checking for a
files existence on the web is caught in a redirect loop?

> 
> Hans van der Meer
> 
> 
> 
> ___________________________________________________________________________________
> If your question is of interest to others as well, please add an entry to the Wiki!
> 
> maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
> webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
> archive  : https://bitbucket.org/phg/context-mirror/commits/
> wiki     : http://contextgarden.net
> ___________________________________________________________________________________
> 

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Files on a server

[Hans van der Meer]

[-- Attachment #1.1: Type: text/plain, Size: 3119 bytes --]

I would be satisfied when a returned 404 error code will be handled within a reasonable (configurable) time delay. As for redirection, there I would not mind if it is not included.
Restrictions like that are not a problem for me, because this is for building a pdf that accesses many internet located files. The check on existence is therefore especially meant for spotting missing or misnamed files.

Hans van der Meer

> On 16 May 2018, at 12:41, Henri Menke <henrimenke@gmail.com> wrote:
> 
> On 05/16/2018 09:31 PM, Hans van der Meer wrote:
>> I tried to determine the existence of a file on the internet. See the
>> following macro call:
>> 
>>         % Test if file exists.
>>         \edef\theurl{\linkprotocol://\urlbase\xmlatt{#1}{link}\thesuffix}
>>         \doiffileelse
>>         {\theurl}
>>         {\verbose{HVDM-PEV-TEST}{file exists: \theurl}}
>>         {\verbose{HVDM-PEV-TEST}{file does not exist: \theurl}}
>> 
>> It seems that \doiffileelse always results in true except for local
>> files where nonexistence is correctly handled.
>> 
>> For example the following url definitely does not exist but nevertheless
>> \doiffileelse reports its existence:
>> HVDM-PEV-TEST   > file exists:
>> http://hansvandermeer.myqnapcloud.com/archive/denhaag/hga-dtb-1869-6040.pdf
>> 
>> Is it possible that the \doiffile macros are extended for files on the
>> internet or a correponding set \doifurl is developed?
> 
> I'm not speaking for the devs here, but I don't think that this is going
> to be included.  It would require that all webpages send proper 404
> codes and timeout after reasonable delay.  Also how would you handle
> redirects?  Should the ConTeXt run be stalled because checking for a
> files existence on the web is caught in a redirect loop?
> 
>> 
>> Hans van der Meer
>> 
>> 
>> 
>> ___________________________________________________________________________________
>> If your question is of interest to others as well, please add an entry to the Wiki!
>> 
>> maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
>> webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
>> archive  : https://bitbucket.org/phg/context-mirror/commits/
>> wiki     : http://contextgarden.net
>> ___________________________________________________________________________________
>> 
> 
> ___________________________________________________________________________________
> If your question is of interest to others as well, please add an entry to the Wiki!
> 
> maillist : ntg-context@ntg.nl <mailto:ntg-context@ntg.nl> / http://www.ntg.nl/mailman/listinfo/ntg-context <http://www.ntg.nl/mailman/listinfo/ntg-context>
> webpage  : http://www.pragma-ade.nl <http://www.pragma-ade.nl/> / http://context.aanhet.net <http://context.aanhet.net/>
> archive  : https://bitbucket.org/phg/context-mirror/commits/ <https://bitbucket.org/phg/context-mirror/commits/>
> wiki     : http://contextgarden.net <http://contextgarden.net/>
> ___________________________________________________________________________________


[-- Attachment #1.2: Type: text/html, Size: 19262 bytes --]

[-- Attachment #2: Type: text/plain, Size: 492 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Files on a server

[Henri Menke]

On 05/16/2018 11:23 PM, Hans van der Meer wrote:
> I would be satisfied when a returned 404 error code will be handled
> within a reasonable (configurable) time delay. As for redirection, there
> I would not mind if it is not included.
> Restrictions like that are not a problem for me, because this is for
> building a pdf that accesses many internet located files. The check on
> existence is therefore especially meant for spotting missing or misnamed
> files.

You can use libcurl via ffi: https://curl.haxx.se/libcurl/

\startluacode
local ffi = require("ffi")

ffi.cdef[[
typedef struct Curl_easy CURL;
typedef int CURLcode;
typedef int CURLoption;
typedef int CURLINFO;
typedef size_t(*callback)(void *buffer, size_t size, size_t nmemb, void
*userp);

CURLcode curl_global_init(long flags);
CURL *curl_easy_init(void);
CURLcode curl_easy_setopt(CURL *curl, CURLoption option, ...);
CURLcode curl_easy_perform(CURL *curl);
CURLcode curl_easy_getinfo(CURL *curl, CURLINFO info, ...);
void curl_easy_cleanup(CURL *curl);
void curl_global_cleanup(void);
]]

-- magic numbers
local CURL_GLOBAL_DEFAULT = 3
local CURLOPT_URL = 10002
local CURLOPT_NOBODY = 44
local CURLOPT_HEADER = 42
local CURLOPT_WRITEFUNCTION = 20011
local CURLINFO_RESPONSE_CODE = 0x200000 + 2

local lcurl = ffi.load("curl")

local function discard(buffer, size, nmemb, userp)
    return size * nmemb
end

function check_status(url)
    lcurl.curl_global_init(CURL_GLOBAL_DEFAULT)

    local curl = lcurl.curl_easy_init()

    lcurl.curl_easy_setopt(curl, CURLOPT_URL, url)
    lcurl.curl_easy_setopt(curl, CURLOPT_NOBODY, ffi.cast("long",1))
    lcurl.curl_easy_setopt(curl, CURLOPT_HEADER, ffi.cast("long",1))
    lcurl.curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION,
ffi.cast("callback",discard));

    local res = lcurl.curl_easy_perform(curl)
    local c_http_code = ffi.new("long[1]")
    lcurl.curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, c_http_code);

    lcurl.curl_easy_cleanup(curl)
    lcurl.curl_global_cleanup()

    return tonumber(c_http_code[0])
end
\stopluacode

\starttext

\ctxlua{context(check_status("http://example.com/"))}

\ctxlua{context(check_status("https://example.com/"))}

\ctxlua{context(check_status("https://example.com/xyz"))}

\stoptext

> 
> Hans van der Meer
> 
>> On 16 May 2018, at 12:41, Henri Menke <henrimenke@gmail.com
>> <mailto:henrimenke@gmail.com>> wrote:
>>
>> On 05/16/2018 09:31 PM, Hans van der Meer wrote:
>>> I tried to determine the existence of a file on the internet. See the
>>> following macro call:
>>>
>>>         % Test if file exists.
>>>         \edef\theurl{\linkprotocol://\urlbase\xmlatt{#1}{link}\thesuffix}
>>>         \doiffileelse
>>>         {\theurl}
>>>         {\verbose{HVDM-PEV-TEST}{file exists: \theurl}}
>>>         {\verbose{HVDM-PEV-TEST}{file does not exist: \theurl}}
>>>
>>> It seems that \doiffileelse always results in true except for local
>>> files where nonexistence is correctly handled.
>>>
>>> For example the following url definitely does not exist but nevertheless
>>> \doiffileelse reports its existence:
>>> HVDM-PEV-TEST   > file exists:
>>> http://hansvandermeer.myqnapcloud.com/archive/denhaag/hga-dtb-1869-6040.pdf
>>>
>>> Is it possible that the \doiffile macros are extended for files on the
>>> internet or a correponding set \doifurl is developed?
>>
>> I'm not speaking for the devs here, but I don't think that this is going
>> to be included.  It would require that all webpages send proper 404
>> codes and timeout after reasonable delay.  Also how would you handle
>> redirects?  Should the ConTeXt run be stalled because checking for a
>> files existence on the web is caught in a redirect loop?
>>
>>>
>>> Hans van der Meer
>>>
>>>
>>>
>>> ___________________________________________________________________________________
>>> If your question is of interest to others as well, please add an
>>> entry to the Wiki!
>>>
>>> maillist : ntg-context@ntg.nl <mailto:ntg-context@ntg.nl> /
>>> http://www.ntg.nl/mailman/listinfo/ntg-context
>>> webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
>>> archive  : https://bitbucket.org/phg/context-mirror/commits/
>>> wiki     : http://contextgarden.net
>>> ___________________________________________________________________________________
>>>
>>
>> ___________________________________________________________________________________
>> If your question is of interest to others as well, please add an entry
>> to the Wiki!
>>
>> maillist : ntg-context@ntg.nl
>> <mailto:ntg-context@ntg.nl> / http://www.ntg.nl/mailman/listinfo/ntg-context
>> webpage  : http://www.pragma-ade.nl
>> <http://www.pragma-ade.nl/> / http://context.aanhet.net
>> <http://context.aanhet.net/>
>> archive  : https://bitbucket.org/phg/context-mirror/commits/
>> wiki     : http://contextgarden.net <http://contextgarden.net/>
>> ___________________________________________________________________________________
> 
> 
> 
> ___________________________________________________________________________________
> If your question is of interest to others as well, please add an entry to the Wiki!
> 
> maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
> webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
> archive  : https://bitbucket.org/phg/context-mirror/commits/
> wiki     : http://contextgarden.net
> ___________________________________________________________________________________
> 

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Files on a server

[Taco Hoekwater]

Or use luasocket, which is included in the luatex binary:

\startluacode
content, status, authinfo = socket.http.request{
    method = "HEAD",
    url = "http://hansvandermeer.myqnapcloud.com/archive/denhaag/hga-dtb-1869-6040.pdf” 
}

print (status)

\stopluacode

prints ‘404’ in this case.

Taco

> On 16 May 2018, at 13:45, Henri Menke <henrimenke@gmail.com> wrote:
> 
> On 05/16/2018 11:23 PM, Hans van der Meer wrote:
>> I would be satisfied when a returned 404 error code will be handled
>> within a reasonable (configurable) time delay. As for redirection, there
>> I would not mind if it is not included.
>> Restrictions like that are not a problem for me, because this is for
>> building a pdf that accesses many internet located files. The check on
>> existence is therefore especially meant for spotting missing or misnamed
>> files.
> 
> You can use libcurl via ffi: https://curl.haxx.se/libcurl/
> 
> \startluacode
> local ffi = require("ffi")
> 
> ffi.cdef[[
> typedef struct Curl_easy CURL;
> typedef int CURLcode;
> typedef int CURLoption;
> typedef int CURLINFO;
> typedef size_t(*callback)(void *buffer, size_t size, size_t nmemb, void
> *userp);
> 
> CURLcode curl_global_init(long flags);
> CURL *curl_easy_init(void);
> CURLcode curl_easy_setopt(CURL *curl, CURLoption option, ...);
> CURLcode curl_easy_perform(CURL *curl);
> CURLcode curl_easy_getinfo(CURL *curl, CURLINFO info, ...);
> void curl_easy_cleanup(CURL *curl);
> void curl_global_cleanup(void);
> ]]
> 
> -- magic numbers
> local CURL_GLOBAL_DEFAULT = 3
> local CURLOPT_URL = 10002
> local CURLOPT_NOBODY = 44
> local CURLOPT_HEADER = 42
> local CURLOPT_WRITEFUNCTION = 20011
> local CURLINFO_RESPONSE_CODE = 0x200000 + 2
> 
> local lcurl = ffi.load("curl")
> 
> local function discard(buffer, size, nmemb, userp)
>    return size * nmemb
> end
> 
> function check_status(url)
>    lcurl.curl_global_init(CURL_GLOBAL_DEFAULT)
> 
>    local curl = lcurl.curl_easy_init()
> 
>    lcurl.curl_easy_setopt(curl, CURLOPT_URL, url)
>    lcurl.curl_easy_setopt(curl, CURLOPT_NOBODY, ffi.cast("long",1))
>    lcurl.curl_easy_setopt(curl, CURLOPT_HEADER, ffi.cast("long",1))
>    lcurl.curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION,
> ffi.cast("callback",discard));
> 
>    local res = lcurl.curl_easy_perform(curl)
>    local c_http_code = ffi.new("long[1]")
>    lcurl.curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, c_http_code);
> 
>    lcurl.curl_easy_cleanup(curl)
>    lcurl.curl_global_cleanup()
> 
>    return tonumber(c_http_code[0])
> end
> \stopluacode
> 
> \starttext
> 
> \ctxlua{context(check_status("http://example.com/"))}
> 
> \ctxlua{context(check_status("https://example.com/"))}
> 
> \ctxlua{context(check_status("https://example.com/xyz"))}
> 
> \stoptext
> 
>> 
>> Hans van der Meer
>> 
>>> On 16 May 2018, at 12:41, Henri Menke <henrimenke@gmail.com
>>> <mailto:henrimenke@gmail.com>> wrote:
>>> 
>>> On 05/16/2018 09:31 PM, Hans van der Meer wrote:
>>>> I tried to determine the existence of a file on the internet. See the
>>>> following macro call:
>>>> 
>>>>         % Test if file exists.
>>>>         \edef\theurl{\linkprotocol://\urlbase\xmlatt{#1}{link}\thesuffix}
>>>>         \doiffileelse
>>>>         {\theurl}
>>>>         {\verbose{HVDM-PEV-TEST}{file exists: \theurl}}
>>>>         {\verbose{HVDM-PEV-TEST}{file does not exist: \theurl}}
>>>> 
>>>> It seems that \doiffileelse always results in true except for local
>>>> files where nonexistence is correctly handled.
>>>> 
>>>> For example the following url definitely does not exist but nevertheless
>>>> \doiffileelse reports its existence:
>>>> HVDM-PEV-TEST   > file exists:
>>>> http://hansvandermeer.myqnapcloud.com/archive/denhaag/hga-dtb-1869-6040.pdf
>>>> 
>>>> Is it possible that the \doiffile macros are extended for files on the
>>>> internet or a correponding set \doifurl is developed?
>>> 
>>> I'm not speaking for the devs here, but I don't think that this is going
>>> to be included.  It would require that all webpages send proper 404
>>> codes and timeout after reasonable delay.  Also how would you handle
>>> redirects?  Should the ConTeXt run be stalled because checking for a
>>> files existence on the web is caught in a redirect loop?
>>> 
>>>> 
>>>> Hans van der Meer
>>>> 
>>>> 
>>>> 
>>>> ___________________________________________________________________________________
>>>> If your question is of interest to others as well, please add an
>>>> entry to the Wiki!
>>>> 
>>>> maillist : ntg-context@ntg.nl <mailto:ntg-context@ntg.nl> /
>>>> http://www.ntg.nl/mailman/listinfo/ntg-context
>>>> webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
>>>> archive  : https://bitbucket.org/phg/context-mirror/commits/
>>>> wiki     : http://contextgarden.net
>>>> ___________________________________________________________________________________
>>>> 
>>> 
>>> ___________________________________________________________________________________
>>> If your question is of interest to others as well, please add an entry
>>> to the Wiki!
>>> 
>>> maillist : ntg-context@ntg.nl
>>> <mailto:ntg-context@ntg.nl> / http://www.ntg.nl/mailman/listinfo/ntg-context
>>> webpage  : http://www.pragma-ade.nl
>>> <http://www.pragma-ade.nl/> / http://context.aanhet.net
>>> <http://context.aanhet.net/>
>>> archive  : https://bitbucket.org/phg/context-mirror/commits/
>>> wiki     : http://contextgarden.net <http://contextgarden.net/>
>>> ___________________________________________________________________________________
>> 
>> 
>> 
>> ___________________________________________________________________________________
>> If your question is of interest to others as well, please add an entry to the Wiki!
>> 
>> maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
>> webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
>> archive  : https://bitbucket.org/phg/context-mirror/commits/
>> wiki     : http://contextgarden.net
>> ___________________________________________________________________________________
>> 
> 
> ___________________________________________________________________________________
> If your question is of interest to others as well, please add an entry to the Wiki!
> 
> maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
> webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
> archive  : https://bitbucket.org/phg/context-mirror/commits/
> wiki     : http://contextgarden.net
> ___________________________________________________________________________________

Taco Hoekwater
Elvenkind BV




___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Files on a server

[Hans van der Meer]

[-- Attachment #1.1: Type: text/plain, Size: 9238 bytes --]

Beste Taco,

Ik probeer jouw oplossing maar er gebeurt nog iets raars. Ik krijg inderdaad 200 resp. 404 terug bij bestaande niet-bestaande Uri’s. Maar ik krijg wel een error die aan het eind van de run gegenereerd wordt. Heb jij enig idee waar ik dat zou moeten zoeken? Als ik de call naar \verbose uitschakel komt er geen foutmelding. Blijft er na de socket call iets achter wat nog opgeruimd had moeten worden?

Hier een deel van de context-log en de code fragmenten in kwestie. <takepersons> is de root node van het bewerkte bestand.

HVDM-PEV-TEST   > return code: 200
HVDM-PEV        > number of events processed: 1
HVDM-PEV        > ============= end of takepersons =============
backend         > xmp > using file '/Users/hansm/TeX/context-45/tex/texmf-context/tex/context/base/mkiv/lpdf-pdx.xml'
pages           > flushing realpage 1, userpage 1, subpage 1
close source    > level 2, order 13, name '/Users/hansm/Documents/Genealogie/familie-event.tex'
close source    > level 1, order 13, name '/Users/hansm/TeX/context-45/tex/texmf-context/tex/context/base/mkiv/cont-yes.mkiv'

lua error       > lua error on line 0 in file :

...-45/tex/texmf-context/tex/context/base/mkiv/core-uti.lua:92: attempt to index a number value (global 'status')

<empty file>


? 
Process aborted

Code in tex:
	\def\URIReturnCode#1{\ctxlua{tex.print(hvdm.urireturncode("#1"))}} 
	\verbose{HVDM-PEV-TEST}{return code: \URIReturnCode{\theurl}}
Code in lua:
	hvdm.urireturncode = function (theuri)
		content, status, authinfo = socket.http.request{
			method = "HEAD",
			url = theuri,
			}
		return tostring(status)
	end

Groet Hans



> On 16 May 2018, at 14:23, Taco Hoekwater <taco@elvenkind.com> wrote:
> 
> 
> Or use luasocket, which is included in the luatex binary:
> 
> \startluacode
> content, status, authinfo = socket.http.request{
>    method = "HEAD",
>    url = "http://hansvandermeer.myqnapcloud.com/archive/denhaag/hga-dtb-1869-6040.pdf” 
> }
> 
> print (status)
> 
> \stopluacode
> 
> prints ‘404’ in this case.
> 
> Taco
> 
>> On 16 May 2018, at 13:45, Henri Menke <henrimenke@gmail.com> wrote:
>> 
>> On 05/16/2018 11:23 PM, Hans van der Meer wrote:
>>> I would be satisfied when a returned 404 error code will be handled
>>> within a reasonable (configurable) time delay. As for redirection, there
>>> I would not mind if it is not included.
>>> Restrictions like that are not a problem for me, because this is for
>>> building a pdf that accesses many internet located files. The check on
>>> existence is therefore especially meant for spotting missing or misnamed
>>> files.
>> 
>> You can use libcurl via ffi: https://curl.haxx.se/libcurl/
>> 
>> \startluacode
>> local ffi = require("ffi")
>> 
>> ffi.cdef[[
>> typedef struct Curl_easy CURL;
>> typedef int CURLcode;
>> typedef int CURLoption;
>> typedef int CURLINFO;
>> typedef size_t(*callback)(void *buffer, size_t size, size_t nmemb, void
>> *userp);
>> 
>> CURLcode curl_global_init(long flags);
>> CURL *curl_easy_init(void);
>> CURLcode curl_easy_setopt(CURL *curl, CURLoption option, ...);
>> CURLcode curl_easy_perform(CURL *curl);
>> CURLcode curl_easy_getinfo(CURL *curl, CURLINFO info, ...);
>> void curl_easy_cleanup(CURL *curl);
>> void curl_global_cleanup(void);
>> ]]
>> 
>> -- magic numbers
>> local CURL_GLOBAL_DEFAULT = 3
>> local CURLOPT_URL = 10002
>> local CURLOPT_NOBODY = 44
>> local CURLOPT_HEADER = 42
>> local CURLOPT_WRITEFUNCTION = 20011
>> local CURLINFO_RESPONSE_CODE = 0x200000 + 2
>> 
>> local lcurl = ffi.load("curl")
>> 
>> local function discard(buffer, size, nmemb, userp)
>>   return size * nmemb
>> end
>> 
>> function check_status(url)
>>   lcurl.curl_global_init(CURL_GLOBAL_DEFAULT)
>> 
>>   local curl = lcurl.curl_easy_init()
>> 
>>   lcurl.curl_easy_setopt(curl, CURLOPT_URL, url)
>>   lcurl.curl_easy_setopt(curl, CURLOPT_NOBODY, ffi.cast("long",1))
>>   lcurl.curl_easy_setopt(curl, CURLOPT_HEADER, ffi.cast("long",1))
>>   lcurl.curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION,
>> ffi.cast("callback",discard));
>> 
>>   local res = lcurl.curl_easy_perform(curl)
>>   local c_http_code = ffi.new("long[1]")
>>   lcurl.curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, c_http_code);
>> 
>>   lcurl.curl_easy_cleanup(curl)
>>   lcurl.curl_global_cleanup()
>> 
>>   return tonumber(c_http_code[0])
>> end
>> \stopluacode
>> 
>> \starttext
>> 
>> \ctxlua{context(check_status("http://example.com/"))}
>> 
>> \ctxlua{context(check_status("https://example.com/"))}
>> 
>> \ctxlua{context(check_status("https://example.com/xyz"))}
>> 
>> \stoptext
>> 
>>> 
>>> Hans van der Meer
>>> 
>>>> On 16 May 2018, at 12:41, Henri Menke <henrimenke@gmail.com
>>>> <mailto:henrimenke@gmail.com>> wrote:
>>>> 
>>>> On 05/16/2018 09:31 PM, Hans van der Meer wrote:
>>>>> I tried to determine the existence of a file on the internet. See the
>>>>> following macro call:
>>>>> 
>>>>>        % Test if file exists.
>>>>>        \edef\theurl{\linkprotocol://\urlbase\xmlatt{#1}{link}\thesuffix}
>>>>>        \doiffileelse
>>>>>        {\theurl}
>>>>>        {\verbose{HVDM-PEV-TEST}{file exists: \theurl}}
>>>>>        {\verbose{HVDM-PEV-TEST}{file does not exist: \theurl}}
>>>>> 
>>>>> It seems that \doiffileelse always results in true except for local
>>>>> files where nonexistence is correctly handled.
>>>>> 
>>>>> For example the following url definitely does not exist but nevertheless
>>>>> \doiffileelse reports its existence:
>>>>> HVDM-PEV-TEST   > file exists:
>>>>> http://hansvandermeer.myqnapcloud.com/archive/denhaag/hga-dtb-1869-6040.pdf
>>>>> 
>>>>> Is it possible that the \doiffile macros are extended for files on the
>>>>> internet or a correponding set \doifurl is developed?
>>>> 
>>>> I'm not speaking for the devs here, but I don't think that this is going
>>>> to be included.  It would require that all webpages send proper 404
>>>> codes and timeout after reasonable delay.  Also how would you handle
>>>> redirects?  Should the ConTeXt run be stalled because checking for a
>>>> files existence on the web is caught in a redirect loop?
>>>> 
>>>>> 
>>>>> Hans van der Meer
>>>>> 
>>>>> 
>>>>> 
>>>>> ___________________________________________________________________________________
>>>>> If your question is of interest to others as well, please add an
>>>>> entry to the Wiki!
>>>>> 
>>>>> maillist : ntg-context@ntg.nl <mailto:ntg-context@ntg.nl> /
>>>>> http://www.ntg.nl/mailman/listinfo/ntg-context
>>>>> webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
>>>>> archive  : https://bitbucket.org/phg/context-mirror/commits/
>>>>> wiki     : http://contextgarden.net
>>>>> ___________________________________________________________________________________
>>>>> 
>>>> 
>>>> ___________________________________________________________________________________
>>>> If your question is of interest to others as well, please add an entry
>>>> to the Wiki!
>>>> 
>>>> maillist : ntg-context@ntg.nl
>>>> <mailto:ntg-context@ntg.nl> / http://www.ntg.nl/mailman/listinfo/ntg-context
>>>> webpage  : http://www.pragma-ade.nl
>>>> <http://www.pragma-ade.nl/> / http://context.aanhet.net
>>>> <http://context.aanhet.net/>
>>>> archive  : https://bitbucket.org/phg/context-mirror/commits/
>>>> wiki     : http://contextgarden.net <http://contextgarden.net/>
>>>> ___________________________________________________________________________________
>>> 
>>> 
>>> 
>>> ___________________________________________________________________________________
>>> If your question is of interest to others as well, please add an entry to the Wiki!
>>> 
>>> maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
>>> webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
>>> archive  : https://bitbucket.org/phg/context-mirror/commits/
>>> wiki     : http://contextgarden.net
>>> ___________________________________________________________________________________
>>> 
>> 
>> ___________________________________________________________________________________
>> If your question is of interest to others as well, please add an entry to the Wiki!
>> 
>> maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
>> webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
>> archive  : https://bitbucket.org/phg/context-mirror/commits/
>> wiki     : http://contextgarden.net
>> ___________________________________________________________________________________
> 
> Taco Hoekwater
> Elvenkind BV
> 
> 
> 
> 
> ___________________________________________________________________________________
> If your question is of interest to others as well, please add an entry to the Wiki!
> 
> maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
> webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
> archive  : https://bitbucket.org/phg/context-mirror/commits/
> wiki     : http://contextgarden.net
> ___________________________________________________________________________________


[-- Attachment #1.2: Type: text/html, Size: 18102 bytes --]

[-- Attachment #2: Type: text/plain, Size: 492 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Files on a server

[Hans Hagen]

On 5/16/2018 2:58 PM, Hans van der Meer wrote:
> Beste Taco,
> 
> Ik probeer jouw oplossing maar er gebeurt nog iets raars. Ik krijg 
> inderdaad 200 resp. 404 terug bij bestaande niet-bestaande Uri’s. Maar 
> ik krijg wel een error die aan het eind van de run gegenereerd wordt. 
> Heb jij enig idee waar ik dat zou moeten zoeken? Als ik de call naar 
> \verbose uitschakel komt er geen foutmelding. Blijft er na de socket 
> call iets achter wat nog opgeruimd had moeten worden?


how about adding local ...

\startluacode
local content, status, authinfo = socket.http.request {
     method = "HEAD",
     url    = 
"http://hansvandermeer.myqnapcloud.com/archive/denhaag/hga-dtb-1869-6040.pdf",
}

print (status)

\stopluacode


> Hier een deel van de context-log en de code fragmenten in kwestie. 
> <takepersons> is de root node van het bewerkte bestand.
> 
> HVDM-PEV-TEST   > return code: 200
> HVDM-PEV        > number of events processed: 1
> HVDM-PEV        > ============= end of takepersons =============
> backend         > xmp > using file 
> '/Users/hansm/TeX/context-45/tex/texmf-context/tex/context/base/mkiv/lpdf-pdx.xml'
> pages           > flushing realpage 1, userpage 1, subpage 1
> close source    > level 2, order 13, name 
> '/Users/hansm/Documents/Genealogie/familie-event.tex'
> close source    > level 1, order 13, name 
> '/Users/hansm/TeX/context-45/tex/texmf-context/tex/context/base/mkiv/cont-yes.mkiv'
> 
> lua error       > lua error on line 0 in file :
> 
> ...-45/tex/texmf-context/tex/context/base/mkiv/core-uti.lua:92: attempt 
> to index a number value (global 'status')
> 
> <empty file>
> 
> 
> ?
> Process aborted
> 
> Code in tex:
> \def\URIReturnCode#1{\ctxlua{tex.print(hvdm.urireturncode("#1"))}}
> \verbose{HVDM-PEV-TEST}{return code: \URIReturnCode{\theurl}}
> Code in lua:
> hvdm.urireturncode = function (theuri)
> content, status, authinfo = socket.http.request{
> method = "HEAD",
> url = theuri,
> }
> return tostring(status)
> end
> 
> Groet Hans
> 
> 
> 
>> On 16 May 2018, at 14:23, Taco Hoekwater <taco@elvenkind.com 
>> <mailto:taco@elvenkind.com>> wrote:
>>
>>
>> Or use luasocket, which is included in the luatex binary:
>>
>> \startluacode
>> content, status, authinfo = socket.http.request{
>>    method = "HEAD",
>>    url = 
>> "http://hansvandermeer.myqnapcloud.com/archive/denhaag/hga-dtb-1869-6040.pdf” 
>>
>> }
>>
>> print (status)
>>
>> \stopluacode
>>
>> prints ‘404’ in this case.
>>
>> Taco
>>
>>> On 16 May 2018, at 13:45, Henri Menke <henrimenke@gmail.com 
>>> <mailto:henrimenke@gmail.com>> wrote:
>>>
>>> On 05/16/2018 11:23 PM, Hans van der Meer wrote:
>>>> I would be satisfied when a returned 404 error code will be handled
>>>> within a reasonable (configurable) time delay. As for redirection, there
>>>> I would not mind if it is not included.
>>>> Restrictions like that are not a problem for me, because this is for
>>>> building a pdf that accesses many internet located files. The check on
>>>> existence is therefore especially meant for spotting missing or misnamed
>>>> files.
>>>
>>> You can use libcurl via ffi: https://curl.haxx.se/libcurl/
>>>
>>> \startluacode
>>> local ffi = require("ffi")
>>>
>>> ffi.cdef[[
>>> typedef struct Curl_easy CURL;
>>> typedef int CURLcode;
>>> typedef int CURLoption;
>>> typedef int CURLINFO;
>>> typedef size_t(*callback)(void *buffer, size_t size, size_t nmemb, void
>>> *userp);
>>>
>>> CURLcode curl_global_init(long flags);
>>> CURL *curl_easy_init(void);
>>> CURLcode curl_easy_setopt(CURL *curl, CURLoption option, ...);
>>> CURLcode curl_easy_perform(CURL *curl);
>>> CURLcode curl_easy_getinfo(CURL *curl, CURLINFO info, ...);
>>> void curl_easy_cleanup(CURL *curl);
>>> void curl_global_cleanup(void);
>>> ]]
>>>
>>> -- magic numbers
>>> local CURL_GLOBAL_DEFAULT = 3
>>> local CURLOPT_URL = 10002
>>> local CURLOPT_NOBODY = 44
>>> local CURLOPT_HEADER = 42
>>> local CURLOPT_WRITEFUNCTION = 20011
>>> local CURLINFO_RESPONSE_CODE = 0x200000 + 2
>>>
>>> local lcurl = ffi.load("curl")
>>>
>>> local function discard(buffer, size, nmemb, userp)
>>>   return size * nmemb
>>> end
>>>
>>> function check_status(url)
>>>   lcurl.curl_global_init(CURL_GLOBAL_DEFAULT)
>>>
>>>   local curl = lcurl.curl_easy_init()
>>>
>>>   lcurl.curl_easy_setopt(curl, CURLOPT_URL, url)
>>>   lcurl.curl_easy_setopt(curl, CURLOPT_NOBODY, ffi.cast("long",1))
>>>   lcurl.curl_easy_setopt(curl, CURLOPT_HEADER, ffi.cast("long",1))
>>>   lcurl.curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION,
>>> ffi.cast("callback",discard));
>>>
>>>   local res = lcurl.curl_easy_perform(curl)
>>>   local c_http_code = ffi.new("long[1]")
>>>   lcurl.curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, c_http_code);
>>>
>>>   lcurl.curl_easy_cleanup(curl)
>>>   lcurl.curl_global_cleanup()
>>>
>>>   return tonumber(c_http_code[0])
>>> end
>>> \stopluacode
>>>
>>> \starttext
>>>
>>> \ctxlua{context(check_status("http://example.com/"))}
>>>
>>> \ctxlua{context(check_status("https://example.com/"))}
>>>
>>> \ctxlua{context(check_status("https://example.com/xyz"))}
>>>
>>> \stoptext
>>>
>>>>
>>>> Hans van der Meer
>>>>
>>>>> On 16 May 2018, at 12:41, Henri Menke <henrimenke@gmail.com 
>>>>> <mailto:henrimenke@gmail.com>
>>>>> <mailto:henrimenke@gmail.com>> wrote:
>>>>>
>>>>> On 05/16/2018 09:31 PM, Hans van der Meer wrote:
>>>>>> I tried to determine the existence of a file on the internet. See the
>>>>>> following macro call:
>>>>>>
>>>>>>        % Test if file exists.
>>>>>>        \edef\theurl{\linkprotocol://\urlbase\xmlatt{#1}{link}\thesuffix}
>>>>>>        \doiffileelse
>>>>>>        {\theurl}
>>>>>>        {\verbose{HVDM-PEV-TEST}{file exists: \theurl}}
>>>>>>        {\verbose{HVDM-PEV-TEST}{file does not exist: \theurl}}
>>>>>>
>>>>>> It seems that \doiffileelse always results in true except for local
>>>>>> files where nonexistence is correctly handled.
>>>>>>
>>>>>> For example the following url definitely does not exist but 
>>>>>> nevertheless
>>>>>> \doiffileelse reports its existence:
>>>>>> HVDM-PEV-TEST   > file exists:
>>>>>> http://hansvandermeer.myqnapcloud.com/archive/denhaag/hga-dtb-1869-6040.pdf
>>>>>>
>>>>>> Is it possible that the \doiffile macros are extended for files on the
>>>>>> internet or a correponding set \doifurl is developed?
>>>>>
>>>>> I'm not speaking for the devs here, but I don't think that this is 
>>>>> going
>>>>> to be included.  It would require that all webpages send proper 404
>>>>> codes and timeout after reasonable delay.  Also how would you handle
>>>>> redirects?  Should the ConTeXt run be stalled because checking for a
>>>>> files existence on the web is caught in a redirect loop?
>>>>>
>>>>>>
>>>>>> Hans van der Meer
>>>>>>
>>>>>>
>>>>>>
>>>>>> ___________________________________________________________________________________
>>>>>> If your question is of interest to others as well, please add an
>>>>>> entry to the Wiki!
>>>>>>
>>>>>> maillist : ntg-context@ntg.nl <mailto:ntg-context@ntg.nl> 
>>>>>> <mailto:ntg-context@ntg.nl> /
>>>>>> http://www.ntg.nl/mailman/listinfo/ntg-context
>>>>>> webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
>>>>>> archive  : https://bitbucket.org/phg/context-mirror/commits/
>>>>>> wiki     : http://contextgarden.net
>>>>>> ___________________________________________________________________________________
>>>>>>
>>>>>
>>>>> ___________________________________________________________________________________
>>>>> If your question is of interest to others as well, please add an entry
>>>>> to the Wiki!
>>>>>
>>>>> maillist : ntg-context@ntg.nl <mailto:ntg-context@ntg.nl>
>>>>> <mailto:ntg-context@ntg.nl> / 
>>>>> http://www.ntg.nl/mailman/listinfo/ntg-context
>>>>> webpage  : http://www.pragma-ade.nl
>>>>> <http://www.pragma-ade.nl/> / http://context.aanhet.net
>>>>> <http://context.aanhet.net/>
>>>>> archive  : https://bitbucket.org/phg/context-mirror/commits/
>>>>> wiki     : http://contextgarden.net <http://contextgarden.net/>
>>>>> ___________________________________________________________________________________
>>>>
>>>>
>>>>
>>>> ___________________________________________________________________________________
>>>> If your question is of interest to others as well, please add an 
>>>> entry to the Wiki!
>>>>
>>>> maillist : ntg-context@ntg.nl <mailto:ntg-context@ntg.nl> / 
>>>> http://www.ntg.nl/mailman/listinfo/ntg-context
>>>> webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
>>>> archive  : https://bitbucket.org/phg/context-mirror/commits/
>>>> wiki     : http://contextgarden.net
>>>> ___________________________________________________________________________________
>>>>
>>>
>>> ___________________________________________________________________________________
>>> If your question is of interest to others as well, please add an 
>>> entry to the Wiki!
>>>
>>> maillist : ntg-context@ntg.nl <mailto:ntg-context@ntg.nl> / 
>>> http://www.ntg.nl/mailman/listinfo/ntg-context
>>> webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
>>> archive  : https://bitbucket.org/phg/context-mirror/commits/
>>> wiki     : http://contextgarden.net
>>> ___________________________________________________________________________________
>>
>> Taco Hoekwater
>> Elvenkind BV
>>
>>
>>
>>
>> ___________________________________________________________________________________
>> If your question is of interest to others as well, please add an entry 
>> to the Wiki!
>>
>> maillist : ntg-context@ntg.nl <mailto:ntg-context@ntg.nl> / 
>> http://www.ntg.nl/mailman/listinfo/ntg-context
>> webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
>> archive  : https://bitbucket.org/phg/context-mirror/commits/
>> wiki     : http://contextgarden.net
>> ___________________________________________________________________________________
> 
> 
> 
> ___________________________________________________________________________________
> If your question is of interest to others as well, please add an entry to the Wiki!
> 
> maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
> webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
> archive  : https://bitbucket.org/phg/context-mirror/commits/
> wiki     : http://contextgarden.net
> ___________________________________________________________________________________
> 


-- 

-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Files on a server

[Taco Hoekwater]

> On 16 May 2018, at 15:41, Hans Hagen <j.hagen@xs4all.nl> wrote:
> 
> On 5/16/2018 2:58 PM, Hans van der Meer wrote:
>> Beste Taco,
>> Ik probeer jouw oplossing maar er gebeurt nog iets raars. Ik krijg inderdaad 200 resp. 404 terug bij bestaande niet-bestaande Uri’s. Maar ik krijg wel een error die aan het eind van de run gegenereerd wordt. Heb jij enig idee waar ik dat zou moeten zoeken? Als ik de call naar \verbose uitschakel komt er geen foutmelding. Blijft er na de socket call iets achter wat nog opgeruimd had moeten worden?
> 
> 
> how about adding local …

yeah.

Taco
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Files on a server

[Ulrike Fischer]

Am Wed, 16 May 2018 14:23:42 +0200 schrieb Taco Hoekwater:

> Or use luasocket, which is included in the luatex binary:

But I'm right that this works only with http and not with https?

 
-- 
Ulrike Fischer 
http://www.troubleshooting-tex.de/

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Files on a server

[Taco Hoekwater]

> On 17 May 2018, at 09:32, Ulrike Fischer <news3@nililand.de> wrote:
> 
> Am Wed, 16 May 2018 14:23:42 +0200 schrieb Taco Hoekwater:
> 
>> Or use luasocket, which is included in the luatex binary:
> 
> But I'm right that this works only with http and not with https?

Works ok for me. Did you test?

Taco

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Files on a server

[Hans van der Meer]

I too have the impression it is working for both.

Hans van der Meer

> On 17 May 2018, at 09:39, Taco Hoekwater <taco@elvenkind.com> wrote:
> 
> 
> 
>> On 17 May 2018, at 09:32, Ulrike Fischer <news3@nililand.de> wrote:
>> 
>> Am Wed, 16 May 2018 14:23:42 +0200 schrieb Taco Hoekwater:
>> 
>>> Or use luasocket, which is included in the luatex binary:
>> 
>> But I'm right that this works only with http and not with https?
> 
> Works ok for me. Did you test?
> 
> Taco
> 
> ___________________________________________________________________________________
> If your question is of interest to others as well, please add an entry to the Wiki!
> 
> maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
> webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
> archive  : https://bitbucket.org/phg/context-mirror/commits/
> wiki     : http://contextgarden.net
> ___________________________________________________________________________________

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Files on a server

[Taco Hoekwater]

> On 17 May 2018, at 09:39, Taco Hoekwater <taco@elvenkind.com> wrote:
> 
> 
> 
>> On 17 May 2018, at 09:32, Ulrike Fischer <news3@nililand.de> wrote:
>> 
>> Am Wed, 16 May 2018 14:23:42 +0200 schrieb Taco Hoekwater:
>> 
>>> Or use luasocket, which is included in the luatex binary:
>> 
>> But I'm right that this works only with http and not with https?
> 
> Works ok for me. Did you test?

Oh, sorry. It seems it is sneakily rewriting the https:// to http://,
and so does not _actually_ work. Which makes sense, now that I think
about it.  Doing https would need openssl support, which is unlikely to 
ever be built into luatex. 

Sorry for being misleading,

Taco



___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Files on a server

[Ulrike Fischer]

Am Thu, 17 May 2018 09:39:47 +0200 schrieb Taco Hoekwater:

>>> Or use luasocket, which is included in the luatex binary:

>> But I'm right that this works only with http and not with https?

> Works ok for me. Did you test?

I tried a few links and always got 301 back regardless if they exist
or not and thought due to the discussion two month ago on the luatex
list about https content that it doesn't work at all. 

I know found https://httpbin.org/ and with it the tests worked:
https://httpbin.org/html gave 200 and https://httpbin.org/htmlxxxx
gave 404.

But the answers from "real" websites are so varied and so often
don't say the truth that I would't describe it as "works ok" ;-). 

-- 
Ulrike Fischer 
http://www.troubleshooting-tex.de/

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Files on a server

[Ulrike Fischer]

Am Thu, 17 May 2018 09:54:25 +0200 schrieb Taco Hoekwater:

>> Works ok for me. Did you test?
> 
> Oh, sorry. It seems it is sneakily rewriting the https:// to http://,
> and so does not _actually_ work.

Ah. This explains why it seemed to work for
https://httpbin.org/html, http://httpbin.org/html exists and why I
get a lot of redirection answers. 



-- 
Ulrike Fischer 
http://www.troubleshooting-tex.de/

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Files on a server

[Taco Hoekwater]

> On 17 May 2018, at 10:20, Ulrike Fischer <news3@nililand.de> wrote:
> 
> Am Thu, 17 May 2018 09:39:47 +0200 schrieb Taco Hoekwater:
> 
>>>> Or use luasocket, which is included in the luatex binary:
> 
>>> But I'm right that this works only with http and not with https?
> 
>> Works ok for me. Did you test?
> 
> I tried a few links and always got 301 back regardless if they exist
> or not and thought due to the discussion two month ago on the luatex
> list about https content that it doesn't work at all. 
> 
> I know found https://httpbin.org/ and with it the tests worked:
> https://httpbin.org/html gave 200 and https://httpbin.org/htmlxxxx
> gave 404.

It only works because it silently discards the https: and replaces
it with http: . Actually, it disregards what is in front of // 
completely, whatever it is.  Even:

  url    = "foobar://www.ntg.nl/watistex.html"

works just 'fine'.


Anyway, perhaps someone can answer me this? I tried the ffi/curl code,
and the network stuff works, but only when I comment out the write
callback:

  lcurl.curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, ffi.cast("callback",discard));

If I dont, I get an error message from ffi that is:

  unable to convert argument 3 from cdata<unsigned long long (*)(void*, unsigned long long, unsigned long long, void*)> to cdata<vararg>
  stack traceback:
	[C]: in function 'curl_easy_setopt'
	[ctxlua]:42: in function 'check_status'
	[ctxlua]:1: in main chunk


And I have no idea what that actually means or how to fix it?

Best wishes,
Taco
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Files on a server

[Hans Hagen]

On 5/17/2018 9:54 AM, Taco Hoekwater wrote:
> 
> 
>> On 17 May 2018, at 09:39, Taco Hoekwater <taco@elvenkind.com> wrote:
>>
>>
>>
>>> On 17 May 2018, at 09:32, Ulrike Fischer <news3@nililand.de> wrote:
>>>
>>> Am Wed, 16 May 2018 14:23:42 +0200 schrieb Taco Hoekwater:
>>>
>>>> Or use luasocket, which is included in the luatex binary:
>>>
>>> But I'm right that this works only with http and not with https?
>>
>> Works ok for me. Did you test?
> 
> Oh, sorry. It seems it is sneakily rewriting the https:// to http://,
> and so does not _actually_ work. Which makes sense, now that I think
> about it.  Doing https would need openssl support, which is unlikely to
> ever be built into luatex.
> 
> Sorry for being misleading,
i have (among a few others such experiments) a util-crl.lua file that 
uses libcurl (using ffi) because it's one of the things we used to test 
ffi support in luatex

i didn't add that to the distribution because i'm not sure what libname 
on windows to use (mojca and i discussed the option to ship some libs 
with the garden distribition at some time) and i don't want to end up in 
endless discussions about lib names on systems

some day i might add this one (if the right lib is found it works ok .. 
performance on my old laptop is is 5000 https://tug.org pages in 100 
sec, 5000 pragma website main pages in 70 sec)

(fwiw: i do use curl but just call the binary which in practice is just 
as efficient because one seldom does a lot in a run)

(and i use the socket lib for other things)

Hans


-----------------------------------------------------------------
                                           Hans Hagen | PRAGMA ADE
               Ridderstraat 27 | 8061 GH Hasselt | The Netherlands
        tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl
-----------------------------------------------------------------
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Files on a server

[luigi scarso]

[-- Attachment #1.1: Type: text/plain, Size: 306 bytes --]

On Thu, May 17, 2018 at 9:54 AM, Taco Hoekwater <taco@elvenkind.com> wrote:

>
>
Doing https would need openssl support, which is unlikely to
> ever be built into luatex.
>
>  We could  provide a ffi wrapper (once ffi is stable)
or a swiglib wrapper  as module.
Something to discuss at meeting.

-- 
luigi

[-- Attachment #1.2: Type: text/html, Size: 897 bytes --]

[-- Attachment #2: Type: text/plain, Size: 492 bytes --]

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Files on a server

[Taco Hoekwater]

> On 17 May 2018, at 10:31, Taco Hoekwater <taco@elvenkind.com> wrote:
> 
> Anyway, perhaps someone can answer me this? I tried the ffi/curl code,
> and the network stuff works, but only when I comment out the write
> callback:
> 
>  lcurl.curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, ffi.cast("callback",discard));
> 
> If I dont, I get an error message from ffi that is:
> 
>  unable to convert argument 3 from cdata<unsigned long long (*)(void*, unsigned long long, unsigned long long, void*)> to cdata<vararg>
>  stack traceback:
> 	[C]: in function 'curl_easy_setopt'
> 	[ctxlua]:42: in function 'check_status'
> 	[ctxlua]:1: in main chunk
> 
> And I have no idea what that actually means or how to fix it?

FYI: fixed by updating luatex.

Best wishes,
Taco

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________

Re: Files on a server

[Henri Menke]

On 05/17/2018 09:03 PM, Taco Hoekwater wrote:
> 
> 
>> On 17 May 2018, at 10:31, Taco Hoekwater <taco@elvenkind.com> wrote:
>>
>> Anyway, perhaps someone can answer me this? I tried the ffi/curl code,
>> and the network stuff works, but only when I comment out the write
>> callback:
>>
>>  lcurl.curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, ffi.cast("callback",discard));
>>
>> If I dont, I get an error message from ffi that is:
>>
>>  unable to convert argument 3 from cdata<unsigned long long (*)(void*, unsigned long long, unsigned long long, void*)> to cdata<vararg>
>>  stack traceback:
>> 	[C]: in function 'curl_easy_setopt'
>> 	[ctxlua]:42: in function 'check_status'
>> 	[ctxlua]:1: in main chunk
>>
>> And I have no idea what that actually means or how to fix it?
> 
> FYI: fixed by updating luatex.

Hi Taco,

Actually you don't really need to set the WRITEFUNCTION callback of
libcurl.  I only set it to the "discard" function which simply return
the number of bytes it got, such that libcurl doesn't print the header
information to stdout.  You could also plug a different function in
there if you wanted to process the headers.  I will soon post an
annotated version of the code with more proper error-checking on the list.

Cheers, Henri

> 
> Best wishes,
> Taco
> 
> ___________________________________________________________________________________
> If your question is of interest to others as well, please add an entry to the Wiki!
> 
> maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
> webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
> archive  : https://bitbucket.org/phg/context-mirror/commits/
> wiki     : http://contextgarden.net
> ___________________________________________________________________________________
> 

___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!

maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage  : http://www.pragma-ade.nl / http://context.aanhet.net
archive  : https://bitbucket.org/phg/context-mirror/commits/
wiki     : http://contextgarden.net
___________________________________________________________________________________