* Files on a server
@ 2018-05-16 9:31 Hans van der Meer
2018-05-16 10:41 ` Henri Menke
0 siblings, 1 reply; 19+ messages in thread
From: Hans van der Meer @ 2018-05-16 9:31 UTC (permalink / raw)
To: NTG ConTeXt
[-- Attachment #1.1: Type: text/plain, Size: 859 bytes --]
I tried to determine the existence of a file on the internet. See the following macro call:
% Test if file exists.
\edef\theurl{\linkprotocol://\urlbase\xmlatt{#1}{link}\thesuffix}
\doiffileelse
{\theurl}
{\verbose{HVDM-PEV-TEST}{file exists: \theurl}}
{\verbose{HVDM-PEV-TEST}{file does not exist: \theurl}}
It seems that \doiffileelse always results in true except for local files where nonexistence is correctly handled.
For example the following url definitely does not exist but nevertheless \doiffileelse reports its existence:
HVDM-PEV-TEST > file exists: http://hansvandermeer.myqnapcloud.com/archive/denhaag/hga-dtb-1869-6040.pdf
Is it possible that the \doiffile macros are extended for files on the internet or a correponding set \doifurl is developed?
Hans van der Meer
[-- Attachment #1.2: Type: text/html, Size: 2292 bytes --]
[-- Attachment #2: Type: text/plain, Size: 492 bytes --]
___________________________________________________________________________________
If your question is of interest to others as well, please add an entry to the Wiki!
maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context
webpage : http://www.pragma-ade.nl / http://context.aanhet.net
archive : https://bitbucket.org/phg/context-mirror/commits/
wiki : http://contextgarden.net
___________________________________________________________________________________
^ permalink raw reply [flat|nested] 19+ messages in thread* Re: Files on a server 2018-05-16 9:31 Files on a server Hans van der Meer @ 2018-05-16 10:41 ` Henri Menke 2018-05-16 11:23 ` Hans van der Meer 0 siblings, 1 reply; 19+ messages in thread From: Henri Menke @ 2018-05-16 10:41 UTC (permalink / raw) To: ntg-context On 05/16/2018 09:31 PM, Hans van der Meer wrote: > I tried to determine the existence of a file on the internet. See the > following macro call: > > % Test if file exists. > \edef\theurl{\linkprotocol://\urlbase\xmlatt{#1}{link}\thesuffix} > \doiffileelse > {\theurl} > {\verbose{HVDM-PEV-TEST}{file exists: \theurl}} > {\verbose{HVDM-PEV-TEST}{file does not exist: \theurl}} > > It seems that \doiffileelse always results in true except for local > files where nonexistence is correctly handled. > > For example the following url definitely does not exist but nevertheless > \doiffileelse reports its existence: > HVDM-PEV-TEST > file exists: > http://hansvandermeer.myqnapcloud.com/archive/denhaag/hga-dtb-1869-6040.pdf > > Is it possible that the \doiffile macros are extended for files on the > internet or a correponding set \doifurl is developed? I'm not speaking for the devs here, but I don't think that this is going to be included. It would require that all webpages send proper 404 codes and timeout after reasonable delay. Also how would you handle redirects? Should the ConTeXt run be stalled because checking for a files existence on the web is caught in a redirect loop? > > Hans van der Meer > > > > ___________________________________________________________________________________ > If your question is of interest to others as well, please add an entry to the Wiki! > > maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context > webpage : http://www.pragma-ade.nl / http://context.aanhet.net > archive : https://bitbucket.org/phg/context-mirror/commits/ > wiki : http://contextgarden.net > ___________________________________________________________________________________ > ___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___________________________________________________________________________________ ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Files on a server 2018-05-16 10:41 ` Henri Menke @ 2018-05-16 11:23 ` Hans van der Meer 2018-05-16 11:45 ` Henri Menke 0 siblings, 1 reply; 19+ messages in thread From: Hans van der Meer @ 2018-05-16 11:23 UTC (permalink / raw) To: NTG ConTeXt [-- Attachment #1.1: Type: text/plain, Size: 3119 bytes --] I would be satisfied when a returned 404 error code will be handled within a reasonable (configurable) time delay. As for redirection, there I would not mind if it is not included. Restrictions like that are not a problem for me, because this is for building a pdf that accesses many internet located files. The check on existence is therefore especially meant for spotting missing or misnamed files. Hans van der Meer > On 16 May 2018, at 12:41, Henri Menke <henrimenke@gmail.com> wrote: > > On 05/16/2018 09:31 PM, Hans van der Meer wrote: >> I tried to determine the existence of a file on the internet. See the >> following macro call: >> >> % Test if file exists. >> \edef\theurl{\linkprotocol://\urlbase\xmlatt{#1}{link}\thesuffix} >> \doiffileelse >> {\theurl} >> {\verbose{HVDM-PEV-TEST}{file exists: \theurl}} >> {\verbose{HVDM-PEV-TEST}{file does not exist: \theurl}} >> >> It seems that \doiffileelse always results in true except for local >> files where nonexistence is correctly handled. >> >> For example the following url definitely does not exist but nevertheless >> \doiffileelse reports its existence: >> HVDM-PEV-TEST > file exists: >> http://hansvandermeer.myqnapcloud.com/archive/denhaag/hga-dtb-1869-6040.pdf >> >> Is it possible that the \doiffile macros are extended for files on the >> internet or a correponding set \doifurl is developed? > > I'm not speaking for the devs here, but I don't think that this is going > to be included. It would require that all webpages send proper 404 > codes and timeout after reasonable delay. Also how would you handle > redirects? Should the ConTeXt run be stalled because checking for a > files existence on the web is caught in a redirect loop? > >> >> Hans van der Meer >> >> >> >> ___________________________________________________________________________________ >> If your question is of interest to others as well, please add an entry to the Wiki! >> >> maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context >> webpage : http://www.pragma-ade.nl / http://context.aanhet.net >> archive : https://bitbucket.org/phg/context-mirror/commits/ >> wiki : http://contextgarden.net >> ___________________________________________________________________________________ >> > > ___________________________________________________________________________________ > If your question is of interest to others as well, please add an entry to the Wiki! > > maillist : ntg-context@ntg.nl <mailto:ntg-context@ntg.nl> / http://www.ntg.nl/mailman/listinfo/ntg-context <http://www.ntg.nl/mailman/listinfo/ntg-context> > webpage : http://www.pragma-ade.nl <http://www.pragma-ade.nl/> / http://context.aanhet.net <http://context.aanhet.net/> > archive : https://bitbucket.org/phg/context-mirror/commits/ <https://bitbucket.org/phg/context-mirror/commits/> > wiki : http://contextgarden.net <http://contextgarden.net/> > ___________________________________________________________________________________ [-- Attachment #1.2: Type: text/html, Size: 19262 bytes --] [-- Attachment #2: Type: text/plain, Size: 492 bytes --] ___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___________________________________________________________________________________ ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Files on a server 2018-05-16 11:23 ` Hans van der Meer @ 2018-05-16 11:45 ` Henri Menke 2018-05-16 12:23 ` Taco Hoekwater 0 siblings, 1 reply; 19+ messages in thread From: Henri Menke @ 2018-05-16 11:45 UTC (permalink / raw) To: ntg-context@ntg.nl >> mailing list for ConTeXt users On 05/16/2018 11:23 PM, Hans van der Meer wrote: > I would be satisfied when a returned 404 error code will be handled > within a reasonable (configurable) time delay. As for redirection, there > I would not mind if it is not included. > Restrictions like that are not a problem for me, because this is for > building a pdf that accesses many internet located files. The check on > existence is therefore especially meant for spotting missing or misnamed > files. You can use libcurl via ffi: https://curl.haxx.se/libcurl/ \startluacode local ffi = require("ffi") ffi.cdef[[ typedef struct Curl_easy CURL; typedef int CURLcode; typedef int CURLoption; typedef int CURLINFO; typedef size_t(*callback)(void *buffer, size_t size, size_t nmemb, void *userp); CURLcode curl_global_init(long flags); CURL *curl_easy_init(void); CURLcode curl_easy_setopt(CURL *curl, CURLoption option, ...); CURLcode curl_easy_perform(CURL *curl); CURLcode curl_easy_getinfo(CURL *curl, CURLINFO info, ...); void curl_easy_cleanup(CURL *curl); void curl_global_cleanup(void); ]] -- magic numbers local CURL_GLOBAL_DEFAULT = 3 local CURLOPT_URL = 10002 local CURLOPT_NOBODY = 44 local CURLOPT_HEADER = 42 local CURLOPT_WRITEFUNCTION = 20011 local CURLINFO_RESPONSE_CODE = 0x200000 + 2 local lcurl = ffi.load("curl") local function discard(buffer, size, nmemb, userp) return size * nmemb end function check_status(url) lcurl.curl_global_init(CURL_GLOBAL_DEFAULT) local curl = lcurl.curl_easy_init() lcurl.curl_easy_setopt(curl, CURLOPT_URL, url) lcurl.curl_easy_setopt(curl, CURLOPT_NOBODY, ffi.cast("long",1)) lcurl.curl_easy_setopt(curl, CURLOPT_HEADER, ffi.cast("long",1)) lcurl.curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, ffi.cast("callback",discard)); local res = lcurl.curl_easy_perform(curl) local c_http_code = ffi.new("long[1]") lcurl.curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, c_http_code); lcurl.curl_easy_cleanup(curl) lcurl.curl_global_cleanup() return tonumber(c_http_code[0]) end \stopluacode \starttext \ctxlua{context(check_status("http://example.com/"))} \ctxlua{context(check_status("https://example.com/"))} \ctxlua{context(check_status("https://example.com/xyz"))} \stoptext > > Hans van der Meer > >> On 16 May 2018, at 12:41, Henri Menke <henrimenke@gmail.com >> <mailto:henrimenke@gmail.com>> wrote: >> >> On 05/16/2018 09:31 PM, Hans van der Meer wrote: >>> I tried to determine the existence of a file on the internet. See the >>> following macro call: >>> >>> % Test if file exists. >>> \edef\theurl{\linkprotocol://\urlbase\xmlatt{#1}{link}\thesuffix} >>> \doiffileelse >>> {\theurl} >>> {\verbose{HVDM-PEV-TEST}{file exists: \theurl}} >>> {\verbose{HVDM-PEV-TEST}{file does not exist: \theurl}} >>> >>> It seems that \doiffileelse always results in true except for local >>> files where nonexistence is correctly handled. >>> >>> For example the following url definitely does not exist but nevertheless >>> \doiffileelse reports its existence: >>> HVDM-PEV-TEST > file exists: >>> http://hansvandermeer.myqnapcloud.com/archive/denhaag/hga-dtb-1869-6040.pdf >>> >>> Is it possible that the \doiffile macros are extended for files on the >>> internet or a correponding set \doifurl is developed? >> >> I'm not speaking for the devs here, but I don't think that this is going >> to be included. It would require that all webpages send proper 404 >> codes and timeout after reasonable delay. Also how would you handle >> redirects? Should the ConTeXt run be stalled because checking for a >> files existence on the web is caught in a redirect loop? >> >>> >>> Hans van der Meer >>> >>> >>> >>> ___________________________________________________________________________________ >>> If your question is of interest to others as well, please add an >>> entry to the Wiki! >>> >>> maillist : ntg-context@ntg.nl <mailto:ntg-context@ntg.nl> / >>> http://www.ntg.nl/mailman/listinfo/ntg-context >>> webpage : http://www.pragma-ade.nl / http://context.aanhet.net >>> archive : https://bitbucket.org/phg/context-mirror/commits/ >>> wiki : http://contextgarden.net >>> ___________________________________________________________________________________ >>> >> >> ___________________________________________________________________________________ >> If your question is of interest to others as well, please add an entry >> to the Wiki! >> >> maillist : ntg-context@ntg.nl >> <mailto:ntg-context@ntg.nl> / http://www.ntg.nl/mailman/listinfo/ntg-context >> webpage : http://www.pragma-ade.nl >> <http://www.pragma-ade.nl/> / http://context.aanhet.net >> <http://context.aanhet.net/> >> archive : https://bitbucket.org/phg/context-mirror/commits/ >> wiki : http://contextgarden.net <http://contextgarden.net/> >> ___________________________________________________________________________________ > > > > ___________________________________________________________________________________ > If your question is of interest to others as well, please add an entry to the Wiki! > > maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context > webpage : http://www.pragma-ade.nl / http://context.aanhet.net > archive : https://bitbucket.org/phg/context-mirror/commits/ > wiki : http://contextgarden.net > ___________________________________________________________________________________ > ___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___________________________________________________________________________________ ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Files on a server 2018-05-16 11:45 ` Henri Menke @ 2018-05-16 12:23 ` Taco Hoekwater 2018-05-16 12:58 ` Hans van der Meer 2018-05-17 7:32 ` Ulrike Fischer 0 siblings, 2 replies; 19+ messages in thread From: Taco Hoekwater @ 2018-05-16 12:23 UTC (permalink / raw) To: mailing list for ConTeXt users Or use luasocket, which is included in the luatex binary: \startluacode content, status, authinfo = socket.http.request{ method = "HEAD", url = "http://hansvandermeer.myqnapcloud.com/archive/denhaag/hga-dtb-1869-6040.pdf” } print (status) \stopluacode prints ‘404’ in this case. Taco > On 16 May 2018, at 13:45, Henri Menke <henrimenke@gmail.com> wrote: > > On 05/16/2018 11:23 PM, Hans van der Meer wrote: >> I would be satisfied when a returned 404 error code will be handled >> within a reasonable (configurable) time delay. As for redirection, there >> I would not mind if it is not included. >> Restrictions like that are not a problem for me, because this is for >> building a pdf that accesses many internet located files. The check on >> existence is therefore especially meant for spotting missing or misnamed >> files. > > You can use libcurl via ffi: https://curl.haxx.se/libcurl/ > > \startluacode > local ffi = require("ffi") > > ffi.cdef[[ > typedef struct Curl_easy CURL; > typedef int CURLcode; > typedef int CURLoption; > typedef int CURLINFO; > typedef size_t(*callback)(void *buffer, size_t size, size_t nmemb, void > *userp); > > CURLcode curl_global_init(long flags); > CURL *curl_easy_init(void); > CURLcode curl_easy_setopt(CURL *curl, CURLoption option, ...); > CURLcode curl_easy_perform(CURL *curl); > CURLcode curl_easy_getinfo(CURL *curl, CURLINFO info, ...); > void curl_easy_cleanup(CURL *curl); > void curl_global_cleanup(void); > ]] > > -- magic numbers > local CURL_GLOBAL_DEFAULT = 3 > local CURLOPT_URL = 10002 > local CURLOPT_NOBODY = 44 > local CURLOPT_HEADER = 42 > local CURLOPT_WRITEFUNCTION = 20011 > local CURLINFO_RESPONSE_CODE = 0x200000 + 2 > > local lcurl = ffi.load("curl") > > local function discard(buffer, size, nmemb, userp) > return size * nmemb > end > > function check_status(url) > lcurl.curl_global_init(CURL_GLOBAL_DEFAULT) > > local curl = lcurl.curl_easy_init() > > lcurl.curl_easy_setopt(curl, CURLOPT_URL, url) > lcurl.curl_easy_setopt(curl, CURLOPT_NOBODY, ffi.cast("long",1)) > lcurl.curl_easy_setopt(curl, CURLOPT_HEADER, ffi.cast("long",1)) > lcurl.curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, > ffi.cast("callback",discard)); > > local res = lcurl.curl_easy_perform(curl) > local c_http_code = ffi.new("long[1]") > lcurl.curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, c_http_code); > > lcurl.curl_easy_cleanup(curl) > lcurl.curl_global_cleanup() > > return tonumber(c_http_code[0]) > end > \stopluacode > > \starttext > > \ctxlua{context(check_status("http://example.com/"))} > > \ctxlua{context(check_status("https://example.com/"))} > > \ctxlua{context(check_status("https://example.com/xyz"))} > > \stoptext > >> >> Hans van der Meer >> >>> On 16 May 2018, at 12:41, Henri Menke <henrimenke@gmail.com >>> <mailto:henrimenke@gmail.com>> wrote: >>> >>> On 05/16/2018 09:31 PM, Hans van der Meer wrote: >>>> I tried to determine the existence of a file on the internet. See the >>>> following macro call: >>>> >>>> % Test if file exists. >>>> \edef\theurl{\linkprotocol://\urlbase\xmlatt{#1}{link}\thesuffix} >>>> \doiffileelse >>>> {\theurl} >>>> {\verbose{HVDM-PEV-TEST}{file exists: \theurl}} >>>> {\verbose{HVDM-PEV-TEST}{file does not exist: \theurl}} >>>> >>>> It seems that \doiffileelse always results in true except for local >>>> files where nonexistence is correctly handled. >>>> >>>> For example the following url definitely does not exist but nevertheless >>>> \doiffileelse reports its existence: >>>> HVDM-PEV-TEST > file exists: >>>> http://hansvandermeer.myqnapcloud.com/archive/denhaag/hga-dtb-1869-6040.pdf >>>> >>>> Is it possible that the \doiffile macros are extended for files on the >>>> internet or a correponding set \doifurl is developed? >>> >>> I'm not speaking for the devs here, but I don't think that this is going >>> to be included. It would require that all webpages send proper 404 >>> codes and timeout after reasonable delay. Also how would you handle >>> redirects? Should the ConTeXt run be stalled because checking for a >>> files existence on the web is caught in a redirect loop? >>> >>>> >>>> Hans van der Meer >>>> >>>> >>>> >>>> ___________________________________________________________________________________ >>>> If your question is of interest to others as well, please add an >>>> entry to the Wiki! >>>> >>>> maillist : ntg-context@ntg.nl <mailto:ntg-context@ntg.nl> / >>>> http://www.ntg.nl/mailman/listinfo/ntg-context >>>> webpage : http://www.pragma-ade.nl / http://context.aanhet.net >>>> archive : https://bitbucket.org/phg/context-mirror/commits/ >>>> wiki : http://contextgarden.net >>>> ___________________________________________________________________________________ >>>> >>> >>> ___________________________________________________________________________________ >>> If your question is of interest to others as well, please add an entry >>> to the Wiki! >>> >>> maillist : ntg-context@ntg.nl >>> <mailto:ntg-context@ntg.nl> / http://www.ntg.nl/mailman/listinfo/ntg-context >>> webpage : http://www.pragma-ade.nl >>> <http://www.pragma-ade.nl/> / http://context.aanhet.net >>> <http://context.aanhet.net/> >>> archive : https://bitbucket.org/phg/context-mirror/commits/ >>> wiki : http://contextgarden.net <http://contextgarden.net/> >>> ___________________________________________________________________________________ >> >> >> >> ___________________________________________________________________________________ >> If your question is of interest to others as well, please add an entry to the Wiki! >> >> maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context >> webpage : http://www.pragma-ade.nl / http://context.aanhet.net >> archive : https://bitbucket.org/phg/context-mirror/commits/ >> wiki : http://contextgarden.net >> ___________________________________________________________________________________ >> > > ___________________________________________________________________________________ > If your question is of interest to others as well, please add an entry to the Wiki! > > maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context > webpage : http://www.pragma-ade.nl / http://context.aanhet.net > archive : https://bitbucket.org/phg/context-mirror/commits/ > wiki : http://contextgarden.net > ___________________________________________________________________________________ Taco Hoekwater Elvenkind BV ___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___________________________________________________________________________________ ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Files on a server 2018-05-16 12:23 ` Taco Hoekwater @ 2018-05-16 12:58 ` Hans van der Meer 2018-05-16 13:41 ` Hans Hagen 2018-05-17 7:32 ` Ulrike Fischer 1 sibling, 1 reply; 19+ messages in thread From: Hans van der Meer @ 2018-05-16 12:58 UTC (permalink / raw) To: NTG ConTeXt [-- Attachment #1.1: Type: text/plain, Size: 9238 bytes --] Beste Taco, Ik probeer jouw oplossing maar er gebeurt nog iets raars. Ik krijg inderdaad 200 resp. 404 terug bij bestaande niet-bestaande Uri’s. Maar ik krijg wel een error die aan het eind van de run gegenereerd wordt. Heb jij enig idee waar ik dat zou moeten zoeken? Als ik de call naar \verbose uitschakel komt er geen foutmelding. Blijft er na de socket call iets achter wat nog opgeruimd had moeten worden? Hier een deel van de context-log en de code fragmenten in kwestie. <takepersons> is de root node van het bewerkte bestand. HVDM-PEV-TEST > return code: 200 HVDM-PEV > number of events processed: 1 HVDM-PEV > ============= end of takepersons ============= backend > xmp > using file '/Users/hansm/TeX/context-45/tex/texmf-context/tex/context/base/mkiv/lpdf-pdx.xml' pages > flushing realpage 1, userpage 1, subpage 1 close source > level 2, order 13, name '/Users/hansm/Documents/Genealogie/familie-event.tex' close source > level 1, order 13, name '/Users/hansm/TeX/context-45/tex/texmf-context/tex/context/base/mkiv/cont-yes.mkiv' lua error > lua error on line 0 in file : ...-45/tex/texmf-context/tex/context/base/mkiv/core-uti.lua:92: attempt to index a number value (global 'status') <empty file> ? Process aborted Code in tex: \def\URIReturnCode#1{\ctxlua{tex.print(hvdm.urireturncode("#1"))}} \verbose{HVDM-PEV-TEST}{return code: \URIReturnCode{\theurl}} Code in lua: hvdm.urireturncode = function (theuri) content, status, authinfo = socket.http.request{ method = "HEAD", url = theuri, } return tostring(status) end Groet Hans > On 16 May 2018, at 14:23, Taco Hoekwater <taco@elvenkind.com> wrote: > > > Or use luasocket, which is included in the luatex binary: > > \startluacode > content, status, authinfo = socket.http.request{ > method = "HEAD", > url = "http://hansvandermeer.myqnapcloud.com/archive/denhaag/hga-dtb-1869-6040.pdf” > } > > print (status) > > \stopluacode > > prints ‘404’ in this case. > > Taco > >> On 16 May 2018, at 13:45, Henri Menke <henrimenke@gmail.com> wrote: >> >> On 05/16/2018 11:23 PM, Hans van der Meer wrote: >>> I would be satisfied when a returned 404 error code will be handled >>> within a reasonable (configurable) time delay. As for redirection, there >>> I would not mind if it is not included. >>> Restrictions like that are not a problem for me, because this is for >>> building a pdf that accesses many internet located files. The check on >>> existence is therefore especially meant for spotting missing or misnamed >>> files. >> >> You can use libcurl via ffi: https://curl.haxx.se/libcurl/ >> >> \startluacode >> local ffi = require("ffi") >> >> ffi.cdef[[ >> typedef struct Curl_easy CURL; >> typedef int CURLcode; >> typedef int CURLoption; >> typedef int CURLINFO; >> typedef size_t(*callback)(void *buffer, size_t size, size_t nmemb, void >> *userp); >> >> CURLcode curl_global_init(long flags); >> CURL *curl_easy_init(void); >> CURLcode curl_easy_setopt(CURL *curl, CURLoption option, ...); >> CURLcode curl_easy_perform(CURL *curl); >> CURLcode curl_easy_getinfo(CURL *curl, CURLINFO info, ...); >> void curl_easy_cleanup(CURL *curl); >> void curl_global_cleanup(void); >> ]] >> >> -- magic numbers >> local CURL_GLOBAL_DEFAULT = 3 >> local CURLOPT_URL = 10002 >> local CURLOPT_NOBODY = 44 >> local CURLOPT_HEADER = 42 >> local CURLOPT_WRITEFUNCTION = 20011 >> local CURLINFO_RESPONSE_CODE = 0x200000 + 2 >> >> local lcurl = ffi.load("curl") >> >> local function discard(buffer, size, nmemb, userp) >> return size * nmemb >> end >> >> function check_status(url) >> lcurl.curl_global_init(CURL_GLOBAL_DEFAULT) >> >> local curl = lcurl.curl_easy_init() >> >> lcurl.curl_easy_setopt(curl, CURLOPT_URL, url) >> lcurl.curl_easy_setopt(curl, CURLOPT_NOBODY, ffi.cast("long",1)) >> lcurl.curl_easy_setopt(curl, CURLOPT_HEADER, ffi.cast("long",1)) >> lcurl.curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, >> ffi.cast("callback",discard)); >> >> local res = lcurl.curl_easy_perform(curl) >> local c_http_code = ffi.new("long[1]") >> lcurl.curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, c_http_code); >> >> lcurl.curl_easy_cleanup(curl) >> lcurl.curl_global_cleanup() >> >> return tonumber(c_http_code[0]) >> end >> \stopluacode >> >> \starttext >> >> \ctxlua{context(check_status("http://example.com/"))} >> >> \ctxlua{context(check_status("https://example.com/"))} >> >> \ctxlua{context(check_status("https://example.com/xyz"))} >> >> \stoptext >> >>> >>> Hans van der Meer >>> >>>> On 16 May 2018, at 12:41, Henri Menke <henrimenke@gmail.com >>>> <mailto:henrimenke@gmail.com>> wrote: >>>> >>>> On 05/16/2018 09:31 PM, Hans van der Meer wrote: >>>>> I tried to determine the existence of a file on the internet. See the >>>>> following macro call: >>>>> >>>>> % Test if file exists. >>>>> \edef\theurl{\linkprotocol://\urlbase\xmlatt{#1}{link}\thesuffix} >>>>> \doiffileelse >>>>> {\theurl} >>>>> {\verbose{HVDM-PEV-TEST}{file exists: \theurl}} >>>>> {\verbose{HVDM-PEV-TEST}{file does not exist: \theurl}} >>>>> >>>>> It seems that \doiffileelse always results in true except for local >>>>> files where nonexistence is correctly handled. >>>>> >>>>> For example the following url definitely does not exist but nevertheless >>>>> \doiffileelse reports its existence: >>>>> HVDM-PEV-TEST > file exists: >>>>> http://hansvandermeer.myqnapcloud.com/archive/denhaag/hga-dtb-1869-6040.pdf >>>>> >>>>> Is it possible that the \doiffile macros are extended for files on the >>>>> internet or a correponding set \doifurl is developed? >>>> >>>> I'm not speaking for the devs here, but I don't think that this is going >>>> to be included. It would require that all webpages send proper 404 >>>> codes and timeout after reasonable delay. Also how would you handle >>>> redirects? Should the ConTeXt run be stalled because checking for a >>>> files existence on the web is caught in a redirect loop? >>>> >>>>> >>>>> Hans van der Meer >>>>> >>>>> >>>>> >>>>> ___________________________________________________________________________________ >>>>> If your question is of interest to others as well, please add an >>>>> entry to the Wiki! >>>>> >>>>> maillist : ntg-context@ntg.nl <mailto:ntg-context@ntg.nl> / >>>>> http://www.ntg.nl/mailman/listinfo/ntg-context >>>>> webpage : http://www.pragma-ade.nl / http://context.aanhet.net >>>>> archive : https://bitbucket.org/phg/context-mirror/commits/ >>>>> wiki : http://contextgarden.net >>>>> ___________________________________________________________________________________ >>>>> >>>> >>>> ___________________________________________________________________________________ >>>> If your question is of interest to others as well, please add an entry >>>> to the Wiki! >>>> >>>> maillist : ntg-context@ntg.nl >>>> <mailto:ntg-context@ntg.nl> / http://www.ntg.nl/mailman/listinfo/ntg-context >>>> webpage : http://www.pragma-ade.nl >>>> <http://www.pragma-ade.nl/> / http://context.aanhet.net >>>> <http://context.aanhet.net/> >>>> archive : https://bitbucket.org/phg/context-mirror/commits/ >>>> wiki : http://contextgarden.net <http://contextgarden.net/> >>>> ___________________________________________________________________________________ >>> >>> >>> >>> ___________________________________________________________________________________ >>> If your question is of interest to others as well, please add an entry to the Wiki! >>> >>> maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context >>> webpage : http://www.pragma-ade.nl / http://context.aanhet.net >>> archive : https://bitbucket.org/phg/context-mirror/commits/ >>> wiki : http://contextgarden.net >>> ___________________________________________________________________________________ >>> >> >> ___________________________________________________________________________________ >> If your question is of interest to others as well, please add an entry to the Wiki! >> >> maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context >> webpage : http://www.pragma-ade.nl / http://context.aanhet.net >> archive : https://bitbucket.org/phg/context-mirror/commits/ >> wiki : http://contextgarden.net >> ___________________________________________________________________________________ > > Taco Hoekwater > Elvenkind BV > > > > > ___________________________________________________________________________________ > If your question is of interest to others as well, please add an entry to the Wiki! > > maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context > webpage : http://www.pragma-ade.nl / http://context.aanhet.net > archive : https://bitbucket.org/phg/context-mirror/commits/ > wiki : http://contextgarden.net > ___________________________________________________________________________________ [-- Attachment #1.2: Type: text/html, Size: 18102 bytes --] [-- Attachment #2: Type: text/plain, Size: 492 bytes --] ___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___________________________________________________________________________________ ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Files on a server 2018-05-16 12:58 ` Hans van der Meer @ 2018-05-16 13:41 ` Hans Hagen 2018-05-16 14:05 ` Taco Hoekwater 0 siblings, 1 reply; 19+ messages in thread From: Hans Hagen @ 2018-05-16 13:41 UTC (permalink / raw) To: mailing list for ConTeXt users, Hans van der Meer On 5/16/2018 2:58 PM, Hans van der Meer wrote: > Beste Taco, > > Ik probeer jouw oplossing maar er gebeurt nog iets raars. Ik krijg > inderdaad 200 resp. 404 terug bij bestaande niet-bestaande Uri’s. Maar > ik krijg wel een error die aan het eind van de run gegenereerd wordt. > Heb jij enig idee waar ik dat zou moeten zoeken? Als ik de call naar > \verbose uitschakel komt er geen foutmelding. Blijft er na de socket > call iets achter wat nog opgeruimd had moeten worden? how about adding local ... \startluacode local content, status, authinfo = socket.http.request { method = "HEAD", url = "http://hansvandermeer.myqnapcloud.com/archive/denhaag/hga-dtb-1869-6040.pdf", } print (status) \stopluacode > Hier een deel van de context-log en de code fragmenten in kwestie. > <takepersons> is de root node van het bewerkte bestand. > > HVDM-PEV-TEST > return code: 200 > HVDM-PEV > number of events processed: 1 > HVDM-PEV > ============= end of takepersons ============= > backend > xmp > using file > '/Users/hansm/TeX/context-45/tex/texmf-context/tex/context/base/mkiv/lpdf-pdx.xml' > pages > flushing realpage 1, userpage 1, subpage 1 > close source > level 2, order 13, name > '/Users/hansm/Documents/Genealogie/familie-event.tex' > close source > level 1, order 13, name > '/Users/hansm/TeX/context-45/tex/texmf-context/tex/context/base/mkiv/cont-yes.mkiv' > > lua error > lua error on line 0 in file : > > ...-45/tex/texmf-context/tex/context/base/mkiv/core-uti.lua:92: attempt > to index a number value (global 'status') > > <empty file> > > > ? > Process aborted > > Code in tex: > \def\URIReturnCode#1{\ctxlua{tex.print(hvdm.urireturncode("#1"))}} > \verbose{HVDM-PEV-TEST}{return code: \URIReturnCode{\theurl}} > Code in lua: > hvdm.urireturncode = function (theuri) > content, status, authinfo = socket.http.request{ > method = "HEAD", > url = theuri, > } > return tostring(status) > end > > Groet Hans > > > >> On 16 May 2018, at 14:23, Taco Hoekwater <taco@elvenkind.com >> <mailto:taco@elvenkind.com>> wrote: >> >> >> Or use luasocket, which is included in the luatex binary: >> >> \startluacode >> content, status, authinfo = socket.http.request{ >> method = "HEAD", >> url = >> "http://hansvandermeer.myqnapcloud.com/archive/denhaag/hga-dtb-1869-6040.pdf” >> >> } >> >> print (status) >> >> \stopluacode >> >> prints ‘404’ in this case. >> >> Taco >> >>> On 16 May 2018, at 13:45, Henri Menke <henrimenke@gmail.com >>> <mailto:henrimenke@gmail.com>> wrote: >>> >>> On 05/16/2018 11:23 PM, Hans van der Meer wrote: >>>> I would be satisfied when a returned 404 error code will be handled >>>> within a reasonable (configurable) time delay. As for redirection, there >>>> I would not mind if it is not included. >>>> Restrictions like that are not a problem for me, because this is for >>>> building a pdf that accesses many internet located files. The check on >>>> existence is therefore especially meant for spotting missing or misnamed >>>> files. >>> >>> You can use libcurl via ffi: https://curl.haxx.se/libcurl/ >>> >>> \startluacode >>> local ffi = require("ffi") >>> >>> ffi.cdef[[ >>> typedef struct Curl_easy CURL; >>> typedef int CURLcode; >>> typedef int CURLoption; >>> typedef int CURLINFO; >>> typedef size_t(*callback)(void *buffer, size_t size, size_t nmemb, void >>> *userp); >>> >>> CURLcode curl_global_init(long flags); >>> CURL *curl_easy_init(void); >>> CURLcode curl_easy_setopt(CURL *curl, CURLoption option, ...); >>> CURLcode curl_easy_perform(CURL *curl); >>> CURLcode curl_easy_getinfo(CURL *curl, CURLINFO info, ...); >>> void curl_easy_cleanup(CURL *curl); >>> void curl_global_cleanup(void); >>> ]] >>> >>> -- magic numbers >>> local CURL_GLOBAL_DEFAULT = 3 >>> local CURLOPT_URL = 10002 >>> local CURLOPT_NOBODY = 44 >>> local CURLOPT_HEADER = 42 >>> local CURLOPT_WRITEFUNCTION = 20011 >>> local CURLINFO_RESPONSE_CODE = 0x200000 + 2 >>> >>> local lcurl = ffi.load("curl") >>> >>> local function discard(buffer, size, nmemb, userp) >>> return size * nmemb >>> end >>> >>> function check_status(url) >>> lcurl.curl_global_init(CURL_GLOBAL_DEFAULT) >>> >>> local curl = lcurl.curl_easy_init() >>> >>> lcurl.curl_easy_setopt(curl, CURLOPT_URL, url) >>> lcurl.curl_easy_setopt(curl, CURLOPT_NOBODY, ffi.cast("long",1)) >>> lcurl.curl_easy_setopt(curl, CURLOPT_HEADER, ffi.cast("long",1)) >>> lcurl.curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, >>> ffi.cast("callback",discard)); >>> >>> local res = lcurl.curl_easy_perform(curl) >>> local c_http_code = ffi.new("long[1]") >>> lcurl.curl_easy_getinfo(curl, CURLINFO_RESPONSE_CODE, c_http_code); >>> >>> lcurl.curl_easy_cleanup(curl) >>> lcurl.curl_global_cleanup() >>> >>> return tonumber(c_http_code[0]) >>> end >>> \stopluacode >>> >>> \starttext >>> >>> \ctxlua{context(check_status("http://example.com/"))} >>> >>> \ctxlua{context(check_status("https://example.com/"))} >>> >>> \ctxlua{context(check_status("https://example.com/xyz"))} >>> >>> \stoptext >>> >>>> >>>> Hans van der Meer >>>> >>>>> On 16 May 2018, at 12:41, Henri Menke <henrimenke@gmail.com >>>>> <mailto:henrimenke@gmail.com> >>>>> <mailto:henrimenke@gmail.com>> wrote: >>>>> >>>>> On 05/16/2018 09:31 PM, Hans van der Meer wrote: >>>>>> I tried to determine the existence of a file on the internet. See the >>>>>> following macro call: >>>>>> >>>>>> % Test if file exists. >>>>>> \edef\theurl{\linkprotocol://\urlbase\xmlatt{#1}{link}\thesuffix} >>>>>> \doiffileelse >>>>>> {\theurl} >>>>>> {\verbose{HVDM-PEV-TEST}{file exists: \theurl}} >>>>>> {\verbose{HVDM-PEV-TEST}{file does not exist: \theurl}} >>>>>> >>>>>> It seems that \doiffileelse always results in true except for local >>>>>> files where nonexistence is correctly handled. >>>>>> >>>>>> For example the following url definitely does not exist but >>>>>> nevertheless >>>>>> \doiffileelse reports its existence: >>>>>> HVDM-PEV-TEST > file exists: >>>>>> http://hansvandermeer.myqnapcloud.com/archive/denhaag/hga-dtb-1869-6040.pdf >>>>>> >>>>>> Is it possible that the \doiffile macros are extended for files on the >>>>>> internet or a correponding set \doifurl is developed? >>>>> >>>>> I'm not speaking for the devs here, but I don't think that this is >>>>> going >>>>> to be included. It would require that all webpages send proper 404 >>>>> codes and timeout after reasonable delay. Also how would you handle >>>>> redirects? Should the ConTeXt run be stalled because checking for a >>>>> files existence on the web is caught in a redirect loop? >>>>> >>>>>> >>>>>> Hans van der Meer >>>>>> >>>>>> >>>>>> >>>>>> ___________________________________________________________________________________ >>>>>> If your question is of interest to others as well, please add an >>>>>> entry to the Wiki! >>>>>> >>>>>> maillist : ntg-context@ntg.nl <mailto:ntg-context@ntg.nl> >>>>>> <mailto:ntg-context@ntg.nl> / >>>>>> http://www.ntg.nl/mailman/listinfo/ntg-context >>>>>> webpage : http://www.pragma-ade.nl / http://context.aanhet.net >>>>>> archive : https://bitbucket.org/phg/context-mirror/commits/ >>>>>> wiki : http://contextgarden.net >>>>>> ___________________________________________________________________________________ >>>>>> >>>>> >>>>> ___________________________________________________________________________________ >>>>> If your question is of interest to others as well, please add an entry >>>>> to the Wiki! >>>>> >>>>> maillist : ntg-context@ntg.nl <mailto:ntg-context@ntg.nl> >>>>> <mailto:ntg-context@ntg.nl> / >>>>> http://www.ntg.nl/mailman/listinfo/ntg-context >>>>> webpage : http://www.pragma-ade.nl >>>>> <http://www.pragma-ade.nl/> / http://context.aanhet.net >>>>> <http://context.aanhet.net/> >>>>> archive : https://bitbucket.org/phg/context-mirror/commits/ >>>>> wiki : http://contextgarden.net <http://contextgarden.net/> >>>>> ___________________________________________________________________________________ >>>> >>>> >>>> >>>> ___________________________________________________________________________________ >>>> If your question is of interest to others as well, please add an >>>> entry to the Wiki! >>>> >>>> maillist : ntg-context@ntg.nl <mailto:ntg-context@ntg.nl> / >>>> http://www.ntg.nl/mailman/listinfo/ntg-context >>>> webpage : http://www.pragma-ade.nl / http://context.aanhet.net >>>> archive : https://bitbucket.org/phg/context-mirror/commits/ >>>> wiki : http://contextgarden.net >>>> ___________________________________________________________________________________ >>>> >>> >>> ___________________________________________________________________________________ >>> If your question is of interest to others as well, please add an >>> entry to the Wiki! >>> >>> maillist : ntg-context@ntg.nl <mailto:ntg-context@ntg.nl> / >>> http://www.ntg.nl/mailman/listinfo/ntg-context >>> webpage : http://www.pragma-ade.nl / http://context.aanhet.net >>> archive : https://bitbucket.org/phg/context-mirror/commits/ >>> wiki : http://contextgarden.net >>> ___________________________________________________________________________________ >> >> Taco Hoekwater >> Elvenkind BV >> >> >> >> >> ___________________________________________________________________________________ >> If your question is of interest to others as well, please add an entry >> to the Wiki! >> >> maillist : ntg-context@ntg.nl <mailto:ntg-context@ntg.nl> / >> http://www.ntg.nl/mailman/listinfo/ntg-context >> webpage : http://www.pragma-ade.nl / http://context.aanhet.net >> archive : https://bitbucket.org/phg/context-mirror/commits/ >> wiki : http://contextgarden.net >> ___________________________________________________________________________________ > > > > ___________________________________________________________________________________ > If your question is of interest to others as well, please add an entry to the Wiki! > > maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context > webpage : http://www.pragma-ade.nl / http://context.aanhet.net > archive : https://bitbucket.org/phg/context-mirror/commits/ > wiki : http://contextgarden.net > ___________________________________________________________________________________ > -- ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl ----------------------------------------------------------------- ___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___________________________________________________________________________________ ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Files on a server 2018-05-16 13:41 ` Hans Hagen @ 2018-05-16 14:05 ` Taco Hoekwater 0 siblings, 0 replies; 19+ messages in thread From: Taco Hoekwater @ 2018-05-16 14:05 UTC (permalink / raw) To: mailing list for ConTeXt users > On 16 May 2018, at 15:41, Hans Hagen <j.hagen@xs4all.nl> wrote: > > On 5/16/2018 2:58 PM, Hans van der Meer wrote: >> Beste Taco, >> Ik probeer jouw oplossing maar er gebeurt nog iets raars. Ik krijg inderdaad 200 resp. 404 terug bij bestaande niet-bestaande Uri’s. Maar ik krijg wel een error die aan het eind van de run gegenereerd wordt. Heb jij enig idee waar ik dat zou moeten zoeken? Als ik de call naar \verbose uitschakel komt er geen foutmelding. Blijft er na de socket call iets achter wat nog opgeruimd had moeten worden? > > > how about adding local … yeah. Taco ___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___________________________________________________________________________________ ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Files on a server 2018-05-16 12:23 ` Taco Hoekwater 2018-05-16 12:58 ` Hans van der Meer @ 2018-05-17 7:32 ` Ulrike Fischer 2018-05-17 7:39 ` Taco Hoekwater 1 sibling, 1 reply; 19+ messages in thread From: Ulrike Fischer @ 2018-05-17 7:32 UTC (permalink / raw) To: ntg-context Am Wed, 16 May 2018 14:23:42 +0200 schrieb Taco Hoekwater: > Or use luasocket, which is included in the luatex binary: But I'm right that this works only with http and not with https? -- Ulrike Fischer http://www.troubleshooting-tex.de/ ___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___________________________________________________________________________________ ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Files on a server 2018-05-17 7:32 ` Ulrike Fischer @ 2018-05-17 7:39 ` Taco Hoekwater 2018-05-17 7:52 ` Hans van der Meer ` (2 more replies) 0 siblings, 3 replies; 19+ messages in thread From: Taco Hoekwater @ 2018-05-17 7:39 UTC (permalink / raw) To: news3, mailing list for ConTeXt users > On 17 May 2018, at 09:32, Ulrike Fischer <news3@nililand.de> wrote: > > Am Wed, 16 May 2018 14:23:42 +0200 schrieb Taco Hoekwater: > >> Or use luasocket, which is included in the luatex binary: > > But I'm right that this works only with http and not with https? Works ok for me. Did you test? Taco ___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___________________________________________________________________________________ ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Files on a server 2018-05-17 7:39 ` Taco Hoekwater @ 2018-05-17 7:52 ` Hans van der Meer 2018-05-17 7:54 ` Taco Hoekwater 2018-05-17 8:20 ` Ulrike Fischer 2 siblings, 0 replies; 19+ messages in thread From: Hans van der Meer @ 2018-05-17 7:52 UTC (permalink / raw) To: NTG ConTeXt I too have the impression it is working for both. Hans van der Meer > On 17 May 2018, at 09:39, Taco Hoekwater <taco@elvenkind.com> wrote: > > > >> On 17 May 2018, at 09:32, Ulrike Fischer <news3@nililand.de> wrote: >> >> Am Wed, 16 May 2018 14:23:42 +0200 schrieb Taco Hoekwater: >> >>> Or use luasocket, which is included in the luatex binary: >> >> But I'm right that this works only with http and not with https? > > Works ok for me. Did you test? > > Taco > > ___________________________________________________________________________________ > If your question is of interest to others as well, please add an entry to the Wiki! > > maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context > webpage : http://www.pragma-ade.nl / http://context.aanhet.net > archive : https://bitbucket.org/phg/context-mirror/commits/ > wiki : http://contextgarden.net > ___________________________________________________________________________________ ___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___________________________________________________________________________________ ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Files on a server 2018-05-17 7:39 ` Taco Hoekwater 2018-05-17 7:52 ` Hans van der Meer @ 2018-05-17 7:54 ` Taco Hoekwater 2018-05-17 8:30 ` Ulrike Fischer ` (2 more replies) 2018-05-17 8:20 ` Ulrike Fischer 2 siblings, 3 replies; 19+ messages in thread From: Taco Hoekwater @ 2018-05-17 7:54 UTC (permalink / raw) To: mailing list for ConTeXt users; +Cc: news3 > On 17 May 2018, at 09:39, Taco Hoekwater <taco@elvenkind.com> wrote: > > > >> On 17 May 2018, at 09:32, Ulrike Fischer <news3@nililand.de> wrote: >> >> Am Wed, 16 May 2018 14:23:42 +0200 schrieb Taco Hoekwater: >> >>> Or use luasocket, which is included in the luatex binary: >> >> But I'm right that this works only with http and not with https? > > Works ok for me. Did you test? Oh, sorry. It seems it is sneakily rewriting the https:// to http://, and so does not _actually_ work. Which makes sense, now that I think about it. Doing https would need openssl support, which is unlikely to ever be built into luatex. Sorry for being misleading, Taco ___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___________________________________________________________________________________ ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Files on a server 2018-05-17 7:54 ` Taco Hoekwater @ 2018-05-17 8:30 ` Ulrike Fischer 2018-05-17 8:42 ` Hans Hagen 2018-05-17 8:46 ` luigi scarso 2 siblings, 0 replies; 19+ messages in thread From: Ulrike Fischer @ 2018-05-17 8:30 UTC (permalink / raw) To: ntg-context Am Thu, 17 May 2018 09:54:25 +0200 schrieb Taco Hoekwater: >> Works ok for me. Did you test? > > Oh, sorry. It seems it is sneakily rewriting the https:// to http://, > and so does not _actually_ work. Ah. This explains why it seemed to work for https://httpbin.org/html, http://httpbin.org/html exists and why I get a lot of redirection answers. -- Ulrike Fischer http://www.troubleshooting-tex.de/ ___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___________________________________________________________________________________ ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Files on a server 2018-05-17 7:54 ` Taco Hoekwater 2018-05-17 8:30 ` Ulrike Fischer @ 2018-05-17 8:42 ` Hans Hagen 2018-05-17 8:46 ` luigi scarso 2 siblings, 0 replies; 19+ messages in thread From: Hans Hagen @ 2018-05-17 8:42 UTC (permalink / raw) To: ntg-context On 5/17/2018 9:54 AM, Taco Hoekwater wrote: > > >> On 17 May 2018, at 09:39, Taco Hoekwater <taco@elvenkind.com> wrote: >> >> >> >>> On 17 May 2018, at 09:32, Ulrike Fischer <news3@nililand.de> wrote: >>> >>> Am Wed, 16 May 2018 14:23:42 +0200 schrieb Taco Hoekwater: >>> >>>> Or use luasocket, which is included in the luatex binary: >>> >>> But I'm right that this works only with http and not with https? >> >> Works ok for me. Did you test? > > Oh, sorry. It seems it is sneakily rewriting the https:// to http://, > and so does not _actually_ work. Which makes sense, now that I think > about it. Doing https would need openssl support, which is unlikely to > ever be built into luatex. > > Sorry for being misleading, i have (among a few others such experiments) a util-crl.lua file that uses libcurl (using ffi) because it's one of the things we used to test ffi support in luatex i didn't add that to the distribution because i'm not sure what libname on windows to use (mojca and i discussed the option to ship some libs with the garden distribition at some time) and i don't want to end up in endless discussions about lib names on systems some day i might add this one (if the right lib is found it works ok .. performance on my old laptop is is 5000 https://tug.org pages in 100 sec, 5000 pragma website main pages in 70 sec) (fwiw: i do use curl but just call the binary which in practice is just as efficient because one seldom does a lot in a run) (and i use the socket lib for other things) Hans ----------------------------------------------------------------- Hans Hagen | PRAGMA ADE Ridderstraat 27 | 8061 GH Hasselt | The Netherlands tel: 038 477 53 69 | www.pragma-ade.nl | www.pragma-pod.nl ----------------------------------------------------------------- ___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___________________________________________________________________________________ ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Files on a server 2018-05-17 7:54 ` Taco Hoekwater 2018-05-17 8:30 ` Ulrike Fischer 2018-05-17 8:42 ` Hans Hagen @ 2018-05-17 8:46 ` luigi scarso 2 siblings, 0 replies; 19+ messages in thread From: luigi scarso @ 2018-05-17 8:46 UTC (permalink / raw) To: mailing list for ConTeXt users [-- Attachment #1.1: Type: text/plain, Size: 306 bytes --] On Thu, May 17, 2018 at 9:54 AM, Taco Hoekwater <taco@elvenkind.com> wrote: > > Doing https would need openssl support, which is unlikely to > ever be built into luatex. > > We could provide a ffi wrapper (once ffi is stable) or a swiglib wrapper as module. Something to discuss at meeting. -- luigi [-- Attachment #1.2: Type: text/html, Size: 897 bytes --] [-- Attachment #2: Type: text/plain, Size: 492 bytes --] ___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___________________________________________________________________________________ ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Files on a server 2018-05-17 7:39 ` Taco Hoekwater 2018-05-17 7:52 ` Hans van der Meer 2018-05-17 7:54 ` Taco Hoekwater @ 2018-05-17 8:20 ` Ulrike Fischer 2018-05-17 8:31 ` Taco Hoekwater 2 siblings, 1 reply; 19+ messages in thread From: Ulrike Fischer @ 2018-05-17 8:20 UTC (permalink / raw) To: ntg-context Am Thu, 17 May 2018 09:39:47 +0200 schrieb Taco Hoekwater: >>> Or use luasocket, which is included in the luatex binary: >> But I'm right that this works only with http and not with https? > Works ok for me. Did you test? I tried a few links and always got 301 back regardless if they exist or not and thought due to the discussion two month ago on the luatex list about https content that it doesn't work at all. I know found https://httpbin.org/ and with it the tests worked: https://httpbin.org/html gave 200 and https://httpbin.org/htmlxxxx gave 404. But the answers from "real" websites are so varied and so often don't say the truth that I would't describe it as "works ok" ;-). -- Ulrike Fischer http://www.troubleshooting-tex.de/ ___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___________________________________________________________________________________ ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Files on a server 2018-05-17 8:20 ` Ulrike Fischer @ 2018-05-17 8:31 ` Taco Hoekwater 2018-05-17 9:03 ` Taco Hoekwater 0 siblings, 1 reply; 19+ messages in thread From: Taco Hoekwater @ 2018-05-17 8:31 UTC (permalink / raw) To: Ulrike Fischer, mailing list for ConTeXt users > On 17 May 2018, at 10:20, Ulrike Fischer <news3@nililand.de> wrote: > > Am Thu, 17 May 2018 09:39:47 +0200 schrieb Taco Hoekwater: > >>>> Or use luasocket, which is included in the luatex binary: > >>> But I'm right that this works only with http and not with https? > >> Works ok for me. Did you test? > > I tried a few links and always got 301 back regardless if they exist > or not and thought due to the discussion two month ago on the luatex > list about https content that it doesn't work at all. > > I know found https://httpbin.org/ and with it the tests worked: > https://httpbin.org/html gave 200 and https://httpbin.org/htmlxxxx > gave 404. It only works because it silently discards the https: and replaces it with http: . Actually, it disregards what is in front of // completely, whatever it is. Even: url = "foobar://www.ntg.nl/watistex.html" works just 'fine'. Anyway, perhaps someone can answer me this? I tried the ffi/curl code, and the network stuff works, but only when I comment out the write callback: lcurl.curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, ffi.cast("callback",discard)); If I dont, I get an error message from ffi that is: unable to convert argument 3 from cdata<unsigned long long (*)(void*, unsigned long long, unsigned long long, void*)> to cdata<vararg> stack traceback: [C]: in function 'curl_easy_setopt' [ctxlua]:42: in function 'check_status' [ctxlua]:1: in main chunk And I have no idea what that actually means or how to fix it? Best wishes, Taco ___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___________________________________________________________________________________ ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Files on a server 2018-05-17 8:31 ` Taco Hoekwater @ 2018-05-17 9:03 ` Taco Hoekwater 2018-05-17 9:49 ` Henri Menke 0 siblings, 1 reply; 19+ messages in thread From: Taco Hoekwater @ 2018-05-17 9:03 UTC (permalink / raw) To: mailing list for ConTeXt users; +Cc: Ulrike Fischer > On 17 May 2018, at 10:31, Taco Hoekwater <taco@elvenkind.com> wrote: > > Anyway, perhaps someone can answer me this? I tried the ffi/curl code, > and the network stuff works, but only when I comment out the write > callback: > > lcurl.curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, ffi.cast("callback",discard)); > > If I dont, I get an error message from ffi that is: > > unable to convert argument 3 from cdata<unsigned long long (*)(void*, unsigned long long, unsigned long long, void*)> to cdata<vararg> > stack traceback: > [C]: in function 'curl_easy_setopt' > [ctxlua]:42: in function 'check_status' > [ctxlua]:1: in main chunk > > And I have no idea what that actually means or how to fix it? FYI: fixed by updating luatex. Best wishes, Taco ___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___________________________________________________________________________________ ^ permalink raw reply [flat|nested] 19+ messages in thread
* Re: Files on a server 2018-05-17 9:03 ` Taco Hoekwater @ 2018-05-17 9:49 ` Henri Menke 0 siblings, 0 replies; 19+ messages in thread From: Henri Menke @ 2018-05-17 9:49 UTC (permalink / raw) To: Taco Hoekwater, ntg-context@ntg.nl >> mailing list for ConTeXt users On 05/17/2018 09:03 PM, Taco Hoekwater wrote: > > >> On 17 May 2018, at 10:31, Taco Hoekwater <taco@elvenkind.com> wrote: >> >> Anyway, perhaps someone can answer me this? I tried the ffi/curl code, >> and the network stuff works, but only when I comment out the write >> callback: >> >> lcurl.curl_easy_setopt(curl, CURLOPT_WRITEFUNCTION, ffi.cast("callback",discard)); >> >> If I dont, I get an error message from ffi that is: >> >> unable to convert argument 3 from cdata<unsigned long long (*)(void*, unsigned long long, unsigned long long, void*)> to cdata<vararg> >> stack traceback: >> [C]: in function 'curl_easy_setopt' >> [ctxlua]:42: in function 'check_status' >> [ctxlua]:1: in main chunk >> >> And I have no idea what that actually means or how to fix it? > > FYI: fixed by updating luatex. Hi Taco, Actually you don't really need to set the WRITEFUNCTION callback of libcurl. I only set it to the "discard" function which simply return the number of bytes it got, such that libcurl doesn't print the header information to stdout. You could also plug a different function in there if you wanted to process the headers. I will soon post an annotated version of the code with more proper error-checking on the list. Cheers, Henri > > Best wishes, > Taco > > ___________________________________________________________________________________ > If your question is of interest to others as well, please add an entry to the Wiki! > > maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context > webpage : http://www.pragma-ade.nl / http://context.aanhet.net > archive : https://bitbucket.org/phg/context-mirror/commits/ > wiki : http://contextgarden.net > ___________________________________________________________________________________ > ___________________________________________________________________________________ If your question is of interest to others as well, please add an entry to the Wiki! maillist : ntg-context@ntg.nl / http://www.ntg.nl/mailman/listinfo/ntg-context webpage : http://www.pragma-ade.nl / http://context.aanhet.net archive : https://bitbucket.org/phg/context-mirror/commits/ wiki : http://contextgarden.net ___________________________________________________________________________________ ^ permalink raw reply [flat|nested] 19+ messages in thread
end of thread, other threads:[~2018-05-17 9:49 UTC | newest] Thread overview: 19+ messages (download: mbox.gz / follow: Atom feed) -- links below jump to the message on this page -- 2018-05-16 9:31 Files on a server Hans van der Meer 2018-05-16 10:41 ` Henri Menke 2018-05-16 11:23 ` Hans van der Meer 2018-05-16 11:45 ` Henri Menke 2018-05-16 12:23 ` Taco Hoekwater 2018-05-16 12:58 ` Hans van der Meer 2018-05-16 13:41 ` Hans Hagen 2018-05-16 14:05 ` Taco Hoekwater 2018-05-17 7:32 ` Ulrike Fischer 2018-05-17 7:39 ` Taco Hoekwater 2018-05-17 7:52 ` Hans van der Meer 2018-05-17 7:54 ` Taco Hoekwater 2018-05-17 8:30 ` Ulrike Fischer 2018-05-17 8:42 ` Hans Hagen 2018-05-17 8:46 ` luigi scarso 2018-05-17 8:20 ` Ulrike Fischer 2018-05-17 8:31 ` Taco Hoekwater 2018-05-17 9:03 ` Taco Hoekwater 2018-05-17 9:49 ` Henri Menke
This is a public inbox, see mirroring instructions for how to clone and mirror all data and code used for this inbox; as well as URLs for NNTP newsgroup(s).