[Edbrowse-dev] startpage

[Edbrowse-dev] startpage

[Karl Dahlke]

I pushed the aforementioned change to comment out my line of code,
because clearly it must not run on this website,
and perhaps it shouldn't run at all.
Startpage with db3 now sends your request with no js errors,
but I always get the "overuse" page back instead of the google response.
Don't know why.

Karl Dahlke

Re: [Edbrowse-dev] startpage

[Chris Brannon]

Karl Dahlke <eklhad@comcast.net> writes:

> I pushed the aforementioned change to comment out my line of code,

Nope, sorry, that dog won't hunt.
Here's a test page:
http://the-brannons.com/form.html
I fetched this in Chromium, setting the text field to "hello, world" and
submitted.
document.writeln wrote the line
"You said: hello, world".
When I view it with edbrowse after applying your recent change, I get an
error.  document.someform.random is undefined.

I don't know how mainstream browsers deal with this problem, either,
but I think we'll probably have to do something like what Adam
mentioned.  In domLink, refuse to overwrite owner.id if owner.id points
to a standard method.
I know how to implement this, if we go that route.

PS.  On another topic, I'm running edbrowse with mozjs-24 as packaged by
Gentoo.  So at least one distribution's package works with what we have.

-- Chris

Re: [Edbrowse-dev] startpage

[Chuck Hallenbeck]

Chris,

Yes, ua1 identifies me as a lynx browser, it lets me in, and the submit
button works as expected. Below is the script:


Script started on Mon 10 Feb 2014 07:34:05 PM EST
[radio@dj ~]$ e
edbrowse ready
ua1
Lynx/2.8.4rel.1 libwww-FM/2.14
db4\b^[[K\b^[[K\r^[[Kb www.startpage.com
33074
1550
/<>
<><Go secure>
i=small squirrels
<small squirrels><Go secure>
db4
i*
* https://www.startpage.com/do/search
submitting form
post cat=web&cmd=process_search&language=english_uk&engine0=v1all&query=small+squirrels&abp=-1
redirect https://www.startpage.com/do/search\x01cat=web&cmd=process_search&language=english_uk&engine0=v1all&query=small+squirrels&abp=-1
curl>
POST /do/search HTTP/1.1
User-Agent: Lynx/2.8.4rel.1 libwww-FM/2.14
Host: www.startpage.com
Accept: */*
Accept-Encoding: deflate, gzip
Accept-Language: en
Content-Length: 89
Content-Type: application/x-www-form-urlencoded

curl<
HTTP/1.1 200 OK
Date: Tue, 11 Feb 2014 00:35:50 GMT
Server: Apache
Content-language: en-gb
Content-Type: text/html
Transfer-Encoding: chunked

.
http code 200
text type is ascii
105059
base href https://startpage.com/
java source https://startpage.com/js/jquery-1.8.0.min.js?changed=1392022869
curl>
GET /js/jquery-1.8.0.min.js?changed=1392022869 HTTP/1.1
User-Agent: Lynx/2.8.4rel.1 libwww-FM/2.14
Host: startpage.com
Accept: */*
Accept-Encoding: deflate, gzip
Accept-Language: en

curl<
HTTP/1.1 200 OK
Date: Tue, 11 Feb 2014 00:35:52 GMT
Server: Apache
Last-Modified: Mon, 10 Feb 2014 09:01:09 GMT
ETag: "2c02d9-16ba2-4f20994a03f40"
Vary: Accept-Encoding
Content-Encoding: gzip
Strict-Transport-Security: max-age=5
Cache-Control: max-age=3456000
Content-Length: 33136
Content-Type: application/javascript

http code 200
execute jquery-1.8.0.min.js?changed=1392022869 at 1
jquery-1.8.0.min.js?changed=1392022869: 2: TypeError: n.setAttribute is not a function
execution complete
java source https://startpage.com/js/parser.js?changed=1392022891
curl>
GET /js/parser.js?changed=1392022891 HTTP/1.1
User-Agent: Lynx/2.8.4rel.1 libwww-FM/2.14
Host: startpage.com
Accept: */*
Accept-Encoding: deflate, gzip
Accept-Language: en

curl<
HTTP/1.1 200 OK
Date: Tue, 11 Feb 2014 00:35:54 GMT
Server: Apache
Last-Modified: Mon, 10 Feb 2014 09:01:31 GMT
ETag: "2c02e3-311e-4f20995eff0c0"
Vary: Accept-Encoding
Content-Encoding: gzip
Strict-Transport-Security: max-age=5
Cache-Control: max-age=3456000
Content-Length: 3507
Content-Type: application/javascript

http code 200
execute parser.js?changed=1392022891 at 1
execution complete
java source https://startpage.com/js/calculator.js?changed=1392022873
curl>
GET /js/calculator.js?changed=1392022873 HTTP/1.1
User-Agent: Lynx/2.8.4rel.1 libwww-FM/2.14
Host: startpage.com
Accept: */*
Accept-Encoding: deflate, gzip
Accept-Language: en

curl<
HTTP/1.1 200 OK
Date: Tue, 11 Feb 2014 00:35:54 GMT
Server: Apache
Last-Modified: Mon, 10 Feb 2014 09:01:13 GMT
ETag: "2c02bc-651-4f20994dd4840"
Vary: Accept-Encoding
Content-Encoding: gzip
Strict-Transport-Security: max-age=5
Cache-Control: max-age=3456000
Content-Length: 714
Content-Type: application/javascript

http code 200
execute calculator.js?changed=1392022873 at 1
execution complete
java source https://startpage.com/js/imagesloaded.js?changed=1392022873
curl>
GET /js/imagesloaded.js?changed=1392022873 HTTP/1.1
User-Agent: Lynx/2.8.4rel.1 libwww-FM/2.14
Host: startpage.com
Accept: */*
Accept-Encoding: deflate, gzip
Accept-Language: en

curl<
HTTP/1.1 200 OK
Date: Tue, 11 Feb 2014 00:35:55 GMT
Server: Apache
Last-Modified: Mon, 10 Feb 2014 09:01:13 GMT
ETag: "2c02d3-1c1d-4f20994dd4840"
Vary: Accept-Encoding
Content-Encoding: gzip
Strict-Transport-Security: max-age=5
Cache-Control: max-age=3456000
Content-Length: 2488
Content-Type: application/javascript

http code 200
execute imagesloaded.js?changed=1392022873 at 1
execution complete
java source https://startpage.com/js/resize.js?changed=1392022871
curl>
GET /js/resize.js?changed=1392022871 HTTP/1.1
User-Agent: Lynx/2.8.4rel.1 libwww-FM/2.14
Host: startpage.com
Accept: */*
Accept-Encoding: deflate, gzip
Accept-Language: en

curl<
HTTP/1.1 200 OK
Date: Tue, 11 Feb 2014 00:35:55 GMT
Server: Apache
Last-Modified: Mon, 10 Feb 2014 09:01:11 GMT
ETag: "2c02e8-3a8-4f20994bec3c0"
Vary: Accept-Encoding
Content-Encoding: gzip
Strict-Transport-Security: max-age=5
Cache-Control: max-age=3456000
Content-Length: 351
Content-Type: application/javascript

http code 200
execute resize.js?changed=1392022871 at 1
execution complete
java source https://startpage.com/js/swfobject.js?changed=1392022878
curl>
GET /js/swfobject.js?changed=1392022878 HTTP/1.1
User-Agent: Lynx/2.8.4rel.1 libwww-FM/2.14
Host: startpage.com
Accept: */*
Accept-Encoding: deflate, gzip
Accept-Language: en

curl<
HTTP/1.1 200 OK
Date: Tue, 11 Feb 2014 00:35:56 GMT
Server: Apache
Last-Modified: Mon, 10 Feb 2014 09:01:18 GMT
ETag: "2c02ed-1596-4f20995299380"
Vary: Accept-Encoding
Content-Encoding: gzip
Strict-Transport-Security: max-age=5
Cache-Control: max-age=3456000
Content-Length: 1822
Content-Type: application/javascript

http code 200
execute swfobject.js?changed=1392022878 at 1
execution complete
java source https://startpage.com/js/image_helper_new.js?changed=1392022863
curl>
GET /js/image_helper_new.js?changed=1392022863 HTTP/1.1
User-Agent: Lynx/2.8.4rel.1 libwww-FM/2.14
Host: startpage.com
Accept: */*
Accept-Encoding: deflate, gzip
Accept-Language: en

curl<
HTTP/1.1 200 OK
Date: Tue, 11 Feb 2014 00:35:56 GMT
Server: Apache
Last-Modified: Mon, 10 Feb 2014 09:01:03 GMT
ETag: "2c02d2-1029-4f2099444b1c0"
Vary: Accept-Encoding
Content-Encoding: gzip
Strict-Transport-Security: max-age=5
Cache-Control: max-age=3456000
Content-Length: 1196
Content-Type: application/javascript

http code 200
execute image_helper_new.js?changed=1392022863 at 1
execution complete
java source https://startpage.com/js/ccspacer.js?w=m
curl>
GET /js/ccspacer.js?w=m HTTP/1.1
User-Agent: Lynx/2.8.4rel.1 libwww-FM/2.14
Host: startpage.com
Accept: */*
Accept-Encoding: deflate, gzip
Accept-Language: en

curl<
HTTP/1.1 200 OK
Date: Tue, 11 Feb 2014 00:35:57 GMT
Server: Apache
Last-Modified: Mon, 10 Feb 2014 09:22:36 GMT
ETag: "2c02bd-6c3-4f209e15b2ced"
Vary: Accept-Encoding
Content-Encoding: gzip
Strict-Transport-Security: max-age=5
Cache-Control: max-age=3456000
Content-Length: 704
Content-Type: application/javascript

http code 200
execute ccspacer.js?w=m at 1
ccspacer.js?w=m: 1: TypeError: can't redefine non-configurable property 'cookie'
execution complete
execute search\x01cat=web&cmd=process_search&language=english_uk&engine0=v1all&query=small+squirrels&abp=-1 at 18
search\x01cat=web&cmd=process_search&language=english_uk&engine0=v1all&query=small+squirrels&abp=-1: 18: ReferenceError: jQuery is not defined
execution complete
execute search\x01cat=web&cmd=process_search&language=english_uk&engine0=v1all&query=small+squirrels&abp=-1 at 67
search\x01cat=web&cmd=process_search&language=english_uk&engine0=v1all&query=small+squirrels&abp=-1: 18: TypeError: i_ob is undefined
execution complete
execute search\x01cat=web&cmd=process_search&language=english_uk&engine0=v1all&query=small+squirrels&abp=-1 at 88
execution complete
execute search\x01cat=web&cmd=process_search&language=english_uk&engine0=v1all&query=small+squirrels&abp=-1 at 130
execution complete
execute search\x01cat=web&cmd=process_search&language=english_uk&engine0=v1all&query=small+squirrels&abp=-1 at 177
execution complete
execute search\x01cat=web&cmd=process_search&language=english_uk&engine0=v1all&query=small+squirrels&abp=-1 at 179
execution complete
execute search\x01cat=web&cmd=process_search&language=english_uk&engine0=v1all&query=small+squirrels&abp=-1 at 181
execution complete
execute search\x01cat=web&cmd=process_search&language=english_uk&engine0=v1all&query=small+squirrels&abp=-1 at 183
execution complete
execute search\x01cat=web&cmd=process_search&language=english_uk&engine0=v1all&query=small+squirrels&abp=-1 at 230
execution complete
execute search\x01cat=web&cmd=process_search&language=english_uk&engine0=v1all&query=small+squirrels&abp=-1 at 232
execution complete
execute search\x01cat=web&cmd=process_search&language=english_uk&engine0=v1all&query=small+squirrels&abp=-1 at 234
execution complete
execute search\x01cat=web&cmd=process_search&language=english_uk&engine0=v1all&query=small+squirrels&abp=-1 at 249
execution complete
docwrite 1563 bytes
<<
<hr>
<form action="https://s18-us2.startpage.com/do/search" id="pnform" name="pnform" method="POST" style="display: inline; width: 30px;" align='center'> <input type="hidden" name="cmd" value="process_search" /> <input type="hidden" name="language" value="english_uk" /> <input type="hidden" name="qid" value="NGLNTMKRSTPK" /> <input type="hidden" name="rcount" value="" /> <input type="hidden" name="rl" value="NONE" /> <input type="hidden" name="abp" class='abp' value="-1" /> <input type="hidden" name="query" value="small squirrels" /> <input type="hidden" name="cat" value="web" /> <input type="hidden" name="engine0" value="v1all"  /> <input id='startat' type="hidden" name="startat" value="" /> &nbsp;1&nbsp;<a href='javascript:;' class='numbers_st' onclick='mysubmit(10); return false;' id='2'>2</a><a href='javascript:;' class='numbers_st' onclick='mysubmit(20); return false;' id='3'>3</a><a href='javascript:;' class='numbers_st' onclick='mysubmit(30); return false;' id='4'>4</a><a href='jav
 ascript:;' class='numbers_st' onclick='mysubmit(40); return false;' id='5'>5</a><a href='javascript:;' class='numbers_st' onclick='mysubmit(50); return false;' id='6'>6</a><a href='javascript:;' class='numbers_st' onclick='mysubmit(60); return false;' id='7'>7</a><a href='javascript:;' class='numbers_st' onclick='mysubmit(70); return false;' id='8'>8</a><a href='javascript:;' class='numbers_st' onclick='mysubmit(80); return false;' id='9'>9</a><a href='javascript:;' class='numbers_st' onclick='mysubmit(90); return false;' id='10'>10</a></form>

>>
execute search\x01cat=web&cmd=process_search&language=english_uk&engine0=v1all&query=small+squirrels&abp=-1 at 367
execution complete
docwrite 821 bytes
<<
<hr>
<form action="https://s18-us2.startpage.com/do/search" method="POST" style="display: inline; width: 85px;" name='nextform' align='left'> <input type="hidden" name="cmd" value="process_search" /> <input type="hidden" name="startat" value="10" /> <input type="hidden" name="language" value="english_uk" /> <input type="hidden" name="qid" value="NGLNTMKRSTPK" /> <input type="hidden" name="rcount" value="" /> <input type="hidden" name="rl" value="NONE" /> <input type="hidden" name="abp" class='abp' value="-1" /> <input type="hidden" name="query" value="small squirrels" /> <input type="hidden" name="cat" value="web" /> <input type="hidden" name="engine0" value="v1all"  /> <a href='javascript:document.nextform.submit();' class='numbers_st' style='width:200px; text-align:left;'>Next &gt;&gt;</a> </form>

>>
execute search\x01cat=web&cmd=process_search&language=english_uk&engine0=v1all&query=small+squirrels&abp=-1 at 409
execution complete
execute search\x01cat=web&cmd=process_search&language=english_uk&engine0=v1all&query=small+squirrels&abp=-1 at 413
search\x01cat=web&cmd=process_search&language=english_uk&engine0=v1all&query=small+squirrels&abp=-1: 419: TypeError: all_links[i].className is undefined
execution complete
execute search\x01cat=web&cmd=process_search&language=english_uk&engine0=v1all&query=small+squirrels&abp=-1 at 579
search\x01cat=web&cmd=process_search&language=english_uk&engine0=v1all&query=small+squirrels&abp=-1: 580: ReferenceError: jQuery is not defined
execution complete
java source https://startpage.com/js/abp.js?adType=1&advertiser=1&advertising=1&changed=1392022892
curl>
GET /js/abp.js?adType=1&advertiser=1&advertising=1&changed=1392022892 HTTP/1.1
User-Agent: Lynx/2.8.4rel.1 libwww-FM/2.14
Host: startpage.com
Accept: */*
Accept-Encoding: deflate, gzip
Accept-Language: en

curl<
HTTP/1.1 200 OK
Date: Tue, 11 Feb 2014 00:35:58 GMT
Server: Apache
Last-Modified: Mon, 10 Feb 2014 09:01:32 GMT
ETag: "2c02b9-a-4f20995ff3300"
Vary: Accept-Encoding
Content-Encoding: gzip
Strict-Transport-Security: max-age=5
Cache-Control: no-cache, no-store, must-revalidate
Content-Length: 30
Content-Type: application/javascript

http code 200
execute abp.js?adType=1&advertiser=1&advertising=1&changed=1392022892 at 1
execution complete
java source https://startpage.com/js/wz_tooltip.js?changed=1392022872
curl>
GET /js/wz_tooltip.js?changed=1392022872 HTTP/1.1
User-Agent: Lynx/2.8.4rel.1 libwww-FM/2.14
Host: startpage.com
Accept: */*
Accept-Encoding: deflate, gzip
Accept-Language: en

curl<
HTTP/1.1 200 OK
Date: Tue, 11 Feb 2014 00:35:58 GMT
Server: Apache
Last-Modified: Mon, 10 Feb 2014 09:01:12 GMT
ETag: "2c02f1-4952-4f20994ce0600"
Vary: Accept-Encoding
Content-Encoding: gzip
Strict-Transport-Security: max-age=5
Cache-Control: max-age=3456000
Content-Length: 6246
Content-Type: application/javascript

http code 200
execute wz_tooltip.js?changed=1392022872 at 1
execution complete
anchorSwap 1
anchors unframed
whitespace combined
6740
=
279
/squirrels/
 <small squirrels> | <Go secure> |

enhanced by

|{Details}











About 1,579,958 results (0.25 seconds)



x









{Startpage does NOT record your IP address!}

{Bookmark this search}
-
{Startpage does NOT record your IP address!}

{Bookmark this search}









* {Any time}

* {Past 24 hours}

* {Past week}

* {Past month}

* {Past year}



* {Web}

* {Pages from the UK}

NEW!

Search tools

{(change settings)}







{See your ad here...}

{Ads related to small squirrels}

1.



{Mini Squirrels at Walmart}



{www.walmart.com/Mini-Squirrels} Shop For Mini Squirrels!

Save Money. Live Better.



walmart.com is rated {47 reviews}
w search.txt
6740
1\r^[[Kq
[radio@dj ~]$ exit

Script done on Mon 10 Feb 2014 07:38:37 PM EST


-- 

 Chuck in Ghent, northeast of Hudson on the Hudson.

Re: [Edbrowse-dev] startpage

[Chris Brannon]

Chuck Hallenbeck <chuckh@ftml.net> writes:

> Can you use elinks for this purpose? Elinks works on startpage.

Unfortunately not, because elinks will not show me the full POST query
or write it to a file.
However, it turns out that startpage is also doing user-agent detection.
Do you have a user-agent definition for lynx in your .ebrc?  It's in the
default config, so I'm guessing that ua1 will set your user-agent string
to lynx.
They like lynx just fine!
Turns out that they're actively denying form submissions that contain
edbrowse in the user-agent string.  I have a third user agent in my
.ebrc, defined as "nunya business", and they like that one just fine as
well.

-- Chris

Re: [Edbrowse-dev] startpage

[Chuck Hallenbeck]

Can you use elinks for this purpose? Elinks works on startpage.

On Mon, 10 Feb 2014, Chris Brannon wrote:

> Karl Dahlke <eklhad@comcast.net> writes:
>
> > There is something else amiss about the way we submit that query.
>
> I get the same thing, even when JavaScript is disabled.
> My guess is that they're expecting some form field that we're not
> sending.  It's supposed to be filled in by JavaScript, and if we don't
> send it, they assume that we're using some kind of automation to access
> their site.  Automation usually isn't "smart" enough to run JavaScript.
> And our JS implementation is hitting a snag somewhere, failing to fill
> in that field.
> I'd try this in Chromium with ChromeVox, but I have no way to see what
> query the browser is sending to the server.  If this were a plain http
> connection, I'd dump the traffic to a file with tcpdump and extract what
> I need.  But it's https.  I'm looking into mitmproxy, which may help me
> to examine the traffic.
>
> -- Chris
> _______________________________________________
> Edbrowse-dev mailing list
> Edbrowse-dev@lists.the-brannons.com
> http://lists.the-brannons.com/mailman/listinfo/edbrowse-dev
>

-- 

 Chuck in Ghent, northeast of Hudson on the Hudson.

Re: [Edbrowse-dev] startpage

[Chris Brannon]

Karl Dahlke <eklhad@comcast.net> writes:

> There is something else amiss about the way we submit that query.

I get the same thing, even when JavaScript is disabled.
My guess is that they're expecting some form field that we're not
sending.  It's supposed to be filled in by JavaScript, and if we don't
send it, they assume that we're using some kind of automation to access
their site.  Automation usually isn't "smart" enough to run JavaScript.
And our JS implementation is hitting a snag somewhere, failing to fill
in that field.
I'd try this in Chromium with ChromeVox, but I have no way to see what
query the browser is sending to the server.  If this were a plain http
connection, I'd dump the traffic to a file with tcpdump and extract what
I need.  But it's https.  I'm looking into mitmproxy, which may help me
to examine the traffic.

-- Chris

Re: [Edbrowse-dev] startpage

[Chuck Hallenbeck]

I can't get onto that site any more. Here is what happens when I try.  The
references to Whittier Place, or Whittier Health Center, are references to
the wifi service here, my only access to the net at present.

Script started on Mon 10 Feb 2014 03:29:54 PM EST
[chuxroom@dj ~]$ e
edbrowse ready
db4
b www.smartpage.com
curl>
GET / HTTP/1.1
User-Agent: edbrowse/3.5.1
Host: www.smartpage.com
Accept: */*
Accept-Encoding: deflate, gzip

curl<
HTTP/1.0 200 OK
Date: Mon, 10 Feb 2014 20:30:31 GMT
Content-Length: 94
X-Powered-By: Servlet/2.4 JSP/2.0
X-Cache: MISS from ghentbarracuda.whittierhealth.com
Via: 1.0 ghentbarracuda.whittierhealth.com:8080 (http_scan/4.0.2.6.19)
Connection: close

http code 200
text type is ascii
94
anchorSwap 1
anchors unframed
whitespace combined
1
=
1
.

ub
94
1
<html><head><title></title><meta name="revised" content="1.1.7" /></head><body></body></html>
q
[chuxroom@dj ~]$ exit

Script done on Mon 10 Feb 2014 03:34:09 PM EST


-- 

 Chuck in Ghent, northeast of Hudson on the Hudson.

[Edbrowse-dev] startpage

[Karl Dahlke]

You aren't being blocked per se; I get the same message,
and I don't believe in coincidences.
There is something else amiss about the way we submit that query.
Probably should run it with db4,
especially noting the redirection to smartpage,
and compare that with what happens in another browser.

Karl Dahlke

[Edbrowse-dev] startpage

[Karl Dahlke]

> It isn't the behavior I see in Chromium, where a form element with
> id=submit does not mask document.form.submit, but a form element with
> id=foo is available as document.form.foo.

Then that is the behavior I should probably replicate, i.e. create
the named link unless it is submit, reset, or action.
I'm pretty sure I wrote that code seven years ago for a reason.
So I'll make that change, with comments, and push.

Here is how I think it all evolves.
Some website that is really popular, like facebook, for example,
has some bogus html or bogus javascript.
Firefox, for example, doesn't want to be the new browwser on the block
that can't run facebook.
So they deliberately program around the bogus js or html code in facebook.
And that gives facebook no insentive to clean up their code.
It just goes on, across many websites that are too big to fail.
Over two decades, all these little exceptions are programmed into
all the browsers out there, and they aren't documented anywhere in
any reliable way,
so if we're serious about edbrowse we have to rediscover all these exceptions
and program them into our DOM.
As Charlie Brown says, oh good grief.

Karl Dahlke

Re: [Edbrowse-dev] startpage

[Chris Brannon]

Karl Dahlke <eklhad@comcast.net> writes:

> Yes I know, exactly, question is, do real web pages do that?

I don't know.  I've been looking for live examples, but I haven't found
one yet.
But here's some text I found that mentions the submit method of form
objects:

"If a form control (such as a submit button)
has a name or id of submit it will mask the form's submit method."
Source: https://developer.mozilla.org/en-US/docs/Web/API/HTMLFormElement.submit

It isn't the behavior I see in Chromium, where a form element with
id=submit does not mask document.form.submit, but a form element with
id=foo is available as document.form.foo.

I also need to get a handle on the DOM specifications.

Wow, what a mess the web is.  I'm amazed it holds together as well as it does.

-- Chris

[Edbrowse-dev] startpage

[Karl Dahlke]

> Here's a test page:

Yes I know, exactly, question is, do real web pages do that?

Karl Dahlke