edbrowse-dev - development list for edbrowse
 help / color / mirror / Atom feed
* [Edbrowse-dev] the amazon saga
@ 2018-01-08 16:11 Chuck Hallenbeck
  2018-01-09  9:22 ` [Edbrowse-dev] possible Referer issue Kevin Carhart
  0 siblings, 1 reply; 5+ messages in thread
From: Chuck Hallenbeck @ 2018-01-08 16:11 UTC (permalink / raw)
  To: Edbrowse Development

Karl,

Well I have a log for you to look at, and I will too, but first:

I followed your steps, and quickly got the initial raw html size, at 
which point the size of /tmp/log was about 2k with only 54 lines, and it 
stayed that way. The rendered size was never displayed. But when, out of 
curiosity, I pressed "=" and enter, it gave a size of about 5k if I 
recall. It will no doubt be in the log.

Next I pressed enter and got "end of buffer" so I typed 1 enter and got 
an html header line.

Undaunted, I typed b enter, and at first it seemed like nothing 
happened. 
However the log file now showed some 80K lines and a multi MB size, so I 
then examined the contents of my edbrowse session, and I found a good 
looking Amazon opening screen.

I found the usual stuff on line 11, did a g2,
added my email address and password, submitted, and got a segfault.

I suppressed my password with a global replace, and posted the log at:

www.panix.com/~chuxroom/amazon.chuck.5.tmp.log

Now I need a cup of coffee.

Chuck


.


-- 
Here In Northeast Ohio also, The Moon is Waning Gibbous (53% of Full)
When your only tool is a hammer, everything looks like a nail.
Sent from Damon's iPhone.

^ permalink raw reply	[flat|nested] 5+ messages in thread

* [Edbrowse-dev] possible Referer issue
  2018-01-08 16:11 [Edbrowse-dev] the amazon saga Chuck Hallenbeck
@ 2018-01-09  9:22 ` Kevin Carhart
  2018-01-09 13:01   ` Chuck Hallenbeck
  0 siblings, 1 reply; 5+ messages in thread
From: Kevin Carhart @ 2018-01-09  9:22 UTC (permalink / raw)
  To: edbrowse-dev



Hi everyone

Here are the 3 highest-up divergences that I can find between 
amazon.wendy.5 and what I get when the login page rejects me.  Do any of 
these ring a bell?

I'm on edbrowse 3.7.1 with curl 7.32.0


(1) There is an HTTP parameter called prepopulatedLoginId.


In Karl's file, this is empty
prepopulatedLoginId=
In my attempt, it has a value.
prepopulatedLoginId=ape%3AZXlKamFYQm9aWElpT2lKcWJIRnBPR3BJVUdwcE5FaFFWMkpNYWpKWFZsSXJNelpSWkRVNFJsWkxUVVZtWlhsTVVtcFhhRzVSUFNJc0luWmxjbk5wYjI0aU9qRXNJa2xXSWpvaWJ6WndkbkJNV1RKR2JucFlRV2xEYldwNU5tUm1kejA5SW4wPQ%3D%3D

In JS, this parameter is located at 
document.forms[0].prepopulatedLoginId.value

I'm playing around with trying to set this equal to "" in jdb, and it 
seems that I can submit with it empty, but the login still fails.  So 
maybe this parameter is not important.


(2) referer/referrer


In Karl's file, the Referer has parameters appended:
Referer: 
https://www.amazon.com/ap/signin?_encoding=UTF8&ignoreAuthState=1.......

In my attempt, there are no parameters appended:

Referer: https://www.amazon.com/ap/signin

I'm playing with the flag 'sr', for send referrer, but this defaults to 
on, isn't that correct?



(3) Cookies


As the curl actions proceed, the POST is followed by the first response.
In Karl's file, this response includes several cookies that it does not 
send me. 
Karl gets 17.  I only get the 10 empties from the beginning of the list, 
give or take one.  I don't get the substantive session-token and the rest 
from there.

My guess is that these tokens are critical, and once the site declines to 
give them to you, it's all over.


Set-Cookie: ap-fid=""; Domain=.amazon.com; Expires=Thu, 01-Jan-1970 
00:00:10 GMT; Path=/ap/; Secure
Set-Cookie: a-ogbcbff=deleted; Domain=.amazon.com; Expires=Thu, 
01-Jan-1970 00:00:10 GMT; Path=/
Set-Cookie: x-main=""; Domain=.www.amazon.com; Expires=Thu, 01-Jan-1970 
00:00:10 GMT; Path=/; Secure
Set-Cookie: session-id=""; Domain=.www.amazon.com; Expires=Thu, 
01-Jan-1970 00:00:10 GMT; Path=/; Secure
Set-Cookie: session-token=""; Domain=.www.amazon.com; Expires=Thu, 
01-Jan-1970 00:00:10 GMT; Path=/; Secure
Set-Cookie: session-id-time=""; Domain=.www.amazon.com; Expires=Thu, 
01-Jan-1970 00:00:10 GMT; Path=/; Secure
Set-Cookie: ubid-main=""; Domain=.www.amazon.com; Expires=Thu, 01-Jan-1970 
00:00:10 GMT; Path=/; Secure
Set-Cookie: at-main=""; Domain=.www.amazon.com; Expires=Thu, 01-Jan-1970 
00:00:10 GMT; Path=/; Secure
Set-Cookie: sess-at-main=""; Domain=.www.amazon.com; Expires=Thu, 
01-Jan-1970 00:00:10 GMT; Path=/; Secure
Set-Cookie: a-ogbcbff=1; Domain=.amazon.com; Expires=Mon, 08-Jan-2018 
08:02:57 GMT; Path=/
Set-Cookie: ubid-main=134-4084505-6559221; Domain=.amazon.com; 
Expires=Sun, 03-Jan-2038 07:48:57 GMT; Path=/
Set-Cookie: 
session-token="gjPmaaR/cmKiNkYFBaBJqKSWz4HPsmtIVrP4kLL99w4L+jACyaWiMMeEuiRP8IByZ3qWLDUEwdbrJ/RKsMonLoBF7alCjxY/DXxqlucbZIvN3OkQqOsDqqtT8Ct09OpYEFR7XhuylLtshrcp4IFAlpkbb2kC5hl0IPmPzDtOuFOz7a03nqgAKfWXOi29hykqT0Yvq86CWXVY0jMKne7QbLf0yCfSaQ2zyBDzdGbSe5Y="; 
Version=1; Domain=.amazon.com; Max-Age=630720000; Expires=Sun, 03-Jan-2038 
07:48:57 GMT; Path=/
Set-Cookie: x-main="KggLFRexG1hFrP8Wjq6TU7wBs?XDfLl6"; Version=1; 
Domain=.amazon.com; Max-Age=630720000; Expires=Sun, 03-Jan-2038 07:48:57 
GMT; Path=/
Set-Cookie: 
at-main=Atza|IwEBILkpNpji4QB_tDibL7Vbc_LUeNK4T9w58lA5USnBv5bgA0mBkadvOphaccTRtKEcmrMmHFKVzrU-msMjzJ17xv519jdxK7a5O1jvDkZovEz8xp16j65WGxDf0BOcOTVAqK5IUCLcNuvNe4LUEdcQNJWCySxNA6ByiffQmZaLGFeRpENhosW2FLduVkp3Mp9-326IaFRtH0VAf9ZVVQqmwiUsoXu64cDNbz-pWr6Jf-aV6r4jRyq8d3EnQE_rKA2JtAZZx3soV6dVa5NEYDzCrB2Kv3uswYNxDyKCLbHGnKmzb6x0IvxGxVFev2FG2Kqw7uHNAqVcE2HPHARdXCrCosZMKa0TG0mbLm3fXLh_e4ntsNcg9hnffEpPHNKVOEPzfmTlCKP2hlSIvUQLhTMybb-F; 
Domain=.amazon.com; Expires=Sun, 03-Jan-2038 07:48:57 GMT; Path=/; Secure; 
HttpOnly
Set-Cookie: sess-at-main="iXWANpU3F/zV0pkuX+V+jjORKtxLfrHqg8EykI/3fPs="; 
Version=1; Domain=.amazon.com; Path=/; Secure; HttpOnly
Set-Cookie: 
sst-main=Sst1|PQFHdUdY7OLERZydn_YdxKORCGs1qWc0tmUGPPUUFJmpAJI_Yh53qu-Nzqo63t9WUxGegIHjDdglyghZv-xmdWlQtu4Tem-iIUKkETTKkVARNYAhQes9S176PlcxM58a1qi6OowPPgtOIhljiil9hje6wSvAct4ch9VXHE5gdtEI9gVTK5WyHkZga7prg-EMU01w1vfIakm65_1VoEm8KXVw6O0Iq7E2LNaMOU1wrPekXpWTe2st33T7T3jQxpUXQAJPVhk2mwjYxLOeUjhT5fxWmw; 
Domain=.amazon.com; Expires=Sun, 03-Jan-2038 07:48:57 GMT; Path=/; Secure; 
HttpOnly
Set-Cookie: lc-main=en_US; Domain=.amazon.com; Expires=Sun, 03-Jan-2038 
07:48:57 GMT; Path=/




^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [Edbrowse-dev] possible Referer issue
  2018-01-09  9:22 ` [Edbrowse-dev] possible Referer issue Kevin Carhart
@ 2018-01-09 13:01   ` Chuck Hallenbeck
  2018-01-09 22:58     ` Kevin Carhart
  0 siblings, 1 reply; 5+ messages in thread
From: Chuck Hallenbeck @ 2018-01-09 13:01 UTC (permalink / raw)
  To: Kevin Carhart; +Cc: edbrowse-dev

Hi Kevin,

I have amazon.wendy.5 here too, and have posted two db5 runs of my own 
failed logins. I used grep to search for prepopulatedLoginID in all 
three files, and found this:

$ grep -c prepopulatedLoginId amazon*
amazon.chuck.5:0
amazon.wendy.5:16
$ grep -c prepopulatedLoginId signing-in.txt 1


My outputs are available from www.panix.com/~chuxroom/signing-in.txt
and www.panix.com/~chuxroom/amazon.chuck.5

However, the file "signing-in.txt" was generated before the segfault was 
detected and fixed, so the amazon.chuck.5 is probably a closer source.

Running here on Debian Sid, edbrowse 3.7.1, curl 7.57.0.

Chuck


-- 
Here In Northeast Ohio also, The Moon is Waning Crescent (44% of Full)
When your only tool is a hammer, everything looks like a nail.
Sent from Sergio's iPhone.

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [Edbrowse-dev] possible Referer issue
  2018-01-09 13:01   ` Chuck Hallenbeck
@ 2018-01-09 22:58     ` Kevin Carhart
  2018-01-10  2:55       ` Chuck Hallenbeck
  0 siblings, 1 reply; 5+ messages in thread
From: Kevin Carhart @ 2018-01-09 22:58 UTC (permalink / raw)
  To: edbrowse-dev



Thanks, Chuck!  Yes.. breaking down multiple factors is a problem.. not 
knowing which of these point of divergence is the important one.  That 
said, I suspect the cookies, and the session-token cookie in particular.
There needs to be a conversation where it is received, sent back out, and 
if the site gives you a new one, you send that one back in your next 
request.  If I understand what I'm reading, this isn't happening in 
amazon.chuck.5 and in my failed attempts, and it does happen in 
amazon.wendy.5.
The only references to session-token in amazon.chuck.5 are outgoing.
It does not appear in any Response.  I bet this is important.  Of 
course I had a totally wrong hypothesis earlier so I could be wrong.


As a side suggestion, I think we can load the login-and-password screen on 
first visit, and maybe save some hops and some bloat. 
Technically we should be able to open a new edbrowse and 'b' the 
login-and-password screen's URL from the getgo.  I don't think we have to 
visit the homepage and 'g' a link first.  This would make the HTTP 
conversations shorter.






On Tue, 9 Jan 2018, Chuck Hallenbeck wrote:

> Hi Kevin,
>
> I have amazon.wendy.5 here too, and have posted two db5 runs of my own failed 
> logins. I used grep to search for prepopulatedLoginID in all three files, and 
> found this:
>
> $ grep -c prepopulatedLoginId amazon*
> amazon.chuck.5:0
> amazon.wendy.5:16
> $ grep -c prepopulatedLoginId signing-in.txt 1
>
>
> My outputs are available from www.panix.com/~chuxroom/signing-in.txt
> and www.panix.com/~chuxroom/amazon.chuck.5
>
> However, the file "signing-in.txt" was generated before the segfault was 
> detected and fixed, so the amazon.chuck.5 is probably a closer source.
>
> Running here on Debian Sid, edbrowse 3.7.1, curl 7.57.0.
>
> Chuck
>
>
> -- 
> Here In Northeast Ohio also, The Moon is Waning Crescent (44% of Full)
> When your only tool is a hammer, everything looks like a nail.
> Sent from Sergio's iPhone.
>

--------
Kevin Carhart * 415 225 5306 * The Ten Ninety Nihilists

^ permalink raw reply	[flat|nested] 5+ messages in thread

* Re: [Edbrowse-dev] possible Referer issue
  2018-01-09 22:58     ` Kevin Carhart
@ 2018-01-10  2:55       ` Chuck Hallenbeck
  0 siblings, 0 replies; 5+ messages in thread
From: Chuck Hallenbeck @ 2018-01-10  2:55 UTC (permalink / raw)
  To: Kevin Carhart; +Cc: edbrowse-dev

Hi Kevin,

I looked at how one gets from the Amazon opening screen to the
sign-in screen by using the capital A command on line 11, and it's
not pretty. I'm not up to tackling that shortcut myself.

About cookies, in my debugging output I see references to expired
cookies, which seems strange.  But I have another concern:

It appears that issuing the db5 command and diverting the resulting
output to a log file changes the behavior of edbrowse.  Let's consider
two cases, first with db1, which is my default in the rcfile.
and second with db5, and the command to redirect  output to a log
file before opening amazon.com.

Cases 1 and 2: Open edbrowse and get a prompt.

Case 2 only: enter db5, then db>/tmp/log

Cases 1 and 2. Enter e www.amazon.com, and get the raw html size,
usually about 400000,

In case 1, but not case 2, we soon get the rendered size, a much
smaller number, and the opening screen is ready to be used.

In case 2, the buffer contains only the raw html, and we must type
the b command to render it, and only then do we see the smaller size
and can use the opening screen.

In both case 1 and case 2, we go to line 11, which contains five
links, we enter g2 to get to the sign-in screen, where we enter our
ID and Password.

For Case 1 only, we press submit and are taken to a new screen
resembling the sign-in screen, except for the messages near the top
saying "An Error Has Occurred" and "Enter a valid email address or
phone number" or some such wording.

The ID you previously typed is shown, but the password field is empty.
That was for case 1 only.

For case 2, pressing the submit button on the sign-in page causes
a segfault.

I would think the program behavior should not be changed by activating
a higher debug level, but the above is consistent and reproducible, and
makes me wonder how reliable the data in the resulting log file are.

Chuck



-- 
Here In Northeast Ohio also, The Moon is Waning Crescent (38% of Full)
When your only tool is a hammer, everything looks like a nail.
Sent from Camille's iPhone.

^ permalink raw reply	[flat|nested] 5+ messages in thread

end of thread, other threads:[~2018-01-10  2:52 UTC | newest]

Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-01-08 16:11 [Edbrowse-dev] the amazon saga Chuck Hallenbeck
2018-01-09  9:22 ` [Edbrowse-dev] possible Referer issue Kevin Carhart
2018-01-09 13:01   ` Chuck Hallenbeck
2018-01-09 22:58     ` Kevin Carhart
2018-01-10  2:55       ` Chuck Hallenbeck

This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).