* [Edbrowse-dev] the amazon saga
@ 2018-01-08 16:11 Chuck Hallenbeck
2018-01-09 9:22 ` [Edbrowse-dev] possible Referer issue Kevin Carhart
0 siblings, 1 reply; 5+ messages in thread
From: Chuck Hallenbeck @ 2018-01-08 16:11 UTC (permalink / raw)
To: Edbrowse Development
Karl,
Well I have a log for you to look at, and I will too, but first:
I followed your steps, and quickly got the initial raw html size, at
which point the size of /tmp/log was about 2k with only 54 lines, and it
stayed that way. The rendered size was never displayed. But when, out of
curiosity, I pressed "=" and enter, it gave a size of about 5k if I
recall. It will no doubt be in the log.
Next I pressed enter and got "end of buffer" so I typed 1 enter and got
an html header line.
Undaunted, I typed b enter, and at first it seemed like nothing
happened.
However the log file now showed some 80K lines and a multi MB size, so I
then examined the contents of my edbrowse session, and I found a good
looking Amazon opening screen.
I found the usual stuff on line 11, did a g2,
added my email address and password, submitted, and got a segfault.
I suppressed my password with a global replace, and posted the log at:
www.panix.com/~chuxroom/amazon.chuck.5.tmp.log
Now I need a cup of coffee.
Chuck
.
--
Here In Northeast Ohio also, The Moon is Waning Gibbous (53% of Full)
When your only tool is a hammer, everything looks like a nail.
Sent from Damon's iPhone.
^ permalink raw reply [flat|nested] 5+ messages in thread
* [Edbrowse-dev] possible Referer issue
2018-01-08 16:11 [Edbrowse-dev] the amazon saga Chuck Hallenbeck
@ 2018-01-09 9:22 ` Kevin Carhart
2018-01-09 13:01 ` Chuck Hallenbeck
0 siblings, 1 reply; 5+ messages in thread
From: Kevin Carhart @ 2018-01-09 9:22 UTC (permalink / raw)
To: edbrowse-dev
Hi everyone
Here are the 3 highest-up divergences that I can find between
amazon.wendy.5 and what I get when the login page rejects me. Do any of
these ring a bell?
I'm on edbrowse 3.7.1 with curl 7.32.0
(1) There is an HTTP parameter called prepopulatedLoginId.
In Karl's file, this is empty
prepopulatedLoginId=
In my attempt, it has a value.
prepopulatedLoginId=ape%3AZXlKamFYQm9aWElpT2lKcWJIRnBPR3BJVUdwcE5FaFFWMkpNYWpKWFZsSXJNelpSWkRVNFJsWkxUVVZtWlhsTVVtcFhhRzVSUFNJc0luWmxjbk5wYjI0aU9qRXNJa2xXSWpvaWJ6WndkbkJNV1RKR2JucFlRV2xEYldwNU5tUm1kejA5SW4wPQ%3D%3D
In JS, this parameter is located at
document.forms[0].prepopulatedLoginId.value
I'm playing around with trying to set this equal to "" in jdb, and it
seems that I can submit with it empty, but the login still fails. So
maybe this parameter is not important.
(2) referer/referrer
In Karl's file, the Referer has parameters appended:
Referer:
https://www.amazon.com/ap/signin?_encoding=UTF8&ignoreAuthState=1.......
In my attempt, there are no parameters appended:
Referer: https://www.amazon.com/ap/signin
I'm playing with the flag 'sr', for send referrer, but this defaults to
on, isn't that correct?
(3) Cookies
As the curl actions proceed, the POST is followed by the first response.
In Karl's file, this response includes several cookies that it does not
send me.
Karl gets 17. I only get the 10 empties from the beginning of the list,
give or take one. I don't get the substantive session-token and the rest
from there.
My guess is that these tokens are critical, and once the site declines to
give them to you, it's all over.
Set-Cookie: ap-fid=""; Domain=.amazon.com; Expires=Thu, 01-Jan-1970
00:00:10 GMT; Path=/ap/; Secure
Set-Cookie: a-ogbcbff=deleted; Domain=.amazon.com; Expires=Thu,
01-Jan-1970 00:00:10 GMT; Path=/
Set-Cookie: x-main=""; Domain=.www.amazon.com; Expires=Thu, 01-Jan-1970
00:00:10 GMT; Path=/; Secure
Set-Cookie: session-id=""; Domain=.www.amazon.com; Expires=Thu,
01-Jan-1970 00:00:10 GMT; Path=/; Secure
Set-Cookie: session-token=""; Domain=.www.amazon.com; Expires=Thu,
01-Jan-1970 00:00:10 GMT; Path=/; Secure
Set-Cookie: session-id-time=""; Domain=.www.amazon.com; Expires=Thu,
01-Jan-1970 00:00:10 GMT; Path=/; Secure
Set-Cookie: ubid-main=""; Domain=.www.amazon.com; Expires=Thu, 01-Jan-1970
00:00:10 GMT; Path=/; Secure
Set-Cookie: at-main=""; Domain=.www.amazon.com; Expires=Thu, 01-Jan-1970
00:00:10 GMT; Path=/; Secure
Set-Cookie: sess-at-main=""; Domain=.www.amazon.com; Expires=Thu,
01-Jan-1970 00:00:10 GMT; Path=/; Secure
Set-Cookie: a-ogbcbff=1; Domain=.amazon.com; Expires=Mon, 08-Jan-2018
08:02:57 GMT; Path=/
Set-Cookie: ubid-main=134-4084505-6559221; Domain=.amazon.com;
Expires=Sun, 03-Jan-2038 07:48:57 GMT; Path=/
Set-Cookie:
session-token="gjPmaaR/cmKiNkYFBaBJqKSWz4HPsmtIVrP4kLL99w4L+jACyaWiMMeEuiRP8IByZ3qWLDUEwdbrJ/RKsMonLoBF7alCjxY/DXxqlucbZIvN3OkQqOsDqqtT8Ct09OpYEFR7XhuylLtshrcp4IFAlpkbb2kC5hl0IPmPzDtOuFOz7a03nqgAKfWXOi29hykqT0Yvq86CWXVY0jMKne7QbLf0yCfSaQ2zyBDzdGbSe5Y=";
Version=1; Domain=.amazon.com; Max-Age=630720000; Expires=Sun, 03-Jan-2038
07:48:57 GMT; Path=/
Set-Cookie: x-main="KggLFRexG1hFrP8Wjq6TU7wBs?XDfLl6"; Version=1;
Domain=.amazon.com; Max-Age=630720000; Expires=Sun, 03-Jan-2038 07:48:57
GMT; Path=/
Set-Cookie:
at-main=Atza|IwEBILkpNpji4QB_tDibL7Vbc_LUeNK4T9w58lA5USnBv5bgA0mBkadvOphaccTRtKEcmrMmHFKVzrU-msMjzJ17xv519jdxK7a5O1jvDkZovEz8xp16j65WGxDf0BOcOTVAqK5IUCLcNuvNe4LUEdcQNJWCySxNA6ByiffQmZaLGFeRpENhosW2FLduVkp3Mp9-326IaFRtH0VAf9ZVVQqmwiUsoXu64cDNbz-pWr6Jf-aV6r4jRyq8d3EnQE_rKA2JtAZZx3soV6dVa5NEYDzCrB2Kv3uswYNxDyKCLbHGnKmzb6x0IvxGxVFev2FG2Kqw7uHNAqVcE2HPHARdXCrCosZMKa0TG0mbLm3fXLh_e4ntsNcg9hnffEpPHNKVOEPzfmTlCKP2hlSIvUQLhTMybb-F;
Domain=.amazon.com; Expires=Sun, 03-Jan-2038 07:48:57 GMT; Path=/; Secure;
HttpOnly
Set-Cookie: sess-at-main="iXWANpU3F/zV0pkuX+V+jjORKtxLfrHqg8EykI/3fPs=";
Version=1; Domain=.amazon.com; Path=/; Secure; HttpOnly
Set-Cookie:
sst-main=Sst1|PQFHdUdY7OLERZydn_YdxKORCGs1qWc0tmUGPPUUFJmpAJI_Yh53qu-Nzqo63t9WUxGegIHjDdglyghZv-xmdWlQtu4Tem-iIUKkETTKkVARNYAhQes9S176PlcxM58a1qi6OowPPgtOIhljiil9hje6wSvAct4ch9VXHE5gdtEI9gVTK5WyHkZga7prg-EMU01w1vfIakm65_1VoEm8KXVw6O0Iq7E2LNaMOU1wrPekXpWTe2st33T7T3jQxpUXQAJPVhk2mwjYxLOeUjhT5fxWmw;
Domain=.amazon.com; Expires=Sun, 03-Jan-2038 07:48:57 GMT; Path=/; Secure;
HttpOnly
Set-Cookie: lc-main=en_US; Domain=.amazon.com; Expires=Sun, 03-Jan-2038
07:48:57 GMT; Path=/
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Edbrowse-dev] possible Referer issue
2018-01-09 9:22 ` [Edbrowse-dev] possible Referer issue Kevin Carhart
@ 2018-01-09 13:01 ` Chuck Hallenbeck
2018-01-09 22:58 ` Kevin Carhart
0 siblings, 1 reply; 5+ messages in thread
From: Chuck Hallenbeck @ 2018-01-09 13:01 UTC (permalink / raw)
To: Kevin Carhart; +Cc: edbrowse-dev
Hi Kevin,
I have amazon.wendy.5 here too, and have posted two db5 runs of my own
failed logins. I used grep to search for prepopulatedLoginID in all
three files, and found this:
$ grep -c prepopulatedLoginId amazon*
amazon.chuck.5:0
amazon.wendy.5:16
$ grep -c prepopulatedLoginId signing-in.txt 1
My outputs are available from www.panix.com/~chuxroom/signing-in.txt
and www.panix.com/~chuxroom/amazon.chuck.5
However, the file "signing-in.txt" was generated before the segfault was
detected and fixed, so the amazon.chuck.5 is probably a closer source.
Running here on Debian Sid, edbrowse 3.7.1, curl 7.57.0.
Chuck
--
Here In Northeast Ohio also, The Moon is Waning Crescent (44% of Full)
When your only tool is a hammer, everything looks like a nail.
Sent from Sergio's iPhone.
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Edbrowse-dev] possible Referer issue
2018-01-09 13:01 ` Chuck Hallenbeck
@ 2018-01-09 22:58 ` Kevin Carhart
2018-01-10 2:55 ` Chuck Hallenbeck
0 siblings, 1 reply; 5+ messages in thread
From: Kevin Carhart @ 2018-01-09 22:58 UTC (permalink / raw)
To: edbrowse-dev
Thanks, Chuck! Yes.. breaking down multiple factors is a problem.. not
knowing which of these point of divergence is the important one. That
said, I suspect the cookies, and the session-token cookie in particular.
There needs to be a conversation where it is received, sent back out, and
if the site gives you a new one, you send that one back in your next
request. If I understand what I'm reading, this isn't happening in
amazon.chuck.5 and in my failed attempts, and it does happen in
amazon.wendy.5.
The only references to session-token in amazon.chuck.5 are outgoing.
It does not appear in any Response. I bet this is important. Of
course I had a totally wrong hypothesis earlier so I could be wrong.
As a side suggestion, I think we can load the login-and-password screen on
first visit, and maybe save some hops and some bloat.
Technically we should be able to open a new edbrowse and 'b' the
login-and-password screen's URL from the getgo. I don't think we have to
visit the homepage and 'g' a link first. This would make the HTTP
conversations shorter.
On Tue, 9 Jan 2018, Chuck Hallenbeck wrote:
> Hi Kevin,
>
> I have amazon.wendy.5 here too, and have posted two db5 runs of my own failed
> logins. I used grep to search for prepopulatedLoginID in all three files, and
> found this:
>
> $ grep -c prepopulatedLoginId amazon*
> amazon.chuck.5:0
> amazon.wendy.5:16
> $ grep -c prepopulatedLoginId signing-in.txt 1
>
>
> My outputs are available from www.panix.com/~chuxroom/signing-in.txt
> and www.panix.com/~chuxroom/amazon.chuck.5
>
> However, the file "signing-in.txt" was generated before the segfault was
> detected and fixed, so the amazon.chuck.5 is probably a closer source.
>
> Running here on Debian Sid, edbrowse 3.7.1, curl 7.57.0.
>
> Chuck
>
>
> --
> Here In Northeast Ohio also, The Moon is Waning Crescent (44% of Full)
> When your only tool is a hammer, everything looks like a nail.
> Sent from Sergio's iPhone.
>
--------
Kevin Carhart * 415 225 5306 * The Ten Ninety Nihilists
^ permalink raw reply [flat|nested] 5+ messages in thread
* Re: [Edbrowse-dev] possible Referer issue
2018-01-09 22:58 ` Kevin Carhart
@ 2018-01-10 2:55 ` Chuck Hallenbeck
0 siblings, 0 replies; 5+ messages in thread
From: Chuck Hallenbeck @ 2018-01-10 2:55 UTC (permalink / raw)
To: Kevin Carhart; +Cc: edbrowse-dev
Hi Kevin,
I looked at how one gets from the Amazon opening screen to the
sign-in screen by using the capital A command on line 11, and it's
not pretty. I'm not up to tackling that shortcut myself.
About cookies, in my debugging output I see references to expired
cookies, which seems strange. But I have another concern:
It appears that issuing the db5 command and diverting the resulting
output to a log file changes the behavior of edbrowse. Let's consider
two cases, first with db1, which is my default in the rcfile.
and second with db5, and the command to redirect output to a log
file before opening amazon.com.
Cases 1 and 2: Open edbrowse and get a prompt.
Case 2 only: enter db5, then db>/tmp/log
Cases 1 and 2. Enter e www.amazon.com, and get the raw html size,
usually about 400000,
In case 1, but not case 2, we soon get the rendered size, a much
smaller number, and the opening screen is ready to be used.
In case 2, the buffer contains only the raw html, and we must type
the b command to render it, and only then do we see the smaller size
and can use the opening screen.
In both case 1 and case 2, we go to line 11, which contains five
links, we enter g2 to get to the sign-in screen, where we enter our
ID and Password.
For Case 1 only, we press submit and are taken to a new screen
resembling the sign-in screen, except for the messages near the top
saying "An Error Has Occurred" and "Enter a valid email address or
phone number" or some such wording.
The ID you previously typed is shown, but the password field is empty.
That was for case 1 only.
For case 2, pressing the submit button on the sign-in page causes
a segfault.
I would think the program behavior should not be changed by activating
a higher debug level, but the above is consistent and reproducible, and
makes me wonder how reliable the data in the resulting log file are.
Chuck
--
Here In Northeast Ohio also, The Moon is Waning Crescent (38% of Full)
When your only tool is a hammer, everything looks like a nail.
Sent from Camille's iPhone.
^ permalink raw reply [flat|nested] 5+ messages in thread
end of thread, other threads:[~2018-01-10 2:52 UTC | newest]
Thread overview: 5+ messages (download: mbox.gz / follow: Atom feed)
-- links below jump to the message on this page --
2018-01-08 16:11 [Edbrowse-dev] the amazon saga Chuck Hallenbeck
2018-01-09 9:22 ` [Edbrowse-dev] possible Referer issue Kevin Carhart
2018-01-09 13:01 ` Chuck Hallenbeck
2018-01-09 22:58 ` Kevin Carhart
2018-01-10 2:55 ` Chuck Hallenbeck
This is a public inbox, see mirroring instructions
for how to clone and mirror all data and code used for this inbox;
as well as URLs for NNTP newsgroup(s).