New issue
Advanced search Search tips

Issue 805887 link

Starred by 3 users

Issue metadata

Status: Closed
Owner:
Closed: Jan 5
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 2
Type: Bug



Sign in to add a comment

Network.getResponseBody fails sporadically

Reported by i...@netreturn.co.za, Jan 25 2018

Issue description

UserAgent: Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/63.0.3239.132 Safari/537.36

Steps to reproduce the problem:
1. git clone https://github.com/ilanc/devtools-bugs
2. cd devtools-bugs
3. npm install
4. node ./getResponseBody.js
5. look for output like this:
  getResponseBody exception 4612.1449 https://files.coinmarketcap.com/generated/search/quick_search.json No data found for resource with given identifier

What is the expected behavior?
Either `Network.getResponseBody` should just work or if it is a cache related issue then I would like to be able to specify which requests should be preserved e.g. something like `Network.preserveResponseBody(requestId)` (which could be called from `Network.requestWillBeSent`)

What went wrong?
* `getResponseBody` fails sporadically - you can't predict which request will fail ahead of time.
* output = No data found for resource with given identifier

Did this work before? No 

Chrome version: 63.0.3239.132  Channel: stable
OS Version: 10.0
Flash Version: Shockwave Flash 28.0 r0

There are some old threads about this:
https://groups.google.com/forum/#!topic/google-chrome-developer-tools/8KWv_gNBfYY

But it looks like they were never resolved
 
Hmm.. it may be that it's my bug. According to this https://github.com/cyrus-and/chrome-remote-interface/issues/260 I need to call `Network.getResponseBody` from `Network.loadingFinished` NOT `Network.responseReceived`. I'll change the code in the repo and retry.
Hander pola?
Kanker pola,
Owner: eostroukhov@chromium.org
Status: Assigned (was: Unconfirmed)
I've been playing with this and want to add some more thoughts - alas I haven't established a reliable repeat case yet.
1. I'm actually using the code on other websites. I can't share this code because I'm working with website which require a login.
2. There are a number of requests (14 in total) across a number of sites (6 in total) where I'm attempting to use getResponseBody. Of these requests some are are more flakey than others - i.e. getResponseBody fails about 50% of the time on the flakiest of the requests.
3. I log out the headers on success and fail for all getResponseBody requests. The headers don't seem to change, i.e. I haven't isolated a header setting which causes getResponseBody to fail 100% of the time.
4. The most flakey requests have these headers:
  Pragma: 'no-cache',
  'Cache-Control': 'no-cache',
5. But some of the reliable ones also have have no-cache

So headers may be a red-herring - maybe  it's down to the volume of requests occurring at the same time? like I originally thought but that's why I used coinmarketcap.com in the first place and it seems more reliable.

Finally I did also run fiddler to capture http traffic during my tests. The flakey requests are being captured by fiddler - i.e. the server is definitely sending a body, it's just that getResponseBody is failing to retrieve it. 
Have you had any luck confirming this?

There is one other possible problem - I've just noticed that I am getting duplicate `Network.responseReceived` & `Network.loadingFinished` events for a single requestId. I've put in code to ensure that I only call getResponseBody once per requestId. I'll update here if this cures the problem.
Hi @eostroukhov have you had a look at this?
I found that invalid characters in the original file (e.g. HTML, JS, etc) stop getResponseBody returning at all. It's not sporadic but is another case where getResponseBody fails to return either the content or an error.

You can find out more in  issue 824174 
Thanks for the feedback. What would constitute an invalid character? The responses I'm trying to capture are ajax request with json payloads. 
An invalid character would be a sequence of bytes that don't conform to UTF8 encoding. This page has some examples:

http://www.cl.cam.ac.uk/~mgk25/ucs/examples/UTF-8-test.txt
Owner: jarhar@chromium.org
ilan@netreturn.co.za I ran your node script but it didn't say anything about getResponseBody. Are you still having any issues? Can you provide better steps to reproduce?

Side note - I would recommend using puppeteer for browser automation stuff like this since it wraps the devtools protocol and our team maintains it: https://pptr.dev
Hi jarhar@chromium.org. It looks like the problem went away. I've searched through 1 year of logs (from the prod code where I use this functionality) and not found any getResponseBody fails since 2018-01-29. Perhaps the `Network.loadingFinished` fix above worked? It's a long time ago now so I can't be sure. Thanks I'm aware of puppeteer but alas it doesn't fit my requirements.
Status: Closed (was: Assigned)

Sign in to add a comment