Issue 831887 link

Starred by 12 users

Issue metadata

Status:	Fixed
Owner:	caseq@chromium.org
Closed:	Oct 26
Cc:	█ lushnikov@chromium.org pfeldman@chromium.org dgozman@chromium.org
Components:	Internals>Headless Platform>DevTools>Network
EstimatedDays:	----
NextAction:	----
OS:	----
Pri:	2
Type:	Bug

DevTools: support download interception

Project Member Reported by caseq@chromium.org, Apr 12 2018

Issue description

This includes:
 - intercept download requests
 - identify intercepted requests as downloads (e.g. isDownload attribute in RequestIntercepted event)
 - expose a convenient method to get the response body as stream and/or cancel the download;

This is mostly needed by headless / puppeteer and would let us get rid of  Page.setDownloadBehavior and DevToolsDownloadManagerDelegate.

This will also address instrumentation of downloads (see issue 767189) once we merge interception and regular network instrumentation.

Project Member

Comment 1 by bugdroid1@chromium.org, Apr 17 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/bc41238800e73da4b0b002011e3700d0944641e8

commit bc41238800e73da4b0b002011e3700d0944641e8
Author: Andrey Kosyakov <caseq@chromium.org>
Date: Tue Apr 17 03:09:11 2018

DevTools: further split different stream implementations from IOContext

- let consumers create streams directly with DevTooolsStreamFoo::Create()
    rather than through different methods of DevToolsIOContext;
- get rid of Close() methods, do all cleanup in destructors;
- eliminate RWStream, let its consumer use DevToolsStreamFile directly;

Bug:  831887 
Change-Id: I1aa28f8dea4a4e00c0323cb4101dc8db274a9bd4
Reviewed-on: https://chromium-review.googlesource.com/1014303
Commit-Queue: Andrey Kosyakov <caseq@chromium.org>
Reviewed-by: Dmitry Gozman <dgozman@chromium.org>
Cr-Commit-Position: refs/heads/master@{#551226}
[modify] https://crrev.com/bc41238800e73da4b0b002011e3700d0944641e8/content/browser/devtools/devtools_io_context.cc
[modify] https://crrev.com/bc41238800e73da4b0b002011e3700d0944641e8/content/browser/devtools/devtools_io_context.h
[modify] https://crrev.com/bc41238800e73da4b0b002011e3700d0944641e8/content/browser/devtools/devtools_stream_blob.cc
[modify] https://crrev.com/bc41238800e73da4b0b002011e3700d0944641e8/content/browser/devtools/devtools_stream_blob.h
[modify] https://crrev.com/bc41238800e73da4b0b002011e3700d0944641e8/content/browser/devtools/devtools_stream_file.cc
[modify] https://crrev.com/bc41238800e73da4b0b002011e3700d0944641e8/content/browser/devtools/devtools_stream_file.h
[modify] https://crrev.com/bc41238800e73da4b0b002011e3700d0944641e8/content/browser/devtools/protocol/io_handler.cc
[modify] https://crrev.com/bc41238800e73da4b0b002011e3700d0944641e8/content/browser/devtools/protocol/tracing_handler.cc

Project Member

Comment 2 by bugdroid1@chromium.org, Apr 19 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/52a8481322219e9b33467f82e46dd2372a5cb170

commit 52a8481322219e9b33467f82e46dd2372a5cb170
Author: Andrey Kosyakov <caseq@chromium.org>
Date: Thu Apr 19 05:36:39 2018

DevTools: added indication of downloadable requests to requestIntercepted

Bug:  831887 
Change-Id: Ib80a0355a22609ea5884a25de734fd268d489b3d
Reviewed-on: https://chromium-review.googlesource.com/1008974
Commit-Queue: Andrey Kosyakov <caseq@chromium.org>
Reviewed-by: Min Qin <qinmin@chromium.org>
Reviewed-by: Dmitry Gozman <dgozman@chromium.org>
Cr-Commit-Position: refs/heads/master@{#551951}
[modify] https://crrev.com/52a8481322219e9b33467f82e46dd2372a5cb170/content/browser/BUILD.gn
[modify] https://crrev.com/52a8481322219e9b33467f82e46dd2372a5cb170/content/browser/devtools/devtools_network_interceptor.cc
[modify] https://crrev.com/52a8481322219e9b33467f82e46dd2372a5cb170/content/browser/devtools/devtools_network_interceptor.h
[modify] https://crrev.com/52a8481322219e9b33467f82e46dd2372a5cb170/content/browser/devtools/devtools_url_interceptor_request_job.cc
[modify] https://crrev.com/52a8481322219e9b33467f82e46dd2372a5cb170/content/browser/devtools/devtools_url_loader_interceptor.cc
[modify] https://crrev.com/52a8481322219e9b33467f82e46dd2372a5cb170/content/browser/devtools/devtools_url_loader_interceptor.h
[modify] https://crrev.com/52a8481322219e9b33467f82e46dd2372a5cb170/content/browser/devtools/protocol/network_handler.cc
[modify] https://crrev.com/52a8481322219e9b33467f82e46dd2372a5cb170/content/browser/devtools/protocol/network_handler.h
[modify] https://crrev.com/52a8481322219e9b33467f82e46dd2372a5cb170/content/browser/devtools/render_frame_devtools_agent_host.cc
[modify] https://crrev.com/52a8481322219e9b33467f82e46dd2372a5cb170/content/browser/devtools/render_frame_devtools_agent_host.h
[modify] https://crrev.com/52a8481322219e9b33467f82e46dd2372a5cb170/content/browser/download/download_manager_impl.cc
[modify] https://crrev.com/52a8481322219e9b33467f82e46dd2372a5cb170/content/browser/frame_host/render_frame_host_impl.cc
[add] https://crrev.com/52a8481322219e9b33467f82e46dd2372a5cb170/content/browser/loader/navigation_loader_util.cc
[add] https://crrev.com/52a8481322219e9b33467f82e46dd2372a5cb170/content/browser/loader/navigation_loader_util.h
[modify] https://crrev.com/52a8481322219e9b33467f82e46dd2372a5cb170/content/browser/loader/navigation_url_loader_network_service.cc
[modify] https://crrev.com/52a8481322219e9b33467f82e46dd2372a5cb170/third_party/WebKit/LayoutTests/http/tests/devtools/network/resources/resource.php
[add] https://crrev.com/52a8481322219e9b33467f82e46dd2372a5cb170/third_party/WebKit/LayoutTests/http/tests/inspector-protocol/network/interception-download-expected.txt
[add] https://crrev.com/52a8481322219e9b33467f82e46dd2372a5cb170/third_party/WebKit/LayoutTests/http/tests/inspector-protocol/network/interception-download.js
[modify] https://crrev.com/52a8481322219e9b33467f82e46dd2372a5cb170/third_party/blink/renderer/core/inspector/browser_protocol.pdl
[modify] https://crrev.com/52a8481322219e9b33467f82e46dd2372a5cb170/third_party/blink/renderer/devtools/front_end/sdk/NetworkManager.js

Project Member

Comment 3 by bugdroid1@chromium.org, Apr 25 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/fd7c7ced20d7d28388479d556ab6b09fb6784bdc

commit fd7c7ced20d7d28388479d556ab6b09fb6784bdc
Author: Andrey Kosyakov <caseq@chromium.org>
Date: Wed Apr 25 19:26:50 2018

DevTools: support returning intercepted response body as a stream

This adds Network.takeResponseBodyForInterceptionAsStream, which returns
a stream handle that can be used with IO.read/IO.close.
Note this is only supported with the Network Service currently, the
legacy URLRequestJob based interception support will be added later.

Bug:  831887 
Change-Id: Ibc5a43e86824f0c86397f200b5a6852171705f0e
Reviewed-on: https://chromium-review.googlesource.com/1026692
Reviewed-by: Dmitry Gozman <dgozman@chromium.org>
Commit-Queue: Andrey Kosyakov <caseq@chromium.org>
Cr-Commit-Position: refs/heads/master@{#553695}
[modify] https://crrev.com/fd7c7ced20d7d28388479d556ab6b09fb6784bdc/content/browser/BUILD.gn
[modify] https://crrev.com/fd7c7ced20d7d28388479d556ab6b09fb6784bdc/content/browser/devtools/devtools_io_context.cc
[modify] https://crrev.com/fd7c7ced20d7d28388479d556ab6b09fb6784bdc/content/browser/devtools/devtools_io_context.h
[modify] https://crrev.com/fd7c7ced20d7d28388479d556ab6b09fb6784bdc/content/browser/devtools/devtools_network_interceptor.h
[modify] https://crrev.com/fd7c7ced20d7d28388479d556ab6b09fb6784bdc/content/browser/devtools/devtools_stream_blob.cc
[modify] https://crrev.com/fd7c7ced20d7d28388479d556ab6b09fb6784bdc/content/browser/devtools/devtools_stream_blob.h
[add] https://crrev.com/fd7c7ced20d7d28388479d556ab6b09fb6784bdc/content/browser/devtools/devtools_stream_pipe.cc
[add] https://crrev.com/fd7c7ced20d7d28388479d556ab6b09fb6784bdc/content/browser/devtools/devtools_stream_pipe.h
[modify] https://crrev.com/fd7c7ced20d7d28388479d556ab6b09fb6784bdc/content/browser/devtools/devtools_url_loader_interceptor.cc
[modify] https://crrev.com/fd7c7ced20d7d28388479d556ab6b09fb6784bdc/content/browser/devtools/devtools_url_loader_interceptor.h
[modify] https://crrev.com/fd7c7ced20d7d28388479d556ab6b09fb6784bdc/content/browser/devtools/protocol/io_handler.cc
[modify] https://crrev.com/fd7c7ced20d7d28388479d556ab6b09fb6784bdc/content/browser/devtools/protocol/network_handler.cc
[modify] https://crrev.com/fd7c7ced20d7d28388479d556ab6b09fb6784bdc/content/browser/devtools/protocol/network_handler.h
[modify] https://crrev.com/fd7c7ced20d7d28388479d556ab6b09fb6784bdc/content/browser/devtools/protocol_config.json
[modify] https://crrev.com/fd7c7ced20d7d28388479d556ab6b09fb6784bdc/content/browser/devtools/render_frame_devtools_agent_host.cc
[modify] https://crrev.com/fd7c7ced20d7d28388479d556ab6b09fb6784bdc/content/browser/devtools/service_worker_devtools_agent_host.cc
[modify] https://crrev.com/fd7c7ced20d7d28388479d556ab6b09fb6784bdc/content/browser/devtools/shared_worker_devtools_agent_host.cc
[add] https://crrev.com/fd7c7ced20d7d28388479d556ab6b09fb6784bdc/third_party/WebKit/LayoutTests/flag-specific/enable-features=NetworkService/http/tests/inspector-protocol/network/interception-take-stream-expected.txt
[modify] https://crrev.com/fd7c7ced20d7d28388479d556ab6b09fb6784bdc/third_party/WebKit/LayoutTests/http/tests/devtools/network/resources/resource.php
[add] https://crrev.com/fd7c7ced20d7d28388479d556ab6b09fb6784bdc/third_party/WebKit/LayoutTests/http/tests/inspector-protocol/network/interception-take-stream-expected.txt
[add] https://crrev.com/fd7c7ced20d7d28388479d556ab6b09fb6784bdc/third_party/WebKit/LayoutTests/http/tests/inspector-protocol/network/interception-take-stream.js
[modify] https://crrev.com/fd7c7ced20d7d28388479d556ab6b09fb6784bdc/third_party/blink/renderer/core/inspector/browser_protocol.pdl
[modify] https://crrev.com/fd7c7ced20d7d28388479d556ab6b09fb6784bdc/third_party/blink/renderer/core/inspector/inspector_protocol_config.json

Project Member

Comment 4 by bugdroid1@chromium.org, May 4 2018

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/6b92b538e19106e1ad916e6f4939433001468be7

commit 6b92b538e19106e1ad916e6f4939433001468be7
Author: Andrey Kosyakov <caseq@chromium.org>
Date: Fri May 04 23:54:13 2018

Network Service: Remove crash expectations for interception-file-url.js

Works for me now, I think the DCHECK it was hitting was removed as a
drive-by here:

https://chromium-review.googlesource.com/c/chromium/src/+/1008974/11/content/browser/devtools/devtools_url_loader_interceptor.cc#b478

TBR: dgozman@chromium.org, lushnikov@chromium.org
Bug:  721408 , 831887 
Cq-Include-Trybots: master.tryserver.chromium.linux:linux_mojo
Change-Id: I886a1e302b94b5de685e4ef4442c10564df78539
Reviewed-on: https://chromium-review.googlesource.com/1044539
Reviewed-by: Dmitry Gozman <dgozman@chromium.org>
Reviewed-by: Andrey Kosyakov <caseq@chromium.org>
Commit-Queue: Andrey Kosyakov <caseq@chromium.org>
Cr-Commit-Position: refs/heads/master@{#556259}
[modify] https://crrev.com/6b92b538e19106e1ad916e6f4939433001468be7/third_party/WebKit/LayoutTests/FlagExpectations/enable-features=NetworkService

Comment 5 by lem...@gmail.com, Jun 11 2018

Any progress on this?

Comment 6 by caseq@chromium.org, Jun 11 2018

Aside from Network.takeResponseBodyForInterceptionAsStreams, everything is done, the streams only work with network service.

Comment 7 by vikramth...@gmail.com, Sep 5

Is there any update on this? It's been over months now and ability to download consistently is one of the expectations of puppeteer where this is causing a major hindrance to users.

Comment 8 by caseq@chromium.org, Oct 26

Status: Fixed (was: Assigned)

Comment 9 by caseq@chromium.org, Oct 26

Note to puppeteer uses: this only works with network service enable, pass --enable-features=NetworkService flag to chrome.

Comment 10 by st...@stevestmartin.com, Oct 29

Can you point me in the right direction for how to use this within puppeteer or how I would go about implementing the PR.

Comment 11 by lushnikov@chromium.org, Oct 29

@steve: I'd strongly prefer to have https://crbug.com/899386 fixed before exposing more interception-related functionality to Puppeteer.

►

Issue 831887 link

Issue metadata

DevTools: support download interception

Issue description

Comment 1 by bugdroid1@chromium.org, Apr 17 2018

Comment 1 Deleted

Comment 2 by bugdroid1@chromium.org, Apr 19 2018

Comment 2 Deleted

Comment 3 by bugdroid1@chromium.org, Apr 25 2018

Comment 3 Deleted

Comment 4 by bugdroid1@chromium.org, May 4 2018

Comment 4 Deleted

Comment 5 by lem...@gmail.com, Jun 11 2018

Comment 5 Deleted

Comment 6 by caseq@chromium.org, Jun 11 2018

Comment 6 Deleted

Comment 7 by vikramth...@gmail.com, Sep 5

Comment 7 Deleted

Comment 8 by caseq@chromium.org, Oct 26

Comment 8 Deleted

Comment 9 by caseq@chromium.org, Oct 26

Comment 9 Deleted

Comment 10 by st...@stevestmartin.com, Oct 29

Comment 10 Deleted

Comment 11 by lushnikov@chromium.org, Oct 29

Comment 11 Deleted

Sign in to add a comment