New issue
Advanced search Search tips

Issue 809308 link

Starred by 3 users

Issue metadata

Status: Fixed
Owner: ----
Closed: Jul 16
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug

Blocked on:
issue 809323
issue 809325
issue 809326



Sign in to add a comment

Long LogDog logs are slow.

Project Member Reported by dgarr...@chromium.org, Feb 6 2018

Issue description

We are getting reports that logdog log streaming speeds for completed builds are unacceptably slow for really long build logs.

A concrete example is here:

https://luci-logdog.appspot.com/v/?s=chromeos/buildbucket/cr-buildbucket.appspot.com/8955839056303699808/+/steps/VMTest__attempt_2_/0/stdout
 

Comment 1 by ihf@chromium.org, Feb 6 2018

We have log files that can exceed 100MB and go to 1GB at times. It is extremely painful to look at them in Chrome. The usual solution is to download them. A feature that I have not discovered with LogDog yet (it streams instead).

Comment 3 by d...@chromium.org, Feb 6 2018

Downloading is easy:

$ logdog cat chromeos/bb/chromeos/../+/....
Where does the logdog command come from? It doesn't seem to be depot_tools.

Comment 5 by ihf@chromium.org, Feb 6 2018

Do you have documentation for this? Google finds other products.

ihf@ql:~$ logdog
logdog: command not found

Comment 6 by d...@chromium.org, Feb 6 2018

Looks like it's in "infra/infra": https://chromium.googlesource.com/infra/infra/+/master/bootstrap/cipd/cipd_linux_amd64.txt#23

Probably should be part of "depot_tools"; TBH I thought it was.

Comment 7 by ihf@chromium.org, Feb 6 2018

Can we make this process simpler for the average developer?

Comment 8 by d...@chromium.org, Feb 6 2018

(Suggestion in #6 would be making it simpler for the average developer)
Agreed.

Further alternatives...
* Add a download link to the logdog header bar.
* Add an additional URL argument that means "display as simple text" ie: "?format=text"

Just thinking.

Who owns these updates now?

Comment 10 by d...@chromium.org, Feb 6 2018

Both of those are great ideas. Would you mind filing bugs for each and assigning LogDog component?

LUCI team still owns LogDog :)
Components: -Infra Infra>Platform>LogDog
Blockedon: 809326 809323 809325
Cc: pwang@chromium.org
Filed. Thanks!
Status: Available (was: Untriaged)
FYI, "cit logdog" works with just depot_tools in PATH.

We'll try to get logdog improvements added to the schedule for next quarter.

Comment 15 by whesse@google.com, Mar 6 2018

Cc: whesse@google.com
One of our users would like Milo to present a link that downloads the log as a plain file, as mentioned in the above comment.

They also think that it is the logdog streaming api that makes it slow, so that when the log has been finished and saved to a single cloud storage object, they want the link to download directly from that location instead, when getting the raw data.

I don't see a separate issue as having been filed with this request yet.
This is the main Milo issue that affects our users' productivity, so I think its priority should be higher than 3.
Cc: jchin...@chromium.org
Labels: -Pri-3 Pri-2
I agree that this should be higher priority than 3. Bumping it to 2.
Cc: hinoka@chromium.org
Status: Fixed (was: Available)
i believe this was fixed by  issue 698429  thanks to hinoka
specifically, 
https://luci-logdog.appspot.com/v/?s=chromeos/buildbucket/cr-buildbucket.appspot.com/8955839056303699808/+/steps/VMTest__attempt_2_/0/stdout#
load data <1.5s and Chrome renders it <5s for me.

please reopen if it is still slow
wrong link: https://luci-logdog.appspot.com/logs/chromeos/buildbucket/cr-buildbucket.appspot.com/8955839056303699808/+/steps/VMTest__attempt_2_/0/stdout

and here is the new viewer for the "long log" in #2:
https://logs.chromium.org/logs/chromeos/bb/chromeos/betty-vmtest-informational/152/+/recipes/steps/VMTest/0/stdout

now the bottleneck is Chrome. If authenticated, you can use curl to load that log, though. Sometimes like

# luci-auth is in depot_tools
luci-auth login
# do the login
curl -H "Authorization: Bearer `luci-auth token`" 'https://logs.chromium.org/logs/chromeos/bb/chromeos/betty-vmtest-informational/152/+/recipes/steps/VMTest/0/stdout' > mylog

Sign in to add a comment