crashpad LUCI tasks frequently drop the tail end of their annotations |
|||
Issue descriptionExample: https://ci.chromium.org/p/crashpad/builders/luci.crashpad.ci/crashpad_mac_rel/1285 If you look at the raw task (https://chromium-swarm.appspot.com/task?id=409d177834d4dd10&refresh=10&show_raw=1&wide_logs=true) you can see that kitchen actually got the full annotation state. However... the last bit of the compile log and the other steps never make their way off the bot. Is it possible that kitchen terminates without waiting for the butler to flush everything?
,
Oct 17
kitchen waits for butler to finish its business https://cs.chromium.org/chromium/infra/go/src/infra/tools/kitchen/cook_logdog.go?q=kitchen/cook_logdog&sq=package:chromium&g=0&l=182&rcl=c44213d85a5a2c7af2e15cf63df3094ea7d15aae
,
Oct 18
I'm not sure where the bug is yet, but dumping some screenshots for postierty. LogDog Collector: https://screenshot.googleplex.com/MQuFNvbMyu7 But catting the annotation protobuf, i see: [I2018-10-17T16:56:11.821757-07:00 25984 0 fetcher.go:366] No logs returned. Sleeping... {"delay":"5s", "index":25} Meaning: 1. The stream isn't terminated (therefore, not archived) 2. The last accessable stream index is 25 (terminal is 36) This agrees: https://screenshot.googleplex.com/K4qUWWL81WT (Terminated, not archived). So this rules out the theory that archiving was happening before termination. Instead it looks like the entry never makes it to BigTable?
,
Oct 18
Actually this was in the swarming output: https://paste.googleplex.com/5167174477938688 This is definitely a butler bug (wrt how invalid UTF8 characters are handled).
,
Oct 18
Specifically: proto: field \"logpb.Text.Line.Value\" contains invalid UTF-8"
,
Oct 18
So I think the summary of this is: * Recent deployment of kitchen includes newer protobuf library which refuses to marshal non-UTF8 into `string` fields. * Test emits non-UTF8 on stdout * Logdog butler bundles this log data, plus the UI data into a single 'bundle' which it attempts to marshal and send to logdog server. * Marshalling this bundle fails outright on the new butler, which drops the WHOLE BUNDLE (including all the rest of the UI updates). * We don't see this in chrome because we have SO MUCH output text that some future bundle DOES succeed, and so we see the latest UI state (even if some test has garbage in the middle) We're going to change logdog to no longer care about the UTF8-ness of its text streams; it will be up to the user agent (browser) to handle display of non-UTF8 data (we may provide a warning icon on the line at some future point to indicate that it contains invalid UTF8 though).
,
Oct 18
The following revision refers to this bug: https://chromium.googlesource.com/infra/luci/luci-go.git/+/3bc8f08ce19949eca45f30957498b69d0e1b2e24 commit 3bc8f08ce19949eca45f30957498b69d0e1b2e24 Author: Ryan Tseng <hinoka@google.com> Date: Thu Oct 18 01:25:16 2018 [logdog] Add prefix logging to all log entries So that we can search the logs for specific prefixes Bug: 896504 Change-Id: Ib5df70a3c45d812b401c1bbdcd56a2ae81b8884b Reviewed-on: https://chromium-review.googlesource.com/c/1286552 Reviewed-by: Robbie Iannucci <iannucci@chromium.org> Commit-Queue: Robbie Iannucci <iannucci@chromium.org> [modify] https://crrev.com/3bc8f08ce19949eca45f30957498b69d0e1b2e24/logdog/server/collector/collector.go |
|||
►
Sign in to add a comment |
|||
Comment 1 by iannu...@google.com
, Oct 17