gs_offloader: Sane reporting of failures |
|||||||
Issue descriptionSee issue 682405 for the barrage of email alerts this is creating right now. TODO: - Drop a file locally on the server containing the set of failed directories / files. Currently, one has to go scrub the alert emails to know what folders to look inside. - Start reporting an error metric to monarch for #of failed uploads. - Drop the email alert.
,
Feb 3 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/6c4ed33bea57d412d2a9891910a30cd1f15138e3 commit 6c4ed33bea57d412d2a9891910a30cd1f15138e3 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Fri Feb 03 20:37:33 2017 gs_offloader: Move logging inside cleanup function. Also change logging level to info. BUG= chromium:686904 TEST=None. Change-Id: I451676e38d3a22fa060810fcb2ba2b5b3135be07 Reviewed-on: https://chromium-review.googlesource.com/434964 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/6c4ed33bea57d412d2a9891910a30cd1f15138e3/site_utils/gs_offloader.py
,
Feb 3 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/867cec5e343e7c923ac3b6caa65a1f1fe3ddaaa2 commit 867cec5e343e7c923ac3b6caa65a1f1fe3ddaaa2 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Fri Feb 03 20:37:33 2017 gs_offloader: Only warn on recoverable errors. BUG= chromium:686904 TEST=None. Change-Id: I74aa3fc907f1493983190f9d7fc4bfc21890e8ef Reviewed-on: https://chromium-review.googlesource.com/434966 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/867cec5e343e7c923ac3b6caa65a1f1fe3ddaaa2/site_utils/gs_offloader.py
,
Feb 3 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/867cec5e343e7c923ac3b6caa65a1f1fe3ddaaa2 commit 867cec5e343e7c923ac3b6caa65a1f1fe3ddaaa2 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Fri Feb 03 20:37:33 2017 gs_offloader: Only warn on recoverable errors. BUG= chromium:686904 TEST=None. Change-Id: I74aa3fc907f1493983190f9d7fc4bfc21890e8ef Reviewed-on: https://chromium-review.googlesource.com/434966 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/867cec5e343e7c923ac3b6caa65a1f1fe3ddaaa2/site_utils/gs_offloader.py
,
Feb 3 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/8973394ff49ab719f979e48b2de27743b892e55d commit 8973394ff49ab719f979e48b2de27743b892e55d Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Fri Feb 03 20:37:33 2017 gs_offloader: Reorder imports. BUG= chromium:686904 TEST=None. Change-Id: I71b62d8f11878766308e5320de55b73e67247386 Reviewed-on: https://chromium-review.googlesource.com/434968 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/8973394ff49ab719f979e48b2de27743b892e55d/site_utils/gs_offloader.py
,
Feb 3 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/beb9e01b8aebdd33b7c866fbdfef64bc77f139e5 commit beb9e01b8aebdd33b7c866fbdfef64bc77f139e5 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Fri Feb 03 20:37:34 2017 gs_offloader: Reorder imports, remove relative import. BUG= chromium:686904 TEST=None Change-Id: Ib88f08f3bcd88cd81de659d9a83be9ab34c3de92 Reviewed-on: https://chromium-review.googlesource.com/434970 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/beb9e01b8aebdd33b7c866fbdfef64bc77f139e5/site_utils/gs_offloader_unittest.py [modify] https://crrev.com/beb9e01b8aebdd33b7c866fbdfef64bc77f139e5/site_utils/gs_offloader.py
,
Feb 3 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/c163e21afb1bb0b72d1cd629c506645ff4425fe6 commit c163e21afb1bb0b72d1cd629c506645ff4425fe6 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Fri Feb 03 20:37:34 2017 gs_offloader: Setup ts_mon metrics for gs_offloader. BUG= chromium:686904 TEST=Local gs_offloader run doesn't blow up. Change-Id: I3c04177b88937aa18258b0a29785357a3454ae9c Reviewed-on: https://chromium-review.googlesource.com/434971 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/c163e21afb1bb0b72d1cd629c506645ff4425fe6/site_utils/gs_offloader.py
,
Feb 3 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/c9856853da581f3ab7c2d30c2f7d2b374aef4169 commit c9856853da581f3ab7c2d30c2f7d2b374aef4169 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Fri Feb 03 20:37:34 2017 gs_offloader: Report metrics for outstanding jobs. Start reporting how many directories are outstanding at the start of an offloading cycle. BUG= chromium:686904 TEST=None. Change-Id: Ic2afa74c47eda46aff2924aab3123150be91f51f Reviewed-on: https://chromium-review.googlesource.com/434972 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/c9856853da581f3ab7c2d30c2f7d2b374aef4169/site_utils/gs_offloader.py
,
Feb 3 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/343d171b1ac05dbf89aab7439a6eb8be3a9c002f commit 343d171b1ac05dbf89aab7439a6eb8be3a9c002f Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Fri Feb 03 20:37:35 2017 gs_offloader: Extract _send_reporting_failure_email. This is a transitional email. Eventually, the goal is to get rid of the function completely. BUG= chromium:686904 TEST=None. Change-Id: I7227a3b9a088d1378c9fd15e7fd1c60d462febc5 Reviewed-on: https://chromium-review.googlesource.com/434973 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/343d171b1ac05dbf89aab7439a6eb8be3a9c002f/site_utils/gs_offloader.py
,
Feb 3 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/ea869738e7219ff6ad91ce958aeb60430e690610 commit ea869738e7219ff6ad91ce958aeb60430e690610 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Fri Feb 03 20:37:35 2017 gs_offloader: Report number of failed jobs to monarch. BUG= chromium:686904 TEST=None. Change-Id: I334b953fefc2c8d3e42b07670da9388023070b45 Reviewed-on: https://chromium-review.googlesource.com/434974 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/ea869738e7219ff6ad91ce958aeb60430e690610/site_utils/gs_offloader.py
,
Feb 3 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/d0409303fcad602e589fdd477f4c3d25c139fc8c commit d0409303fcad602e589fdd477f4c3d25c139fc8c Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Fri Feb 03 20:37:35 2017 gs_offloader: Replace statsd metrics with monarch. BUG= chromium:686904 TEST=None. Change-Id: I7774a05edf5b0855884b7384cd53feb84446398f Reviewed-on: https://chromium-review.googlesource.com/434975 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/d0409303fcad602e589fdd477f4c3d25c139fc8c/site_utils/gs_offloader.py
,
Feb 3 2017
,
Feb 3 2017
Not all CLs have landed yet.
,
Feb 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/268d3d06f45bcfe453126e68976ab017d8fe50a4 commit 268d3d06f45bcfe453126e68976ab017d8fe50a4 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Mon Feb 06 18:37:43 2017 gs_offloader: Report metrics about some error conditions. BUG= chromium:686904 TEST=None Change-Id: I03f9fc5bd8a6ef61d7ad96955f31aeb87ae45cbd Reviewed-on: https://chromium-review.googlesource.com/434976 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/268d3d06f45bcfe453126e68976ab017d8fe50a4/site_utils/gs_offloader.py
,
Feb 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/23fcb47649dff800f5f8b837e2a7eea9b0f4a52d commit 23fcb47649dff800f5f8b837e2a7eea9b0f4a52d Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Mon Feb 06 18:37:44 2017 gs_offloader: Drop a redundant statsd metric. BUG= chromium:686904 TEST=None Change-Id: I7317cef462bdccf5ede15aba361c88115366dadb Reviewed-on: https://chromium-review.googlesource.com/435098 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/23fcb47649dff800f5f8b837e2a7eea9b0f4a52d/site_utils/gs_offloader.py
,
Feb 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/eeaa7ef307202c9494ec72fffd102f970842f659 commit eeaa7ef307202c9494ec72fffd102f970842f659 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Mon Feb 06 18:37:44 2017 gs_offloader: Replace statsd timer with monarch metrics. BUG= chromium:686904 TEST=None Change-Id: Icc8408657f48a8dccbdd62480afeb94d4df40091 Reviewed-on: https://chromium-review.googlesource.com/434977 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/eeaa7ef307202c9494ec72fffd102f970842f659/site_utils/gs_offloader.py
,
Feb 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/4b5f75199b91b2f59de403a0eb31f91661f18ab0 commit 4b5f75199b91b2f59de403a0eb31f91661f18ab0 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Mon Feb 06 18:37:44 2017 gs_offloader: Drop references to autotest_stats BUG= chromium:686904 TEST=None Change-Id: I6f89195380ffe103bb452c2f4185c381bd38dd6b Reviewed-on: https://chromium-review.googlesource.com/435099 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/4b5f75199b91b2f59de403a0eb31f91661f18ab0/site_utils/gs_offloader.py
,
Feb 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/fda271aecfd1402eaee58d0ff56e2b9bde192865 commit fda271aecfd1402eaee58d0ff56e2b9bde192865 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Mon Feb 06 18:37:45 2017 Make _format_job_for_failure_reporting a (private) module function. BUG= chromium:686904 TEST=None Change-Id: I1adae391a7ef50bb606817e29b0bb3d9c22dfb26 Reviewed-on: https://chromium-review.googlesource.com/435100 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/fda271aecfd1402eaee58d0ff56e2b9bde192865/site_utils/gs_offloader.py
,
Feb 6 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/16f9e5c9cbeadbfb030d2b2cf90c0816712d43d9 commit 16f9e5c9cbeadbfb030d2b2cf90c0816712d43d9 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Mon Feb 06 20:47:05 2017 gs_offloader: Locally log the list of failed jobs. We now maintain a log file on disk with the list of failed gs_offloader jobs. Before this CL, a developer had to go scrub emails to find out what jobs had failed from being offloaded. Now, they can look at the file on disk on the server where gs_offloader runs. This prepares the way to stop sending those annoying emails. BUG= chromium:686904 TEST=None. Change-Id: I691f04de07aafa01d606041ee0dc4809a60daf30 Reviewed-on: https://chromium-review.googlesource.com/435101 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/16f9e5c9cbeadbfb030d2b2cf90c0816712d43d9/site_utils/gs_offloader_unittest.py [modify] https://crrev.com/16f9e5c9cbeadbfb030d2b2cf90c0816712d43d9/site_utils/gs_offloader.py
,
Feb 8 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/80dfb1e9366b74f6c5a928e33793012e5a8ebc5a commit 80dfb1e9366b74f6c5a928e33793012e5a8ebc5a Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Wed Feb 08 04:14:12 2017 gs_offloader: Migrate unittests to new reporting method. This CL replaces the unittest for email reporting from gs_offloader to instead verify it's replacement -- the locally dropped file containing information about failed offloader jobs. BUG= chromium:686904 TEST=unittests. Change-Id: Ia5b75a4bf6fffde82fe4f7353a7ed713bdf8bee7 Reviewed-on: https://chromium-review.googlesource.com/435102 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/80dfb1e9366b74f6c5a928e33793012e5a8ebc5a/site_utils/gs_offloader_unittest.py [modify] https://crrev.com/80dfb1e9366b74f6c5a928e33793012e5a8ebc5a/site_utils/gs_offloader.py
,
Feb 8 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/33e3e1ad214e6d681f64eb01835b18bb045ce463 commit 33e3e1ad214e6d681f64eb01835b18bb045ce463 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Wed Feb 08 04:14:12 2017 gs_offloader: Stop sending emails on offloading failures. Better monarch based reporting is in place. A list of failed jobs is now dropped on the server for debugging purposes. This CL also removes the unittests around email reporting, since the logic related to deciding when an email report should be sent is now gone. BUG= chromium:686904 TEST=None Change-Id: I2db1463ac72d9c51e598345f02871aba65de744e Reviewed-on: https://chromium-review.googlesource.com/435103 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> [modify] https://crrev.com/33e3e1ad214e6d681f64eb01835b18bb045ce463/site_utils/gs_offloader_unittest.py [modify] https://crrev.com/33e3e1ad214e6d681f64eb01835b18bb045ce463/site_utils/gs_offloader.py
,
Feb 8 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/ca4815909f753b9117e5e87045dda7f088ba8e6f commit ca4815909f753b9117e5e87045dda7f088ba8e6f Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Wed Feb 08 04:14:13 2017 gs_offloader: Remove is_reportable from JobDirectory. We no longer report failures via emails, so the is_reportable logic is no longer needed. BUG= chromium:686904 TEST=None Change-Id: I2d5b0ee1aeb9a3d7a973b9fade818cf6b6a6d0f9 Reviewed-on: https://chromium-review.googlesource.com/435104 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/ca4815909f753b9117e5e87045dda7f088ba8e6f/site_utils/job_directories.py [modify] https://crrev.com/ca4815909f753b9117e5e87045dda7f088ba8e6f/site_utils/gs_offloader_unittest.py
,
Feb 8 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/051787b99dbe8804ec41cbd3c1566a303c2fbbfe commit 051787b99dbe8804ec41cbd3c1566a303c2fbbfe Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Wed Feb 08 04:14:13 2017 gs_offloader: Initialize variables outside try block. ... because they're used in the corresponding finally block. BUG= chromium:686904 TEST=None Change-Id: I66e08e91869dde09df29e2c637666ae8c6925cea Reviewed-on: https://chromium-review.googlesource.com/435575 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/051787b99dbe8804ec41cbd3c1566a303c2fbbfe/site_utils/gs_offloader.py
,
Feb 8 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/autotest/+/8f85cd2c7660da3f9498b9495e99f2eb39fe45f8 commit 8f85cd2c7660da3f9498b9495e99f2eb39fe45f8 Author: Prathmesh Prabhu <pprabhu@chromium.org> Date: Wed Feb 08 04:14:13 2017 gs_offloader: Replace logging.warn with logging.warning. BUG= chromium:686904 TEST=None Change-Id: Icddfe319ac787161b21946bea85d4b578c94a519 Reviewed-on: https://chromium-review.googlesource.com/435577 Commit-Ready: Prathmesh Prabhu <pprabhu@chromium.org> Tested-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Prathmesh Prabhu <pprabhu@chromium.org> Reviewed-by: Allen Li <ayatane@chromium.org> [modify] https://crrev.com/8f85cd2c7660da3f9498b9495e99f2eb39fe45f8/site_utils/gs_offloader.py
,
Mar 4 2017
,
May 30 2017
,
Aug 1 2017
,
Jan 22 2018
|
|||||||
►
Sign in to add a comment |
|||||||
Comment 1 by pprabhu@chromium.org
, Jan 30 2017