New issue
Advanced search Search tips

Issue 904876 link

Starred by 1 user
Status: Fixed
Owner:
perezju@chromium.org
Closed: Nov 20
Cc:
benjhayden@chromium.org
EstimatedDays: ----
NextAction: ----
OS: ----
Pri: 2
Type: Bug

Blocking:
issue 900218



Sign in to add a comment

[soundwave] Do not guess DataFrame dtypes from data

Project Member Reported by perezju@chromium.org, Nov 13 
In many places when building DataFrame's we let pandas guess the
dtypes of columns based on data returned from APIs.

This is error prone an leads to unexpected behaviors (e.g. when
an API returns no data for a selected date range).

Instead we should create empty DataFrame's with the correct types,
and incrementally add rows of data from the API responses.


* Found this when trying to switch soundwave to /timeseries2 API.
Project Member

Comment 1 by bugdroid1@chromium.org, Nov 20 

The following revision refers to this bug:
  https://chromium.googlesource.com/catapult/+/d11507507daae2e80b82614c6bbf07975e8843b7

commit d11507507daae2e80b82614c6bbf07975e8843b7
Author: Juan Antonio Navarro Perez <perezju@chromium.org>
Date: Tue Nov 20 11:09:19 2018

[soundwave] Use explicit dtype's for DataFrame columns

Otherwise letting pandas guess the dtypes is brittle when, e.g., an API
returns no rows of data.

For this we adapt the existing _EmptyDataFrame used to create the
database table schemas and adapt it to be able to also receive rows
of data.

Bug:  chromium:904876 
Change-Id: I69ef496e2b1ae33a516a219ff27dbec1181528d0
Reviewed-on: https://chromium-review.googlesource.com/c/1341929
Commit-Queue: Juan Antonio Navarro Pérez <perezju@chromium.org>
Reviewed-by: Ben Hayden <benjhayden@chromium.org>

[modify] https://crrev.com/d11507507daae2e80b82614c6bbf07975e8843b7/experimental/soundwave/soundwave/pandas_sqlite_test.py
[modify] https://crrev.com/d11507507daae2e80b82614c6bbf07975e8843b7/experimental/soundwave/soundwave/worker_pool_test.py
[modify] https://crrev.com/d11507507daae2e80b82614c6bbf07975e8843b7/experimental/soundwave/soundwave/tables/__init__.py
[modify] https://crrev.com/d11507507daae2e80b82614c6bbf07975e8843b7/experimental/soundwave/soundwave/tables/timeseries.py
[modify] https://crrev.com/d11507507daae2e80b82614c6bbf07975e8843b7/experimental/soundwave/soundwave/pandas_sqlite.py
[modify] https://crrev.com/d11507507daae2e80b82614c6bbf07975e8843b7/experimental/soundwave/soundwave/tables/timeseries_test.py
[modify] https://crrev.com/d11507507daae2e80b82614c6bbf07975e8843b7/experimental/soundwave/soundwave/tables/alerts.py
[modify] https://crrev.com/d11507507daae2e80b82614c6bbf07975e8843b7/experimental/soundwave/soundwave/tables/bugs.py

Comment 2 by perezju@chromium.org, Nov 20

Status: Fixed (was: Assigned)
Project Member

Comment 3 by bugdroid1@chromium.org, Nov 20 

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/8162ce12d9d4042c35716db084dd7d88f2869947

commit 8162ce12d9d4042c35716db084dd7d88f2869947
Author: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Date: Tue Nov 20 13:42:43 2018

Roll src/third_party/catapult 1e54003639e3..d11507507daa (1 commits)

https://chromium.googlesource.com/catapult.git/+log/1e54003639e3..d11507507daa


git log 1e54003639e3..d11507507daa --date=short --no-merges --format='%ad %ae %s'
2018-11-20 perezju@chromium.org [soundwave] Use explicit dtype's for DataFrame columns


Created with:
  gclient setdep -r src/third_party/catapult@d11507507daa

The AutoRoll server is located here: https://autoroll.skia.org/r/catapult-autoroll

Documentation for the AutoRoller is here:
https://skia.googlesource.com/buildbot/+/master/autoroll/README.md

If the roll is causing failures, please contact the current sheriff, who should
be CC'd on the roll, and stop the roller if necessary.

CQ_INCLUDE_TRYBOTS=luci.chromium.try:android_optional_gpu_tests_rel;luci.chromium.try:linux_optional_gpu_tests_rel;luci.chromium.try:mac_optional_gpu_tests_rel;luci.chromium.try:win_optional_gpu_tests_rel

BUG= chromium:904876 
TBR=sullivan@chromium.org

Change-Id: Icb6e1555aca0dee52cf934d4c489034ae6b0d033
Reviewed-on: https://chromium-review.googlesource.com/c/1342690
Reviewed-by: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Commit-Queue: chromium-autoroll <chromium-autoroll@skia-public.iam.gserviceaccount.com>
Cr-Commit-Position: refs/heads/master@{#609682}
[modify] https://crrev.com/8162ce12d9d4042c35716db084dd7d88f2869947/DEPS
►

Sign in to add a comment