After several days of running CTS P the sqld crashes with a corrupt innodb file. I see that in each case the ibdata1 file has grown very large, much larger than the size of the actual tables in the db.
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/139c030c3f55a2373eccbb2d007a5179b6dc8ace commit 139c030c3f55a2373eccbb2d007a5179b6dc8ace Author: Keith Haddow <haddowk@chromium.org> Date: Wed Aug 29 04:12:46 2018 [moblab] Kill long running sql queries There is a sql connection/transaction leak in moblab causing it to crash. As the investigation continues to the root cause add a limit to the length of any one sql query so the moblab will not crash BUG= chromium:878425 TEST=ad hoc testing on moblab Change-Id: I3a454b4c99a0c02b9b17afd3f8bd65c5695f1e38 Reviewed-on: https://chromium-review.googlesource.com/1194186 Commit-Ready: Keith Haddow <haddowk@chromium.org> Tested-by: Keith Haddow <haddowk@chromium.org> Reviewed-by: Keith Haddow <haddowk@chromium.org> Reviewed-by: Matt Mallett <mattmallett@chromium.org> [add] https://crrev.com/139c030c3f55a2373eccbb2d007a5179b6dc8ace/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-kill-long-running-sql-query.conf [add] https://crrev.com/139c030c3f55a2373eccbb2d007a5179b6dc8ace/project-moblab/chromeos-base/chromeos-bsp-moblab/chromeos-bsp-moblab-0.0.1-r46.ebuild
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/bce1f8f5c92099df81084c873dab43de610559ec commit bce1f8f5c92099df81084c873dab43de610559ec Author: Keith Haddow <haddowk@chromium.org> Date: Wed Aug 29 18:43:20 2018 [moblab] Kill long running sql queries There is a sql connection/transaction leak in moblab causing it to crash. As the investigation continues to the root cause add a limit to the length of any one sql query so the moblab will not crash BUG= chromium:878425 TEST=ad hoc testing on moblab Change-Id: I3a454b4c99a0c02b9b17afd3f8bd65c5695f1e38 Reviewed-on: https://chromium-review.googlesource.com/1194186 Commit-Ready: Keith Haddow <haddowk@chromium.org> Tested-by: Keith Haddow <haddowk@chromium.org> Reviewed-by: Keith Haddow <haddowk@chromium.org> Reviewed-by: Matt Mallett <mattmallett@chromium.org> (cherry picked from commit 139c030c3f55a2373eccbb2d007a5179b6dc8ace) Reviewed-on: https://chromium-review.googlesource.com/1194201 Commit-Queue: Keith Haddow <haddowk@chromium.org> Trybot-Ready: Keith Haddow <haddowk@chromium.org> [add] https://crrev.com/bce1f8f5c92099df81084c873dab43de610559ec/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-kill-long-running-sql-query.conf [add] https://crrev.com/bce1f8f5c92099df81084c873dab43de610559ec/project-moblab/chromeos-base/chromeos-bsp-moblab/chromeos-bsp-moblab-0.0.1-r46.ebuild
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/def38cb02c400cb5e8b509e5e81edcf050d391bf commit def38cb02c400cb5e8b509e5e81edcf050d391bf Author: Keith Haddow <haddowk@chromium.org> Date: Wed Aug 29 18:43:22 2018 [moblab] Kill long running sql queries There is a sql connection/transaction leak in moblab causing it to crash. As the investigation continues to the root cause add a limit to the length of any one sql query so the moblab will not crash BUG= chromium:878425 TEST=ad hoc testing on moblab Change-Id: I3a454b4c99a0c02b9b17afd3f8bd65c5695f1e38 Reviewed-on: https://chromium-review.googlesource.com/1194186 Commit-Ready: Keith Haddow <haddowk@chromium.org> Tested-by: Keith Haddow <haddowk@chromium.org> Reviewed-by: Keith Haddow <haddowk@chromium.org> Reviewed-by: Matt Mallett <mattmallett@chromium.org> (cherry picked from commit 139c030c3f55a2373eccbb2d007a5179b6dc8ace) Reviewed-on: https://chromium-review.googlesource.com/1194200 Commit-Queue: Keith Haddow <haddowk@chromium.org> Trybot-Ready: Keith Haddow <haddowk@chromium.org> [add] https://crrev.com/def38cb02c400cb5e8b509e5e81edcf050d391bf/project-moblab/chromeos-base/chromeos-bsp-moblab/files/init/moblab-kill-long-running-sql-query.conf [add] https://crrev.com/def38cb02c400cb5e8b509e5e81edcf050d391bf/project-moblab/chromeos-base/chromeos-bsp-moblab/chromeos-bsp-moblab-0.0.1-r46.ebuild
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/5be2fd6ffd2471526f4433ee32aee4dc23a976e9 commit 5be2fd6ffd2471526f4433ee32aee4dc23a976e9 Author: Keith Haddow <haddowk@chromium.org> Date: Fri Sep 07 09:58:37 2018 [moblab] Tweak the mysql server parameters. Ran a tool to analize usage, updated the server parameters based on the recomendations of the tool. BUG= chromium:878425 TEST=ad hoc testing on moblab Change-Id: I5d8cebedb597236c0c20296886b6486e10af067b Reviewed-on: https://chromium-review.googlesource.com/1210482 Commit-Ready: Keith Haddow <haddowk@chromium.org> Tested-by: Keith Haddow <haddowk@chromium.org> Reviewed-by: Matt Mallett <mattmallett@chromium.org> [rename] https://crrev.com/5be2fd6ffd2471526f4433ee32aee4dc23a976e9/project-moblab/chromeos-base/chromeos-bsp-moblab/chromeos-bsp-moblab-0.0.1-r47.ebuild [modify] https://crrev.com/5be2fd6ffd2471526f4433ee32aee4dc23a976e9/project-moblab/chromeos-base/chromeos-bsp-moblab/files/mysql_defaults_extra.cnf [delete] https://crrev.com/a8fc36018acf86d8189b2a895e381f94765ebee5/project-moblab/chromeos-base/chromeos-bsp-moblab/chromeos-bsp-moblab-0.0.1-r46.ebuild
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/0350e2be97a15163651a4350ad5c0822f67ec2e7 commit 0350e2be97a15163651a4350ad5c0822f67ec2e7 Author: Keith Haddow <haddowk@chromium.org> Date: Fri Sep 07 22:30:16 2018 [moblab] Tweak the mysql server parameters. Ran a tool to analize usage, updated the server parameters based on the recomendations of the tool. BUG= chromium:878425 TEST=ad hoc testing on moblab Change-Id: I5d8cebedb597236c0c20296886b6486e10af067b Reviewed-on: https://chromium-review.googlesource.com/1210482 Commit-Ready: Keith Haddow <haddowk@chromium.org> Tested-by: Keith Haddow <haddowk@chromium.org> Reviewed-by: Matt Mallett <mattmallett@chromium.org> (cherry picked from commit 5be2fd6ffd2471526f4433ee32aee4dc23a976e9) Reviewed-on: https://chromium-review.googlesource.com/1213768 Reviewed-by: Keith Haddow <haddowk@chromium.org> Commit-Queue: Keith Haddow <haddowk@chromium.org> Trybot-Ready: Keith Haddow <haddowk@chromium.org> [rename] https://crrev.com/0350e2be97a15163651a4350ad5c0822f67ec2e7/project-moblab/chromeos-base/chromeos-bsp-moblab/chromeos-bsp-moblab-0.0.1-r47.ebuild [modify] https://crrev.com/0350e2be97a15163651a4350ad5c0822f67ec2e7/project-moblab/chromeos-base/chromeos-bsp-moblab/files/mysql_defaults_extra.cnf [delete] https://crrev.com/229a7536e6ab47a5ab6303ce8c417d59b5110302/project-moblab/chromeos-base/chromeos-bsp-moblab/chromeos-bsp-moblab-0.0.1-r46.ebuild
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/bc82af3648a8cc51e7a3e47a226c1445e2b75d39 commit bc82af3648a8cc51e7a3e47a226c1445e2b75d39 Author: Keith Haddow <haddowk@chromium.org> Date: Fri Sep 07 22:30:18 2018 [moblab] Tweak the mysql server parameters. Ran a tool to analize usage, updated the server parameters based on the recomendations of the tool. BUG= chromium:878425 TEST=ad hoc testing on moblab Change-Id: I5d8cebedb597236c0c20296886b6486e10af067b Reviewed-on: https://chromium-review.googlesource.com/1210482 Commit-Ready: Keith Haddow <haddowk@chromium.org> Tested-by: Keith Haddow <haddowk@chromium.org> Reviewed-by: Matt Mallett <mattmallett@chromium.org> (cherry picked from commit 5be2fd6ffd2471526f4433ee32aee4dc23a976e9) Reviewed-on: https://chromium-review.googlesource.com/1213769 Reviewed-by: Keith Haddow <haddowk@chromium.org> Commit-Queue: Keith Haddow <haddowk@chromium.org> Trybot-Ready: Keith Haddow <haddowk@chromium.org> [rename] https://crrev.com/bc82af3648a8cc51e7a3e47a226c1445e2b75d39/project-moblab/chromeos-base/chromeos-bsp-moblab/chromeos-bsp-moblab-0.0.1-r47.ebuild [modify] https://crrev.com/bc82af3648a8cc51e7a3e47a226c1445e2b75d39/project-moblab/chromeos-base/chromeos-bsp-moblab/files/mysql_defaults_extra.cnf [delete] https://crrev.com/4955c913fec27a01ae7b0eaa9cad7fb0b7c475cc/project-moblab/chromeos-base/chromeos-bsp-moblab/chromeos-bsp-moblab-0.0.1-r46.ebuild
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/overlays/board-overlays/+/865b3cc8214ccf38028021e3b3d603fa13082bd9 commit 865b3cc8214ccf38028021e3b3d603fa13082bd9 Author: Keith Haddow <haddowk@chromium.org> Date: Fri Sep 07 22:30:19 2018 [moblab] Tweak the mysql server parameters. Ran a tool to analize usage, updated the server parameters based on the recomendations of the tool. BUG= chromium:878425 TEST=ad hoc testing on moblab Change-Id: I5d8cebedb597236c0c20296886b6486e10af067b Reviewed-on: https://chromium-review.googlesource.com/1210482 Commit-Ready: Keith Haddow <haddowk@chromium.org> Tested-by: Keith Haddow <haddowk@chromium.org> Reviewed-by: Matt Mallett <mattmallett@chromium.org> (cherry picked from commit 5be2fd6ffd2471526f4433ee32aee4dc23a976e9) Reviewed-on: https://chromium-review.googlesource.com/1213767 Commit-Queue: Keith Haddow <haddowk@chromium.org> Trybot-Ready: Keith Haddow <haddowk@chromium.org> Reviewed-by: Keith Haddow <haddowk@chromium.org> [rename] https://crrev.com/865b3cc8214ccf38028021e3b3d603fa13082bd9/project-moblab/chromeos-base/chromeos-bsp-moblab/chromeos-bsp-moblab-0.0.1-r47.ebuild [modify] https://crrev.com/865b3cc8214ccf38028021e3b3d603fa13082bd9/project-moblab/chromeos-base/chromeos-bsp-moblab/files/mysql_defaults_extra.cnf [delete] https://crrev.com/f7e67a52c8db64c8b2bdce33ff81dfdf1ef6e47c/project-moblab/chromeos-base/chromeos-bsp-moblab/chromeos-bsp-moblab-0.0.1-r46.ebuild
Comment 1 by bugdroid1@chromium.org
, Aug 29