Revise Flashrom test scripts
Reported by
dhend...@chromium.org,
Jun 20 2016
|
||||||||||||
Issue descriptionI'm in the process of re-writing the Flashrom test scripts with the goal of making them much easier for real test scenarios we care about. Hopefully this will prove useful to others (including upstream) and enable automated testing. Feel free to chip in ideas and feature requests as we go. I uploaded the first patch for this effort with basic command-line handling and general interface, other functionality is yet-to-be-implemented. Here is my WIP patch: https://chromium-review.googlesource.com/#/c/353912 For now, the show_help() function in test_v2.sh describes in rough detail how the script can be used and is the only real documentation... Use cases I am targeting: - Developer wishes to test flashrom on the system which they are currently running on. For folks who like to live dangerously (and upstream devs). - Developer wishes to test flashrom on remote host via SSH. - Developer wishes to test flashrom on a remote host via SSH and verify the result using an external programmer such as Servo, Dediprog, RaspPi, etc. The developer will be able to tell the script which regions may be utilized for testing via different region access modes: "clobber" where everything is fair game, "descriptor" where we avoid locked regions according to the Flash Descriptor (Intel systems), "flashmap" where we use the RW region for testing (ChromeOS devices), and "layout" where the user supplies a layout file. The script is intended to control one or two programmers, referred to as primary and secondary programmers. The primary programmer may used on either the machine which is running the script, or on a remote host specified via -r/--remote-host. The secondary programmer is always assumed to be connected to the machine which is running the script. These assumptions allow us to avoid messy logic and command-line switches to deal with which programmer is connected to which machine. TODO (near future to help with Souvik's work): - Basic functionality - Ability to test read/erase/write in "clobber" mode (wipe out whole chip) - "Flashmap mode" to narrow it down to testing regions which are known to be writeable. - Edge cases (partial reads/writes across block boundaries, etc). - Documentation, probably in the form of a wiki page on flashrom.org. Other goals: - Add other modes (descriptor, layout) - Add non-destructive tests that are suitable for an autotest. - Add basic write-protect testing. (Advanced WP testing might be best done thru whitebox tests). - Add a way to upload results to automate supported chipset/flash chip reporting (like board_status.sh in coreboot).
,
Jun 21 2016
Sounds good. A stress test mode will definitely be useful when adding chips to the AVL. We can call it "endurance mode" (--endurance) and take an integer as an argument (<=0 loops forever) and stop if we run into any errors, and report how many iterations we went thru in the end. Performance measurement would be useful, but might be kind of tricky. I'm not sure how useful comparing the two programmers is - We always expect the "internal" programmer on the host to be much faster than Servo, for example. However, it would be very useful to compare a new flashrom version's performance against a previous version. A couple things to consider: 1. For this script we expect a new version of flashrom to be tested against an old version. Comparing performance should be easy, so I'll add a --benchmark mode to do that. 2. For autotest, will we have the ability to fetch an old version of flashrom from somewhere to compare? I figured we'd probably need to use the same version as both the "old" and "new" since we might not have an older binary available. Another approach could be to tie this in with the last item on my list and have the script utilize (and perhaps maintain) a text file that resides in the flashrom directory. When benchmarking, it can grep the text file for platform name (or SoC or something) and look up performance data in one of the columns. So in this case we'd have two benchmark modes, one which compares a new binary's performance relative to a given old binary, and and one which compares a new binary's performance relative to "canonical" data stored in git.
,
Jul 6 2016
,
Jul 6 2016
I had some time to hack on the test script recently and it's shaping up quite nicely! As of patch set 20 it supports testing on a remote host, dual programmers (the secondary programmer for additional verification), targeting regions via the flashmap, and a bunch of other stuff. And all this basic stuff comes in at just over 900 lines of shell ;-) https://chromium-review.googlesource.com/#/c/353912/ I also wrote up a manual with illustrated examples for how to use it: https://goo.gl/3jNoL7 I'll try to polish up the patch in its current state and get the first version committed. After that we can focus on updating/adding autotests and features to help with automated testing In addition to what Danny suggested, we should also add a read-only test and perhaps also add a new dummy programmer that writes to a file so that we can test write logic without wearing out the ROM. In the meantime, Souvik can help test/debug the script by using it for his work getting us sync'd with upstream (issue #478356).
,
Jul 20 2016
Danny, Souvik and I met briefly and came up with a few more things that need to be addressed: - How to plug script into the Autotest framework. We took a cursory glance at some tests but didn't see a particularly good example of how we'll invoke the script from an autotest control file and collect what needs to be collected. So for now Souvik and I will work on step-by-step documentation to help get a control file written which we can then use as a template for more tests. - How often to run "destructive" tests (anything that involves clobbering the flash) in order to avoid hardware failure due to endurance limits. - We definitely need to test host/BIOS firmware and EC firmware reads/writes/WP. How do we test PD and other chips? For PD we may even need to match charger to system. - Logging. Souvik is currently importing an upstream flashrom patch that adds logfile support (CL:361822), so we'll have a verbose flashrom log created as a file (no need to explicitly capture stdout/stderr). The test script itself will produce output which can be captured by stdout, though perhaps we can replace its print statements (echo and printfs) with a wrapper that also directs to a file. That way we'll have a flashrom log and a test report from the script.
,
Jul 22 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/flashrom/+/40d8acb0232a6bb64db4ad72bd51efd4349e992a commit 40d8acb0232a6bb64db4ad72bd51efd4349e992a Author: David Hendricks <dhendrix@chromium.org> Date: Sun Jun 19 04:51:09 2016 Move test scripts from util/ to tests/tests_v1 Moving old tests to make room for new ones. BUG=chromium:621715 BRANCH=none TEST=see follow-up patches Signed-off-by: David Hendricks <dhendrix@chromium.org> Change-Id: I4859994e8d6106cc626537df1848d569cb4a57c7 Reviewed-on: https://chromium-review.googlesource.com/353911 [rename] https://crrev.com/40d8acb0232a6bb64db4ad72bd51efd4349e992a/tests/tests_v1/wp-range.sh [rename] https://crrev.com/40d8acb0232a6bb64db4ad72bd51efd4349e992a/tests/tests_v1/chip_size.sh [rename] https://crrev.com/40d8acb0232a6bb64db4ad72bd51efd4349e992a/tests/tests_v1/do_tests.sh [rename] https://crrev.com/40d8acb0232a6bb64db4ad72bd51efd4349e992a/tests/tests_v1/partial_reads.sh [rename] https://crrev.com/40d8acb0232a6bb64db4ad72bd51efd4349e992a/tests/tests_v1/partial_writes_ec.sh [rename] https://crrev.com/40d8acb0232a6bb64db4ad72bd51efd4349e992a/tests/tests_v1/common.sh [rename] https://crrev.com/40d8acb0232a6bb64db4ad72bd51efd4349e992a/tests/tests_v1/README [rename] https://crrev.com/40d8acb0232a6bb64db4ad72bd51efd4349e992a/tests/tests_v1/wp-toggle.sh [rename] https://crrev.com/40d8acb0232a6bb64db4ad72bd51efd4349e992a/tests/tests_v1/partial_writes_x86_bios.sh
,
Jul 22 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/flashrom/+/5b972c94025639e58ac1a02c994bc264ed7fa602 commit 5b972c94025639e58ac1a02c994bc264ed7fa602 Author: David Hendricks <dhendrix@chromium.org> Date: Sun Jun 19 04:14:37 2016 rename tests.py Make way for new tests! BUG=chromium:621715 BRANCH=none TEST=see follow-up patches Signed-off-by: David Hendricks <dhendrix@chromium.org> Change-Id: Ifa33ef555d50e88f4cbc0ae09d40cabc29a282c8 Reviewed-on: https://chromium-review.googlesource.com/353910 [rename] https://crrev.com/5b972c94025639e58ac1a02c994bc264ed7fa602/tests/fdt_test.py
,
Jul 25 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/flashrom/+/1aee44a63b98d1c305215f7b2acbd6d3159415e9 commit 1aee44a63b98d1c305215f7b2acbd6d3159415e9 Author: David Hendricks <dhendrix@chromium.org> Date: Sun Jun 19 19:53:22 2016 New test script for flashrom Shiny new testing capabilities for Flashrom: - Region awareness - Remote testing - Primary and secondary programmer support - Autotest-friendly Long-term goals: - Performance measurement - Endurance testing BUG=chromium:621715 BRANCH=none TEST=this *is* the test Change-Id: Ic643cf2edaf7d8772c63ce8119363d882d1b116d Signed-off-by: David Hendricks <dhendrix@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/353912 [add] https://crrev.com/1aee44a63b98d1c305215f7b2acbd6d3159415e9/tests/tests_v2/test_v2.sh [add] https://crrev.com/1aee44a63b98d1c305215f7b2acbd6d3159415e9/tests/tests_v2/cmd.sh
,
Jul 30 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/flashrom/+/915e74112423c47738210b81367c7e06e246a4da commit 915e74112423c47738210b81367c7e06e246a4da Author: David Hendricks <dhendrix@chromium.org> Date: Thu Jul 21 22:45:54 2016 test_v2: Add support for custom hooks file This adds an option, --custom-hooks, which allows the user to specify a script which will be sourced for custom hooks which will be called before and after each command is run. The primary use case is to run extra programs needed before flashrom is used. BUG=chromium:621715 BRANCH=none TEST=Used with servo which requires running "dut-control" before and after a test to enable and disable SPI buffers on its flex cable. Change-Id: Id5c3829b6b30cebf00dff41e6b3cab0eff6357e2 Signed-off-by: David Hendricks <dhendrix@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/362388 [modify] https://crrev.com/915e74112423c47738210b81367c7e06e246a4da/tests/tests_v2/test_v2.sh [modify] https://crrev.com/915e74112423c47738210b81367c7e06e246a4da/tests/tests_v2/cmd.sh
,
Aug 29 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/flashrom/+/2aaacdfb783a19bea973fb2f4df121c4af1e4cea commit 2aaacdfb783a19bea973fb2f4df121c4af1e4cea Author: Souvik Ghosh <souvikghosh@google.com> Date: Wed Jul 27 02:40:15 2016 Building on test script - endurance/wp modes, logging, code cleanup Adds endurance and write-protect test types to the test script. Also, logs of the script output and any failed flashrom runs are stored in the temporary directory. A future feature might be to auto-upload these results to gerrit or the flashrom website. BUG=chromium:621715 BRANCH=none TEST=needs testing Change-Id: Iaffee4cf4cfcfcb59209ba7ac17a29a47adf9e92 Signed-off-by: Souvik Ghosh <souvikghosh@google.com> Reviewed-on: https://chromium-review.googlesource.com/363585 Commit-Ready: David Hendricks <dhendrix@chromium.org> Reviewed-by: David Hendricks <dhendrix@chromium.org> [modify] https://crrev.com/2aaacdfb783a19bea973fb2f4df121c4af1e4cea/tests/tests_v2/test_v2.sh [modify] https://crrev.com/2aaacdfb783a19bea973fb2f4df121c4af1e4cea/tests/tests_v2/cmd.sh
,
Aug 29 2016
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/flashrom/+/04b1cc2f708d9b14879c4c25b9c0046862b3a9ed commit 04b1cc2f708d9b14879c4c25b9c0046862b3a9ed Author: Souvik Ghosh <souvikghosh@google.com> Date: Sat Aug 13 19:35:49 2016 test_v2: add wp hooks files for servo-v2 CL:363585 added the functionality for wp-hooks, or the functions that an external programmer can use to enable/disable write protection and turn the current wp setting on/off. This is an example file for how the wp hooks might look. This file corresponds to the servo-v2 commands for running on veyron_jaq. BUG=chromium:621715 BRANCH=none TEST=none Change-Id: I2002ee0966eb7f333dab0826e417daee50adeff0 Signed-off-by: Souvik Ghosh <souvikghosh@google.com> Reviewed-on: https://chromium-review.googlesource.com/368882 Commit-Ready: David Hendricks <dhendrix@chromium.org> Tested-by: David Hendricks <dhendrix@chromium.org> Reviewed-by: David Hendricks <dhendrix@chromium.org> [add] https://crrev.com/04b1cc2f708d9b14879c4c25b9c0046862b3a9ed/tests/tests_v2/servo_wp_hooks_jaq.sh
,
Sep 21 2016
,
Oct 6 2016
Issue 529347 has been merged into this issue.
,
Jan 11 2017
,
Jan 13 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/flashrom/+/a71ee6b44aec85565aac4fa4a932df1344f2da75 commit a71ee6b44aec85565aac4fa4a932df1344f2da75 Author: David Hendricks <dhendrix@chromium.org> Date: Wed Nov 09 21:33:47 2016 test_v2: Trivial cleanup - Get rid of a couple unused globals. - Get rid of a stray semicolon. - Add a print to indicate when backup image is being read. - Add a FIXME to utilize the secondary programmer, if applicable, in double_read_test(). BUG=chromium:621715 BRANCH=none TEST=script still works. Change-Id: I5316be716e61fd940c2b6d6962c8a8f645fdb6e6 Signed-off-by: David Hendricks <dhendrix@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/410146 Reviewed-by: Martin Roth <martinroth@chromium.org> [modify] https://crrev.com/a71ee6b44aec85565aac4fa4a932df1344f2da75/tests/tests_v2/test_v2.sh
,
Jan 13 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/flashrom/+/9cc415d4e8db33e4c569223208efbf598fa38f47 commit 9cc415d4e8db33e4c569223208efbf598fa38f47 Author: David Hendricks <dhendrix@chromium.org> Date: Wed Nov 09 21:35:16 2016 test_v2: Add -v/--voltage arg and make Servo hooks generic This allows the user to pass a voltage value into the test script which will then get passed into the Servo hooks. Although it adds slightly more complexity to the command-line, it actually simplifies things from a scripting perspective since the user doesn't need to select a different set of hooks depending on the voltage of the chip. They just need to pass in the same voltage which they use to enable the SPI buffers via dut-control. BUG=chromium:621715 BRANCH=none TEST=tested with follow-up patches Change-Id: Iac34dd89c7241a4171818a81126088fbdd01f4cf Signed-off-by: David Hendricks <dhendrix@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/410147 Reviewed-by: Martin Roth <martinroth@chromium.org> [modify] https://crrev.com/9cc415d4e8db33e4c569223208efbf598fa38f47/tests/tests_v2/test_v2.sh [rename] https://crrev.com/9cc415d4e8db33e4c569223208efbf598fa38f47/tests/tests_v2/servo_wp_hooks.sh
,
Jan 13 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/flashrom/+/d4b7e58564774781518cc8f68a001a0f4a455917 commit d4b7e58564774781518cc8f68a001a0f4a455917 Author: David Hendricks <dhendrix@chromium.org> Date: Thu Nov 10 21:23:12 2016 test_v2: Add a sanity check to wp hooks This adds a simple self-test step for write-protect hooks to ensure dut-control has the necessary functionality. This prevents us from seeing misleading failures later on if the user doesn't have everything set up just right. BUG=chromium:621715 BRANCH=none TEST=tested using Reef Change-Id: Ia283c95c3d4386aff32abe7ff8aded1a2ea7c6ef Signed-off-by: David Hendricks <dhendrix@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/410148 Reviewed-by: Martin Roth <martinroth@chromium.org> [modify] https://crrev.com/d4b7e58564774781518cc8f68a001a0f4a455917/tests/tests_v2/test_v2.sh [modify] https://crrev.com/d4b7e58564774781518cc8f68a001a0f4a455917/tests/tests_v2/servo_wp_hooks.sh
,
Jan 13 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/flashrom/+/488afc56f21e550da11a1eb4740da195ec33fc23 commit 488afc56f21e550da11a1eb4740da195ec33fc23 Author: David Hendricks <dhendrix@chromium.org> Date: Thu Dec 22 00:07:41 2016 test_v2: Fixes to flashrom_log_scmd() This fixes a few things with flashrom_log_scmd(): - Update function args to take $LOCAL/$REMOTE into account. - Propagate return code to caller. - Declare tmpdir locally and set it conditionally rather than defaulting to $LOCAL_TMPDIR (making it a bit easier to follow where it gets set when 'sh -x' is used to invoke the script). BUG=chromium:621715 BRANCH=none TEST=tested with follow-up patches Signed-off-by: David Hendricks <dhendrix@chromium.org> Change-Id: I7474d1beca8941bb91c94023935c476a667bffdb Reviewed-on: https://chromium-review.googlesource.com/410149 Reviewed-by: Martin Roth <martinroth@chromium.org> [modify] https://crrev.com/488afc56f21e550da11a1eb4740da195ec33fc23/tests/tests_v2/test_v2.sh
,
Jan 13 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/flashrom/+/c5365d7105921b6b8ff24d662f7ffeed8d538a05 commit c5365d7105921b6b8ff24d662f7ffeed8d538a05 Author: David Hendricks <dhendrix@chromium.org> Date: Wed Dec 21 23:59:05 2016 test_v2: Simplify logic in save_wp_state() This attempts to simplify the logic used to scrape the output from "--wp-status". BUG=chromium:621715 BRANCH=none TEST=tested script on veyron_mighty Signed-off-by: David Hendricks <dhendrix@chromium.org> Change-Id: I03503a11570771e6073b347378f9260f819bb1a9 Reviewed-on: https://chromium-review.googlesource.com/422914 Reviewed-by: Martin Roth <martinroth@chromium.org> [modify] https://crrev.com/c5365d7105921b6b8ff24d662f7ffeed8d538a05/tests/tests_v2/test_v2.sh
,
Jan 13 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/flashrom/+/f7cffb7369908722fc719237d72acf80db296b55 commit f7cffb7369908722fc719237d72acf80db296b55 Author: David Hendricks <dhendrix@chromium.org> Date: Sat Dec 17 04:19:55 2016 test_v2: Skip cleanup if a test fails BUG=chromium:621715 BRANCH=none TEST=tested using Reef Change-Id: I2a404efc8b4077ff4188d5e16983f87f22d1e454 Signed-off-by: David Hendricks <dhendrix@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/422943 Reviewed-by: Martin Roth <martinroth@chromium.org> [modify] https://crrev.com/f7cffb7369908722fc719237d72acf80db296b55/tests/tests_v2/test_v2.sh
,
Jan 13 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/flashrom/+/815a6b3db8705bc51d56c7bfb141611c5e934c2e commit 815a6b3db8705bc51d56c7bfb141611c5e934c2e Author: David Hendricks <dhendrix@chromium.org> Date: Sat Dec 17 01:00:42 2016 test_v2: Add 1sec delay after each dut-control command This seems to make it somewhat more reliable. BUG=chromium:621715 BRANCH=none TEST=tested with follow-up patches Change-Id: Ic4dd90c14af70986d0098fcd227cc84d1194f5e6 Signed-off-by: David Hendricks <dhendrix@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/422944 Reviewed-by: Martin Roth <martinroth@chromium.org> [modify] https://crrev.com/815a6b3db8705bc51d56c7bfb141611c5e934c2e/tests/tests_v2/servo_wp_hooks.sh
,
Jan 13 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/flashrom/+/9cfa12bfd75c897e18010ddd06694e0a220c692c commit 9cfa12bfd75c897e18010ddd06694e0a220c692c Author: David Hendricks <dhendrix@chromium.org> Date: Tue Dec 20 02:27:00 2016 test_v2: Make servo hooks more generic This makes it so that servo_hooks.sh can be used as the argument for both --custom-hooks and --wp_hooks: - Add preflash_hook() and postflash_hook(). - Add variables that can be set in user's environment to control dut-control: - SERVO_PORT: Port to listen on (default: 9999) - SERVO_SPI: SPI channel to use (default: spi2) - Voltage gets passed in as an environment variable, so no need to pass it in explicitly as an argument to wp_enable_hook. - Rename servo_wp_hooks.sh to servo_hooks.sh. BUG=chromium:621715 BRANCH=none TEST=none Change-Id: I8c69c98d4e6cc997833bcc9a82a968933053db29 Signed-off-by: David Hendricks <dhendrix@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/422945 Reviewed-by: Martin Roth <martinroth@chromium.org> [modify] https://crrev.com/9cfa12bfd75c897e18010ddd06694e0a220c692c/tests/tests_v2/test_v2.sh [delete] https://crrev.com/d4b7e58564774781518cc8f68a001a0f4a455917/tests/tests_v2/servo_wp_hooks.sh [add] https://crrev.com/9cfa12bfd75c897e18010ddd06694e0a220c692c/tests/tests_v2/servo_hooks.sh
,
Jan 13 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/flashrom/+/89252419206a683278ba0f14cc01cbf7470e4f42 commit 89252419206a683278ba0f14cc01cbf7470e4f42 Author: David Hendricks <dhendrix@chromium.org> Date: Tue Dec 20 04:20:03 2016 test_v2: Fix layout path for secondary programmer Secondary programmer should always use LOCAL_TMPDIR for layout path. BUG=chromium:621715 BRANCH=none TEST=Tested script using remote host with Servo attached. Change-Id: I5c9a2399c378a0565515b088de73114c281b5e9b Signed-off-by: David Hendricks <dhendrix@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/422946 Reviewed-by: Martin Roth <martinroth@chromium.org> [modify] https://crrev.com/89252419206a683278ba0f14cc01cbf7470e4f42/tests/tests_v2/test_v2.sh
,
Jan 13 2017
The following revision refers to this bug: https://chromium.googlesource.com/chromiumos/third_party/flashrom/+/52bbeeb28c086d381b213cee6c37a3ea8434f49c commit 52bbeeb28c086d381b213cee6c37a3ea8434f49c Author: David Hendricks <dhendrix@chromium.org> Date: Thu Dec 22 01:52:05 2016 test_v2: Add custom_hook_sanity_check() This adds a sanity check for when we're using custom hooks. The first use case is to check that dut-control can obtain certain values that are set by other some other hooks. The hook is intended to run early so that we don't get confusing errors several seconds after the script begins. BUG=chromium:621715 BRANCH=none TEST=ran with servod listening on a non-default port, script erred out sooner rather than later. Change-Id: I63d83f6d6fc9e26caadfb4c05c73e07f4018c0f9 Signed-off-by: David Hendricks <dhendrix@chromium.org> Reviewed-on: https://chromium-review.googlesource.com/422997 Reviewed-by: Martin Roth <martinroth@chromium.org> [modify] https://crrev.com/52bbeeb28c086d381b213cee6c37a3ea8434f49c/tests/tests_v2/cmd.sh [modify] https://crrev.com/52bbeeb28c086d381b213cee6c37a3ea8434f49c/tests/tests_v2/servo_hooks.sh
,
Jan 13 2017
,
Feb 3 2017
,
Aug 18 2017
,
Aug 20
This issue has been Available for over a year. If it's no longer important or seems unlikely to be fixed, please consider closing it out. If it is important, please re-triage the issue. Sorry for the inconvenience if the bug really should have been left as Available. For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot
,
Aug 20
,
Oct 15
,
Nov 2
|
||||||||||||
►
Sign in to add a comment |
||||||||||||
Comment 1 by dchan@google.com
, Jun 21 2016