In https://chromium-review.googlesource.com/c/chromium/src/+/1351481, I was able to add a non existing benchmark to the shard map and have no problem submitting it. 

tools/perf/PRESUBMIT should prevent this from happening 


We already have a validator script: ./tools/perf/validate_perf_json_config that is run on PRESUBMIT. We can extend this script to error on such case.