New issue
Advanced search Search tips
Starred by 1 user

Issue metadata

Status: Duplicate
Merged: issue 908474
Owner:
Closed: Dec 21
Cc:
Components:
EstimatedDays: ----
NextAction: ----
OS: Linux
Pri: 1
Type: Bug



Sign in to add a comment
link

Issue 914776: deterministic linux dbg build is failing

Reported by ellyjo...@chromium.org, Dec 13 Project Member

Issue description

The step in this build that checks whether two builds are identical is failing. Here's the first failing build: <https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Deterministic%20Linux%20%28dbg%29/3813> and the last succeeding build: <https://ci.chromium.org/p/chromium/builders/luci.chromium.ci/Deterministic%20Linux%20%28dbg%29/3812>

From the stdout of the compare_build_artifacts step here's the first distressed file:

  nacl_bootstrap_x64/obj/native_client/src/trusted/service_runtime/linux/nacl_bootstrap_lib/nacl_bootstrap.o                                                            : 74 out of 18944 bytes are different (0.39%)
  0x2180    : 2f52656c6561736500011100250e1305030e10171b0e11011206000000260000 '/Release....%................&..'
              2f52656c656173652e3200011100250e1305030e10171b0e1101120600000026 '/Release.2....%................&'
                              ++++                                                      ++

It appears to me that something changed that caused nacl to embed part of the outdir path in its binaries?

The got_buildtools_revision and got_nacl_revision are the same for both good and bad builds though.

Over to thakis@ - you seem like you might have an idea of what this bot is/does :)
 

Comment 1 by thakis@chromium.org, Dec 13

nacl_bootstrap_x64/obj/native_client/src/trusted/service_runtime/linux/nacl_bootstrap_lib/nacl_bootstrap.o                                                            : 74 out of 18944 bytes are different (0.39%)


That's true but uninteresting. Or at least it's not why the bot is red. The problem is this:


Unexpected files with diffs:

  libboringssl_fuzzer.so
  libboringssl_fuzzer_no_fuzzer_mode.so
  merkle_integrity_source_stream_fuzzer
  renderer_fuzzer
  renderer_proto_tree_fuzzer
  renderer_tree_fuzzer


(and then it prints diffs of inputs of these files).


My money is on https://chromium-review.googlesource.com/c/chromium/src/+/1374777 ; I'd try reverting that.

Comment 2 by thakis@chromium.org, Dec 13

Owner: robert...@chromium.org
I think the main reason is that capture_context_linux.o is in all these fuzzers, and .S files currently aren't deterministic. Fixing is only blocked on a clang roll (https://bugs.chromium.org/p/chromium/issues/detail?id=908474#c6), so adding these 6 files to tools/deterministic_build/deterministic_build_whitelist.pyl in the meantime would be fine with me too. Over to Robert for doing that.

Comment 3 by ellyjo...@chromium.org, Dec 13

Cc: ellyjo...@chromium.org
I'm reverting 1374777 as <https://chromium-review.googlesource.com/c/chromium/src/+/1375433>.

Comment 4 by st...@chromium.org, Dec 13

Components: Tools>CodeCoverage
Many thanks thakis@ for looking into this and the suggestion! This helps a lot to the code coverage project!

Comment 5 by robertocn@google.com, Dec 13

Status: Started (was: Assigned)
Thanks folks, so if I understand correctly, it should be okay to reland if we include the whitelisting of these affected fuzzers in the same patch. right?

Comment 6 by st...@chromium.org, Dec 14

Is luci.chromium.try/linux_chromium_clobber_deterministic different from luci.chromium.ci/Deterministic Linux (dbg) ?

In the trybot run of the reland (https://chromium-review.googlesource.com/c/chromium/src/+/1376321), the diffs are different from those above, and has as many as 292 files:

  accessibility_unittests
  app_shell_unittests
  appcache_fuzzer
  appcache_manifest_parser_fuzzer
  audio.service
  audio_decoder_fuzzer
  aura_unittests
  autofill_phone_number_i18n_fuzzer
  ax_table_fuzzer
  ax_tree_fuzzer
  background_service_manager_test_service.service
  blink_common_unittests
  blink_fuzzer_unittests
  blink_harfbuzz_shaper_fuzzer
  blink_heap_unittests
  blink_html_tokenizer_fuzzer
  blink_http_parsers_fuzzer
  blink_json_parser_fuzzer
  blink_platform_unittests
  blink_png_decoder_fuzzer
  blink_text_codec_UTF_8_fuzzer
  blink_text_codec_WINDOWS_1252_fuzzer
  boringssl_arm_cpuinfo_fuzzer
  boringssl_bn_div_fuzzer
  boringssl_bn_mod_exp_fuzzer
  boringssl_cert_fuzzer
  boringssl_client_fuzzer
  boringssl_client_no_fuzzer_mode_fuzzer
  boringssl_crypto_tests
  boringssl_dtls_client_fuzzer
  boringssl_dtls_server_fuzzer
  boringssl_pkcs12_fuzzer
  boringssl_pkcs8_fuzzer
  boringssl_privkey_fuzzer
  boringssl_read_pem_fuzzer
  boringssl_server_fuzzer
  boringssl_server_no_fuzzer_mode_fuzzer
  boringssl_session_fuzzer
  boringssl_spki_fuzzer
  boringssl_ssl_ctx_api_fuzzer
  boringssl_ssl_tests
  browser_tests
  capture_unittests
  captured_sites_interactive_tests
  cast_message_fuzzer
  cast_unittests
  cbcs_decryptor_fuzzer
  cc_unittests
  cenc_decryptor_fuzzer
  chrome
  chrome_app_unittests
  chromedriver
  chromedriver_unittests
  clear_site_data_fuzzer
  command_buffer_perftests
  components_browsertests
  components_perftests
  components_unittests
  compositor_unittests
  connect_test_app.service
  connect_test_class_app.service
  connect_test_exe
  connect_test_package.service
  connect_test_singleton_app.service
  content_browsertests
  content_security_policy_fuzzer
  content_shell
  content_unittests
  crashpad_tests
  cronet_tests
  cronet_unittests
  crypto_unittests
  css_parser_fast_paths_fuzzer
  css_parser_proto_fuzzer
  csv_reader_fuzzer
  ctap_response_fuzzer
  device_unittests
  display_unittests
  events_unittests
  extensions_browsertests
  extensions_unittests
  feature_policy_fuzzer
  fido_ble_frames_fuzzer
  fido_cable_handshake_handler_fuzzer
  fido_hid_message_fuzzer
  filesystem.service
  filesystem_service_unittests
  fingerprint_fuzzer
  form_structure_fuzzer
  gcm_unit_tests
  gfx_unittests
  gl_tests
  gl_unittests
  google_apis_unittests
  gpu_angle_fuzzer
  gpu_angle_passthrough_fuzzer
  gpu_fuzzer
  gpu_perftests
  gpu_raster_angle_fuzzer
  gpu_raster_fuzzer
  gpu_raster_swiftshader_fuzzer
  gpu_swiftshader_fuzzer
  gpu_unittests
  headless_browsertests
  headless_unittests
  hid_report_descriptor_fuzzer
  hit_test_manager_fuzzer
  hit_test_query_fuzzer
  host_test_service.service
  html_preload_scanner_fuzzer
  indexed_ruleset_fuzzer
  interactive_ui_tests
  ipc_tests
  jingle_unittests
  language_detection_fuzzer
  latency_unittests
  leveldb.service
  leveldb_service_unittests
  libaddressinput_parse_address_fields_fuzzer
  libaddressinput_parse_format_rule_fuzzer
  libblink_deprecated_test_plugin.so
  libblink_test_plugin.so
  libclearkeycdm.so
  libcorb_test_plugin.so
  libcrashpad_snapshot_test_module.so
  libcrashpad_snapshot_test_module_large.so
  libcrashpad_snapshot_test_module_small.so
  libjingle_xmpp_unittests
  libmojo_core.so
  libpower_saver_test_plugin.so
  libppapi_tests.so
  libsrtp_fuzzer
  lifecycle_unittest_app.service
  lifecycle_unittest_exe
  lifecycle_unittest_package.service
  lifecycle_unittest_parent.service
  media.service
  media_bit_reader_fuzzer
  media_blink_unittests
  media_container_names_fuzzer
  media_h264_parser_fuzzer
  media_perftests
  media_pipeline_integration_fuzzer
  media_service_unittests
  media_unittests
  media_vp8_parser_fuzzer
  media_vp9_parser_encrypted_fuzzer
  media_vp9_parser_fuzzer
  media_vpx_video_decoder_fuzzer
  media_webm_muxer_fuzzer
  mediasource_MP3_pipeline_integration_fuzzer
  mediasource_MP4_AV1_pipeline_integration_fuzzer
  mediasource_MP4_FLAC_pipeline_integration_fuzzer
  mediasource_MP4_OPUS_pipeline_integration_fuzzer
  mediasource_WEBM_OPUS_VP9_pipeline_integration_fuzzer
  mediasource_WEBM_OPUS_pipeline_integration_fuzzer
  mediasource_WEBM_VORBIS_pipeline_integration_fuzzer
  mediasource_WEBM_VP8_pipeline_integration_fuzzer
  mediasource_WEBM_VP9_pipeline_integration_fuzzer
  merkle_integrity_source_stream_fuzzer
  message_center_unittests
  mhtml_parser_fuzzer
  midi_message_queue_fuzzer
  midi_unittests
  midi_webmidi_data_validator_fuzzer
  mojo_core_channel_fuzzer
  mojo_core_node_channel_fuzzer
  mojo_core_port_event_fuzzer
  mojo_core_user_message_fuzzer
  mojo_parse_message_fuzzer
  mojo_parse_message_proto_fuzzer
  mojo_unittests
  nacl_helper
  nacl_helper_bootstrap
  nacl_loader_unittests
  native_theme_unittests
  net_auth_challenge_tokenizer_fuzzer
  net_brotli_source_stream_fuzzer
  net_cert_normalize_name_fuzzer
  net_cert_parse_certificate_fuzzer
  net_cert_verify_name_in_subtree_fuzzer
  net_cert_verify_name_match_fuzzer
  net_data_job_fuzzer
  net_dns_hosts_parse_fuzzer
  net_dns_query_parse_fuzzer
  net_dns_record_fuzzer
  net_dns_response_fuzzer
  net_ftp_ctrl_response_fuzzer
  net_ftp_directory_listing_fuzzer
  net_get_domain_and_registry_fuzzer
  net_gzip_source_stream_fuzzer
  net_host_resolver_impl_fuzzer
  net_hpack_decoder_fuzzer
  net_http2_frame_decoder_fuzzer
  net_http_chunked_decoder_fuzzer
  net_http_proxy_client_socket_fuzzer
  net_http_security_headers_expect_ct_fuzzer
  net_http_security_headers_hsts_fuzzer
  net_http_server_fuzzer
  net_http_stream_parser_fuzzer
  net_http_transport_security_state_static_fuzzer
  net_mime_sniffer_fuzzer
  net_ntlm_ntlm_client_fuzzer
  net_parse_cookie_line_fuzzer
  net_parse_data_url_fuzzer
  net_parse_ip_pattern_fuzzer
  net_parse_proxy_bypass_rules_fuzzer
  net_parse_proxy_list_fuzzer
  net_parse_proxy_list_pac_fuzzer
  net_parse_proxy_rules_fuzzer
  net_parse_url_hostname_to_address_fuzzer
  net_perftests
  net_quic_crypto_framer_parse_message_fuzzer
  net_quic_stream_factory_fuzzer
  net_quic_transport_parameters_fuzzer
  net_reporting_header_parser_fuzzer
  net_socks5_client_socket_fuzzer
  net_socks_client_socket_fuzzer
  net_spdy_session_fuzzer
  net_unescape_url_component_fuzzer
  net_unittests
  net_uri_template_fuzzer
  net_url_request_ftp_fuzzer
  net_url_request_fuzzer
  net_websocket_deflate_stream_fuzzer
  net_websocket_extension_parser_fuzzer
  net_websocket_frame_parser_fuzzer
  origin_policy_parser_fuzzer
  origin_trial_token_fuzzer
  paint_op_buffer_fuzzer
  password_manager_form_parser_fuzzer
  password_manager_form_parser_generic_fuzzer
  password_manager_form_parser_proto_fuzzer
  password_manager_form_parser_proto_generic_fuzzer
  payment_method_manifest_fuzzer
  payment_web_app_manifest_fuzzer
  pdf_unittests
  performance_browser_tests
  ppapi_unittests
  preg_parser_fuzzer
  printing_unittests
  remoting_unittests
  renderer_fuzzer
  renderer_proto_tree_fuzzer
  renderer_tree_fuzzer
  sequence_manager_fuzzer
  service_manager_unittest_embedder.service
  service_manager_unittest_target
  service_manager_unittests
  services_unittests
  shell_dialogs_unittests
  shutdown_client.service
  shutdown_service.service
  signed_exchange_certificate_chain_fuzzer
  signed_exchange_envelope_fuzzer
  signed_exchange_signature_header_field_fuzzer
  skia_unittests
  snapshot_unittests
  storage_unittests
  stylesheet_contents_fuzzer
  sync_integration_tests
  template_url_parser_fuzzer
  test_ime_driver.service
  test_ws.service
  text_resource_decoder_fuzzer
  transfer_cache_fuzzer
  ui_base_unittests
  ui_touch_selection_unittests
  unit_tests
  url_unittests
  usb_descriptors_fuzzer
  usrsctp_fuzzer
  v4_get_hash_protocol_manager_fuzzer
  v8_serialized_script_value_fuzzer
  video_capture.service
  views_mus_interactive_ui_tests
  views_mus_unittests
  views_perftests
  views_unittests
  viz_unittests
  vr_common_perftests
  vr_common_unittests
  vr_pixeltests
  web_icon_sizes_fuzzer
  webcrypto_ec_import_key_pkcs8_fuzzer
  webcrypto_ec_import_key_raw_fuzzer
  webcrypto_ec_import_key_spki_fuzzer
  webcrypto_rsa_import_key_pkcs8_fuzzer
  webcrypto_rsa_import_key_spki_fuzzer
  webkit_unit_tests
  wm_unittests
  xml_parser_fuzzer

Comment 7 Deleted

Comment 8 Deleted

Comment 9 by bugdroid1@chromium.org, Dec 14

Project Member
The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1460da851dc26572b5ac6e16bf8b387f26bbffb6

commit 1460da851dc26572b5ac6e16bf8b387f26bbffb6
Author: Roberto Carrillo <robertocn@google.com>
Date: Fri Dec 14 17:10:39 2018

Reland "Reland of: [code coverage] Add all fuzzer targets"

This is a reland of 80cee65bec02ee6edbd88670120867bc0af8e028

Original change's description:
> Reland of: [code coverage] Add all fuzzer targets
>
> Now with a fix for the issue in GenerateIsolates().
>
> R=jbudorick,stgao,liaoyuke
>
> > Change-Id: I78ffec6525cd1d808a4b431ae1e2729b76357d46
> > Reviewed-on: https://chromium-review.googlesource.com/c/1372917
> > Commit-Queue: Shuotao Gao <stgao@chromium.org>
> > Reviewed-by: Shuotao Gao <stgao@chromium.org>
> > Reviewed-by: John Budorick <jbudorick@chromium.org>
> > Cr-Commit-Position: refs/heads/master@{#615834}
>
> Change-Id: I4ae0ef2588026bb6e0e5501e9ff39a7852d89afa
> Reviewed-on: https://chromium-review.googlesource.com/c/1374777
> Reviewed-by: Yuke Liao <liaoyuke@chromium.org>
> Reviewed-by: John Budorick <jbudorick@chromium.org>
> Commit-Queue: Roberto Carrillo <robertocn@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#616169}

Change-Id: I360303f0c4ecb2b818c4ad6e936d3d4b1dfb1807
Bug:  914776 
Reviewed-on: https://chromium-review.googlesource.com/c/1376321
Reviewed-by: Shuotao Gao <stgao@chromium.org>
Reviewed-by: Nico Weber <thakis@chromium.org>
Commit-Queue: Shuotao Gao <stgao@chromium.org>
Cr-Commit-Position: refs/heads/master@{#616716}
[modify] https://crrev.com/1460da851dc26572b5ac6e16bf8b387f26bbffb6/testing/buildbot/chromium.fyi.json
[modify] https://crrev.com/1460da851dc26572b5ac6e16bf8b387f26bbffb6/testing/buildbot/gn_isolate_map.pyl
[modify] https://crrev.com/1460da851dc26572b5ac6e16bf8b387f26bbffb6/testing/buildbot/test_suites.pyl
[modify] https://crrev.com/1460da851dc26572b5ac6e16bf8b387f26bbffb6/testing/buildbot/tryserver.chromium.linux.json
[modify] https://crrev.com/1460da851dc26572b5ac6e16bf8b387f26bbffb6/testing/buildbot/waterfalls.pyl
[modify] https://crrev.com/1460da851dc26572b5ac6e16bf8b387f26bbffb6/tools/determinism/deterministic_build_whitelist.pyl
[modify] https://crrev.com/1460da851dc26572b5ac6e16bf8b387f26bbffb6/tools/mb/mb.py
[modify] https://crrev.com/1460da851dc26572b5ac6e16bf8b387f26bbffb6/tools/mb/mb_unittest.py

Comment 10 by thakis@chromium.org, Dec 21

Mergedinto: 908474
Status: Duplicate (was: Started)
This looks like the "targets with .S files aren't deterministic" bug.

Sign in to add a comment