New issue
Advanced search Search tips

Issue 899442 link

Starred by 1 user
Status: Fixed
Owner:
shinyak@chromium.org
Closed: Nov 27
Cc:
tikuta@chromium.org
most...@vewd.com
shinyak@chromium.org
r...@chromium.org
ukai@chromium.org
yyanagisawa@chromium.org
sque@chromium.org
Components:
EstimatedDays: ----
NextAction: ----
OS: Windows
Pri: 3
Type: Bug

Blocking:
issue 899438



Sign in to add a comment

Two files in skia aren't deterministic due to goma preprocessor bug

Project Member Reported by thakis@chromium.org, Oct 27 
I built chrome twice on my win box, once in out\gn, once in out\gn2. args.gn both times was just:

symbol_level = 1
use_goma = true

Then I ran

python tools\determinism\compare_build_artifacts.py --first-build-dir=out\gn --second-build-dir=out\gn2 -t win

with the script hacked up so that get_files_to_compare() is just

  return set(['content.dll'])


The output contained:


From issue 899438:

: different size: 1468 != 1469
  obj/skia/skia/GrRadialGradientLayout.obj                                                                                                                                : different size: 761521 != 761524
  obj/skia/skia/GrGLShaderStringBuilder.obj                        


What's interesting is this part of the diff of GrGLShaderStringBuilder.obj (also attached):

000647b0: 636c 7564 652f 636f 7265 5c2e 2e2f 7072  clude/core\../pr
000647c0: 6976 6174 652f 536b 4174 6f6d 6963 732e  ivate/SkAtomics.
000647d0: 6800 5f00 4100 7400 6f00 6d00 6900 6300  h._.A.t.o.m.i.c.


vs

000647b0: 636c 7564 652f 635c 2e2e 2f70 7269 7661  clude/c\../priva
000647c0: 7465 2f53 6b41 746f 6d69 6373 2e68 005f  te/SkAtomics.h._
000647d0: 0041 0074 006f 006d 0069 0063 005f 006c  .A.t.o.m.i.c._.l


This seems to be in the middle of string literals for asserts. Note that one says include/core\../private/SkAtomics.h while the other says include/c\../private/SkAtomics.h


That looks a bit like it might be a StringRef memory stomper somewhere in the compiler? (The total list of different files is small, see blockee of this bug.)
GrGlShaderStringBuilder.1.obj
912 KB Download
GrGlShaderStringBuilder.2.obj
912 KB Download

Comment 1 by thakis@chromium.org, Oct 27 

Here's some of the identical stuff preceding this:

000646e0: 7420 3e20 6652 6573 6572 7665 2900 6173  t > fReserve).as
000646f0: 7365 7274 2853 6b54 4669 7473 496e 3c69  sert(SkTFitsIn<i
00064700: 6e74 3e28 7265 7365 7276 6529 2900 6173  nt>(reserve)).as
00064710: 7365 7274 286d 6f20 3d3d 2073 6b5f 6d65  sert(mo == sk_me
00064720: 6d6f 7279 5f6f 7264 6572 5f72 656c 6178  mory_order_relax
00064730: 6564 207c 7c20 6d6f 203d 3d20 736b 5f6d  ed || mo == sk_m
00064740: 656d 6f72 795f 6f72 6465 725f 7365 715f  emory_order_seq_
00064750: 6373 7420 7c7c 206d 6f20 3d3d 2073 6b5f  cst || mo == sk_
00064760: 6d65 6d6f 7279 5f6f 7264 6572 5f61 6371  memory_order_acq
00064770: 7569 7265 207c 7c20 6d6f 203d 3d20 736b  uire || mo == sk
00064780: 5f6d 656d 6f72 795f 6f72 6465 725f 636f  _memory_order_co
00064790: 6e73 756d 6529 002e 2e2f 2e2e 2f74 6869  nsume).../../thi
000647a0: 7264 5f70 6172 7479 2f73 6b69 612f 696e  rd_party/skia/in


It's interesting that there's a wide char literal following that difference; maybe that's triggering something.


The assert strings come from https://cs.chromium.org/chromium/src/third_party/skia/include/private/SkAtomics.h?type=cs&sq=package:chromium&g=0&l=77 (via https://cs.chromium.org/chromium/src/third_party/skia/include/core/SkTypes.h?type=cs&sq=package:chromium&g=0&l=53 / https://cs.chromium.org/chromium/src/third_party/skia/include/core/SkPostConfig.h?type=cs&q=SK_ABORT&sq=package:chromium&g=0&l=128) and the include is probably one of these two:

https://cs.chromium.org/search/?q=%5C.%5C./private/skatomics.h&type=cs

Comment 2 by thakis@chromium.org, Oct 27 

Oh, and GrRadialGradientLayout looks basically the same.

Comment 3 by thakis@chromium.org, Oct 27 

Build command 2:

..\..\third_party\llvm-build\Release+Asserts\bin\clang-cl.exe /nologo -imsvc..\..\third_party\depot_tools\win_toolchain\vs_files\3bc0ec615cf20ee342f3bc29bc991b5ad66d8d2c\win_sdk\Include\10.0.17134.0\um -imsvc..\..\third_party\depot_tools\win_toolchain\vs_files\3bc0ec615cf20ee342f3bc29bc991b5ad66d8d2c\win_sdk\Include\10.0.17134.0\shared -imsvc..\..\third_party\depot_tools\win_toolchain\vs_files\3bc0ec615cf20ee342f3bc29bc991b5ad66d8d2c\win_sdk\Include\10.0.17134.0\winrt -imsvc..\..\third_party\depot_tools\win_toolchain\vs_files\3bc0ec615cf20ee342f3bc29bc991b5ad66d8d2c\win_sdk\Include\10.0.17134.0\ucrt -imsvc..\..\third_party\depot_tools\win_toolchain\vs_files\3bc0ec615cf20ee342f3bc29bc991b5ad66d8d2c\VC\Tools\MSVC\14.14.26428\include -imsvc..\..\third_party\depot_tools\win_toolchain\vs_files\3bc0ec615cf20ee342f3bc29bc991b5ad66d8d2c\VC\Tools\MSVC\14.14.26428\atlmfc\include -DV8_DEPRECATION_WARNINGS -DUSE_AURA=1 -DNO_TCMALLOC -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DCHROMIUM_BUILD -DFIELDTRIAL_TESTING_ENABLED "-DCR_CLANG_REVISION=\"344066-1\"" -D_HAS_EXCEPTIONS=0 -DCOMPONENT_BUILD -D__STD_C -D_CRT_RAND_S -D_CRT_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_DEPRECATE -D_ATL_NO_OPENGL -D_WINDOWS -DCERT_CHAIN_PARA_HAS_EXTRA_FIELDS -DPSAPI_VERSION=1 -DWIN32 -D_SECURE_ATL -D_USING_V110_SDK71_ -DWINAPI_FAMILY=WINAPI_FAMILY_DESKTOP_APP -DWIN32_LEAN_AND_MEAN -DNOMINMAX -D_UNICODE -DUNICODE -DNTDDI_VERSION=0x0A000002 -D_WIN32_WINNT=0x0A00 -DWINVER=0x0A00 -D_DEBUG -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DWTF_USE_DYNAMIC_ANNOTATIONS=1 -DSK_IGNORE_LINEONLY_AA_CONVEX_PATH_OPTS -DSK_HAS_PNG_LIBRARY -DSK_HAS_WEBP_LIBRARY -DSK_HAS_JPEG_LIBRARY -DSKIA_DLL -DGR_GL_IGNORE_ES3_MSAA=0 -DSKCMS_API=__declspec(dllexport) -DSK_SUPPORT_GPU=1 "-DSK_GPU_WORKAROUNDS_HEADER=\"gpu/config/gpu_driver_bug_workaround_autogen.h\"" -DGR_GL_FUNCTION_TYPE=__stdcall "-DSK_FREETYPE_MINIMUM_RUNTIME_VERSION=(((FREETYPE_MAJOR) * 0x01000000) | ((FREETYPE_MINOR) * 0x00010000) | ((FREETYPE_PATCH) * 0x00000100))" -DSKIA_IMPLEMENTATION=1 -DSK_GAMMA_SRGB -DSK_GAMMA_CONTRAST=0.5 -DSK_DEFAULT_FONT_CACHE_LIMIT=20971520 -DSK_DEFAULT_FONT_CACHE_COUNT_LIMIT=256 -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE -DWEBP_EXTERN=extern -DPNG_USE_DLL -DPNG_NO_MODULEDEF "-DFT_CONFIG_CONFIG_H=\"freetype-custom-config/ftconfig.h\"" "-DFT_CONFIG_MODULES_H=\"freetype-custom-config/ftmodule.h\"" "-DFT_CONFIG_OPTIONS_H=\"freetype-custom-config/ftoption.h\"" -DPDFIUM_REQUIRED_MODULES -DUSE_LIBJPEG_TURBO=1 -I../.. -Igen -I../../skia/config -I../../skia/ext -I../../third_party/skia/include/c -I../../third_party/skia/include/config -I../../third_party/skia/include/core -I../../third_party/skia/include/docs -I../../third_party/skia/include/effects -I../../third_party/skia/include/encode -I../../third_party/skia/include/gpu -I../../third_party/skia/include/pathops -I../../third_party/skia/include/ports -I../../third_party/skia/include/utils -I../../third_party/skia/include/codec -I../../third_party/skia/src/gpu -I../../third_party/skia/src/sksl -I../../third_party/skia/modules/skottie/include -I../../third_party/skia/include/codec -I../../third_party/skia/include/private -I../../third_party/skia/include/client/android -I../../third_party/skia/src/codec -I../../third_party/skia/src/core -I../../third_party/skia/src/image -I../../third_party/skia/src/images -I../../third_party/skia/src/opts -I../../third_party/skia/src/pdf -I../../third_party/skia/src/ports -I../../third_party/skia/src/shaders -I../../third_party/skia/src/shaders/gradients -I../../third_party/skia/src/sfnt -I../../third_party/skia/src/utils -I../../third_party/skia/src/lazy -I../../third_party/skia/third_party/gif -I../../third_party/skia/src/effects/gradients -I../../third_party/skia/include/utils/win -I../../third_party/skia/src/utils/win -I../../third_party/skia/modules/sksg/include -I../../third_party/libwebp/src -I../../third_party/libpng -I../../third_party/zlib -I../../third_party/freetype/include -I../../third_party/freetype/src/include -I../../third_party/harfbuzz-ng/src/src -I../../third_party/libjpeg_turbo -I../../third_party/sfntly/src/cpp/src /utf-8 /X -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -fcolor-diagnostics -fmerge-all-constants -Xclang -mllvm -Xclang -instcombine-lower-dbg-declare=0 -Xclang -fdebug-compilation-dir -Xclang . -no-canonical-prefixes -fcomplete-member-pointers /Gy /FS /bigobj /d2FastFail /Zc:sizedDealloc- -fmsc-version=1911 -m64 /Brepro /Od /Ob0 /GF /MDd -Xclang -add-plugin -Xclang find-bad-constructs -Xclang -plugin-arg-find-bad-constructs -Xclang enforce-in-thirdparty-webkit -Xclang -plugin-arg-find-bad-constructs -Xclang check-enum-max-value /wd4244 /wd4267 /wd4341 /wd4345 /wd4390 /wd4554 /wd4748 /wd4800 /W3 /wd4800 /wd4267 /wd4996 -Wno-unused-variable -Wno-unused-lambda-capture /WX /wd4091 /wd4127 /wd4251 /wd4275 /wd4312 /wd4324 /wd4351 /wd4355 /wd4503 /wd4589 /wd4611 /wd4100 /wd4121 /wd4244 /wd4505 /wd4510 /wd4512 /wd4610 /wd4838 /wd4995 /wd4996 /wd4456 /wd4457 /wd4458 /wd4459 /wd4200 /wd4201 /wd4204 /wd4221 /wd4245 /wd4267 /wd4305 /wd4389 /wd4702 /wd4701 /wd4703 /wd4661 /wd4706 /wd4715 -Wno-missing-field-initializers -Wno-unused-parameter -Wno-c++11-narrowing -Wno-unneeded-internal-declaration -Wno-undefined-var-template -Wno-null-pointer-arithmetic -Wno-nonportable-include-path -Wno-ignored-pragma-optimize -Wno-defaulted-function-deleted -Wno-undefined-bool-conversion -Wno-tautological-undefined-compare /TP /GR- /wd4577 /c ../../third_party/skia/src/gpu/gl/builders/GrGLShaderStringBuilder.cpp /Foobj/skia/skia/GrGLShaderStringBuilder.obj /Fd"obj/skia/skia_cc.pdb"


(gomacc stripped manually)

Note actual `-I../../third_party/skia/include/c` in there!



Build command 1:

"..\..\third_party\llvm-build\Release+Asserts\bin\clang-cl.exe" -imsvc..\..\third_party\depot_tools\win_toolchain\vs_files\3bc0ec615cf20ee342f3bc29bc991b5ad66d8d2c\win_sdk\Include\10.0.17134.0\um -imsvc..\..\third_party\depot_tools\win_toolchain\vs_files\3bc0ec615cf20ee342f3bc29bc991b5ad66d8d2c\win_sdk\Include\10.0.17134.0\shared -imsvc..\..\third_party\depot_tools\win_toolchain\vs_files\3bc0ec615cf20ee342f3bc29bc991b5ad66d8d2c\win_sdk\Include\10.0.17134.0\winrt -imsvc..\..\third_party\depot_tools\win_toolchain\vs_files\3bc0ec615cf20ee342f3bc29bc991b5ad66d8d2c\win_sdk\Include\10.0.17134.0\ucrt -imsvc..\..\third_party\depot_tools\win_toolchain\vs_files\3bc0ec615cf20ee342f3bc29bc991b5ad66d8d2c\VC\Tools\MSVC\14.14.26428\include -imsvc..\..\third_party\depot_tools\win_toolchain\vs_files\3bc0ec615cf20ee342f3bc29bc991b5ad66d8d2c\VC\Tools\MSVC\14.14.26428\atlmfc\include -DV8_DEPRECATION_WARNINGS -DUSE_AURA=1 -DNO_TCMALLOC -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DCHROMIUM_BUILD -DFIELDTRIAL_TESTING_ENABLED "-DCR_CLANG_REVISION=\"344066-1\"" -D_HAS_EXCEPTIONS=0 -DCOMPONENT_BUILD -D__STD_C -D_CRT_RAND_S -D_CRT_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_DEPRECATE -D_ATL_NO_OPENGL -D_WINDOWS -DCERT_CHAIN_PARA_HAS_EXTRA_FIELDS -DPSAPI_VERSION=1 -DWIN32 -D_SECURE_ATL -D_USING_V110_SDK71_ -DWINAPI_FAMILY=WINAPI_FAMILY_DESKTOP_APP -DWIN32_LEAN_AND_MEAN -DNOMINMAX -D_UNICODE -DUNICODE -DNTDDI_VERSION=0x0A000002 -D_WIN32_WINNT=0x0A00 -DWINVER=0x0A00 -D_DEBUG -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DWTF_USE_DYNAMIC_ANNOTATIONS=1 -DSK_IGNORE_LINEONLY_AA_CONVEX_PATH_OPTS -DSK_HAS_PNG_LIBRARY -DSK_HAS_WEBP_LIBRARY -DSK_HAS_JPEG_LIBRARY -DSKIA_DLL -DGR_GL_IGNORE_ES3_MSAA=0 -DSKCMS_API=__declspec(dllexport) -DSK_SUPPORT_GPU=1 "-DSK_GPU_WORKAROUNDS_HEADER=\"gpu/config/gpu_driver_bug_workaround_autogen.h\"" -DGR_GL_FUNCTION_TYPE=__stdcall "-DSK_FREETYPE_MINIMUM_RUNTIME_VERSION=(((FREETYPE_MAJOR) * 0x01000000) | ((FREETYPE_MINOR) * 0x00010000) | ((FREETYPE_PATCH) * 0x00000100))" -DSKIA_IMPLEMENTATION=1 -DSK_GAMMA_SRGB -DSK_GAMMA_CONTRAST=0.5 -DSK_DEFAULT_FONT_CACHE_LIMIT=20971520 -DSK_DEFAULT_FONT_CACHE_COUNT_LIMIT=256 -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE -DWEBP_EXTERN=extern -DPNG_USE_DLL -DPNG_NO_MODULEDEF "-DFT_CONFIG_CONFIG_H=\"freetype-custom-config/ftconfig.h\"" "-DFT_CONFIG_MODULES_H=\"freetype-custom-config/ftmodule.h\"" "-DFT_CONFIG_OPTIONS_H=\"freetype-custom-config/ftoption.h\"" -DPDFIUM_REQUIRED_MODULES -DUSE_LIBJPEG_TURBO=1 -I../.. -Igen -I../../skia/config -I../../skia/ext -I../../third_party/skia/include/c -I../../third_party/skia/include/config -I../../third_party/skia/include/core -I../../third_party/skia/include/docs -I../../third_party/skia/include/effects -I../../third_party/skia/include/encode -I../../third_party/skia/include/gpu -I../../third_party/skia/include/pathops -I../../third_party/skia/include/ports -I../../third_party/skia/include/utils -I../../third_party/skia/include/codec -I../../third_party/skia/src/gpu -I../../third_party/skia/src/sksl -I../../third_party/skia/modules/skottie/include -I../../third_party/skia/include/codec -I../../third_party/skia/include/private -I../../third_party/skia/include/client/android -I../../third_party/skia/src/codec -I../../third_party/skia/src/core -I../../third_party/skia/src/image -I../../third_party/skia/src/images -I../../third_party/skia/src/opts -I../../third_party/skia/src/pdf -I../../third_party/skia/src/ports -I../../third_party/skia/src/shaders -I../../third_party/skia/src/shaders/gradients -I../../third_party/skia/src/sfnt -I../../third_party/skia/src/utils -I../../third_party/skia/src/lazy -I../../third_party/skia/third_party/gif -I../../third_party/skia/src/effects/gradients -I../../third_party/skia/include/utils/win -I../../third_party/skia/src/utils/win -I../../third_party/skia/modules/sksg/include -I../../third_party/libwebp/src -I../../third_party/libpng -I../../third_party/zlib -I../../third_party/freetype/include -I../../third_party/freetype/src/include -I../../third_party/harfbuzz-ng/src/src -I../../third_party/libjpeg_turbo -I../../third_party/sfntly/src/cpp/src /utf-8 /X -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -fcolor-diagnostics -fmerge-all-constants -Xclang -mllvm -Xclang -instcombine-lower-dbg-declare=0 -Xclang -fdebug-compilation-dir -Xclang . -no-canonical-prefixes -fcomplete-member-pointers /Gy /FS /bigobj /d2FastFail /Zc:sizedDealloc- -fmsc-version=1911 -m64 /Brepro /Od /Ob0 /GF /MDd -Xclang -add-plugin -Xclang find-bad-constructs -Xclang -plugin-arg-find-bad-constructs -Xclang enforce-in-thirdparty-webkit -Xclang -plugin-arg-find-bad-constructs -Xclang check-enum-max-value /wd4244 /wd4267 /wd4341 /wd4345 /wd4390 /wd4554 /wd4748 /wd4800 /W3 /wd4800 /wd4267 /wd4996 -Wno-unused-variable -Wno-unused-lambda-capture /WX /wd4091 /wd4127 /wd4251 /wd4275 /wd4312 /wd4324 /wd4351 /wd4355 /wd4503 /wd4589 /wd4611 /wd4100 /wd4121 /wd4244 /wd4505 /wd4510 /wd4512 /wd4610 /wd4838 /wd4995 /wd4996 /wd4456 /wd4457 /wd4458 /wd4459 /wd4200 /wd4201 /wd4204 /wd4221 /wd4245 /wd4267 /wd4305 /wd4389 /wd4702 /wd4701 /wd4703 /wd4661 /wd4706 /wd4715 -Wno-missing-field-initializers -Wno-unused-parameter -Wno-c++11-narrowing -Wno-unneeded-internal-declaration -Wno-undefined-var-template -Wno-null-pointer-arithmetic -Wno-nonportable-include-path -Wno-ignored-pragma-optimize -Wno-defaulted-function-deleted -Wno-undefined-bool-conversion -Wno-tautological-undefined-compare /TP /GR- /wd4577 /c ../../third_party/skia/src/gpu/gl/builders/GrGLShaderStringBuilder.cpp /Foobj/skia/skia/GrGLShaderStringBuilder.obj /Fd"obj/skia/skia_cc.pdb"


Also has the -I../../third_party/skia/include/c


Huh, if I rebuild the first build dir, the output is suddenly the same as in the other flie. Maybe it's a goma bug then, since I built without goma now?

Comment 4 by thakis@chromium.org, Oct 27

Cc: tikuta@chromium.org shinyak@chromium.org
Summary: Two files in skia aren't deterministic due to goma preprocessor bug (was: Two files in skia aren't deterministic)
Confirmed: If I do

C:\src\chrome\src\out\gn>set GOMA_USE_LOCAL=false

C:\src\chrome\src\out\gn>C:\src\goma\goma-win64/gomacc.exe "..\..\third_party\llvm-build\Release+Asserts\bin\clang-cl.exe" -imsvc..\..\third_party\depot_tools\win_toolchain\vs_files\3bc0ec615cf20ee342f3bc29bc991b5ad66d8d2c\win_sdk\Include\10.0.17134.0\um -imsvc..\..\third_party\depot_tools\win_toolchain\vs_files\3bc0ec615cf20ee342f3bc29bc991b5ad66d8d2c\win_sdk\Include\10.0.17134.0\shared -imsvc..\..\third_party\depot_tools\win_toolchain\vs_files\3bc0ec615cf20ee342f3bc29bc991b5ad66d8d2c\win_sdk\Include\10.0.17134.0\winrt -imsvc..\..\third_party\depot_tools\win_toolchain\vs_files\3bc0ec615cf20ee342f3bc29bc991b5ad66d8d2c\win_sdk\Include\10.0.17134.0\ucrt -imsvc..\..\third_party\depot_tools\win_toolchain\vs_files\3bc0ec615cf20ee342f3bc29bc991b5ad66d8d2c\VC\Tools\MSVC\14.14.26428\include -imsvc..\..\third_party\depot_tools\win_toolchain\vs_files\3bc0ec615cf20ee342f3bc29bc991b5ad66d8d2c\VC\Tools\MSVC\14.14.26428\atlmfc\include -DV8_DEPRECATION_WARNINGS -DUSE_AURA=1 -DNO_TCMALLOC -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DCHROMIUM_BUILD -DFIELDTRIAL_TESTING_ENABLED "-DCR_CLANG_REVISION=\"344066-1\"" -D_HAS_EXCEPTIONS=0 -DCOMPONENT_BUILD -D__STD_C -D_CRT_RAND_S -D_CRT_SECURE_NO_DEPRECATE -D_SCL_SECURE_NO_DEPRECATE -D_ATL_NO_OPENGL -D_WINDOWS -DCERT_CHAIN_PARA_HAS_EXTRA_FIELDS -DPSAPI_VERSION=1 -DWIN32 -D_SECURE_ATL -D_USING_V110_SDK71_ -DWINAPI_FAMILY=WINAPI_FAMILY_DESKTOP_APP -DWIN32_LEAN_AND_MEAN -DNOMINMAX -D_UNICODE -DUNICODE -DNTDDI_VERSION=0x0A000002 -D_WIN32_WINNT=0x0A00 -DWINVER=0x0A00 -D_DEBUG -DDYNAMIC_ANNOTATIONS_ENABLED=1 -DWTF_USE_DYNAMIC_ANNOTATIONS=1 -DSK_IGNORE_LINEONLY_AA_CONVEX_PATH_OPTS -DSK_HAS_PNG_LIBRARY -DSK_HAS_WEBP_LIBRARY -DSK_HAS_JPEG_LIBRARY -DSKIA_DLL -DGR_GL_IGNORE_ES3_MSAA=0 -DSKCMS_API=__declspec(dllexport) -DSK_SUPPORT_GPU=1 "-DSK_GPU_WORKAROUNDS_HEADER=\"gpu/config/gpu_driver_bug_workaround_autogen.h\"" -DGR_GL_FUNCTION_TYPE=__stdcall "-DSK_FREETYPE_MINIMUM_RUNTIME_VERSION=(((FREETYPE_MAJOR) * 0x01000000) | ((FREETYPE_MINOR) * 0x00010000) | ((FREETYPE_PATCH) * 0x00000100))" -DSKIA_IMPLEMENTATION=1 -DSK_GAMMA_SRGB -DSK_GAMMA_CONTRAST=0.5 -DSK_DEFAULT_FONT_CACHE_LIMIT=20971520 -DSK_DEFAULT_FONT_CACHE_COUNT_LIMIT=256 -D_CRT_NONSTDC_NO_WARNINGS -D_CRT_NONSTDC_NO_DEPRECATE -DWEBP_EXTERN=extern -DPNG_USE_DLL -DPNG_NO_MODULEDEF "-DFT_CONFIG_CONFIG_H=\"freetype-custom-config/ftconfig.h\"" "-DFT_CONFIG_MODULES_H=\"freetype-custom-config/ftmodule.h\"" "-DFT_CONFIG_OPTIONS_H=\"freetype-custom-config/ftoption.h\"" -DPDFIUM_REQUIRED_MODULES -DUSE_LIBJPEG_TURBO=1 -I../.. -Igen -I../../skia/config -I../../skia/ext -I../../third_party/skia/include/c -I../../third_party/skia/include/config -I../../third_party/skia/include/core -I../../third_party/skia/include/docs -I../../third_party/skia/include/effects -I../../third_party/skia/include/encode -I../../third_party/skia/include/gpu -I../../third_party/skia/include/pathops -I../../third_party/skia/include/ports -I../../third_party/skia/include/utils -I../../third_party/skia/include/codec -I../../third_party/skia/src/gpu -I../../third_party/skia/src/sksl -I../../third_party/skia/modules/skottie/include -I../../third_party/skia/include/codec -I../../third_party/skia/include/private -I../../third_party/skia/include/client/android -I../../third_party/skia/src/codec -I../../third_party/skia/src/core -I../../third_party/skia/src/image -I../../third_party/skia/src/images -I../../third_party/skia/src/opts -I../../third_party/skia/src/pdf -I../../third_party/skia/src/ports -I../../third_party/skia/src/shaders -I../../third_party/skia/src/shaders/gradients -I../../third_party/skia/src/sfnt -I../../third_party/skia/src/utils -I../../third_party/skia/src/lazy -I../../third_party/skia/third_party/gif -I../../third_party/skia/src/effects/gradients -I../../third_party/skia/include/utils/win -I../../third_party/skia/src/utils/win -I../../third_party/skia/modules/sksg/include -I../../third_party/libwebp/src -I../../third_party/libpng -I../../third_party/zlib -I../../third_party/freetype/include -I../../third_party/freetype/src/include -I../../third_party/harfbuzz-ng/src/src -I../../third_party/libjpeg_turbo -I../../third_party/sfntly/src/cpp/src /utf-8 /X -Wno-builtin-macro-redefined -D__DATE__= -D__TIME__= -D__TIMESTAMP__= -fcolor-diagnostics -fmerge-all-constants -Xclang -mllvm -Xclang -instcombine-lower-dbg-declare=0 -Xclang -fdebug-compilation-dir -Xclang . -no-canonical-prefixes -fcomplete-member-pointers /Gy /FS /bigobj /d2FastFail /Zc:sizedDealloc- -fmsc-version=1911 -m64 /Brepro /Od /Ob0 /GF /MDd -Xclang -add-plugin -Xclang find-bad-constructs -Xclang -plugin-arg-find-bad-constructs -Xclang enforce-in-thirdparty-webkit -Xclang -plugin-arg-find-bad-constructs -Xclang check-enum-max-value /wd4244 /wd4267 /wd4341 /wd4345 /wd4390 /wd4554 /wd4748 /wd4800 /W3 /wd4800 /wd4267 /wd4996 -Wno-unused-variable -Wno-unused-lambda-capture /WX /wd4091 /wd4127 /wd4251 /wd4275 /wd4312 /wd4324 /wd4351 /wd4355 /wd4503 /wd4589 /wd4611 /wd4100 /wd4121 /wd4244 /wd4505 /wd4510 /wd4512 /wd4610 /wd4838 /wd4995 /wd4996 /wd4456 /wd4457 /wd4458 /wd4459 /wd4200 /wd4201 /wd4204 /wd4221 /wd4245 /wd4267 /wd4305 /wd4389 /wd4702 /wd4701 /wd4703 /wd4661 /wd4706 /wd4715 -Wno-missing-field-initializers -Wno-unused-parameter -Wno-c++11-narrowing -Wno-unneeded-internal-declaration -Wno-undefined-var-template -Wno-null-pointer-arithmetic -Wno-nonportable-include-path -Wno-ignored-pragma-optimize -Wno-defaulted-function-deleted -Wno-undefined-bool-conversion -Wno-tautological-undefined-compare /TP /GR- /wd4577 /c ../../third_party/skia/src/gpu/gl/builders/GrGLShaderStringBuilder.cpp /Foobj/skia/skia/GrGLShaderStringBuilder.obj /Fd"obj/skia/skia_cc.pdb"


then I get the include/core\../private/SkAtomics.h output, if I do the same with  `set GOMA_USE_LOCAL=true` then I get the include/c\../private/SkAtomics.h output (I checked on localhost:8088 that the "wrong" output came in via goma while the "right" one, at least the one matching clang locally, was with a local run).


-I../../third_party/skia/include/c is in front of  -I../../third_party/skia/include/core in the command, so the `/c\` version does seem more correct.

shinyak, could this be an issue in goma's preprocessor? Or does the order if -I flags change on the server side or something?

Comment 5 by thakis@chromium.org, Oct 27 

*the order _of_ -I flags...

Comment 6 by thakis@chromium.org, Oct 27 

Let me summarize the comments above:

- third_party/skia/include/private/SkAtomics.h [1] uses SkASSERT, which uses __LINE__

- SkAtomics is includes as `#include "../private/SkAtomics.h"`

- The cpp file (transitively) pulling in SkAtomics passes both `-I../../third_party/skia/include/c  -I../../third_party/skia/include/core` (in that order)

- __FILE__ resolves to the relative include concatenated to the used include path. Both ../../third_party/skia/include/c/../private/SkAtomics.h and ../../third_party/skia/include/core/../private/SkAtomics.h "work" and refer to the same file, but the first one is correct since that uses the earlier -I flag

- Running clang locally produces the right output, but running it on goma produces the wrong output

- Sometimes the local fallback wins, leading to nondeterministic .obj output (which in turn leads to nondeterministic binaries, which hurts swarming's dedup cache)


1: https://cs.chromium.org/chromium/src/third_party/skia/include/private/SkAtomics.h?type=cs&sq=package:chromium&g=0&l=77

Comment 7 by thakis@chromium.org, Oct 28

Blocking: 899438

Comment 8 by shinyak@chromium.org, Oct 29

Components: Infra>Goma
I take a look.

Comment 10 by shinyak@google.com, Oct 29 

Basic: goma's include processor just lists all source/header files which will be used in a compile. So, if this is goma's include processor's bug, the files that goma lists and the files clang lists differ.
Goma's include processor won't do the exact same thing as clang (cpp) does, but as long as the file list is the same, remote compile should work fine. Even if __LINE__ or __FILE__ implementation is wrong in goma, as long as the file list is the same, the compile result must be the same.

Usually, __LINE__ and __FILE__ are used just to print. They won't change the result of include processor. So, I'm a bit skeptical they affect the result of include processor.

Anyway, the candidate is:
(1) goma's file list is wrong
(2) remote compile result is wrong

I'll check what is happening...

Comment 11 by shinyak@google.com, Oct 29 

According to goma's log, the following three SkAtomics.h are listed from the result of include processor.

..\..\third_party\skia\include\c\..\private\SkAtomics.h
..\..\third_party\skia\include\core\..\private\SkAtomics.h
..\..\third_party\skia\include\private\SkAtomics.h

The problem was:
> then I get the include/core\../private/SkAtomics.h output, if I do the same with  `set GOMA_USE_LOCAL=true` then I get the include/c\../private/SkAtomics.h output (I checked on localhost:8088 that the "wrong" output came in via goma while the "right" one, at least the one matching clang locally, was with a local run).

So, include processor correctly lists ..\..\third_party\skia\include\c\..\private\SkAtomics.h.
Then the problem won't be include processor.

Next we have to check
(1) whether include order is not changed in goma somewhere,
(2) and how they are included in the goma server.

Comment 12 by shinyak@google.com, Oct 29 

ExecReq compiler_proxy generated seems correct, then this might be server side bug? investigating.

Comment 13 by thakis@chromium.org, Oct 30 

For reference, there are now 3 obj files (out of tens of thousands) in a chrome build that depend on the name of the build directory: The two here, and one that probably has a simple fix ( issue 899437 ). It'd be good to have this fixed soon, so that we can make the Windows deterministic bot use two different build directories and make sure we don't regress.

Comment 14 by thakis@chromium.org, Oct 30

Owner: shinyak@chromium.org
shinyak, it looks like you're investigating (thanks!), so giving this to you.

Comment 15 by shinyak@google.com, Oct 31 

I could repro this easily, but I couldn't yet understand why this is happening. Let me give time...

Comment 16 by most...@vewd.com, Nov 8

Cc: most...@vewd.com
I also see this bug with our own goma backend, also in skia (discovered when trying GOMA_VERIFY_PREPROCESS_CODE=true).

Comment 17 by thakis@chromium.org, Nov 8 

Update from the internal bug: Fixed, fix will be in next server release.

Comment 18 by most...@vewd.com, Nov 8 

Any hints to what the server-side bug was?

Comment 19 by tikuta@chromium.org, Nov 8 

Do you make directory in your backend for
..\..\third_party\skia\include\c and
..\..\third_party\skia\include\core
in addition to ..\..\third_party\skia\include\private\SkAtomics.h ?

Actual fix for our backend is bit more complex, but I wonder whether your backend makes (seems) unnecessary intermediate directory.

Comment 20 by most...@vewd.com, Nov 8 

I fixed a similar sounding issue last week actually (I found a case where not having unused intermediate directories would fail to compile, fixed by doing the equivalent of mkdir -p before any other processing on the directory path), but haven't retested this since. Thanks for the tip!

Comment 21 by shinyak@google.com, Nov 15 

I believe the root cause of this issue has been fixed in goma side.
I've just rolled goma servers, but there might still be a wrong cache result.
It should be eventually fixed.

Comment 22 by shinyak@google.com, Nov 15 

As far as I checked, GrGLShaderStringBuilder.obj now contains the correct string (include/c\../private/SkAtomic.h)

thakis, could you also check?

Comment 23 by thakis@chromium.org, Nov 26

Status: Verified (was: Untriaged)
Looks good over here too. Thanks!

Comment 24 by thakis@chromium.org, Nov 26

Status: Started (was: Verified)
Hmm, I'm trying the same on linux now. `goma_ctl.py update` says I'm on the latest version.

This looks like a pretty similar thing. My args.gn are:

symbol_level = 1
is_component_build = true
is_debug = true
use_goma = true


Let me confirm that GOMA_USE_LOCAL triggers this...
skiagoma.png
478 KB View Download

Comment 25 by thakis@chromium.org, Nov 26 

Yes, looks like this is still an issue:

thakis@thakis:~/src/chrome/src$ export GOMA_USE_LOCAL=false
thakis@thakis:~/src/chrome/src$ ninja -C out/gn obj/skia/skia_core_and_effects/SkBlurImageFilter.o
ninja: Entering directory `out/gn'
[1/1] CXX obj/skia/skia_core_and_effects/SkBlurImageFilter.o
thakis@thakis:~/src/chrome/src$ cp out/gn/obj/skia/skia_core_and_effects/SkBlurImageFilter.o .
thakis@thakis:~/src/chrome/src$ mv SkBlurImageFilter.o SkBlurImageFilter_nolocal.o
thakis@thakis:~/src/chrome/src$ rm out/gn/obj/skia/skia_core_and_effects/SkBlurImageFilter.o
thakis@thakis:~/src/chrome/src$ export GOMA_USE_LOCAL=true
thakis@thakis:~/src/chrome/src$ ninja -C out/gn obj/skia/skia_core_and_effects/SkBlurImageFilter.o
ninja: Entering directory `out/gn'
[1/1] CXX obj/skia/skia_core_and_effects/SkBlurImageFilter.o
thakis@thakis:~/src/chrome/src$ cp out/gn/obj/skia/skia_core_and_effects/SkBlurImageFilter.o .
thakis@thakis:~/src/chrome/src$ mv SkBlurImageFilter.o SkBlurImageFilter_local.o
thakis@thakis:~/src/chrome/src$ cmp SkBlurImageFilter_*
SkBlurImageFilter_local.o SkBlurImageFilter_nolocal.o differ: byte 41, line 1


This time on linux.

Comment 26 by shinyak@google.com, Nov 27 

I take a look again...

Since the solution I used for Windows was already applied for Linux,
there might be another issue that affects only for Linux (and probably Mac, too)

X(

Comment 27 by shinyak@google.com, Nov 27 

I tried repro, but I couldn't.
In my environment with args.gn written in Comment 24, remote and local objects are the same.

If you have time, could you try with GOMA_STORE_ONLY=true?

Then we will have 3 objects:
  (1) local build (build without goma)
  (2) remote build with cache (GOMA_USE_LOCAL=false)
  (3) remote build without cache (GOMA_USE_LOCAL=false GOMA_STORE_ONLY=true)

If (2) and (3) are different, the reason is the cache. We need to purge cache.
If they are the same, goma's executor is culprit.

shinyak@shinyak [~/work/chrome/src/out/DebugSymbolLevel1]
$ cat args.gn 
symbol_level = 1
is_component_build = true
is_debug = true
use_goma = true
shinyak@shinyak [~/work/chrome/src/out/DebugSymbolLevel1]
$ ninja -t clean                                                                         
Cleaning... 3 files.
shinyak@shinyak [~/work/chrome/src/out/DebugSymbolLevel1]
$ GOMA_USE_LOCAL=true ninja obj/skia/skia_core_and_effects/SkBlurImageFilter.o
[3/3] CXX obj/skia/skia_core_and_effects/SkBlurImageFilter.o
shinyak@shinyak [~/work/chrome/src/out/DebugSymbolLevel1]
$ mv ./obj/skia/skia_core_and_effects/SkBlurImageFilter.o ~/tmp/SkBlurImageFilter_local.o
shinyak@shinyak [~/work/chrome/src/out/DebugSymbolLevel1]
$ GOMA_USE_LOCAL=false ninja obj/skia/skia_core_and_effects/SkBlurImageFilter.o      
[1/1] CXX obj/skia/skia_core_and_effects/SkBlurImageFilter.o
shinyak@shinyak [~/work/chrome/src/out/DebugSymbolLevel1]
$ mv ./obj/skia/skia_core_and_effects/SkBlurImageFilter.o ~/tmp/SkBlurImageFilter_remote.o
shinyak@shinyak [~/work/chrome/src/out/DebugSymbolLevel1]
$ cmp ~/tmp/SkBlurImageFilter_local.o ~/tmp/SkBlurImageFilter_remote.o

Comment 28 by thakis@chromium.org, Nov 27

Status: Fixed (was: Started)
Huh, now I can't repro either. As far as I know nothing has changed over here. Well, let's close this again, and I'll shout if I see it again.

Comment 29 by shinyak@google.com, Nov 28 

OK, maybe the cache is gone.
►

Sign in to add a comment