New issue
Advanced search Search tips
Note: Color blocks (like or ) mean that a user may not be available. Tooltip shows the reason.

Issue 675648 link

Starred by 2 users

Issue metadata

Status: Verified
Owner:
Closed: Dec 2016
Cc:
EstimatedDays: ----
NextAction: ----
OS: Chrome
Pri: 1
Type: Bug



Sign in to add a comment

Internal Compiler Error: Segmentation Fault - veyron_fievel SimpleChrome

Project Member Reported by afakhry@chromium.org, Dec 19 2016

Issue description

Steps to reproduce:
(1) cros chrome-sdk --board=veyron_fievel

(2) gn gen --root=./src src/out_veyron_fievel/release --args='use_ozone = true custom_toolchain = "//build/toolchain/cros:target" arm_float_abi = "hard" ozone_platform_gbm = true cros_v8_snapshot_cc = "/Work/chromiumos/src/third_party/llvm-build/Release+Asserts/bin/clang" cros_target_cxx = "armv7a-cros-linux-gnueabi-g++ -B/Work/chromiumos/.cros_cache/chrome-sdk/tarballs/veyron_fievel+9087.0.0+target_toolchain/usr/x86_64-pc-linux-gnu/armv7a-cros-linux-gnueabi/binutils-bin/2.25.51-gold" cros_host_ar = "/Work/chromiumos/src/third_party/binutils/Linux_x64/Release/bin/ar" use_v4lplugin = true cros_host_cxx = "/Work/chromiumos/src/third_party/llvm-build/Release+Asserts/bin/clang++" cros_v8_snapshot_ar = "/Work/chromiumos/src/third_party/binutils/Linux_x64/Release/bin/ar" use_cups = true use_system_harfbuzz = true arm_use_neon = true cros_target_extra_cppflags = "" use_evdev_gestures = true use_goma = true cros_host_extra_ldflags = "" cros_target_extra_cxxflags = "-pipe -march=armv7-a -mtune=cortex-a15 -mfpu=neon -mfloat-abi=hard -D__google_stl_debug_vector=1" target_sysroot = "/Work/chromiumos/.cros_cache/chrome-sdk/tarballs/veyron_fievel+9087.0.0+sysroot_chromeos-base_chromeos-chrome.tar.xz" clang_use_chrome_plugins = false symbol_level = 2 cros_host_extra_cppflags = "" cros_host_is_clang = true cros_target_extra_cflags = "-pipe -march=armv7-a -mtune=cortex-a15 -mfpu=neon -mfloat-abi=hard" cros_v8_snapshot_extra_cflags = "" system_libdir = "lib" use_cras = true cros_host_cc = "/Work/chromiumos/src/third_party/llvm-build/Release+Asserts/bin/clang" v8_snapshot_toolchain = "//build/toolchain/cros:v8_snapshot" cros_v8_snapshot_cxx = "/Work/chromiumos/src/third_party/llvm-build/Release+Asserts/bin/clang++" is_clang = false cros_v8_snapshot_is_clang = true use_v4l2_codec = true cros_host_extra_cflags = "" host_toolchain = "//build/toolchain/cros:host" use_xkbcommon = true icu_use_data_file = true cros_v8_snapshot_extra_ldflags = "" enable_nacl = true linux_use_bundled_binutils = true cros_v8_snapshot_ld = "/Work/chromiumos/src/third_party/llvm-build/Release+Asserts/bin/clang++" enable_remoting = true ozone_auto_platforms = false cros_target_ld = "armv7a-cros-linux-gnueabi-g++ -B/Work/chromiumos/.cros_cache/chrome-sdk/tarballs/veyron_fievel+9087.0.0+target_toolchain/usr/x86_64-pc-linux-gnu/armv7a-cros-linux-gnueabi/binutils-bin/2.25.51-gold" target_os = "chromeos" use_system_minigbm = true cros_host_extra_cxxflags = "" cros_target_ar = "armv7a-cros-linux-gnueabi-ar" use_debug_fission = true cros_v8_snapshot_extra_cxxflags = "" target_cpu = "arm" cros_target_extra_ldflags = "-Wl,-O1 -Wl,-O2 -Wl,--as-needed" is_asan = false is_debug = false cros_host_ld = "/Work/chromiumos/src/third_party/llvm-build/Release+Asserts/bin/clang++" cros_v8_snapshot_extra_cppflags = "" ozone_platform = "gbm" goma_dir = "/Work/chromiumos/.cros_cache/common/goma+2" cros_target_cc = "armv7a-cros-linux-gnueabi-gcc -B/Work/chromiumos/.cros_cache/chrome-sdk/tarballs/veyron_fievel+9087.0.0+target_toolchain/usr/x86_64-pc-linux-gnu/armv7a-cros-linux-gnueabi/binutils-bin/2.25.51-gold" rtc_use_h264=true proprietary_codecs=true ffmpeg_branding="ChromeOS" symbol_level=2 enable_profiling=true gdb_index=true v8_enable_backtrace=true treat_warnings_as_errors=false is_debug=false is_official_build=true is_chrome_branded=true use_goma=true'

(3) ninja -C src/out_veyron_fievel/release -j 1000 chrome chrome_sandbox nacl_helper

===============================================================================================

--->>> If using GOMA, the proxy server will crash midway, saying:
FAILED: v8_snapshot/obj/v8/v8_nosnapshot/extras-libraries.o 
/usr/local/google/home/afakhry/goma/gomacc /Work/chromiumos/src/third_party/llvm-build/Release+Asserts/bin/clang++ -MMD -MF v8_snapshot/obj/v8/v8_nosnapshot/extras-libraries.o.d -DV8_DEPRECATION_WARNINGS -DUSE_UDEV -DUI_COMPOSITOR_IMAGE_TRANSPORT -DUSE_AURA=1 -DUSE_NSS_CERTS=1 -DUSE_OZONE=1 -DUSE_PROPRIETARY_CODECS -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DOFFICIAL_BUILD -DGOOGLE_CHROME_BUILD -DENABLE_MEDIA_ROUTER=1 -DCR_CLANG_REVISION=289575-1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -DNO_UNWIND_TABLES -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DV8_I18N_SUPPORT -DENABLE_HANDLE_ZAPPING -DV8_USE_EXTERNAL_STARTUP_DATA -DV8_TARGET_ARCH_ARM -DCAN_USE_ARMV7_INSTRUCTIONS -DCAN_USE_VFP3_INSTRUCTIONS -DCAN_USE_VFP32DREGS -DCAN_USE_NEON -DUSE_EABI_HARDFLOAT=1 -I../.. -Iv8_snapshot/gen -I../../v8 -fno-strict-aliasing --param=ssp-buffer-size=4 -fstack-protector -g -fno-unwind-tables -fno-asynchronous-unwind-tables -fPIC -pipe -B../../third_party/binutils/Linux_x64/Release/bin -fcolor-diagnostics -fdebug-prefix-map=/Work/chromiumos/src=. -m32 -msse2 -mfpmath=sse -mmmx -momit-leaf-frame-pointer -pthread -mstack-alignment=16 -mstackrealign -fno-omit-frame-pointer -g2 -gsplit-dwarf -Wheader-hygiene -Wstring-conversion -Wall -Wno-unused-variable -Wno-missing-field-initializers -Wno-unused-parameter -Wno-c++11-narrowing -Wno-covered-switch-default -Wno-deprecated-register -Wno-unneeded-internal-declaration -Wno-inconsistent-missing-override -Wno-shift-negative-value -Wno-undefined-var-template -Wno-nonportable-include-path -Wno-address-of-packed-member -Wsign-compare -Winconsistent-missing-override -O3 -fno-ident -fdata-sections -ffunction-sections -fvisibility=default -fno-threadsafe-statics -fvisibility-inlines-hidden -std=gnu++11 -fno-rtti -fno-exceptions -Wno-deprecated -c v8_snapshot/gen/v8/extras-libraries.cc -o v8_snapshot/obj/v8/v8_nosnapshot/extras-libraries.o
E1216 21:07:15.899263 12517 goma_ipc.cc:225] GOMA: Unexpected end-of-file
E1216 21:07:15.899770 12517 gomacc.cc:330] GOMA: compiler proxy not working?


--->>> If not using GOMA, you'll see the actual compiler error:
../../v8/src/compiler/js-native-context-specialization.cc: In member function 'v8::internal::compiler::JSNativeContextSpecialization::ValueEffectControl v8::internal::compiler::JSNativeContextSpecialization::InlineApiCall(v8::internal::compiler::Node*, v8::internal::compiler::Node*, v8::internal::compiler::Node*, v8::internal::compiler::Node*, v8::internal::compiler::Node*, v8::internal::compiler::Node*, v8::internal::compiler::Node*, v8::internal::Handle<v8::internal::SharedFunctionInfo>, v8::internal::Handle<v8::internal::FunctionTemplateInfo>)':
../../v8/src/compiler/js-native-context-specialization.cc:1609:1: internal compiler error: in decompose_normal_address, at rtlanal.c:5799
 }
 ^

../../v8/src/compiler/js-native-context-specialization.cc:1609:1: internal compiler error: Segmentation fault
{standard input}: Assembler messages:
{standard input}:1619: Warning: end of file not at end of a line; newline inserted
{standard input}: Error: open CFI at the end of file; missing .cfi_endproc directive
armv7a-cros-linux-gnueabi-g++.real.elf: internal compiler error: Segmentation fault (program cc1plus)



--->>> Restarting GOMA after the crash sends a crash report with ID = dcc8184080000000.


Reverting this CL https://codereview.chromium.org/2578563002 locally fixes the issue.

 

Comment 1 by laszio@chromium.org, Dec 19 2016

Owner: laszio@chromium.org
Owner: cmt...@chromium.org
cmtice is going to triage.

Comment 3 by cmt...@chromium.org, Dec 19 2016

I downloaded and set a chrome source tree.  Then I did step 1 & step 2.

It died for me on step 2 with the following error:

ERROR at //printing/BUILD.gn:147:22: Script returned non-zero exit code.
      cups_version = exec_script("cups_config_helper.py",
                     ^----------
Current dir: /usr/local/google2/cmtice/chrome-src-top/src/out_veyron_fievel/release/
Command: python -- /usr/local/google2/cmtice/chrome-src-top/src/printing/cups_config_helper.py --api-version /Work/chromiumos/.cros_cache/chrome-sdk/tarballs/veyron_fievel+9087.0.0+sysroot_chromeos-base_chromeos-chrome.tar.xz
Returned 1 and printed out:

cups-config not found: /Work/chromiumos/.cros_cache/chrome-sdk/tarballs/veyron_fievel+9087.0.0+sysroot_chromeos-base_chromeos-chrome.tar.xz/usr/bin/cups-config

See //BUILD.gn:268:7: which caused the file to be included.
      "//printing:printing_unittests",
      ^------------------------------

Is there some other configure or set up step that I'm missing?  (I don't normally build Chrome outside a ChromeOS chroot, so this process is a bit new to me...)

Comment 4 by cmt...@chromium.org, Dec 19 2016

 $ find . -name cups-config

./src/build/linux/debian_wheezy_i386-sysroot/usr/bin/cups-config
./.cros_cache/chrome-sdk/tarballs/veyron_fievel+9095.0.0+sysroot_chromeos-base_chromeos-chrome.tar.xz/usr/bin/cups-config


For some reason the script that is giving the error seems to be looking in the wrong chromeos-chrome tarball.  It is looking in version "+9087.0.0", whereas the file actually appears to be in "+9095.0.0".

Two questions:

1).  Any idea why it is being looked for in the wrong place?
2).  Any idea how to fix this, so I can progress with triaging this issue?

Comment 5 by cmt...@chromium.org, Dec 19 2016

Never mind; I've figured it out (I need to fix the version number in the step 2 stuff rather than straight cur & paste).
Yes, sorry. I should have written step 2 as:

gn gen --root=./src src/out_$BOARD/release --args='$GN_ARGS rtc_use_h264=true proprietary_codecs=true ffmpeg_branding="ChromeOS" symbol_level=2 enable_profiling=true gdb_index=true v8_enable_backtrace=true treat_warnings_as_errors=false is_debug=false is_official_build=true is_chrome_branded=true use_goma=true'

Comment 7 by cmt...@chromium.org, Dec 20 2016

I cannot reproduce your problem.  I did the three steps above; I ran until I got the goma error messages (unexpected end of file; compiler proxy not working).

Then I ran the compiler command for the file that you got the error on, and it worked just fine for me:

$ cd src/out_veyron_fievel/release
 $ /usr/local/google2/cmtice/chrome-src-top/src/third_party/llvm-build/Release+Asserts/bin/clang++ -MMD -MF v8_snapshot/obj/v8/v8_nosnapshot/extras-libraries.o.d -DV8_DEPRECATION_WARNINGS -DUSE_UDEV -DUI_COMPOSITOR_IMAGE_TRANSPORT -DUSE_AURA=1 -DUSE_NSS_CERTS=1 -DUSE_OZONE=1 -DUSE_PROPRIETARY_CODECS -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DOFFICIAL_BUILD -DGOOGLE_CHROME_BUILD -DENABLE_MEDIA_ROUTER=1 -DCR_CLANG_REVISION=289575-1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -DNO_UNWIND_TABLES -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DV8_I18N_SUPPORT -DENABLE_HANDLE_ZAPPING -DV8_USE_EXTERNAL_STARTUP_DATA -DV8_TARGET_ARCH_ARM -DCAN_USE_ARMV7_INSTRUCTIONS -DCAN_USE_VFP3_INSTRUCTIONS -DCAN_USE_VFP32DREGS -DCAN_USE_NEON -DUSE_EABI_HARDFLOAT=1 -I../.. -Iv8_snapshot/gen -I../../v8 -fno-strict-aliasing --param=ssp-buffer-size=4 -fstack-protector -g -fno-unwind-tables -fno-asynchronous-unwind-tables -fPIC -pipe -B../../third_party/binutils/Linux_x64/Release/bin -fcolor-diagnostics -fdebug-prefix-map=/Work/chromiumos/src=. -m32 -msse2 -mfpmath=sse -mmmx -momit-leaf-frame-pointer -pthread -mstack-alignment=16 -mstackrealign -fno-omit-frame-pointer -g2 -gsplit-dwarf -Wheader-hygiene -Wstring-conversion -Wall -Wno-unused-variable -Wno-missing-field-initializers -Wno-unused-parameter -Wno-c++11-narrowing -Wno-covered-switch-default -Wno-deprecated-register -Wno-unneeded-internal-declaration -Wno-inconsistent-missing-override -Wno-shift-negative-value -Wno-undefined-var-template -Wno-nonportable-include-path -Wno-address-of-packed-member -Wsign-compare -Winconsistent-missing-override -O3 -fno-ident -fdata-sections -ffunction-sections -fvisibility=default -fno-threadsafe-statics -fvisibility-inlines-hidden -std=gnu++11 -fno-rtti -fno-exceptions -Wno-deprecated -c v8_snapshot/gen/v8/extras-libraries.cc -o v8_snapshot/obj/v8/v8_nosnapshot/extras-libraries.o
$ ls -l v8_snapshot/obj/v8/v8_nosnapshot/extras-libraries.o-rw-r--r-- 1 cmtice eng 42608 Dec 20 08:28 v8_snapshot/obj/v8/v8_nosnapshot/extras-libraries.o
$
$ /usr/local/google2/cmtice/chrome-src-top/src/third_party/llvm-build/Release+Asserts/bin/clang++ --version
clang version 4.0.0 (trunk 289944)
Target: x86_64-unknown-linux-gnu
Thread model: posix
InstalledDir: /usr/local/google2/cmtice/chrome-src-top/src/third_party/llvm-build/Release+Asserts/bin
$

Is it possible you are using a different version of the compiler?  Or is there something else I need to do in order to reproduce this?
Try building without goma; i.e. change the gn args to have 'use_goma=false', and rebuild. That's how you can see the build failure.

Comment 9 by cmt...@chromium.org, Dec 20 2016

The following change causes this error to occur  (I tested reverting the change in my source tree, and then the file compiled just fine).  I'm still trying to figure out why the compiler is getting an internal error (it should not), but in the meantime, you might want to look at the change and verify that you don't see anything wrong there:

https://codereview.chromium.org/2578563002

Cc: epertoso@chromium.org
(Adding the person who created the CL that causes the problem).
Just for clarification, the actual compiler command that generates the error is:

armv7a-cros-linux-gnueabi-g++ -B/usr/local/google2/cmtice/chrome-src-top/.cros_cache/chrome-sdk/tarballs/veyron_fievel+9095.0.0+target_toolchain/usr/x86_64-pc-linux-gnu/armv7a-cros-linux-gnueabi/binutils-bin/2.25.51-gold -MMD -MF obj/v8/v8_base/js-native-context-specialization.o.d -DV8_DEPRECATION_WARNINGS -DUSE_UDEV -DUI_COMPOSITOR_IMAGE_TRANSPORT -DUSE_ASH=1 -DUSE_AURA=1 -DUSE_NSS_CERTS=1 -DUSE_OZONE=1 -DENABLE_WAYLAND_SERVER=1 -DUSE_PROPRIETARY_CODECS -DFULL_SAFE_BROWSING -DSAFE_BROWSING_CSD -DSAFE_BROWSING_DB_LOCAL -DOFFICIAL_BUILD -DGOOGLE_CHROME_BUILD -DENABLE_MEDIA_ROUTER=1 -D_FILE_OFFSET_BITS=64 -D_LARGEFILE_SOURCE -D_LARGEFILE64_SOURCE -DNO_UNWIND_TABLES -DOS_CHROMEOS -DNDEBUG -DNVALGRIND -DDYNAMIC_ANNOTATIONS_ENABLED=0 -DV8_I18N_SUPPORT -DENABLE_HANDLE_ZAPPING -DV8_USE_EXTERNAL_STARTUP_DATA -DV8_TARGET_ARCH_ARM -DCAN_USE_ARMV7_INSTRUCTIONS -DCAN_USE_VFP3_INSTRUCTIONS -DCAN_USE_VFP32DREGS -DCAN_USE_NEON -DU_USING_ICU_NAMESPACE=0 -DU_ENABLE_DYLOAD=0 -DU_STATIC_IMPLEMENTATION -DICU_UTIL_DATA_IMPL=ICU_UTIL_DATA_FILE -I../.. -Igen -I../../v8 -I../../v8/include -I../../third_party/icu/source/common -I../../third_party/icu/source/i18n -fno-strict-aliasing -g -fno-unwind-tables -fno-asynchronous-unwind-tables -fPIC -pipe -B../../third_party/binutils/Linux_x64/Release/bin -march=armv7ve -mfloat-abi=hard -mtune=generic-armv7-a -pthread -mfpu=neon -mthumb -fno-omit-frame-pointer -g2 -gsplit-dwarf --sysroot=../../../.cros_cache/chrome-sdk/tarballs/veyron_fievel+9095.0.0+sysroot_chromeos-base_chromeos-chrome.tar.xz -Wno-psabi -Wno-unused-local-typedefs -Wno-maybe-uninitialized -Wno-missing-field-initializers -Wno-unused-parameter -O3 -fno-ident -fdata-sections -ffunction-sections -fvisibility=default -fno-threadsafe-statics -fvisibility-inlines-hidden -std=gnu++11 -fno-rtti -fno-exceptions -Wno-deprecated -Wno-narrowing -Wno-literal-suffix -pipe -mcpu=cortex-a12 -mtune=cortex-a12 -mfpu=neon -mfloat-abi=hard -D__google_stl_debug_vector=1 -c ../../v8/src/compiler/js-native-context-specialization.cc -o obj/v8/v8_base/js-native-context-specialization.o -v -save-temps
It appears that if you compile at -O2 rather than -O3, the bug does not happen.
Continuing to investigate.  
Owner: epertoso@chromium.org
Status: Assigned (was: Untriaged)
Ok, here's what's happening.  The problem piece of code is here (in the new code added by the CL above, in InlineApiCall, in js-native-context-specialization.cc:

  // Add CallApiCallbackStub's register argument as well.
  Node* inputs[11] = {
      code, target, data, receiver /* holder */, function_reference, receiver};
  if (value != nullptr) {
    inputs[6] = value;
  }

At -O3, the compiler tries to do some fancy optimizations, combining all of this into one complex instruction:

(insn 459 458 460 20 (set (mem:V4SI (plus:SI (plus:SI
                 (if_then_else:SI (reg:SI 112 [ D.405243 ])
                   (const_int 28 [0x1c])
                   (const_int 24 [0x18]))
                (reg/f:SI 102 sfp))
             (const_int -48 [0xffffffffffffffd0])) 
    [0 MEM[(struct Node *[11]*)vectp.2397_175]+0 S16 A32]
  )
        (unspec:V4SI [
                (reg:V4SI 262 [ vect_cst_.2395 ])
            ] UNSPEC_MISALIGNED_ACCESS)) ../../v8/src/compiler/js-native-context-specialization.cc:1599 836 {*movmisalignv4si_neon_store}
     (expr_list:REG_DEAD (reg:V4SI 262 [ vect_cst_.2395 ])
        (nil)))


The problem is that a later part of the compiler that interprets address calculations was written with an assumption that if_then_else clauses would NOT be part of an address calculation.

The bad news is that the optimization that generates the if_then_else clause is in a very messy, large optimization pass, which we really don't want to mess with, particularly as we are trying to migrate away from using GCC altogether; we would much rather focus our efforts on the LLVM migration.

The good news is that you can get the compiler to NOT try to do this complex combining optimization by changing the source code just a little bit (move the two statements apart slightly).  The following patch does this, and with this patch I was able to build all of Chrome with the Simple Chrome workflow, for veyron_fievel, without any issues. 

I am going to reassign the bug to epertoso@, who submitted the original problem CL, so that he/she can apply this fix.

$ git diff js-native-context-specialization.cc
diff --git a/src/compiler/js-native-context-specialization.cc b/src/compiler/js-native-context-specialization.cc
index 408a6cd..2d9b93e 100644
--- a/src/compiler/js-native-context-specialization.cc
+++ b/src/compiler/js-native-context-specialization.cc
@@ -1592,14 +1592,14 @@ JSNativeContextSpecialization::InlineApiCall(
   // Add CallApiCallbackStub's register argument as well.
   Node* inputs[11] = {
       code, target, data, receiver /* holder */, function_reference, receiver};
-  if (value != nullptr) {
-    inputs[6] = value;
-  }
   int index = 6 + argc;
   inputs[index++] = context;
   inputs[index++] = frame_state;
   inputs[index++] = effect;
   inputs[index++] = control;
+  if (value != nullptr) {
+    inputs[6] = value;
+  }
 
   Node* effect0;
   Node* value0 = effect0 =



An alternative change would be to add an attribute to the function header so that this particular function gets compiled at -O2 instead of at -O3:

diff --git a/src/compiler/js-native-context-specialization.h b/src/compiler/js-native-context-specialization.h
index 24a9ef6..7a04057 100644
--- a/src/compiler/js-native-context-specialization.h
+++ b/src/compiler/js-native-context-specialization.h
@@ -152,7 +152,8 @@ class JSNativeContextSpecialization final : public AdvancedReducer {
       Node* receiver, Node* context, Node* target, Node* frame_state,
       Node* parameter, Node* effect, Node* control,
       Handle<SharedFunctionInfo> shared_info,
-      Handle<FunctionTemplateInfo> function_template_info);
+      //    Handle<FunctionTemplateInfo> function_template_info);
+      Handle<FunctionTemplateInfo> function_template_info)  __attribute__((optimize ("O2")));
 
   Graph* graph() const;
   JSGraph* jsgraph() const { return jsgraph_; }

Owner: cmt...@chromium.org
Status: Fixed (was: Assigned)
Thanks cmtice@ for looking into this. Should be fixed with http://crrev.com/2598463003. Can you confirm?

Comment 17 by cmtice@google.com, Dec 21 2016

Confirmed.
Isn't that just a workaround rather than an actual fix of the compiler error? I mean, we might end up with a similar compiler crash if the trigger is the same.

So, my question is: Is that enough to consider this bug "Fixed"?
It is true that the actual compiler bug is not truly fixed, but (for various reasons) we have no intention of trying to fix it, so this is as "fixed" as it will get.

Comment 20 by dchan@google.com, Mar 4 2017

Labels: VerifyIn-58

Comment 21 by dchan@google.com, Apr 17 2017

Labels: VerifyIn-59

Comment 22 by dchan@google.com, May 30 2017

Labels: VerifyIn-60
Labels: VerifyIn-61
Status: Verified (was: Fixed)
Closing. Please reopen it if its not fixed. Thanks!

Sign in to add a comment