Thanks for reporting this. It was fixed recently in https://bugs.chromium.org/p/chromium/issues/detail?id=773561. 

@laszio, I don't think this is the same issue (ie: it is not a duplicate).

From what we discussed, what they are reporting here is that local images they build perform better than the images we release. which should not be the case. 

on email I asked them to remove the the non-verification flag for the build_image command. But I dont see it being used here. 

@hong.zheng. 

is your build_image command at 7) complete? are you sure you are not turning off the image verification?

I agree with llozano this is another issue. 
@llozano, command 7) is the last command to build image. Because root user can't remount file system, I think the verification should be enabled. could the issue be caused by manifest difference, paladin manifest and offical manifest?

Just to add context:
In the doc I commented:

"It is pretty strange that you are getting better performance. What I meant by disabling write protection is that you are using --noenable_rootfs_verification. We have seen performance differences because of this. Please try without this option."

And Hong Zheng replied:

"We delete --noenable_rootfs_verification and --adjust_part='ROOT-A:+1G', but performance gap still exists. 

We notice there are several manifests for one revision in paladin folder, for example, for revision 9887.0.0, there are 9887.0.0-rc1.xml, 9887.0.0-rc2.xml and 9887.0.0-rc3.xml. But there is only one 9887.0.0 partner image. Do you also use manifests in paladin folder to build partner image? If yes, which one do you use, rc1/rc2/rc3? If not, what manifest do you use?"

So, it looks like you are not using --noenable_rootfs_verification anymore. 

The only 2 differences I can think of are:
1) USE="chrome_internal". I think you cannot try this. So, we would have to try building without this USE flag on our side.
2) AFDO for Kernel. @laszio: is reef one of the boards being optimized with Kernel AFDO?

About your question about paladin manifests, I don't know the answer. I will have to dig deeper but I don't think the problem is coming from here. 

@laszio, do you have any other possible explanations?

reef didn't participate in the kernel experiment. Some random guesses:

1. The afdo profile might be different; Can you check which autofdo profile is used in your local build? I can check whether it is good or not. The profile used in the official build is bad :(

$ cat src/third_party/chromiumos-overlay/chromeos-base/chromeos-chrome/chromeos-chrome-9999.ebuild | grep LLVM | grep '\.afdo'


2. There might be difference in USE flags.
These are relevant flags when building chrome in build_packages:

[ebuild  N     ] chromeos-base/chromeos-chrome-62.0.3198.0_rc-r1::chromiumos to /build/reef/ USE="accessibility afdo_use authpolicy autotest build_tests buildcheck chrome_debug chrome_internal chrome_remoting clang cups debug_fission evdev_gestures fonts gold highdpi nacl opengles runhooks v4l2_codec vaapi xkbcommon -app_shell -asan -chrome_debug_tests -chrome_media -component_build -goma -hardfp -internal_gles_conform -mojo (-neon) -opengl -thinlto -v4lplugin -verbose -vtable_verify" OZONE_PLATFORM="gbm -caca -cast -egltest {-test}" OZONE_PLATFORM_DEFAULT="gbm -caca -cast -egltest {-test}" 0 KiB

Would you mind to find out what it is in your logs?


3. Would you mind to check the toolchain versions? The official image was built on Aug.28 and there might be some difference in the toolchain.

$ (in chroot) x86_64-cros-linux-gnu-clang -v
$ (in chroot) x86_64-cros-linux-gnu-gcc -v
$ (in chroot) x86_64-cros-linux-gnu-ld -v

4. FWIW, I also checked recent changes in the compiler wrapper and none of them look suspicious.

@laszio, thanks for your reply, I collect some information depending on your suggestion, please review
1. about afdo
AFDO_FILE_LLVM["amd64"]="chromeos-chrome-amd64-61.0.3163.67_rc-r1.afdo"
AFDO_FILE_LLVM["x86"]="chromeos-chrome-amd64-61.0.3163.67_rc-r1.afdo"
AFDO_FILE_LLVM["arm"]="chromeos-chrome-amd64-61.0.3163.67_rc-r1.afdo"

2. about USE flags
[ebuild   R   *] chromeos-base/chromeos-chrome-9999::chromiumos to /build/reef/ USE="accessibility afdo_use authpolicy autotest build_tests buildcheck chrome_debug chrome_remoting clang cups debug_fission evdev_gestures fonts gold highdpi nacl opengles runhooks v4l2_codec vaapi xkbcommon -app_shell -asan -chrome_debug_tests -chrome_internal -chrome_media -component_build -goma -hardfp -internal_gles_conform -mojo (-neon) -opengl -thinlto -v4lplugin -verbose -vtable_verify" OZONE_PLATFORM="gbm -caca -cast -egltest {-test}" OZONE_PLATFORM_DEFAULT="gbm -caca -cast -egltest {-test}" 0 KiB

3. about toolchain versions
1) x86_64-cros-linux-gnu-clang -v
Chromium OS 5.0_pre305632_p20170806-r3 clang version 5.0.0 (/var/cache/chromeos-cache/distfiles/host/egit-src/clang.git 30060bff5b4cb49e17c27672d1aa60e6bc7a95e8) (/var/cache/chromeos-cache/distfiles/host/egit-src/llvm.git b903fddc562ccc622cabc4f08f5df2af90ceb251) (based on LLVM 5.0.0svn)
Target: x86_64-cros-linux-gnu
Thread model: posix
InstalledDir: /usr/bin
Found candidate GCC installation: /usr/bin/../lib/gcc/i686-pc-linux-gnu/4.9.x
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-cros-linux-gnu/4.9.x
Found candidate GCC installation: /usr/bin/../lib/gcc/x86_64-pc-linux-gnu/4.9.x
Found candidate GCC installation: /usr/bin/../lib64/gcc/x86_64-cros-linux-gnu/4.9.x
Found candidate GCC installation: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/4.9.x
Selected GCC installation: /usr/bin/../lib64/gcc/x86_64-pc-linux-gnu/4.9.x
Candidate multilib: .;@m64
Candidate multilib: 32;@m32
Selected multilib: .;@m64

2) x86_64-cros-linux-gnu-gcc -v
Using built-in specs.
COLLECT_GCC=/usr/x86_64-pc-linux-gnu/x86_64-cros-linux-gnu/gcc-bin/4.9.x/x86_64-cros-linux-gnu-gcc.real
COLLECT_LTO_WRAPPER=/usr/libexec/gcc/x86_64-cros-linux-gnu/4.9.x/lto-wrapper
Target: x86_64-cros-linux-gnu
Configured with: /var/tmp/portage/cross-x86_64-cros-linux-gnu/gcc-4.9.2-r163/work/gcc-4.9.2/gcc-4.9/configure --prefix=/usr --bindir=/usr/x86_64-pc-linux-gnu/x86_64-cros-linux-gnu/gcc-bin/4.9.x --datadir=/usr/share/gcc-data/x86_64-cros-linux-gnu/4.9.x --includedir=/usr/lib/gcc/x86_64-cros-linux-gnu/4.9.x/include --with-gxx-include-dir=/usr/lib/gcc/x86_64-cros-linux-gnu/4.9.x/include/g++-v4 --mandir=/usr/share/gcc-data/x86_64-cros-linux-gnu/4.9.x/man --infodir=/usr/share/gcc-data/x86_64-cros-linux-gnu/4.9.x/info --with-python-dir=/share/gcc-data/x86_64-cros-linux-gnu/4.9.x/python --build=x86_64-pc-linux-gnu --host=x86_64-pc-linux-gnu --target=x86_64-cros-linux-gnu --enable-languages=c,c++ --enable-__cxa_atexit --disable-canonical-system-headers --enable-checking=release --enable-linker-build-id --with-bugurl=http://code.google.com/p/chromium-os/issues/entry --with-pkgversion=4.9.2_cos_gg_4.9.2-r163-0c5a656a1322e137fa4a251f2ccc6c4022918c0a_4.9.2-r163 --disable-libatomic --disable-multilib --enable-libgomp --disable-libcilkrts --disable-libitm --disable-libmudflap --disable-libquadmath --disable-libssp --enable-frame-pointer --enable-poison-system-directories --with-sysroot=/usr/x86_64-cros-linux-gnu
Thread model: posix
gcc version 4.9.x 20150123 (prerelease) (4.9.2_cos_gg_4.9.2-r163-0c5a656a1322e137fa4a251f2ccc6c4022918c0a_4.9.2-r163) 

3) x86_64-cros-linux-gnu-ld -v
GNU gold (binutils-2.27-85fafaf039799ebc8053bf36ce1c6e6df7adbbec_cos_gg 2.27.0.20170315) 1.12

Thank you, Hong. I think this is the reason:

> 1. about afdo
> AFDO_FILE_LLVM["amd64"]="chromeos-chrome-amd64-61.0.3163.67_rc-r1.afdo"
> AFDO_FILE_LLVM["x86"]="chromeos-chrome-amd64-61.0.3163.67_rc-r1.afdo"
> AFDO_FILE_LLVM["arm"]="chromeos-chrome-amd64-61.0.3163.67_rc-r1.afdo"

The profiles we generated for most of R62 and R63 are broken. You may think that the partner image was built WITHOUT autofdo. Your local image was built WITH autofdo with a old profile.

So it is still valid to merge this into 773561 :)

I'll check other settings before closing this again.

@laszio, I notice the  bug 773561 's impact begins from revision 9901, but our issue is in revision 9887. They might be not the same issue. FWIW, I will sync the latest code to verify the issue.

You are right. Although the compiler upgrade happens 3 weeks before 9987, the profile was not updated for another reason. Let me investigate.

Hmm, the compiler and linker versions are the same. The only difference that I can tell is the USE flag "chrome_internal", which builds a Google branded Chrome rather than Chromium.

CC'ing Ben just in case this is a more general performance problem and not specific to toolchain.

yes, the only difference I see now is "chrome_internal". Ting-Yuan can you please try getting crosperf run for +chrome_internal and -chrome_internal? See if we can see the performance differences?

I verify the issue in revision 10100.0.0 and the gap still exists. @laszio and @llozano, do you have any update?

Hi, we are trying to compare the performance difference with and without USE=chrome_internal and should have some result by tomorrow.

It looks like that USE=chrome_internal doesn't make a difference.

Benchmark:  page_cycler_v2.typical_25;  Iterations: 3
                                 keys -chrome_internal  (pass:3 fail:0)                   +chrome_internal  (pass:3 fail:0)
                                 Keys  Amean  StdDev StdDev/Mean  Amean  StdDev StdDev/Mean GmeanSpeedup p-value
cold@@timeToOnload_avg__summary (ms)  2308.90  57.21        2.5% 2261.95  28.43        1.3%        +2.1%    0.36
warm@@timeToOnload_avg__summary (ms)  2020.71  12.19        0.6% 1991.20  26.23        1.3%        +1.5%    0.22

We are actually interested in tracking this down as we see performance differences in locally built images as opposed to the official image too, though it is the opposite where the local image performance worse. We are working on bisection tools where we want to achieve the same performance levels as the official builds too.

I collected pagecycler_v2 trace and found some difference between local image and partner image, for example there is GpuMemoryThread in renderer process in partner image, but not in local image. I don't know whether the difference can cause performance gap. But it may be a clue for this issue. Do you know why there is difference between traces, or the purpose of GpuMemoryThread?

Deleted: local-http___allrecipes_com_Recipe_Pull_Apart_Hot_Cross_Buns_Detail_aspx.html 5.0 MB

local-http___allrecipes_com_Recipe_Pull_Apart_Hot_Cross_Buns_Detail_aspx.html 5.0 MB View Download

Deleted: partner-http___allrecipes_com_Recipe_Pull_Apart_Hot_Cross_Buns_Detail_aspx.html 5.2 MB

partner-http___allrecipes_com_Recipe_Pull_Apart_Hot_Cross_Buns_Detail_aspx.html 5.2 MB View Download

[It appears that a bunch of old cros issues bulk-added the "Infra" component recently, but they should probably be "Infra>Client>ChromeOS".]

Hi, we are bringing up with some updates.

On 65-10191.0.0 image, we can still observe ~3% performance leadership on local ChromiumOS over official ChromeOS images by running page_cycler_v2.typical_25. We did some breakdown and below are the findings so far.

'Flickr.com' has relatively larger performance gap (~20%) between official and local images. Through our analysis, we find out that one of the main differences of two images is in 'SSL_CONNECT' part, shown as the attached files.

For official ChromeOS image, the read event for SSLSocket is not active on I/O until a long period of waiting (~50ms); while for local ChromiumOS, the event for SSLSocket is active just shortly after it was added for monitor on I/O. Browser would use Libevent as the event notification library, and further use 'epoll' method inside Libevent for Chromium/ChromeOS. The main difference in 'SSL_CONNECT' is the different event active timings from 'epoll'. As 'epoll' is a Linux kernel system call for I/O event notification mechanism, we suspect the difference might be related with kernel.

We also notice that in the chromium kernel design doc (https://www.chromium.org/chromium-os/chromiumos-design-docs/chromium-os-kernel), it mentions that kernel configs are different between ChromeOS and ChromiumOS - 'a smaller one to support the official Google Chrome OS devices, and a larger file to enable broader support for Chromium OS'. 

We wonder if we could get the official kernel configs for comparison? If not, would it be possible that you help us to do a comparison and provide us with some hints on this issue? Maybe then we can do some experiments according to these hints.

Do you have any comments based on our findings?

Deleted: official-flickr-trace.png 190 KB

	official-flickr-trace.png 190 KB View Download

Deleted: local-flickr-trace.png 211 KB

	local-flickr-trace.png 211 KB View Download

that config section dates back to 2010 and has been accurate for a while.  there are no private/differences between Chrome OS & Chromium OS when it comes to kernel settings.  i'll delete the section.

Thanks for the comments, vapier@. 
Do you mean that the kernel used in Chrome OS and Chromium OS are the same and the difference we observed in C#21 is not kernel related? If yes, do you know what the possible causes of such kind of difference between these 2 images are?

yes, we only have public kernels & configs for all our devices.  you have the same exact set of sources & kernel settings as our bots.  you can even verify by doing `modprobe configs` and `zcat /proc/config.gz`.

we've been turning on afdo for the kernel for more boards, but those profiles too are public, so you should be able to get the same build.

if you wanted, you could rule out the kernel entirely by just copying over the kernel partition (and the modules in the rootfs).

Thanks for the comments, vapier@.
We've verified the kernel settings as you suggested and there are no difference between ChromeOS and ChromiumOS images. Also 'kernel_afdo' is already used in our local build.
But when we tried to replace the official ChromeOS kernel with our local build one (the same kernel version used) by running 'update_kernel.sh', after reboot it went wrong and didn't boot normally. Do you know what we did wrong in replacing the kernel? 

We've tried to replace browser/kernel in official image, and the results showed that they were not the culprits of such performance gap in page_cycler_v2. Do you have any suggestions on what we can do to track down this issue in next step?

bccheng@, as you mentioned in C#17, we were wondering if you have any updates on this issue that can share with us. Do you also use 'page_cycler_v2.typical_25' to analyze such performance gaps?

Not yet unfortunately.

Do you know which CPU family shows the largest performance difference between the builds?

We found this issue on reef board, and the CPU family is 6. We'll try to verify it on more platforms later.

One more question, do you use two different machines to compare? As there is another issue that different USB ethernet dongle might have impact on the performance benchmarks.

No, we just use the same machine but flash different images for comparison.