Call stack looks like:

Thread 0 (crashed)
 0  chrome!void std::__1::__sort<AppListModelUpdater::GetFirstAvailablePositionInternal(std::__1::vector<ChromeAppListItem*, std::__1::allocator<ChromeAppListItem*> > const&)::$_0&, ChromeAppListItem**>(ChromeAppListItem**, ChromeAppListItem**, AppListModelUpdater::GetFirstAvailablePositionInternal(std::__1::vector<ChromeAppListItem*, std::__1::allocator<ChromeAppListItem*> > const&)::$_0&) [ordinal.h : 290 + 0x1]
    rax = 0x00001b328fcb08c0   rdx = 0x0000000000000002
    rcx = 0x00001b328fbdf900   rbx = 0x00001b328fcb08a8
    rsi = 0x00001b328e730c43   rdi = 0x00001b328f20f1e0
    rbp = 0x00007ffcd6eedab0   rsp = 0x00007ffcd6eeda20
     r8 = 0x000000008d78610a    r9 = 0x00001b328fbc1e21
    r10 = 0x0000000000000009   r11 = 0x000077e59f4771a0
    r12 = 0x0000000000000002   r13 = 0x00001b328fcb08c8
    r14 = 0x00001b328fcb08a8   r15 = 0x0000000000000003
    rip = 0x00005a0cb3b0c104
    Found by: given as instruction pointer in context
 1  chrome!AppListModelUpdater::GetFirstAvailablePositionInternal(std::__1::vector<ChromeAppListItem*, std::__1::allocator<ChromeAppListItem*> > const&) [algorithm : 4117 + 0xb]
    rbx = 0x00001b328fcb0900   rbp = 0x00007ffcd6eedb30
    rsp = 0x00007ffcd6eedac0   r12 = 0x0000000000000060
    r13 = 0x00001b328fcb08a0   r14 = 0x000000000000000c
    r15 = 0x00001b328fcb0900   rip = 0x00005a0cb3b0b27d
    Found by: call frame info
 2  chrome!ChromeAppListModelUpdater::GetFirstAvailablePosition() const [chrome_app_list_model_updater.cc : 383 + 0x8]
    rbx = 0x00007ffcd6eedc10   rbp = 0x00007ffcd6eedbb0
    rsp = 0x00007ffcd6eedb40   r12 = 0x00001b328d82fa80
    r13 = 0x00001b328d82fa60   r14 = 0x00001b328f6a2ec0
    r15 = 0x00001b328f884ab0   rip = 0x00005a0cb39392ec
    Found by: call frame info
 3  chrome!ChromeAppListItem::SetDefaultPositionIfApplicable(AppListModelUpdater*) [chrome_app_list_item.cc : 163 + 0xb]
    rbx = 0x00001b328e786a00   rbp = 0x00007ffcd6eedca0
    rsp = 0x00007ffcd6eedbc0   r12 = 0x00001b328f6a2eb0
    r13 = 0x00007ffcd6eeddb0   r14 = 0x00001b328f57d420
    r15 = 0x00007ffcd6eedc10   rip = 0x00005a0cb39369b4
    Found by: call frame info
 4  chrome!ArcAppModelBuilder::OnAppRegistered(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, ArcAppListPrefs::AppInfo const&) [arc_app_item.cc : 29 + 0xf]
    rbx = 0x00001b328f6a2eb0   rbp = 0x00007ffcd6eedd00
    rsp = 0x00007ffcd6eedcb0   r12 = 0x00007ffcd6eedea8
    r13 = 0x00007ffcd6eeddb0   r14 = 0x00001b328f57d420
    r15 = 0x00001b328f6a2be0   rip = 0x00005a0cb3957483
    Found by: call frame info
 5  chrome!ArcAppListPrefs::AddAppAndShortcut(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, bool, bool, bool, bool, bool, bool) [arc_app_list_prefs.cc : 1063 + 0x2]
    rbx = 0x0000000000000001   rbp = 0x00007ffcd6eedef0
    rsp = 0x00007ffcd6eedd10   r12 = 0x0000000000000000
    r13 = 0x002ed4ddcc838b0f   r14 = 0x0000000000000001
    r15 = 0x00001b328d119480   rip = 0x00005a0cb394e89a
    Found by: call frame info
 6  chrome!ArcAppListPrefs::AddApp(arc::mojom::AppInfo const&) [arc_app_list_prefs.cc : 1240 + 0xf]
    rbx = 0x0000000000000001   rbp = 0x00007ffcd6eee090
    rsp = 0x00007ffcd6eedf00   r12 = 0x00001b328d119480
    r13 = 0x00001b328e65dfe0   r14 = 0x00007ffcd6eee0a0
    r15 = 0x00001b328e65dfe8   rip = 0x00005a0cb39508b0
    Found by: call frame info
 7  chrome!ArcAppListPrefs::OnPackageAppListRefreshed(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::vector<mojo::StructPtr<arc::mojom::AppInfo>, std::__1::allocator<mojo::StructPtr<arc::mojom::AppInfo> > >) [arc_app_list_prefs.cc : 1294 + 0x8]
    rbx = 0x0000000000000000   rbp = 0x00007ffcd6eee210
    rsp = 0x00007ffcd6eee0a0   r12 = 0x00001b328d119480
    r13 = 0x00001b328e65dfe0   r14 = 0x00007ffcd6eee0a0
    r15 = 0x00001b328e65dfe8   rip = 0x00005a0cb3950ba8
    Found by: call frame info
 8  chrome!arc::mojom::AppHostStubDispatch::Accept(arc::mojom::AppHost*, mojo::Message*) [app.mojom.cc : 1339 + 0xb]
    rbx = 0x00007ffcd6eee290   rbp = 0x00007ffcd6eee4d0
    rsp = 0x00007ffcd6eee220   r12 = 0x00001b328d119488
    r13 = 0x00001b328e8e4400   r14 = 0x00007ffcd6eee540
    r15 = 0x00001b328ebf5cf8   rip = 0x00005a0caf7085c6
    Found by: call frame info

Change history between successful and failing builds are here:

https://chromium.googlesource.com/chromium/src/+log/34893e97f49e129ceb545f4edb3758e4b1bee1e5..edc217230468753d86cac298adf0632867170e8d

There are no obvious culprits at a first glance.

Oops, i was wrong, there are two pages of changes. This is the clear culprit:

https://chromium-review.googlesource.com/1231193

The build fails at a CHECK [1], so one fix is to filter out those items with invalid position.

[1] https://cs.chromium.org/chromium/src/components/sync/base/ordinal.h?q=ordinal.h&sq=package:chromium&dr=CSs&l=290

Ordinals should never be invalid (thus the check there), I wonder if the failure is actually a symptom of an invalid pointer? Although I don't see an obvious error in that logic.

I would add a bunch of extra CHECKs in a patch and use 'cros tryjob' to test it maybe? Also we should increase the test coverage and see if we can catch it that way.

I followed the instruction here to run command 
"cros tryjob -g 1237256 samus-release-afdo-use-tryjob",

but it fails with error message:
15:09:56: DEBUG: Cache dir lookup.
15:09:56: DEBUG: Configured cache_dir to '/usr/local/google/home/weidongg/chromium/src/third_party/.cache'
15:09:56: INFO: Verifying patches...
15:09:56: DEBUG: No netrc file or Appengine credentials found.
15:09:56: DEBUG: GET https://chromium-review.googlesource.com/a/changes/1237256/detail?o=CURRENT_REVISION&o=CURRENT_COMMIT
15:09:56: DEBUG: Cookie: HIDDEN
15:09:56: DEBUG: User-Agent: chromite.lib.gob_util cros 6dd0074561b786e22ce16b1e2233f6bef47b850a
15:09:57: INFO: Scheduling remote tryjob(s): caroline-tot-chrome-pfq-informational-tryjob
15:09:57: ERROR: cros tryjob failed before completing.
cros: Unhandled exception:
Traceback (most recent call last):
  File "/usr/local/google/home/weidongg/chromium/src/third_party/chromite/bin/cros", line 169, in <module>
    DoMain()
  File "/usr/local/google/home/weidongg/chromium/src/third_party/chromite/bin/cros", line 165, in DoMain
    commandline.ScriptWrapperMain(FindTarget)
  File "/usr/local/google/home/weidongg/chromium/src/third_party/chromite/lib/commandline.py", line 912, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/usr/local/google/home/weidongg/chromium/src/third_party/chromite/scripts/cros.py", line 66, in main
    code = _RunSubCommand(subcommand)
  File "/usr/local/google/home/weidongg/chromium/src/third_party/chromite/scripts/cros.py", line 51, in _RunSubCommand
    return subcommand.Run()
  File "/usr/local/google/home/weidongg/chromium/src/third_party/chromite/cli/cros/cros_tryjob.py", line 668, in Run
    return RunRemote(site_config, self.options, patch_pool)
  File "/usr/local/google/home/weidongg/chromium/src/third_party/chromite/cli/cros/cros_tryjob.py", line 315, in RunRemote
    args += PushLocalPatches(patch_pool.local_patches, user_email)
  File "/usr/local/google/home/weidongg/chromium/src/third_party/chromite/cli/cros/cros_tryjob.py", line 273, in PushLocalPatches
    manifest = git.ManifestCheckout.Cached(constants.SOURCE_ROOT)
  File "/usr/local/google/home/weidongg/chromium/src/third_party/chromite/lib/git.py", line 750, in Cached
    md5 = cls._GetManifestHash(manifest_path)
  File "/usr/local/google/home/weidongg/chromium/src/third_party/chromite/lib/git.py", line 501, in _GetManifestHash
    with open(source, 'rb') as f:
IOError: [Errno 2] No such file or directory: '/usr/local/google/home/weidongg/chromium/src/third_party/.repo/manifest.xml'

Do you know how to fix that?

After I clean up the local code base, the error become:

weidongg@weidongg:~/chromium/src/third_party/chromite$ cros tryjob caroline-tot-chrome-pfq-informational-tryjob -g 1237256 --debug
15:37:10: DEBUG: Cache dir lookup.
15:37:10: WARNING: The location of Chrome's cache dir has changed. The old path at /usr/local/google/home/weidongg/chromium/.cros_cache can safely be removed.
15:37:10: DEBUG: Configured cache_dir to '/usr/local/google/home/weidongg/chromium/src/build/cros_cache'
15:37:11: INFO: Verifying patches...
15:37:11: DEBUG: No netrc file or Appengine credentials found.
15:37:11: DEBUG: GET https://chromium-review.googlesource.com/a/changes/1237256/detail?o=CURRENT_REVISION&o=CURRENT_COMMIT
15:37:11: DEBUG: Cookie: HIDDEN
15:37:11: DEBUG: User-Agent: chromite.lib.gob_util cros 6dd0074561b786e22ce16b1e2233f6bef47b850a
15:37:13: INFO: Scheduling remote tryjob(s): caroline-tot-chrome-pfq-informational-tryjob
15:37:13: ERROR: cros tryjob failed before completing.
cros: Unhandled exception:
Traceback (most recent call last):
  File "/usr/local/google/home/weidongg/chromium/src/third_party/chromite/bin/cros", line 169, in <module>
    DoMain()
  File "/usr/local/google/home/weidongg/chromium/src/third_party/chromite/bin/cros", line 165, in DoMain
    commandline.ScriptWrapperMain(FindTarget)
  File "/usr/local/google/home/weidongg/chromium/src/third_party/chromite/lib/commandline.py", line 912, in ScriptWrapperMain
    ret = target(argv[1:])
  File "/usr/local/google/home/weidongg/chromium/src/third_party/chromite/scripts/cros.py", line 66, in main
    code = _RunSubCommand(subcommand)
  File "/usr/local/google/home/weidongg/chromium/src/third_party/chromite/scripts/cros.py", line 51, in _RunSubCommand
    return subcommand.Run()
  File "/usr/local/google/home/weidongg/chromium/src/third_party/chromite/cli/cros/cros_tryjob.py", line 668, in Run
    return RunRemote(site_config, self.options, patch_pool)
  File "/usr/local/google/home/weidongg/chromium/src/third_party/chromite/cli/cros/cros_tryjob.py", line 315, in RunRemote
    args += PushLocalPatches(patch_pool.local_patches, user_email)
  File "/usr/local/google/home/weidongg/chromium/src/third_party/chromite/cli/cros/cros_tryjob.py", line 273, in PushLocalPatches
    manifest = git.ManifestCheckout.Cached(constants.SOURCE_ROOT)
  File "/usr/local/google/home/weidongg/chromium/src/third_party/chromite/lib/git.py", line 748, in Cached
    search=search)
  File "/usr/local/google/home/weidongg/chromium/src/third_party/chromite/lib/git.py", line 572, in _NormalizeArgs
    raise OSError(errno.ENOENT, "Couldn't find repo root: %s" % (path,))
OSError: [Errno 2] Couldn't find repo root: /usr/local/google/home/weidongg/chromium/src/third_party

You need to run 'cros tryjon' in a chromeos/src directory.

We used to be able to do it using cbuildbot with -G to specify a chromium CL. But "cros tryjob" seems not working with that.

Ah, I see, it works in chromiumos/src/ folder. Thanks!

Triage nag: This Chrome OS bug has an owner but no component. Please add a component so that this can be tracked by the relevant team.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/0c7979fa86ca386d95b209931612ccf90f2a173c

commit 0c7979fa86ca386d95b209931612ccf90f2a173c
Author: Weidong Guo <weidongg@chromium.org>
Date: Mon Oct 01 16:10:50 2018

Reland "Install app to first available position in app list"

This is a reland of e39c47e5986f7927d7330d923364aae8a4a5d861

Original change's description:
> Install app to first available position in app list
>
> Changes:
> 1. Calculate the first available position in AppListModelUpdater for
>    newly added item.
> 2. Add test coverage.
>
> Bug: 883939
> Test: AppListSyncableServiceTest.FirstAvailablePosition
> Change-Id: Ibdd2f27455ffd0c76f45906adbc025baab1ff319
> Reviewed-on: https://chromium-review.googlesource.com/1231193
> Commit-Queue: Weidong Guo <weidongg@chromium.org>
> Reviewed-by: Xiyuan Xia <xiyuan@chromium.org>
> Reviewed-by: Steven Bennetts <stevenjb@chromium.org>
> Cr-Commit-Position: refs/heads/master@{#592539}

Bug: 883939, 887636 
Change-Id: Id630fd99023a670ddcec52c977f20f3e1092f8b6
Reviewed-on: https://chromium-review.googlesource.com/1237256
Reviewed-by: Xiyuan Xia <xiyuan@chromium.org>
Reviewed-by: Steven Bennetts <stevenjb@chromium.org>
Commit-Queue: Weidong Guo <weidongg@chromium.org>
Cr-Commit-Position: refs/heads/master@{#595461}
[modify] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/ash/app_list/views/apps_grid_view.cc
[modify] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/ash/public/cpp/app_list/app_list_config.cc
[modify] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/ash/public/cpp/app_list/app_list_config.h
[modify] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/chrome/browser/chromeos/login/session/user_session_manager.cc
[modify] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/chrome/browser/ui/BUILD.gn
[modify] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/chrome/browser/ui/app_list/app_context_menu_unittest.cc
[add] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/chrome/browser/ui/app_list/app_list_model_updater.cc
[modify] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/chrome/browser/ui/app_list/app_list_model_updater.h
[modify] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/chrome/browser/ui/app_list/app_list_syncable_service_factory.cc
[modify] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/chrome/browser/ui/app_list/app_list_syncable_service_unittest.cc
[modify] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/chrome/browser/ui/app_list/arc/arc_app_item.cc
[modify] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/chrome/browser/ui/app_list/chrome_app_list_item.cc
[modify] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/chrome/browser/ui/app_list/chrome_app_list_item.h
[modify] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/chrome/browser/ui/app_list/chrome_app_list_model_updater.cc
[modify] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/chrome/browser/ui/app_list/chrome_app_list_model_updater.h
[add] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/chrome/browser/ui/app_list/chrome_app_list_model_updater_browsertest.cc
[modify] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/chrome/browser/ui/app_list/crostini/crostini_app_item.cc
[modify] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/chrome/browser/ui/app_list/crostini/crostini_app_model_builder.cc
[modify] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/chrome/browser/ui/app_list/extension_app_item.cc
[modify] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/chrome/browser/ui/app_list/internal_app/internal_app_item.cc
[modify] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/chrome/browser/ui/app_list/internal_app/internal_app_item.h
[modify] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/chrome/browser/ui/app_list/internal_app/internal_app_model_builder.cc
[modify] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/chrome/browser/ui/app_list/test/fake_app_list_model_updater.cc
[modify] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/chrome/browser/ui/app_list/test/fake_app_list_model_updater.h
[modify] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/chrome/test/BUILD.gn
[add] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/chrome/test/data/extensions/app_list_oem/Extensions/emfkafnhnpcmabnnkckkchdilgeoekbo/1.0/icon_128.png
[add] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/chrome/test/data/extensions/app_list_oem/Extensions/emfkafnhnpcmabnnkckkchdilgeoekbo/1.0/main.html
[add] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/chrome/test/data/extensions/app_list_oem/Extensions/emfkafnhnpcmabnnkckkchdilgeoekbo/1.0/manifest.json
[add] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/chrome/test/data/extensions/app_list_oem/Preferences
[modify] https://crrev.com/0c7979fa86ca386d95b209931612ccf90f2a173c/chrome/test/data/extensions/theme/Cached Theme.pak