This used to work fairly well as long as the xref you were looking for was compiled on Linux.  But now all kinds of xrefs seem to be missing.  This has gotten much worse some time recently (in the last month).

Also, xrefs are completely broken on cs-staging. I don't get links when I hover over symbols, despite the layer being enabled.

Upping priority.  All kinds of cross-references are broken and using codesearch has been much more hassle for me for weeks.  This is the biggest tools issue in my day-to-day work.

In case it's useful, here's another broken case: https://cs.chromium.org/chromium/src/components/omnibox/browser/in_memory_url_index.cc?rcl=0&l=130 , "HistoryItemsForTerms" is not clickable.

Dave, you are still set as owner on this.  Are you actively working on it?

If not, Emma, can you take a look?  I agree with the P1 classification here.

I'll have a look. 

I'm ramping up on the code base so I'll likely need some assistance from Dave and others. 

What I found so far:

- I can reproduce the problem for https://cs.chromium.org/chromium/src/components/omnibox/browser/in_memory_url_index.cc?rcl=0&l=130 but not the earlier two examples

- Calling the Xref service I don't see method entries for the line 130 in /components/omnibox/browser/in_memory_url_index.cc. That is, looks like one or more refs are missing in the computed Xref data.

- Xref jobs seem to run fine.

- Trying to run Xref computation locally results in an error. This may be the cause of the problem. I have reached out to the team responsible to investigate further.

After more digging the problem is likely in the chromium extraction where some files are not included in the index pack. That is, they are listed by https://cs.chromium.org/chromium/src/tools/clang/scripts/run_tool.py in the compilation database provided to https://cs.chromium.org/chromium/build/scripts/slave/chromium/package_index.py but the file data is not available.

For the example error, the HistoryItemsForTerms method without a ref is called on an instance of the type URLIndexPrivateData defined in components/omnibox/browser/url_index_private_data.h, and this file includes components/omnibox/browser/in_memory_url_index_cache.pb.h for which the cc file data is missing.

The problem could be that generated files are not generated, but I haven't fully verified this. This seems related to https://bugs.chromium.org/p/chromium/issues/detail?id=633865, where the need for a "null build" was discussed, that is, a build only building generators and using those to generate files. 

The problem is not just with generated files. My example in the initial comment still doesn't work for me, and it doesn't involve any generated files.

Sorry, I initially diagnosed this as one issue (reproduced with pkasting@'s example), but there is more than one issue at play here. 

mattm@, I see the problem you are referring to (missing uses) and I need to investigate this further. I suspect something platform related, but that's just an initial guess.

 Issue 644153  has been merged into this issue.

Adding info from 644153: this also affects the src/cc directory such as "SetScrollOffset" and "TakeDebugInfo" in src/cc/layers/layer.cc

Logging into the bot running https://build.chromium.org/p/chromium.infra.cron/builders/Chromium%20Linux%20Codesearch the compilation database generated by ninja includes a compilation unit for gen/components/omnibox/browser/in_memory_url_index_cache.pb.cc but the actual file is missing. 

However, the compile step of the Builder has been failing for at least the last 200 builds (back to Aug 18: https://build.chromium.org/p/chromium.infra.cron/builders/Chromium%20Linux%20Codesearch/builds/2813). The builder issue is likely related to crbug/621828 which is being addressed for the code search builder in https://codereview.chromium.org/2332283002/.

Emma, can you also look into adding alerts for when the codesearch builders start failing reliably?  Getting a follow up bug on file is fine, but, uh, failing for at least 200 builds without us knowing it isn't a good status quo :)

Yes, good point. I'll discuss possible alerts with Dave. 

The Builder as a whole is green while the compile step is orange. I think we would get an alert if the overall Builder failed, or if the step failed. I'll investigate if we can change this to make the step go red and make the Builder fail.

There's  issue 637807  to add alerts for this, from the last postmortem :(

At least there is a consistent need for it? 😐

FYI this has been strongly affecting my entire team for over a month. An extremely valuable piece of developer infrastructure should not be down for this long. You may want to consider bumping this to P0.

There has been a long running problem with the compiler step in the code search builders. We are reviewing how to improve our alerting.

The issue causing the compiler step to fail is goma related and once this change (https://codereview.chromium.org/2353083005/) is landed that may be resolved.

The goma fix (https://codereview.chromium.org/2353083005/) fixed the problems in the compile step: https://uberchromegw.corp.google.com/i/chromium.infra.cron/builders/Chromium%20Linux%20Codesearch/builds/3026

The run_translation_unit_clang_tool step is still failing but less than before: was Sucess=29319/Failure=474 and is now Success=29639/Failure=167. Reviewing the failures they are mostly on mojom files which suggests that they are not being generated in the compile step. Previously there were a lot of errors due to missing generated proto files, theses seem to now have gone away.

Once the results of this more successful build goes all the way to the Xref service, the result can be inspected in more detail via code search.

maxboque: I agree with you. The infra team is investing more in code search to make it more stable and to improve those parts of it most critical to users, for instance, Xrefs. Do you have an example of an Xref that haven't worked for your team?

Sure. Of the four DCHECKs in the SharedModelTypeProcessor::OnSyncStarting method here:

https://cs.chromium.org/chromium/src/components/sync/core/shared_model_type_processor.cc?l=104

the symbols CalledOnValidThread, start_callback_, and error_handler all don't highlight properly when moused over, and clicking on the OnSyncStarting method does not properly display that it overrides the one in ModelTypeChangeProcessor.

Sometimes code inside #if defined(OS_CHROMEOS) blocks should highlight. For example, this line does not:

https://cs.chromium.org/chromium/src/ash/common/system/tray/system_tray.cc?sq=package:chromium&dr=CSs&l=186

However, there is other code that only compiles on Chrome OS that does have xref highlighting:

https://cs.chromium.org/chromium/src/chromeos/settings/timezone_settings.h?sq=package:chromium&dr=CSs&rcl=1474529952&l=24

The code search builder for chromium was fully green for the first time in a while yesterday. https://codereview.chromium.org/2373463002/ resolved the issue with the missing generated files. There is still some issue with the Chrome OS code search builder that needs to be investigated.

Verfied that the now generated files resolve the issue with missing links like that for HistoryItemsForTerms (example from pkasting above): https://cs.chromium.org/chromium/src/components/omnibox/browser/in_memory_url_index.cc?rcl=0&l=130

Verified that the first issue with the Xrefs of CreateDefault is resolved. That is, clicking on CreateDefault on https://cs.chromium.org/chromium/src/net/cert/cert_verifier.h?rcl=1474932129&l=177 includes /chrome/browser/io_thread.cc in the result list.

Verfied the Xrefs of https://cs.chromium.org/chromium/src/components/sync/core/shared_model_type_processor.cc?rcl=0&l=101

I'd like to move problems with Chrome OS specific Xrefs to  issue 652215 . These problems are part of the multi-platform support we would like to provide (on the road map).

The majority of the symptoms in this bug where related to the long-running failure in the code search builder for Linux. This issue has now been resolved.