andrewhayden@, could you please take a look?

I'm confused how this is a regression. CLD2 is unchanged since August 2015, and there is absolutely nothing in the regression range that looks like it could be related to this. I can't access the testcase link (for no good reason that I can imagine).

Anyhow here is the data from the crash:

third_party/cld_2/src/internal/utf8statetable.cc:300:18: runtime error: left shift of negative value -2
    #0 0x4e0174 in CLD2::UTF8GenericPropertyBigOneByte(CLD2::UTF8StateMachineObj const*, unsigned char const**, int*) third_party/cld_2/src/internal/utf8statetable.cc:300:18
    #1 0x4e4507 in CLD2::GetUniHits(char const*, int, int, CLD2::ScoringContext*, CLD2::ScoringHitBuffer*) third_party/cld_2/src/internal/cldutil.cc:224:19
    #2 0x4dd7b4 in CLD2::ScoreCJKScriptSpan(CLD2::LangSpan const&, CLD2::ScoringContext*, CLD2::DocTote*, std::__1::vector<CLD2::ResultChunk, std::__1::allocator<CLD2::ResultChunk> >*) third_party/cld_2/src/internal/scoreonescriptspan.cc:1192:23
    #3 0x4ddc72 in CLD2::ScoreOneScriptSpan(CLD2::LangSpan const&, CLD2::ScoringContext*, CLD2::DocTote*, std::__1::vector<CLD2::ResultChunk, std::__1::allocator<CLD2::ResultChunk> >*) third_party/cld_2/src/internal/scoreonescriptspan.cc:1327:5
    #4 0x4cc1c8 in CLD2::DetectLanguageSummaryV2(char const*, int, bool, CLD2::CLDHints const*, bool, int, CLD2::Language, CLD2::Language*, int*, double*, std::__1::vector<CLD2::ResultChunk, std::__1::allocator<CLD2::ResultChunk> >*, int*, bool*) third_party/cld_2/src/internal/compact_lang_det_impl.cc:1938:5
    #5 0x4c17ca in CLD2::ExtDetectLanguageSummaryCheckUTF8(char const*, int, bool, CLD2::CLDHints const*, int, CLD2::Language*, int*, double*, std::__1::vector<CLD2::ResultChunk, std::__1::allocator<CLD2::ResultChunk> >*, int*, bool*, int*) third_party/cld_2/src/internal/compact_lang_det.cc:339:19
    #6 0x42de68 in DetermineTextLanguage components/translate/core/language_detection/language_detection_util.cc:118:18
    #7 0x42de68 in translate::DeterminePageLanguage(std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> > const&, std::__1::basic_string<unsigned short, base::string16_char_traits, std::__1::allocator<unsigned short> > const&, std::__1::basic_string<char, std::__1::char_traits<char>, std::__1::allocator<char> >*, bool*) components/translate/core/language_detection/language_detection_util.cc:234

I would imagine that if this were a serious problem we would have known about it 6+ months ago, so I am dubious about Pri-1. Downgrading to Pri-2 and moving myself to CC and reassigning to Jason Riesa, who owns CLD2.

Detailed report: https://cluster-fuzz.appspot.com/testcase?key=5333430950952960

Fuzzer: libfuzzer_language_detection_fuzzer
Job Type: libfuzzer_chrome_ubsan
Platform Id: linux

Crash Type: Undefined-shift
Crash Address: 
Crash State:
  CLD2::UTF8GenericPropertyBigOneByte
  CLD2::GetUniHits
  CLD2::ScoreCJKScriptSpan
  
Regressed: https://cluster-fuzz.appspot.com/revisions?job=libfuzzer_chrome_ubsan&range=395640:395746

Minimized Testcase (0.00 Kb): https://cluster-fuzz.appspot.com/download/AMIfv9557CmH7o7mLRFqZZKCUElP9LF8gkw4oNLhj0TpcXMoAfZ7tVCu9KrEiIXAR1CMR4hQ42ulR0bceqk5Jrux3lbLzf-TXLiNiAOWUEGDYVKXvQFnc8BJjgeyHqPTddvQdNXdLJtot9Ou9Fyg292ItgYbn-vldQ?testcase_id=5333430950952960

Filer: kavvaru

See https://chromium.googlesource.com/chromium/src/+/master/testing/libfuzzer/reproducing.md for more information.

Detailed report: https://cluster-fuzz.appspot.com/testcase?key=6269761982038016

Fuzzer: libfuzzer_language_detection_fuzzer
Job Type: libfuzzer_chrome_ubsan
Platform Id: linux

Crash Type: Undefined-shift
Crash Address: 
Crash State:
  CLD2::UTF8GenericPropertyBigOneByte
  CLD2::GetUniHits
  CLD2::ScoreCJKScriptSpan
  
Regressed: https://cluster-fuzz.appspot.com/revisions?job=libfuzzer_chrome_ubsan&range=395640:395746

Minimized Testcase (0.00 Kb): https://cluster-fuzz.appspot.com/download/AMIfv9631u2Fu7KPTDd5UZVNXTt5NOdUuMtZraPo4DZqGKjwl9IAgbWYdmAnlVOemtQSi0bA-bDSdmuzhcMW9HNUtrVdS_2w3seoPMX7BNBBBivjtQl4TlDJd3McRqXxtl9HvLHU-MMAked33_YZF6h2tTIl0Srsdw?testcase_id=6269761982038016

Filer: rnimmagadda

See https://chromium.googlesource.com/chromium/src/+/master/testing/libfuzzer/reproducing.md for more information.

Gentle Ping.

@riesa: Could you please provide some update on this issue.

Thank you.

I think my comments in #3 still stand. We are also replacing CLD2 with CLD3 actively, so I'm marking this as WontFix. If you disagree, let me know. 

Issue 624904 is the launch tracker for the new feature, details on the timeline are there.

ClusterFuzz has detected this issue as fixed in range 411637:411850.

Detailed report: https://cluster-fuzz.appspot.com/testcase?key=6269761982038016

Fuzzer: libfuzzer_language_detection_fuzzer
Job Type: libfuzzer_chrome_ubsan
Platform Id: linux

Crash Type: Undefined-shift
Crash Address: 
Crash State:
  CLD2::UTF8GenericPropertyBigOneByte
  CLD2::GetUniHits
  CLD2::ScoreCJKScriptSpan
  
Regressed: https://cluster-fuzz.appspot.com/revisions?job=libfuzzer_chrome_ubsan&range=395640:395746
Fixed: https://cluster-fuzz.appspot.com/revisions?job=libfuzzer_chrome_ubsan&range=411637:411850

Minimized Testcase (0.00 Kb): https://cluster-fuzz.appspot.com/download/AMIfv9631u2Fu7KPTDd5UZVNXTt5NOdUuMtZraPo4DZqGKjwl9IAgbWYdmAnlVOemtQSi0bA-bDSdmuzhcMW9HNUtrVdS_2w3seoPMX7BNBBBivjtQl4TlDJd3McRqXxtl9HvLHU-MMAked33_YZF6h2tTIl0Srsdw?testcase_id=6269761982038016

See https://chromium.googlesource.com/chromium/src/+/master/testing/libfuzzer/reproducing.md for more information.

If you suspect that the result above is incorrect, try re-doing that job on the test case report page.

Removing EditIssue view restrictions from ClusterFuzz filed bugs. If you believe that this issue should still be restricted, please reapply the label.

For more details visit https://www.chromium.org/issue-tracking/autotriage - Your friendly Sheriffbot