+CrashReporting, since this *may* be related to the recent spike of No Files To Process reports from ChromeOS (issue 680342) - perhaps someone from Breakpad can confirm whether low-memory would b0rk crash uploads?

Crash IDs from this morning's instance of this crash-out (I think).

Crash ID Chrome (Server ID: 6da71fb580000000)
Crash report uploaded on Friday, 24 February 2017 at 11:58:17

Crash ID Chrome (Server ID: 76fcef2d40000000)
Crash report uploaded on Friday, 24 February 2017 at 11:48:58

Crash ID Chrome (Server ID: d786423ec0000000)
Crash report uploaded on Friday, 24 February 2017 at 11:41:34

Crash ID Chrome (Server ID: bc376c3ec0000000)
Crash report uploaded on Friday, 24 February 2017 at 11:33:39

Crash ID Chrome (Server ID: 3534a03300000000)
Crash report uploaded on Friday, 24 February 2017 at 11:27:48

Crash ID Chrome (Server ID: 3d25efb580000000)
Crash report uploaded on Friday, 24 February 2017 at 11:25:45

Just spotting the Browser process suddenly ballooning again (see first screenshot).  On a hunch, I closed my Play Music tab, and Chrome Remote Desktop, and Browser memory immediately recovered to normal (see second screenshot).

I've hit this issue five times so far today; we shouldn't promote this Dev version to Beta, so RB-Beta.

Deleted: Screenshot 2017-02-24 at 15.36.50 - Display 2.png 26.8 KB

	Screenshot 2017-02-24 at 15.36.50 - Display 2.png 26.8 KB View Download

Deleted: Screenshot 2017-02-24 at 15.38.37 - Display 2.png 29.2 KB

	Screenshot 2017-02-24 at 15.38.37 - Display 2.png 29.2 KB View Download

This happened again just now, with Chrome Remote Desktop and YouTube Music both active, but this time I closed both and Chrome still balloon (this time to ~5GB of RAM, which is pretty impressive on a 4GB device ;)

Adding some folks in the hope of getting pro-tips on where to look for clues as to what is going wrong... (so far this has happened ~10 times today)

Is this same as  issue 689678 ? Can you check that the fix is in your version?

Re #5: Definitely not, since this is on ChromeOS, not Windows.

Possibly a dupe of issue 690517 (related to  issue 533648 , which I believe is a browser memory leak in IndexedDB).

steel@, WDYT? 

I have no context on this issue past decoding the stack on one of the crash reports :)
Adding a few more people who might know more.

Re #9: OK; I'm speculatively de-duping to issue 690517 since the issue I was seeing definitely sounds like that. :)

Issue 690517 is currently restricted, so re-opening this for public tracking of this issue.

Issue 690517 has been merged into this issue.

A few notes from the other places (b/35647846 and crbug.com/690517) we've talked about this:

---

1. How this reproduces is different depending on your particular machine

Remember that:

  * OOMs or "low memory notify" messages happen when physical memory is running out.
  * "Out of memory" error message in Chrome logs happens when Chrome's virtual address space is running out.  
    On a 32-bit machine you theoretically have 4GB of "virtual" address space, but in reality some of this 
    virtual address space is used for other things, so you really get between 2.5GB and 3GB.

Concrete examples:

  2  GB physical memory, 32-bit userspace: Lots of OOMs expected, maybe sometimes "Out of memory" in Chrome 
  4  GB physical memory, 32-bit userspace: Not too many OOMs, lots of "Out of memory" in Chrome
  4+ GB physical memory, 64-bit userspace: Lots of OOMs expected, no "Out of memory" in Chrome at all.


It is unknknown if this bug reproduces on 64-bit systems with really large physical memory.

---

2. Indications are that this isn't limited to Chrome OS

There were indications in other bugs that this was not limited to Chrome OS, but in Mac / Windows the bug is masked by the fact that we have swap to the hard disk.  We'll just chew through that instead of getting an out of memory.

---

3. Problem happens even on R55

We have reports of this problem with people on a Chromebook Flip on R55 stable channel.  It's not a new R56 issue.

---

4. We know that crashes aren't symbolized very well on crash servers.

There was a bug making these things very hard to look at on crash servers.  I believe it is fixed on newer Chrome OS versions.

Reference: https://chromium-review.googlesource.com/c/442284/

---

5. Symbolized stack trace looked like this on 3/3 systems (32-bit userspace where the browser aborted due to "out of memory") I looked at:

#3  0xb397905e in base::debug::BreakDebugger() () at ../../../../../../../home/chrome-bot/chrome_root/src/base/debug/debugger_posix.cc:249
#4  0xb3985a9a in logging::LogMessage::~LogMessage() () at ../../../../../../../home/chrome-bot/chrome_root/src/base/logging.cc:748
#5  0xb3999318 in base::(anonymous namespace)::OnNoMemorySize(unsigned int) [clone .constprop.4] () at ../../../../../../../home/chrome-bot/chrome_root/src/base/process/memory_linux.cc:35
#6  0xb2627b76 in operator new(unsigned int, std::nothrow_t const&) () at ../../../../../../../home/chrome-bot/chrome_root/src/base/allocator/allocator_shim.cc:80
#7  0xb440f914 in leveldb::ReadBlock(leveldb::RandomAccessFile*, leveldb::ReadOptions const&, leveldb::BlockHandle const&, leveldb::BlockContents*) () at ../../../../../../../home/chrome-bot/chrome_root/src/third_party/leveldatabase/src/table/format.cc:124
#8  0xb440ad86 in leveldb::Table::BlockReader(void*, leveldb::ReadOptions const&, leveldb::Slice const&) () at ../../../../../../../home/chrome-bot/chrome_root/src/third_party/leveldatabase/src/table/table.cc:189
#9  0xb440c15a in leveldb::(anonymous namespace)::TwoLevelIterator::InitDataBlock() () at ../../../../../../../home/chrome-bot/chrome_root/src/third_party/leveldatabase/src/table/two_level_iterator.cc:165
#10 0xb440c356 in leveldb::(anonymous namespace)::TwoLevelIterator::Seek(leveldb::Slice const&) () at ../../../../../../../home/chrome-bot/chrome_root/src/third_party/leveldatabase/src/table/two_level_iterator.cc:93
#11 0xb440a894 in leveldb::IteratorWrapper::Seek(leveldb::Slice const&) () at ../../../../../../../home/chrome-bot/chrome_root/src/third_party/leveldatabase/src/table/iterator_wrapper.h:47
#12 0xb440a8d2 in leveldb::(anonymous namespace)::MergingIterator::Seek(leveldb::Slice const&) () at ../../../../../../../home/chrome-bot/chrome_root/src/third_party/leveldatabase/src/table/merger.cc:53
#13 0xb43ff59a in leveldb::(anonymous namespace)::DBIter::Seek(leveldb::Slice const&) () at ../../../../../../../home/chrome-bot/chrome_root/src/third_party/leveldatabase/src/db/db_iter.cc:280
#14 0xb3145d32 in content::LevelDBIteratorImpl::Seek(base::BasicStringPiece<std::string> const&) () at ../../../../../../../home/chrome-bot/chrome_root/src/content/browser/indexed_db/leveldb/leveldb_iterator_impl.cc:45
#15 0xb3146756 in content::LevelDBTransaction::TransactionIterator::Seek(base::BasicStringPiece<std::string> const&) () at ../../../../../../../home/chrome-bot/chrome_root/src/content/browser/indexed_db/leveldb/leveldb_transaction.cc:230
#16 0xb31184ba in content::IndexedDBBackingStore::Cursor::FirstSeek(leveldb::Status*) () at ../../../../../../../home/chrome-bot/chrome_root/src/content/browser/indexed_db/indexed_db_backing_store.cc:3181
#17 0xb311c830 in content::IndexedDBBackingStore::OpenObjectStoreCursor(content::IndexedDBBackingStore::Transaction*, long long, long long, content::IndexedDBKeyRange const&, blink::WebIDBCursorDirection, leveldb::Status*) () at ../../../../../../../home/chrome-bot/chrome_root/src/content/browser/indexed_db/indexed_db_backing_store.cc:3976
#18 0xb312e136 in content::IndexedDBDatabase::OpenCursorOperation(std::unique_ptr<content::IndexedDBDatabase::OpenCursorOperationParams, std::default_delete<content::IndexedDBDatabase::OpenCursorOperationParams> >, content::IndexedDBTransaction*) () at ../../../../../../../home/chrome-bot/chrome_root/src/content/browser/indexed_db/indexed_db_database.cc:1673
#19 0xb312cf06 in base::internal::Invoker<base::internal::BindState<void (content::IndexedDBDatabase::*)(std::unique_ptr<content::IndexedDBDatabase::OpenCursorOperationParams, std::default_delete<content::IndexedDBDatabase::OpenCursorOperationParams> >, content::IndexedDBTransaction*), scoped_refptr<content::IndexedDBDatabase>, base::internal::PassedWrapper<std::unique_ptr<content::IndexedDBDatabase::OpenCursorOperationParams, std::default_delete<content::IndexedDBDatabase::OpenCursorOperationParams> > > >, void (content::IndexedDBTransaction*)>::Run(base::internal::BindStateBase*, content::IndexedDBTransaction*&&) () at ../../../../../../../home/chrome-bot/chrome_root/src/base/bind_internal.h:214
#20 0xb3143904 in content::IndexedDBTransaction::ProcessTaskQueue() () at ../../../../../../../home/chrome-bot/chrome_root/src/base/callback.h:64
#21 0xb27af4d2 in base::debug::TaskAnnotator::RunTask(char const*, base::PendingTask const&) () at ../../../../../../../home/chrome-bot/chrome_root/src/base/callback.h:64
#22 0xb27a4c58 in base::MessageLoop::DoWork() () at ../../../../../../../home/chrome-bot/chrome_root/src/base/message_loop/message_loop.cc:405
#23 0xb27a516a in base::MessagePumpDefault::Run(base::MessagePump::Delegate*) () at ../../../../../../../home/chrome-bot/chrome_root/src/base/message_loop/message_pump_default.cc:35
#24 0xb399d5ce in base::RunLoop::Run() () at ../../../../../../../home/chrome-bot/chrome_root/src/base/run_loop.cc:35
#25 0xb39b0cce in base::Thread::ThreadMain() () at ../../../../../../../home/chrome-bot/chrome_root/src/base/threading/thread.cc:333
#26 0xb39aee32 in base::(anonymous namespace)::ThreadFunc(void*) () at ../../../../../../../home/chrome-bot/chrome_root/src/base/threading/platform_thread_posix.cc:71

---

6. It wasn't one big allocation.

The bit of code that was doing the allocation was this allocation:

      char* ubuf = new char[ulength]; 
      if (!port::Snappy_Uncompress(data, n, ubuf)) { 

Presuming my assembly level debugging skills were good enough, I found:

In two of the crashes I analyzed (both from the same system, which was continually crashing):

ulength: 0x00542a57
n:       0x001092df

In the other crash I analyzed (from a different system):

ulength: 0x0053d614
n:       0x0011215e


The above allocation was about 5MB, so it wasn't that we were doing one giant allocation.

---

7. Tracing is hard to gather

The crash happens so fast that it's hard to gather a trace and save it off.

Someone suggested "trace on tap" as a fast way to gather a trace:
  https://chrome.google.com/webstore/a/google.com/detail/traceontap/amlnjoelabmgbmhkgejofjdchfcfogdg

Last I tried it my system crashed right away, but YMMV.


In theory, it might be easier to try to grab a trace on Mac or Windows if you can reproduce there, since they (presumably) will just eat up lots of disk space for swap and won't die quite as hard.

---

8. You can catch the problem by using "top"

You can see when your system is about to crash by watching "top".  You'll see the browser process growing in memory usage at an incredible rate, then you go boom.

There's a video of this on b/35647846, comment 48

---

9. Unknown how to get into this state, but when you're in this state you crash a lot

* People seem to get into this state and they can reproduce every 5-15 minutes, even across machine reboots.

* It's unclear how to get into this state.

* In at least one case, someone seemed to get out of this state eventually, even without turning "offline docs sync" off.

---

10. Nobody has managed to repro with "offline docs sync" turned off

---

11. It seems like we need some better tools for this.

Filed bug #692592 with some ideas for how we can catch these leaks more programatically.  Needs an owner.

 Issue 693299  has been merged into this issue.

 Issue 697606  has been merged into this issue.

 Issue 696853  has been merged into this issue.

kathrelkeld@ -

From Dianders@ explanation above:

3. Problem happens even on R55

We have reports of this problem with people on a Chromebook Flip on R55 stable channel.  It's not a new R56 issue.

So doesn't look like a regression. I am thinking this shouldn't be RBB in that case unless you are seeing this repro on M57 consistently.

Right, it seems hard to justify a release block for this, but easy to justify a high Priority

Please see  crbug.com/696853  - I was able to repro on 57

@25: Yup, it certainly _affects_ R57, but it is not a regression that is new for R57.

...so while this is a very high priority to get fixed, it would be pointless to hold up releases of R56 and R57 waiting for a fix.  The problem is not new.

Do we have a sense of relative frequency of occurrences per milestone?  Personally, I didn't notice this until M57 - and my corp machine is unusable right now.

+Olga for conops insight

For anyone able to reproduce, when you see the memory footprint shoot up, could you go to docs.google.com/offline/debug/console and capture the output?
Also, chrome team is looking for IndexedDB traces to help investigate: See b/35795717#comment4

@27: Yup.  Your experience matches everyone else who has run into this where suddenly a machine is affected and it becomes unusable.  Because of the sudden occurrence, the problem always feels like it is a "new" thing, but whatever causes it doesn't seem to be related to a given version of the OS.

+1 to #27. I never saw it in 56. All of my repro's were on 57. 
- I think metrics to support that "this is not a new regression in 57" is missing.

@30:

I suppose one could assert that it somehow became more common in R57, but IMHO it is more likely it became more common due to some other external change (like a change in Google Docs).  At a given date these issues started spiking, but you can certainly see in bug #690517 that clearly we had a M55 report of this (beyond any doubt it was R55, since it was a googler reporting it and we worked closely with him).  Also beyond a doubt we have had many instances of this on R56 on Kevin, Caroline, and other machines.  The original report of this bug that I'm aware of (http://crosbug.com/p/62731) was on R56 kevin.

There may be no official metrics here, but definitely the bug reproduces across all 3 versions.  Also, if there were some metrics but the bug was induced by an external change (like a Change in the Google Drive Extension), then it would like like the bug didn't exist in old versions of the browser because nobody ran the old browser with the new version of the extension.

Hmmm; we likely won't see this issue reflected in LevelDB crash report signatures, since the OOMing often seems to prevent a crash report being captured and uploaded, but have we seen a recent spike in OOM crashes reported via UMA?

Looking at crash reports, I notice that the "No files to process" signature has graphs showing it peaking in M54->M55.0.2883.87 (Stable), M56.0.2924.76 (Beta) and M57.0.2987.13 (Dev) - that could support the idea of an external change around late November / early-mid December.

Issue 696555 has been merged into this issue.

Issue 695581 has been merged into this issue.

Issue 696711 has been merged into this issue.

+oshima for another set of eyes

See also b/35927616

per dianders@ in #26 above moving this to M58.

Is @cmumford still the right owner ? Don't see any action from him yet. (just being paranoid as this was a terrible experience for me and I don't want anyone to see this).

Update: this is still being worked by multiple people, and considered the top priority by the storage team. Evidence suggests this is not a regression, but a latent bug triggered by a server side change to offline docs sync. Sorry, nothing more definitive to report at present.

Issue 694891 has been merged into this issue.

Any news to share?

We have a document, that when sync'ed can trigger this, but reproducible is somewhat low. Trying to refine the repro steps to facilitate debugging.

Look like problem still exists in M58. Feedback from sauquet@'s  Caroline:

CHROME VERSION=58.0.3027.0 dev
CHROMEOS_ARC_ANDROID_SDK_VERSION=25
CHROMEOS_ARC_VERSION=3775402
CHROMEOS_RELEASE_VERSION=9331.0.0

<4>[ 2036.217034] IndexedDB invoked oom-killer: gfp_mask=0x280da, order=0, oom_score_adj=-1000

Issue 700575 has been merged into this issue.

Browser memory leak? GPU does not appear to be involved, nor offline docs sync.

Tabs open at around 29,000K and immediately start climbing in memory usage. Through the course of a few hours the tabs use more and more memory, climbing through 200,000K and then 300,000K. Kept task manager open. Machine froze, eventually "blacked out" and recovered - just as the machine has done and similar to reports above. Filed automated report with this bug number listed in the text. Grabbed photo of frozen task manager on screen: largest task was using 442,816K. I do not know if this is the same bug the team is chasing, but this looks like what I know as a memory leak. I had previously removed the offline docs extension. GPU never budged from 94,000K. This is a browser memory leak, not a misbehaving extension, no? 

Specs of machine on which blackout/recovery occurred:
LG ChromeBase 22CV241 Intel Celeron 2955U 1.4 GHz with 2 GB DDR3L SDRAM, 16 GB Solid-State Drive. Version 56.0.2924.110 (64-bit) Google_Monroe.4921.17.0

Deleted: IMG_20170312_214950005.jpg 821 KB

	IMG_20170312_214950005.jpg 821 KB View Download

Re #5- This bug is specifically about a multi-GB Browser process (the entry roughly in the middle of your screenshot, which shows 233MB usage by that process, which is actually fairly normal).

Renderer memory usage is tracked by other bugs, but I'd suggest filing a new bug with a summary of any specific steps (e.g. how long is the system up for, how long were the bloated pages actually open, which extensions do you use) to see the issue.  Thanks!

cmumford@ I understand you're heads down on this, but is there any way we can get more frequent updates. This issue is absolutely devastating to Chrome OS users - it's the #1 issue we're facing at the moment.

Can we get a copy of the document mentioned in #47 so we can investigate in parallel or in some other way help in the investigation.

We've identified that switching from iterator continue/update to store get/put works around this bug - at least in our test page that causes a CrOS crash. Docs has a change to switch to get/put ready to roll out. Right now they are examining the results of the IDB team's findings before deployment.

Note, this is not yet a fix to Chrome, so only a workaround.

Thanks. A workaround on the docs side would be fabulous.

[b/35927616 seems to be the internal bug on this where some work is happening]

#CBC-RS/TC-watchlist 

Thanks Royans, looks good. Anything we can formally share broader?

since cmumford@ is out for a bit, passing on to dmurph@

(but do watch the internal bug for more timely updates)

Issue 699908 has been merged into this issue.

 Issue 693129  has been merged into this issue.

Ah! Who can repro this? Is this still happening?

If so, while it's happening can you go to about://tracing and record a trace?

1. about://tracing
2. uncheck everything (including system profiling) and check IndexedDB and leveldb
3. click record
4. wait until you're close to a crash
5. click 'stop'
6. (wait for it to process - so maybe give it a bit of time before the crash)
7. click 'save' and save it before the crash

I had 4 crashes with this stack on my Windows box today. (I had 0 docs tabs open, but I do have the docs offline extension installed.) Crash IDs from today: 394b930e60000000 1c2b8f9480000000 ae76d0f660000000 f8dfba6640000000

It's very unpredictable, so there's no "while it's happening" or "close to a crash" really. Chrome just disappears from under me every now and then.

Updates from internal b/35927616: The Docs workaround fix is currently in dogfood. ETA for reaching prod is still Wed 3/22.

Is it expected that this happens even if no docs tabs are open?

Yes, the bug is manifesting when syncing is happening in the background via the extension.

A customer is reporting that they are occasionally still experiencing crashing with the docs offline extension removed.
Please advise if there is some the customer can gather to assist.

Device log: https://drive.google.com/a/google.com/file/d/0B9a1yrkxazgMbVlVM0RsQVpadFU/view?usp=sharing 

@gbirtchnell, I couldn't find crash IDs in the attached log. Can you collect latest Crash IDs from after disabling Docs Offline? It may be a different issue.

FYI we have seen this occur multiple times on the ChromeOS platform where it triggers a kernel panic and device crash.

More info/fbreports/logs in the bug here is useful:  https://buganizer.corp.google.com/issues/36532260

Updates from internal b/35927616: Docs workaround blocked on the prod push of Explorer. Once Explorer is pushed we can start to rollout the flag.

Issue 704435 has been merged into this issue.

we are GSuite Business customer. For several weeks multiple users have been reporting regular crashes in Chrome. Clearing Chrome cache and recreating Chrome user profile has had some success. I recreated user profile and the crashes stopped for about 2 weeks and have now restarted. Four incidents yesterday morning where Chrome just closed down, was able to Restore, have reported each incident via crash reports. Switched off Google Docs Offline and Google Docs extensions yesterday and crashes have stopped for now, unclear if other affected users have same extensions. Last crash report yesterday at 13.48 - b6faf47c-d5a2-4335-8c7e-aeb270536b1f

@75: Sorry for your troubles.  I think we're nearly at the end of this one.  As I understand it production rollout has started, so very soon now everyone should have the fix.

Workaround fix is now rolled out to 10%. ETA for full rollout is Monday (3/27).

 Issue 661768  has been merged into this issue.

Issue 693142 has been merged into this issue.

We have over 1800 non-touch devices with Intel chips in our schools. None of these devices have experienced this problem. 

We have 45 touch devices with ARM processors. Some of these devices have experienced problems. To date, the only solution is to disable the Google Docs offline sync extension. 

 Issue 700357  has been merged into this issue.

Updates from internal b/35927616: Docs workaround now rolled out to 100%. ETA for it to reach all clients is 7:30 PM ET.

Thanks Ralph, we've asked any users who see frequent crashes after 8pm ET to capture logs and Crash IDs. Any other data you'd like to see from them? (fingers crossed that won't be necessary)

One piece of logs that might be useful to us is:
1) Visit docs.google.com
2) Open Devtools (Ctrl-Shift-I) -> Console
3) Paste & execute: localStorage.getItem('docs-tasksStats_default')
4) Capture the output

I don't have anything easier at the moment. We can also try looking at the logs from our side on a case by case basis, if they're still experiencing this.

Filed b/36655835 against bhthompson to split that off into a separate bug/discussion. thanks everyone.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/ec764054b12a8e21e1fe958c3f1e4d47df671066

commit ec764054b12a8e21e1fe958c3f1e4d47df671066
Author: dmurph <dmurph@chromium.org>
Date: Mon Mar 27 23:12:47 2017

[IndexedDB] Pool and evict leveldb iterators, to save memory

Caps the total number of open cursors per database. Ideally the cap
would be on the # of bytes held by the cursors, but this data is deeply
hidden, and this is good enough for an initial mitigation.

Note: This could potentially cause worse memory churn if the number of
cursors allowed is too low.

R=jsbell,pwnall,cmumford
BUG= 696055 , 702787 

Review-Url: https://codereview.chromium.org/2760163002
Cr-Commit-Position: refs/heads/master@{#459927}

[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/BUILD.gn
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/indexed_db_backing_store.cc
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/indexed_db_backing_store_unittest.cc
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/indexed_db_class_factory.cc
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/indexed_db_class_factory.h
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/indexed_db_cleanup_on_io_error_unittest.cc
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/leveldb/leveldb_database.cc
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/leveldb/leveldb_database.h
[add] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/leveldb/leveldb_iterator.cc
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/leveldb/leveldb_iterator.h
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/leveldb/leveldb_iterator_impl.cc
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/leveldb/leveldb_iterator_impl.h
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/leveldb/leveldb_transaction.cc
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/leveldb/leveldb_transaction.h
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/leveldb/leveldb_transaction_unittest.cc
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/leveldb/leveldb_unittest.cc
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/mock_browsertest_indexed_db_class_factory.cc
[modify] https://crrev.com/ec764054b12a8e21e1fe958c3f1e4d47df671066/content/browser/indexed_db/mock_browsertest_indexed_db_class_factory.h

It looks like crash numbers are way down since the Docs mitigation push on Monday evening:

https://crash.corp.google.com/browse?q=OMIT%20RECORD%20IF%20SUM%28CrashedStackTrace.StackFrame.FunctionName%3D%27content%3A%3AIndexedDBDatabase%3A%3AOpenCursorOperation%28std%3A%3Aunique_ptr%3Ccontent%3A%3AIndexedDBDatabase%3A%3AOpenCursorOperationParams%2Cstd%3A%3Adefault_delete%3Ccontent%3A%3AIndexedDBDatabase%3A%3AOpenCursorOperationParams%3E%20%3E%2Ccontent%3A%3AIndexedDBTransaction%20*%29%27%29%20%3D%200&ignore_case=false&enable_rewrite=true&omit_field_name=CrashedStackTrace.StackFrame.FunctionName&omit_field_value=content%3A%3AIndexedDBDatabase%3A%3AOpenCursorOperation%28std%3A%3Aunique_ptr%3Ccontent%3A%3AIndexedDBDatabase%3A%3AOpenCursorOperationParams%2Cstd%3A%3Adefault_delete%3Ccontent%3A%3AIndexedDBDatabase%3A%3AOpenCursorOperationParams%3E%20%3E%2Ccontent%3A%3AIndexedDBTransaction%20*%29&omit_field_opt=%3D#samplereports:5,day:30

I'm going to switch this to Fixed (also because of our own patch above), and start on a postmortem.

For clarification, push was on 2017/03/27 at 8pm, and the percent went from 0.58% on Monday to 0.16% on Tuesday.

Enterprise customer has confirmed no crashes this week so I think we're all set here.

We are not seeing this as fixed with the patch in OS 56.  Opened a case with Google today and they recommended going to OS 57 and see if that helps.  Our dilemma is that our state testing in KS will not support 57.  They do not guarantee that it will work with 56, but they will try to help if we run into issues.  Why is the patch possibly not working?  We saw no issues until last Friday.

@ddellere: The fix for this was rolled out as Drive offline update (not chromeos update). We will find the case you created and investigate your issue as a separate issue.

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/1f18ca937688b4e0ee570b4d506f5ac21ead5b93

commit 1f18ca937688b4e0ee570b4d506f5ac21ead5b93
Author: dmurph <dmurph@chromium.org>
Date: Tue Apr 11 21:51:08 2017

[IndexedDB] Closing mojo connections when renderer exits.

Changes IndexedDBDispatcherHost to be owned by a unique_ptr instead of
ref counted, and makes the mojo connections for the database and the
cursor interfaces strongly stored (owned) by the dispatcher host.

The dispatcher host now supports weak pointers, which are used by the
IndexedDB.*Callbacks (still refcounted, and probably by necessity) to
ignore responses when the host is gone. The host also observes renderer
exit - which can happen well before the render process host is
destroyed - to destruct the bindings and clear the weak pointers.

Finally, minor cleanup includes minimizing dependencies, adding the
connection error listener on IndexedDBDatabaseCallbacks to cut that off
as early as possible, and formally create the IDBHelper on the
dispatcher host to make the idb thread separation cleaner.

BUG= 696055 

Review-Url: https://codereview.chromium.org/2727733004
Cr-Commit-Position: refs/heads/master@{#463786}

[modify] https://crrev.com/1f18ca937688b4e0ee570b4d506f5ac21ead5b93/content/browser/indexed_db/cursor_impl.cc
[modify] https://crrev.com/1f18ca937688b4e0ee570b4d506f5ac21ead5b93/content/browser/indexed_db/cursor_impl.h
[modify] https://crrev.com/1f18ca937688b4e0ee570b4d506f5ac21ead5b93/content/browser/indexed_db/database_impl.cc
[modify] https://crrev.com/1f18ca937688b4e0ee570b4d506f5ac21ead5b93/content/browser/indexed_db/database_impl.h
[modify] https://crrev.com/1f18ca937688b4e0ee570b4d506f5ac21ead5b93/content/browser/indexed_db/indexed_db_callbacks.cc
[modify] https://crrev.com/1f18ca937688b4e0ee570b4d506f5ac21ead5b93/content/browser/indexed_db/indexed_db_callbacks.h
[modify] https://crrev.com/1f18ca937688b4e0ee570b4d506f5ac21ead5b93/content/browser/indexed_db/indexed_db_database_callbacks.cc
[modify] https://crrev.com/1f18ca937688b4e0ee570b4d506f5ac21ead5b93/content/browser/indexed_db/indexed_db_database_callbacks.h
[modify] https://crrev.com/1f18ca937688b4e0ee570b4d506f5ac21ead5b93/content/browser/indexed_db/indexed_db_database_unittest.cc
[modify] https://crrev.com/1f18ca937688b4e0ee570b4d506f5ac21ead5b93/content/browser/indexed_db/indexed_db_dispatcher_host.cc
[modify] https://crrev.com/1f18ca937688b4e0ee570b4d506f5ac21ead5b93/content/browser/indexed_db/indexed_db_dispatcher_host.h
[modify] https://crrev.com/1f18ca937688b4e0ee570b4d506f5ac21ead5b93/content/browser/indexed_db/indexed_db_factory_unittest.cc
[modify] https://crrev.com/1f18ca937688b4e0ee570b4d506f5ac21ead5b93/content/browser/indexed_db/indexed_db_unittest.cc
[modify] https://crrev.com/1f18ca937688b4e0ee570b4d506f5ac21ead5b93/content/browser/indexed_db/mock_indexed_db_callbacks.cc
[modify] https://crrev.com/1f18ca937688b4e0ee570b4d506f5ac21ead5b93/content/browser/indexed_db/mock_indexed_db_database_callbacks.cc
[modify] https://crrev.com/1f18ca937688b4e0ee570b4d506f5ac21ead5b93/content/browser/renderer_host/render_process_host_impl.cc
[modify] https://crrev.com/1f18ca937688b4e0ee570b4d506f5ac21ead5b93/content/browser/renderer_host/render_process_host_impl.h
[modify] https://crrev.com/1f18ca937688b4e0ee570b4d506f5ac21ead5b93/mojo/public/cpp/bindings/associated_binding.h
[add] https://crrev.com/1f18ca937688b4e0ee570b4d506f5ac21ead5b93/mojo/public/cpp/bindings/strong_associated_binding_set.h