Link to uploaded files: https://pantheon.corp.google.com/storage/browser/chromiumos-test-logs/bugfiles/cr/591871/?pli=1

Clarification in regards to video freeze on the second screen attached to guado, this happened sometime after the # of participants was dropped down to 15. I did not observe this in the second test that ran for 1hr.

Another observation - when there are 25 participants in the session, there is a noticeable lag in remote operation as well. It becomes glitchy with minor delay from when you press a button on the remote to when it reflects on the UI. This is not guado specific and is reproducible on Panther i7 as well.

In yet another (3rd) test hangout session with 25 participants, Guado celeron w/ kepler is now mostly showing 720p per HUD as seen in https://pantheon.corp.google.com/m/cloudstorage/b/chromiumos-test-logs/o/bugfiles/cr/591871/Guado-Celeron-25Participants.mp4 but as can be seen in the video there is still some pixelation and glitches.

Is this specific to this configuration of running dual monitors.
Can you please also confirm, that a single monitor set up is working ok?


Yung, who should this be assigned to ?

c#4 pixelation issue is tracked at issue 501915.  Not particular to single or dual monitor issue.

I do see some difference in video quality with 1 monitor vs 2. With only 1 monitor connected quality visually looks better and HUD also shows 720p (for all sender->reflector->receiver) but FPS still in mid teens. When I add 2nd monitor, HUD rarely gets better than 540p.

I am also consistently able to reproduce the 2 jabras not being in sync issue and something muting/unmuting from hangouts UI does not reflect on the speakers either.

Also consistently able to reproduce lag in remote operation issue with large number of participants, very noticeable on Guado.


I'll file separate bugs for jabra out of sync and remote operation lag.




Some additional cpu/memory usage data points from another long running hangout session:

Here is a graph from a 4hr run. The red sections on the chart are with 6 participants and yellow section is with 25 participants. 
https://pantheon.corp.google.com/m/cloudstorage/b/chromiumos-test-logs/o/bugfiles/cr/591871/Guado-Panther-4hr-run.jpg


Here is a graph from Guado showing the cpu/memory usages after the device was left in a hangout session with 6 participants for over 20hrs.
- Notice the memory usage is almost 100% though the CPU is mid 70's. 
https://pantheon.corp.google.com/m/cloudstorage/b/chromiumos-test-logs/o/bugfiles/cr/591871/Guado%20Celeron%20Kepler%20-%206%20participants%20session%20after%2020hr%20run%20time.png

Both Panther and Guado+Kepler seem to have some memory leak if you look at the memory graph from https://pantheon.corp.google.com/m/cloudstorage/b/chromiumos-test-logs/o/bugfiles/cr/591871/Guado-Panther-4hr-run.jpg

In 2 hours, the memory grew by 8%, so in 20 hours, I can see it can grow (4%/hr x 20 = 80%) into saturation.  Same can be said for the Panther but at much lower rate. (about 1.5%/hr rate)

Pawel, Ville-Mikko, is there a way to track memory leak?  After looking at the graphs, I think it's reasonable to assume there's some memory leak introduced by kepler (directly or indirectly)

Harpreet, can we have the same CPU/memory graph running the same OS version w/ Guado+Kepler but only with 1 screen?  I am curious if the memory leak is caused by adding the 2nd screen somehow or an independent issue.  Thanks!

I'll run the test with 1 screen overnight tonight with few participants and get you the results tomorrow morning.

Below link is for the graph showing 20min data sample collected after 20hrs of continuously running hangout session on guado with 6 participants. There definitely seems like memory leak as memory usage is close to 100% with only 6 participants.

https://pantheon.corp.google.com/m/cloudstorage/b/chromiumos-test-logs/o/bugfiles/cr/591871/Guado%20Celeron%20Kepler%20-%206%20participants%20session%20after%2020hr%20run%20time.png

To track available memory globally you can graph it just like harpreet@ did it. If there is leak, you should see amount of allocated memory climb up gradually. Once we see harpreet@'s result.

There is similar bug to this that was already present in M47: https://bugs.chromium.org/p/chromium/issues/detail?id=524568

There are also ways to instrument code with ASAN/valgrind but they are invasive in a sense that they cause everything to slow down quite a bit. Especially if it looks like this would only happen on large scale Hangouts, this will be very difficult to debug because Hangout tries to adapt when it realizes a host is high on CPU.

Here are the links to the graphs from another test run which lasted 17hrs with 6 participants in the session:

Panther i7: https://docs.google.com/a/google.com/spreadsheets/d/1hI9a5En0id31LGDvQd5lQ08kVaGQO6g2tOcrMTUwChs/pubchart?oid=538815746&format=interactive

Rikku Celeron Kepler: https://docs.google.com/a/google.com/spreadsheets/d/1hI9a5En0id31LGDvQd5lQ08kVaGQO6g2tOcrMTUwChs/pubchart?oid=1171912365&format=interactive

Guado Celeron Kepler: https://docs.google.com/a/google.com/spreadsheets/d/1hI9a5En0id31LGDvQd5lQ08kVaGQO6g2tOcrMTUwChs/pubchart?oid=1387492733&format=interactive

Clearly some memory leak on all the units but not as drastic as with larger number of participants. I am going to kick off another run with 25 participants in the session for the first 1hr and then drop the number to 6 participants (same as the test from last night) and let it run for few hours.



Also, no jabra mute/volume sync issues with smaller number (6) of hangout participants even after 17hr session.

Another run with same exact setup except for 1 monitor vs. 2.

With 1 Monitor - https://docs.google.com/a/google.com/spreadsheets/d/1hI9a5En0id31LGDvQd5lQ08kVaGQO6g2tOcrMTUwChs/pubchart?oid=556648709&format=interactive

With 2 Monitors - https://docs.google.com/a/google.com/spreadsheets/d/1hI9a5En0id31LGDvQd5lQ08kVaGQO6g2tOcrMTUwChs/pubchart?oid=1143998446&format=interactive


From this it is pretty clear that 2 monitors do cause more memory leak.



More data tomorrow morning.

Summary from extensive performance testing over the last few days with large room setup using guado/rikku celeron w/ kepler and panther i7 devices:

- Memory leak is a non-issue; what we observed in the initial graphs was due to HUD being turned on

- Pixelation with large number of participants; also observed on panther i7 with 25 participant hangout session - tracking bug: https://bugs.chromium.org/p/chromium/issues/detail?id=501915

- USB peripherals stop working after some time (usually within few minutes of joining large hangout session) when connected via lindy USB extension's which are sold as part of the large room bundle - tracking bug: https://bugs.chromium.org/p/chromium/issues/detail?id=593187

- 2 jabra's don't stay in sync with mute/volume up/down operations soon after joining a large hangout session; this happens on Guado / Rikku celeron w/ kepler devices and not on panther i7. This only happens when the jabra's are connected via Lindy USB extensions (so could be related to the above issue) - tracking bug: https://bugs.chromium.org/p/chromium/issues/detail?id=593551

- Noticeable lag in remote operation with large number of participants, this is consistently reproducible on Guado celeron w/ kepler - https://bugs.chromium.org/p/chromium/issues/detail?id=593556



I'll make this bug blocked on all the bugs listed above.

Regarding the corruption issue, I thought webrtc team would try to help us.  But there were no firm commitment in the past.  Any update to this ?

Here is some more data with and without large 25 participant feature.

Interestingly the memory usage is higher overall with only 10 active participants (rest are avatars) in a 25 participant session when compared to all 25 being active for Guado and Rikku. CPU usage is few percentage points lower. Rest of the issues as mentioned in https://bugs.chromium.org/p/chromium/issues/detail?id=591871#c15 still apply.


25 Participants w/ only 10 active participants + 2 monitors
Guado - https://docs.google.com/a/google.com/spreadsheets/d/1hI9a5En0id31LGDvQd5lQ08kVaGQO6g2tOcrMTUwChs/pubchart?oid=886550911&format=interactive
Rikku - https://docs.google.com/a/google.com/spreadsheets/d/1hI9a5En0id31LGDvQd5lQ08kVaGQO6g2tOcrMTUwChs/pubchart?oid=1773536682&format=interactive
Panther - https://docs.google.com/a/google.com/spreadsheets/d/1hI9a5En0id31LGDvQd5lQ08kVaGQO6g2tOcrMTUwChs/pubchart?oid=1206110610&format=interactive


25 Participants w/ all 25 active + 2 monitors (ignore data after ~3:15pm mark - that is w/ HUD turned on)
Guado - https://docs.google.com/a/google.com/spreadsheets/d/1hI9a5En0id31LGDvQd5lQ08kVaGQO6g2tOcrMTUwChs/pubchart?oid=1901980672&format=interactive
Rikku - https://docs.google.com/a/google.com/spreadsheets/d/1hI9a5En0id31LGDvQd5lQ08kVaGQO6g2tOcrMTUwChs/pubchart?oid=327879705&format=interactive
Panther - https://docs.google.com/a/google.com/spreadsheets/d/1hI9a5En0id31LGDvQd5lQ08kVaGQO6g2tOcrMTUwChs/pubchart?oid=2044780626&format=interactive


Feedback reports from the 25 participant session with 10 active participants:
Guado: http://feedback/#/Report/6859532904
Rikku: http://feedback/#/Report/6859535328
Panther: http://feedback/#/Report/6859475636

Guado celeron with kepler is not used / sold as a CFM device. Closing.