New issue
Advanced search Search tips
Starred by 4 users

Issue metadata

Status: Released
Owner: ----
Closed: May 2015



Sign in to add a comment

Submit is very slow with 2.11 compared to 2.10

Reported by org...@gmail.com, May 12 2015

Issue description

Affected Version: 2.11

What steps will reproduce the problem?
1. Submit a change

What is the expected output? What do you see instead?
With 2.9 on the same server and settings it used to take 1-2 seconds.

With 2.11 it takes about 6-8 seconds.

jstack trace with 2.11-gce6c4bc7: https://gist.github.com/orgads/942671d3e2a9ee11ac15

See discussion on https://gerrit-review.googlesource.com/67653
 
Project Member

Comment 1 by edwin.ke...@gmail.com, May 12 2015

Which submit type is used? Is it slow for all submit types?

Comment 2 by org...@gmail.com, May 12 2015

Cherry-Pick. I don't use other submit types, and can't change it for this project (other projects are not as slow).
Project Member

Comment 3 by david.os...@gmail.com, May 12 2015

Title of this issue is "compared to 2.10". In description you are saying:

"With 2.9 on the same server and settings it used to take 1-2 seconds."

Could you test the submit operation for the same project and setting on recently released 2.10.4 and post the result?

Comment 4 by org...@gmail.com, May 12 2015

Is it possible to downgrade gerrit? How?
Project Member

Comment 5 by david.os...@gmail.com, May 12 2015

How many open changes do you have on the branch your are submitting to? Can you close all open changes and test submit performance again?

Comment 6 by org...@gmail.com, May 12 2015

I have 116 open changes. Closing all the changes fixes performance. I can try running 2.10 on my test server.
Project Member

Comment 7 by david.os...@gmail.com, May 12 2015

>Is it possible to downgrade gerrit? How?

This would be hard.

* 2.10: schema version 98
* 2.11: schema version 107

You would need to undo these version upgrades. Don't you have some backups from your 2.9 or 2.10 production databases, so that you can restore this in test environment?

Comment 8 by org...@gmail.com, May 12 2015

I have a backup of the DB from 2.9. I'll use it on my test environment.

Comment 9 by org...@gmail.com, May 12 2015

By the way. On the branch I tested against there are only 11 open changes. I doubt it only reindexes this branch.

Comment 10 by org...@gmail.com, May 12 2015

Works much better with 2.10 (back to ~2 seconds).
Project Member

Comment 11 by ziv...@gmail.com, May 12 2015

Hmm... here you say that it works much better with 2.10. Which 2.10? 2.10.4?
Do we have the same issue in 2.10?
Is the  issue 3300  still relevant?

Comment 12 by org...@gmail.com, May 12 2015

2.10 => origin/stable-2.10 (that is v2.10.4).

> Is the   issue 3300   still relevant?

Well, I opened it against 2.11, not 2.10 :) It was changed later.
Project Member

Comment 13 by ziv...@gmail.com, May 12 2015

OK then 3300 can be closed (again).
Project Member

Comment 14 by david.os...@gmail.com, May 12 2015

> Works much better with 2.10 (back to ~2 seconds)

Like I said in my comment in [1] I suspect that's because of reindexing of all open changes during push. That's the only difference between 2.10 and 2.11. JGit version in 2.10.4 and in stable-2.11 is exactly the same: 4.0.0.201505050340-m2. Can you patch the 2.11 and disable Git ReferenceUpdatedListener and report if this solved your performance problem on 2.11?

Here is the diff:

diff --git a/gerrit-server/src/main/java/com/google/gerrit/server/config/GerritGlobalModule.java b/gerrit-server/src/main/java/com/google/gerrit/server/config/GerritGlobalModule.java
index 6ad264b..5bd86cc 100644
--- a/gerrit-server/src/main/java/com/google/gerrit/server/config/GerritGlobalModule.java
+++ b/gerrit-server/src/main/java/com/google/gerrit/server/config/GerritGlobalModule.java
@@ -255,7 +255,7 @@ public class GerritGlobalModule extends FactoryModule {
     DynamicSet.setOf(binder(), ProjectDeletedListener.class);
     DynamicSet.setOf(binder(), HeadUpdatedListener.class);
     DynamicSet.setOf(binder(), UsageDataPublishedListener.class);
-    DynamicSet.bind(binder(), GitReferenceUpdatedListener.class).to(ReindexAfterUpdate.class);
+    //DynamicSet.bind(binder(), GitReferenceUpdatedListener.class).to(ReindexAfterUpdate.class);
     DynamicSet.bind(binder(), GitReferenceUpdatedListener.class)
         .to(ProjectConfigEntry.UpdateChecker.class);
     DynamicSet.setOf(binder(), EventListener.class);

Interessting enough, with the diff applied all unit tests are still passing.

Project Member

Comment 15 by david.os...@gmail.com, May 12 2015

Status: AwaitingInformation
OK just checked the code:

ListeningExecutorService getBatchIndexExecutor(
      @IndexExecutor(INTERACTIVE) ListeningExecutorService interactive,
      @GerritServerConfig Config config,
      WorkQueue workQueue) {
    if (batchExecutor != null) {
      return batchExecutor;
    }
    int threads = config.getInt("index", null, "batchThreads", 0);
    if (threads <= 0) {
      threads = config.getInt("changeMerge", null, "threadPoolSize", 0);
    }
    if (threads <= 0) {
      return interactive;
    }
    return MoreExecutors.listeningDecorator(
        workQueue.createQueue(threads, "Index-Batch"));
  }

The problem is probably, that both:

* index.batchThreads
* changeMerge.threadPoolSize

aren't set on your site, in which case the reindexing would happen interactively. In your case as you reported, you have 116 open changes on this branch. 6-8 sec. to reindex 116 changes is fast.

Just set index.batchThreads (Note that changeMerge.threadPoolSize is deprecated) to some sane value and you should be OK.

We should probably warn in ReleaseNotes, that since 2.11 this setting is critical, otherwise the user would see dramatic performance degradation on push operation. And this degradation is linear to the number of open changes on the branch.

Comment 16 by org...@gmail.com, May 13 2015

Thanks! Now it's down to ~4 seconds, which is much better (in the original post it was 6-8, recently it became closer to 15 seconds).

I suggest setting a sensible default (2? auto-detection by number of processors?) in addition to adding this information to the release notes.
Project Member

Comment 17 by david.os...@gmail.com, May 13 2015

Status: ChangeUnderReview
https://gerrit-review.googlesource.com/67861
Project Member

Comment 18 by david.os...@gmail.com, May 13 2015

>>Is it possible to downgrade gerrit? How?
>
>This would be hard.
>
>* 2.10: schema version 98
>* 2.11: schema version 107
>
>You would need to undo these version upgrades.

Turn out, that OpenStack guys already did that and downgraded from 2.10 t0 2.8:

https://review.openstack.org/#/c/182113/2/tools/gerrit-rollback-2.10-2.8.sql
Labels: FixedIn-2.11.1
Status: Submitted
Status: Released

Sign in to add a comment