Postponing inserting of merge commit into chromium/src to after the production freeze.

Also, see this https://groups.google.com/a/google.com/d/msg/chrome-infrastructure-team/D2Gk3A0RmV4/Gg0O-0qADQAJ 

The NextAction date has arrived: 2019-01-03

It looks like tandrii is working on this (per email thread). Assigning to him (if not, please send it back and we'll sync tomorrow :)).

Yeah, since I promised to do this, I'll handle it.

Hi Andrii,

May I confirm with you that after migration, the old codebase still works? Since we have release branch on the old codebase, just want to make sure that we could still use the old codebase if needed. Thanks.

The NextAction date has arrived: 2019-01-10

wtlee@ can you elaborate what "old codebase works" mean?

I can tell you that:
* old git repo will remain in place with all its history and branches
* ideally, we change ACLs such that old repo becomes read only.


Also, I'm sorry, but i'm behind on my Pri0..Pri1 tasks this week, and I'm not ready to allocate ~1 hour minutes for this work today (and tomorrow i'm OOO).
However, if wtlee@ or someone else is willing to prepare the commits to be merged, including merged commit, I can assist with changes of permissions. The plan has been already outlined in https://groups.google.com/a/google.com/d/msg/chrome-infrastructure-team/D2Gk3A0RmV4/6Az7zTlRDwAJ 

I can help with preparing the merge commit, do you mean you could grant the forge committer permission for me so that I could submit the merge commit as any other CL? Thanks.

You should be able to prepare this by pushing your merge commit to 

git push https://chromium.googlesource.com/chromium/src/ HEAD:refs/experimental/wtlee@google.com/<something>

(maybe s/google.com/chromium.org, depending on which account use use)

LMK how it goes.

It seems that I don't have permissions to push merge commit even on an experimental branch. Could you suggest?

The logs:

=======================================================================
To https://chromium.googlesource.com/chromium/src/
 ! [remote rejected]           HEAD -> refs/experimental/wtlee@chromium.org/ChromeCameraAppMigration (commit d918ba2: you are not allowed to upload merges)
error: failed to push some refs to 'https://chromium.googlesource.com/chromium/src/'
=======================================================================

Argh, acls were wrong all this time :( fixing... https://chromium-review.googlesource.com/c/chromium/src/+/1406569

Landed. Should work now, in theory...

Thanks, it seems succeed.

=======================================================================
To https://chromium.googlesource.com/chromium/src.git
 * [new branch]                HEAD -> refs/experimental/wtlee@chromium.org/ChromeCameraAppMigration
=======================================================================

But I cannot view it on "https://chromium.googlesource.com/chromium/src/+/refs/experimental/wtlee@chromium.org/ChromeCameraAppMigration"

Do I push the merge commit right?

Indeed, I don't see ref as is in gitiles, maybe some @ or smth else is specially treated. However, I can see it by hash:


which looks good, but two missing things in description:

1. Bug: 918091
2. Cr-Commit-Position: refs/heads/master@{#TBD}
 it's TBD because we'll need to manually fill it in right before landing based on previous commit number.
3. (optionally) -- maybe add more details about where repo used to be?

oops, forgot actual link to your merge commit: https://chromium.googlesource.com/chromium/src/+/3680a9db7fcf50545eced9cfe41d0b3700484946 

Got it, thanks. I'll change the commit message on Monday. And does that mean we need to change the commit message again to fill the {#TBD} part right before landing?

Yes, exactly right.

I just found that the branch I pushed does not contain the commit logs very well. So I tried a different way and push again, this time the logs seems to be well preserved. 

The new branch is: 
refs/experimental/wtlee@chromium.org/ChromeCameraAppMigrationV2

And the link is:
https://chromium.googlesource.com/chromium/src/+/3491b151a6b14d3b5e64e7df391b484bd6d85c2f/


However, I am not sure how to update the merge commit to make one of its parent becomes the latest commit in chromium. Could you help me to update it or could you teach me how to do that? Thanks!

Copy-pasting plan from email:

On flag day
1. Send PSA to chops-all@
2. Troopers heads up.
3. Close tree.
4. Massage commit position footer of merged commit.
5. Grant myself push merge access on refs/heads/master && refs/for/refs/heads/master, as well as force author/committer
6. Push.
7. Re-open tree.

Btw, this doesn't need to be internal.

Revoking submit ACLs + Granting myself push + forge + merge access: https://crrev.com/c/1410363 https://crrev.com/c/1410018 
disabling git numberer 
https://chromium-review.googlesource.com/c/chromium/src/+/1410083

Tree closed. ACLs landed.

Commit pushed as https://chromium.googlesource.com/chromium/src/+/a23510b0284104fd2ba2d87986503ac65ad7109c

ACLs reverted.
Tree opened.

As vadimsh@ pointed out, https://luci-scheduler.appspot.com/jobs/chromium/master-gitiles-trigger/9089737662828758816 has triggered many other builds, though in practice most builders merge triggers, so only last commit will count.

Still, something we didn't plan for. I keep bug opened till tomorrow to de-dup anything we might have broken today.

Verified. The codebase seems to be migrated successfully. Thanks a lot for your help :)

The following revision refers to this bug:
  https://chromium.googlesource.com/chromium/src.git/+/a23510b0284104fd2ba2d87986503ac65ad7109c

commit a23510b0284104fd2ba2d87986503ac65ad7109c
Author: Andrii Shyshkalov <tandrii@chromium.org>
Date: Tue Jan 15 00:56:27 2019

Migrate Chrome Camera App to chromium

This CL migrates the codebase of Chrome Camera App from "apps/camera" to
"chromium/src/chrome/browser/resources/chromeos/camera".

Bug: 918091
Cr-Commit-Position: refs/heads/master@{#622663}
Change-Id: Ib4f0ddbf93336e259abdef8972fed3bbde879205

There is one issue observed that some folders in chromium have incorrect history.

e.g.
https://chromium.googlesource.com/chromium/src/+log/master/chrome/browser

Start from "chrome/browser", those folders only contain history that are committed after migration, which is incorrect. Do you have any idea how to fix this? Thanks.

I think this is a bug in gitiles, filed http://b/122868467

And another issue is that the folder history of Chrome Camera App is not migrated. I think the reason is that I did not successfully bring all folder history to the preparation CL. The details are as following:


Originally, I:

(In the old Chrome Camera App repo)
1. Pull the origin/master branch
2. Commit a local CL to move all files under root directory to "chrome/browser/resources/chromeos/camera"

(In the Chromium repo)
1. Add the local Chrome Camera App repo as a remote repo
2. Fetch
3. Merge that local branch to chromium
4. Push


But it seems that the folder history failed to migrate. 
So I have another idea:

(In the old Chrome Camera App repo)
1. Pull the origin/master branch
2. Do "git filter-branch" to rewrite the history of the local branch as they born to be under "chrome/browser/resources/chromeos/camera"

(In Chromium repo)
0. (Revert the previous merge commit on chromium)
1. Add the local Chrome Camera App repo as a remote repo
2. Fetch
3. Merge that local branch to chromium
4. Push



I pushed this new approach on my experimental branch:
https://chromium.googlesource.com/chromium/src/+/2b14e406953d5c91ce9d3cefc2d046344d282fbb

The folder history of "chrome/browser" is still missing, which might be a bug of gitiles as you have mentioned:
https://chromium.googlesource.com/chromium/src/+log/2b14e406953d5c91ce9d3cefc2d046344d282fbb/chrome/browser

And the folder history of "chrome/browser/resources/chromeos/camera" is back:
https://chromium.googlesource.com/chromium/src/+log/2b14e406953d5c91ce9d3cefc2d046344d282fbb/chrome/browser/resources/chromeos/camera


Do you think this approach is doable? Note that since the merge commit only affects directory "chrome/browser/resources/chromeos/camera", which is only used by us, the revert of merge commit should be safe enough.

> it seems that the folder history failed to migrate. 

WDYM? The hashes in chromium/src are exactly as they were in old repo, so I don't understand what you mean by fail.


Dealing with merge commits in chromium is PITA :( It cost me 1.5 hours yesterday, I'm not really willing to do it again. I also doubt anyone ever looks at gitiles history of folders, but if you see the same problem in codesearch (which is used by devs), then importance of redoing this may be high, assuming that what you changed actually will solve the problem.

The origin migration, which we did on yesterday, result in missing folder history  in "chrome/browser/resources/chromeos/camera":

https://chromium.googlesource.com/chromium/src/+log/a23510b0284104fd2ba2d87986503ac65ad7109c/chrome/browser/resources/chromeos/camera



And the test migration that I pushed to my experimental branch is like following, which preserves the folder history:

https://chromium.googlesource.com/chromium/src/+log/2b14e406953d5c91ce9d3cefc2d046344d282fbb/chrome/browser/resources/chromeos/camera



This fix only solve the issue inside "chrome/browser/resources/chromeos/camera", but did not fix the gitiles bug you filed, which can also be observed on codesearch. 

I think the first step we can do is to make sure the missing history in code search could be fixed. If it can be fixed by some operation from our side, we can prepare it as well before next migration.

I understand it costs much time to handle merge commits, I also spend a couple days on this issue. However, since it is a hard requirement of making Chrome Camera App a component app, which is the first step to bring awesome camera features made by Google to Chrome OS users, we really need and appreciate your help. 

+ajp@ 

I see. So, you've actually edited git history to make it appear as if development has always been done inside "chrome/browser/resources/chromeos/camera", but at the cost of changed hashes, e.g.:
  old https://chromium.googlesource.com/apps/camera/+/a00d564590f5490ee6da9db29eb7364d190565aa
  new https://chromium.googlesource.com/chromium/src/+/57e8544b1e1d16970e8787ca395ba4cb28c3cf7e

This is exactly choice #2 in my proposal early on in [1], but apparently I wasn't clear enough about implications, so the resulting choice (#1) wasn't actually the desired one.



Good news is that at least now what you want, and we have all CLs referenced here and could be re-used as is :) I also think the proposal to land a kind of revert and yet another merge commit isn't bad, but I'm not 100% sure it won't cause yet more side effects in JGit and other scripts (e.g., we broke naive bisect script somewhere, pending a bug filing).

But, I can't allocate any more of my time to such tasks, I've already spent >6 hours on it. If another git admin is willing to pick it up, great. Otherwise, you'd need to convince ajp@ that this is important enough that it gets scheduled as work outside of git admin rotation.


[1] https://groups.google.com/a/google.com/d/msg/chrome-infrastructure-team/D2Gk3A0RmV4/Gg0O-0qADQAJ

Thank you so much Andrii for helping the camera app repo migration!

Sorry that we didn't realize the implication on the git history when we prepared the branch for migration.

With the current setup, all the files' and directories' history is not visible in neither `git log` nor `gitiles. One has to manually add `--follow` to the `git log` to perform an expensive parsing to get the original history. The missing history would cause a lot of pain to the engineers, so we'd really appreciate it if we can have some help from the git admins to fix the issue.

ajp@ - Is there anything we can do to make it easier for your team to fix the issue?

Andy - Could you provide some guidance on how we can move forward?

It's really not well defined what is in the scope of what git-admins should handle and what is out of scope. And staffing projects outside of that rotation is tough because we don't actually have staffing for git. I want to discuss this with jparent@.

But, we definitely want to support making ChromeOS awesome so if this is a necessary step to that, we'll definitely figure out the right way to support it.

One logistical question... I assume you're making more changes to the Camera App after the existing merge commit. Is that going to make any revert + land the new version of the merge (with changed hashes) going to be harder / problematic because of all the new changes? Or will you update the new commit to have everything up to whenever we can do this again?

Talked with jparent@. We agreed this falls in the scope of what git admins should do, and they should be planning their work to account for the fact that being admin may take up significant time when scheduled. Worth aligning on that with everyone and their managers. Though there is definitely a point at which things become projects that I must staff without dedicated headcount. 

Andrii you've already done this once. If it's not too hard for someone to ramp up and continue where you've left off I think it's reasonable to ask for the next admin to do the follow up. But please help if we need guidance on how to do all of this. Current admin is Edward so I'm assigning this to Edward for now.  

Thanks for following up, Andy!

We've put all our CLs on-hold at the moment until the migration is done.

We're looking to migrate CCA to component app in M-73, so we can follow the standard Chrome/Chrome OS release process to release our camera app. Currently it's a lot of pain for all the teams involved to release a new version of camera app because the app is not part of the OS image.

Would it be feasible for git admins to finish the migration before M-73 branch?

To make sure we're all on the same page it looks like M-73 branch is on Jan 24 correct? (I'm looking at https://chromiumdash.appspot.com/schedule).

Edward - Talking with Andrii and based on this bug it seems all the CLs are ready we mostly just need to coordinate when to do the work, send PSAs, etc. Is that something you could coordinate to do first thing next week to make the M-73 branch date?

Yes, I'll do this first thing next week.

Yes, M-73 branch is on Jan 24.

Thanks for helping, Edward!

Hi Edward, 

Thanks for helping. Do you need me to help preparing merge commit? What I plan to do is:


(Prepare old CCA repo)
    mkdir OldCCA && cd OldCCA
    git clone https://chromium.googlesource.com/apps/camera
    cd camera
    git checkout -b revised_master
    git filter-branch --tree-filter "mkdir -p /tmp/camera; mv * /tmp/camera; mkdir -p ./chrome/browser/resources/chromeos; mv /tmp/camera ./chrome/browser/resources/chromeos/" HEAD
    git for-each-ref --format="%(refname)" refs/original/ | xargs -n 1 git update-ref -d

(In Chromium repo, assume it is at ToT)
    git checkout -b CCAMigration

(Revert the original merge commit)
    git revert -m 1 a23510b0284104fd2ba2d87986503ac65ad7109c

(Migrate again)
    git remote add CCA [path of OldCCA/camera]
    git fetch CCA
    git merge --allow-unrelated-histories CCA/revised_master
(Revise the commit message and put the correct Cr-Commit-Position)

(Push)
    git push origin [My experimental branch]



Please let me know if you need any more information. Thanks!

I think I'll only need you to do (in Chromium)

    git remote add CCA [path of OldCCA/camera]
    git fetch CCA
    git push origin CCA/revised_master:[Your experimental branch]

I can do the revert and the merge myself.

The branch is pushed to here:
https://chromium.googlesource.com/chromium/src/+/9ef0720703fb97624b0814d04f0fe78d96c66dd3

Please help to confirm that the folder history of "chrome/browser/resources/chromeos/camera" contains 10+ commits to make sure it is successfully migrated after merged. 

And please let me know the time you plan to do the merge if you need me to stand by. Thanks a lot for helping!

I update the branch to include a pending CL and now the ToT of that experimental branch (refs/experimental/wtlee@chromium.org/PrepareCCAMigration) is:
https://chromium.googlesource.com/chromium/src/+/1e8936a96dd223b1f58ed4c593093f8c

I plan to do this at 4pm PST today.

It should be done now :)

I can see the folder history now, thanks for your help!