I’d like to tell a tale of a git-astrophe that I caused in the hope that others can learn from my mistakes. Git is awesome but also very feature-ful, which can lead to learning about some of those features at the worst times. In this episode, I abused my knowledge of git rebase, learned how the -f flag to git push works, and narrowly avoided learning about git reflog/fsck in any great detail.

Often times, you will need to rebase your feature branch against master (or production, in this case, it was a puppet controlrepo) before submitting a pull request for someone else to review. This isn’t just a chance to rewrite your commit history to be tidy, but to re-apply the changes in your branch against an updated main branch.

For instance, you created branch A from production on Monday morning, at the same time as your coworker created a branch B. Your coworker finished up her work on the branch quickly and submitted a PR that was merged on Monday afternoon. It took you until Tuesday morning to have your PR ready. At this time, it is generally adviseable to rebase against the updated production to ensure your branch behaves as desired after applying B‘s changes. Atlassian has a great tutorial on rebasing, if you are not familiar with the concept.

When you rebase, your history is rewritten. When you push your changes to a remote, the commit history of your local A is different than that of the remote A and an error is generated:

[rnelson0@build01 app]$ git rebase -i origin/master
...
Successfully rebased and updated refs/heads/sshkey.
[rnelson0@build01 controlrepo]$ git push origin sshkey
To git@github.com:rnelson0/controlrepo.git
 ! [rejected]        sshkey -> sshkey (non-fast-forward)
error: failed to push some refs to 'git@github.com:rnelson0/controlrepo.git'
To prevent you from losing history, non-fast-forward updates were rejected
Merge the remote changes before pushing again.  See the 'Note about
fast-forwards' section of 'git push --help' for details.

Since you have rewritten your history, this is expected (if you have not done a rebase and see this error, you should investigate!). To push the changes upstream anyway, you want to force the push by appending the -f flag. Here’s what the man page for git-push says about the flag (emphasis mine):

Usually, the command refuses to update a remote ref that is not an ancestor of the local ref used to overwrite it. This flag disables the check. This can cause the remote repository to lose commits; use it with care.

The remote’s branch is no longer an ancestor of our local reference, as the commit history has been rewritten. This is our branch A so we know the above error is legitimate and that using the force flag is the correct way to proceed. This is what the output looks like:

[rnelson0@build01 controlrepo]$ git push origin sshkey -f
Counting objects: 15, done.
Compressing objects: 100% (9/9), done.
Writing objects: 100% (9/9), 1.65 KiB, done.
Total 9 (delta 3), reused 0 (delta 0)
To git@github.com:rnelson0/controlrepo.git
 + 39f0d4a...52dc92b sshkey -> sshkey (forced update)

Notice that I appended the flag after the branch name! If you leave out the branch name, you get… different behavior:

 [rnelson0@build01 app]$ git push origin -f
Counting objects: 2519, done.
Compressing objects: 100% (803/803), done.
Writing objects: 100% (2312/2312), 368.70 KiB, done.
Total 2312 (delta 1530), reused 2205 (delta 1424)
To git@github.com:rnelson0/app.git
 + f6a6c07...87e5f14 sshkey -> sshkey (forced update)
 + 5068146...c29d6c8 production -> production (forced update)

Uhoh. The remote’s production has been overwritten with your production as your history was different than the upstream’s. If you forget to git fetch and git pull onto production, the merged PR of your coworker’s branch B has now been overwritten with your copy of master from before the PR!

When I did this, I was very fortunate – another coworker had done a fetch / pull against production and so we had a copy of the branch that I had overwritten. That branch was force pushed to the remote and all was well. Branch B was also still around and could have been pushed as production in the worst case – almost the same history but without the commit indicating the PR merge.

If everyone else had deleted their branches and no-one had an updated production branch, it’s possible that git reflog or git fsck could have saved our butts, but that’s an entirely different headache to deal with, well beyond the scope of this article. It’s best to simply avoid these problems through judicious use of the force flag. I suggest avoiding shorthand git commands by always specifying the remote and branch (i.e. git push origin sshkey instead of git push) and only using the force flag after receiving an error by hitting the up arrow and adding -f to ensure it’s applied against the same remote/branch.

To recap:

• When you use -f you overwrite history.
• If you push to a branch (git push origin feature -f), you overwrite that branch.
• When you push without a branch (git push origin -f), you overwrite all remote branches that are not ancestors of the corresponding local branches – potentially all branches, including the main branch (master/production).
• If this happens, you may have modified the history of your repo. If someone merged PR10 on the main branch but you have not done a git fetch/pull on master since PR3, 7 PRs were lost. You may have also triggered a webhook to perform a build against the “new” commit.
• Simple Recovery: have someone else who did a fetch/pull after PR10 do a git push origin master -f.
• Complex recovery: use git reflog or git fsck to recover as much of the lost commits as possible.
• Prevention: Investigate if your git upstream allows you to disable force pushes to branches. GitHub Enterprise does and regular GitHub can be done if you email support@github.com (no UI option).