Git-filter-repo is a tool that you can use to rewrite your GIT history and remove a file from every commit that it was involved with.
- How to Install git-filter-repo using pip?
- Action items to be followed before starting cleaning the secrets
- How to Clone fresh repository for secrets cleaning process?
- Identify all the secrets in the repository and place it in inside a text file
- Time for some action
- Pushing the changes to the GIT Remote Repository
How to Install git-filter-repo using pip?
Pre-requisite:
- Python3 or Python latest version should be installed and added to PATH
Now Open command prompt and execute the below command
pip install git-filter-repo In this case, I have already installed git-filter-repo in my system.
Action items to be followed before starting cleaning the secrets
- Ask the team members to stash their working items separately.
- Ask the team members to not commit/push any code in to the repository.
- For the Cleaning process, one should not use the exising working repository. We should always use the new clone repository.
How to Clone fresh repository for secrets cleaning process?
Create new folder in your drive and open GitBash.
Clone the repository using the below command
git clone https://github.com/vigneshram20/SecretsRemovalTest.git
Navigate into the folder and execute the below command to pull all the meta info about the commits and tags from the remote.
git pull --all --tags
Identify all the secrets in the repository and place it in inside a text file
If you haven’t read the previous post on How to Scan the repo for secrets, Please follow the below link and execute Step-2 .
Okay , So now we have all the secrets identified in the JSON file which we got as an output file from the above Step-2 outcome, The file will look something like this
Now whatever the values that present in the array incidents[] and with a key “match”, We have to collect all the unique values and store it in the new text file in the below format
{Secret}==>{ReplacingText}For Example:
This is the Base64 secret SGkgVGhpcyBpcyBhIFRlc3Q= which i want to replace from the history with the identifier _REMOVED , We can also keep the right hand side as empty if we don’t want to replace it with any other text.
So the format has to be the below one
SGkgVGhpcyBpcyBhIFRlc3Q==>_REMOVEDOR the below one
SGkgVGhpcyBpcyBhIFRlc3Q==>We can keep all the unique secrets one by one in the new line as shown below.
Time for some action
- Copy the file to any directory,
- DON’t copy paste this file into your git repository as it could be accidentally commited .
- Now, Navigate to the newly cloned git repository folder and Open GitBash
- Execute the below comand
git filter-repo --replace-text D:\\replacement.txt --forceAs you can see from the below screenshot that the scan is completed and objects are parsed and completed successfully.
But Wait!
The secrets are all replaced in our local repository only, and that needs to be pushed into the remote,
but before that we can check ourselves whether the replacement happened successfully in our local repository.
Now Open the the cloned repository in your eclipse and identify the file which underwent the secret replacement process.
You can see that the Base64 Tokens present in the Java is no more visible in the class anymore, we can also check the commits in history, there also it won’t be visible to the user.
We have now successfully removed the secret from the file and also rewritten the GIT commit history.
Now time to push the changes to the remote!
Pushing the changes to the GIT Remote Repository
Return back to the GitBash command window
Execute the below commands in sequence
git push --tags --force https://github.com/vigneshram20/SecretsRemovalTest.git
git push --all --force https://github.com/vigneshram20/SecretsRemovalTest.git
Vola, We have successfully completed the PUSH now.
Now all we have to do is go and check the changes that are reflected or not in the GITHUB.
So now we have identified that the secret is not only removed from the current existing file, it also removed it’s existence from the commit history as well, now nobody can see the accidental secret commited which was there before. You are safe now!
