When
Still a long downtime but was a good lesson.
What
Blog deleted…
How
Manually added policy on wrong bucket, and it kicked in removing the content of the bucket. Recovered by triggering the deployment pipeline via a new PR to production.
Follow up
Things I would like to improve and watch out for in the future
- GH actions to deploy
- Separate branches for develop, staging and prod
- Get the buckets into terraform and run infra from another pipeline
- Automate IaC best practices checking and deployment
- New Alert Webhook to Github to check and force redeploy
- Figure out how to trigger the pipeline when the “if” is set to a merge request and
github.event.pull_request.merged_by != ''used as the if trigger - create separate workflow with different checks for manual/webhook deployments