Building a house on sand is a fundamentally unsound endeavour, as the shifting and unstable nature of the foundation poses a constant threat to the structure's stability. The sand's inability to provide a solid, dependable base becomes evident when adverse weather conditions, such as heavy rains or flooding, occur. In such situations, the house is at risk of sinking, tilting, or even collapsing, highlighting the importance of a strong, well-anchored foundation as the bedrock of any secure and reliable structure.

Good job ChatGPT. Right guys, we're outta here. There's nothing more to add. No, wait - we forgot to talk about the Rugs.

Welcome to another exciting blog post, where we will dive deep and explore how we helped a long-standing AWS customer regain control over their data lake, consisting of over 3 Petabytes (PB) of S3 data.

This customer embarked on their cloud journey over a decade ago, embracing the scalability and flexibility offered by AWS. Nevertheless, as time went by, they found themselves grappling with a growing challenge – an ever-increasing pool of data that had accumulated within their S3 buckets. Data management practices had adjusted with growth, however, the customer had no clear visibility into which data was essential to retain and which could be safely removed.

How can we decide what needs to be retained if we don't have visibility of the inventory we have?

As organizations embarked on their cloud adoption journeys, the established practice was to treat the cloud as a separate domain, similar to other specialized areas such as networking, user management, or databases. This approach involved dedicated and specialized teams responsible for the ownership and operation of yet another single vertical within the organization; cloud. However, this model has proven to be more of a hindrance than a long-term success, resulting in more barriers to adoption and ever increasing costs.

What if I told you that delivering a modern asset management lifecycle could be achieved using automation and infrastructure-as-code in a way that could be added to your existing DevOps pipelines? Traditional DevOps pipelines and managing digital assets are a lot more alike than you think;

Clear benefits​

The benefits of adopting an infrastructure-as-code (IAC) approach to provisioning and managing resources is widely accepted. The benefits unlocked by using Hashicorp's Terraform for this are also well documented, in short;

1. Version Control: Terraform code can be stored in a version control system, making it easy to track changes and collaborate with others.
2. Repeatability: Terraform IAC can be executed repeatedly, ensuring consistent and predictable infrastructure deployment.
3. Idempotency: Terraform ensures that the desired state of infrastructure is always met, regardless of how many times the workflow is run.
4. Automation: Terraform can help automate the provisioning and management of infrastructure, reducing manual effort and the risk of human error.
5. Scalability: Terraform can manage multiple AWS resources and scale infrastructure up or down as needed.
6. Improved visibility: Terraform provides clear and concise documentation of the infrastructure and its configuration, making it easier to understand and manage.
7. Cross-provider support: Terraform can provision and manage infrastructure across multiple cloud providers, including AWS, making it a versatile tool for managing infrastructure.

At what cost and risk?​

The challenge to unlocking those benefits though is to find an approach will be cost effective and minimise your risks.