{"id":4634,"date":"2024-03-14T14:20:10","date_gmt":"2024-03-14T14:20:10","guid":{"rendered":"https:\/\/staging.elixirrdigital.com\/us\/?p=4634"},"modified":"2025-03-17T12:14:11","modified_gmt":"2025-03-17T12:14:11","slug":"creating-and-managing-a-data-mesh-in-aws-with-lake-formation","status":"publish","type":"post","link":"https:\/\/staging.elixirrdigital.com\/us\/2024\/03\/14\/creating-and-managing-a-data-mesh-in-aws-with-lake-formation\/","title":{"rendered":"Creating and Managing a Data Mesh in AWS with Lake Formation"},"content":{"rendered":"

If you\u2019re interested in building a Data-Mesh on AWS, engineers from Elixirr Digital have built an AWS CDK project to enable you to rapidly set up your own Data-Mesh proof of concept. If you\u2019re interested in seeing a demo, please get in touch.<\/em><\/p>\n

Intro<\/h1>\n

In our\u00a0last post\u00a0we explained what a Data Mesh is and how it can enable organisations to think about their data as a collection of valuable products. These Data Products will be owned and managed by Data Producers typically split across lines of business. A Data Governance layer manages all the Data Products from all the Data Producers. The Governance layer has two roles:<\/p>\n

1.Security layer<\/strong>\u00a0where admins secure data and ensure it is only shared with Data Consumers and third parties who have appropriate privileges.<\/p>\n

2.Data Mart<\/strong>\u00a0where Data Consumers can discover all of the organisation\u2019s Data Products and request access to them.<\/p>\n

By structuring your organisations data into a Data Mesh, we can more easily avoid the pitfalls of a central data lake managed by one team. Over time the central data-lake can become a Data Swamp with terabytes of data but with no rapid way to control, access or gain actionable insight from that data.<\/p>\n

In this post we look at turning Data Mesh theory into practice. Elixirr Digital supports multiple cloud platforms and technologies such as Snowflake and DataBricks, and all of them can be used to create a functioning Data Mesh. However, for this post we\u2019ve selected AWS and we\u2019ll explain the practicalities and the challenges of building a Data Mesh in AWS using the AWS Lake Formation service<\/a>.<\/p>\n

Communication and Buy-In<\/h1>\n

It would be foolish to jump straight to looking at the technology without mentioning the importance of listening to the concerns, and to generate enthusiasm and buy-in from the teams that will be involved in the project. Successful projects depend on aligned, motivated teams without which any project will struggle to launch. We can help you help you define, deliver and manage projects to ensure maximum success.<\/p>\n

Data Mesh Architecture on AWS<\/h1>\n

To illustrate, we are going to imagine a Bank as a customer. We will first define the high-level architecture for a Data Mesh, which will be structured within AWS.<\/p>\n

Our first architectural decision is to split our Data Producer, Data Governance and Data Consumers across 3 different AWS accounts. Our rationale:<\/p>\n

1.Simplified management<\/strong>\u00a0of users and roles and importantly allows each account to evolve separately to meet their particular use case.<\/p>\n

2.Simplified tracking<\/strong>\u00a0AWS billing and attaching AWS charges to the correct cost centre.<\/p>\n

3.Simplified importing, securing and sharing of data products<\/strong>, as Lake Formation is designed with cross account data-sharing in mind.<\/p>\n

\"\"<\/p>\n

For our bank customer, we will likely have multiple Producer accounts arranged along the lines of business, e.g. Investments, Insurance, Retail Banking, Credit Cards \u2013 each with their own Data Product Owner. Similarly, we could have multiple consumer accounts, e.g. Business Intelligence Team, Machine Learning Team, C-Level Executive team, each with different privileges and rights to view sensitive data products.<\/p>\n

Let\u2019s look at each account type in detail.<\/p>\n

Data Producer Account<\/h1>\n

At a high-level each Producer account will have the same shape, utilising these core elements:<\/p>\n