A Complementary Partnership
“Data is the new currency.”— has gained immense popularity in recent years as data is now a highly valuable and sought-after resource. Overtime data continues to be accumulated and is becoming increasingly abundant. The focus has now shifted from acquiring data to effectively managing and protecting it. As a result, the design and structure of data systems have become a crucial area of interest, and research into the most effective methods for unlocking its potential is ongoing.
While innovation and new ways keep coming to the fore, the best of the ideas currently consists of two distinct approaches in the form of data mesh and data fabric. Although both aim to address the challenge of managing data in a decentralized and scalable manner, they have different approaches and benefits, and they differ in their philosophy, implementation, and focus.
Data Mesh
The architectural pattern was introduced by Zhamak Dehghani for data management platforms that emphasize decentralized data ownership, discovery, and governance. It is designed to help organizations achieve data autonomy by empowering teams to take ownership of their data and provide them with the tools to manage it effectively. Data mesh enables organizations to create and discover data faster through data autonomy. This contrasts with the more prevalent monolith and centralized approach where data creation, discovery, and governance are the responsibility of just one or a few domain-agnostic team(s). The goal of data mesh is to promote data-driven decision-making and increase transparency, break down data silos, and create a more agile and efficient data landscape while reducing the risk of data duplication.
Building Blocks of Data Mesh
Data Mesh Architecture
Since data mesh involves a decentralized form of architecture and is heavily dependent on the various domains and stakeholders, the architecture is often customized and driven as per organizational needs. The technical design of a data mesh thus becomes specific to an organization’s team structure and its technology stack. The diagram below depicts a possible data mesh architecture.
It is crucial that every organization designs its own roadmap to data mesh with conscious and collective involvement of all the teams, departments, and line of Business (LoBs), with a clear understanding of their own set of responsibilities in maintaining the data mesh.
Data Fabric
Data Fabric is not an application or software package; it’s an architectural pattern that brings together diverse data sources and systems, regardless of location, for enabling data discovery and consumption for a variety of purposes while enforcing data governance. A data fabric does not require a change to the ownership structure of the diverse data sets like in a data mesh. It strives to increase data velocity by overlaying an intelligent semantic fabric of discoverability, consumption, and governance on a diverse set of data sources. Data sources can include on-prem or cloud databases, warehouses, and data lakes. The common denominator in all data fabric applications is the use of a unified information architecture, which provides a holistic view of operational and analytical data for better decision-making. As a unifying management layer, data fabric provides a flexible, secure, and intelligent solution for integrating and managing disparate data sources. The goal of a data fabric is to establish a unified data layer that hides the technical intricacies and variety of the data sources it encompasses.
Data Fabric Architecture
It is an architectural approach that simplifies data access in an organization and facilitates self-service data consumption. Ultimately, this architecture facilitates the automation of data discovery, governance, and consumption through integrated end-to-end data management capabilities. Irrespective of the target audience and mission statement, a data fabric delivers the data needed for better decision-making.
Principles of Data Fabric
| Parameters | Data Mesh | Data Fabric |
|---|---|---|
Data Ownership |
Decentralized |
Agnostic |
Focus |
High data quality and ownership based on expertise |
Accessibility and integration of data sources |
Architecture |
Domain-centric and customized as per organizational needs and structure |
Agnostic to internal design with an intelligent semantic layer on top of existing diverse data sources |
Scalability |
Designed to scale horizontally, with each team having their own scalable data product stack |
Supports unified layer across an enterprise with the scalability of the managed semantic layer abstracted away in the implementation |
Both data mesh and data fabric aim to address the challenge of managing data in a decentralized and scalable manner. The choice between the two will depend on the specific needs of the organization, such as the level of data ownership, the focus on governance or accessibility, and the desired architecture.
It is important to consider both data mesh and data fabric as potential solutions when looking to manage data in a decentralized and scalable manner.
Enhancing Data Management: The Synergy of Data Mesh and Data Fabric
A common prevailing misunderstanding is that data mesh and data fabric infrastructures are exclusive to each other i.e., only one of the two can exist. However, fortunately, that is not the case. Data mesh and data fabric can be architected to complement each other in a way that the perquisites of both technologies are brought to the fore to the advantage of the organization.
Organizations can implement data fabric as a semantic overlay to access data from diverse data sources while using data mesh principles to manage and govern distributed data creation at a more granular level. Thus, data mesh can be the architecture for the development of data products and act as the data source while data fabric can be the architecture for the data platform that seamlessly integrates the different data products from data mesh and makes it easily accessible within the organization. The combination of a data mesh and a data fabric can provide a flexible and scalable data management solution that balances accessibility and governance, enabling organizations to unlock the full potential of their data.
Data mesh and data fabric can complement each other by addressing different aspects of data management and working together to provide a comprehensive and effective data management solution.
In conclusion, both data mesh and data fabric have their own strengths but are complementary and thus can coexist synergistically. The choice between the two depends on the specific needs and goals of the organization. It’s important to carefully evaluate the trade-offs and consider the impact on the culture and operations of the organization before making a decision.
