The data fabric and data mesh are two concepts frequently mentioned in conversations surrounding enterprise data and analytics. While they may seem similar on the surface, they are in fact very different in purpose for information flow. The ramifications too have a growing impact on businesses today and in the future, as more elements of business — product delivery, customer engagement, business development, and even financial accounting — involve data integration and management.
Over the past two decades, enterprises have managed data by oscillating through cycles of centralization, decentralization, databasing, data warehousing, cloud data stores, and data lakes. The list goes on. At present, we have cloud-based-hyper scalers like Amazon Web Services (AWS), Microsoft Azure, and Google Cloud Platform (GCP) as instantly recognisable choices.
Despite the abundance of options, the conundrum remains — businesses want data to be in one place, and easy to find. Collecting all the data into a single location continues to be a challenge. Data fabric and data mesh designs can help businesses solve these challenges in different ways.
For businesses today, finding a future-proof data framework that meets the changing needs of commerce is essential. Today, let’s take out the abstract understanding of data management and see how to approach data as new sources of revenue and value creation.
Decentralization — the Way Out
Physically containing data in one single repository can be challenging in today’s environment where business units operate in silos. This means connecting to necessary data sources that might be stored in different formats, sizes, privacy restrictions, or other metadata traits.
Through logical data integration, business users harness virtualization to connect and unify data and avoid issues relating to physically replicating data for ingestion. In logical data integration architectures, users do not access data directly, but through shared semantic models. These solutions provide virtualized representations of the data and leave the source data untouched. This is important as more stakeholders— executives, and key decision-makers — are involving themselves with the source data to draw more accurate macro-understandings of the business.
It is important to note that logical data fabric and data mesh are two very different architectural approaches. Data fabric being a data infrastructure stack while data mesh focuses on process orientation, that intend to solve data integration, management and delivery in a distributed environment.
Data Fabric for Business Intelligence Analysis
Let’s use an everyday analogy: like different threads in the fabric of our clothing, a data fabric encompasses data from different locations, formats, and types weaved together. In this configuration, the data is still understood to be physically integrated through traditional replication. A logical data fabric replaces physical data integration with a logical data integration component. Data virtualization makes this process possible. This creates a logical data fabric.
This logical data fabric gives business users the option to layer business semantics on top without affecting the underlying data sources. Business decision-makers and data analysts can build custom-tailored virtual data stores, without moving, or without fears of unintentionally modifying or corrupting the underlying data sources.
For business leaders, their team of data scientists can use their preferred business intelligence tools and iteratively build their data models. This means less project management complications in collecting, replicating, and cleaning up the data for analysis. Logical data fabric makes data ready and accessible for use.
Toyota-Astra Motor (TAM) Indonesia, currently the market leader in the Indonesian automotive industry applies data virtualisation as a core component of its enterprise-wide logical data fabric. Executives, data scientists, and business users can now easily use their business intelligence tool of choice. And more importantly, the introduction of the logical data platform ensured that security policies were able to be managed centrally and the overall trust and confidence in data were restored.
The logical data fabric removes complexities in data access and integration, enabling business users to make timelier decisions and cut down on product or service development cycles.
Data Mesh for Macro Visibility
While a data fabric serves as an integration solution, a data mesh is an organizing solution to structure data, individual access privileges, processes, and workflows within a singular enterprise.
In a data mesh, data ownership and management belong to assigned “data domains” that correspond to the department or function of the enterprise. Stakeholders within each data domain will package their data together as products to be delivered throughout the enterprise. Each line of the enterprise creates and maintains their own data products — consumer data products, asset data products, and finance data products amongst others.
When key leadership lines of the enterprise and owners of the data domains need to create their own “views” or data products, granting access privileges may require complex and costly workarounds. This is where a logical data mesh bridges the needs of top-view visibility by business stakeholders with the enterprise’s existing IT architecture design.
Through a logical data mesh, enterprises can connect the networks of relevant data available in an orderly and secure way to users, analysts, developers and applications that need it. Data virtualization enables enterprises to continue working with existing data assets, services, and project management workflows without disruptive trade-offs.
The logical data mesh grants business intelligence teams access privileges and business metadata in a separate mutually exclusive layer. New semantic layers can be created for data domains without fears of modification or corrupting the data sources. Data virtualization is also a building block for creating new data domains that could be packaged for internal enterprise use or external customer-facing products for sale.
Logical Data Fabric, Data Mesh or Both?
Which approach works better? It depends on the business’s operational workflow and the size of the organization. Logical data fabric is an intelligent, powerful way to integrate data, manage and deliver data which can be applicable to all shapes and sizes of businesses. A data mesh architecture is an equally intelligent way to organise data within an entire enterprise but usually is more effective for large businesses with many lines of business units and complex organizational structures. So, it’s not whether data fabric or data mesh, but rather a question as to whether a business requires both data fabric and data mesh implementation, based on their needs.
Data virtualization empowers organizations to use logical data integration and adopt the benefits of both approaches — data fabric and data mesh. Data virtualization capabilities also provide a future-proofing data framework to meet the evolving needs of businesses. Data virtualization enables businesses with a future-proof data framework to meet the changing needs and landscape of business, today and in the days ahead.
By Shanmuga Sunthar Muniandy, Director of Data Architecture, Denodo APAC
