Data virtualisation has evolved considerably. During its infancy, data virtualisation overcame legacy infrastructure limitations, consolidated structured and unstructured data, and liberated data from silos.
Today, data virtualisation’s larger mission is managing data — enabling data to be used to the fullest. By working in new ways that extend beyond its beginnings, it is surpassing assumptions about its purpose in technology, industry, and the broader business world.
Data Virtualisation: Overcoming Limitations
Data virtualisation started out as a way to connect to disparate sources of data – to retrieve and manipulate data without requiring technical details or physically replicating the data from the source. But as computational needs grew more sophisticated, connecting to data has become more complex.
First, there was a need to connect to singular sources. Hence, the data warehouse was established as a central repository for integrating data. However, the growing volume of unstructured data called for a new storage form that could accommodate unstructured data. Thus, data lakes were born. But soon, data lakes hit their own limitations. Stakeholders stored datasets in multiple formats within data lakes, making it difficult to access data in the required format without data processing.
To overcome this limitation, companies turned to data virtualisation to connect to these diverse data sources within data lakes, and analysts responded favorably.
The connectivity offered by data virtualisation employs a logical integration rather than a physical one. This is crucial, as physical data integration means slower time-to-value, higher cost, and rigid, repetitive coding to accommodate any changes in the data infrastructure. As more companies realised that there is no one-size-fits-all solution to connectivity, data virtualisation, through its logical integration approach, became the way forward.
“Enterprise and software architects are familiar with the myth that monoliths are simpler,” says Gartner. “More often than not, monoliths are inherently complex and fragile due to unnecessary dependencies, as architectural principles, modularity and decoupling do not seem to be compatible with a single platform for everything.”
Data Virtualisation: Creating Futures
Today, data virtualisation is meeting the growing needs of data management by supporting metadata management, data catalogs, unified semantics, and data governance.
Data virtualisation is also used to power newer forms of data architectures such as logical data fabric or data mesh, a distributed, decentralised data management design. This design is built on data virtualisation and overcomes the limitations of traditional centralised architecture.
The data virtualisation market has grown in recent years, with more players joining in to take advantage of the growing demand for data integration services. Nonetheless, decision makers should be prudent when selecting a data virtualisation vendor.
A common pitfall would be entrusting your data to companies that purportedly offer a full spectrum of data management services, yet do not. With many newcomers still focusing on the basics of data virtualisation, odds are, they are unable to live up to the promises of delivering active metadata, advanced query optimisation, or a unified security layer.
Data virtualisation has progressed to a stage of advanced data management. It now needs to be able to support the demands of today’s business operations and services using emerging technologies such as platform-as-a-service (PaaS) deployments, logical data fabric, and data mesh.
Whether for marketing professionals evaluating customer insights, migrations to the cloud, or the analysis of sensor readings, data virtualisation today connects widgets, services, and infrastructure in real time. Strong data integration, management, and delivery capabilities provide data analysts with a holistic view over all enterprise data. By leveraging data virtualisation to maximise the power of connecting to data, companies can surpass their competition by leaps and bounds.
By Ravi Shankar, SVP & CMO at Denodo
