Heard about data fabrics, data lakes and data mesh solutions? Wondering what all the fuss is about? Find out here.

Data access, the innovation imperative

24/7 eCommerce means that every business faces stiff competition from competitors located around the world. Brand experience has become the primary competitive weapon.  It means every business needs to be digital with decisions based on data, not conjecture.  Companies that don’t harness data and modern innovations like artificial intelligence, blockchain, bots, and 3D visualization, etc. face extinction.

Why do you need a data fabric?

The organization and exploitation of business data are central to the effectiveness of any business seeking to thrive in the digital era.  Before data can be fully harnessed, it needs to be harvested and blended into a common structural design, where every table and row has its place. This ‘layer’ of composable data is also known as a Data Fabric.  A data fabric is the foundation stone to achieve digital ambitions. It is the key to implementing digital transformation at scale–and specifically to the edge of the enterprise, where it is arguably most needed.

Unbundling data from data silos and disparate data sources

Data is commonly managed and stored by the IT systems that produce it. In a modern IT enterprise architecture, businesses will use a variety of SaaS apps, in addition to Cloud-Native applications, and platforms, like Google, Amazon, Office365, Facebook, LinkedIn, etc.  It means that data is fragmented across data silos, making it all but impossible for business users to access data when they need it.  The rising authority of departmental leaders to make their own IT decisions over the past decade has only served to increase data complexity, adding to the challenges of IT and Data Security leaders charged with protecting and leveraging data.  In most organizations, business systems and data systems are the same thing.

The need to re-think data management

Enterprise data delivery is broken; workers today lack access to the data they need to do their jobs, while executives lack insights to answer what-if questions and make data-driven decisions.  For data engineers, the first problem in any new project is to bring data together from its current source, cleanse, de-dupe, and normalize it, etc. before any serious work can begin to answer new questions, solve problems and build apps.  In most cases, spreadsheets become the only accessible tool to organize data into a useful format. Meanwhile, data governance capabilities remain woefully poor.  This has led to boardroom discussions on digital transformation to pivot towards data access, data sharing, and how to decouple data from the data silos to make it more reusable. Improving data consumption and data quality are now business-critical issues.

Data fabric: what it is and what it looks like 

A data fabric is a conceptual data layer that spans your enterprise, releasing data from the applications and systems where data assets are found, to pre-process and organize it in such a way that it can be made useful for composing reports and be consumed by applications to facilitate machine-to-machine automation. 

A data fabric solution creates a unifying data umbrella layer across your enterprise. It means data is presented in a ready-to-use format. Its main elements include: 

1. Data Integration Tools to connect existing systems that hold data to the data ecosystem. 

2. Extract, Transform and Load Technology to harvest and integrate with data sources, with added tools to design Master Data repositories. 

3. Data Mashup Technology to pluck data from existing data sources into new structures and enrich with new data structures to meet new data demands. 

4. Monitoring, Validation, and Data Integrity Tooling to ensure that data pipelines are working correctly to present data from endpoints while preventing corrupted or high-risk data from being accepted into the data fabric. 

5. Analytical Tooling and Applications Development Software to consume data and present it for different uses. 

A data fabric enables a new standard in data management, presenting data of high integrity that data scientists can use to automate processes and answer questions. 

Micro-Portals • Forms • Reports • Training Dashboards • Charts • Maps • Tables Checklists • Onboarding • Risk Registers • Presentations • eBooks

Data Vault, Data Fabric, Data Lake, or Data Mesh—what’s the difference? 

The terminology used in the data industry can be confusing.  Here we try to unbundle the terms commonly used in data architectural discussions.

Data Mesh

A data mesh is more about philosophy and the recognition that businesses have previously seen value in owning data (and thereby trying to control and manage it internally to their business), whereas now, it’s increasingly recognized that harnessing third-party data and sharing it can yield more data asset value.   A data mesh exists therefore within, across, and beyond the enterprise as a federated data ecosystem. 

What are the benefits of a data fabric architecture? 

Strategic ambitions 

A key strategic element of a data fabric architecture is the separation of the data layer from the application layer, fostering greater re-usability of both. Businesses looking to adopt a data fabric approach site one or more of the following objectives. To: 

• Liberate data from old, inflexible legacy core systems 
• Transfer the ownership of data from IT to the business 
• Bring data transparency to create a curious, learn fast/fail fast, data-driven decisioning culture 
• Speed time to value of new projects and digital services 

Hyperautomation and digital transformation 

For organizations determined to reduce headcount and drive down back-office costs, removing the human in the loop is a prime driver for data fabrics. 

Becoming data driven 

While the importance of liberating data from data silos, spreadsheets, and increasing the pace of innovation are all good reasons for a data fabric architecture, there’s no question that most organizations will adopt it because it’s the only way for an enterprise to become truly data driven. 

Case stories and examples 

Ask anyone whether they feel well served by data and they will probably have their own horror story of having to use spreadsheets to harvest and manually crunch data to answer a question or solve a problem. Others will be using a spreadsheet as a quasi-business application because their IT team is too busy to find a robust alternative.  

While the ambition to create a data fabric that spans the enterprise is a desirable one, getting to it can be costly. There will always be a need to achieve quick wins along the way. It’s likely that early-stage solutions will be solving problems at a departmental or functional level.  

In this section, we’ve pulled together some examples of how data fabric architectures are being used to solve business problems. Applications for decoupling span the enterprise, although justifications for projects can originate at a departmental level, as illustrated by the examples below. 

Scanning the market horizon, and matching resources to opportunities 

Power and Energy is a fast-changing market. In the professional services industry, becoming adept at surfacing new advisory opportunities–also knowing what advisory services to offer and how to resource them–is critical to success. Find out how one global advisory firm used a decoupling architecture to gain a competitive advantage. 

Missing building blocks of enterprise data management 

When organizational priorities for IT are dictated by departmental managers seeking to address operational challenges with point-specific solutions, this inevitably leads to more data silos and a whole series of missing pieces in data architectures. 

One of these is the vital framework of data bridges that link data elements together. It’s not uncommon, for example, to find an enterprise resource planning and customer relationship management solution using different fields to identify a customer. Equally, financial systems may not be correctly configured to identify product or service line profitability. When such data relationships are absent, it’s not possible to ask ‘what-if’ questions that combine multiple data sources, without first bringing data together into a new structure. 

Another missed opportunity is the ability of an enterprise to understand its digital DNA. This is the data that helps an organization define itself, such as the organizational structures, locations, people, processes, suppliers, systems, risks, and data elements that define an organization’s makeup and capability. 

Data Lakes are not the answer to every problem 

Whilst the technology is certainly catching the headlines, it would be wrong to assume that harvesting everything into a data lake is a complete or fool proof solution.  There are many wheels to the data fabric wagon and getting data into a single ‘place’ may be a misguided outcome priority for some organizations.  Sometimes, there are quicker wins to be gained by delivering data fabrics across departments and solving a handful of high-priority projects.

DIGITAL DOCUMENTS REMASTERED

Micro-Portals • Forms • Reports • Training Dashboards • Charts • Maps • Tables Checklists • Onboarding • Risk Registers • Presentations • eBooks

Related content