Robin Gower
Schedule a discovery call

Data Products

Your organisation has plenty of data. The problem is insight — can people actually find it, trust it, or use it for anything beyond the original purpose it was built for?

The idea of a “data product” is to manage data as a product, not just a resource: widening the focus from the data itself to the tooling that helps people understand and use it.

We shouldn’t think of data as a resource

The problem is that we tend to think of data as a resource - a means to an end.

The transactional data needed to run your organisation is necessarily managed with database systems so that processes can be automated with software (accounts managed, events processed etc). This software is really only designed to serve those day-to-day processes.

The data is just the fuel powering the machinery.

Analysing these transactions in aggregate provides us with a strategic picture of the organisations performance, helping to highlight problems that need attention. This tends to involve the re-interpretation of the transactional data. This too is usually only done for the purpose of a certain report or dashboard. If someone wants to dig into the numbers to get a different breakdown it can be prohibitively difficult.

This approach isn’t necessarily wrong. It just treats data (and analysis) as a means to an end which tends to preclude alternate endings.

Inevitably we suffer when data is locked into a specific purpose:

We should think of data as a product

What if the data weren’t just a means to an end but an end in itself? Instead of treating it as a resource we can build it like a product, supporting users to understand and use the data.

Data products can help solve the problems that arise from treating data like a resource, unlocking it for new purposes:

How to treat data as a product

There’s no one right way to go about this but common principles and standards are emerging.

Adopt a product mindset

Generally speaking, a product mindset encourages you to start by thinking from the perspective of your customers. Talk to the users (and potential users) of your data. Identify the decisions they need to make and problems they’re suffering. Learn about their experiences and workflows. Study your “target market” and how this fits within your organisations strategy. We can work backwards from their decisions to design data models and analyses.

A related pattern for digital products and services is to take an agile approach. Instead of trying to figure out a perfect design before building anything we prototype and iterate. This acknowledges the fact that even users don’t know exactly what they want until they’ve had chance to try things out. Feedback helps us learn what works and what doesn’t so we can optimise our design. Prototypes also let us start delivering results fast.

Manage data as a product

If we’re going to manage data as a product then we need someone to take responsibility for the product and its customers.

A product manager is the named, accountable owner for the product.

There may be a broader governance structure to set the strategy but products need a steward to implement that strategy on a day-to-day basis. The strategy can define goals and situate the product within the organisation’s overall roadmap, helping the manager decide which features should be priorities.

Making the ownership of the product clear helps others understand who they can turn to if they have problems or requests. As such it can also help to explain the team structure and communication channels. Ideally the team responsible for the product includes the domain experts who create the data and models as they have the most context.

One key area of communication is between data producers and data consumers. The manager can set expectations and provide each party with some certainty by creating a data contract. This can include:

A data contract can help prevent breaking changes and simplify the consumer’s work by obviating the need to defend against invalid data. Many parts of a data contracts can be automated ensuring fast feedback and giving producers and consumers the confidence to make bold changes without fear of causing problems for the other.

As with any product, we also need to invest in support for consumers.

Perhaps the most important concern is discoverability. Data products need to be findable. Data-as-a-resource is hidden in the background. Data products ought to be on stage in the spotlight.

This means thinking about positioning and branding. At the very least you should be able to define how the product helps its users. You might even want more elaborate marketing materials - product demos, example reports, tutorials (see below for more on documentation).

In larger organisations with bigger collections this also means “advertising the product”, for example in metadata catalogues and data stores. These let potential consumers search for the data products they need. The catalog record for each product can explain the content and data contract.

Engineer reproducibility

These principles lead us towards some technical design choices drawing on lessons from modern software development. The architecture you adopt for data engineering can produce those artifacts and tools your users need to work with your data products. Providing users with these tools helps them work autonomously and independently, reducing the need for coordination.

Rather than solving a requirement with a one-off procedure, we extract patterns into a software library. Don’t copy SQL statements around or create single-use notebooks. Write software. Library code is structured for re-use. We give components names and document their inputs and outputs (e.g. “sales-net must subtract both tax and discounts from the price”). The library defines the transformations we apply to the data and models we fit to it.

This is the living definition of the data that can be repurposed. You can look-up how something was done and then re-use that code to do it yourself. If you do something new you can contribute it to the library, sharing back for others to benefit.

We can then build our data pipelines as applications on top of this software library. This is the part that actually executes to generate the datasets. We separate out the execution from the implementation like this to ensure re-use. This also makes it trivial to add new pipelines.

The library and pipelines should have tests. These explain how the software works in practice. They can be run to confirm things still fit together while the world (and it’s external data sources) changes around you.

You can go beyond test-driven development and practice documentation-driven design. If you start by trying to explain how to use the data then you’ll tend to write code that’s designed for re-use. Executable documentation stays up to date (a failing build will catch regressions) and serves as a tutorial showing exactly how to acquire and handle the data. You should provide:

We can close the feedback loop with continuous data delivery. Automate everything. Changes to the data sources, software library or pipeline code trigger a build process. The build runs all the tests, regenerates the documentation and runs the pipelines. The pipelines extract, transform and load the data. You can then deploy everything (e.g. insert data into a database, publish reports on the web, release artifacts etc).

An end-to-end delivery process like this can substantially accelerate iteration. The investment delivers the best returns when we have it in place from the outset (when things are least complicated) as it can evolve with the product.

Automation requires us to be completely explicit about the process. We might containerise the pipeline, documenting the dependencies and providing a solution others can adopt. Ultimately engineering a data product is about making things reproducible.

Maximise the value of your data

A product approach maximises the value of your data by making it easier to find, more trustworthy, readily combinable and extendible to future needs.

The approach isn’t a specific technology or technique but rather a mindset. Organisations that adopt it can answer stakeholder questions faster and with more confidence than those who treat data as exhaust.

Ready to get started? Book a discovery call and I’ll take you through an audit of your current setup and set out a roadmap for building better data products.