Your organisation has plenty of data. The problem is insight — can people actually find it, trust it, or use it for anything beyond the original purpose it was built for?
The idea of a “data product” is to manage data as a product, not just a resource: widening the focus from the data itself to the tooling that helps people understand and use it.
Read the post →
Your data has potential. An evidence-based decision-making process leads to better outcomes because you’re acting on verifiable facts not fallible opinions or risky guesswork.
In order to realise this potential you need to put your data to work by creating a model. I’m not talking about database schemas, I mean a digital twin that simulates your domain.
A data model provides a simplified view of the complicated reality of your services and products. You can use this abstract representation to breathe life into your data.
At it’s heart this is a set of mathematical functions describing the processes that generate your data. Exploring the parameters and feeding-in alternate inputs provides insights about alternate scenarios. We can use models to help us optimise our choices.
Read the post →
A scientific approach to data promises to let us take decisions on the basis of empirical evidence. This ought to be objective and so more correct than managing our work according to opinions.
But how do we make sure we’ve got the right data? How do we ensure our analytical approach will enable us to take the right decision?
If the data is lacking then we’ll inevitably find ourselves falling back to opinions.
I believe the secret to data-driven decision making is to start at the end…
Read the post →
People have built up intuitions for computers as we’ve lived with them over the decades. Our mental model is that they behave something like a database. We expect computers to record data for us precisely. Inaccuracy is usually attributed to the human system surrounding the computer.
This intuition is at odds with the reality of our new AI oracles. Large language models don’t store facts like databases. They are prone to hallucination (the tendency to make things up during generation). They contain ambiguity and uncertainty very much like the human systems surrounding them.
This post demonstrates this uncertainty and explains how it arises before discussing some implications to help revise your expectations. We conclude with recommendations to address hallucination by returning to the strengths of structured data.
Read the post →
Have you ever written the perfect program? Has it still run unchanged 6 months later? Can your colleagues run it without you?
Just because your analysis is executable, it doesn’t mean the results are reproducible.
Data ages. Libraries change. Machines differ. Servers fail. Bits rot.
Entropy is inescapable.
We can learn how to engineer reproducibility by drawing on techniques from functional programming and software development.
Read the post →
CSV is an accessible format for encoding data but it can be less than accessible for the human reading it.
CSV tables can be hard to interpret as they only have brief column headings and are shorn from context.
CSV on the Web (CSVW) is a standard for describing and clarifying the content of CSV tables.
Read the post →