Lineage is reality: plug the gaps, plan to succeed
By Philip Miller, Co-Founder and Chief Innovation Officer
In the first part of this series on data lineage, we talked about how lineage is a tool for context and how if we use it first, we get better outcomes. In this second instalment, let’s take this a bit further and say that without lineage, governance is a false economy.
I consider myself a bit of an all-around geek, not just of data but of technology in general and of ideas. There is power in ideas, which is why one of my favourite films is Inception.
You’ll see how this relates to lineage and your work in a moment. Until then, please bear with me and enjoy the ride.
Back to Inception
For the uninitiated (how?), Inception’s premise is that there can be a tiny moment of inspiration that can affect everything. Another is one is The Terminator (and Terminator 2 – that is as far as I am willing to go!), where the protection of one single life echoes into the future.
These two films have much in common, but they clearly demonstrate the value of understanding reality.
To an extent, both deal with time and the proposition of ‘what if’, raising questions such as…
Inception: what if I question and change someone’s reality?
The Terminator: what if I could kill Hitler before he committed his crimes?
Christopher Nolan, director of the first, spent almost ten years polishing off a grand idea and created a clear closed loop in which he could simply explain the whole premise.
His original diagram is shown below (there is a cleaned-up version at the end):
For those unfamiliar with the film, there are many levels of simulated realities in which the hero characters have to unlock the secrets to a character’s deepest desires. There is a messiness as one of the characters (Cobb) hasn’t disclosed some of his own secrets, leaving the reality at risk.
In The Terminator a ‘mad’ computer decides to wipe out his only nemesis by killing his mother before he was born. The computer goes into this with partial information and in the end ensures that his enemy is actually born – whoops. From there, the Terminator film series goes on to prove what happens when you don’t plan properly. As you can see below, there is a real confusion about reality, and anyone trying to pick up the pieces and add in more lore is left with less and less sense.
Back to lineage
Let’s bring this all back to the point in hand – lineage.
Lineage is an expression of reality, it is a record of what is actually happening. If you take time to build up context as you are going along, you find yourself in a good place where you can show what’s actually happening around you.
The Law of Unintended Consequences comes into play here – The Terminator’s mission is to travel back in time and kill Sarah Connor, the mother of John Connor, who is destined to lead humanity in a future war against machines. By doing this, he hopes to change the course of history and prevent his own destruction at John’s hands. The Terminator didn’t have enough information and brought about its own demise. It was missing reference data (context) – the identity of his foe’s father!
Lineage is important to the film Inception because it helps to explain the origins of the characters and their relationships. It also provides a sense of continuity, which is essential when trying to piece together the complex plot of the film. Ultimately, Cobb didn’t disclose all his relationships to his teammates and created a situation where he created risk to his mission.
Data lineage applications
Now let’s apply these lessons.
There are relationships everywhere; they can be simple, they can be complex, but they all have context. When you spend the time to build up these connections, you find that you can describe what’s going on better and better. The less in the way of holes there are in your lineage view, the more chance you have to manipulate the system.
For example, the machines missed the fact that a child has a father as well as a mother and never sought to complete that information (which we know was freely available from the later films). We know that no one really questioned why Cobb didn’t dream more than superficially; those that did, didn’t grasp the implications properly.
We should always be looking for gaps in our reality, noting them and investigating them. To do this we need a mechanism that exposes the gaps clearly, visually, transparently. If you don’t know, you are missing an important piece of information and will end up making bad decisions. Sometimes these bad decisions will compound over time – building up until they cause big problems, often catastrophic and wide-reaching in nature. Acting on incomplete intelligence is sometimes needed, but it’ll more likely than not to lead to a paradox of some sort.
The world is literally overflowing with examples of not taking lineage seriously. It shows up as unmanaged risks, cyber defence failures, over budget/unsuccessful transformation projects, corporate failures.
Ultimately, who pays? In the end it’s penny-wise, pound foolish to ignore this. It’s always the innocent that pay – with their cash, their jobs, their prospects. Does that sound dramatic? Well, in 26 years of IT projects, I don’t think there’s one that wouldn’t have been more efficient, more sustainable, more pleasant without lineage baked in.
The reality of lineage is that when you do it right, you’re planning to succeed.
If Cobb were working in data governance, I think he’d say: “Solidatus specializes in a very specific type of governance. Lineage governance!”
If Kyle Reese were in the same field, he might comment: “The lineage is not set. There is no governance but what we make for ourselves.”
…………………………………………………….
Below: simplified Inception diagram
Written by: Philip Dutton
Co-Founder & CEO at Solidatus
Philip is a Senior System Architect and Project Manager with over 20 years’ experience within Financial Services.
