Salma Bakouk: Avoiding Data Catastrophes

Insights from data you can’t trust are worth nothing. When was the last time you saw a software engineer comfortable pushing code to production without testing it?

Exactly.

Engineers develop apps that change all the time but they still need to be working well at all times. No one would push code to production without testing. So why are we comfortable pushing data to production without testing? If data means money, we want to detect data issues before they turn into costly mistakes. We need to go from firefighting to fire prevention and be able to resolve issues when they occur. Today we dive into data observability with Salma Bakouk, CEO and co-founder of Sifflet.

Salma has an engineering degree in applied mathematics from the École Polytechnique in Paris. Before Sifflet she was leading key data initiatives in the sales and trading division of one of the largest US banks. She spent most of her time on the trading floor trying to turn data into money. That's where she learned that knowing when a data pipeline is about to brake is super valuable. She realised that the only way to do this well is to have a system that tracks the health of data as it flows from source to consumption. She also learned that both technical and non-technical people must collaborate to solve data issues. That’s when she got the idea of building a full-stack intelligence layer that made data and metadata more accessible.

What's observability?

The concept of “observability” is a measure of how well we can estimate the internal state of the system by looking at some of its outputs. It was first developed in control theory but it applies to all systems.

For example, take the NASA mission Artemis. Engineers delay the launch of the rocket because when they see that something is wrong. They do this to avoid a catastrophe and burn billions of dollars on the launch pad. They can do this thanks to a system that continuously monitors the state of the rocket, and helps them resolve issues. NASA needs observability.Another example is software development. As apps and codebases become more complex, the probability of bugs and errors increases. The only way to keep the app running is to track the health status of all systems and notify the engineers if there is an anomaly. So why data observability? Because data observability allows data teams to track the health status of the entire data stack. It helps them assess issues, and automate data quality monitoring and anomaly detection. It gives them the context to root-cause issues and helps them speed up incident resolution. Also, they help consumers be proactive and to know when to wait and when to escalate a request.

Salma shared this story that stuck with me. She said: "I spoke once to a data leader of a large public organization that told me, I am fed up with this data quality issues. I think I'm just gonna go and fire the whole data team." What this story speaks to is the loss of trust. This applies at all levels of maturity, especially at early stages. Exploratory is a critical phase. If you can't trust the data at that stage, you're never gonna get to show value to the business and have a chance to scale the team.

If you are a small team you should use basic testing. And you should keep track of your tech debt to make sure that you don't end up with a bunch of spaghetti SQL.You are small? Use DBT tests, the de-facto way for data modelling and data transformation. It's great, it works but it's not enough because we need to see what's going on end to end. That's when we can prevent issues and resolve them quickly. The impact of this is huge and extends beyond just serving data to a downstream dashboard or app. Think about the reusability of data assets for example. Being able to see how things are used is essential to stop reinventing the wheel and start multiplying our business impact.

From Salma, I learned that there are three pillars of data observability: metrics, metadata, and lineage. Metrics help you measure the quality of the data assets the usability of the data asset, the performance of databases and data pipelines. Metadata tells you how data is being used and how it connects to other assets (other models, dashboards, APIs. End-to-end lineage tells us how the data flows from ingestion to insights, and it's essential to track dependencies.

What should "data observability" look like?‍

It should be a holistic view of the whole data stack, to help engineers and consumers work with confidence. It should have anomaly detection across the whole data stack. Once the system detects an anomaly it should make it easy to understand its impact on the business. How is the issue disrupting the workflow of the data consumer? To me the most interesting part of this conversation is how observability can help manage tech debt. Tech debt sneaks in. You don't see it and then all of a sudden you realize that you have been cutting corners for too long. And now what could have become a repository of assets, it's actually unusable. It's important to start small and evolve a system as the need arises, but it's also important to know the tech that we are accumulating.

So many insights in E042, tune in if this stuff is relevant to you and join the conversation on LinkedIn, Twitter, or all other channels.‍

Coming Next

‍Next week we'll be looking into how observability impacts accountability and how to measure the ROI of data observability. That’s going to be episode 043. Thanks for reading, stay tuned and stay curious!