BD #11 - 3 Insidious Reasons Data Teams Struggle and How to Avoid Them

There are many reasons that data teams fail to deliver the impact they're capable of. This list is by no means exhaustive, but I wanted to capture some of the more insidious that I've encountered, how to spot them, and the best safeguards I've used to counteract them. We discussed how to help underperforming teams in this post here but today we're going to talk about some of the issues data leaders don't spot until it's too late.

The thing is, data is a growing field with a plethora of challenging requirements. The ever-changing technical landscape makes it easy to get lost! To make matters worse, trends across the industry never seem to stick around long, as the flavour of the week tools and approaches rise and fall or get stuck in eternal blog post wars that have been going on for decades (check the data modelling communities and you'll see what I mean).

These influences manifest in unrealised potential, wasted effort, frustrated professionals, and managers looking at their carefully planned data initiatives and wondering where it all went wrong.

There's no paint-by-numbers that makes this stuff work every time (yet!) so people are left to learn the hard way - expensive trial and error or painful experience.

Well I've felt my share of pain and I've had the privelege of building, leading, and scaling data teams at over 30 organisations now - either internally as Director or Head of, or for my customers as a consultant.

Here are some of the less obvious and more dangerous reasons that I've spotted in my time, and my best advice for avoiding them.

Fake it 'till you break it

This issue is rarely because of any malicious intent - instead being caused by good intentions and fairly innocent parties in a tricky situation.

The thing is, not many people know what they need or how to even start when it comes to building data teams. The story goes like this:

• A well meaning hiring manager spots an opportunity to deliver more value from data

• They go through some arduous process to get the budget approved to hire (maybe just one person)

• Without the experience in the data space they guess at the best role for the job by trusting media from their industry that convinced them in the first place

• This is almost always a story about some huge organisation delivering value from data science - so they hire a data scientist first, in hope of emulating those results

• They don't really know what a data scientist does (no one does it seems, see this post here) so they copy a job advert they find online

• Through best efforts and good intentions someone lands in the data role and begins working on the solutions.

But this is where the issue really starts. Very few people are confident enough or secure enough to put their hand up and admit they're well and truly out of their depth. But that's where more data scientists find themselves. They land at an organisation with low data maturity where there's a significant body of work to build some kind of data and AI platform before their solution will ever be usable.

So they fake it.

They spend the next 18 months learning things like SQL, databases, ETL, cloud infrastructure, automation, APIs, reporting tools - you name it. They're delivering something, but often a fraction of the speed for a premium cost of someone that already has those skills.

What then tends to happen is both sides start getting frustrated - where's the AI-driven wonder solution we needed 18 months ago? If this frustration starts to creep into performance reviews then there's a chance this data scientist might leave - and you've just given them 18 months of on-the-job training with a load of new skills that'll open plenty of doors for them.

The hiring manager gets burned and the organisation loses confidence in ever doing this stuff again.

So what do you do about it?

Solution: start in the right places, ensure your data platform is ready to support the more advanced, complex use-cases. You wouldn't hire a glazer to install the windows on your new house before the foundations were built.

CV-driven development

Slightly more sinister is something that I like to call CV-driven development (CVDD). This is especially dangerous for people that haven't come from a technical background leading and managing data teams.

CVDD is when the data professional suggests tools and solutions primarily because they want to learn them and based on how appropriate they are for the problem.

I've seen this in the wild. At one organisation there was a requirement for some analysis and reports - the data was large enough to need parallelism and the reports needed shared. When I asked how we were going to implement them I was told by my team lead at the time "we'll be doing this in Dask and Plotly because I want to learn Dask and Plotly". This baffled me then because the organisation was paying for Databricks and had a Power BI development team.

It makes sense though - there are too many skills to master in the world of data and learning on the job is the best approach, if you can engineer a way to learn and get paid you're on to a winner.

Solution: ensure the wool isn't being pulled over your eyes and that the many, seemingly random tool choices are actually aligned to the problems at hand. Again, this can be difficult without the right experience or technical background.

Wolves in wolves' clothing

Not just a NOFX album, this is a problem in general about the data space. Namely, be careful of who you go to for help. With such a complex and difficult landscape to navigate you might be tempted to grab a free map from a friendly fellow traveler - just don't be surprised when that map leads you straight to a toll road that's difficult to get back from.

It should come as no surprise that most of the advice and guidance out there that's given for free is being touted by vendors that want to convince you their solution is the best. The outcomes of this range all the way from all round positive to outright disasterous depending on who you trust. Be very wary of people that make money from all-in-one data and AI platforms that require you do everything through their tool. Many of these are great tools and sometimes they're the perfect solution for you, but be aware of what you're getting into - an ecosystem of products that make it harder and harder to leave as they increase their grip on your data team and platform.

Solution: remain vigilant and wary of the advice you do get. Take a step back and try to understand why someone is being helpful. Many people in tech are lovely and really do want to help you because they're good people and enjoy sharing their knowledge. Some of them also think their product is the solution you need and a lot of their opinions and advice will be geared to convince you of the same. The best counter for this is to read more from varying sources and not get dazzled in the headlights.

Final thoughts

How do you know this stuff if you don't have the experience?

Reaching out to the data community can be the best ROI in this regard (just be vigilant of who gets back to you 😉). Failing that, some consultancies specialise in helping organisations get started on their journey and will help you map your data and analytics journey in a way that makes the most sense (I'm fully aware of the irony of suggesting a consultancy at this point, this really wasn't meant to be a veiled pitch for my consulting firm, I promise).

It's a tough journey, be careful where you tread and who you travel with.

All the bestAdam

If you're looking for help, here are a few options:

• CDO-as-a-Service: I'll help you build and shape your data function for a fraction of the cost of a permanent Head of Data or Director hire; get in touch about this or project delivery using my consultants here [email protected]

• Coaching: I offer 1-2-1 coaching; book a coaching session here

• Free, daily insights: on LinkedIn here or Twitter here