BD #6 - Why Your Data Scientist Will Quit

Many data scientists are leaving to become data engineers. This has been a growing trend for several years. Unfortunately, the reality in many organisations doesn't match the dream most data scientists were sold.

We've all heard the hype around data science roles, and many were lured in by the promise of challenging work that would significantly impact our employers. I'm included in that group - my sights were set squarely on another career before I got caught up in the excitement.

For me, at least, I wanted to leverage my existing technical skills by applying them to solving novel and exciting business problems.

I loved being a data scientist, but it was far from what I had initially expected. 

Wrong expectations

Many learning materials for breaking into data science focus very heavily on machine learning and very little else. This poses a significant problem - data science requires so much more.

Thinking data science is all about machine learning is much like thinking plumbing is all about wrenches - you've missed the point.

Many people land in their new data science role and do one of two things.

Some blindly apply machine learning to any dataset they can get their hands on (even getting the data can be a nightmare, more on that later). They don't take the time to understand the domain context deeply, ask the stakeholders the right questions, or try other solutions to deliver the desired outcome. This leads to talented people spending a lot of time and effort developing the wrong solutions or wasting time getting to what matters.

Others never get that far and spend all their time learning a host of other skills to try and get their hands on the data in the first place.

Limited technical foundations

The truth is most organisations aren't ready to be deploying machine learning solutions. But unfortunately, these companies are also victims of the hype - oversold the promise of what machine learning and AI can do for them. Some are even worse, jumping on the bandwagon because of FOMO and trying to follow the path of competitors without really understanding what they're getting in for.

This is difficult - there's a lot of conflicting advice out there, some genuine, some pure marketing. It takes a lot to deliver machine learning solutions successfully.

Many smaller organisations look to hire data scientists in the hope they will shape the newly formed data function. This isn't always a bad approach but often leads to talented and expensive data scientists doing work that's not in their interest. I've seen data scientists that did nothing but build pipelines and databases for two years because there was nothing in the way of data architecture when they started.

In these cases, you might be lucky and get a data scientist that's keen to learn the new tools and approaches. Just remember you're probably paying them over the odds to learn the core tools of a different role and not getting as much value out of the core skillset that makes them expensive in the first place.

Lack of leadership buy-in

Another core killer of enthusiasm in your data scientists is a complete lack of engagement by the wider business.

I've seen it time and again where one well-meaning manager has landed some budget to bring in a data team. They've spent months getting the right person in and, at first, things are going well. This data scientist even manages to build some models that will make a significant impact.

Then the problems start to come. 

These models look promising but getting them from the exploration stage to production requires more investment. Or perhaps the models are in production but the end users don't like the idea of changing their workflow because of them.

Things start to grind to a halt. The well-meaning manager can't get the engagement they need to drive this project to success. The leadership still see this as an experiment or pet project. 

Seeing your hard work and skills go to waste is a surefire way to drive away strong data science talent.

The rise of the data engineer

One of the themes across the points above is a lack of self-sufficiency - data science just sits relies on too many other components being suitable to work. These obstacles might be political and people-focused or they might be technical and data-focused. 

When it comes to technical issues, many of them are solved by building better pipelines and expanding the data platform. This is typically the work of the data engineer. 

As more organisations realise how many data engineers they need the market for them gets hotter and hotter. For some data scientists, this is great - they've been in adjacent roles and maybe even had the chance to learn the requisite skills on the job.

Why not leap into a promising career that's able to impact a wider set of use cases and build the foundations that data science will eventually need?

Good luck,

Adam

If you're looking for help, here are a few options:

• For data professionals seeking help or wanting some 1-2-1 coaching, reach out to me on LinkedIn here or book a coaching session here

• If you're an organisation looking for some assistance, email me at [email protected]

• If you're interested in some packaged training or consulting services then... watch this space

And finally, if you've enjoyed this or know someone who might find it useful, I'd greatly appreciate you sharing it with them 🙏All the best.