Post Data Science Hype
by Stella
We are Stella and Amy. We share firsthand stories and perspectives that are either lost in translation or simply inaccessible to you. Together, we bridge cultural divides and bring the world a little closer—one post at a time.
Post Data Science Hype
2024 has been a busy year for us. We kicked off the year with our Mandarin podcast "数据女孩的中年危机" in January and later launched "The Cocoons", this Substack English newsletter. When asked why we started these projects, we often explain that they stemmed from frustrations with our careers as data scientists. This answer surprises many, given the once-glamorous reputation of the field. But for those of us "in the water," the reality can be worrying.
The Data Science Hype
I started my data science career over a decade ago, at the dawn of the data science hype. Before transitioning to data science, I worked as a civil engineer, focusing on traffic volume modeling and forecasting. In 2012, Harvard Business Review published an article titled “Data Scientist: The Sexiest Job of the 21st Century”. Like many others at the time, I grew bored with traditional engineering jobs, recognized that data science was fundamentally similar to mathematical modeling, and decided to reinvent myself, joining the first wave of data scientists.
Ten years later, in 2022, Harvard Business Review revisited the topic with an article titled “Is Data Scientist Still the Sexiest Job of the 21st Century?”. The article explored how the field had evolved and ended on a positive note, claiming that more opportunities lay ahead. Yet, after reading it, the question lingers: Is data science still a sexy job?
The Start of the Hype
Being a data scientist used to be incredibly exciting. It felt like the perfect job for me. Data science was a catch-all phrase—any problem involving data fell under its umbrella. Colleagues from various business units eagerly sought our expertise for their most pressing challenges. There was an almost magical expectation that data scientists could wave their proverbial wand and solve the toughest problems.
Of course, reality didn’t match these expectations. Stakeholders often came with high hopes for immediate deliverables, unaware that data collection alone could take months. By the time data scientists reached the modeling phase, stakeholders might have already lost interest, and left with disappointment.
I’ve seen this cycle play out too many times. Data scientists would tirelessly explain why projects took so long while struggling to provide accurate timelines. The advice often given by experienced data scientists was to “start with low-hanging fruit.” However, “low-hanging fruit” usually meant building data dashboards, even better if they included tooltips and sliders. Stakeholders loved dashboards, but once you went down that path, it became increasingly difficult to pivot to more complex modeling work. What began as a simple “low-hanging fruit” often turned into a steady stream of dashboard requests, sidelining deeper data science ambitions.
In the end, neither party is happy. The stakeholder never sees the advanced models that could “transform the business” and could be bitter about “over-investing” in data science. On the other hand, data scientists become dissatisfied because they never get to fully leverage their skills.
The Evolution of Data Science
In its early days, being a data scientist meant having a broad skill set. Every data scientist was a generalist, responsible for everything—from data collection and ETL processes to mathematical modeling, prototyping, and production. But today, the definition and scope of data science have shifted. When people talk about data science now, they usually refer to reporting, ad hoc analysis, explanatory modeling, and etc.
While there’s nothing inherently wrong with these types of work, they’re largely supportive roles. Consequently, the data scientist’s role has become more of a supporting function, gradually losing its autonomy. Increasingly, we hear about the importance of “storytelling skills” for data scientists, reflecting the time and effort spent persuading stakeholders and leadership about the insights within the data. Data is objective, but insights often need to be communicated in ways that resonate with diverse audiences. With the exact same data, there are stories stakeholders like, and stories that are disliked.
The more innovative aspects of data science have spun off into specialized roles: Machine Learning Engineer, NLP Engineer, AI Engineer—you name it. Each title corresponds to a specific project type and focuses on a narrower domain. These roles require expertise in specific model tuning techniques and familiarity with domain-specific knowledge. While they may offer less room for exploration than before, they reflect the natural maturing of a field that has carved out its niches. Data science has gradually lost its early-day excitement, and become more well-defined, closer to the more traditional engineering jobs that we ran away from.
What’s Next for Data Science?
The fading excitement aside, is data science still a viable career? Has it evolved into just another skill rather than a standalone profession? It’s like learning English as a second language: while it can open career doors, it’s hard to build an entire career solely on that skill. Similarly, reporting and writing SQL queries are now baseline competencies—akin to Excel proficiency, which was once highly valued but is rarely mentioned on tech resumes today.
Additionally, AI has made data science skills more accessible to those without formal training. Companies are increasingly adopting AI-powered analytics tools, enabling business units and stakeholders to self-serve analytics with little or no input from analysts.
With all the changes in the field, how can data scientists secure their roles in this evolving landscape? Should we specialize in a domain and deepen our expertise to remain indispensable, or pivot to data engineering or software engineering, focusing on AI readiness or tool development?
A Positive Ending Note
Over the holidays, I rewatched Moneyball on a flight. The film was released in 2011, at the peak of data science hype. It reminded me of the excitement and creativity that defined the field’s early days. While the hype is in the past, I see now as a period of dormancy—a pause before data science reinvents itself. We’ve witnessed how it transformed industries, and I’m confident it will rise again in the AI era, bringing fresh opportunities and challenges.
We've stayed through the highs; we might as well stay through the lows.
Next week, Amy will write about her take on the post data science hype.
Please subscribe The Cocoons for more exclusive conversation snippets and stories.
We are Stella and Amy. We share firsthand stories and perspectives that are either lost in translation or simply inaccessible to you.
Thank you for sharing this thought-provoking article. I feel that an article like this, from data scientists like yourselves, could be even more impactful if it included data-driven evidence to support the arguments. Thank you again for sparking such an insightful discussion!