How AI is disrupting the demand for software program engineers: knowledge from 20M job postings
It’s been a miserable job marketplace for tech employees. How miserable? In keeping with a current ballot, 25% of engineers said it took them a year to find a new job.
I do know lots of you assume this hiring slowdown gained’t final, however what if AI is inflicting a long run shift in demand for sure expertise? What if AI is partially chargeable for all these layoffs?
I’m a extremely curious individual and I wanted solutions. So I made a decision to dig into the chilly, exhausting knowledge and analyze 20 million tech job openings, from November 1, 2022 to Feb 21, 2024 to know how AI has impacted the demand for sure tech expertise and engineers. And to reply the query: Are these tech layoffs pushed by AI in any means?
Right here’s what I’ll analyze:
- Which engineering roles (ie. machine studying, knowledge science, frontend, backend) have grown and declined essentially the most in demand
- How salaries for sure engineering roles have modified prior to now 12 months
- Which expertise/applied sciences/languages have grown and declined essentially the most in demand
- Whether or not corporations are hiring way more AI expertise after layoffs than earlier than
The supply of my knowledge and my detailed methodology are within the very finish of this put up, however my principal dataset was 20M job openings crawled instantly from over 50,000 firm web sites, supplied by Revealera (an information supplier for job postings) and the checklist of corporations that did layoffs have been from this list from Techcrunch.
With out additional to do, let’s bounce into the info!
Which engineering roles have grown and declined essentially the most in demand?
First off, how has demand modified for various engineers?
To search out out, I made a decision to check the uncooked variety of new job openings for every kind of engineer from Nov 1, 2022 to Feb 21, 2023 to the # of latest job openings from Nov 1, 2023 to Feb 21, 2024.
My conclusion? The job marketplace for engineers is a story of two very totally different worlds.
On one finish, the demand for AI analysis scientists and machine studying engineers has boomed. The # of job openings grew 80% for AI scientists and 70% for machine studying engineers. There’s no secret that these individuals are extremely coveted proper now.
Then again, the demand for each different kind of engineer has slowed down. Job openings for cell engineers, frontend engineers and knowledge engineers all dropped greater than 20% from a 12 months in the past. They’re the ugly duckling no one needs to rent proper now.
I don’t assume anybody is shocked that AI engineers and scientists are in large demand, however what’s fascinating is the decline in demand for different engineers isn’t uniform. There are variations in demand , which means that it isn’t all resulting from tech layoffs. AI might need some impression right here.
For example, job openings for backend engineers declined simply 14% vs 24% for frontend engineers. I feel there could also be an “AI impact” right here, as corporations want steady and scalable backend infrastructure to deploy machine studying fashions like LLMs. Then again, there’s no want for robust frontend expertise in the event you’re investing in AI. Whether or not you’re utilizing Angular or React has little impression on how nicely your machine studying fashions carry out.
Secondly, the truth that hiring for knowledge scientists is extra resilient could possibly be attributed to the truth that they complement AI efforts by offering knowledge preparation, cleansing and evaluation. However on the similar time, they aren’t as excessive in demand as ML engineers as a result of knowledge scientists are likely to concentrate on conventional fashions that aren’t based mostly on deep studying or LLMs.
How salaries for sure engineering roles have modified prior to now years
How have salaries modified for every position modified prior to now 12 months?
To search out out, I made a decision to investigate the salaries in these job postings, targeted ONLY on excessive price of residing cities like SF, NYC and Seattle. It’s vital to notice that these are public wage ranges listed in job postings, *not* precise salaries of individuals presently employed in these roles. And I’m taking a look at solely base salaries, *not* complete compensation.
Excellent news: Salaries haven’t declined. However they haven’t gone up a lot both. In actual fact, if we regulate for inflation, salaries have been flat, even amongst AI scientists and ML engineers.
What this implies is that, sure there’s been an enormous spike in demand for AI expertise, however there isn’t a scarcity of provide of machine studying engineers and scientists both. To offer you some context, the # of job openings for machine studying engineers prior to now 4 months continues to be round 15% decrease than what it was in the course of the hiring craze from 2020-2021. And AI and ML engineers + scientists make up simply 5% out of all engineering associated job postings.
So the unhealthy information is in the event you’re pondering of switching gears and transitioning to changing into a machine studying engineer, there’s a number of competitors for comparatively few job openings, in comparison with the remainder of software program engineering.
It’ll be fascinating to see how salaries modified within the subsequent few years, however it may be attainable that salaries for engineers would possibly stay stagnant for awhile.
Which expertise and languages have grown and declined essentially the most in demand
Subsequent, I made a decision to investigate which machine studying expertise have elevated in demand essentially the most.
NLP (pure language processing) had by far the biggest development in demand with a 155% enhance in jobs that point out “NLP”. On the floor, this is sensible as a result of the #1 killer use case for LLMs are in creating chatbots for customer support. Mentions of laptop imaginative and prescient, then again grew half as a lot, and that’s seemingly as a result of the use circumstances for laptop imaginative and prescient are way more specialised (ie self-driving autos) and never as standard.
In fact, the one “ability” lacking right here is LLMs, which I couldn’t embody within the chart as a result of it will merely overwhelm every little thing else. Mentions of LLMs in job postings elevated a whopping 3000% 12 months over 12 months, as evidenced by this chart.
OK, let’s put apart AI and ML expertise for a second. Which conventional engineering expertise reminiscent of programming languages have declined essentially the most in demand? I checked out 4 standard backend languages and 4 standard frontend languages, to see which languages have been the largest losers.
A few takeaways: First, Rust is a large winner. The variety of job openings mentioning Rust went up 32% from a 12 months in the past, which is kind of wonderful contemplating job openings for backend engineers went down. Perhaps there’s a provide/demand imbalance price digging into right here.
2nd, although demand for React went down, it’s nonetheless clearly stealing market share from Angular and Vue. third, Ruby on Rails’ recognition continues to wane and was the largest loser. Lastly, Python’s robust resiliency is usually as a result of truth it’s the de-facto language for machine studying.
Any relationship between corporations doing layoffs and their hiring for AI expertise
Now, for the final word query: Is there any relationship between all these tech layoffs and AI? Are corporations shedding folks to allow them to spend extra money on hiring AI expertise as a substitute?
I made a decision an inexpensive means of discovering this reply was to compile a listing of corporations that did mass layoffs in 2023, and evaluate the # of AI jobs they posted within the 3 months earlier than the layoffs and the three months after the layoffs. If there was a noticeable enhance after the layoffs (in comparison with the change in general jobs post-layoffs), it appeared affordable there was a minimum of an off-the-cuff relationship.
I looked at 50 companies (big and small) that did layoffs and what did the info say? There’s merely no relationship between layoffs and hiring for extra AI expertise. Firms that did layoffs in 2023 had on common 20% extra job postings associated to AI within the 3 months after the layoff announcement vs pre-layoffs, however additionally they had 24% extra job postings general (for any position) within the 3 months afterwards.
Right here’s a pattern of some Huge Tech corporations that did layoffs final 12 months, and their AI job openings pre and put up layoffs.
This doesn’t kill the narrative that corporations are shedding folks to spend extra on AI (in any case, they could possibly be utilizing that $$$ to purchase GPUs), however it does make it weaker. Sure, corporations are specializing in AI, and sure a couple of corporations are seemingly shedding low stage employees that may be changed by AI. However the narrative that the majority corporations are shedding folks to allow them to rent much more AI expertise isn’t backed by any knowledge in any respect. It’s disingenuous.
What’s extra seemingly is that corporations are shedding headcount as a result of they over-hired because the pandemic, rates of interest are nonetheless excessive, and Wall Avenue is rewarding cost-efficiency measures. Not as a result of they should dramatically shift their company technique to AI.
What’s fascinating is that corporations are placing up extra job postings after they announce layoffs than earlier than, which could lead credence to the concept corporations could need to layoff low performers and substitute them with excessive performers. However that’s a narrative for an additional article.
Conclusion
It stays to be seen whether or not demand for different kinds of engineers in addition to AI consultants picks up, or whether or not that is the “new regular”. Sure, hiring is means down from late 2020-2021, however tech job openings are mainly on par with what it was pre-2020. The hiring surge that occurred when rates of interest was 0% could merely be a mirage that can by no means occur once more.
The excellent news is it doesn’t look like corporations are shedding non-AI engineers to allow them to use the cash to rent extra AI engineers. On the similar time, AI is altering the demand for sure kinds of engineers, and reducing the demand for others. The extra associated your ability set is to AI, the extra in demand you’re. And this could possibly be a long run pattern.
However what do you have to do in the event you’re a software program engineer that may’t discover a job, or anxious about being laid off? Whereas I wouldn’t suggest you shift gears instantly and rebrand your self as an AI engineer, it wouldn’t damage to be taught some machine studying on the aspect, so you can begin contributing to machine studying and AI initiatives.
For instance, in the event you’re a devops engineer, you’ll be able to learn to construct CI/CD pipelines for ML initiatives. Likewise, in the event you’re a safety engineer, you’ll be able to learn to construct menace modeling techniques with machine studying, or methods to safeguard AI fashions from tampering. The secret’s to combine ideas of ML into your skillset naturally. So you’ll be able to turn out to be a extra full well-rounded engineer. I daresay the definition of a “full stack engineer” would possibly change to incorporate AI and ML expertise.
Perhaps the AI hype dies down. Perhaps AI is usually an answer in quest of an issue. Who is aware of. But it surely doesn’t damage to be taught the talents wanted in case AI turns into a disruptive expertise (as cell and cloud was a decade in the past)
Subscribe to get extra data-driven analysis on hiring, labor and AI tendencies
Future articles will embody: outsourcing tendencies, our greatest try to make use of knowledge to seek out out which careers are least seemingly to get replaced by AI, and extra! No fluff, no armchair predictions, simply refreshing analysis backed by DATA
Methodology (the boring stuff)
I analyzed 20M job postings from 50,000 corporations (5000 enterprises and 45,000 smaller ones, with nearly each public firm > 1B market cap). These job postings have been supplied by Revealera.com, an information supplier of job postings knowledge.
I constructed a machine studying classifier that learn the job description/title for every of those job openings and categorized them to an engineering job title (ML engineer, AI scientist, backend engineer, frontend engineer, and so on). I additionally used an entity extraction algorithm to extract the talents current in every job posting. I listed all of this knowledge into an Elasticsearch cluster the place I carried out easy time interval aggregations to seek out out the quantity of job openings for every kind of engineer and ability.
Salaries have been taken from the midpoint of the wage vary listed in every job with a location in both Seattle, New York, or San Francisco. If there have been a number of wage ranges, the min was the min of all of them, likewise for the max. Every firm was counted simply as soon as so if there have been a number of jobs for an organization, I took the common of all of them for that firm. Solely salaries with a USD forex have been analyzed.