Pierre was telling me customers are reaching out asking for help to adopt Generative AI because they are now afraid to be late to the game.
In my humble opinion, this FOMO effect (Fear Of Missing Out) is merely a consequence of the unknown. Fear is also a strategy some AI companies have been wrongly using to market their products or services.
As I stated, FOMO should not be driving an AI initiative and has never been a good enough reason for investing. Fear also has such negative effects on our brain, I would think it could have the opposite effect and freeze the initiatives.
Really, today we should just consider the evidence some studies have brought to light, that Generative AI technology is a recognized boost of productivity. And I think we have not seen that many technologies which could pretend to bring two digits productivity gains.
So let’s have a look at a few papers and their findings, shall we ?
BCG consultants productivity
New study from MIT on generative AI impact on highly skilled workers finds that when artificial intelligence is used within the boundary of its capabilities, it can improve a worker’s performance by as much as 40% compared with workers who don’t use it. At the same time, workers performance drops 19% when AI is used outside that boundary.
But what exactly are the upper limits of AI’s abilities ? Let’s try and understand the details.
The Study involved 700 consultants of the Boston Consulting Group. It involved 2 groups, one suited for GPT-4 capabilities, one received assignments designed so GPT-4 would make an error. Each subgroup was subdivided into 3 conditions : no access to AI, access to GPT-4 and GPT-4 guided access.
‘fit for AI’ assignments involved a new product design and presentation, including several actions, from pitch to launch, like marketing slogan, 2500 word article, and lessons learned. Both AI enabled subgroups performed better with 38% and 42.5% compared to no AI group. They also acknowledge it benefitted the bottom-half lower performers the most, suggesting AI is also good at upskilling.
‘outside the AI abilities’ assignments involved 3 brands, digesting their financial data and preparing an investment strategy memo with justifications. Both AI enabled subgroups underperformed with 13% and 24% compared to no AI group. AI researchers observed performance decrease was due to people “kind of switch off their brains and follow what AI recommends”.
Researchers go on to make some recommendations on how to prevent people following blindly very credible, yet could be erroneous AI generated answers.
Customer service agents productivity
Another study by the National Bureau of Economic Research found that generative AI can increase workers’ productivity by 14%. This increase was most pronounced for less-experienced and lower-skilled workers (34% productivity improvement).The AI tool also improved customer sentiment, decrease of managerial escalation and reduced worker attrition.
The study looked at 5179 customer service agents who used a tool built on OpenAI’s Generative Pre-trained Transformer (GPT) large language model (LLM). The LLM helped agents to respond more quickly, answer more chats per hour, and resolve chats more successfully. In addition to improving productivity, AI can also help to improve customer satisfaction.
The experiment was conducted by a Fortune 500 software firm that provides business process software. The firm staggered the rollout of a generative AI-based conversational assistant to 5,179 customer support agents. The AI tool monitored 3 million customer chats and provided agents with 1.2 million real-time AI suggestions for how to respond. The researchers used a difference-in-differences regression to isolate the causal impact of access to AI recommendations on agent productivity.
Access to AI-assistance may impact how customers treat agents, as it may improve the tenor of conversations by helping agents set customer expectations or resolve their problems more quickly. This also results in less instances of swearing, verbal abuse and yelling, in turns, leading to less burnout and attrition among customer service workers.
This paper results also raise questions about how workers, particularly top performers, should be compensated for the data that they provide to AI systems (as they are the one contributing most best in class training examples and benefiting less productivity gains).
college-educated professionals productivity on writing tasks
Overall, the Noy and Zhang article mentioned people were 40% faster and their quality rose 18%, when using ChatGPT. Workers exposed to ChatGPT during the experiment were also 2 times as likely to report using it in their real job 2 weeks after the experiment and 1.6 times as likely 2 months after the experiment.
The paper discusses the results of an experiment that was conducted to study the effects of ChatGPT on writing tasks. The experiment involved 444 participants, college-educated professionals, who were randomly assigned to one of two groups. The participants in the first group were randomly given ChatGPT, while the participants in the second group were given a different writing tool.
The participants were asked to write two essays. A randomly-selected 50% of our participants are instructed to sign up for ChatGPT between the first and second essay. The essays were then graded by two independent raters. The raters found that the essays written with ChatGPT were significantly better than the essays written with the other tool. The essays written with ChatGPT were also written much faster.
The authors of the article argue that the results of their experiment provide strong evidence that generative AI can have a significant positive impact on productivity. They argue that ChatGPT could be used to automate many writing tasks, such as writing emails, reports, and marketing materials. They also argue that ChatGPT could be used to improve the quality of writing, such as by helping to identify and correct grammatical errors.
The article concludes by discussing the potential implications of generative AI for the workforce. The authors argue that generative AI could lead to job displacement, as some jobs that are currently performed by humans could be automated. However, they also argue that generative AI could create new jobs, as new industries and products emerge.
Overall, the article is a positive assessment of the potential of generative AI to improve productivity. The authors provide strong evidence that generative AI can have a significant positive impact on writing tasks, and they argue that this is just one example of the many ways in which generative AI could be used to improve productivity.
Software developers productivity
The article outlines a study conducted by McKinsey to assess the impact of generative AI-based tools on developer productivity. The study involved over 40 McKinsey developers with different levels of software development experience.
For the study, participants were instructed to perform common software development tasks in three categories: code generation, refactoring, and documentation. The testing spanned several weeks, during which each task was undertaken by a test group with access to two generative AI-based tools and a control group without AI assistance. Developers alternated between the test group and control group for different tasks.
The study employed a comprehensive methodology, combining surveys, time tracking, task surveys, judge evaluations, automated code reviews, and post-experiment surveys to thoroughly evaluate the impact of generative AI-based tools on developer productivity across various software development tasks.
The study is showing potential for a significant increase in productivity (~40% improvement for code generation), depending on task complexity and developer experience. Time savings were less than 10 percent for tasks deemed high in complexity, especially when developers lacked familiarity with a specific programming framework. In some cases, developers with less than a year of experience sometimes took 7 to 10 percent longer to complete tasks with AI tools compared to without.
Code quality in terms of bugs, maintainability, and readability was marginally better in AI-assisted code. Developers iterated with the tools to achieve this quality, suggesting that the technology is best suited to augmenting rather than replacing developers.
The article recommends a structured approach for maximizing productivity gains and minimizing risks when deploying generative AI-based tools, such as generative AI training and coaching, careful use case selection, workforce upskilling, and implementing risk controls.
Another study run during the Amazon CodeWhisperer preview, Amazon ran a productivity challenge. Participants who used CodeWhisperer were 27 percent more likely to complete tasks successfully and did so an average of 57 percent faster than those who didn’t use CodeWhisperer.
Conclusion : Generative AI productivity
Multiple studies are showing evidence generative AI has significant positive productivity impact on different kinds of tasks. It is not that often technology brings two-digit types of productivity improvements. This is the case with GenAI and that should be enough IMHO to justify you investigate it closely to see how to leverage it in your domain.
Comments