Amid the buzz surrounding generative AI, a recent study conducted by researchers at MIT has shed light on the impact of this technology in the workplace. The study revealed that generative AI, specifically in the form of ChatGPT support chatbot, significantly increased worker productivity in tasks such as writing letters, presentation documents, sensitive emails, and cost-benefit analysis.
Although the tasks assigned in the study were not exact replicas of real work and did not require factual accuracy or context about specific company goals or customer preferences, many participants felt that they closely resembled their actual job responsibilities. They reported substantial benefits from using ChatGPT support chatbot. Workers who had access to the chatbot were able to complete tasks 40% faster compared to those without it. Moreover, independent evaluators rated the quality of results produced with ChatGPT assistance 18% higher.
The researchers hope that their study, published in open-access format in the journal Science MIT News, will provide insights into how AI tools like ChatGPT can impact the workforce.
Shakked Noy, a PhD student at MIT’s Department of Economics and co-author of the paper alongside Whitney Zhang ’21, stated that “generative AI will have a big effect on administrative work.” He emphasized that while this technology is undoubtedly useful for administrative tasks, it is too early to determine whether its Wholly impact will be positive or negative.
The concept of new technologies leading to mass automation and job losses has been a concern throughout history. However, it is essential to acknowledge that technological advancements also create new jobs and can enhance worker productivity, ultimately benefiting the economy.
Noy explained that economists prioritize productivity when assessing technological developments: “The classical view in economics is that technological advances increase productivity by allowing us to produce economic results more efficiently.”
To examine how generative AI affects worker productivity specifically, researchers enlisted 453 college-educated professionals from various fields such as marketing, grant writing, consulting, data analysis, HR, and management. These participants were given two writing tasks related to their respective occupations. The tasks included writing grant application cover letters, organizational restructuring emails, and analysis plans based on customer data for push notification decisions. Each presentation was evaluated by professionals experienced in the same occupation as the participant, without knowing which ones were created with ChatGPT’s assistance.
Half of the participants had access to ChatGPT-3.5 chatbot developed by OpenAI for their second task. This group completed their tasks 11 minutes faster than the control group, and their average quality ratings increased by 18%.
The study also revealed that using ChatGPT reduced performance inequality among workers. Participants who scored lower on the first task benefited more from utilizing ChatGPT for the second task.
While these results show promise for generative AI technology like ChatGPT, there are limitations to consider. The researchers pointed out that due to using anonymous participants, they couldn’t require contextual knowledge about specific companies or customers. They also had to provide explicit instructions for each task while real-world assignments tend to be more open-ended. Additionally, fact-checking the accuracy of results was not feasible during this study—an issue prevalent in current generative AI technologies.
Despite these limitations potentially impacting ChatGPT’s productivity benefits in real-world scenarios, workers exposed to ChatGPT during the experiment reported being twice as likely to continue using it in their actual jobs two weeks later.
Noy acknowledged that although speed benefits might be less significant in practice due to fact-checking and additional writing requirements, “the experiment shows that it provides significant speed benefits.” He stressed that further research is needed regarding how society should respond to the increasing use of generative AI.
The researchers plan to expand their investigation beyond specific writing tasks and explore how generative AI impacts the economy as a whole. While acknowledging that many other factors influence wages, employment rates, and sectoral changes, they believe the time saved and quality improvements demonstrated in their study are revolutionary for certain types of work.
Zhang emphasized the importance of considering downstream economic and political effects to shape appropriate policies: “If we think this will increase the wages of the lowest-paid workers, that’s a very different implication than if it’s going to increase wage inequality by raising the wages of already high-earners.”
In summary, while generative AI tools like ChatGPT show significant potential to enhance worker productivity, there is still much research to be done on how society should adapt to this technology. Understanding its implications for wages, employment rates, and economic inequality is crucial. As technology continues to advance rapidly, policymakers must consider these factors when formulating appropriate responses.
The study received support from Emergent Ventures, the Mercatus Center at George Mason University, the George and Obie Shultz Fund, the MIT Department of Economics, and a National Science Foundation Graduate Research Fellowship MIT News.
