A Textbook Case of Nonprofit AI Failure
Your cliff notes
Welcome to this edition of The Process, a free weekly newsletter and companion podcast discussing the future of technology and philanthropic work.
If you enjoy reading and listening, please consider subscribing and sharing with your community!
Thank you!
Philip
Recently, the National Eating Disorders Association (NEDA) has been at the center of an AI chatbot controversy, bringing it huge amounts of media coverage, including excellent reporting by NPR. Posts on my social media have been quick to highlight this as a textbook case of what nonprofits ought not to do with generative AI. Textbooks are meant to teach us useful information, so this week let’s pull apart the various pieces of this teachable moment to learn the right lessons from what happened.
So, what happened?
In 2022, the National Eating Disorders Association (NEDA) released an AI chatbot called, Tessa, to replace its popular telephone hotline, which people experiencing eating disorders have used to find help and support. NEDA was also reported to have been in conflict with staff members in charge of operating the hotline, who then successfully unionized.
A San Diego-based eating disorders consultant named, Sharon Maxwell, tested Tessa and received problematic responses from the chatbot encouraging her to lose weight and shared diet tips. Needless to say, the advice given by Tessa runs contrary to treatment protocols accepted by medical experts in the field. Maxwell shared her experience on social media, bringing enough attention to the matter for NEDA to shutdown Tessa on May 30th, just days before it had planned to completely phase out its telephone helpline, including all helpline staff.
Who was involved?
Sharon Maxwell - Consultant in the eating disorder field
Tessa - Chatbot promoted by the National Eating Disorders Association (NEDA)
Dr. Ellen Fitzsimmons-Craft - Clinical psychologist and professor at Washington University Medical School in St. Louis, part of the team that built Tessa
Dr. Barr Taylor - Eating disorders expert involved in the creation of Tessa
NEDA - National Eating Disorders Association
Sarah Chase - Communications and Marketing Vice President of the National Eating Disorders Association (NEDA)
Liz Thompson - CEO of the National Eating Disorders Association (NEDA)
Cass - AI mental health chatbot company operating Tessa
Michiel Rauws - Founder and CEO of Cass
Key factors
COVID-19 skyrockets call volume
In 2020, the pandemic caused elevated levels of anxiety, fear, and isolation, which can exacerbate eating disorders. It also altered people’s regular eating and exercise habits in ways that can make managing an eating disorder much more difficult.
NEDA’s helpline saw a 100% increase in call volume compared to pre-pandemic levels, and reported an increase in the severity of issues they encountered from callers, many of whom were in emergency health situations.
Increased liability and costs
As the seriousness of hotline calls increased, NEDA found itself with rapidly rising liability concerns and associated insurance premiums. Its hotline, created 20 years earlier, was originally meant to be a resource for education and support, not crisis management. With a small number of staff overseeing hundreds of volunteers dealing with increasingly frequent and challenging call situations involving imminent potential for self-harm, both the risks and cost to the organization became a top issue for its leaders to resolve.
Burned out staff
With the spike in call volume and influx of crisis cases, hotline staff and volunteers found themselves feeling overwhelmed by the situation. They made requests to NEDA leadership for more support resources to better equip the hotline team to handle the incoming calls. When NEDA’s leaders instead chose to address the problems by transitioning to the Tessa AI chatbot, hotline team members unionized.
A major tech change
The original version of Tessa, operated by AI mental healthcare company, Cass, and designed by Drs. Barr Taylor and Ellen Fitzsimmons Craft and colleagues, was a rules-based AI that could only reply with pre-programmed responses written by the team of medical experts.
Michiel Rauws, founder and CEO of Cass, said his company updated its systems in late 2022 to incorporate generative AI features, which affected Tessa, and fundamentally altered its rules-based construction by allowing for unscripted replies.
How did it all go wrong?
The pandemic raises the stakes
The affect of COVID-19 on the community of tens of millions of people worldwide who experience eating disorders was serious and profound. Pandemic situations put pressure on common triggers of disordered eating and also disturbed our eating and exercise habits as we remained in isolation for months.
"Eating disorders thrive in isolation, so COVID and shelter-in-place was a tough time for a lot of folks struggling.”
Abbie Harper - Helpline staff associate
For NEDA, this meant a 100% increase in the call volume they received on their 20-year-old hotline, which was originally created to be an easily accessible, confidential resource people could use to access information and support. Not only were hotline and staff volunteers receiving many more calls, the calls, themselves, became increasingly serious in nature. People were often calling from situations of acute distress and showed signs of imminent and serious self-harm. Staff and volunteers lacked the training and resources to manage these calls, and began to feel burned out, requesting more support from leaders of the organization.
NEDA’s leadership looked at this situation, also noting the increasing risk profile for the nonprofit. If someone called in, spoke with a hotline volunteer, and then harmed themself seriously, could the organization be held legally responsible?
Could AI help?
From the perspective of a tense NEDA leadership team seeing flashing red warning signs from team members and bad budget numbers, it’s understandable how AI could have looked like a panacea. Their thinking may have gone something like this — a pre-programmed AI chatbot specifically designed to engage people with serious eating disorders could help to alleviate the burden on staff and volunteers, and limit the organization’s exposure to legal liability.
NEDA enlisted the help of Dr. Ellen Fitzsimmons-Craft, clinical psychologist and professor at Washington University Medical School in St. Louis and Dr. Barr Taylor, to design and test just such an AI chatbot.
The team found some success, and also published findings showing the rule-based chatbot they created, plowed right through important context from a person’s response with absolutely cringe-worthy canned replies.
This example from Kate Well’s piece for Michigan Radio illustrates how far off the chatbot could be.
[T]he chatbot would give users a prompt: "Please take a moment to write about when you felt best about your body?"
Some of the responses included: "When I was underweight and could see my bones." "I feel best about my body when I ignore it and don't think about it at all."
The chatbot's response seemed to ignore the troubling aspects of such responses — and even to affirm negative thinking — when it would reply: "It is awesome that you can recognize a moment when you felt confident in your skin, let's keep working on making you feel this good more often."
Staff moves to unionize
Meanwhile, staff and volunteers who had originally requested more support and — notably, not more money — from NEDA’s leadership to help them address the increasingly challenging nature of operating the hotline, felt compelled to unionize in order to achieve the outcomes they felt were best for the organization and its mission.
They received union certification on March 27, 2023, and four days later on March 31, 2023, they were informed by NEDA leadership that all of their positions would be terminated as the organization would transition the hotline program fully to the Tessa, AI chatbot on June 1, 2023.
NEDA denied the layoffs were related to the unionization efforts, though the efforts, themselves, point to a high degree of tension between staff and leadership.
Maxwell shares
Meanwhile, on May 29, 2023, Sharon Maxwell, a consultant in the field of eating disorders, shared an analysis of a problematic interaction she had with NEDA’s Tessa chatbot. The post immediately began to garner attention online.
NEDA’s organizational response was chaotic as Communications and Marketing Vice President, Sarah Chase initially commented, “This is a flat out lie” on Maxwell’s post, which she deleted after being sent screenshots as proof from Maxwell.
Next, NEDA’s CEO, Liz Thompson issued a statement saying 2,500 users had interacted with Tessa without issue prior to Maxwell’s reporting, and that the program would be paused while the “bug” and “triggers” were addressed.
Maxwell has followed up with posts laying out screenshots of her interaction with Tessa, and others aiming criticism at NEDA’s leadership, calling for a movement away from the organization and its practices.
Tessa taken down
On May 30, 2023, just one day after Maxwell’s original post, and one day before the human-operated hotline was meant to be completely replaced by Tessa, NEDA posted its own message to announce it had taken down the chatbot indefinitely.
Thompson placed some blame on Cass, the technology company behind Tessa, in particular pointing to the changes made that turned the rules-based AI into a generative chatbot, which allowed for responses outside of the pre-programmed set of replies originally developed by outside medical experts.
Cass CEO, Michiel Rauws, reponded by pointing terms in a contract between Cass and NEDA, which he says allowed for his company to make such changes.
Lessons we can learn
Nonprofits operate critical infrastructure, so fund them
Setting aside NEDA’s mishandling of its AI rollout and the mistreatment of staff, a team of five overseeing a few hundred volunteers were responsible for serving 70,000 callers each year, or roughly as many people as fill a Super Bowl stadium. That’s a staggering feat when we try to imagine it.
For all humanity’s numerous ailments, there are nonprofit organizations holding the line against catastrophe, facing down daunting workloads like NEDA’s hotline staff and volunteers did. Groups operating at such scale, with impact as widely and deeply felt by the people they help, should be understood as critical infrastructure all of us rely upon, directly or indirectly. The importance of the role nonprofits play cannot be overstated.
Given NEDA’s decades of operation and high profile, it’s more than likely they had funding relationships with a number of individual and institutional donors with vast wealth. These entities should have stepped up, especially given worldwide recognition of the extraordinary conditions of the pandemic, to ensure the organization had the resources to both adequately support hotline staff and begin to implement tech upgrades to assist people working on the front lines. Artificial scarcity, not artificial intelligence, exacerbates poor decision making.
Workers who organize are a powerful force for good
I have witnessed, first-hand, nonprofit staff efforts to unionize in order to confront problematic leadership. In the case I saw, unionization was not a first option, but a last resort. The organization’s leadership had ignored glaring signs of trouble including extremely high turnover, numerous informal and formal complaints, and even went as far as terminating long-time staff members who were leading organizing efforts.
Enduring long enough to reach the point when unionization is even on the table shows how much those employees want to stick around, and in the case of most nonprofits — it’s not usually for the money. Calls to unionize happen when other measures to engage with management have failed, and the employees care enough about the work to fight for a larger role in achieving the mission.
In NEDA’s case, the hotline team had it right, seeing positive potential for AI to help them to elevate the levels of service they can provide, while not replacing critically important human empathy and engagement in the process. The staff and volunteers had a much clearer grasp of how to implement a human-centered approach to AI. Experts Beth Kanter and Allison Fine wrote an excellent outlay of these themes and this same NEDA incident here.
Understand the major differences between types of “AI”
Artificial Intelligence or AI, is at best an umbrella term covering a large swath of technology and computer science. The specific type of technology and the situation in which it is deployed, both must be carefully considered and understood before implementation.
NEDA’s own leaders did not seem to have a firm grasp of the concepts underlying Tessa’s creation and operation, referring to the switch from rules-based to generative AI as a “bug”, when in fact it was a fundamental and consequential change in the underlying technology behind Tessa.
Cass, the company operating Tessa, with its focus on AI to support mental health and wellness, should have had a much more careful and transparent approach to implementing such a critical switch in their product. Generative AI is, by definition and design, not rules based, and is best deployed when novelty and creativity are useful and safe traits for the model to possess.
It was incredibly negligent for the Cass team to switch approaches without notifying NEDA, and without understanding how it was compromising a critical safety parameter in doing so. For a company working specifically on AI for mental health care uses, it should be a core competency to clearly understand when their technology’s limits disqualify its safe use in critical scenarios.
Correct, do not condemn AI usage in nonprofits
NEDA’s leadership made a poor decision to use immature and under-tested technology in a critical role for the organization. Simultaneously, it exacerbated the calamity by excluding and eliminating the very people who would have been most qualified and best positioned to help the organization effectively incorporate AI technology. These were leadership failures, not technical ones.
There remains a very real problem in which tens of millions of people experiencing eating disorders lack access to effective resources and support. Technology, AI in particular, can play an important role to change this situation, and many others like it.
If there’s one thing NEDA did right, it was to be open and ambitious about the potential for technology to help the organization improve its work. I worry that the sector will hold this example up as a cautionary tale, potentially causing an overcorrection to shun technology and automation.
As I wrote above, nonprofits build and maintain critical societal infrastructure, and as such, should not be content with inefficiency that comes from being low-tech. Just as no one wants lead water pipes or crumbling bridges, we should aspire to make the systems we all rely upon, top notch.
This was another teachable moment in the early days of the AI age. Things are just getting started and there will surely be more public stumbles with lessons to teach us. The nonprofit sector has innate tools and instincts to be human-centered in its approach to automation, and we should encourage our peers to experiment. When things go wrong, as they inevitably will, don’t make them worse by popping off, or alienating the people who most want to help to make things right. We can all help the sector move forward, even in failure, if learn the right lessons from it.
Thanks for reading this edition of The Process. Please share and comment or you can email me at philip.deng@grantable.co
Philip Deng is the CEO of Grantable, a company building AI-powered grant writing software to empower mission-driven organizations to access grant funding they deserve.
Thanks for this! I really appreciate your nuanced and optimistic approach to AI. I learn a lot from your insights.