AI is being used in social services鈥攂ut we must make sure it doesn't traumatize clients
Late last year, ChatGPT was used by a Victorian child protection worker to draft documents. In a glaring error, ChatGPT referred to a "doll" used for sexual purposes as an "age-appropriate toy." Following this, the Victorian information commissioner .
Unfortunately, many will not garner such public visibility. It's crucial that people who use social services鈥攕uch as employment, homelessness or domestic violence services鈥攁re aware they may be subject to AI. Additionally, service providers should be well informed about how to use AI safely.
Fortunately, emerging regulations and tools, such as our , can help to reduce AI harm.
How do social services use AI?
AI has captured global attention with promises of better service delivery. In a , AI promises to reduce backlogs, lower administrative burdens and allocate resources more effectively while enhancing services. It's no surprise a range of social service providers are using AI in various ways.
Chatbots simulate human conversation with the use of voice, text or images. These programs are increasingly used for a range of tasks. For instance, they can provide mental health support or offer employment advice. They can also speed up data processing or help quickly create reports.
However, chatbots can easily produce harmful or inaccurate responses. For instance, the United States National Eating Disorders Association deployed the chatbot Tessa to support clients experiencing eating disorders. But it was quickly pulled offline when advocates flagged Tessa .
Recommender systems use AI to make personalized suggestions or options. These could include targeting job or rental ads, or educational material based on data available to service providers.
But recommender systems can be discriminatory, such as . They can also reinforce existing anxieties. For instance, pregnant women have been recommended .
Recognition systems classify data such as images or text to compare one dataset to another. These systems can complete many tasks, such as face matching to verify identity or transcribing voice to text.
Such systems can raise , , inaccuracy and concerns. stopped using facial recognition cameras because they risked privacy breaches鈥攊t's difficult to obtain informed consent from mentally unwell or intoxicated people using the shelter.
Risk-assessment systems use AI to predict the likelihood of a specific outcome occurring. Many systems have been used to calculate the risk of child abuse, long-term unemployment, or tax and welfare fraud.
Often data used in these systems can recreate societal inequalities, causing harm to already-marginalized peoples. In one such case, a tool in the US used for identifying risk of child mistreatment unfairly targeted , and .
A Dutch risk assessment tool seeking to identify childcare benefits fraud was , while .
The need for a trauma-informed approach
Concerningly, our research shows using AI in social services can cause or perpetuate trauma for the people who use the services.
The defines trauma as an emotional response to a range of events, such as accidents, abuse or the death of a loved one. Broadly understood, and be passed down through generations. people in Australia as a result of colonization is an example of group trauma.
Between 57% and 75% of Australians experience at least one traumatic event in their lifetime.
Many social service providers have long adopted a trauma-informed approach. It prioritizes trust, safety, choice, empowerment, transparency, and cultural, historical and gender-based considerations. A trauma-informed service provider understands the impact of trauma and recognizes signs of trauma in users.
Service providers should be wary of abandoning these core principles despite the allure of the capabilities of AI.
Can social services use AI responsibly?
To reduce the risk of causing or perpetuating trauma, social service providers should carefully evaluate any AI system before using it.
For AI systems already in place, evaluation can help monitor their impact and ensure they are operating safely.
We have developed a that helps service providers to assess the safety of their planned or current use of AI. The toolkit is based on the principles of trauma-informed care, case studies of AI harms, and design workshops with service providers. An online version of the toolkit is about to be piloted within organizations.
By posing a series of questions, the toolkit enables service providers to consider whether risks outweigh the benefits. For instance, is the AI system co-designed with users? Can users opt out of being subject to the AI system?
It guides service providers through a series of practical considerations to enhance the safe use of AI.
Social services do not have to avoid AI altogether. But social service providers and users should be aware of the risks of harm from AI鈥攕o they can intentionally shape AI for good.
Provided by The Conversation
This article is republished from under a Creative Commons license. Read the .
