Funny, I was just reading comments in another thread about people with mental health problems proclaiming how terrific it is. Especially concerning is how they had found value in the recommendations LLMs make and “trying those out.” One of the commenters described themselves as “neuro diverse” and was acting upon “advice” from generated LLM responses.

And for something like depression, this is deeply bad advice. I feel somewhat qualified to weigh in on it as somebody who has struggled severely with depression and managed to get through it with the support of a very capable therapist. There’s a tremendous amount of depth and context to somebody’s mental condition that involves more deliberate probing to understand than stringing together words until it forms sentences that mimic human interactions.

Let’s not forget that an LLM will not be able to raise alarm bells, read medical records, write prescriptions or work with other medical professionals. Another thing people often forget is that LLMs have maximum token lengths and cannot, by definition, keep a detailed “memory” of everything that’s been discussed.

It’s is effectively self-treatment with more steps.