Natural language processing (NLP) allows us to tag, parse, and even extract information from text. But we believe it also has the potential to help address major challenges facing the world. Recently, we have been working on applying NLP to a serious global health issue: mental illness. In the U.S. alone, 43.6 million adults (18.1%) experience mental illness each year. Fortunately, mental health conditions can often be treated with counseling and psychotherapy, and in recent years there has been rapid growth in the availability of these treatments thanks to technology-mediated counseling. The goal of our project was to better understand how to conduct counseling sessions, which we have done through a large-scale study of crisis counseling conversations.

So far, most research on counseling has been small-scale and qualitative due to the difficulty of obtaining data. We partnered with a nonprofit organization that offers crisis counseling via text messages to apply techniques from data mining and NLP on a dataset of over 80,000 counseling sessions. In our analysis, we searched for linguistic aspects of conversations that were correlated with the outcomes of the conversations (whether the person texting felt better afterwards).

The Data

The text-based counseling service offers free, 24/7 counseling for anyone in crisis (depression, self-harm, suicidal thoughts, anxiety, etc.). Anyone who texts the public number will be matched with a counselor and undergo a counseling session completely via SMS. At the end of the session, the texter receives a follow-up question: “How are you feeling now? Better, same, or worse?” Texting-based counseling is particularly effective with teenagers, allows for privacy (nobody can overhear your conversation), and is much easier to access than other forms of counseling. Each day, the service conducts hundreds of conversations and (on average) instigates at least one active rescue of a texter who’s thought to be in immediate danger of suicide. Carefully anonymized data collected from these conversations is made available with the hope of facilitating research on counseling. You can learn more about accessing the data here.

Our Research

Our study was conducted on about 15,000 conversations (660,000 messages) that had a response to the follow-up question. On average, the conversations were 43 messages long with around 20 words per message. There are many questions that could be investigated with this data, but we were most interested in learning what characterizes a successful conversation. Although a counseling session is free-from and without strict rules, it involves many choices that could make a difference in someone’s life. To answer this question, we developed techniques to quantify aspects of the conversations and determine which ones were associated with successful counselors. There are five “strategies” we found more prevalent in successful counselors (i.e., those who have a higher rate of texters saying they felt better in the follow-up):

1. Adaptability: Successful counselors are aware of how the conversation is going and react accordingly.

2. Dealing with ambiguity: Successful counselors clarify situations by writing more, reflecting back to check understanding, and making their conversation partner feel more comfortable through affirmation.

3. Creativity: Successful counselors respond in a creative way, not using too generic or “templated” responses.

4. Making Progress: Successful counselors are quicker to get to know the main issue and are faster to move on to collaboratively solving the problem.

5. Change in Perspective: We found that people in distress are more likely to be more positive, think about the future, and consider others, when the counselors bring up these concepts. This kind of perspective change is associated with positive conversations, a finding that is consistent with psychological theories of depression.

Although some of these are obvious in hindsight, this is to the best of our knowledge the first time someone has been able to perform a large-scale analysis of these strategies. We hope that this research will lead to a better understanding of how to provide quality counseling services.

Some of our Findings

Here is a summary of some of our findings. See our paper for the full set of experiments and analyses.

Being Adaptable

Thanks to the post-conversation question, we know the outcomes of the counseling sessions. But are the counselors themselves aware of how the conversation is going? And if a conversation is going badly do they react? We investigated this question by looking for language differences between positive (i.e. the texter says they feel better at the end) and negative conversations. In particular, we computed a distance measure between the language counselors use in positive conversations and the language counselors use in negative conversations and observed how this distance changes over time. The results are shown below.

At the beginning of the conversation, the language used in positive and negative conversations is quite similar, but then the distance in language increases over time. This increase in distance is much larger for more successful counselors than less successful ones, suggesting they are more aware of when conversations are going poorly and adapt their counseling more in an attempt to remedy the situation.

Reacting to Ambiguity

We also analyzed how counselors react to ambiguous situations. Ambiguity arises most at the beginning of conversations. We looked at the counselors' responses to the first long message by the texter (typically a response to a “Can you tell me more about what is going on?” question by the counselor). Based on counselor training materials, we hypothesized successful counselors would

• Write more themselves
• Use more check questions (statements that tell the conversation partner that you understand them while avoiding the introduction of any opinion or advice e.g., “that sounds like...”),
• Check for suicidal thoughts
• Thank the texter for showing the courage to talk to them
• Use more hedges (mitigating words used to lessen the impact of an utterance; e.g., “maybe”, “fairly”)
• Be less likely to respond with surprise (e.g., “oh, this sounds really awful”)

	More successful counselors	Less successful counselors
Counselor message length (in words)	15.8	11.8
Counselor responds with check question	12.6%	4.1%
Counselor responds with suicide check	13.5%	10.3%
Counselor responds with thanks	6.3%	2.4%
Counselor responds with hedges	41.4%	36.8%
Counselor responds with surprise	3.3%	2.8%

We found there to be statistically significant differences in all of these aspects except for showing surprise, suggesting these methods from counselor training do indeed help.

Being Creative

Interestingly, although more successful counselors tend to more often use structured responses like check questions, their responses also tended to be more unique. We measured the uniqueness of responses by clustering counselor messages and then counting how many close neighbors the messages tended to have. Messages from more successful counselors tended to have fewer neighbors, suggesting they were being more creative or personalized in their responses. This tailoring of messages requires more effort from the counselor, which is consistent with the results in the above table showing that more successful counselors put in more effort in composing longer messages as well.

Facilitating Perspective Change

Prior work on counseling suggests that certain perspectives are associated with depression, such as having a negative view of the future or being self-focused. We quantified the concept of perspective change by measuring the frequency of different word categories (provided by LIWC) over time in the conversation. The results for time are shown below (we also explored focusing on oneself and having a positive or negative perspective).

Texters start explaining their issues largely in terms of the past and present, but over time switch to talking about the future. Additionally, texters writing more about the future are more likely to feel better after the conversation. This suggests that changing the perspective from issues in the past towards the future is associated with a higher likelihood of successfully working through the crisis. We also investigated whether counselors could instigate this perspective change, and found that texters were more likely to talk about the future if the counselor brought up the subject.

Conclusion

As NLP techniques become more effective and data becomes more available, it is becoming increasingly useful as a tool for investigating pressing issues that our societies face. We think mental health is one such problem and we hope our research on counseling will inspire future work on the area, leading to new insights that could benefit treatments for mental illness. Such research could improve counselor training and lead to tools that help counselors be more successful.

Acknowledgements

Thanks to Jure Leskovec for advising the research and Chris Manning for providing helpful feedback.

---