Ijsfontein

During the last years, cognitive behavioural therapy started experimenting with new technologies, including sensors or even Virtual Reality. Cognitive behavioural therapy is a type of psychotherapy where negative thinking patterns are challenged to modify behaviour patterns or mood disorders such as depression.

IJsfontein - a company that designs and develops playful (digital) learning, “serious games” and interactive experiences - developed a Virtual Reality (VR) experience for eGGZ / ARQ, a centre that wants to promote large-scale integration of e-mental health in The Netherlands and treat the consequences of shocking events and psycho-trauma.

The set-up simulates the experience of being a waiter in a café. The patients take the role of serving other people and communicating with them. The therapists “accompany” the users by watching the events on an external screen, and directing the narrative that the patient is experiencing.

People suffering from shock and trauma may perceive some social situations in very negative ways. The main purpose of this experience is to simulate real-life conversations and challenge the patient’s way of thinking. By ‘accompanying’ the patient through the VR process, the therapist can adjust the treatment by addressing challenges and issues particular to this specific patient.

More than meets the eye

In order to participate in a VR simulation, patients must wear a headset which covers a great part of their face, including the eyes. The therapist is, therefore, unable to ‘read’ the patient’s expressions. Also, what you say is not always how you feel.

So, the project stakeholders were interested in exploring personalization even further. They wanted to investigate how biofeedback could be used in the therapy as a means of adapting narratives and conversations depending on the patient’s emotions. In other words, more than just registering what the patients said or how they acted, therapists wanted to know what kind of emotional responses the patients had to certain situations.

Wearing your heart in your sleeve

Two types of data were chosen to provide insight into the state of mind of the patient: heart rate and galvanic skin response.

Because a VR environment requests mobility, wireless and light-weight technology was preferred. Gloves (EasyPulse & GSR) turned out to be our wearable of choice to attach the bio-feedback sensors that we would use for the prototype.

We programmed the sensors (both attached to finger tips) to collect the desired data and created an algorithm to visualise it in two ways:

• on the therapist screen, so that it’s easier for the therapist to follow the patient’s emotions;
• and in the VR experience of the user, in the shape of an experimental creature which empathizes with the patients.

Initially, we thought that giving a visualisation of the feedback to the user would be a good idea. It would inform the patients of their emotional condition and help them be more aware of their reactions.

However, soon it became clear this had too many variables that rose questions. An important one was the possibility that the feedback visualisation could create a loop of emotions of the user. In order to not compromise the results, we took a step back and decided to provide the visualisation of the data exclusively to the therapist.

Just the beginning...

The project was an early exploration of how to integrate technology in this medical field. Personalisation is key in these types of therapies and bio-feedback could indeed be beneficial in achieving that.

Although we have used proxy users for our tests, we see great potential in studying the influence of extending VR with insights into the patient’s emotional state on the therapy progress. Furthermore, we see great opportunities in exploring how the interpretation of bio-feedback could tweak narratives in storytelling according to the reactions of the recipient.

Finally, yet importantly, we want to emphasise that our study setup could support in creating an accessory for more authentic user testing results.