Sentiment Analysis of Web-Scraped Near-Death Experience Narratives

The recent Surviving Death documentary on Netflix is an interesting look at some of the evidence for the survival hypothesis in parapsychology. The first program in the series deals with Near-Death Experiences (NDEs). Accounts of NDEs are of interest to scientists, philosophers and others because of the possible insights they can provide into the nature of the mind-brain relationship. NDEs are also of interest because of their effects. NDEs are often profoundly transformative, having long-lasting and major effects on a persons attitudes and values. There is some research that shows that just learning about NDEs can bring psycho-spiritual benefits.

I have just had a paper published in the Journal of Near-Death studies in which I used a computational technique known as sentiment analysis to measure the sentiment polarity of the words with which people described their NDEs.

Sentiment analysis as a technique has been around for some time now but, as far as I know, it has never been applied to NDE narratives before. In fact there seems to be very little research applying any type of text mining technique to first person accounts of NDEs. I found only one study in the scientific literature. Rense Lange and colleagues in a paper published in 2015, used Latent Semantic Analysis to demonstrate concurrent validity of the Near-Death Experience Scale (a psychometric instrument commonly used to assess NDEs developed by Dr. Bruce Greyson).

Since I did the review for this paper, Vanessa Charland-Verville and colleagues published a paper last year in which they analysed the most frequently occurring words in their sample of NDE descriptions (n=158) and did a hierarchical cluster analysis to determine relationships between them. These researchers expressed the opinion that computational linguistics represents a promising approach to the study of NDE narratives.

It seems to me that we can expect to see more analyses of NDEs using computational approaches since there are several advantages to these approaches:

  • The results are reliable since they are algorithmically derived.
  • Computational methods can scale well. Lexicon based sentiment analysis scales well, certainly in comparison to human analysts.
  • These methods are much faster than human analysts.

A key question of course is whether the results of such analyses are valid. That was something I wanted to examine in this paper, which represents a first attempt at analysing the sentiment contained in description of NDEs.

Scraping the Narratives

To obtain narratives for analysis I scraped first person narrative accounts from the Near Death Experience Research Foundation website. Dr Jeffrey Long, administrator of this site, kindly helped me by providing the entry numbers for nearly 600 NDE accounts, that had been accompanied by a score of 7 or more on the NDE scale, when submitted to the site. The NDE score is important since this is the criteria typically used to determine whether an experience qualifies as an NDE or not.

I used the R package Rcrawler to crawl and scrape the site and was able to successfully extract 557 first person narratives which were then subject to sentiment analysis. Web scraping represents an easy way of obtaining NDE narratives for analysis but it’s worth keeping in mind the following:

  • For data quality purposes it’s probably best to only scrape narratives where the experiencer has taken the NDE scale and scored at least 7.
  • Relying only on webscraped narratives as a data source possibly introduces selection bias into the sample. NDE accounts submitted to public repositories may not be representative of the phenomenon as a whole.
  • Ideally the account needs to contain a certain minimum level of description about the experience itself (as opposed to description of the circumstances of the experience).

Sentiment Analysis

For the sentiment analysis I used the package sentimentr since it incorporates valence shifters, including negators, in the analysis.

Full details of the results are available in the paper. In brief the results indicated that sentiment of NDEs is indeed mainly positive (around two thirds of the documents obtained an overall positive score) though this study showed less overall positive sentiment per document than might be expected. This is both because of the topic (for example the word ‘death’ denotes negative sentiment in a standard sentiment word lexicon) and also because many of the narratives contained a lot of details around the circumstances of the NDE, which was often a distressing or unpleasant situation.

The above might suggest that future approaches to this task should:

  • Consider using a sentiment lexicon modified for use with NDE narratives.
  • In the data preprocessing stage include the extraction of NDE features as aspects of the text, which can then be analysed for sentiment separately from the description of the circumstances of the NDE.

Emotion Analysis

Recently I carried out some extra analysis of this dataset. This analysis isn’t contained in the paper. Specifically I carried out an emotion analysis using the same package as for the sentiment analysis. Emotion mining with sentimentr involves the use of a lexicon to identify words corresponding to the eight primary emotions defined by Robert Plutchik. The analysis was carried out at sentence level then totalled for the collection as a whole. I used the default emotion lexicon in sentimentr. This is a filtered version of the word-emotion lexicon created by Mohammad and Turney (2010)

Total numbers of Emotion words present in web scraped collection of Near Death Experience narratives categorised by emotion type.

The diagram above shows how many words in total in each emotion category were present in the entire collection of 557 documents. With regard to frequency of emotion words there are possibly 3 groups: Trust words and Anticipation words occur most often, followed by Fear, Joy and Sadness words (which occur at similar rate overall) with Anger, Surprise and Disgust words occurring least frequently.

The most frequent emotive word types found were those in the Trust and Anticipation categories. Some words are contained in multiple categories. There were 556 different words found in the Trust category with the three most commonly occurring being “hospital”, “feeling” and “God”. There were 417 Anticipation words found. The most common were “time“, “thought” and “feeling“.

As one would expect there were lower incidences of negated emotion present. The most common negated emotions identified were Fear and Anticipation. The graph below shows the frequency of these.

Total numbers of negated emotion words in the document collection categorised by negated emotion type.

As with the sentiment analysis, I carried out the emotion analysis on the full narratives. Excluding text from the narratives extraneous to the description of the NDE should be considered before this type of analysis also, if one wants to understand what emotion words people use to describe a near-death experience specifically.

I think computational linguistic work such as this can be useful, since it can give researchers a more fine-grained measure of the sentiment and emotions associated with NDEs, than the typical pleasant/unpleasant categorisation often used up till now does. In future work, researchers might consider whether particular characteristics of the NDEr or the circumstances the NDE occurs in, are associated with the strength or type of sentiment and emotion felt by the NDEr during their experience, as measured by the analysis of the words they use to describe it.

Leave a Reply

This site uses Akismet to reduce spam. Learn how your comment data is processed.