Sentiment Analysis on Text Using Neural Networks

The process of analyzing and classifying the words based on the tags associated with the words is called sentiment analyzes. As the above statement clearly states that we will be using the lexicon-based approach, i.e. there exists a dictionary containing the bag of words (tags) needed for the neural network to analyze whether the sentiment in the text is a positive one or a negative one. Sentiment analyzes has found its applications in many areas of analyzing movie reviews, spam email detection, the success of a product through trend prediction and so on. Naïve Bayes classifier is a probabilistic classifier which analyzes the presence of a given sentiment in a sentence. A tokenizer is used to tokenize the sentence and pass the semantic vector value to the neural network. The neural network then uses a Naïve Bayes classifier to classify whether the overall sentiment of the sentence is positive or negative. The dataset used is corpus which contains a labeled bag of words.

Peer review under responsibility of International Conference on Sustainable Computing in Science, Technology and Management.

The research on sentiment analysis has been going on through ages and the main reason for that is that data is readily available in the form of reviews, feedback comments, etc. But this information can be harnessed by a machine using a deep learning neural network. A deep learning code combined with a neural network makes the system more dynamic and adaptable. This helps the neural network in analyzing various convoluted patterns and also classifying it in presence of noise. A neural network was designed based on the observation of complexity present in the neurons of biological organisms. This neuron mostly consisted of 3 parts, i.e. the dendrites, stoma, and terminal. There are various types of neural networks available and based on their structure and adaptability they can be used for a plethora of activities. For example, a CNN is mostly used for pattern detection and classification in an image, whereas an RNN is used for text due to its exhaustive iterations and high adaptability in presence of noise.

The neural network we will be using in this project will be Naïve Bayes classifier. The main reason we are using Naïve Bayes is that it is made for sentiment analyzes of huge datasets. The naïve Bayes classifier classifies each vector tag based on their maximum likelihood. Since it is probabilistic it considers the presence of a feature independent of another feature. The maximum likelihood is an estimation of the probability of a particular condition, that is the probability that a particular event will occur. Since it uses a dictionary which in our case is corpus it follows supervised learning. Supervised learning is the process in which the neural network learns in presenceof a teacher (image, data, or in our case tags/ labeled data in the bag of words). Based on the semantic vector classified by the neural network the sentiment is accordingly analyzed.

Sentiment analyzes has found its application in a variety of application like text classification for spam detection, topic detection, and recommendation, etc. Also, sentiment analyzes when applied with the help of combinations of the neural network can carry out various complicated tasks like automated replies whose relevancy increases with the decrease inavailable time resource. Our proposed model was able to analyze the sentiments from the corpus dataset with an accuracy of 73%.

Literature Review

Sentiment analysis has received the attention of many authors since past two decades. In recent years, many researchers have contributed towards different neural network models which are the parts of Artificial Neural Networks(ANN). These models include Convolutional Neural Networks (CNN), Recursive Neural Networks(RNN), Deep Neural Networks(DNN), Recurrent Neural Networks(RNN) and Deep Belief Networks(DBN)[1].

Convolutional Neural Networks (CNN)

The CNN (Convolutional Neural Network) [25] includes pooling layers and sophistication as it gives a standard architecture to map the sentences of variable length into sentences of fixed size scattered vectors.

The textual sentiment analysis has been done by authors[27] on twitter using deep learning. The worked on avoiding the requirement of the new feature by initializing the weight of parameters of CNN and it seemed critical to train the model. The neural language and big unsupervised group of tweets were used for initialization and training of words respectively. The sentiment analysis was done on message level and phrase levels and both resulted in average rank 1 across all test sets.

Sentiment analysis on Micro-blogs was overviewed by [28] in detail. CNN was used to get opinions and attitudes of users about hot events. The problem of explicit feature extraction was overcome by using CNN. Implicit learning was accomplished by CNN. To collect the data from the target, the input URL and focused crawler have been used, 1000 to 1500 blog/review comments were collected as a corpus and divided into three labels, i.e., neutral sentiment, negative sentiment, and positive sentiment. The proposed model has been compared with the previous studies as those had studies used CRF, SVM, and additional traditional algorithms to perform sentiment analysis at a high price. However, the performance proves that the proposed model is reasonable and sufficient to enhance the accuracy in terms of sentiment analysis.

The combined textual and visual sentiment analysis proposed by researchers of [29] motivated the need for controlling comprehensive social multimedia content. A convolutional neural network (CNN) is one of the neural networks which perform brilliantly when dealing with images. This model was a rule-based sentiment classifier called ‘VADER’. Tweets were gathered through Twitter API. The sentiment labels for the chosen tweets where made using the Mechanical Turk (AMT) and crowd intelligence. The results recommend that the joint textual-visual model has performed better than both single visual and textual sentiment analysis models.

In the study by [15], the researchers have represented a seven-layer framework to analyze the sentiments of sentences. This framework depends on CNN and Word2vec for sentiment analysis and to calculate vector representation, respectively. The Dropout technology, Normalization, and Parametric Rectified Linear Unit (PRLU), have been used to progress the correctness and generalizability of the proposed model. The framework was verified on the data set from rottentomatoes.com which contains movie review excerpts’ corpus, the dataset consists of five labels positive, somewhat positive, neutral, negative and somewhat negative. By comparing the proposed PRLU model with previously mentioned models such as Matrix-Vector recursive neural network (MV-RNN) and recursive neural network (RNN) we can observe that the proposed PRLU model outperforms the previous Matrix-Vector models with the 45.5 % accuracy.

Recursive neural network (RNN) and Recurrent neural network (Rec NN)

The Recursive Neural Network (RNN) [25] lies in supervised learning. It has a tree-like structure which settles before training and the nodes have random matrices. There is no need for reconstruction of input in RNN.

In the study [31], Sentiment Treebank has been introduced. It includes fine-grained sentiment labels for phrases in the parse trees of sentences. To address them, the Recursive Neural Tensor Network is been introduced. When the proposed model is trained on the new Treebank, this model outperforms all previously mentioned methods. The combination of new model and data results in a system for single sentence sentiment detection that pushes the state of the art by 5.4% for positive/negative sentence classification. Apart from this standard setting, the dataset also poses important new challenges and allows for new evaluation metrics. For instance, the RNTN obtains an accuracy of 80.7% on fine-grained sentiment prediction across all sentences used and also captures negation of different sentiments.

The research by [25], the idea of sentiment analysis with the help of different architectures of Recursive – Recurrent neural networks has been proposed. They separated each sentence from one another in the review and gave it to a Recursive neural network (RNN). From this, the class is decided and the average semantic vector is then analyzed by the neural network to find the sentiment of the statement. This research paper compared various techniques and the conclusion is drawn was Support Vector Machine classifier (SVM) –Linear is more accurate than RNN and RecNN ie. Recursive and Recurrent Neural Network Architecture respectively.

Deep Neural Networks (DNN)

In this study [33], the author has proposed a model for sentiment analysis considering both visual and textual contents of social networks. This new scheme used a deep neural network model such as Denoising autoencoders and skip gram which is the base scheme of CBOW (Continuous Bag-Of-Words) model. The proposed model consisted of two parts CBOW-LR (logistic regression) for textual contents analyzes which were then expanded to the CBOW-DA-LR. This model was able to classify sentiments based on the polarity of visual and textual information. Four datasets were evaluated, i.e., Sanders Corpus, Sentiment140, SemEval2013, and SentiBank Twitter dataset from which the proposed model outperformed the CBOWS+SVM and FSLM (fully supervised probabilistic language model). Perhaps the extended fully supervised probabilistic language model in term of small training data had outperformed the current model. As predicted, both feature learning and skip grams required large datasets for obtaining its optimum performance.

Deep Belief Networks (DBN)

Deep belief networks (DBNs) [38] includes several hidden layers, composed by RBM (restricted Boltzmann machines). DBN has been proved efficient for feature representation which utilizes the unlabeled datasets (or raw data) and fixes the deficiencies of labeled analysis issues.

In this paper [39], a new deep neural network structure has been presented termed as WSDNNs (Weakly Shared Deep Neural Networks). The main objective of WSDNNs is using two languages to share sentiment labels. The features of language specific and interlanguage have been presented through building multiple weakly shared layers of features. The datasets from Pretten Hofer and Stein have been used containing four languages French, German, English and Japanese. In comparison with existing studies, the proposed work addresses the challenge of shortening overlap among feature spaces of both source and target language data through cross-lingual information transfer process using backpropagation. DNNs are used for the transformation of information from the source language to the target language. This experiment has been conducted for sentiment classification for cross multilingual product reviews on many commercial websites including Amazon. From which the proposed model’s approach proved to be more effective and powerful in terms of cross-lingual sentiment classification than the previous models.

Hybrid Neural Networks

In this research study [40] a hybrid model has proposed which consists of Probabilistic Neural Network (PNN) and a two-layered Restricted Boltzmann (RBM). The main objective of proposing this hybrid deep learning model is to attain better accuracy for the process of sentiment classification. The polarity, i.e., negative and positive reviews vary according to the different context in order to solve this type of problem this model performs well, neutral reviews are not considered. Research has been undertaken regarding the same with datasets of Pang and Lee and Blitzer, et al., and binary classification was implemented on every dataset. The accuracy was enhanced for datasets by comparing them with the existing state-of-the-art Dang, et al. [13]. There are no outer resources in the proposed approach such as POS tagger and sentiment dictionary etc, therefore, it is faster too than the competitor. To attain a reduced number of features the dimensionality reduction has been implemented as the previous study used a complex strategy for feature selection.

This study [9] has proposed two deep learning techniques for the sentiment classification of Thai Twitter data, i.e., Convolutional Neural Network (CNN) and Short Term Memory (LSTM). In this study, data was collected from its users, i.e. its followers of Thai Twitter. After filtering and cleansing the data, only the users with Thai tweets and tweets with Thai characters were only selected. Consecutive experiments were conducted accordingly. Based on this research, the parameters needed for comparison of deep learning with classical techniques were analyzed and they were able to achieve the importance of words sequence. Three-fold cross-validation was used to verify the process and the results proved that the accuracy is high in DNN than in LSTM and both the techniques of deep learning are higher in accuracy than SVM and Naive Bayes but lesser than that of Maximum Entropy. Higher accuracies were found in original sentences than to that of shuffled sentences basically, the words sequence also seems to play an important role.

Other Neural Networks

In this study [41] to overcome the complexity in word level models, the character-level model has been proposed. The motivation of proposed model CDBLSTM is an existing model that is DBLSTM neural networks [42]. The goal of this research is only on textual content and on the polarity analysis of tweets in which a tweet is classified into two classes, i.e., positive and negative. The tweets encoded from this level are trained with the use of CCE (categorical cross-entropy) which is very effective when dealing with noisy data. Experiments were conducted on two datasets from which the first one is the latest benchmark dataset for SemEval 2016 and the next one was provided by the GO dataset. Several classifiers have been used such as SVMs (Support Vector Machine), RBFNet (Random Forest, Radial Basis Function Neural Network), Naive Bayes, J48 (Decision Tree), CART, JRip, Logistic Regression (LR) and Multi-layer Perceptron (MLP) methods for the process of classification of data and promising results have been found either through increase complexion or through combination of different neural network.

Observation

Based on the following implementation, we have found to observed to receive an accuracy of 72.39%. We have successfully performed the testing and training upon the bag of words and obtained the output whether it is a positive or negative review.

Positive Review

We have successfully showed that the review is positive.

Negative Review

We have successfully shown the review is negative.

Conclusion

From the following, we conclude that a higher rate of testing obtained, higher is the accuracy obtained, better performed of the system. We implement using Naive Bayes method for performance of sentimental analysis as it is ideal for a larger dataset and is able to classify the sentiment better in smaller iterations but it certainly gets better with a decrease in the learning rate and an increase in the epoch.