Skip to main content
Source Link
Leevo
  • 6.5k
  • 3
  • 19
  • 52

I'll go through your questions one by one:


Question - So is it correct to add it this way?

It's a potentially good approach. A potential improvement (but that's a personal opinion) is to change the overall polarity scores with something else. I think that approach is too much "bag of words" (i.e. it looses the sequence information).

Since you are familiar with Neural Networks, I would suggest you to employ RNNs. Here's why: when you are working with natural language, sometimes the order of words is more important than the meaning of the word itself. Sometimes the order of words itself determines their meaning! That's why NLP achieves state-of-the-art performances when models that are able to understand sequences are employed (such as RNNs).


I could also convert these polarities to binary values to mark if the overall sentiment is positive, negative or neutral using 0 and 1.

That's a great point. In this way, you can train an RNN to read the sequence, and based on both word representations and their sequence, you can come to a classification.

The model I have in mind would start with recurrent layers (LSTM or GRU), that would learn a representation of the sequential information. Later, this information would be passed to dense layers, that can then perform the classification task.


Note - I am representing 3 categories(positive, negative or neutral) using 2 columns to avoid dummy variable trap.

If you are working with Neural Networks, I suggest you to use three output nodes with a softmax activation function (for the last layer).


But am I doing it right? If not, kindly let me know how can I collaborate the two mentioned above, i.e. NLTK Vader results with Machine Learning classifiers.

Your model can be good, I only suggested potential improvements.

Another thing I'd suggest is to use word embeddings, such as word2vec or Glove, instead of tf-idf representation of words. That is because Neural Networks love dense vectors, while they tend to work not so well on sparse vectors (such as tf-idf). You can find pre-trained datasets online just googling them.


Good luck with your task!