I collected almost one million tweets on six US stocks, turned them into daily sentiment signals with Doc2Vec and VADER, and added these signals to the HAR model, a standard benchmark for forecasting realized volatility. For the two most-discussed stocks, the Doc2Vec signal cut forecast errors by 25–33%.
01 / Overview
Stock market volatility is notoriously hard to predict. The HAR-RV model is a widely used benchmark: it forecasts tomorrow's realized volatility from the average volatility of the past day, week, and month.
Investor attention and opinion move markets, and much of that discussion happens on social media. This project asks whether a sentiment signal extracted from tweets, added as a fourth component, improves the one-day-ahead forecast.
02 / Approach
A three-step pipeline bridges raw social media text and a structured time-series model. A dictionary-based VADER feature serves as a simple comparison.
Doc2Vec converts every tweet into a dense 10-dimensional vector; the vectors of a trading day are averaged into one daily vector.
XGBoost maps the previous day's vector to a single sentiment score by predicting the next day's realized volatility.
The score enters the HAR model as a fourth regressor. All models are re-estimated after each out-of-sample prediction.
03 / Data
Using the Twitter API (tweepy), I collected all tweets mentioning each stock's cashtag (e.g. $AAPL) between December 2, 2020 and March 26, 2021. Each trading day covers the tweets from one closing time to the next.
Target variable
Daily realized volatility is computed from intraday prices at 15-minute intervals: the square root of the sum of the 26 squared log returns of a trading day.
04 / Sentiment Features
A fixed sentiment dictionary versus embeddings learned from the tweets themselves.
Dictionary approach
VADER assigns each tweet a compound sentiment score between −1 and 1, based on a fixed lexicon that also accounts for emojis, punctuation, and capitalization.
A daily average would be misleading: both very positive and very negative days can signal high volatility. The feature is therefore the variance of the compound scores within a trading day, a measure of disagreement.
Learned embeddings
Doc2Vec extends Word2Vec with a document vector, learning a dense representation of each tweet from the corpus itself. Two variants were compared:
05 / Forecasting Model
The HAR model regresses tomorrow's realized volatility on its daily, weekly (5-day), and monthly (22-day) averages. The Sentiment-HAR adds yesterday's sentiment score F:
F is either the VADER variance or the XGBoost score from the PV-DM or PV-DBOW vectors. For XGBoost, the true value is added to the training set after every prediction, so the model is continuously re-fitted.
Evaluation
Each sentiment model is compared to the plain HAR model and to a HAR model with trading volume. Accuracy is measured with the root mean squared and mean absolute forecast errors (RMSFE, MAFE).
06 / Results
PV-DBOW delivers the largest improvements for the two most-tweeted stocks. For Apple, RMSFE drops by 25.5% and MAFE by 27.9%. For Alphabet, RMSFE drops by 30% and MAFE by 33.2%.
VADER and trading volume bring marginal gains at best. For stocks with fewer than 40k tweets, no text-based model improves significantly on the benchmark.
RMSFE by model
| Stock | HAR | VADER | PV-DBOW | PV-DM | Tweets |
|---|---|---|---|---|---|
| Apple | 0.00886 | 0.00866 | 0.00660 | 0.00747 | 534k |
| Alphabet | 0.00737 | 0.00761 | 0.00513 | 0.00586 | 273k |
| Nvidia | 0.01634 | 0.01655 | 0.01751 | 0.01704 | 78k |
| J&J | 0.00409 | 0.00458 | 0.00398 | 0.00395 | 39k |
| Mastercard | 0.00690 | 0.00668 | 0.00721 | 0.00701 | 20k |
| Visa | 0.00652 | 0.00655 | 0.00790 | 0.00984 | 19k |
Highlighted: lowest RMSFE per stock. Lower is better.
07 / Conclusion
Twitter sentiment can improve realized volatility forecasts, but only when enough tweets are available for Doc2Vec to learn meaningful embeddings.
Limitations and next steps