Predicting Trends

Predicting Trends Beforehand

In this age of constant advertisement and brand placement, trending topics on Twitter have become a great free way for advertisers to get their message in front of more potential customers. The only problem is that no one can predict what will be come a trending topic, at least until now. A professor at M.I.T. in conjunction with one of his students, developed an algorithm that they claim will be 95% accurate in predicting those trending topics as much four to five hours before they are trending.


Picture: Unmadindu (CC)

Via M.I.T.

At the Interdisciplinary Workshop on Information and Decision in Social Networks at MIT in November, Associate Professor Devavrat Shah and his student, Stanislav Nikolov, will present a new algorithm that can, with 95 percent accuracy, predict which topics will trend an average of an hour and a half before Twitter’s algorithm puts them on the list — and sometimes as much as four or five hours before.

The algorithm could be of great interest to Twitter, which could charge a premium for ads linked to popular topics, but it also represents a new approach to statistical analysis that could, in theory, apply to any quantity that varies over time: the duration of a bus ride, ticket sales for films, maybe even stock prices.

Shah and Nikolov’s experiments used a relatively small data set to “train” the algorithm. The team loaded 200 tweets containing topics that ended up trending and 200 that did not. After “training” the algorithm they ran it using real-time random tweets, and found that they had a 95% success rate and only a 4% false positive rate. Shah believes that this algorithm’s accuracy will increase over time as the data sets used to “train” it are increased. While this algorithm doesn’t seem to have the privacy implications as William Binney’s algorithms used by the NSA do, it is worth noting that it could be applied to any data set with regular updates such as Facebook or G+, and there is every reason to believe that it will be.