I had another meeting with my supervisor. I presented the current state of the evaluation and we went through the gathered data. I had 11 annotators annotating 258 tweets based on their feeling of credibility for a tweet. When i went through the data, i found that most of them did not agree with the label of TweetCred. This led to another annotation of tweets by six “Power Users”. This users were more familiar with Twitter and the style of writing of tweets (@mentions, #hashtags, RT keyword). The dataset was extracted from the 258 tweets where no agreement between the user and TweetCred for the “credible / not credible” tweets was.

The data set consisted of 48 tweets and already had one annotation from the previous annotation round. To get a majority agreement the six Power Users annotated the tweets again leading to three annotations per tweet. The annotation for a tweet with the majority label was choosen. This led to following confusion matrix:

The data shows that the majority of users do not agree with the results of TweetCred. For example, altough TweetCred defines 37 as not credible, users agree only with 4, leaving 29 with ‘credible’ and 4 with a ‘dont know’.

Next task is to dig deeper why users did not agree / agree and to evaluate possible feature sets to create a classifier based on the evaluation results.