High level of agreement across different news domain quality ratings

One prominent approach to quantifying the amount of misinformation people share is by looking at the quality of the news domains they share. However, different fact-checkers and researchers have produced different sets of news domain quality ratings, raising questions about the degree of agreement across these ratings. Here, we compared 6 sets of ratings and found that they generally correlated highly with one another. To create a comprehensive set of domain ratings that researchers can use, we used an ensemble “wisdom of experts” approach by performing imputation together with principal component analysis to generate aggregate ratings. Our new set of ratings comprises 11,520 domains—the most extensive coverage to date—and it correlates well with other rating sets with limited coverage. These results suggest that experts generally agree on the quality of domains and our new set of ratings offers a tool for evaluating content quality and the efficacy of misinformation interventions.