engleski

A Topic Coverage Approach to Evaluation of Topic Models

Topic models are widely used unsupervised models capable of learning topics – weightedlists of words and documents – from large collections of text documents. When topic models are used fordiscovery of topics in text collections, a question that arises naturally is how well the model-induced topicscorrespond to topics of interest to the analyst. In this paper we revisit and extend a so far neglected approachto topic model evaluation based on measuring topic coverage – computationally matching model topics witha set of reference topics that models are expected to uncover. The approach is well suited for analyzingmodels’ performance in topic discovery and for large-scale analysis of both topic models and measures ofmodel quality. We propose new measures of coverage and evaluate, in a series of experiments, different typesof topic models on two distinct text domains for which interest for topic discovery exists. The experimentsinclude evaluation of model quality, analysis of coverage of distinct topic categories, and the analysis of therelationship between coverage and other methods of topic model evaluation. The paper contributes a newsupervised measure of coverage, and the first unsupervised measure of coverage. The supervised measureachieves topic matching accuracy close to human agreement. The unsupervised measure correlates highlywith the supervised one (Spearman’s ρ≥ 0.95). Other contributions include insights into both topic modelsand different methods of model evaluation, and the datasets and code for facilitating future research on topiccoverage.

Topic coverage ; Topic coherence ; Topic discovery ; Topic models ; Topic model evaluation ; Topic model stability

nije evidentirano