04-08-2011, 03:42 PM
Text Mining
The explosion of on-line information has given rise to many query based search engines and manually constructed topic hierarchies. But with the current growth rate in the amount of information, query results grow incomprehensibly large and manual classification in topic hierarchies creates an immense bottleneck. Search engines return millions of relevant sites but sites referring to similar content are not grouped. Cluster search, groups similar sites, giving users a greater chance of finding more sites relevant to their search.
In this dissertation, we address these problems with a system for topical information space navigation that combines the query-based and taxonomic approaches. Our system Racimo enables the creation of dynamic hierarchical document clustering based on full text of articles. A major challenge in document clustering is the extremely high dimensionality. For example, the vocabulary for a document set can easily be thousands of words. On the other hand, each document often contains a small fraction of words in the vocabulary. These features require special handlings. Another requirement is hierarchical clustering where clustered documents can be browsed according to the increasing specificity of topics. In this system, we propose to use the notion of frequent itemsets, which comes from association rule mining, for document clustering. The intuition of our clustering criterion is that each cluster is identified by some common words, called frequent itemsets, for the documents in the cluster. Frequent itemsets are also used to produce a hierarchical topic tree for clusters. By focusing on frequent items, the dimensionality of the document set is drastically reduced. We show that this method outperforms best existing methods in terms of both clustering accuracy and scalability.
