org.galagosearch.core.eval
Class RetrievalEvaluator

java.lang.Object
  org.galagosearch.core.eval.RetrievalEvaluator

public class RetrievalEvaluator
extends java.lang.Object

A retrieval evaluator object computes a variety of standard information retrieval metrics commonly used in TREC, including binary preference (BPREF), geometric mean average precision (GMAP), mean average precision (MAP), and standard precision and recall. In addition, the object gives access to the relevant documents that were found, and the relevant documents that were missed.

BPREF is defined in Buckley and Voorhees, "Retrieval Evaluation with Incomplete Information", SIGIR 2004.

Author:

Trevor Strohman

Nested Class Summary

static class	RetrievalEvaluator.Document This class represents a document returned by a retrieval system.
static class	RetrievalEvaluator.Judgment This class represents a relevance judgment of a particular document for a specific query.

Constructor Summary

RetrievalEvaluator(java.lang.String queryName, java.util.List<RetrievalEvaluator.Document> retrieved, java.util.Collection<RetrievalEvaluator.Judgment> judgments)
Creates a new instance of RetrievalEvaluator

Method Summary

double	averagePrecision() Returns the average precision of the query.
double	binaryPreference() The binary preference measure, as presented in Buckley, Voorhees "Retrieval Evaluation with Incomplete Information", SIGIR 2004.
java.util.ArrayList<RetrievalEvaluator.Document>	irrelevantRetrievedDocuments() This method returns a list of all documents that were retrieved but assumed to be irrelevant.
java.util.ArrayList<RetrievalEvaluator.Document>	judgedIrrelevantRetrievedDocuments()
protected double	normalizationTermNDCG(int documentsRetrieved)
double	normalizedDiscountedCumulativeGain() Normalized Discounted Cumulative Gain
double	normalizedDiscountedCumulativeGain(int documentsRetrieved) Normalized Discounted Cumulative Gain
double	precision(int documentsRetrieved) Returns the precision of the retrieval at a given number of documents retrieved.
java.lang.String	queryName() Returns the name of the query represented by this evaluator.
double	recall(int documentsRetrieved) Returns the recall of the retrieval at a given number of documents retrieved.
double	reciprocalRank() Returns the reciprocal of the rank of the first relevant document retrieved, or zero if no relevant documents were retrieved.
java.util.ArrayList<RetrievalEvaluator.Document>	relevantDocuments() Returns a list of all documents judged relevant, whether they were retrieved or not.
java.util.ArrayList<RetrievalEvaluator.Document>	relevantMissedDocuments() Returns a list of documents that were judged relevant that were not retrieved.
int	relevantRetrieved(int documentsRetrieved) The number of relevant documents retrieved at a particular rank.
java.util.ArrayList<RetrievalEvaluator.Document>	relevantRetrievedDocuments() Returns a list of retrieved documents that were judged relevant, in the order that they were retrieved.
java.util.ArrayList<RetrievalEvaluator.Document>	retrievedDocuments()
double	rPrecision() Returns the precision at the rank equal to the total number of relevant documents retrieved.

Methods inherited from class java.lang.Object

clone, equals, finalize, getClass, hashCode, notify, notifyAll, toString, wait, wait, wait

Constructor Detail

RetrievalEvaluator

public RetrievalEvaluator(java.lang.String queryName,
                          java.util.List<RetrievalEvaluator.Document> retrieved,
                          java.util.Collection<RetrievalEvaluator.Judgment> judgments)

Creates a new instance of RetrievalEvaluator

Parameters:

retrieved - A ranked list of retrieved documents.

judgments - A collection of relevance judgments.

Method Detail

queryName

public java.lang.String queryName()

Returns the name of the query represented by this evaluator.

precision

public double precision(int documentsRetrieved)

Returns the precision of the retrieval at a given number of documents retrieved. The precision is the number of relevant documents retrieved divided by the total number of documents retrieved.

Parameters:

documentsRetrieved - The evaluation rank.

recall

public double recall(int documentsRetrieved)

Returns the recall of the retrieval at a given number of documents retrieved. The recall is the number of relevant documents retrieved divided by the total number of relevant documents for the query.

Parameters:

documentsRetrieved - The evaluation rank.

rPrecision

public double rPrecision()

Returns the precision at the rank equal to the total number of relevant documents retrieved. This method is equivalent to precision(relevantDocuments().size()).

reciprocalRank

public double reciprocalRank()

Returns the reciprocal of the rank of the first relevant document retrieved, or zero if no relevant documents were retrieved.

averagePrecision

public double averagePrecision()

Returns the average precision of the query.

Suppose the precision is evaluated once at the rank of each relevant document in the retrieval. If a document is not retrieved, we assume that it was retrieved at rank infinity. The mean of all these precision values is the average precision.

binaryPreference

public double binaryPreference()

The binary preference measure, as presented in Buckley, Voorhees "Retrieval Evaluation with Incomplete Information", SIGIR 2004. This implemenation is the 'pure' version, which is the one used in Buckley's trec_eval.

The formula is: 1/R \sum_{r} 1 - |n ranked greater than r| / R where R is the number of relevant documents for this topic, and n is a member of the set of first R judged irrelevant documents retrieved.

normalizedDiscountedCumulativeGain

public double normalizedDiscountedCumulativeGain()

Normalized Discounted Cumulative Gain

This measure was introduced in Jarvelin, Kekalainen, "IR Evaluation Methods for Retrieving Highly Relevant Documents" SIGIR 2001. I copied the formula from Vassilvitskii, "Using Web-Graph Distance for Relevance Feedback in Web Search", SIGIR 2006. Score = N \sum_i (2^{r(i)} - 1) / \log(1 + i) Where N is such that the score cannot be greater than 1. We compute this by computing the DCG (unnormalized) of a perfect ranking.

normalizedDiscountedCumulativeGain

public double normalizedDiscountedCumulativeGain(int documentsRetrieved)

Normalized Discounted Cumulative Gain

This measure was introduced in Jarvelin, Kekalainen, "IR Evaluation Methods for Retrieving Highly Relevant Documents" SIGIR 2001. I copied the formula from Vassilvitskii, "Using Web-Graph Distance for Relevance Feedback in Web Search", SIGIR 2006. Score = N \sum_i (2^{r(i)} - 1) / \log(1 + i) Where N is such that the score cannot be greater than 1. We compute this by computing the DCG (unnormalized) of a perfect ranking.

normalizationTermNDCG

protected double normalizationTermNDCG(int documentsRetrieved)

relevantRetrieved

public int relevantRetrieved(int documentsRetrieved)

The number of relevant documents retrieved at a particular rank. This is equivalent to n * precision(n).

retrievedDocuments

public java.util.ArrayList<RetrievalEvaluator.Document> retrievedDocuments()

Returns:

The list of retrieved documents.

judgedIrrelevantRetrievedDocuments

public java.util.ArrayList<RetrievalEvaluator.Document> judgedIrrelevantRetrievedDocuments()

Returns:

The list of all documents retrieved that were explicitly judged irrelevant.

irrelevantRetrievedDocuments

public java.util.ArrayList<RetrievalEvaluator.Document> irrelevantRetrievedDocuments()

This method returns a list of all documents that were retrieved but assumed to be irrelevant. This includes both documents that were judged to be irrelevant and those that were not judged at all. The list is returned in retrieval order.

relevantRetrievedDocuments

public java.util.ArrayList<RetrievalEvaluator.Document> relevantRetrievedDocuments()

Returns a list of retrieved documents that were judged relevant, in the order that they were retrieved.

relevantDocuments

public java.util.ArrayList<RetrievalEvaluator.Document> relevantDocuments()

Returns a list of all documents judged relevant, whether they were retrieved or not. Documents are listed in the order they were retrieved, with those not retrieved coming last.

relevantMissedDocuments

public java.util.ArrayList<RetrievalEvaluator.Document> relevantMissedDocuments()

Returns a list of documents that were judged relevant that were not retrieved.