Learning to rank (software, datasets)

Posted 2020-09-30 energy1010

Datasets for ranking (LETOR datasets)

• MSLR-WEB10k and MSLR-WEB30k You’ll need much patience to download it, since Microsoft’s server seeds with the speed of 1 Mbit or even slower.
  The only difference between these two datasets is the number of queries (10000 and 30000 respectively). They contain 136 columns, mostly filled with different term frequencies and so on. (but the text of query and document are available)

• Apart from these datasets,  LETOR3.0 and LETOR 4.0 are available, which were published in 2008 and 2009. Those datasets are smaller. From LETOR4.0 MQ-2007 and MQ-2008 are interesting (46 features there).  MQ stays for million queries.

• Yahoo! LETOR dataset, from challenge organized in 2010. There are currently two versions: 1.0(400Mb) and 2.0 (600Mb). Here is more info about two sets within this data

There is also

Yandex imat’2009

(Интернет-Математика 2009) dataset, which is rather small. (~100000 query-pairs in test and the same in train, 245 features). 

 

 

Algorithms

There are plenty of algorithms on wiki and their modifications created specially for LETOR (with papers).

Implementations

There are many algorithms developed, but checking most of them is real problem, because there is no available implementation one can try. But constantly new algorithms appear and their developers claim that new algorithm provides best results on all (or almost all) datasets.

This of course hardly believable, specially provided that most researchers don’t publish code of their algorithms. In theory,  one shall publish not only the code of algorithms, but the whole code of experiment.

However, there are some algorithms that are available (apart from regression, of course).

1. LEMUR.Ranklib project incorporates many algorithms in C++ http://sourceforge.net/projects/lemur/ the best option unless you need implementation of something specific. Currently contains
   MART (=GBRT), RankNet, RankBoost, AdaRank, Coordinate Ascent, LambdaMART and ListNet
2. LEROT: written in python online learning to rank framework. Also there is less detailed, butlonger list of datasets: https://bitbucket.org/ilps/lerot#rst-header-data
3. IPython demo on learning to rank
4. Implementation of LambdaRank (in python specially for kaggle ranking competition)
5. xapian-letor is part of xapian project, this library was developed at GSoC 2014. Though I haven’t found anythong on ranking in documentation, some implementations can be found in C++ code: https://github.com/xapian/xapian/tree/master/xapian-letor https://github.com/v-hasu/xapian/tree/master/xapian-letor

 

• Metric learning to rank (mlr) for matlab
• SVM-Rank implementation in C++
• ListMLE, ListNET rankers (probably these were used in xapian)
• SVM-MAP implementation in C++

Comparison from http://www.ke.tu-darmstadt.de/events/PL-12/papers/07-busa-fekete.pdf,
though paper was about comparison of nDCG implementations.