Learners : Matchbox recommender : Command-line runners

Evaluators

Evaluators exist for each one of the four prediction modes - rating prediction, item recommendation, find related users and find related items.

Rating prediction evaluator

Rating prediction can be evaluated using the EvaluateRatingPrediction argument to Learner Recommender. It takes in a test set and predictions for this test set, and produces a report of evaluation metrics, which include mean absolute error (MAE), and root mean squared error (RMSE). These metrics as well as the way the evaluator works are explained in the corresponding evaluation API section.

Required parameters

  • test-data - dataset to make predictions for; typically produced by the command-line data splitter
  • predictions - predictions to evaluate; typically produced by Learner Recommender PredictRatings
  • report - evaluation report file

Example

Learner Recommender EvaluateRatingPrediction --test-data TestSet.dat
                                             --predictions RatingPredictions.dat 
                                             --report RatingPredictionEvaluation.txt

Item recommendation evaluator

Item recommendations can be evaluated using EvaluateItemRecommendation argument to Learner Recommender.  It takes in a test set and predictions for this test set, and produces a report which specifies the computed normalized discounted cumulative gain (NDCG). This metric as well as the way the evaluator works is explained in the corresponding evaluation API section.

Required parameters

  • test-data - dataset to make predictions for; typically produced by the command-line data splitter
  • predictions - predictions to evaluate; typically produced by Learner Recommender RecommendItems
  • report - evaluation report file

Example

Learner Recommender EvaluateItemRecommendation --test-data TestSet.dat
                                               --predictions ItemRecommendations.dat 
                                               --report ItemRecommendationEvaluation.txt

Related user evaluator

Related users can be evaluated using the EvaluateFindRelatedUsers argument to Learner Recommender. It takes in a test set and predictions for this test set, and produces a report of evaluation metrics, which include NDCG with L1 and L2 similarities as gains. These metrics as well as the way the evaluator works are explained in the corresponding evaluation API section.

Required parameters

  • test-data - dataset to make predictions for; typically produced by the command-line data splitter
  • predictions - predictions to evaluate; typically produced by Learner Recommender FindRelatedUsers
  • report - evaluation report file

Optional parameters

  • min-common-items - minimum number of items that the query user and the related user should have rated in common; defaults to 5

Example

Learner Recommender EvaluateFindRelatedUsers --test-data TestSet.dat
                                             --predictions RelatedUsers.dat 
                                             --report RelatedUserEvaluation.txt

Related item evaluator

Related items can be evaluated using the EvaluateFindRelatedItems argument to Learner Recommender. It takes in a test set and predictions for this test set, and produces a report of evaluation metrics, which include NDCG with L1 and L2 similarities as gains. These metrics as well as the way the evaluator works are explained in the corresponding evaluation API section.

Required parameters

  • test-data - dataset to make predictions for; typically produced by the command-line data splitter
  • predictions - predictions to evaluate; typically produced by Learner Recommender FindRelatedItems
  • report - evaluation report file

Optional parameters

  • min-common-items - minimum number of users that the query item and the related item should have been rated by in common; defaults to 5

Example

Learner Recommender EvaluateFindRelatedItems --test-data TestSet.dat
                                             --predictions RelatedItems.dat 
                                             --report RelatedItemEvaluation.txt

©2009-2015 Microsoft Corporation. All rights reserved.  Terms of Use | Trademarks | Privacy Statement