Summary of Learning to Rank with Apache Spark: A Case Study in Production ML with Adam Davidson & Anna Bladzich

00:00:00 - 00:25:00

In the video, Apache Spark is used to learn to rank articles. The data used in the algorithm is from daily web browsing logs. The first step is to remove any traffic generated through the collaborative filtering algorithm's own recommendations. The next step is to generate item-based collaborative filtering recommendations. The final step is to hold on to the collaborative filtering score and look at popularity of articles as determined by the number of people who have downloaded or viewed an article on ScienceDirect.

• 00:00:00 Apache Spark is used to develop a recommendation engine for ScienceDirect, a website with over 15 million articles and millions of unique users. The Spark recommendation engine is collaborative filtering and learning to rank.
• 00:05:00 In this video, Apache Spark is used to improve the quality of recommendations given by a collaborative filtering algorithm. The data used in the algorithm is from daily web browsing logs. The first step is to remove any traffic generated through the collaborative filtering algorithm's own recommendations. The next step is to generate item-based collaborative filtering recommendations. The final step is to hold on to the collaborative filtering score and look at popularity of articles as determined by the number of people who have downloaded or viewed an article on ScienceDirect.
• 00:10:00 Apache Spark is used to learn to rank articles, which is important for recommending content to users. The model takes features associated with the article, the recommendation, and the user's behavior, and predicts the relevance of the recommendation.
• 00:15:00 The video describes how Apache Spark can be used to learn to rank with a model that is able to predict the relevance of recommendations for a particular article. The video describes how the system works, including an a/b test that showed a statistically significant improvement in recommendations using the LTR model.
• 00:20:00 In this presentation, Apache Spark is used to perform collaborative filtering and rank recommendations. The presentation also includes an a/b test of the models and a description of how Spark enables collaboration between engineers and scientists.
• 00:25:00 The video discusses how Apache Spark can be used to learn to rank articles, and how the learning to rank model can be improved by considering the user's behavior.