Optimized Movie Recommendation Using Hybrid Jaccard-Based K-Means Clustering Using Apache Spark

Abstract

The need for recommendations specific to each person's interests is growing every day, making recommendation systems increasingly important in people's lives. Researchers have paid close attention to the rapid expansion of e-commerce businesses and online video streaming services like Hotstar, YouTube, and Netflix. The collaborative filtering-based RS system aims to suggest such films or videos to users based on their past viewing habits. However, RS has challenges with cold start, sparsity, and scalability, which the proposed hybrid system must handle. This data is often represented using a rating matrix. These scores, however, frequently differ since some individuals provide harsher evaluations while others are more forgiving. As a result, the RS cannot suggest customized films to demanding viewers. This research suggests a collaborative filtering recommendation system based on K-means clustering, movie clustering, and normalization in order to address the problem. K-means clustering and movie recommendation are carried out using Apache Spark. First, the algorithm clusters similar movies using Jaccard distance, and then normalization is carried out to eliminate the small ratings in the utility matrix. Further, the K-means technique is implemented to recommend the top possible movies to the users.