Download Harvard Course Recommender System and more Slides Humanities in PDF only on Docsity!
Harvard Course Recommender System
Justin Kaashoek, Nicolas Lepore, Natalie Margulies
CS136, November to December 2019
1 Introduction
Recommender systems are prevalent throughout our daily lives, from YouTube recom- mending what video to watch next, to Spotify recommending what new song to listen to. Still, many areas remain that could make use of recommendation systems, one of which is a Course Recommendation system. The idea of course recommendation systems is not a novel idea, but it has yet to be applied to Harvard. In this paper, we propose different approaches to create a course recommenda- tion system that is specific to Harvard College. We believe this would be a valuable resource to Harvard undergraduates. There are many courses available to under- graduates, and it can often be overwhelming trying to select classes every semester. Students must also seek to fulfill requirements every semester, which adds another layer of difficulty in selecting courses. Especially considering the transition to dif- ferent General Education requirements, students are often unsure what requirements they have yet to fulfill. With a course recommendation system, we seek to alleviate some of the stress of choosing classes. We hope to be able to eventually create a system that recommends classes to students that a student would find interesting while fulfilling the necessary requirements to graduate. To accomplish this goal, we first discuss a theoretical implementation of a course recommendation system at Harvard. We discuss a recommendation system that is purely content-based and recommends classes based on a student’s previous courses. We then discuss a system that is purely collaborative-based, where classes are rec- ommended to a student based on how other students have rated similar classes. We conclude the section by discussing the theory behind a hybrid implementation of the two models as well as expectations and potential issues of an actual implementation. We then seek to implement a recommendation system based on self-generated data. We implement these purely content-based and a purely collaborative-based system, leaving a hybrid system for future work. Finally, we evaluate the results of each.
2 Theoretical
In this section, we discuss the theoretical implementation of a course recommendation system for Harvard.
2.1 Content-Based using Previous Courses
In a content-based recommender, user i’s rating of class j is estimated based on finding classes that are similar to classes that user i likes (or rates highly) and dissimilar to classes that user i dislikes (or rates poorly). Therefore, we can use a student’s previous courses, along with their ratings for such courses, to determine an effective recommendation of future courses. In this content-based approach, we can define each class to have a content profile wj = (wj 1 , wj 2 , ..., wjK ) ∈ [0, 1]K^ , for K different attributes, and each attribute k has a weight wjk in the range [0,1], indicating the degree to which attribute k is associated with class j. For example, the attributes could represent the subject of a course, where the weights are 1 if the subjects match. We chose the following attributes for our content profile: the subject (STEM or humanities), department the course falls under, whether there’s a lab component, lectures per week, time slot, difficulty level, class size, whether it has a final exam or final project, number of midterms, and number of homework assignments, as well as a few other attributes.
2.1.1 Similarity Heuristic
Each user i can be modeled to have their own content profile wi = (wi 1 , wi 2 , ..., wiK ) ∈ [0, 1]K^ , and each attribute k has a weight wik in the range [0,1], indicating the degree to which user i likes classes with attribute k. To build our content user profile, we could survey a student about which subjects they enjoy, or infer based on their concentration. We could also use their previous ratings to infer their content profile by looking at their Q scores for each class that fits a particular genre and counting the number of likes vs. dislikes to construct a 0- vector to describe a user by threshold entries. High like-to-dislike would map to a 1 and low like-to-dislike would map to a 0. We then estimate the rating ˆrij on an class j for user i based on the distance between the user and class content profiles, wi and wj. We can use the cosine similarity measure: rˆij = cos(wi, wj ) =
wi · wj ||wi|| 2 ||wj || 2
where the numerator is the dot product and the denominator is the product of each Euclidean distance.
- Estimate user i’s rating for item j as the similarity-weighted average rating on item j by users in neighborhood Ni, by
rˆij =
∑ i′∈Ni sim(i, i ′)ri′,j ∑ i′∈Ni |sim(i, i′)|
where sim(i, i′) is the similarity measure between users i and i′.
In this way, we can create a neighborhood of like-minded students, and we can test different neighborhood sizes to see which is the best. We modeled this problem for a small number of students, but it is also interesting to explore how this scales to a few thousand students. We can use optimization techniques to decide the best neighborhood size, and from there, create accurate ratings.
2.2.2 Item-Item
For item-item collaborative filtering, we estimate user i’s rating on class j in the following way:
- Find the neighborhood Nj of classes, which consists of classes that are similar to class j, and have been rated by user i.
- Estimate user i’s rating for class j as the similarity-weighted average rating by user i on the classes in neighborhood Nj ,
ˆrij =
∑ j′∈Nj sim(j, j ′)ri,j′ ∑ j′∈Nj |sim(j, j′)|
where sim(j, j′) is the similarity measure between classes j and j′.
Here, we can create a neighborhood of similar courses, and we can test different neighborhood sizes to see which is the best. We modeled this problem for a small number of courses, but it is also interesting to explore how this scales to a few hundred courses. We can use optimization techniques to decide the best neighborhood size, and from there, create accurate ratings. An issue here is that courses are can be more uniquely different than students. Each course at Harvard is meant to cover a different topic, so it is sometimes hard to draw similarities between them, even if they are in the same department. On the other hand, many students can have very similar courseloads, and this helps us figure out the kind of neighborhood they fall into.
2.3 Hybrid Approach
There are four typical approaches in combining content-based and collaborative fil- tering in recommendation systems.
- Ensemble approach: We take some combination of recommendations from the different systems. We could also use different systems in different situations. For class recommendations, this could look like recommending the class that has the highest average recommendation out of the following: a content-based recom- mendation using a cosine similarity heuristic, a content-based recommendation system using a Naive Bayes Classifier, and a user-user collaborative filtering recommendation.
- Calculate a recommendation based on student-student collaborative filtering where student profiles are content-based, or do the opposite, and calculate course-course collaborative filtering where course profiles are content-based.
- Hard code domain knowledge into the recommendation system. For example, that stem majors may not like humanities classes as much.
- Combine course attributes and student profiles in a single framework and predict how much a student will like a class using a deep learning approach.
It is unclear which of these approaches would be best in practice.
2.4 Incorporating Requirements
Any of the systems previously discussed can easily be adapted to take into account requirements by removing classes from the recommendations that fulfill a requirement that has already been fulfilled. A drawback to this approach is that we may remove classes that a student would find very interesting. We could solve this problem by allowing a user to input a preference into the system that toggles whether we factor requirements into the recommendation. If the student wants to see only interesting classes regardless of requirements, we run the system as normal. If they want to take required classes that semester, we remove any classes that do not fulfill a requirement that the student has not already fulfilled.
Another option to incorporate requirements is as follows. Firstly, we can eliminate all classes that a student has taken before, similar to how Netflix does not recommend movies that have already been viewed. Then, in the content-based approach, we can give their content profile a value of 0 for attributes that represent that a class fulfills a certain GenEd. For concentration requirement classes that are typical to take in freshman or sophomore year, we can add an extra weighting in favor of these classes for underclassmen. In addition, we must factor in pre-requirements for other classes, and only show classes in which a student satisfies the pre-reqs. We can even show the classes that then become available once a student fulfills pre-requirements.
effect that Amazon observed in its recommender system. This may not be advantageous for Harvard’s business model, and could hurt academia. However, the Netflix recommender system accounts for this by randomly rec- ommending rather serendipitous titles, and our recommender system could do this as well.
3 Implementation
3.1 Algorithm
We implemented this theoretical framework as described below.
3.1.1 Content-Based Implementation
We created class and student data for our content-based approach. The class data was randomly assigned the attributes stem/humanities, department, level (0, 100s, 200s), class size, time slot (just categorical values of the order of time slot), lab component (whether it has a lab component), times per week, final exam (whether it has a final exam), midterms (number of midterms), final project (whether it has a final project), and homeworks (number of homeworks). We then put these values in the range of [0,1] by using dummy variables if the attribute was categorical, and a range between [0,1] if it was quantitative. The student data was generated based on sample Q Guide data that we randomly generated. We initially assign random weights for all attributes that classes are rated on, showing with what intensity they value a certain attribute. We then update the random weights based on two factors:
- Major: If the student is declared as a STEM major, we assign a higher weight for STEM classes. If they are a humanities or social science major, we assign a lower weight.
- GPA: If the student has a low GPA, we assign a higher weight for classes below the 200 level. We reason that students who are doing poorly in their classes will not want to take difficult upper-level classes. We assume that students with a high GPA could value classes at all levels equally.
There are a number of other factors that we could have considered to manipulate weights, but we believe these two to be the most important. Lastly we used cosine similarity to see the best matches between students and classes, and we showed which classes we would recommend to students in descending order.
3.1.2 Collaborative Filtering Implementation
We randomly created a ratings matrix, and replaced 0’s with NaNs to signify that the user had not rated that item. For user-user (student-student) and item-item (course-course), we centered and normalized our ratings to obtain our baseline rating, then we compared these baseline ratings to other students for user-user and other courses for item-item. After finding the neighborhood in order of similarity, we took the closest few neighbors to calculate the ratings, which was the sum of the similarities between our user/item and the neighbor user/item multiplied by their rating. We then divided this product by the sum of absolute value similarity between our user/item and the neighbor user/item, following the formula given in Section 2.2.1 and 2.2.2. This gave us our rating for a specific user and item.
3.2 Results: Analysis of Algorithm Accuracy
We analyzed the efficacy of our each recommender systems by testing the theoretical recomender values against the randomly generated actual data.
3.2.1 Accuracy of Content-Based Filtering
Unfortunately, analyzing the efficacy of content-based filtering with given weights wj = (wj, 1 , wj, 2 , ...) is largely impossible without verified data because we don’t know the weights wj that each student assigns each criterion. We found that modifying weights based on concentration and GPA had no impact on recommendations. This indicates a shortcoming of our content-based recommendations and is likely due to the fact that there are so many class attributes. Changing only a few weights, then, has limited impact on the final outcome. Given limitations in Q Guide data collection and bias in survey data, we are unable to produce a verifiable actual ranking of classes for each student in the randomly generated data set. Verification of content-based filtering would require a deeper understanding of the weights each student ascribes to each descriptor.
3.2.2 Accuracy of Collaborative Filtering
Although we didn’t have verified historic data to assess the accuracy of collaborative filtering rankings, we decided to compare the original rankings with the projected outputs of both collaborative filtering models. We first converted each of the rank profiles into a weak preference ordering and then compared the accuracy of two sets of rankings by taking the square root of the average of the squared deviations of the ranking positions from their final value. We compare the efficacy of the recommen- dation in the collaborative filtering recommendation system across niterations = 15
3.2.3 Comparison of User-User and Item-Item Collaborative Filtering
We compare the efficacy of in recommendation between user-user and item-item col- laborative filtering recommendation system across niterations = 15 on a randomly generated sample size of n = 10 students in order to test which algorithm gives more consistent results with the actual ranking. Since rating were randomly generated in range (1, 5) with equal probability, we would expect user-user and item-item collab- orative filtering to yield similar mean rankings and mean differences across multiple bootstrap trials by the Law of Large Numbers. Figure 2 shows that the mean ratings across both collaborative filtering systems are consistent across multiple trials. The mean difference between predicted and actual rankings straddles 0, with an average difference of 0.112 across all iterations.
Figure 2: Comparison of Mean Difference and Mean Ratings across User-User and Item-Item Based Collaborative Filtering
3.3 Results: Theoretical Implementation
In 3.2, we used all theoretical rankings produced by collaborative filtering in order to verify the accuracy of the algorithm. In this section, we walk through a theoretical implementation of how the algorithm would work. Once we generate predictions of all classes, we would use only the NaN predictions to generate a preference ordering for classes the students has not taken as shown below in Figure 3 through red highlighting of untaken classes.
Figure 3: Conversions of NaNs to Rankings
We see that user-user and item-item recommendations generally agree with each other, which is as expected given our results in section 3.2.3.
4 Conclusion
We are successfully able to provide course recommendations for content-based and collaborative filtering systems. Although it is difficult to draw any meaningful con- clusions based on these recommendations, the fact that we were able to generate this system is a success in itself. The system is a long way from being used in practice, however. It is possible that the separate content-based and collaborative filtering sys- tems would work well independently, but in practice, we would want to use a hybrid approach.
One of the biggest issues that we ran into was the lack of data. We spent a lot of time generating our own data, hoping to make it somewhat meaningful, whereas if we had available data from the Q Guide or our own survey of students, we could have made major improvements to the system. Before a system such as this is used in practice, we would have to test it on such data. Despite these shortcomings, we believe that we have laid both the theoretical and practical groundwork for a system that could actually be used in the future.
