Are you a Foodie looking for New Cookies to try out? Better not ask an LLM

Diversity often helps in personalisation and generating more user engagement [9, 13], for example, the average number of podcasts streamed per user increased by 28.90% on Spotify by embracing diversity based recommendation strategy. Additionally, diversity also improves catalogue coverage [12] because more items are displayed to the users who may then choose to buy these items. Diversity has also been shown to aid in cold start problem for new users and newly launched products by promoting a broader range of recommendations [7, 26]. Diversity in retail recommendation systems provides several advantages. It helps counter popularity bias, ensuring that less popular items receive adequate exposure, thus preventing dissatisfaction among both sellers and customers. Additionally, diversity enhances personalization by allowing customers to explore and understand their preferences. It aligns with the long tail theory, where revenue is generated from a broad range of items, not just the popular ones. Moreover, diversity improves catalogue coverage, addressing the issue of limited representation of items.

Some works predict that Large Language Models (LLMs) will serve as recommender systems in future [10, 24], while others foresee current recommender systems to be augmented with LLMs for better recommendations [6, 14]. In either case, it is important to measure the level of diversity that an LLM's recommendations contain. In this work, we study the comparative performance of the two state-of-the-art LLMs, GPT-3.5 and GPT-4, in terms of providing more diverse recommendations. This is of significant interest to the retail organisations as it can help to gain and retain customers and generate more revenue by providing diverse recommendations to the users. The next part of the work talks about the experimentation framework, followed by the comparative results and their analysis.

Recommender systems are currently popularly used by large retail chains for suggesting products to customers based on their prior search history and helping them to do shopping from the convenience of their homes in few clicks. They help these companies in higher customer onboarding and retention and a much better and faster user experience. With the recent hype around LLMs, they are being used as part of these recommender systems. More details about the application of LLMs in recommender systems can be found out in various prior works in this context, such as [6, 14].

Now, there is a community interest and significant proof of the effectiveness of LLMs in recommender systems, and bias and diversity of recommended products is an important consideration for a retail organisation in order to give good recommendation to the customers for high customer retention, customer onboarding and higher revenues. So, it is important to find out if the recommendations provided by the state-of-the-art LLMs are diverse enough. We study the recommendation results from the two most popular LLMs – GPT-3.5 and GPT-4 for this purpose. An overview of our experimental setup is shown in Figure 1. The steps involved in this process are briefly as follows:

Data Collection: At first, necessary data needs to be collected for evaluation. We consider 4 popular retail product categories: Electronics, Home Appliances, FMCG (Fast Moving Consumer Goods) - Toiletries and FMCG - Edible. For each product category, based on some reliable sources of reference, we scrap some popular products in that category and some popular qualifiers (search adjectives for the product names). [1, 2, 3, 4, 5, 8, 11, 15, 16, 17, 18, 19, 21, 22, 25] are the data sources we have used to scrape this necessary data. The top products and corresponding qualifiers for the four different product categories have been identified from these sources and are illustrated in Table 1.

Prompt: A very small, simple yet effective prompt is used to generate responses from both GPT-3.5 and GPT4. The format of the prompt is shown below:

"Name <N> <qualifier> <product> available in market as list of (brand name, company name) tuple format."

Here, N is the size of the tuple-list, qualifiers and products are already described above.

Evaluation: For each product and the corresponding relevant qualifiers, we compute the diversity scores of the recommendations from both GPT-3.5 and GPT-4 models. We perform this evaluation for various temperatures for each product and search qualifier combination search term from temperature 0 to 2 with intervals of 0.1.

For performing the comparative evaluation of GPT-3.5 and GPT-4 models with respect to their ability to provide diverse recommendations, we conduct a 3-step experimentation.

First, for each product search term (product category with corresponding qualifier), for both GPT-3.5 and GPT-4 models, the diversity scores are computed for each temperature from 0 to 2 with an interval of 0.1. The diversity scores are averaged for each temperature for each model and plotted. The corresponding plot is illustrated in Figure 2(a). The plot clearly shows that for each temperature, GPT-3.5 model has a higher diversity score compared to GPT-4 model. This clearly indicates that GPT-3.5 model produces more diverse results, when compared temperature wise.

Next, we focused on temperature 0.4 for diversity comparison between GPT-3.5 and GPT-4. GPT-3.5 consistently showed higher diversity scores. Box plots in Figure 2(b) highlighted this, affirming GPT-3.5’s superiority in delivering diverse recommendations over GPT-4 across various statistics.

Now, we perform the diversity score comparison in terms of the number of product items in each category where each of GPT-3.5 and GPT-4 models give more recommendation. This comparative plot is shown in Figure 2(c), which again clearly depicts the superiority of GPT-3.5 compared to GPT-4 in terms of providing more diverse recommendations.

Also, we perform another experiment. Sometimes, if the original search term from the user is slightly rephrased, the diversity in the recommendations might improve significantly. We perform this analysis on some product term searches and our hypothesis proves to be true, as shown in Table 2.

Table 2 shows that for many of the product search terms, like ’Cookies’, GPT-4 model recommendations show a lower diversity compared to GPT-3.5.

This work provides an experimental framework to evaluate multiple LLMs in terms of diversity in their recommendations. It shows that GPT-4 might be superior in terms of providing more optimal recommendations and results in general but, at least in the context of diversity of retail products, GPT-3.5 surprisingly turns out to be superior. Further, it would be interesting to do more such analysis in different other product domains.