One of the most important features in an e-commerce retail solution is the capability to provide high-quality recommendations for buying other products/items.
A system responsible for this feature is known as a Product Recommendations (PR) system.
A good PR system can significantly increase the website profits by driving additional product sales for the existing customers and also by increasing the customer base.
Ultimately, from the end-user point of view, the product recommendation is that feature that provides this type of information:
• Customers who bought this also bought this;
• This product is usually bought together with this;
• These are related recommended products
Product recommendations systems can be found everywhere in the modern e-commerce world; below some relevant numbers (source):
- Research conducted by Barilliance in 2018 concluded that product recommendations accounted for up to 31% of ecommerce revenues. On average, customers saw 12% of their overall purchases coming from products that were recommended to them.
- A Salesforce study of product recommendations concluded that visits where the shopper clicked a recommendation comprise just 7% of total site traffic but make up 24% of orders and 26% of revenue.
- The conversion rate for visitors clicking on product recommendations was found to be 5.5x higher than for visitors who didn’t click.
- An Accenture report says personalization increases the likelihood of a prospect purchasing from you by 75 percent.
As a proof of the importance of having good product recommendation systems, a few years ago Netflix organized a challenge (the “Netflix prize”) where the goal was to produce a recommender system that performs better than its own algorithm, with a prize of 1 million dollars to win.
So let’s go together through the main concepts used in building recommendation systems, what is at the core of such systems, and why using our multi-model database – CoraDB is the right technology choice to build a product recommendations solution.
There are three main models used to design and implement a product recommendation system:
• Content-based filtering: Recommendations rely on product properties (e.g., actor/director) but ignore complex user-product relationships, usually implemented with relational databases.
• Collaborative filtering: Recommendations are based on user-item relationships and similarity to other users, accounting for complex connections, typically implemented using a graph database.
• Hybrid models combine content and collaborative filtering for best results, despite increased complexity.
The usual components we see in a product recommendation system are:
• E-commerce relational database (RDBMS).
• Graph database for collaborative PR models and fast complex queries.
• ETL system populates graph database from relational source. This means that the PR solution needs to have an ETL tool/module responsible for:
- Populating the graph database;
- Populating specific relational data structures needed for content-based PR models;
- Populating back the e-commerce relational database with the recommendations data.
Depending on how often the data is updated, there are two types of PR systems:
• Offline systems update recommendation data periodically or on demand.
• Real-time systems use graph databases and parallel processing for instant recommendation updates.
Real-time systems offer accuracy but face implementation limits due to:
• Capacity constraints.
• Data size limitations.
• High operating costs.
Create from scratch a full e-commerce solution
The assumption is that you have selected your e-commerce application solution (such as Magento) and what you need on top of it is a high-quality PR system. There are two main choices to select from:
- Populating the graph database
- Choose CoraDB as the unified backend. Use its relational engine for the core e-commerce solution and content-based filtering, and its graph capabilities for collaborative filtering PR models.
Improve your existing e-commerce solution
- Fully replace your current relational database with ArniaDB. ArniaDB is compatible with the standard SQL ANSI databases, so the replacement will go smoothly. Use ArniaDB graph database capabilities to implement the PR collaborative filtering models.
- Keep your current relational database along with your current content-based PR system and use ArniaDB graph database capabilities to implement PR collaborative filtering models.
The main differentiator between using CoraDB and other databases is the multi-model capabilities built-in in the CoraDB.
Having both relational and graph apabilities in the same unified database engine provides important advantages toward design, implementation, maintenance, performance, and overall costs.
The current market solutions design for a hybrid model is usually the following:
CoraDB simplifies hybrid design, boosting performance and maintenance by storing all PR data in a single, unpolluted database.
1. Analysis
We will do a complete analysis of your specific requirements and will recommend the selection of methods and algorithms that will best fit your needs. For content-based filtering implementations we will analyze the fit of the most used models, such as:
- Vector space models
- Probabilistic models (Bayes Classifiers)
- Decision trees
- Neural networks
SELECT c.CustomerId,
p.Model,
COUNT(p.Model) as FrequentBuy
FROM …
GROUP BY
p.Model,
c.CustomerId
ORDER BY FrequencyBuy DESC
For collaborative filtering, the analysis will target the fit for known models such as correlation-based, cosine-based, dimensionality reduction techniques, latent semantic methods, regression, clustering, neural networks, particle filtering and others.
MATCH
(customer:Customer {name: ‘Joe’})-[:BOUGHT]->(:Basket)<-[:IN]-
(product:Product)
RETURN product, count(product)
ORDER BY count(product) DESC LIMIT 10
Content Filtering
- Memory-based
- TF-IDF & cosine similarity
- Model-based
- Neural networks
- Bayesian classifiers
Collaborative Filtering
- Memory-based
- KNN & cosine similarity
- Model-based
- Latent factor model
2. Solution architecture & Design
Based on the analysis results, we will propose the best solution, and we will build the solution architecture and the system design. This includes the design of the data storage (graph database & relational database, as applicable).
As a side note, a good solution must address the typical issues associated with PR systems:
- Cold-start
- Data sparsity
- Scalability
- Trust
- Synonymity
3. Development of the ETL module(s)
For both models, in particular for the collaborative filtering model which requires a graph database back-end, we need to select, prepare, and load the data needed to run the algorithms (i.e. populate the dedicated graph and relational storages). This is achieved by implementing an ETL (Extract-Transform-Load) module/application.
The ETL solution will be executed at the scheduled periods or continuously, for real-time solutions. For example, it is very important to have an ETL batch process that will calculate in parallel the various similarities between the system entities in scope (such as users, items) when you scale to big data (millions of entities), in order to provide fast response times.
4. Product recommendations engine implementation
For both models, in particular for the collaborative filtering model which requires a graph database back-end, we need to select, prepare, and load the data needed to run the algorithms (i.e. populate the dedicated graph and relational storages). This is achieved by implementing an ETL (Extract-Transform-Load) module/application.
- Collaborative module implementation: Graph-based algorithms implementation.
- Content module implementation. Relational (or NoSQL) algorithms implementation.
In this phase we will implement the product recommendations engine, using one or multiple (hybrid) model(s).
5. Solution maintenance and support
Our product team will ensure proper system functionality, periodic updates, solution improvements and evolution.
Not the least, as part of the solution, we will implement tools to support specific metrics and efficiency data analysis. The quality of the PR system must be constantly measured, in order to be able to act for improvements.
It is not just users and products/items that can be used in a recommendation system.
A recommendation system is a more general domain than just what we see in e-commerce product recommendation solutions – it is any system that provides a recommendation, prediction, opinion, or user-configured list of items that assists the user in evaluating items.
What is important is that the same theory, models, techniques, and solutions used for e-commerce PR systems can be applied in other domains and scenarios, beyond the typical e-commerce online store, such as for:
• Data mining
• Business intelligence
• Online advertising
• Content recommenders for social media platforms
• Web content recommenders
• Restaurants
• Online dating
• Media streaming
Recommendation prediction is inexact, dependent on factors
such as:
- Weather
- Geography and culture
- Product description keywords
- Fashion tendencies
All these can influence the quality of the recommendations, which is measured by the sales volume increase. To build a more flexible PR system that can learn by itself over time and make connections that are not immediately spotted by the usual models/algorithms, more and more we see Machine Learning and Artificial Intelligence engines added to the PR systems.
We can put to work our rich experience in designing and building ML/AI solutions to create or enhance new or existing recommendations systems.