Understanding the customer base goes a long way in powering a number of recommendation models throughout our apps.

Personalisation models make use of various signals and behaviors to build an understanding of user preferences while making sure business criteria are also met. Let us explore what entails such an exploration.

In the previous blog, we discussed key components that make up a personalisation setup with the sole purpose of improving customer experience and helping our business bottomline. Customers are at the very center of any business. At Delivery Hero they are our heroes and no personalisation journey can work without understanding the customers. Let’s dive deeper into this topic and how it shaped our personalisation journey.

It is very important to understand and answer questions related to users’ behavior, motivations, preferences and requirements. Exploratory analysis is a key step which powers not just the initial set of tasks but is an on-going activity which helps throughout the journey. This step generates a number of key insights, ideas and helps identify potential challenges which we try to handle in subsequent phases as the system matures.

To understand how this helps, let us first talk about how the exploratory analysis works out. The most widely used and obvious method is to leverage all the data we have about our customers. This can be in the form of transactions, the kind of items they order, their restaurant and price point preferences and more. This form of analysis makes use of implicit signals which we leverage as feedback to understand different customer behaviors. This also helps in developing personas and categorization of usage patterns.

“Customers in Singapore and Thailand Love 💖 Indian vendors “ for late-night dinners

The second way of analyzing customer behavior is through explicit feedback. When customers use our apps, they are presented with the opportunity to rate the packaging, delivery, rider, restaurant/vendor and more such characteristics. They can also provide textual feedback in the form of reviews. These explicit signals are even more impactful towards building an understanding of the customer base. Most modern recommendation systems make use of both implicit and explicit feedback to improve overall experience.

“Malaysian customers just Love💖 Burger vendors for their evenings”

The final piece in the list for methods for exploratory analysis is User Research (UXR). This is by far the most meticulous and tedious task but it provides a direct connection with our customers. This is also very costly to conduct, in terms of time, effort and monetary aspects. At Delivery Hero, we typically run multiple UXR studies to understand how our customers use the app, what they use it for, their major pain points and what they love :).

A combination of these three analysis methods helped us develop an understanding of our customers.  For instance, UXR studies helped us understand that our customers in certain parts of the world just love bubble teas at certain times of the day over other beverages. This helped us prioritize the visibility of bubble tea serving locations to help our customers find the best ones quickly. Similarly, analysis of implicit signals pointed to hyper-local preference of restaurants. Helping us have a more focused approach to improve our recommendations.

On an average, most customers just love💖 to order again from vendors they have already ordered from #nostalgia #cravings

EDA’s are not just helpful in understanding customer behavior. We analyze data to improve upon existing models and system behaviors as well. This acts as a feedback loop to ensure our models are indeed working as intended and to identify gaps to improve further. The following is a high-level overview of a recent analysis to understand the impact of position and what motivates a customer to order from a specific vendor.

This analysis helped us understand what type of vendors were being rated high by our models versus which appealed to our customers better. It helped us narrow down to a list of features which helped in improving our model performance.

Modeling User Behaviour

The insights from the exploratory phase power the modeling phase to a great extent. Right from helping Data Scientists decide on what features to use to even choice of models, evaluation criteria and more. There are a number of intricate details which we are skipping here for the sake of brevity, but in a nutshell, the modeling phase is split into two main components described next.

A New Beginning: Cold Start

If you know someone, it’s comparatively easy to guess what they would like to eat or from where they would like to eat, as opposed to someone whom you just met. The same applies to the recommendation models as well. 

The coldstart models are geared towards analyzing generic and macro patterns in a population to provide better recommendations than random listing. The aim is to try to present what a new user might be interested in rather than just a plain random set of choices. The evaluation criteria and business metrics are also specifically defined separately for new customers. Coldstart models mitigate the challenge of not knowing much about new customers by trying to use aggregate information and signals/behaviors from long term customers. Some of our models even make use of platform (our app) usage behavior to tailor recommendations for new customers.

Don’t Worry, We Got You: Personalized Model

Once we know a customer’s behavior based on their interactions with our system, we try to provide more fine-tuned recommendations, tailored to individual tastes and preferences. The more we know about you, the better our recommendations get. The “know you” part is where the whole data science magic happens. Data Scientists leverage both explicitly shared feedback and implicitly deduced behaviors to power such models. Just like every other component in this setup, personalized models also go through an iterative process of improvement. Right from a basic rule based setup where we leverage insights from EDA and domain understanding of experts to our current set of machine learning based models, we have been in constant state of improvement. A resilient and robust infrastructure setup powers this highly iterative data driven ecosystem. 

Understanding the data/signals and developing models to improve concerned metrics is only one part of the picture. Another important aspect is “how does one know which model to use?” or “is the latest model any better than what we currently have in production?”. Evaluation of recommendation models which are highly temporal in nature is quite different from evaluation of usual machine learning models. We make use of offline evaluations as well as online A/B tests to evaluate our models using a number of well defined metrics. We will cover more on this in upcoming posts.

Conclusion

Personalisation is an interesting and important aspect of our platform. Understanding user preferences is one of the key steps in developing solutions powered by recommendation engines. It is a continuous and iterative process which keeps evolving as the user preferences evolve. We discussed how implicit and explicit feedback helps in developing an understanding of our customers along with UX Research which involves far more effort and depth. We touched upon a few interesting insights from exploratory analysis of different populations as well. The blog post presented how we personalize experience for new and existing customers by leveraging different approaches. We concluded by hinting about standard methods that help in evaluation of recommendation systems. Stay tuned as we introduce more components and go more in depth in up-coming blog posts.

Would you like to become a Hero? We have good news for you, join our Talent Community to stay up to date with what’s going on at Delivery Hero and receive customized job alerts!

At Delivery Hero, personalisation of the user experience is at the very core. Right from identifying the right bakery for your dessert cravings to your favourite curry place for lunch, we get it all to your fingertips. Let us see what goes behind the scenes to make this all happen.

More often than not, when we talk about a recommendation system, the spotlight is on the algorithm used. However, in the last few years while working on our own recommendation and personalization system we discovered that there is more to it than meets the eye. Like any other complex and sophisticated system around us, recommendation systems are composed of a number of building blocks and moving parts. Not paying attention to each of these components can cause even the best algorithm to fail. 

In this post, we will discuss the building blocks of our personalisation system along with our learnings. Let’s get started!

A Team of Heroes

The first and the most important building block of this overall setup is a highly-skilled, motivated cross-functional team. No wonder we call them “Heroes” @DH. In our experience, having a cross-functional data focussed team that has the autonomy to take independent decisions about the system being developed is very important. The setup that has worked for us, is a good mix of broad and deep expertise across roles and levels. To envision the system and plan, we have our own strategist (product manager). A highly skilled set of Data Scientists from varying backgrounds such as engineering, mathematics and physics keeps us grounded and enables us to challenge the status quo of ML-first approaches. A quick and talented set of investigators (data analysts) who love to dig into things and are curious to the core. To make all the ML/AI magic and plumbing work seamlessly (data pipelines, APIs, storage and more), an ingenious set of software and data engineers. And finally, a business owner to help unblock a lot of operational challenges keeping the team aligned to the vision. 

Interesting Fact: Even though our team is spread between Berlin and Singapore Offices, we have been able to work closely in a highly collaborative and supportive environment.

Vision, Strategy and Roadmap

Now, this might sound a bit superficial and flashy but it all begins with a goal or a mission statement. A goal to answer open questions and solve for gaps in offerings. A high-level goal/need for a team’s existence and its future are typically summarized as Mission and Vision statements.

First and foremost, you need to set up the right vision and strategy for the team. It is important to spend some time understanding the current state of the system. In our case, our ranking system was a rule based system a few years back and there was very little personalization. Our aim was to strive for an improved customer experience. To improve the experience, personalization was the obvious next step. Our main hypothesis was that by learning more about our users, understanding their tastes and needs we should be able to improve the relevance for our end customers.  
One important thing to think about is your north star metric/KPI. The driving question to ask is “why are we doing this?”. What does it mean to improve relevance? How will we be able to measure it and what tradeoffs are there to consider? In our case, our focus was towards measuring our KPIs through A/B tests, results of which can be attributed to our system.

The mission and vision statements are what drive a team, help in aligning stakeholders and provide guard-rails throughout the life-cycle of such teams. Make sure you give it enough thought, challenge the plan and vet it multiple times before the actual work begins.

The Architecture

Every product is like an organism that grows and improves as it progresses in its lifecycle. A recommendation system is no different. The following figure introduces a high-level schematic of our recommendation system (simplified for ease of understanding and discussion)

We will go into a lot more detail in a later blog where we plan to explain each of these systems. For now, the following is a gist of some learnings which we discovered over time:

1. The ability to serve multiple models in parallel is very important to empower research and improve the speed of experimentation. 
2. A configurable re-ranking layer can enable the team to test quite a few hypotheses without your data scientists spending a lot of time on them. 
3. A feature store which can be used across multiple models not only helps you bring standardization but also makes model generation faster. 

Automation is super important. Lesser human intervention at each step means faster time to market and also lesser errors. It is no surprise that a system of such scale and complexity immensely benefits from CI/CD pipelines and monitoring/alerting mechanisms.

Conclusion

We hope we were able to intrigue you enough about recommendation systems. Building one such system of our own has been an exciting and insightful experience. Our aim was to highlight the learnings and share them. We discussed composition of the team, vision-mission and strategic stuff and finally a glimpse of the overall architecture. Stay tuned, we have a number of detailed follow-ups coming shortly covering aspects of understanding customers, modeling for new and existing users, setting up online AB experiments, offline evaluation framework and more.

If you like what you’ve read and you’re someone who wants to work on open, interesting projects in a caring environment, we’re on the lookout for Data Engineers, Data Scientists, Data Science Managers, and Data Engineering Managers. Check out our full list of open roles here – from Backend to Frontend and everything in between. We’d love to have you on board for an amazing journey ahead.