Episode #5: Incrementality Projects with George Natsvlishvili

Or listen on:

Apple Podcasts
Spotify
Google Podcasts

Key takeaways from the episode:

• George has more than 14 years of experience in the mobile app industry. He has worked with reputable multinational organizations like Microsoft, Zalando SE, Schibsted Media Group and is currently working as the head of organic growth for Glovo. George joined Glovo two years back and has been a source of remarkable change and growth for the organization. 
  
• As George joined Glovo, there was no marketing team, and therefore little to no organic traffic. He began with off-page optimization, keyword updating, and running tests for horizontal and vertical reviews. He also conducted digital campaigns on the Google Play Store and updated screenshots that featured the most popular products.
  
• It’s better to enter a fresh market and take hold of the first position than enter an established one and compete from the 8th or 10th position. 
  
• Setting up a proper dashboard can be immensely helpful in tracking. The dashboard is set up such that it tracks visits from any user. The automation ensures that any and every visitor gets a properly structured page that communicates well through visuals.
  
• KPIs might differ from country to country. For example, in some regions, TV holds a larger share and is more impactful when used, and it goes vice versa for others. Mostly this can be observed through the linear regression model, which is utilized to observe the correlation between TV and incremental campaigns. For most countries, coefficients are used to eliminate the different impacts. However, this is not always the case, and sometimes more specific tools are required.
  
• For incremental growth, it took George about a year to find the right approach. This also led him to discover the right channel for generating organic traffic. Different statistical models are often applied to know which works best. These are often prone to change based on trends and regions.
  
• There cannot be a specific time frame when implementing incremental variables to ensure growth, incrementality can be quite complex, so one can never be 100% sure. The timeframe is mostly approximated, but it might take at least 1 to 1 and a half months to see results.
  
• George shares the differences between SEO and ASO. SEO requires a proper technical team of engineers that need to monitor it consistently. On the other hand, ASO does not need a large team but rather efficient minds who know how to set better targets. While SEO is generally a longer-term growth channel, ASO can bring visible results in just a couple of weeks.

“Don’t stop learning. If you think you’re an expert, you are a dead man.”

George Natsvlishvili

Full Transcript:

Esther: All right, so I’m joined here by George, whose last name I don’t say, well, so I’m not going to try to say it. I’ll let him say it. George runs organic growth at Glovo. George, tell us a bit about yourself.

George: Hi, guys, thank you for inviting me. I’m George from Georgia, actually, more than 14 years of experience and among them more than eight years in mobile industry, with different international companies such as the Glovo company, Microsoft, The Londo, Shifted, and now I’m at Glovo as the head of organic growth. I enjoy my time in Barcelona.

Esther: Amazing. Just to kick us up, George, if you could tell us a bit about, we’re going to talk about a couple of topics here. In general just share the KPIs that you most hope to impact when you go through these efforts. What are the bottom line KPIs that we’re looking for?

George: There are different direction that my team is working towards, two main direction right now. It’s not what [unintelligible 00:01:28] even more. One is EB and one is web. Of course, the main KPIs for me is driving the organic traffic and increasing the conversion rate. If you talk about organic conversion rate, is people who make order versus who will visit our web page or for apps, people who visit our app page to, who download it and who make order.

Esther: Can you share just to give us an idea of the scale that we’re looking at about how many downloads do you guys have for the global app?

George: In terms of, I can’t share these kinds of information with you guys but I can say that one of our projects, the apps optimization brought at least $4 million marketing savings for Glovo, and we are super proud of that. We did our best to make the same happen and we are super happy about that.

Esther: Amazing. Let’s jump right in and talk about the efforts that are able to make such monumental changes, what’s an example?

George: When I joined the company, it was approximately two years ago. There was no any organic team, there was a small marketing team and there was no one who even touch extra communication point, touch organic traffic. There was no any web even. Website was super generic, a normal website for ordinary people or whatever. Since we start with the project, with apps optimization, we update the keywords, we make some several test in terms of visuals, we add horizontal and vertical video on App Store.

We put typical visuals from two campaigns on Google Play Store and we updated screenshots based on the most popular products that we had, and most popular partners that our users were searching for. We also add some additional features on the screenshots such as the [unintelligible 00:03:56] because in past people most of them even now doesn’t know that we are not only food delivery, versus our competitors but we can deliver everything actually.

We have a courier service, we can deliver whatever you want. If you, for instance, forgot– If you don’t want to go to the shop, and you know what product you have to buy, even not only grocery, I’m not talking about grocery but something like, I don’t know, shoes or clothes. You can just order through the courier and he or she going to bring it to your house or to your work.

Esther: I have to complain for a minute that you’re not live in Tel Aviv, because I dream about this all the time.

George: Actually, we were suspicious because probably our competitors are super active there in terms of food and we are not going to the market where it’s 10 or 15 competitors because we experienced this. It’s better to open in markets where it’s one, one main competitor, or there is no one. There’s still in the world, there are such markets and there are lots of lots of markets in. It’s better for us to go to such markets and taking top positions versus to be number 8 or number 10 in Israel for instance, unfortunately.

We experience this, and at the end of the day, we experience with different countries. For instance, with Brazil, we entered the high competitive market in Brazil, with one of our main competitors. We spend a lot of money and this effort that we make for this country, for these efforts we could open a couple countries. We decided after that not to spend too much time on market data conquered by our competitors, but go into the market that have only one main competitor, or even doesn’t have anyone.

Esther: You were discussing a lot of ASO activities, you had keywords, you had incorporating different video sizes and orientations, screenshots, and TV. How do you structure that effort so that you’re able to maybe understand what’s causing the impact? You throw everything in there at once and hope for the best, are you going one at a time?

George: One by one image. When I joined the company, still we had even more and more countries that right now, with us, and even now we have more than 22 countries and we are represented in extra in grocery stores. Imagine one person for 22 countries doing apps optimization, it’s a miracle for other companies. Other companies that have such scale and such big, big markets and several markets, they have at least three or four people in apps optimization addition as well.

I was using my previous experience, how we can optimize stuff. In terms of hiring additional people, we move to automation of this stuff as well, and it helps us to be much more efficient versus our competitors.

Esther: What kind of automation?

George: For instance of the automation of dashboards, our KPIs, which is fully automated and we can track, it’s in one place. Whenever you open the dashboard, everything is automated, and we can track actually the stuff there. For that we don’t need an [unintelligible 00:08:01] for whatever for instance. There are also a couple of tools here.

Esther: By the way, once you have that dashboard set up, how are you using that to decide what to do?

George: We track different stuff, in terms of, for instance, implementing visuals, we track the conversion rate before and after and also we are testing on Google Play Store as well, because you can do a test. You can do this proper testing on Apps Store because you have to use the third party tools to test it. I don’t want to tell anything. Even the Google test on Goggle Play is not ideal because our conversion rate is super low, 90%. Statistically maybe it should be 99%, meaning 95% and 90% is very low.

We double check each time the results up and before implementation, what’s going on here. Of course taking into account other activities that are going for instance, TV campaign, or pushing our UA campaigns. We have to take this into account and see the results and trade the results. In terms of keywords, we are tracking the third party tool as well. Through API getting the information, and it does both from this tool is approximately per non branded keywords, how many organic traffic we have per country and per keywords as well.

Esther: One last question on the ASO before we move on to more incrementality side. You mentioned you’ve got a lot of other events that are going on. You have TV campaigns that of course will shift the way things are performing and you have UA. I assume that’s going on which also will. It depends in the organic site. How do you create that isolation and be able to say, “Yes, this was our version release. This was our screenshots, or these were the keywords that we released that were causing the difference.”?

George: It depends on the country because in some countries, the share of TV is huge, in some countries, not. In some countries, people are interested to watch TV campaign. In some countries, they don’t care about TVs. It depends on the country, so we built a couple of models. We start with simple regression and linear regression model to see a correlation between TV campaign and incremental organic traffic and we get the coefficient. We try to eliminate, remove this TV campaign impact using this coefficient.

That works in most of the countries, then we moved to more sophisticated model like not too linear but multi-regression model. Also, in past, with another company, we also did with TV agency because they also tracked how many organic traffic brings specifically this TV campaign. We also do a couple of things with that.

Esther: Are there any metrics that you associate specifically with TV versus, say, a screenshot release? Metrics that you would expect see a TV campaign impacting that you wouldn’t expect to see a screenshot change impacting?

George: In general, a TV campaign have a huge impact on organic growth, but also, you have to do a very, very good TV campaign because if you do, for instance, shooting for a different channels at the same time, the TV campaign, the impact will be not as good as to shooting in a different time. You have to take also this into consideration. In terms of visuals, of course, not only visuals on TV campaign but also ongoing campaign something special that you want to offer people.

For instance, I don’t know, for COVID, when everything was shut down, we put on the screenshot like a grocery. It had a big impact on conversion rate because people want to order the grocery online. They didn’t want to go to offline shop because obviously they were super scared about everything. It helps to increase even further our conversion rate during the crisis.

Esther: Okay. Let’s shift a little bit into incrementality. You touched on it just now. Obviously, it’s something that is, I’d say, on a lot of people’s mind in the industry. It’s also something that is incredibly complex. I don’t think anybody has got it 100% mastered. I’d love to hear from you, maybe first, what hasn’t worked? Have you had any efforts that really failed? [chuckles]

George: A lot of stuff didn’t work for us, a lot of– I don’t know, we spend one year to find out the appropriate approach. We start with the correlation model and the regression model as well per country to find out per each channel, per each UA channel, through Facebook for, I don’t know, universal campaign what is the correlation between the spending and increasing the spending and traffic, including the traffic of paid traffic and the organic traffic. We made conclusions that there is no any model that fit per country, at least, in our case.

Esther: You’re trying the same model on every single region basically?

George: The approach was different there. We tried different statistical models here, not only linear but also multi-models, like, I don’t know, time series as well and whatever. Almost whatever exists in statistical, in data sciences, we try almost everything. Then instead of doing this, we changed our approach. We launch incrementality test per channel for Facebook and for Google. As you know, Google is super geo-based and Facebook is user-based. This is a different test and different incrementality you can get and different results as well.

Whenever we started to launch this incrementality and we still is ongoing country by country, we found out the very interesting stuff, and some of the stuff was even surprising versus what we expected. I can’t, unfortunately, tell what stuff it was then, but it was-

Esther: You read my mind on the next question.

[laughter]

George: -super interesting. Actually, after thinking, it was very obvious. It should be and we didn’t think about it, but it was obvious conclusion that we made based on this test. We’re still doing them, doing tests in different, not even in different regions but different countries because countries are even super different in terms on incrementality within the same channel and within the similar city actually.

Esther: What timeframe do you feel like you really need to be able to understand the true impact of incrementality, meaning, three days probably isn’t enough. [laughs]

George: No, no, no. Actually, like 100% you can say it’s just like, you can get coefficient and key factor, for approximate impact. The approach is approximately we are doing two months [inaudible 00:16:23] or, at least, you have to do one and a half month or one month.

Esther: Okay, so I’ll try to push my luck again. I know you can’t share the greatest strategies but maybe–

George: No, I can’t. I can’t.

Esther: I’m sure. I won’t make you, I promise. You get to proofread this before it goes out, so you’re 100% safe.

George: If I share this, there will not be any unique– I can’t sell myself to– I can’t sell this topic anymore, so this would be not interesting anymore.

Esther: I guess what I would say is, let’s go not necessarily for specific strategies, but how do you approach designing an incrementality test? What is it that tells you, “We need to do this now,” and how do you set it up?

George: I will not tell you exactly what, but I can tell the approach here. The approach is to take the similar regions and to do a different direction. One, is to shut down completely the user acquisition and see the results, and one is to put even more money, double the money or triple the money on the user acquisitions and to see the results. The details, I can’t tell you, but from this, you can get the starting point. Then you can get the logic behind and you will improve into further iterations.

Esther: Awesome. That works. I’m happy with that. It’s enough information for me. Okay, let’s take you back two years in time. You just start at Glovo once again. What would you do differently? What do you feel like, if you got to start again, that you would have changed?

George: I don’t know. In terms of what I did, so far, everything was successful. First of all, I did extra optimization. It was super successful, like more than $4 million marketing savings. Second, we sold the project search engine optimization that no one in the company understood. There was no any focus on web because it was just ad-based and mobile-based company, so people don’t, in past, didn’t take any to close this proposal, but at the end of the day, we managed it. Now the strategy is one of our key players.

Esther: Is web, is it an extension of mobile or is it its entirely owned product? Do you have users who are in web who don’t touch the app whatsoever, or is it just different avenue for the same goal?

George: Actually, we have a web, especially the mobile, I am super surprised because actually, from web, you can get even more traffic than from mobile. Our competitors actually, they are super old-fashioned and they are web-based first, and then they move to app. They still get a lot of traffic from web actually. The main loyal users they have from web. Also, we see that the web, especially the mobile, is a huge opportunity because even this old fashion company that have web-based, they are mostly on desktop. They are very good, but in mobile you can still conquer the world.

If you start from the scratch, if you have a good PMof SEO or web, in general, and good engineers that are super experienced, that’s it, you don’t need anything else actually.

Esther: I have another question just because you mentioned it. Let’s compare SEO to ASO for a second. Similar and in theory, definitely, do you find that you can copy-paste strategies from one to the other or are they completely different animals?

George: No, not really. You can’t because for instance, SEO, a technical SEO, you are super dependent on the engineers that you can’t move whatever, without engineers, without testing, without dedicate person. It’s low-key. You have ideas to implement it, but you have a limitation of the resource because in– In ASO, it’s different. Here, you own on your hand everything, almost everything, not everything but almost everything. You can do whatever you want. All roadmap of implementations that you have, you can implement it by your team and by yourself. You are more independent here from tech team.

In some of cases, you are, but still you are super independent versus SEO because SEO, it’s mostly technical SEO. Technical SEO, without engineering, there is no possibility to move forward.

Esther: Yes, so if you have that independence, do you find that ASO– Are you able to move the needle more in ASO, or is it even with the dev-side SEO shows more high-level impact?

George: You mean that which shows biggest– In short terms, of course. In short terms and if you want a quick wins, of course, you have to prove, put here on ASO. SEO is more long-term because again, it’s super– It’s not that you implement stuff and the next day, you get the result. With ASO, it’s [unintelligible 00:22:08] specific, but with SEO, you have to wait, I don’t know, half a year, like three month until Google will start to index your site until you get the popularities, blah, blah. Whatever technical stuff you implemented, you will see the results minimum in two months. You know what I mean?

Esther: Yes. It’s a long game.

George: It’s longer. It’s a long-term bet. ASO, it’s more like short-term bet and it’s more quick wins actually in ASO. In terms of healthy or whatever, so healthy, I don’t know, stabilized traffic, of course, SEO has a bit more potential because whenever you fix everything in SEO, that’s it. You are just like doing nothing and-

Esther: Autopilot. You get to see that. Yes.

George: -getting the traffic. Exactly. With ASO, it’s like changing every one year or half a year, they change something so if you are not tracking eyes on that, that’s it. You are gone whatever you did impart.

Esther: Very true. ASO, you never get to let go of, so you have the more immediate satisfaction that you put the change in place, two weeks later, you can already see the benefits quite clearly, but you never get to relax.

George: Correct.

Esther: It’s always going.

George: That was the main challenge when we start with SEO because the people like management was super used to get a quick wins in one day and in one week, maximum in one month. Especially with the ASO implementation, they would go super quick. We did everything and for super quick win. With SEO, it’s like after one month you did something when we start then they approach me like, “George, what’s going on?”

Esther: Where’s our numbers? Tell us about the performances.

George: Exactly. “Wait a minute, what did you do in the SEO? What’s going on?” I was like, “Come on, guys, this is a different world. You have to understand that this a web-based, not ad-based. They behave with different rules how to play inside here.”

Esther: Yes, for sure. Now we’re moving on to the quick-fire round. These are question that everyone gets asked by me during this podcast. Okay, first, you could give just one tip to an aspiring growth marketer, what would it be?

George: Don’t stop learning this stuff, guys. Don’t stop. If you think that you are expert, you’re a dead man actually.

Esther: It’s a good tip. Your favorite Mobile Growth resource?

George: In terms of ASO stuff, there is a [unintelligible 00:24:51] channel with Phiture or I don’t know, the company.

Esther: Yes, Phiture.

George: A very, very good one. Exactly. Like, managing anything by Moritz and company, actually. Not only mine but other guys, they are super good, and hi to them. My kudos to you, guys, once again.

Esther: Who is the person in the Mobile Growth world that you’d most want to take for lunch, assuming it wasn’t Corona and you could go somewhere for lunch?

George: For lunch, of course, my favorite person is Thomas Petit. Thomas Petit, actually is my HR, because of him, I found the job at Glovo actually, and because of him, probably, I will find another job.

Esther: That’s a good answer. That’s the right person to take for lunch.

George: Yes, exactly. That’s the right, exactly. He’s my HR.

Esther: Okay, last one. It’s the most important question. What is your favorite flavor of pancake?

George: I don’t know. Brown? Something like this. I don’t know.

Esther: You can take your favorite breakfast, if you’re not a pancake guy.

George: I don’t know, tea and Eastern European and all coffee or tea. I preferred tea because coffee, it’s too strong to start the day. Coffee should be after lunch or something, they don’t, before lunch to wake you up.

Esther: In Tel Aviv, coffee is for anything. Israelis drink coffee to sleep. They drink coffee to wake up. They drink coffee to relax.

George: Yes, coffee before sleeping is not the best idea, because if coffee is super strong, you will not sleep for couple of hours.

Esther: If I have coffee ice cream before that, I’m awake all night so no, but that’s how you know I’m not a native Israeli. I actually can’t tolerate it. One last one now that’s not related to the quick fire round, but if people want to see what you’re up to, is there anywhere they can find you online?

George: Yes, I’m in actually usually in my other [unintelligible 00:27:07] group. You can find whatever that you want in most of mobile conferences. You can approach me in LinkedIn and we can have a conversation.

Esther: Amazing. Georgie, thank you so much.George: It was super fun here, Esther. Thank you for inviting me.