Navigating the Future of AdTech & Measurement

[music]

Andrew Covato: If you are doing the right type of experimentation that is specifically design experiment, test and control, and you're understanding the incremental impact on a high LTV user acquisition, you can still find and seek out those kind of users, just doing it in a different way, a better way, in my opinion, and certainly a more scientifically sound way.

[music]

Producer 1: Hello, and welcome to Good Data, Better Marketing, the ultimate guide to driving customer engagement. Today's episode features an interview with Andrew Covato, Founder and Managing Director of Growth by Science. But first, a word from our sponsors.

Producer 2: This podcast is brought to you by Twilio Segment. 92% of businesses today are using AI-driven personalization to drive growth. However, successful AI-driven engagement is only as good as your data, be it targeting your top accounts with relevant ads, or delighting customers with personalized experiences both online and in-store, Segment has helped thousands of companies like Intuit, Fox, IBM, and Levi's, be AI-ready by laying a foundation of data that they can trust. Want to get your data AI-ready? Learn more at segment.com.

[music]

Kailey Raymond: The benefits of AI in advertising and marketing are indisputable. Automating mundane tasks, improving creative processes, and optimizing campaigns are all possible, so long as we're feeding AI the right type of data. And if marketers wanna drive progress and innovation forward, we must create and promote a culture of scientifically-backed measurement strategies and data-driven decision-making. Growth by Science is putting their money where their mouth is. Today, I'm speaking with the Founder and Managing Director, Andrew Covato. We discuss the resurgence of media mix modeling, AI-driven advertising platforms, and the importance of scientific analysis and marketing strategies.

[music]

Kailey Raymond: Today I am joined by Andrew Covato. He is the Founder and Managing Director of Growth by Science. Andrew has over 15 years of AdTech and MarTech experience at companies like eBay, Google, Meta, Netflix, Snap, to name a few, you've probably heard of those. He's also been advising company from startups to Fortune 500 enterprises since 2012. He is on a speaking circuit, and he also is on various AdTech industry boards and working groups. Wealth of information. Andrew, welcome to the show.

Andrew Covato: Thanks. Thank you for having me. Great to be here.

Kailey Raymond: I'm excited to dive in. You're gonna school me today about advertising measurements. It's certainly an area where I could grow and learn, so go easy on me, Andrew.

[chuckle]

Kailey Raymond: I wanna learn in your own words, what's your career journey? How did you get to where you are today in this role of advising and having your own company?

Andrew Covato: Yeah. I would say that like a lot of folks out there, it's not something I pre-planned, it sort of happened along the way. I've always been a technical, analytical guy, I did engineering as an undergrad, I got a master's in finance. And I just really didn't know what to do with all that math that I learned in engineering, I didn't wanna be a hands-on technical, mechanical engineer. So I ended up working in market research and analytics. And by a turn of several events, random events, a lot of them fueled, frankly, by one of my most favorite tech companies, LinkedIn, one of the few I haven't worked at. But thanks to LinkedIn and the connections I made there, I ended up falling into tech, and there seemed to be a natural connection between the experimentation work that I had done in college and that I learned about design experiments and analytics and that kind of thing. And it seemed like marketing didn't have a lot of that going on when I started at first, probably still doesn't have enough, but I found a strange unforecasted intersection of passions and have seen in there since. But it's been a great journey, I've been really lucky along the way, and I'm excited for what the future holds for the industry, and hopefully my small part in as well.

Kailey Raymond: I like that the math guy goes towards the advertising area of marketing. It makes perfect sense, it's definitely the most scientific place I think that exists in marketing today. I'm curious, you've been doing this a while, you've probably seen some trends come and go in advertising. What are some of the things that you're looking at right now as it relates to AdTech and measurement? What's on your radar?

Andrew Covato: Yeah. The particular area of focus for me in advertising is measurement ad efficacy and I've always worked around that and had projects where that was at the core of it. And so, I think that's an area that there are some interesting trends that are emerging in the last couple of years. Obviously, sure we'll touch on this, in great depth as we talk. But privacy changes, ability to access data, all of that has been in absolute turmoil over the last couple of years and even before that. Started off really with regulatory changes and then OS changes from Apple, and Google [0:05:16.7] ____ having this year. That has thrown a monkey wrench into the traditional way that advertisers have been understanding the impact of their marketing. A lot of what advertisers have done in the past is look at really granular user level path to conversion type data, path to purchase type data, and that, that's been broken.

Andrew Covato: One of the major trends that I'm seeing is going back to the old-school, if you will, MMM, marketing mix modeling, media mix modeling. That's a traditional way that media companies or rather that advertisers used to understand the impact of their advertising spend and their advertising campaigns. That's making a comeback, and we're seeing a change from the old-school, "takes a year to build" type models to a lot nimbler things that are being able to be calibrated, that can be tweaked and refreshed on a weekly or even more frequently basis. So that's I would say the canary in the coal mine for big changes in ad measurement. I think it started with MMM, but really it's a general shift toward aggregate measurement, maybe not relying so much on deterministic user level data, looking at more statistical techniques aggregate style measurement. And so, that's been great. My one caution around that is that any type of measurement that doesn't consider causality, design experiments in particular, and their ability to understand, the cause and effect of ads and subsequent purchases, anything that doesn't do that can be pretty detrimental and not really that much better than what existed in the past.

Andrew Covato: While there is a swing to these more aggregate style approaches, I think it's important for marketers as they're getting on this train to understand that, hey, there's a big piece of causality that needs to underpin all of it. And that's really what, the work that I'm doing in industry right now is trying to help move that along. Long-winded way of saying, hopefully better measurement. I don't think it's getting adopted quite as much as I think it should be, but it's one interesting trend. Second trend is this concept of advertising platforms taking away control from advertisers. And, in the past you used to be able to tweak things, you could still do it to a certain extent, but there's definitely a trend now for these AI campaigns. You've got PMax, you've got Advantage Plus with Meta, that are saying, "Hey, give us some very high-level parameters and we're gonna black box the whole thing and just trust us... "

[overlapping conversation]

Kailey Raymond: I don't know...

Andrew Covato: "To the little shiny gold tooth. Trust us, we got it, we're gonna make it perform." My opinion, the jury's still out as to, if that trust is warranted, and I'm sure we'll talk about this as well, but that's an area of opportunity for advertisers to dive into and dig in there.

Kailey Raymond: This is great. Yeah, I'm gonna definitely ask your take about whether or not we should trust that. I wanna go back to the first one, which is, how many times did Google change when cookies were actually gonna be deprecated? Like 25?

[laughter]

Kailey Raymond: It's finally happening this year. I know that, well, a couple of percent of people are having cookies deprecated. Let's see if it actually ends up fully happening by the end of the year. I think it will.

Andrew Covato: I think so too.

Kailey Raymond: What advice do you have for companies that are looking to make this shift right now? What are the options available for them as they're moving away from third-party cookies and thinking about technologies like DMPs, which have historically relied on third-party cookies? Obviously, a lot of advertisers have been relying on them to make sure that they're creating the right audiences as well. So both sides of it, what do you see is happening? What are some of the outlets for them to make the change?

Andrew Covato: Yeah. I think the first thing that I would throw out there and say that advertisers need to wrap their head around is that, user level data is dead. And maybe not 100% dead, that's obviously an exaggeration, there's still some around and maybe some will persist. But if you approach your growth program with the mindset of, "It's dead, I'm not gonna have access to it, I'm gonna have diminished access and continued diminished access over the next 12 to 24 months," then I think that puts you into a different headspace and it makes you think of different totally out-of-the-box approaches for building a growth program. And I can say from personal experience, I know that it is 100% possible to not use pixels, not use CAPI integrations, not have custom third-party audiences and all of that, and you can build quality growth program even in a performance marketing context without any of that stuff using old-school methods. Great targeting, great creative reach and frequency management, and incrementality-based measurements, and you can build a growth program.

Andrew Covato: It's not easy, and doesn't happen, it's not gonna happen overnight, especially if you've got a whole MarTech stack and methodologies that are all based around user level data. It's gonna shift and it's gonna take some time to shift. But it is possible. And I would say that when you get there... There are a number of advertisers, some of the biggest ones that are out there actually, you'd be surprised are operating in that type of paradigm. But when you get there, it's such a relief that you don't have to worry about any of this stuff and you're robust against all of the turmoil that exists in user level data. So, I would strongly recommend taking what I said, which arguably is extreme, but giving it some serious consideration as an advertiser, and starting to understand, "How can I decouple myself from this addiction of user level data when it comes to growth marketing?" And that's not to say, your own internal data that your users are providing for you and maybe they're logged in or whatever. I'm not saying ignore any of that stuff, obviously that's really important for growth, especially from a product perspective. But from a marketing perspective, I do think it's different and it can operate a little bit differently.

Kailey Raymond: That's what I was gonna ask. I was like, "Okay, well, what about customer profiles? What about known behaviors as it relates to how you might build an audience?" I think that is an extremely rich source to be able to mine from and is a place where we've seen a lot of folks really have an impact on their advertising campaigns, leveraging behavioral customer data from, yeah, sure, signed in, but then, once you know a lot about that person, there's inferences that you can make about who's coming to your site, so you can even match a little bit of the unknown data from unlogged in. So those, really those customer profiles I do think have a place here now.

Andrew Covato: I think so. But I would say, even how you're describing it, I wouldn't think of it in exactly that way. I do think it's very important to understand your customer and from any information that you can with 100%, yeah, you know that you're allowed to and you have access to, it's typically your first party data, obviously that should be mind and understood and profiled to the greatest extent possible. However, all of these changes, cookie deprecation or cookie apocalypse, some call it, for what it is, and IDFA apocalypse and all that, what that prevents you from doing is even something like, it either prevents or severely limits your ability to take that first party data, dump it into some third-party platform, make lookalikes, retarget, all of that kind of stuff. That whole ecosystem is dying, is terminal right now, for sure. Not necessarily dead, but it's on life support, in my opinion. So, the analytics that you'll need to do will be slightly different. Instead of, "These are my high LTV customers, I'm gonna throw them into a platform and tell it to make lookalikes," instead, you'll need to dig a little bit further and understand, "What are the types of campaign parameters that I can assemble that don't require user level data that will generate an increase in incremental high LTV users?"

Andrew Covato: And so, you're still trying to get those high LTV users, but you're doing it in a different way, and the experimentation that you're doing and the analytics that you're doing around that is different. Some people might think, "Well, that's extremely restrictive. If I can't directly target them, what's the point?" It's not. I think if you understand all the levers that you still have at your disposal, just even think of a few though, the creative, the messaging, the reach, the frequency, the channel mix, all of that can be assembled in an infinite number of combinations. And if you're doing the right type of experimentation that is specifically design experiment, test and control, and you're understanding the incremental impact on high LTV user acquisition, you can still find and seek out those kind of users just doing it in a different way, a better way, in my opinion, and certainly a more scientifically sound way. At the end of the day, do you really trust that we can get into this? But should you be trusting platforms to be delivering those high LTV users? My answer is no. And I'll just... Coincidentally, the state that I've worked at a lot of these platforms...

[laughter]

Andrew Covato: Totally unrelated to that last statement.

Kailey Raymond: Yeah. We'll decouple those things in our minds really quickly. And then we'll...

[chuckle]

Andrew Covato: 100%.

[laughter]

Kailey Raymond: And then we'll ask the question, which is, you already brought up ad platforms pushing us into these AI-based campaigns, and it seems like there's a little bit of hesitancy from your side on that, it doesn't seem like you're buying it. Why? What's the crux of the fear?

Andrew Covato: Yeah. Context, at the start of the dawn of digital advertising, way back in the day, you think about what were the first types of ads that were out there, it was search ads, maybe some banner ads, things like that. There wasn't this massive ecosystem of connected devices, cross-device, mobile phones weren't really a thing, certainly not, and you weren't getting ads on them the way we are today. And so in that world, it made sense that if you clicked on an ad or you saw an ad, then you did something online, but the ad probably made you do that. So it wasn't like a huge leap of, logic to say, "Okay, post-click conversions means that those ads are working." And so, great, people went all in on that. The ecosystem blew out and evolved and got super sophisticated, and machine learning and data got into it, and the whole thing went...

Andrew Covato: But one thing did not evolve, and that's the logic behind, I saw or interacted with an ad, and then I converted. That's the underpinning of all AdTech, still is to this day. Anything that... Any platform that says otherwise is really just layering stuff on top of that, but really at the core of it, that's what's the truth set... That's what's the objective function of optimization of ad delivery is essentially that. If AdTech is built on this premise of post-exposure, which is what I like to term it, post-exposure attribution, and you let the machine learning run wild and optimize with that in mind, what's it gonna do? It's gonna find people that have a high propensity to convert and show them ads. And funny enough, that's exactly what's happening. So let's take a second to let that gel, right?

Kailey Raymond: Yeah.

Andrew Covato: Platforms are really, really good... Think about the billions and trillions of data points that platforms like Meta, for example, have, on individuals, their behavior from the decades that it's been around. They are really, really good at predicting user behavior. And so when you show an ad to somebody that already has a high propensity to buy something and the ad is for that particular item of interest, then of course you're going to see people convert more and you're gonna get this self-fulfilling prophecy. And this is something that I go into a lot with a lot of my clients on the buy and sell side, this self-fulfilling prophecy of finding people that have a high intrinsic conversion probability, showing them ads and then saying, "Hey, because they saw that ad, or after they saw that ad they converted, so give credit to that ad." It's cyclical, it doesn't make any sense. Then, by the way, and I'm not saying that this is 100% wrong, in fact, that's why the ecosystem hasn't even passed this, is because it does work for some advertisers and it does work for some users, some people who just need a little bump or maybe they don't have awareness of a product, something like that. However, it is suboptimal at best. And so now you've got this backdrop. And so what you're saying is, "AI, take the wheel."

[laughter]

Andrew Covato: "Just take the wheel. Do this stuff. And here's the parameters." And the parameters are not, find people who will be influenced by this ad to purchase. It's find people who have a high intrinsic interest for the product and show them an ad. And if you compare those two components, you can see, this is a perfect example of the "dangers of AI," you give it that with no bounds, it's gonna be really, really good at finding people who have intrinsic product interest. And guess what's gonna happen? The incrementality of those campaigns is not gonna be impressive. And by the way, I work with a number of measurements partners, measurements platforms, who are my clients that are doing research into this, and they have incrementality-based measurements, so they do a true testing control. And the research that is being shown is actually verifying the point that I just made.

Andrew Covato: The performance that you see on the platforms is way, way higher than the performance that you see from a true incrementality perspective for these AI campaigns. Now, not to say they don't ever work. It's not to say you shouldn't test it. It's not to say that you can't make them work. But out-of-the-box, you just need to tread very, very cautiously as an advertiser with these campaigns, and you have to make sure that you really, truly understand what they're designed to do, and have a testing program that helps you put them on the right track. So there have been, to be fair, to give the counterpoint to what I just said, there are instances where those AI campaigns can perform really, really well, but it shouldn't be taken for granted, and much like any new ad product, you've gotta test it, and you've gotta test it scientifically.

Kailey Raymond: I'm lightly following the Supreme Court social media case, which obviously hasn't come to its conclusion, but I just wanted to get your take, we're talking about AI right now, I think that this is a really interesting place where there is a lot of bot activity. There's potential issues with First Amendment rights that change the face of the internet depending on what decisions they make. And obviously, this intersects with a lot of the platforms that we're talking about right now. What are you watching? Are there any pros for advertisers at all if we go in this direction? Give me your take.

Andrew Covato: Yeah, this is a tough one. There's so much politics injected into the whole thing, and as everyone's aware, this political spectrum shifts left and right, left and right, all over the place. I think we're in a certain shift right now where there's just a bit more focus on things like viewpoints being expressed, and to what extent do you allow any type of viewpoints, regardless of what they might be? And what is the jurisdiction, if you will, of private enterprise to be able to control that? Is it editorial content or is it a First Amendment thing? There's a lot of complexities to it. At the end of the day, the platforms are never going to be able to get it 100% right. They're never going to be able to make everybody feel good about whatever policies and methodologies they put out there. It's really a tough call on... What do you expect them to do? I would say, you can look at X, I think it's taken a little bit more of an aggressive stance against a moderation, and that could be potentially looked at as a use case or a, whatever you wanna call it, a petri dish for what it could look like if moderation really took a backseat or was intentionally put to the side or deprioritized.

Andrew Covato: And so really, I would seek out the advertiser's own opinions. Some advertisers may think that's great and feel good about advertising on platforms like that, and others may not. But ultimately, that is what is going to drive what the platforms do is to have dollars. So, advertisers will speak with their dollars. If advertisers like the type of moderation that exists, they're gonna spend more on those platforms. If they don't, they're gonna pull it away. And I think it's a little early to see how that pans out for X. They've got a lot of things going on, and admittedly, I'm not very deep in the weeds on that platform, the politics around it. But it's suffice to say, everybody's never gonna be happy, [chuckle] about whatever moderation approach is taken.

Kailey Raymond: Yeah. No, it's gonna be fascinating to watch. You talk about cookies being the big thing that everybody's talking about this year. If this is a decision that's made, then the ad world gets thrown into a completely new loop. And nothing would happen for a while, the government tends to move really, really, really slow. So there might be a pocket of time. But, yeah, I'm fascinated to see. Especially, I was reading this article this week, it came from the software company CHEQ, which really... They monitor bot traffic. And it was claiming that on the Super Bowl, on the day of the Super Bowl, that the majority of traffic that X was sending to their client sites, 76% bot traffic. That is astounding.

Andrew Covato: That's wild.

Kailey Raymond: Unbelievable. And so I'm wondering, how much are you seeing bot traffic impact a lot of your ability to measure this? Is this front and center with what you're doing right now?

Andrew Covato: To be honest, it's not at the front of what I'm doing. But the methodologies that I advocate for, they wouldn't necessarily highlight a bot problem, but they would highlight, ultimately, a lack of performance problem, which, once you investigate that, potentially could uncover bot issues or fraudulent traffic or whatever. But we think of in incrementality perspective, how would bot traffic or significant amount of bot traffic manifest, ultimately, it would drive the price up and obviously bots aren't gonna be buying anything, and if they are, well, then we're really in trouble, the AI has become sentient and a consumer beyond that. But if... And we would see an inflated price relative to the incrementality. And ultimately, again, I'm a believer in the markets really driving innovation and progress and change in ad platforms. And if you consistently see advertisers say, "Hey, I'm not gonna be paying 2X what I should be paying for, for these incremental conversions, so I'm pulling back my budget." That's gonna spur like, "Hey," on the platform side, "Well, why is that happening? Let's dig into it." "Oh, wow, three-quarters of our traffic is bots. Okay, well, maybe we should look into that."

Producer 1: No kidding. Okay. We've been talking about AI and some pretty scary, negative, perhaps [0:24:35.5] ____ lights for the past few minutes. You started this, Andrew, very optimistic about the future of AdTech and marketing and so on, I want your take on what some of those opportunities are related to AI and where we could head that you are interested in pursuing.

Andrew Covato: Yeah, totally. Look, there's no question that AI is going to be a huge part of marketing and already is a huge part, it's gonna be an even bigger part, a more pervasive part. What I'm most excited for is alleviating the burden of rote work. That's, I think, the first primary opportunity. And so that's in copywriting potentially. And not to say that it would maybe 100% replace the creativity that a human brings to copywriting, but it can certainly augment it. Same thing with some of the graphic components and some of the video editing and video generation technologies in the last couple of weeks has been pretty incredible. I don't think it's quite there yet, and it still takes a lot of work to make it really high gloss and ready to be out there. But, you can tell the advances that have happened just even in the last six months, it's coming quick, and eventually stock photos will be 100% replaced with that, stock videos, same thing, and then the editing and the evolution of that will just take off. I think from a creative side, it's the first really, really big opportunity for that.

Andrew Covato: And then ultimately, if we can change the parameters of optimization, I think there is a place for these AI campaigns. It's just, AI is only as good as the data that you feed it. A few years ago I read some anecdote about, you give it the task of, "Make as much paper as you possibly can." That's the parameter that you give it. Eventually, the planet will be totally razed to the ground and there'll be nothing, there'll be just manufactured tree farms and big paper printing things or machines that are churning out and people are exterminated because they're in the way of the paper process. And so you can see how you give it the wrong type of parameters and it can go and do the wrong thing. And obviously I'm not saying that, I'm not worried about that scenario happening, not too worried, not worried about it, but in ads, that is what's happening with the optimization. Again, going back to, give it the right data, give it the right objective functions, and I think it can be a hugely powerful tool. I am hopeful, I really am. I think... And actually, just on that topic, maybe I have said a bunch of doomsday type things and...

Kailey Raymond: We always do. Don't worry. On the show, it's all doomsday for AI. [chuckle]

Andrew Covato: Oh, we do? Hey, you know what? It's always darkest before the dawn, as they say.

Kailey Raymond: There you go.

Andrew Covato: And I think we're in a place right now in tech and maybe even beyond tech where things are hard, there's a lot of issues that are bubbling up, a lot of conflict, a lot of really strongly opposing opinions, and I think it's out of this turmoil and that kind of tension that I think real innovation comes about. And so, I truly believe that. I think that once we get through this 2024, which will be a dark [0:27:35.7] ____ shed for a year for a lot of reasons, I think we'll find a balance and we'll be able to move forward positively.

Kailey Raymond: I totally agree. I'm already seeing a lot of the right spots of where AI is being used from the creative outlet that you're sharing. And there's this really interesting, I'm forgetting which AI software it is, so apologies for not being able to quote it, but there's this one that allows you to change languages. You can take your voice and if you record it for a minute or whatever, it has it down pat and then it translates it and then you could speak Japanese or German or whatever.

Andrew Covato: That's cool.

Kailey Raymond: Which is super cool. And it's also really interesting to think about, well, it could be applied in a lot of different places, but certainly in media where we'd be able to spend a whole lot less money for that creative process and be able to put it out into the world a lot quicker.

[music]

Producer 2: This podcast is brought to you by Twilio Segment. According to Twilio Segment's State of Personalization Report, 69% of businesses are increasing their investment in personalization. And with the world going cookie-less, first party data is the most reliable, trustworthy, and compliant way to create tailored experiences. Imagine, suppressing consumers that just purchased that pair of shoes from your ads and decreasing acquisition costs by 5X or sending the perfect text promoting your other beauty lines and converting 50% of customers. That's the power of Twilio Segment. Learn what's possible at segment.com.

[music]

Kailey Raymond: You mentioned basically the sentiment of garbage in, garbage out, when you were talking about AI.

Andrew Covato: Yeah.

Kailey Raymond: And so, the principle of this show is that you really do require good data to have good marketing outcomes. And so I'm wondering, you have a thought around what the definition of good data is?

Andrew Covato: Yeah, for sure. Good data, as it relates to marketing, is data that can help you understand cause and effect. And I actually think a lot of marketers overthink this. It's not about necessarily the quantity of data. It's not about the granularity of the data and all of that, and there was a time when that might've helped, but right now, the more important thing is how it's collected and how you set up design experiments around the questions and the hypotheses that you have, and do you have the right analytics that is being applied to that data? A lot of the growth clients that I work with, we don't have pixels, and I mentioned this, we don't have pixels, we don't have CAPIs, we look at aggregate data, and the best data that I can feed into the models that we use and the synthetic control, creation and all of that, is just really high fidelity regional data and aggregate outcome data aggregate on the ad side and on the purchase side or business KPI side. Getting all of that pieced together, parsed and clean.

Andrew Covato: Once you have data at that level of resolution, which isn't super detailed, possibilities are endless, you could build a really robust econometric model on that, you can do design experiment, you can impose variation into the ad side and look at how that impacts the outcomes and have following around that. And all of that, the difference between that and really super granular user level data is you set it up with the intention of measuring causality. When you look at really granular super user level data, you think you forget that, and you mistake quantity for quality. And, yeah, maybe you can get some great nuggets under that, but I wouldn't... And I've seen this in a lot of organizations that I've had exposure to, get all this data and it just sits there. It just sits there.

Kailey Raymond: I 100% agree with that. You have to know the use case of why you're collecting that data. Years ago, IBM put up a study that was like, "Data doubles every 18 months." Data is gonna double every two months now, it's insane. There's just an unbelievable amount of information going out into the world. Social media, TikTok alone. [chuckle] I'm sure with the video content, that is an unbelievably rich source of information. And so, to your point, I do think it's all about, what's the use case that you're looking to apply, and what are the traits that you need to understand and collect or about this particular set of people to actually drive that outcome? Absolutely.

Andrew Covato: And more about, be creative in how you're designing your experiments. And granular data does not automatically mean more insights and better insights, there are other ways that you can get granular insights without granular data. If you design an experiment correctly, you don't have to measure all of the little components and the mechanics of how it goes out. You can look at the incremental outcome of that experiment. If you have all of the cells designed correctly and you set the variation for your model correctly and all of that, you can get really, really powerful insights out of that. And the best thing is if you do it right, you actually get scientific insights. You don't get correlative things and, "Well, I think this is the case because I see these trends working together," you get a... You actually have causality that's baked into the analysis. So I think one of the toughest things for organizations today, and it's really been something I've seen for 15-plus years or whatever that I've been working with them is this culture of experimentation, culture of science that has to exist in organizations for them to be successful.

Andrew Covato: And that's why my company, we call ourselves Growth by Science. We don't want alchemy. I love to call it Growth Alchemy. There's a lot of folks that say, "Well, my gut says this and I know this and I was on this blog and here's this hack, growth hacking," and all this stuff. No, growth is a scientific approach and there's a science behind it, there's a method to it. And if you set... If you apply a scientific method to what you're doing and that's... Exists from the top down, from the executive team all the way through, then you avoid these radicals of like over-collecting data, misusing it, not understanding causality. But it's deceptively simple, it's hard just to be... To stay with it, and it's hard to adhere to scientific principles. But it's pretty easy. You develop a hypothesis, you create a test that can validate or disprove the hypothesis, you run the test, and then you iterate. You create a follow-up hypothesis. You have a course of action. The problem is, people I think are really good at doing the first, they're not bad at doing the testing and they're getting better, but once they get at that result, it's like, "Well, I don't trust it... "

Kailey Raymond: "Could have been this, could have been that." [laughter]

Andrew Covato: Yeah. It's not even that. Even if you get a clear result, but it's contrary to say your gut or what the CMO thinks or what they've done for the last 20 years, it's like, "Yeah, that's all well and good, but I don't trust it because it doesn't align with my preconceived notion of what should happen. Therefore, we're not gonna do it, or we're gonna only dip our toe into it and blah, blah, blah." And so when you go down that rabbit hole, you've not really adopted science. You've adopted half science and half intuition, and unfortunately, that's not good enough. Intuition can help you generate the hypotheses, but science generates the conclusions.

Kailey Raymond: That's a good way to frame it. I was gonna ask you, I was like, "Okay, so where does gut fit in here?" Because you're talking about creativity and some of the incredible things that you can do with that in terms of your message, in terms of the actual visual and the creative, that's I think where that, some of that intuition...

Andrew Covato: Totally.

Kailey Raymond: Comes into place. Of course, you can test as well. But there is a balance of gut and science, but you're right, it's where you apply it.

Andrew Covato: Yes. That is totally right. I love how you put that. It's where you apply your gut, your intuition, and where you apply the science. And it's really hard because I think it's a natural human instinct. Everyone has confirmation bias, everybody has preconceived notions and they don't wanna be proven wrong. And certainly, old-school CMOs who I would call out as some of the most egregiously guilty of this, like, "Oh, I've done this for 25 years, I know better than whatever these numbers that you're putting out there." Right now, in this day and age, if a CMO does not have, at least I would say, more than a cursory understanding of causality, of incrementality, of the scientific method that goes behind testing, cause, and effect, then I would say that they're probably not gonna survive in a modern digital forward advertiser. I just don't think that they're cut out for that job. I do think CMOs need to adapt and evolve. And there certainly are good ones out there, but I do think there's still a lot of folks who are relying on the martini lunch style, Mad Men...

Kailey Raymond: Bring me back. I was like... Come on.

[laughter]

Kailey Raymond: I wanna be that CMO. That sounds like a...

Andrew Covato: You know what? If you do that with science, then I'm all for it. But the problem is, the martinis usually preclude the science.

Kailey Raymond: Andrew's bringing his pie charts to lunch.

Andrew Covato: Yeah. [chuckle]

Kailey Raymond: I love it.

Andrew Covato: Nobody wants that.

Kailey Raymond: That's so funny. And it's interesting that you're saying, in advertising, I would just open that up and say pretty much every part of marketing right now, I think...

Andrew Covato: Yeah. Totally.

Kailey Raymond: The past two years, the past five years, absolutely, there's plenty of tactics that I've run that do not work anymore. Or the opposite, that, I think that didn't used to work and now do. I think the biggest one is, obviously with COVID, every event marketer will tell you they've tried 17 different things in the past two years and it's different every time they do it. So it's like, if you don't have that ability to put yourself out there constantly and test and try new things and fall on your face sometimes, then you're not gonna move anywhere, and actually hit those goals and be innovative, gotta do things differently. Let's wrap this up, Andrew. Tons of great advice, really interesting, on the industry and your insights from the past 15 years of working with a lot of these platforms and companies. I'm wondering if you had any steps and recommendations that you might have if somebody is looking to up-level their measurement strategies. What's zero to one for them to get started? And then if you really want a teaser trailer, is there a one to two that you would give them as well?

Andrew Covato: Yeah. Yeah. Zero to one and depending on where you are in the org chart, but I would say it applies to varying degrees across the org chart, from CMO all the way down to operators, would be, understand the concept of incrementality, understand basic high-level analytical methods to drive that. And that doesn't mean you need to know all the stats and how to build things, but understand that to ascertain cause and effect, you need a test group and a control group. Get that concept really, really ingrained in everything you do. And even if you don't have the capability right away to start implementing that, just keep it in the back of your mind that when you're looking at data, and it's not coming from a test versus control context, you can't understand causality. And so really go going back to the absolute first principles, that it should be the foundation of marketing.

Andrew Covato: And I think, let's say somebody is working day-to-day in marketing and they've got this concept, you're gonna realize that most of what you're seeing out there is very deceptive, correlative, doesn't really guide you as much as you think it does. And that's gonna be your eye-opening, "Oh my gosh, wow, this is so pervasive, it's like, yeah, I don't know where this... They came up with this before." But it's almost like the standard American diet. Sadly, it's so hard to step out of that, I'm trying myself here, but it is a challenge because you have to make sincere efforts to not do the wrong thing when it comes to your health and nutrition. And think about health and nutrition of data and analytics and causality, you have to try to not do the wrong thing and it's really, really challenging.

Kailey Raymond: Andrew, thank you so much for being here, I really appreciate you sharing your insights and your stories and your hot takes. I appreciate that.

Andrew Covato: Thank you. I really appreciate being on here. It's a great conversation. Thanks for having me.

Kailey Raymond: Yeah. Absolutely.

[music]