Inside Overstock's personalization engine

Alex Maguire: 00:15 Hello everyone and welcome. Thank you so much for joining us today. Can everyone hear me okay? And can you see the slide? If you have any issues, please feel free to let us know in the chatbox. Great. Seems like we're good to go. Thank you again, everyone for joining us today for an inside look into the Overstock personalization machine. Just a reminder that at the end of today's session, um, we will take a couple of questions from the audience. If you have any throughout the presentation, please feel free to ask in the chat box. That should be at the bottom of your screen under Q&A.  

Alex Maguire: 01:16 We're here today presenting to you from the mParticle office in the Big Apple, New York City, as well as the Overstock offices in beautiful Utah.  My name is Alexandra Maguire and I'm in marketing for mParticle, although I do have a background leading in-house acquisition teams for ecommerce, most recently coming from Gilt. We're also so excited to be joined by our special guest and partner Craig Kelly, who is group product manager in marketing for Overstock.com. Welcome Craig, and thank you so much for joining us, it's our pleasure. Overstock has definitely remained relevant over more than two decades now, I think with a winning business strategy and has managed to stay relevant with expanded inventory and personalized user experience. So we're so excited to dive into and learn more about the personalization engine.  

Alex Maguire: 02:43 With more and more optionality cluttering user experience, it's increasingly necessary to curate the shopping experience for each individual shopper. For online retailers with deep inventory the opportunity to impact the bottom line with personalization is greater now than it's ever been before.  

Alex Maguire: 03:11 This year, Gartner predicts organizations with fully functioning online personalization will outsell companies without it by more than 30 percent. Plus, the mobile-centric world has increased online users' expectations to the point where 74 percent of customers feel frustrated when website content is not personalized to them according to Infosys, and this helps make companies profitable. A smart personalization engine does not only drive a positive user experience, but according to Gartner, they can increase profits up to 15 percent. So how to begin? Identity is the foundation that makes personalization possible. A single organized user view is necessary for personalized, one-to-one experiences to become possible. Identity enables marketing growth by leveraging first party data on an individualized basis as opposed to broader segmentation. With unique user identity at the center of the marketing stack, machine learning can function across devices and across marketing channels as well. Now, having set the stage, let's hear from Greg about how Overstock succeeded at bringing personalization to life. So Craig, do you mind starting off by telling us about your role with Overstock?

Craig Kelly: 04:50 I'm the group product manager for our marketing division at Overstock. I come from a background mostly in adtech and it's kind of been interesting for me over the past year here how the Martech world is kind of catching up with some of  those learnings from ad tech over the past several years and how you have these technologies that I think generally started in the, you know, the dollar and spend efficiency world are starting to be used to create real-time personalization and a real time coordination across all of the various marketing pieces. So, my role here is working with a group of just absolutely amazing developers and product managers, We work on all of our push and email technology or ad technology, which includes an in-house DSP functionality and API based campaign management. We have, you know, a large infrastructure team, data science teams, a content and UX technology team, as well as promotions team, and co-op advertising. They kind of all fall into the marketing umbrella and, you know, my role is really focused on how to create efficient systems that can enable all of these various programs to speak to one another in a way that creates the best experience for our customers.

Alex Maguire: 06:29 Great. What goals are you trying to accomplish currently in that role?

Craig Kelly: 06:37 Yeah, so I would say the mission that I was given when I started at Overstock was probably pretty typical of any marketing world. But it's been unachievable for a pretty long time in the digital world and that's just boiled down to assessment. It's really to deliver the right message to the right person at the right time in the right place. And we think that that's something that the CDP space, mParticle in particular, really helps us to do. When we talk about the more macro level, what are the pieces around that? Obviously everybody's trying to do that, right? So,  there's when we dig into a—what are we trying to do specifically to achieve that big view and what's the macro context of what we have to been able to do in order to deliver on that promise. One thing I would say is that we have an insanely competitive landscape, uh, and it gets more competitive by the month.

Craig Kelly: 08:01 So one piece of that is we have to beat our competitors. And so how do we do that? How do we become better than our competitors at, you know, achieving that mission of right person, right message, right time, right place. That to me is all about speed to market and efficiency. So how can we be most efficient in everything we do, especially our ads? Then when we talk about the types of dollars that our competitors are spending, we have to do it, uh, just as well in and more efficiently. We have to have the right data structures in place to actually... when you have all these various sources of data, you have to have a combined data structure that enables all of the channels to talk to each other and then you kind of have to have the identity piece too.

Craig Kelly: 08:51 And so the vast majority, our traffic is web traffic, but we also have a massive email infrastructure and we have a rapidly growing a app install base. So how do we manage identities across different, you know, not just the fingerprint thing but across multiple different types of identities that we have and that we rely on. And how do we standardize them into a data structure that allows us to pick the best identity for the right person, um, so that we can follow that person as closely as possible in order to deliver a deliver just what she wants. And so that is kind of the high level and specific goals that we're really trying to solve here. Yeah.

Alex Maguire: 09:46 Set goals. I think some other folks, may share those goals? I have a feeling.  

Alex Maguire: 09:55 What prompted you to look into a customer data platform?

Craig Kelly: 10:05 So this one, we did stumble upon it by accident. I think the fact that we've invested so heavily in the customer data platform space and all of the pieces around it has really enabled us to accelerate our progress more rapidly than I ever could have expected a year ago when we were first looking at this. When we're just reviewing those martech landscape diagrams and they're absolutely insane and we're kind of looking at how it's been growing year over year from five years ago there might have been like 300 players in the Martech landscape, a lumascape type sheet, uh, and now you know, you have, I think 1500 last year. So the whole space is growing so rapidly and so there's this process internally here of, okay, we know what spaces do we want to play in, so not only has the vendor, the number of vendors increased, but the number of spaces has also increased rapidly.

Craig Kelly: 11:17 And what spaces do we want to play in? And then also within each space, what vendor do we want to select and there and there are so, so many choices. And it kind of brought us back to, okay, well, like how do you actually keep up with that? And that's, that's been something that's kind of, you know, sitting in the back of our heads for a while is everything's changing so fast and we have limited resources just like everybody else in the world. So how do we actually build something that can keep up with the pace of change? So we're on the landscape sheet, and there's the customer data platform space and we started to look into it and it's like, "Aha! This is the piece." While we don't believe in silver bullets here, and certainly, nothing is ever perfect, but it's the piece that will build the promise of being able to actually keep up with the pace of change.

Craig Kelly: 12:20 And then as we dug into it more and more, we realized all of these additional benefits of things like a cross-channel communication and taking unstructured data and being able to have generic data structures that will enable us to keep pace with everything. So, you know, there are lots of various pieces that prompted us to look for it. That was kind of by accident. And then when we did start looking into it, it really is, "How do you keep up?"

Alex Maguire: 13:03 That makes a lot of sense. It's certainly the times that we're in today. So with that said, would you talk a little bit to us about your market?

Craig Kelly: 13:24 One thing that's interesting about the past year is that we've basically built our marketing stack from the ground up in here. That includes pieces for push automation and email automation. It includes where we have all kind of these disparate systems. We've upgraded in certain cases and connected the pieces in other cases. But within a year, what, eight months? In eight months in integrating mParticle, we've been able to build an entire interconnected system where our channels actually talk to each other, which is not often heard of. And as we're building, we always keep these five principles in mind, which is what does the marketing stack have to be, or what should it be to reach that ideal state and with the context that we talked about, a big piece of it has to be future-proof.

Craig Kelly: 14:28 And future-proof means that it's modular and it's successful. And so we purposefully lean towards best in class docs as opposed to marketing clouds because we said, okay, the world's changing so fast, no company can actually keep up with the bleeding edge of technology in every single area. So how do we pick the right vendors and then create a system where it's actually easily easy to swap out vendors when one falls behind. So that's when we talk about future proof and it also has to be real-time. Even if we don't have a real-time use case today in a particular area, anything that we build, has to be real-time ready. That's absolute because as we get further into this world, we can't end up in a state where we have to rebuild everything or rebuild pieces from a platforms that were not real-time ready.

Craig Kelly: 15:35 As we go into machine learning and deep learning and trying to create experiences in real-time, everything has to be there. Another piece of that is that it has to be cross-channel. So, you know, we have to really get away from the world get away from the world of siloed channels and it is certainly easier said than done. Especially with limitations around identity, but this is something that has helped us to say we can have 30 percent of our known users, as a random number, talking to one another and creating efficiencies across channels where we can identify them on multiple channels. Well, there's huge value in just that. And then the challenge becomes not having the tools in place to be cross-channel ready and it becomes, "How do we get more pieces of the identity puzzle to get that number higher?"

Craig Kelly: 16:38 It's a mindset that helps us focus on the right problems, but everything has to be built for cross-channel. Another piece is everything has to be rapidly deployable. So when we look at where we want to build our marketing stack versus partner on our marketing stack, we prefer to go with vendors and then we prefer to spend our resources on problems that have not been solved in the market yet or that we don't feel this have been solved sufficiently. And so rapid deployment is really about how do we minimize our development time spent on setting up vendors or integrating systems. To give you an example here, I'll give a shoutout to Justin Vito, who I know was on the line here, but we recently started doing some testing with Braze and when we said we want to have real-time, one-to-one push messaging and an email coordination.

Craig Kelly: 17:53 We've worked with a number of email providers over the years, but because we had mParticle already in place we were able to get Braze up and ready to send emails in eight days. That's a rapid deployment. And when we're not working connected to mParticle, or connected to a CDP layer in general, these are projects that take in the magnitude of years, you know, one year to 18 months. Braze is a great system, and I think the real piece here that stands out to me is with mParticle we're able to have multiple use cases of being able to stand things up in orders and orders of magnitude faster than we were able to before and we can just move on to the next thing.

Craig Kelly: 18:51 And that's how you stay relevant in a rapidly changing world. Another example, when we were doing machine learning on a problem, it would generally take us at least a few weeks to deploy a model into production and habits, then start to talk to our marketing channels. But because of mParticle and the existing integration, since all the channels and new data structure of ideas with attributes, we were able to stand up new machine learning models and have them immediately start talking to production channels or our idea channels or email providers, within a day. So these are the types of changes that this has actually enabled in how we operate in the market. I think that goes back to the market conditions; we are in such heated competition, you know? There are certainly macroeconomic situations, sheet money, that make it a situation where everybody is investing right now into their platforms.

Craig Kelly: 19:57 Ah, and so we have to be able to keep up with that rapid pace of change. And the thing that enables is arbitrage. If we look at ad spend, the advertising side, you really only get super efficient dollars in customer acquisition. When you have arbitrage opportunities and arbitrage is available for some short period of time, and those periods of time are getting shorter and shorter. So how do we create a system where we can jump at new opportunities faster than anybody else and capture those arbitrage opportunities. That's really what we're focused on there.

Alex Maguire: 20:44 Yeah the connection available for the identity of the users within this entire data process. Is that accurate?

Craig Kelly: 20:55 Yeah, I would say it's the identity and it's the single stream of data. So yeah, whether we're getting, if you look at the slide on the screen, it's all of the interactions from every channel and from our website or using the same data structure going into mParticle.  And so when you actually push that into our machine learning engine, you have a unified data set ready. So what it does is it eliminates all of the data engineering needs for much of the day that engineering needs, because you're able to roll up consistent data into machine learning features because you have that consistent identifier at the end of your machine learning process. You're able to write machine learned attributes back into mParticle  that you can take action off of immediately. So you have those persisting connections. You have the data structure and then you have the identity and kind of those three pieces enable this role.

Alex Maguire: 21:57 That's great. Super interesting. This work is amazing.

Craig Kelly: 22:05 We certainly didn't do it ourselves as the feeler, the only two cases, other cases in our principals that are when we need consistent measurement, that goes back to the data. And so when we think about why do we need consistent measurement I'm testing, mParticle has some features or tapping automation, and the velocity of testing is actually the most important thing. So when we actually know; I'm a human being, when I come up with the test that we should do, the chances are the test is not going to help. So if we assume that most of our tests will be wrong, not because of some inherent issue in how we come up with tests, but because of the fact that most tests are wrong, you actually can't improve, your end goals by increasing the quality of your tests. The thing that you actually have to change in order to grow and make improvements faster is to improve the velocity of your testing. So, having those consistent data structures and the ability to toggle on and off as quickly as possible increases our velocity assessing and then, when I'm wrong, 75 percent of the time, it doesn't matter and we can assume that we have some baseline of incorrect test, but as long as we test faster, we will grow faster than our competitors. And then the final piece that we talk about in our marketing stack is identity and that's critical with mParticle, there's a lot of pieces that go into it, but the critical piece there on the CDP layer is how we resolve those identities against each other because we have all these different types of identities from different sources. How do we create a data structure that allows us to pull the poll user information by whichever identity we choose?

Alex Maguire: 24:18 Especially in a business where you're working with a web product as well as mobile. I'm sure that would otherwise be quite challenging.

Craig Kelly: 24:31 It would be a mess otherwise to be perfectly honest, without being able to ping based on the identity that we have at the time based on the context and pull back the same user attributes,  as a user of Amazon web or mobile or vice versa. It keeps the experience consistent for the user will enable us enabling us to take action on that identity in the context in which the user is in.

Alex Maguire: 25:10 Makes sense. So how do you use that? You touched upon this briefly before, but how do you use machine learning models to inform the marketing specifically?

Craig Kelly: 25:18 Yeah, yeah, know there's the question of how do we do it technologically? And so technologically what we do is we stream data from mParticle into our data warehouse. In the data warehouse, we have automated feature roll up so that's kinda where we create user features and we have a pool of features built on mParticle or mParticle data. That's, you know, in the many hundreds to a thousand features, those features then become available to any model. Then from our data warehouse, we have our machine learning platform that plugs into our data warehouse and can pull those features and write back the outputs of whatever model we want back into mParticle as an attribute. That to me is where we start to realize, sorry, we start to push the boundaries and realize the full capabilities of the CDP layer is when we write attributes, rip machine learned attributes into mParticle.

Craig Kelly: 26:32 And so we use that for things like knowing when to send an email, what content to show user, knowing when to recommend the loyalty program, what channel to advertise to somebody on. So for example, if somebody is a consistent opener of emails, we'd probably want to pull them out of audiences on mParticle or on, sorry, on Facebook or Google Ads. And so in mParticle enables us to do that because when we write back into mParticle we can write attributes back there based on their propensity and use that to pull people in and out of audiences on paid channels. So we're not spending money on users who would convert through owned channels otherwise. That would be one example of it. We use mParticle and machine learning for our, in house DSP.

Craig Kelly: 27:44 So we're passing data back into mParticle that gets pushed into a Redis cluster for real-time bidding and so we have that empirical data set. It allows us to identify things like how likely a user is to purchase, so we can adjust how much we're bidding for them in a real-time bidding context. We know what the expected value is of their purchases and that combined with channel propensities really gives us a good concept of how much value there is from a particular user. So we use mParticle there for any machine learning models that are user centric. The output of those models goes into mParticle and it kind of plugs into how all of the different channels speak to each other and then how those different channels behave and the audiences functionality has been really valuable for us there. Not only can we go do things like, say in mParticle, we could say only a few people in this audience who have a conversion propensity over eighty percent and maybe we want to exclude those people and bid lower on them because the incremental value of an ad impression is lower.

Craig Kelly: 29:06 And when we get into more advanced use cases, what we can do is we can just say from our machine learning platform, what audiences this user is in even if there's no bullying string you could put together to determine what audience or users should be in. You can still use that real-time audience compilation feature by defining what audience or users should be in through machine learning and then using mParticle to actually activate those audiences. So I think there's a lot of really interesting use cases there that have been pretty fruitful for us. Yeah, yeah,

Alex Maguire: 29:49 That makes a lot of sense is very sophisticated. It sounds like that must have tremendous impact on, on marketing outcomes as well.

Craig Kelly: 30:07 Yeah. We've seen very significant improvements in efficiency across all of our marketing spend since we started this project about, about eight months ago.

Alex Maguire: 30:15 That's fantastic. So what models are you using for a lot of this machine learning and all the attributes that you spoke of?

Craig Kelly: 30:31 We get into today, we have lots of various efforts going on in the data science side and one thing to just point out there is that we have, we've gotten to a point where our data scientists are doing data science and they're not doing data engineering. So that's really enabled us to dramatically increased the breadth of different problems we're trying to solve with machine learning in the marketing context as far as, you know, some of the ones we're using today. For example, today we have a time of day model, that writes the ideal time of day to contact the user back into mParticle. We use that or push messaging, we use it for email, we use it for our DSP, we have channel propensity models going in, so we know which channels you're most likely to interact with and we use that to pull people in and out of audiences.

Craig Kelly: 31:47 We have models for your expected purchase value, uh, and we use that to determine how much we should bid on you on pay channels and kind of grouped people into audiences have similar values. One huge issue on the API channels, like Facebook and Google Ads, is that you could create a pixel-based the audience, but everybody gets clustered into the same bid. How do you, so you might have users who have a really high expected value versus end users who have a really low expected value and have different capacities all getting the same bid. And so being able to group users with a similar expected value into the same audience, custom audiences on facebook, for example, allows us to give all of those users a similar bid because they have the same those ad impressions are going to create a similar value.

Craig Kelly: 32:54 That's one that we think is huge for creating efficiency. We also look at things like category and product affinity, so when we're sending an email to somebody, what should we show them and there we look at, you know, what categories are they most interested in? What products, product recommendations, and I would say probably for most of the attendees, the concept of personalization started with recommendations, product recommendations, and I think a lot of the world is still still operating in that paradigm. But for us, we've taken allowed us to take a different approach to this, which is that like literally everything should be personalized. Um, and so how do we get that? Are we get there with all of these different types of models? And then we take the outputs of those models. Uh, and we use those in new models of, ah, what experience we should give a user at any given time.

Craig Kelly: 34:00 So there's all these different pieces and I think they're a nice interim play, uh, in many cases and they're nice and safe play in some cases. Um, but as we move towards the world of artificial intelligence, uh, you know, it is coming and we all have to be ready for it. You know, even though if we think of ourselves as retailers or you know, a pure play, ecommerce, whatever it might be, a artificial intelligence is coming and we kind of have to be ready for when it does. Uh, so this is Kinda the first step towards getting to that world and being able to operate effectively in that world with true one to one personalization.

Alex Maguire: 34:43 It's just incredible the level of orchestration to that's involved in this case because you're really looking across a very large and robust marketing program that it's definitely, um, you know, it's, it's incredible that it is possible to orchestrate it. That magnitude.

Craig Kelly: 35:02 Yeah. Yeah, I guess so the uh, the power of the cloud, the elasticity of compute resources kind of has changed the world for us in that respect to enable to, in terms of, ah, being able to scale these programs effectively and, and pretty efficiently. And you know, I think mParticle has been really helpful there.

Alex Maguire: 35:31 It's obvious from user experience just shopping on overstock.com. Would you be able to talk to us a little bit about the, what impact this has had because the marketing programs as a whole?

Craig Kelly: 35:45 Yeah. I would say the most important thing that has changed or the most important impact that mParticle has had on marketing, it's not a headline a for sure, is that it has changed our mindset about what we can actually do and what is possible. As soon as that mindset shifted and we started to say, okay, you have a database of every single user who's visited your site, all of the attributes associated with him or her and all of the events associated with him or her, what could you do with that? The change in mindset has been just staggering and what that's enabled us to do is really come up with a vision for where we want to take our marketing tech and understand, you know, instead of having all of our really smart people in silos in their channels, now we have all of our absolutely best in class people thinking about everything as a singular problem or a singular vision to achieve.

Craig Kelly: 37:10 And no, we're at a point now where, yeah, we're then particles throughout the, you know, working with mParticle and many of our partners, but really having this mindset of what data for marketing might look like a has enabled us to do. We know now where we want to be three years from now and we know exactly how to get there. Um, and that's been the biggest change, uh, things that have kind of come from that. A speed of deployment. Uh, we, you know, as I said, we've basically changed our entire marketing stack in eight months and that includes data warehouses, machine learning platforms, including core infrastructure and core linking. It includes the platforms that we use for messaging. It includes things that we're doing on the content side.

Craig Kelly: 38:09 All of that has changed with a relatively small team and has gotten to a world that's going to enable us to achieve our vision of delivering just what she wants in every single case. We have seen upticks in our conversion rate. Some of our models have had multimillion dollar effects on our bottom line. And I would say the fact that we've had that is because our channels talk to each other. We've had massive improvements in our spend efficiency. We spend a good chunk of dollars every year on advertising and we've been able to improve our efficiency by about 10 percent in the past year through these programs. Which, you know, 10 percent of our budget is a lot of money and that's allowed us to really invest in pushing this platform forward and growing our customer base. Which in the context of our market growth is by far the most important thing right now. To have those extra dollars to be able to push towards that has really been invaluable.

Alex Maguire: 39:35 That's incredible. And so just wondering what are your future plans may look like or kind of where you plan to take this amazing vision next?

Craig Kelly: 39:50 No, I think there's certainly when we think about the whole picture, there's the question of who is the user and all of the details that go along with that. And you know, identity is certainly the hardest part of, uh, of bringing that up to where it needs to be. Um, so I think there will be, we will invest a lot more than identity. Um, we will, uh, continue to invest in the user side. There's this whole other question, okay, all this information about a user, what do you do with it? How do you change their experience given this data? And that's really where the majority of our effort is focused on today is how do we take all this information and connect it to create the absolute best customer experience on the planet.

Alex Maguire: 41:05 In the coming quarters.

Alex Maguire: 41:11 Yeah, we've got a lot of good stuff up our sleeves.

Alex Maguire: 41:15 Absolutely. Well, is there anything else that you want to share or should I, should I start looking at some of the questions came from the audience?

Alex Maguire: 41:23 Yeah, I think we can move on to questions.

Alex Maguire: 41:28 Um, so I think we got one around the machine learning platform and whether not it's in house or a, a third party. Um, also if there are extensions of that particle for machine learning,

Craig Kelly: 41:48 Extensions for machine learning, what we use to get data back into article from our machine learning platform we just see as a content stream. So it's actually really simple. Our machine learning platform, in marketing we work primarily with Databricks, We have some custom applications built on top of that for versioning and amount of deployment, but for the most of the work for the data scientist occurs in Databricks.

Alex Maguire: 42:34 Okay, great. We have one other question. What would you say is the next big thing in marketing technology that overstock is considering?

Craig Kelly: 42:52 I think the next big thing is, is going to be universal identity. We're certainly looking at lots of ways to better identify our customers. I think we were pretty well at the front of the pack now in terms of the systems that we have for creating attributes for users and acting on those attributes. But,  we're still not where we want to be in terms of the percentage of our customers that we can identify in the percentage of our customers that we can speak back to channels with a common identity. So I think there's lots of investments there and that I would say the experience is that it's great to have all of this data, but if it's not informing the customer experience then then that's a problem. And so right now it informs certain ways in which we interact with customers, but there's just so much more that we can do there. And I think that's kind of where we can no a play set ourselves apart in terms of, you know, building things that are unique to.

Alex Maguire: 44:17 Sounds good. And we actually just got in, have the time. Seems like they do. A lot of these questions are questions, which is great. Um, how do you combine attribution data into an article?

Craig Kelly: 44:40 So as far as attributes and data and mParticle, we do all of our attribution outside of mParticle today. And we, you know, we feed that into bidding systems, um, but the attribution and mParticle are disconnected today.

Alex Maguire: 44:55 Yeah. Got It. Okay.

Alex Maguire: 44:59 Yeah, yeah, definitely. And just as an FYI to the attendee that asked that question, when I had been the end user of the mParticle services for Gilt, we did actually use mParticle for our attribution because it allowed us to have actually a lot of similar reasons for, for flexibility in terms of testing various partners as well as, you know, looking at pivoting measurement models. So similar reasons to you Craig, and having that flexibility and future proofing, is definitely big piece of it. And then just one last question that also has come in since the last one. How many data scientists work at overstock is the first part of the question and how has that changed over time and how did the leadership decide to invest in those resources?

Craig Kelly: 46:18 As far as how we increase our investment in data science quite a bit, I think especially over the last year. As far as how leadership they decided to invest in those resources, I don't know if I can speak to that a specifically just because I'm not sure what the executive thought process was behind that. Besides we have all this data and no data science and we want to improve both our efficiency and automation and I think we come from a heavy automation mindset here and I think machine learning plays a lot into being able to automate things effectively. Um, no, I think as far as how there's probably more to that which is kind of how do you. So leadership is  investing in data science resources, and to me that having clear direction as what the output of data science should be and having good problems for it. Um, and I think as we've come up with a couple use cases where our data scientists performed extraordinarily and made big impacts on the business. We've been able to have someone to talk to us, identify more use cases for data science and, and help us see the impact that investment can have.

Alex Maguire: 48:06 It looks like audience creation and orchestration, between them and how that works given the platforms that the mParticle platforms as well as Braze have powerful segmentation capabilities.

Craig Kelly: 48:36 It's good for me to have worked with Andre a couple times, over the past several years. I think they're a bit different. One, as far as Braze goes, who'd be communicated via push or any push an email service, we don't want to segment that. And so we don't. When I think segmentation, I think audiences, we want it to be one to one, and so we don't place any limitations on who can be contacted via email and push. What we do want is to have coordination between the very channels and making sure that users get their pushes in the right place. And we're not just like sending a web push and an email all at the same time to the same user. Then mParticle, as far as our paid and API audiences, we do all those through mParticle. I wouldn't pretend to be an expert on where's the best place to do that, certainly you have multiple very capable platforms. For us it's about creating our audiences closer to the machine learning platform. And that's why we've done most of that with mParticle.

Alex Maguire: 50:28 Thank you so much for taking us through. And for showing us this once in a lifetime, look into the personalization engines such interesting stuff and to all of our listeners as well, if you have any other questions that you didn't get time to ask or have any new ones that come up, please feel free to reach out. And I hope everyone has a wonderful rest of their day.