Exploring 8x8’s Cloud Native Journey with Chief Product Officer Dejan Deklich
Emily and Dejan cover the following points:
8x8’s journey to a leading cloud technology provider.
Why 8x8 decided to migrate to Kubernetes, a move that gave them the flexibility to run workloads wherever they want.
Dejan’s thoughts on the Kubernetes migration, and how it’s helped the company improve its operations. For example, Kubernetes has helped 8x8 migrate away from several legacy systems.
The biggest challenges and surprises that the 8x8 team experienced during their migration journey, such as getting engineering teams to embrace a culture built around monitoring, observability, and documentation.
How 8x8 has avoided “feature bloat” and maintained a product that performs at a high level, while staying true to the features that are important for its core customer base.
The strategy of obtaining buy-in from stakeholders and fellow executives by focusing on business problems, instead of technical issues. This included cost, velocity of innovation, global scale, and so on.
How 8x8’s cloud-native architecture has made it faster and easier to scale.
Transcript
Announcer: Welcome to
The Business of Cloud Native
podcast where we explore how end users talk and think about the transition to Kubernetes and cloud-native architectures.
Emily: Welcome to
The Business of Cloud Native
. I'm Emily Omier, and I am talking with Dejan Deklich, from
.
Dejan: So, I'm the Chief Product Officer at 8x8. To give you an idea, 8x8 is now 16 or 1700 employees worldwide, 450 million in revenue, give or take, offices all over the world, customers all over the world. I'm responsible for all product management, engineering, QA, project management operations for all the products worldwide for 8x8.
Emily: Can you give me a little bit of an idea of 8x8’s history in the Cloud?
Dejan: So, 8x8 has been around, probably, a lot longer than most companies you're talking about. We've been public 30 years, give or take. We have been in the business of communication and collaboration since early 2000s. As you can imagine, we have gone through so many different tech stacks, architectures, and so on, that it is pretty amazing.
We have, in the last several years, done a massive cleanup and rebuild of our software stack. We rebuilt pretty much all of the mobile apps, desktop apps, web apps. We rebuilt the platform starting with billing and provisioning all the way down to how the voice traverses the world. So, it's been a incredible couple of years, incredible journey where I would argue we have gone from the early versions of hosted service to early versions of Cloud, maybe 10 years ago, and we are now what I would like to call a proper cloud technology company. And it's been a very interesting, difficult journey. We learned a lot. We messed up a lot of things, then we learned some more than they did it correctly.
Emily: When you first moved to Kubernetes, and the modern public cloud, what was the rationale? What were their business reasons?
Dejan: Those multiple steps there. We moved to public cloud I don't know, five, six, seven years ago. We ran a lot of things in Amazon. And to be fair, we still also have data centers around the world. So, let me explain quickly what we actually running because I think it's important. So, we have, I think 16 data centers around the world, and then we run in pretty much every region of Amazon, we use Google Cloud extensively, and we have now shifted a lot of workloads to Oracle Cloud. At the same time, business is threatening me with Alibaba Cloud and Tencent Cloud as something that might be coming our way in the next couple of quarters. So, data centers are there because on the networking layer, the Cloud does not yet give us what we need for the realtime voice and video transmission.
We actually are the best voice provider in the industry. We have proven that, and that's where your milliseconds really matter, therefore networking still sits in data centers. As soon as the backbone can be moved into Amazon, and we are told that could happen in the next three to four years, we will move likely everything to the Cloud. So, what we have generally in the Cloud are different applications, and the reason for that is simply the velocity of deploying and scaling them.
So, what matters to us is, on one hand, the global reach: we have customers in 150 countries around the world. We have to have data centers close to the customers. And the applications need to be as close to the customer as possible, therefore all the different regions of Amazon, and Google, and whatnot. So, as you can imagine, managing all of that, monitoring all of that is a non-trivial exercise.
So, we moved to Kubernetes, in large reason, simply because it is one underlying framework that allows us to run workloads wherever we want. So, to give you an idea, we launched a video meetings product to compete with Zoom. We had, on launch, a couple of hundred thousand users, nothing really. And then, this COVID-19 happened, and within a period of weeks, we now hit 15 million users. The only way you can scale a system like that is if you have a properly built underlying architecture, everything horizontally scalable.
I was blown away, everything really worked. People were super busy, but by having proper cloud architecture, we were able to actually scale, and fulfill the demand that we have seen worldwide. Now, the nice thing is, as you put more and more workloads on top of Kubernetes, you can shift them between clouds as you want, or data centers as you want. And I think that's number one reason why we went with Kubernetes.
I love Amazon, I love Google, and nothing makes me happier than writing them a million-dollar checks, but I also want to be able to move the workloads wherever I can run them cheaply. And, to me, that's very important. I don't have unlimited budget; I have to be able to play the game and get the most compute and the most bandwidth for the lowest cost that I can, and Kubernetes lets me do that.
Emily: And would you say that Kubernetes was a technical decision or a business decision or both?
Dejan: That's a good question. I think normally, the way we operate at 8x8, you start with the business problem. The business problem was we don't want to be locked into one cloud. We want to be able to run wherever we want to run, and on top of that, we have customers in Europe who are not very friendly towards Amazon, and want us to run on other clouds. And then, we took a peek: what can we do? What's the fastest and easiest way to do it? Turned out it was Kubernetes, so that's the way we went.
Emily: What did the move to Kubernetes, what was it like? What were some of the surprises?
Dejan: It was very interesting. It is still very interesting. So, on one hand, the good thing was we have already broken the monoliths in the past God knows how many years, into services. But to get things running properly in Kubernetes, you have to go a bit deeper, you actually have to really clean up your code, and so on, and so on. So, one thing that I thought was incredibly useful was this allowed us to, for the first time in 8x8 history, create a proper template for a service where all your monitoring, logging, debugging, all of your stuff is standard for all the services.
So, as you can imagine, when you have a company that's been around for a long time, and you have a lot of services, and through various acquisition, you end up with different languages, different platforms, and so on, this now allowed us to clean up a lot of legacy and actually get to what I consider a really impressive engineering velocity. So, super challenging in many ways, because people had to say goodbye to some of the legacy tools, or ways of thinking, and so on. On the other hand, super-positive from the velocity we are now seeing. And honestly, if we haven't done some of these things, I doubt our platform would have survived the spike of the last two months.
Emily: What were some surprises in moving to Kubernetes? And you mentioned that you made some mistakes and learned a lot. What were some of those?
Dejan: It's funny, the good old story of automation and orchestration comes to bite you in the rear end every time, again. It's all nice when you have 10, 20 services. People can keep things in their mind. Once you are getting to hundreds or thousands of services that are running all over the world, things that you normally never want to do, such as documentation, such as fully automated CD/CI really become an absolute necessity. They stop being something that you learned about in your software engineering 101, and they become simply something that you really have to hardcore do.
And getting the engineering teams to accept that and changing the culture towards more documentation, more monitoring, more observability, was the key. It was also really interesting to see all these different monitoring tools, how they either couldn't scale, or if they were able to scale the cost would explode, or getting the insight out of the data that was being collected was next to impossible, or you needed somebody who spent their whole life looking into Splunk or something like that in order to figure out what's really going on. The release process had to be completely changed. We completely rebuilt—we blew up the CI/CD pipelines completely, rebuilt everything cleanly, and so on, and so on.
So, lots of very difficult engineering challenges, and I would argue the hardest thing was selling some of that work to the business. Business is always coming to you and saying, “Hey, we need feature XYZ, we need this product, we need that product.” And we would have to go back and say, “Well, actually, what we need is the availability, and the uptime, and stability, and we need to know what is really going on with the system so we can continue scaling it.” And looking back, I am really glad we did what we did because if we haven't started down this way, the 15 million users would not have happened.
Emily: Tell me a little bit more about making that sale to the business. Was there anything that's lost in translation, more about that conversation with the business about why this needed to happen?
Dejan: Right. So, it is very, very interesting. Most—and I have seen that pretty much in every company I've ever worked. There is always this feature bloat, and we are all guilty of that. And for sales, it is very easy to go to a prospect and go, “Hey, how do you like this?” Then prospects go, “Well, how about this and that feature?” sales says, “Absolutely.” And you end up with a roadmap which becomes this incredible sea-urchin-like monstrosity that goes in every possible dimension, and the cleanliness of the original product vision disappears over the years.
You end up with this extremely complex enterprise product, which does everything for everybody, and does everything badly for everybody. So, I don't believe that's the right way to do business. And I was lucky enough that my CEO and the board fully agreed with my view that we need to go back to the core; we need to figure out what's really important to our customers, and as bad as this COVID-19 is, and as much chaos as this caused around the world, it was really gratifying for us inside 8x8 to see that all the work we have done in the last few years was actually done correctly with a correct view of the future. Think about it, a few years ago, everybody was always debating, “Oh, should I have engineers working from home, or not? Are people more efficient working from home or not?”
Now, everybody's working from home. There is no more choice. There is no more talking about that topic. You have to be able to function from home. 8x8 builds the tools that ultimately allow enterprises to have their people function from home, both on the unified communication, and the contact center, and the video meeting side. So, you go, this now opens up the whole universe; I can hire people in anywhere around the world, we can do business anywhere around the world. And to me, that's the right way to go forward, but my job was to articulate what might happen, and what is likely to happen in the world and the economy in the coming five to 10 years, and this COVID-19 thing just pulled all of this work from home thing into now and not into next two, three, five years.
Emily: When you were talking to business stakeholders about moving to Kubernetes and why it was important. Was there anything that you felt like was lost in translation?
Dejan: They don't care about Kubernetes at all, and I never talked to them about Kubernetes. I talked to them about the scale and performance. Your CEO, your CFO, your Chairman of the Board, they don't care about Kubernetes or any technology. They care about the business problem. And the business problem is cost, velocity of innovation, global scale, global delivery, size of the engineering team, funding for R&D, you have to articulate what you want to do in the terms that matter to the business.
Technology is one of many ways to solve the problem. If you could have trained monkeys do something super fast and super cheap, it might be a valid way to solve a problem. As we are lacking trained, cheap monkeys, we have to use technology. And for a lot of us technology is a great, great toy that we love to play with, and we enjoyed dealing with it, so it is sort of a way we solve the problem. The way a business solves the problem is effectively through the P&L.
So, this translation between technical and business has to happen, and if you can't tell the story correctly, you will not get the funding you need, you will not get the project that you need, and ultimately the business will fail. And I think that's where the challenge lies. I see a lot of engineers and product people talk about the features, about how cool this or that would be, but they fail to forget that in the end, in a publicly-traded company, CFO and CEO go on the stage quarterly, and say, “This is the money we made. This is the money we spent.” And if you can help tell your story and the story of a product and engineering in the terms that they can use with the outside world, everybody's life becomes much easier.
Emily: How did that conversation progress as you progressed on the technical transition?
Dejan: So, it's like every other conversation. First time I said, “Yo, guys, we need to stop the feature work.” Obviously I got completely blank looks from everybody along the lines of, “Why would you ever want to do that?” after a couple of quarters of improving the engineering velocity, improving the product quality, stability, and so on, and so on. I think the whole company is now on the bandwagon. I think everybody by now understood why we are doing what we are doing.
Which is great. If initially, there was some pull back from Sales—Sales loves more features, it makes their life easier, but there is also, I have to tell you now, a very visible and noticeable pride in the whole company about what we were able to do, about the scale we have achieved, and how quickly and painlessly we actually got there. So, it was not easy, first few months, I have to tell you that.
Emily: What would you say were the biggest challenges?
Dejan: Changing the culture. Changing the culture for the whole company to stop chasing features and start chasing elegance, and speed, and stability, and uptime, and global scale, on one hand; on the engineering side, it was about, stop committing to featurettes, and start thinking about larger architectural projects that will massively improve the company; on the sales side, it was about how do I stop talking about the future roadmap and start talking about the vision of the company and the vision of the product. Those are very noticeable and hard to pull off changes, and it took a lot of effort from all the teams to actually get on the bandwagon.
Emily: Was there anything that was much easier than you expected?
Dejan: [laughs]. Probably the core technical parts. The nice thing with good engineers is that they always love to try the latest and greatest. So, I was worried that people here will have objections to changing some of the frameworks and so on. Turned out that that was absolutely not the problem. I was delighted that the engineers and PMs were able to jump on the latest and greatest and just go run with it, learn what they had to learn, then go forward. In previous companies, I have seen that there's much more of a challenge than at 8x8.
Emily: And then, from a technical perspective, what was more challenging than you expected, or what was a pain point?
Dejan: The CI/CD pipeline. It was really interesting how—it's almost once you have gone down the cloud way, you have to clean up how you build and release software. And most companies get to this sort of in-between state where there is a part of automation, but not everything is fully automated, and so on, and so on. You have to really talk to the people at really big companies to get to the point where there is proper really end-to-end CI/CD.
So, we were also in this in-between world. Yes, it's automated, yes it’s CI/CD, but not really. And we spent a tremendous amount of time and energy into cleaning that up and finding all the edge cases. And honestly, I don't think that work will ever end. It seems to be as soon as we finish one project, we find other things that can be improved and cleaned up, and then we go and do that. So, by far, the biggest challenge was the Continuous Integration. Getting that to the point where it really works is a lot, a lot of work.
Emily: Would you consider that among your biggest continuing challenges? Are there any other sort of challenges that you haven't quite figured out yet?
Dejan: I think this is one of the bigger ones because we are not the only AWS, or only Google Cloud, or only data center. We have a mix of everything, and a mix of different applications and platform components that have to reside all over the world. And it is stunning how much work goes into this. The other very interesting set of problems that we are encountering is on the data side, and figuring out—I would argue we haven't yet really fully figured out the data storage requirements.
So, to give you an idea—and it's not really so much even us figuring it out, it's the world figuring it out—every country has its own set of compliance regulations, security directives, they're all good, but how do you actually comply with all of them if you are providing service in 150 countries is a different problem. It almost feels like somebody on sand hill should invest in a startup which builds legal-as-a-service, something where the security and compliance will be done at the file system level so my engineers don't have to worry about it. To give you an idea, say you have a conversation, you and me: one of us sits in Europe, one of us sits in US. Who’s data privacy laws win? Where can the data reside? How do you store the data? What happens if you have an employee of a US company which resides in Europe? How is that data handled? So, there's a lot of these unknowns on the data security side, which are really interesting, which are really complicated. And I think if somebody manages to build a service around that, they will do very, very well.
Emily: Yeah. You have the employee of a US company who lives in Europe but is on vacation in Africa.
Dejan: Exactly. And you go, “Okay, does GDPR win? Does the US win? Does—” how do we do this?
Emily: Tell me just a little bit more about how Kubernetes and how the cloud-native architecture has made it possible to scale in the past month or so?
Dejan: Well, by removing all manual steps, and by having very clean containers that can be deployed anywhere, we were able to scale roughly 100x. And then, to make it even better, we were able to move from Amazon to Oracle Cloud. We did a very interesting collaboration with Oracle, where we agreed that we want to provide the highest possible security to our video product. We felt we can get that security and enterprise presence through Oracle Cloud, and not only did we scale up video product in the last I would say two months by 100x, we also moved very large parts over to Oracle in order to manage cost and get increased security. 100 percent possible through modern, latest and greatest architecture and automation. Otherwise, I don't know how we wouldn't have done it.
Emily: Has there been anything that surprised you over the course of this past two months, as you've seen the scaling. Any problems you encountered, for example, that you hadn't anticipated?
Dejan: It was interesting to see some of the monitoring tools have issues, as well as seeing the cost of all of these things really explode. It was also interesting to see the response of various cloud providers. As soon as we saw the cost explode, we reached out to everybody. It was really interesting to see how some are much more open to collaborating, others are less open to collaborating. It's been a very interesting two months.
But purely on the technical side, it's the monitoring that actually showed how important it is to be able to know what is really going on everywhere, and monitoring couple of thousand instances around the world, while possible is not necessarily easy.
Emily: In terms of things that went wrong, or things that went right, any surprises or any notable examples?
Dejan: Nothing—I was amazed that nothing really went wrong. I mean, we really worked our butts off, don't get me wrong. I mean, people have been up 16, 18 hour days for weeks to make sure everything works, and so on. But the fact that there were no serious problems still is amazing to me. I mean, 100x ramp in two months is a very nice ramp.
I was amazed with how well the team reacted to this. There was really no complaining. This is what we were all working towards: sort of the best possible technical time mixed with the worst possible personal time because of the virus. So, it's been really interesting to see how people all over the world really stepped up, and kicked butt, and helped us move forward.
Emily: Any comments you would have for other companies that are maybe a little behind 8x8 on the cloud transition?
Dejan: Yes, figure out your platform. If you don't have a good underlying platform for everything, you will end up paying for it. I'm hoping most people are done forklifting legacy products into Cloud. If you are still in that phase, really start thinking about building cloud-native. And it's not cheap, it's not easy, it's not trivial, but you have to do it if you want to survive. Otherwise, when something like this virus happens to you, you will not be able to scale, and you will fail at the worst possible time. Cloud-native is the way to go. It's the future. Embrace it, but don't think it will all be roses along the way.
Emily: Any other thoughts you have about the business of cloud-native, anything else you'd like to add?
Dejan: I think the very interesting thing on cloud-native is, cloud-native has many different forms, and thinking across multiple clouds, both public and private, I think is the way to go. I'm sure there are applications out there that are perfectly fine running in a single cloud and scaling in a single cloud. But I would imagine for the vast majority of enterprises, they are in a very similar world to 8x8, where you will be faced with challenges, across the whole world, which are better solved in different cloud providers at a different time. And the sooner you start thinking about your multi-cloud strategy, and how you will shift from one to the other, the better off you will be. And then, if you can throw in the private cloud into all of that, that's where I believe things become truly interesting, and probably the future is, in some form or the other.
Emily: All right. Well, I think we can go ahead and wrap it up there. Thank you so much for joining me. This was really great.
Announcer: Thank you for listening to
The Business of Cloud Native
podcast. Keep up with the latest on the podcast at
and subscribe on iTunes, Spotify, Google Podcasts, or wherever fine podcasts are distributed. We'll see you next time.
This has been HumblePod production. Stay humble.