CERN’s Transition to Containerization and Kubernetes with Ricardo Rocha

The European Organization for Nuclear Research (CERN), has produced some staggering accomplishments over the years, such as the discovery of the famous Higgs Boson, or “God particle,” and the internet. What often gets overlooked, though, is the sheer amount of data the organization is creating on a daily basis, and how the organization manages it.

In this episode of The Business of Cloud Native, host Emily Omier talks with Ricardo Rocha who is a software engineer and architect at CERN. Ricardo provides an overview of how CERN is managing data today, as well as what led the organization to explore containerization and Kubernetes, and the benefits and challenges that it has produced for their staff members.

Some of the highlights of the show include: 

  • The challenges that CERN was facing when storing, processing, and analyzing data, and why it pushed them to think about containerization.

  • CERN’s evolution from using mainframes, to physical commodity hardware, to virtualization and private clouds, and eventually to containers. Ricardo also explains how the migration to containerization and Kubernetes was started.

  • Why there was a big push from groups that focus on reproducibility to explore containerization.

  • How end users have responded to Kubernetes and containers. Ricardo talks about the steep Kubernetes learning curve, and how they dealt with frustration and resistance.

  • Some of top benefits of migrating to Kubernetes, and the impact that the move has had on their end users.

  • Current challenges that CERN is working through, regarding hybrid infrastructure and rising data loads. Ricardo also talks about how CERN optimizes system resources for their scientists, and what it’s like operating as a public sector organization.

  • How CERN handles large data transfers.

Links

Transcript

Emily: Hi everyone. I’m Emily Omier, your host, and my day job is helping companies position themselves in the cloud-native ecosystem so that their product’s value is obvious to end-users. I started this podcast because organizations embark on the cloud naive journey for business reasons, but in general, the industry doesn’t talk about them. Instead, we talk a lot about technical reasons. I’m hoping that with this podcast, we focus more on the business goals and business motivations that lead organizations to adopt cloud-native and Kubernetes. I hope you’ll join me.

Emily: Welcome to the Business of Cloud Native. I'm your host, Emily Omier, and today I'm here with Ricardo Rocha. Ricardo, thank you so much for joining us.

Ricardo: It's a pleasure.

Emily: Ricardo, can you actually go ahead and introduce yourself: where you work, and what you do?

Ricardo: Yeah, yes, sure. I work at CERN, the European Organization for Nuclear Research. I'm a software engineer and I work in the CERN IT department. I've done quite a few different things in the past in the organization, including software development in the areas of storage and monitoring, and also distributed computing. But right now, I'm part of the CERN Cloud Team, and we manage the CERN private cloud and all the resources we have. And I focus mostly on networking and containerization, so Kubernetes and all these new technologies.

Emily: And on a day to day basis, what do you usually do? What sort of activities are you actually doing?

Ricardo: Yeah. So, it's mostly making sure we provide the infrastructure that our physics users and experiments require, and also the people on campus. So, CERN is a pretty large organization. We have around 10,000 people on-site, and many more around the world that depend on our resources. So, we operate private clouds, we basically do DevOps-style work. And we have a team dedicated for the Cloud, but also for other areas of the data center. And it's mostly making sure everything operates correctly; try to automate more and more, so we do some improvements gradually; and then giving support to our users.

Emily: Just so everyone knows, can you tell a little bit more about what kind of work is done at CERN? What kind of experiments people are running?

Ricardo: Our main goal is fundamental research. So, we try to answer some questions about the universe. So, what's dark matter? What's dark energy? Why don't we see antimatter? And similar questions. And for that, we build very large experiments. 

So, the biggest experiment we have, which is actually the biggest scientific experiment ever built, is the Large Hadron Collider, and this is a particle accelerator that accelerates two beams of protons in opposite directions, and we make them collide at very specific points where we build this very large physics experiments that try to understand what happens in these collisions and try to look for new physics. And in reality, what happens with these collisions is that we generate large amounts of data that need to be stored, and processed, and analyzed, so the IT infrastructure that we support, it’s larger fraction dedicated to this physics analysis.

Emily: Tell me a little bit more about some of the challenges related to processing and storing the huge amount of data that you have. And also, how this has evolved, and how it pushed you to think about containerization.

Ricardo: The big challenge we have is the amount of data that we have to support. So, these experiments, each of the experiments, at the moment of the collisions, it can generate data in the order of one petabyte a second. This is, of course, not something we can handle, so the first thing we do, we use these hardware triggers to filter this data quite significantly, but we still generate, per experiment, something like a few gigabytes a second, so up to 10 gigabytes a second. And this we have to store, and then we have large farms that will handle the processing and the reconstruction of all of this. So, we've had these sort of experiments since quite a while, and to analyze all of this, we need a large amount of resources, and with time. 

If you come and visit CERN, you can see a bit of the history of computing, kind of evolving with what we used to have in the past in our data center. But it's mostly—we used to have large mainframes, that now it's more in the movies that we see them, but we used to have quite a few of those. And then we transitioned to physical commodity hardware with Linux servers. Eventually introduced virtualization and private clouds to improve the efficiency and the provisioning of these resources to our users, and then eventually, we moved to containers and the main motivation is always to try to be as efficient as possible, and to speed up this process of provisioning resources, and be more flexible in the way we assign compute and also storage. 

What we've seen is that in the move from physical to virtualization, we saw that the provisioning and maintenance got significantly improved. What we see with containerization is the extra speed in also deployment and update of the applications that run on those resources. And we also see an improving resource utilization. We already had the possibility to improve quite a bit with virtualization by doing things like overcommit, but with containers, we can go one step further by doing more efficient resource sharing for the different applications we have to run.

Emily: Is the amount of data that you're processing stable? Is it steadily increasing, have spikes, a combination?

Ricardo: So, the way it works is, we have what we call ‘beam’ which is when we actually have protons circulating in the accelerator. And during these periods, we try to get as much collisions as possible, which means as much data as possible because this is the main ingredient we need to do physics analysis. And then with the current experiment, actually, we are quite efficient. So, we have a steady stream of data we generate from raw data is something like 70 petabytes a year, and then out of that data, we generate a lot more data be it like pre-processing, or physics analysis. So, in terms of data, we can pretty much calculate how much storage we need. What we see spikes is in the computing part.

Emily: And is the amount of data, are you constantly having to increase your data storage capacity?

Ricardo: Yeah, so, yeah, that's one aspect is we never—from the raw data, we keep it forever. So, actually, we need to increase what we do, which is kind of particular to the way CERN has been doing things is we rely a lot on tape for archival. And this has allowed us to—because states still evolve, and the capacity of the tape still evolves, this actually allows us to increase the capacity on our data center without having to get more space or more robots. We just increase the capacity of the tape storage. And then for the on-disk storage, though, we do have to increase capacity regularly.

Emily: Who spearheaded the push to containerization and Kubernetes? Was it within the IT? Who in the organization was really pushing for it?

Ricardo: Yeah. That's an interesting question because it happens gradually in different places. I wouldn't say it's a decision taken from top; bottom. It's more like a movement that starts slowly. And there's two main triggers, I would say. 

One of them is from the IT department, the people run the services because some of our engineers are exposed to these new technologies frequently, and there's always someone that sees some benefit, does a prototype, and then presents to their colleagues and this kind of trigger. So, it's more on the sense of simplifying operations, deployment, all the monitoring that is required, and also improve resource usage. So, this was one of the triggers. 

The other trigger is actually from our end users. One big aspect of science, and specifically here—physics—is that there's a big need to share a setup. When someone performs some analysis, it's quite common that they want to share the exact same setup with their colleagues, and make sure that this analysis is reproducible. So, there was a huge push from groups that focus on reproducibility to explore containerization and the possibility to just wrap all the dependencies in one piece of software and make sure that even if things change dramatically in the infrastructure, they still have one blob that can be run at any moment, be it in five years, ten years, and that they also can share easily with their colleagues.

Emily: Do you think that that's been accomplished? Has it made it easier for the physicists who collaborate?

Ricardo: I think, yeah, from the experience that I have working with some groups in this area, I think that that has been accomplished. And actually last year, we did some exercise that had two goals: one was to make sure that we could scale out using these technologies to a large amount of resources, and the second was that we could prove that this technology can be used for reusability and reproducibility. And we took, as an example, the analysis that originated the Nobel Prize in 2013, which was the analysis that found the Higgs boson at the time. So, this was code that was pretty old, and the run for this analysis was done in 2012, with old hardware, old code, and what we did is we wrapped the code as it was at the time in a container, and we could run it in infrastructure that is modern today using Kubernetes, and containers, and we could reproduce the results in a much faster way because we have better resources and all this new technologies, but using exactly the same code that was used at the time.

Emily: Wow. And how would you say that moving to Kubernetes and to containers has changed the experience for end-users for the scientists?

Ricardo: Yeah. So, there, I think it depends on which—how to put it—which step of the transition you're at, I think. There's clearly—like, the people that are familiar with Kubernetes, and at ease with these kind of technologies, they will tell about the benefits quite easily. But one thing that we saw is that unlike the transition from physical to virtualization where the resources still felt the same—you would still SSH to your machines, and the tools inside the virtual machines were still very similar—an, like, that—the transition to containerization and to Kubernetes has a steep learning curve. So, we have both some resistance from some people and also some frustration at the start. So, we promote all these trainings, and we do webinars internally to try to help with this, this jump. 

But then the measure of benefits once you start using this are quite clear. Like if you're managing a service, you see that deploying updates becomes much much easier. And then if you have to monitor, restart, collect metrics, all of this is well integrated because of this way of declaring how applications should look like, and how they should work, and just delegating to the orchestrator on how to handle this. I think yeah, people have seen this benefit quite a lot from the infrastructure part, from the service management as well.

Emily: What do you think has been surprising or unexpected in the transition to Kubernetes and containers? I think the first thing that I'm interested in is unpleasant surprises: things that were harder than expected.

Ricardo: Yep. I think this steep learning curve is one. Yeah, when you start using containers, it's not that complicated. Like if you—I don’t know—build a Docker file, create your container is not that complicated, but when you jump in the full system, we do see some people having some kind of resistance, I would say, at the start. And it's quite important to have this kind of dissemination activities internally. 

And then the other thing that we realized—but it's kind of specific to our infrastructure because I mentioned that we have all these needs for computing capacity, and because we have a large data center, but it's not enough, we actually have a network of institutes around the world and data centers that collaborate with us, and that we developed this technology over the last almost 20 years now. And we had our own ways to, for example, distribute software to these sites. And by moving to containers, this poses a challenge because then the software now is wrapped in this well—defined image; it's distributed by registries, and this doesn't really match all the systems we had before. So, we have to rethink all these aspects of a very large, distributed infrastructure that has limitations, but we know about them and we know how to deal with them, and really learn how to do this when we start using containers, how to distribute the software, how to distribute the images and all the tracking that is required around this. So, for us in this area, it's not something we thought at the start, but it became pretty obvious after a while.

Emily: Makes sense. What specific resistance did you meet, and how did you overcome it?

Ricardo: Yeah, so I think we really need to show the benefit in some cases. We have people that are very proactive, of course, into exploring these technologies, but in some cases—because for many people, this is not their core work. So, if you're a physicist, you want to execute your analysis or get your workloads done. You don't necessarily want to know all the details about infrastructure, so you need to hide this. But if you are involved in this, you kind of have to relearn a bit, how to do system operations and service, how you do your tasks of service manager. 

So, you really need a learning process to understand how containers work, how you deploy things in Kubernetes. You have to redefine a lot of the structure of how your application is deployed. If you had a configuration management system that you were using—Puppet, Ansible, whatever—you have to transition to this new model of deploying things. So, I think if you don't see this as part of your core work, you are less motivated to do the transition. But yeah, I think this happens. It happened in the past already, like when we transition to automated configuration management, when we change configuration management systems, there’s always a change that is required. 

Yeah, we are also lucky that in this research area, people are open-minded. So, yeah, we do organize a lot of different activities in-house. We do what we call ‘container office hours’ weekly, where people can come and ask questions. We will organize regular webinars in different topics, and we have a lot of internal communication systems where we try to be responsive.

Emily: What do you think are the benefits that, when you encounter somebody who's not super excited about this—or, when you encountered—what are the benefits that you usually highlight?

Ricardo: Yeah. So, actually, we've been collecting from some users that have finished this transition what do they think, and the main benefits they say is in the speed of provisioning. In our previous setup, sometimes you can think that provisioning a cluster can take up to an hour, even in some cases more, to provision a full cluster, while now provisioning a full Kubernetes cluster, even a large one can take less than 10 minutes. And then even more when you do the deployment of the applications or when you do an update to an application, because of the way things are structured, you can really trust that this will happen within seconds and that you can apply these changes very quickly. If you have a large application that is distributed in many nodes, even if you have an automation system, this can take several minutes, or up to an hour or more, for that change to be applied everywhere, while with this containerization and orchestration offered by Kubernetes we can reduce that to just a couple of seconds and be sure that the state is correct; we don't have to do some extra modifications; we just trust the system in most cases. I think those have been the two main aspects. 

And then for more advanced use cases, I would say that one benefit is that in some cases, we have tools that are deployed at CERN, but they are also deployed in other sites—as I mentioned, we have this large network—and one thing, because of the how popular Kubernetes became and these technologies, is that they can develop the application at CERN, rapid, and make sure that it runs well on our Kubernetes system, and in most cases, they can just take that, deploy in a different site or in a public cloud, and with no or very little modifications, they will get a similar setup running there. So, that's also a key aspect, is that we reduced the amount of work that is going into custom deployments here and there.

Emily: And would that make it easier for a scientist to, say, collaborate with somebody at a different location?

Ricardo: Yeah, I think that's also one aspect, is that the systems start feeling quite familiar because they're pretty similar in all these places. This is something that is happening in our own infrastructure, but if we look at public clouds, all of them offer some kind of managed Kubernetes service that in some cases have slight differences but deal—in its overall functionality, they're pretty similar. They also have the right abstractions on computing, and also on storage and network so that you don't have to know, necessarily, the details underneath.

Emily: And does this also translate to physicists being able to complete their research faster?

Ricardo: Yeah. So, that's another aspect that we've been investigating is that on-premise, we have this capability to improve our resource usage. So, hopefully, we can do more with the same amount of hardware, but what we've seen is that we can also explore more on-demand resources. So, I mentioned earlier this trial that we did with this Higgs analysis, and the original analysis takes almost a day. We containerized and tried to wrap it all, and we managed to run a subset of the analysis on our infrastructure—but of course, it's a production infrastructure, so we can just ask for all the 300,000 cores to run a test; we get a very small fraction of that if we want to do something like this—but what we showed is that by going, for example, to the public cloud, because the systems are offered in the same way, we took the exact same application, we deployed it in a public cloud, in this case into GCP, using GKE, and we can trust, or we can imagine that they're offering us infinite capacity as long as we pay, but we only pay for what we use. 

And basically, we could scale-out, and in this experiment, we scaled out to 25,000 cores. And we could reduce what took initially one day, and then eight hours, we reduced it to something less than six minutes. And that's all also all we pay. Like when we finished the analysis, we just get rid of the resources and stop paying. So, this ability to scale out in this way, explore more resources in different places—we do the same with the HPC computers as well—kind of opens up a lot of possibilities for our physicists to still rely on our large capacity, but explore this possibility of getting extra capacity and do their analysis much faster if they have access to this extra resources.

Emily: Do you ever feel like there's miscommunications or mistranslations that happen between your team—the cloud team, and the end-users—the physicists?

Ricardo: Well, I think their main goal is to get access to resources, so miscommunications usually happen, when the expectations are not met somehow. And this is usually because we have a large amount of users, and we have to prioritize resources. So, sometimes, yeah, sometimes they will want their resources a bit faster than they can get. So, hopefully, this ability to explore more resources from other places will also help in this area.

Emily: Since starting the Cloud transition, what do you think has been easier than expected, either for your team or for the scientists that are using them?

Ricardo: Well, we've done the transition to Cloud itself already back in 2013, offering this API based provisioning of resources. I think one benefit that we had was that in the initial transition to the Cloud, we rely on a software stack of OpenStack for the private cloud, and also, we now rely on Kubernetes and other products that are part of CNCF. For the containerization part, I think one big thing here has been that we always got very involved with the upstream communities, and this has paid off quite a lot. We also contribute a lot to the upstream projects. I think that's the key aspect is that we suddenly realized that things that we used to have to build in-house because we always had big data, and we had to have systems that process all this amount of data, but as big data becomes a more generic problem for other industries and companies, these communities have been built to handle this, and this has been one key aspect that we realize now that we are not alone. We can just join these communities, participate, and contribute to them, learn with them, and we give some but we get a lot more back from these communities.

Emily: What are you looking for from these communities now? And what I'm trying to ask is what's the next step, the problems that you haven't solved yet that you're actively working on?

Ricardo: Yeah, so our main problem right now—well, it's not a problem. It's a challenge I would say—it's that even if we have a lot of data, we are actually having a big upgrade of our experiment in just a few years, where the amount of data could well be increased by something like 100 times, so this gets us to amounts of data that are again pushing the boundaries, and we need to figure out how to handle that. But the big challenge right now is to really understand how to have hybrid infrastructure where our workloads—what we try to do is hide this fact that the infrastructure is maybe not all in our data center, but some bits are in public clouds or other centers, we try to hide this to our user. So, this hybrid approach with the clusters spanning multiple providers is something that still has open issues in terms of networking, but also because we depend so much on this amount of data, we need to understand how to distribute this data in an efficient way and make sure that the workloads run in this places correctly. If you've used public clouds before, you realize quickly that it's very easy to get things up and running, it's very easy to push data, but then when you want to have a more distributed infrastructure and move data around, there's a lot of challenges, both technically and financially. So, these are challenges that we'll have to tackle in the next couple of months or years.

Emily: Yeah, in fact, I was going to ask about how you manage data transfers. I know that could be expensive, and just difficult to manage. Is that something that you feel like you've solved or…?

Ricardo: Yeah, so, technically we know how to do it because we have this past experience with large distributed infrastructures, like this large collection of centers around the world that I mentioned earlier. We have dedicated links between all the sites, and we know how to manage this, and how to efficiently move the data around. And we also are quite well connected to big hubs, to the scientific network hubs in Europe like [JANET], but also direct links to some cloud providers. So, technically we've experimented this, we can have efficient movement of the data we have built over the years to systems that can handle this, and we have the physical infrastructure as well. I think the challenge is understanding the cost models of this: when things are worth moving around, or just move the workloads to where the data is, and how to tackle this. So, we need to understand better how we do this, and also how we account our users when this is happening. This is not a problem that we’ve solved yet.

Emily: Do you also have to have any way to put guardrails on what scientists can and cannot do? To what extent do you need to manage centrally and then let the scientists do whatever they want, but within a certain parameter?

Ricardo: Yeah, so our scientists are grouped in what we call experiments, which are linked to physical experiments underground. These groups are well defined, and the amount of resources that they get are constantly renegotiated, but they are well defined. So, we know which share go to each group of users, but one very important point for us is that we need to try to optimize resource usage. So, if for some reason, a group that has a certain amount of resources allocated is not using them, we allow other groups to just increase a bit—temporarily—their usage, but then we need this mechanism of fair share to balance the usage over a large amount of time. So, using this model on-premises, we know how to do the accounting and management of this. Integrating this to a more flexible infrastructure where we can explore resources from different cloud providers and centers, yeah, it's something we are working on. But we do have this requirement that all of it has to be accounted, and over time, we have to make sure that everyone gets their share.

Emily: Do you think everybody feels like the goals that you set out in moving to containers and Kubernetes, that you've met those goals? I mean, do you think everybody’s, sort of, satisfied that it was a success?

Ricardo: Yeah, I'm not sure we are at that point, yet. I think when we present these exercises, like, what we did that very large scale, that there is a clear improvement. I think people get excited about this. Now, we need to make sure that all of this is offered in a very easy way so that people are happy. But what it also shows—this kind of experiments, and I think it's where physicists get more interested, also, in this kind of technology, is that it makes it much easier to explore not only generic computing resources, but exploring dedicated resources such as accelerators, GPUs, or more custom accelerators like TPUs and IPUs, that we wouldn't have access, necessarily, in our data center, when we offer the potential to have a much easier way to explore these resources and have some sort of central accounting, then, yeah, they get excited because they know that this on-demand offering of GPUs has a lot of potential to speed up their analysis, and without having to overprovision our internal resources, which is something that we wouldn't be allowed to do.

Emily: I'm imagining you don't really have much of a comparison, but I'm thinking about what's unique about being a public sector organization using this technology or being part of the open-source community and what's different from being in a private company.

Ricardo: Yeah, I think one aspect is that we can't necessarily easily choose our providers because it's public sector, so maybe the tendering process is a bit different from a private company. I don’t know exactly, but I could imagine that there are differences in this area. But also the type of workloads we do are quite different. One area that is kind of appearing from industry that is similar to what we do is machine learning, because there's this large batch workloads for machine learning model training. This is a lot of what we do, the majority of what we do. 

So, a traditional IT company in the industry would be more interested in providing services. In our case, we do that as well, but I would say 80 percent of our computing is not that. It's large batch submissions that need to execute as fast as possible, so the model is very different. This is especially important when we start talking about multi-cluster because our requirements are much simpler for multi-cluster—or multi-cloud then the requirements of a company that needs to expose services to their end-users.

Emily: Is there anything else that you'd like to add that I haven't thought to ask?

Ricardo: No, I think we covered most of the motivations behind our transition. I would say that I mentioned that one of the challenges is the amount of data that we will have in this upgrade. But I think we did—it could now happen, but it looks like it might happen—we did is that we need to find new computing models to do our analysis than the traditional way we used to do, and this seems to involve, these days, a lot of machine learning. So, I think one thing that we will see that we will benefit a lot, also, from what's happening in the industry is all these frameworks for improving machine learning. And for us, it's also nice to see that a lot of these frameworks are actually relying on this cloud-native Kubernetes type of technologies because it will mean that we are doing something right on the infrastructure, and we might benefit also to upper layers very soon.

Emily: All right, one last question. What is an engineering tool that you can't live without, your favorite tool?

Ricardo: Ah, okay. That’s… well, I would say it's my desktop setup, I would say. I've been a longtime Linux user, and the layout I have, and the tools I use daily are the ones that make me productive, be it to manage my email, calendar, whatever. I think that's one. Yeah, I've learned to rely on Docker for quick prototyping of different environments as well, testing different setups. Yeah, I haven't thought about that. But I would say something in this area.

Emily: And where could listeners connect with you or follow you?

Ricardo: They can contact me by my CERN email which is ricardo.rocha@cern.ch. Or I also have a Twitter account, which is at @ahcorporto. I can pass the name after. But yeah, those are probably the two main points of contact.

Emily: Okay, fabulous. Well, thank you so much for joining me.

Ricardo: Thank you. This was great. Thanks.

Emily: Thanks for listening. I hope you’ve learned just a little bit more about the business of cloud native. If you’d like to connect with me or learn more about my positioning services, look me up on LinkedIn: I’m Emily Omier, that’s O-M-I-E-R, or visit my website which is emilyomier.com. Thank you, and until next time.

Announcer: This has been a HumblePod production. Stay humble.

The Business of Cloud ...