Adriana geeks out with the one and only Kelsey Hightower on Kubernetes, open source, and making tech accessible. Kelsey delves into Kubernetes' rise as an essential yet complex ecosystem, and shares his thoughts on where things might be headed. He switches gears a bit to talk about Kubernetes and Nomad: how they're similar and different, and how they benefit each other. The discussion also touches upon the significance of community engagement in open source projects, as Kelsey emphasizes the responsibilities of contributors and maintainers amidst the constant evolution and monetization of open-source software. Finally, Kelsey talks about the importance of learning and making complex topics accessible, punctuated by his own commitment to demystifying Kubernetes and encouraging progress over perfection in the tech field.
About our guest:
Kelsey Hightower has worn every hat possible throughout his career in tech, and enjoys leadership roles focused on making things happen and shipping software. Kelsey is a strong open source advocate focused on building simple tools that make people smile. When he is not slinging Go code, you can catch him giving technical workshops covering everything from programming to system administration.
Find our guest on:
Find us on:
Show Links:
Additional Links:
Transcript:
ADRIANA: Hey, y'all, welcome to Geeking Out, the podcast about all geeky aspects of software delivery, DevOps, Observability, reliability, and everything in between. I'm your host, Adriana Villela, coming to you from Toronto, Canada. And today I have the pleasure of geeking out with me, Kelsey Hightower. Welcome, Kelsey.
KELSEY: Happy to be here.
ADRIANA: And where are you calling in from today?
KELSEY: I'm in Washington state, so on the border of Portland, Oregon, and Washington.
ADRIANA: Awesome. Well, let us get to it with the warm up questions. Are you ready?
KELSEY: I am.
ADRIANA: Okay, first question. Are you a lefty or a righty?
KELSEY: Right handed.
ADRIANA: All right. iPhone or Android?
KELSEY: iPhone forever. And I've tried android. Given that I've worked at Google for almost eight years, I've tried, but I'm an iPhone person.
ADRIANA: Yeah, I'm an iPhone person too. I never tried android. I went straight from BlackBerry to iPhone.
KELSEY: I think BlackBerry was definitely...I was a BlackBerry person. I was also a Nokia person. But I think once iPhone really dialed in the ability to have third party apps in the App Store, iPhone all day.
ADRIANA: Yeah, I'm the same way. That was like one big sticking point. And for us in Canada, when the iPhone first came out, we didn't even have access to the App Store. So if you wanted any apps, you had to jailbreak your iPhone until it finally became available...because we get everything a little bit late here.
KELSEY: Awesome.
ADRIANA: Okay, next question. What's your favorite programming language?
KELSEY: The one that I can get things done in. So, at one point it was Bash, then it was Python, then it was Ruby when I worked at Puppet Labs, and then it's been Goblin, probably for the last ten years.
ADRIANA: Cool. Awesome. And Mac, Linux, or Windows?
KELSEY: Mac on my desktop. Linux on the server.
ADRIANA: All right, next question. Dev or Ops?
KELSEY: They're one and the same.
ADRIANA: I love it. Okay. JSON or YAML?
KELSEY: JSON. If I had to program against it, YAML if I had to write it.
ADRIANA: By oh, yeah, I definitely agree. I do find, like, manipulating JSON in Python is nicer, but YAML is more readable.
KELSEY: Yeah. To all the people that are like, JSON over YAML, let me watch you write it and see how fast you change your opinion.
ADRIANA: Yes, I totally agree with you there. Okay, this one's a little more controversial, and you can thank one of my previous guests for hinting at it. Spaces or tabs?
KELSEY: I don't care. I actually don't care if Python makes me uses Spaces and my IDE does the right thing. I'm totally fine, actually.
ADRIANA: I'm down for that. Okay, two more questions. Do you prefer to consume video or text when you're consuming content?
KELSEY: It depends. If I'm trying to learn, I need to read it, I need to see it, I need to be able to kind of backscan read it twice. But I do like video in terms of when people are really good at the human side of it. Right? Like, if they're expressing or showing me something, like, I want to see the code run. I want to see where they click. I want to see how they start. But when it's like learning something in the programming world, I need text. People are pretty bad at video and programming lessons.
KELSEY: Like, oh, just write these three lines of code. I'm like, can you please scroll up so I can see what you imported to make this work? So when it comes to seeing code, I want to see no snippets. I want to see as much as possible, but if I'm just going through for the first time to get the flow, video.
ADRIANA: I totally agree with you. And you landed on one of my big pet peeves. When consuming content for learning stuff, which is the code snippets, because I have been and I'm sorry, Hashi people, but this is a crime on the Hashi docs that I see all the time is that I get code snippets, and I don't get to see a full example on the site, and it drives me bananas. And I'm like, what does this apply to? Give me a full fledged code example? Link me to a GitHub repo at some point.
KELSEY: I'm always asking, why are people writing docs out there giving me hints to a murder mystery?
ADRIANA: Yes.
KELSEY: Show me the whole thing. I don't need it to be cute. I don't need it to fit perfectly in your style guide. I just need to see the whole thing and what's going on. So I think people do it out of style. There's really no substance when I'm trying to learn.
ADRIANA: Yeah, I completely agree. I do find it very frustrating. That's why, for me personally, whenever I do technical docs, I give excruciating detail. All right, final question. What is your superpower?
KELSEY: My superpower? I think one thing that I've learned over the years when it comes to mentoring, specifically, I used to be all about sharing my expertise, my background, my learning. And I've noticed that I changed my approach to holding up a mirror in front of other people and convincing them to like what they see and the number of people who actually like what they end up seeing and follow up with me. I really felt like that is a superpower, that you can actually have that impact on people. So that would be my superpower.
ADRIANA: That is such an incredible superpower. And I think it's so relevant to our industry, too, because we have a lot of smart people who suffer from impostor syndrome. And I think showing people that you are actually as good as you think you are is such a huge thing. Right? I mean, we've got some amazing stuff happening. I have some coworkers who are brilliant, and they're like, oh, my God, I feel like I'm just a hack. I'm like, Are you kidding me? I can't even keep up with some of the stuff that you're telling me right now.
KELSEY: Yeah. And I try to get people to understand that sometimes you aren't as good as you want to be. And that's okay too, right? I think there's okay with making progress, entering to new domains, and just helping people just relieve the pressure. Ideally, if you're any good at this thing, you're going to always feel this way forever because you're humble enough to keep learning, so you shouldn't feel so bad about it.
ADRIANA: Yeah, that's true. That's a very excellent point. So let's get into the meaty bits. One of the things that I wanted to share with our audience was how you came to be on the podcast. We met at KubeCon North America in Chicago this year, and you were doing a book signing. And I came, stood in line, the long line. It was totally worth it. And I was wearing this mask that had the sticker for the podcast, Geeking Out, and you said, "Oh, what is that?" And I said, "Oh, that's my podcast."
ADRIANA: And you said, "Oh, I could be on your podcast." So I am so stoked that you were able to join. And yeah, I mean, I've admired your work from afar for many years. I find your approach to Kubernetes very accessible, especially because it's such a complex subject matter. So I wanted to start off with how did you get into this field in the first place? Where did you find your calling to make things technical things, gnarly technical things so accessible to folks?
KELSEY: I want to answer that question, but I want to address this advice that I give to my former self and to people that I run into all the time. And they say, how do you put in the effort to make sure good things happen to you in your career and in your life in general? And so you at that book signing with the podcast on your mask. You're advertising to the world, this is my podcast, this is what I'm doing, and you're advertising what needs to happen next. And so for someone like me, I can see that clearly. I understand in that limited interaction that there is this opportunity that I could actually be on your podcast, because now I know you have one. I think a lot of people really confuse luck and that kind of effort, right? When you put that kind of effort forward, you tend to make things happen. And so I just want to highlight that part of you having that as part of your strategy of going to KubeCon, making the best use of your time and every human interaction. So kudos to you for doing that.
KELSEY: But it's a perfect example of how people kind of design their own careers and create the world that they want. So that's perfect. Now to your question about this whole idea of explaining things simply to other people. When I was getting into tech, a lot of people come from various backgrounds I come from the...fast food was my only job background, and I didn't go to college. And so for me, learning technology was like a pivotal life-making decision. I need to get into this field. I admire people that are in this field.
KELSEY: I don't know anyone that's in this field. And so I would go get all the books and just flip through them. I remember the first book I think I bought was the A+ certification guide. I was like, I'll start there. And you just go through all of this stuff and you look at all your notes, right? You're trying to simplify all the information to truly demonstrate that you understand it. And everyone knows that feeling of the A-ha! moment where you take something that is complex to you and you finally understand it, and your confidence level just goes up. It immediately goes up. And so that feeling, I've always enjoyed having that feeling because it felt so empowering.
KELSEY: So whenever I had the opportunity to speak at a meetup, I've noticed that some people at meetups or conferences, they speak, and it's just like, overwhelming. Hey, here's this computer science diagram. Here's this map that you cannot understand what's happening, and they are happy with just leaving it as a mystery to everyone. And you're like, what the hell was that? You had this opportunity to let me have my light bulb moment, but you chose not to. You chose to try to overwhelm me with your vast understanding of things that I don't. And so I've tried to say, what if I can make people feel like I felt whenever I learned a new subject? So this is why I've always said, hey, now that I understand this thing, I want to show it to you as well. But before I can, I have to give you context where I came from, my understanding beforehand, and then what led me to that understanding. And then let me show it to you.
KELSEY: And I try to use analogies and simple terms, and you can see the light bulb moment go off for people in the audience, and then it becomes a game changer for their own career. So for me, I think I got addicted to that. Like, hey, I don't want to talk. I don't want to write a tutorial if it doesn't have that impact on people.
ADRIANA: I absolutely love that because I can completely relate to that feeling, the euphoria, the high that you get from solving a problem, especially something where you've had to really put on the detective skills hat and try extra, extra hard to solve it. So that is so wonderful. I love that so much, and I think it's so important because making learning accessible to people, I think, makes it fun too, because I agree with you. Like those gnarly architecture diagrams that just look overly complicated, and then your brain starts to wander, and then you miss some important thing, and then that's it your opportunity for learning. That thing is gone if you're watching that lecture, because it's just, like, way over your head. So I think that's so great. Such an awesome approach to really disseminating information across the industry, especially these are not easy topics to unravel, right? So, Kubernetes, for example, how did you come upon doing your work with Kubernetes?
KELSEY: You know how you walk in on someone watching some hit TV show, they're on season six, right? And you ask them, what's going on? Why is this person not like this other person? They're like, I got to recap season three for you to understand what's going on on the screen right now. And so I think for a lot of people, Kubernetes was my season six, right? I had always been in tech trying to share information. If you would have caught me 15 years ago, you would have saw me at a Python conference teaching people about packaging Python applications. If you saw me maybe six years after that, I was at Puppet Labs trying to contribute to configuration management tools using Ruby. And so when I get to Kubernetes, there's a whole career behind me of trying to build similar systems without the terminology or the experience. You just know that there has to be a better way of doing things. So when I saw Kubernetes for the first time and really got hands on time with it, there was an a-ha! moment. I was like, you know what? All the scripts, tools, philosophies, techniques, it has now been serialized into this one checkpoint, and the industry has finally given it a name.
KELSEY: And so when I got that feeling, you know what was next, right? It was like, hey, I can't wait to go to a meetup to show people this thing. And I think the reason why I was able to resonate with so many people is because I had that previous background of doing things manually, trying different automation tools. And so I was just so excited. Like, I think we finally found the thing we've been all trying to build, and it looks like this. And so I think a lot of people got to see that season. It was like, oh, he's the Kubernetes guy. But there's so much historical context that goes into why I was ready to have that conversation, make those contributions at that time.
ADRIANA: That's basically the classic case of, like, everything you've done up until that point has led you to that moment, and now you're ready to take on that thing. Right?
KELSEY: I became a better speaker than I had ever been prior. I became a better engineer than I had ever been prior. And I've gone through all of that experience, and I was able to really articulate what was important. And I think for a lot of people who have been on this DevOps journey for a decade, nothing is working. We're doing all of the things: CI/CD pipeline, infrastructure-as-code. We're missing something here. And I think the industry had overly focused on automation and not abstraction. And Kubernetes was that final thing that you could touch to say, there is a difference between automation and abstraction.
KELSEY: And I think when people saw those new APIs, in many ways, I told people Kubernetes was like this type system to infrastructure. It was like a standard library that we'd never had. It's not like a thing that if you just install, it solves all your problems. But it's definitely a much better checkpoint than what people were doing before.
ADRIANA: Yeah, absolutely. And it's one of those things where I feel like it's a bit of a love hate relationship with Kubernetes. Right. Because in some ways, it makes life so much easier, and then in other ways, it's like, oh, my God, this thing is so complex to try to unravel in your mind. Right.
KELSEY: I want to address that a lot, because there are some people that think I am the biggest Kubernetes fan in the world, and I am not. I actually spent the last four years working on replacements. I spent so much time at Google Cloud working on serverless just to make Kubernetes go away. I learned everything about it because I think the best people that will replace it are the people who understand it the most. And the way I look at Kubernetes is very different. People look at it as a tool that is competing with their other favorite tool or some alternative ways of doing things. To me, Kubernetes is just another word in the dictionary, and my focus has always been, what does it mean? And as a contributor, what should it mean? And when I think about it as an aggregation of the previous ten year set of techniques, and you push them all together, you get this thing. And I study that thing for, like, wow, we've come a long way since those days.
KELSEY: Also, you can see what's missing. And I think that part is where, for me, that's inspiring. Oh, this is what's missing. So this is where the opportunity space is. Go work there and solve that problem. But I think a lot of people get into, oh, this thing is too complex. And I always ask them, but do you understand it? If you don't understand it and you say it's complex, then I think that's a mislabeling of the situation. You can just say, I don't understand it, therefore, I don't know why I would use it.
KELSEY: And I think that's a fair way to start the conversation. I think a lot of people are just dismissing it because it's complex, and I can do something much simpler, and then they tell me what they're doing. I'm like, that sounds like a Rube Goldberg machine. You just named 25 pieces of custom tooling so you can avoid using Kubernetes. I don't know if that adds up.
ADRIANA: Yeah, that makes a lot of sense. I think it makes sense too, like what you said earlier about looking for something that could potentially replace Kubernetes, because I also think that we tend to get into this sort of rut where we think, well, it all ends with Kubernetes. But we all know that software has evolved so much in the last 20 years. Even everyone was talking about Heroku is this awesome thing, and now, yeah, Heroku is still in the picture, but other things have come and kind of taken our attention. So where are we moving towards then in this space?
KELSEY: I think in some ways things haven't changed very much in 20 years. You write code, you build the code, and you try to do some process to get it on the server so people can use the code. About 20 years, people have been doing exactly that thing. Now, how people have gone about doing that thing, that's changed at different speeds. Some people are still writing KornShell scripts right now as we speak, deploying apps at their company, and it probably works well. Then you have some people that are still using tools like Capistrano because they want to use something that's written in their favorite programming language, in that case, Ruby. And so they just want to keep everything within their problem domain. And then you have some people who prefer platforms like Heroku, Cloud Foundry, you name it.
KELSEY: I think the challenge has been is lots of people have been looking for that one solution for everything. I remember when Cloud Foundry, like the Heroku competitor that you could run yourself, it was like, look, twelve factor apps are the way to go, and you can write everything as long as you use Java and Spring Boot. You do that, you're done, you're great. And then it's like, okay, that's fine. What about my batch jobs? Where do I run those? Not there. What about my databases? Where do I run those? Not there? And then what happens is you end up having to bring in a second or third platform. And that's where the harsh reality of all of this stuff is, is that whenever we don't have one solution to solve everything, you end up having to complicate your infrastructure. And I think complicated infrastructure just the actual norm at this point.
KELSEY: What the world wants in terms of if you have a public facing website, you're probably going to have a cache, you're going to have Cloudflare DDoS protection. Various security concerns that Kubernetes versus Heroku is such a small part of the decision making process that even if you got that layer right, it is such a small part of the equation that thinking that's where the complexity is, ignores the big picture, where I think things like Kubernetes are 1/100th of the equation.
ADRIANA: Right. That makes a lot of sense. Now, on a similar vein of Kubernetes like product, you've also done some work with HashiCorp Nomad, right? How would you compare Kubernetes to Nomad for folks who aren't familiar with both?
KELSEY: Respect to everyone that contributes. Because I've written lots of code myself, and you do the best you can. So we just got to make sure we get that out of the way. We're not attacking people here. So if you have a HashiCorp logo tattooed on your body or Kubernetes logo tattooed on your body, this is not about you at all. When I first saw Nomad, I remember, is when they announced it in Portland at one of the smaller first HashiConfs. And I was scheduled to give a talk about Kubernetes, and I changed the talk the night before to do Nomad versus Kubernetes. And I remember Mitchell, Armon and so many people from HashiCorp standing there watching the talk. Everyone's crowded in to watch the talk.
KELSEY: And look, I'm not a mean person, so I'm not someone that's naively attack a project that I'm not working on. Doesn't make any sense. But I did learn it, got it installed. And the things I liked about Nomad, you got this single binary written in Golang. You just put it on the server and it's almost immediately ready to go, starting getting value, right? That part around, just go get a binary and just have it run on the server. It really, really made that easy. The part that wasn't great, though, is the API. You look at it and it's like, what is this thing? Right? I think I get it.
KELSEY: And it felt like, oh, you're trying to copy the Borg white paper about what a task is, but you haven't used Borg enough to know that this is not what you want to copy. And so it was a good serialization of that knowledge that was out there. They built a very high performance fast scheduler. They optimized for scheduling, speed, and performance. But the thing I think that they missed was the ecosystem. This space now is about collaboration. So you have lots of people who want to build infrastructure, automation, tools. And the one problem we've had over time, in my opinion, is that you have to glue them all back together.
KELSEY: And scripting only gets you so far when you have to glue together all these various APIs. So Kubernetes takes a different approach. Kubernetes says these things are all related. Your load balancer and your app and your IPs, and your storage, your secrets, all of it is related. And they depend on each other. And so Kubernetes felt like it lived a life where the maintainers or the people of that project had been using Borg for a decade or two and said, what would we fix? And they come into a popular ecosystem like Docker and all these pieces, and they aggregate them. And when you look at the API, you can see the experience peek through. Right here is a pod.
KELSEY: A pod has to have multiple containers because most apps that people deploy in reality, need things like NGINX or sidecars or logging daemons. And so I felt like Kubernetes had so much more experience baked into it than just being a faster, easier to manage system for deploying things. So given that, it was really nice to see over time that both communities kind of learn from each other. I remember when Nomad started adding things like volumes, sidecars, or other things that you would typically see in Kubernetes. So I think some people like Nomad because of its simplicity. I kind of lean towards the simplicity side of the house, so I kind of resonate with the whole Nomad thing. But watching people kind of glue together, like vault console, and all these other pieces to try to get a whole system, I'm like, man, at this point, now Kubernetes starts to look a little better.
ADRIANA: Yeah, I definitely agree. I worked at a job where so I had come from a Kubernetes background and worked at a job where it was a Hashi shop, and they're like, oh, we're using Nomad. So I'm like, oh, my God. How do I translate this? And when I learned that Nomad is not fully equal to Kubernetes, that you have to still stitch these other pieces together, I'm like, oh, okay, that complicates things. But I definitely agree with you. One of the things that I do appreciate about Nomad is that certain things seem a little bit simpler. And I did find the learning curve not too bad. Maybe it was because I also knew Kubernetes at the time, so maybe that helped and it allowed me to translate.
ADRIANA: But there's definitely a lot of stuff that I appreciate about Nomad, and I'm glad that I've had exposure to both ways of doing things, because I think that's really cool. And like what you were saying, both communities learning from each other rather than, like, let's hoard our secrets, because that way you can end up with better products overall, right?
KELSEY: 100%.
ADRIANA: Now, one thing that I wanted to ask you about was your famous Hashinetes tutorial. What motivated you to put this together? And also, if you can just share with folks what this Hashinetes thing is.
KELSEY: I remember the Hashinetes talk, because that was the year I was like, okay, all of these tools have been out for a while. Vault is out. Consul is out. Nomad is out. Kubernetes is out. Now what? How do you think about all these things? What do you even do with them? And I remember that year I wanted to have fun, right? Previous years, it's more about, what are these things? And then maybe years after that, it's like, it's in production. But I was like, you know what? I want to have a irresponsible talk. I remember starting to talk off: "Today we're going to be irresponsible."
KELSEY: "Do not do this in production." "Do not go to work and say Kelsey said anything." This is just having fun. Okay, and so I remember having a Kubernetes cluster or maybe even Nomad, and said, all right, we're going to install Nomad as an app to see how it works. And I just started adding different layers and components one by one. Number one, teaching people how all of these things actually fit together and how another scheduler could actually arrange them and put them into place. And then I think people had so much fun with the talk. It's like, wow, look how powerful these tools are that they can actually deploy and manage each other if you really wanted to.
KELSEY: And look how they're similar in some ways. And I think a lot of people were like, oh, these are just you need to pick one or the other. And at that time, there was a blog post of a company using Kubernetes for some stuff and then using Nomad for some of their batch jobs that would benefit from the Nomad way of doing things. I thought that was just, like, the right way to think about it. So that talk Hashinetes is like, what happens if you push Kubernetes and all the HashiCorp tools together, like using Vault for secrets instead of the thing that was built into Kubernetes, because I think Vault was a far superior secrets management product and API. And then what if you were to use Consul instead of Kubernetes built-in service-discovery? What would you get? And then let's just say you really do like Nomad. What if you were to run that inside of Nubernetes, too, and let that become the scheduler instead of Kubernetes doing the scheduling? And I think when people kind of saw that talk, they understood how to really fairly evaluate those tools. So we just had a bunch of fun.
ADRIANA: What do you think was the biggest learning from putting this talk together for yourself?
KELSEY: I think, honestly, if you just live 100% in Kubernetes land, all you know is config, maps, secrets, and you have an idea in your mind that there's no other way of thinking about these problems. Right? Everything must be a CRD. Kubernetes, Kubernetes, Kubernetes. But I think people forget I was a contributor to Kubernetes. I knew how some of the inner workings worked. And so it's like, how do you get Vault to work nicely inside of Kubernetes? Then you have to rethink the APIs, and you start, oh, the Kubernetes secret management API isn't that great at all? And so when you bring in Vault and you have to stitch it in and bake it into the whole process, you really do gain empathy for gluing all of these parts together yourself. So I think the biggest learning for me is that, number one, you can do it. There are situations where it does make sense.
KELSEY: Think about it. If you have multiple clusters and you want to have multi cluster service discovery, you cannot do that with Kubernetes alone. When you add something like Consul, you can have Consul be the place that takes over DNS. And guess what? Voilà, you can now address multiple clusters using one service discovery tool. And so it's like, oh, okay. So even though Kubernetes hasn't solved all the problems, it doesn't mean that you can't bring in all these alternative tools to step in and fill that gap.
ADRIANA: And it's nice to see that everything plays nice in that little ecosystem and that you can, I guess, take advantage of each tool superpowers, right, to sort of give that boost to Kubernetes Awesome. Now, on the Hashi front, I also wanted to talk to you briefly about a talk that you gave at KubeCon EU, "From Community to Customers". And I attended that talk, and I really enjoyed the talk. I thought it was very interesting how you were talking about this fine line of what to keep open source versus what not to keep open source. One example that you cited was HashiCorp, and then shortly thereafter, HashiCorp changed their licensing. So what are your thoughts around that?
KELSEY: Yeah, I actually had this question come up a few times, and I always tell people from a place of empathy, I had a project, Confd. It became a little popular. I remember going to FOSSDEM on the other side of the world in Europe, and watching someone give a talk about using Confd, this miniature configuration management tool, and how they were using it and why they thought it was one of the greatest projects ever. Like, as a maintainer of an open source project, you'd love to see a community form around the thing you've built. But as a solo maintainer, you also know how hard it is to say no. And you wake up on, like, a Saturday morning and it's like, hey, I work at a huge company that makes tons of profit, and I get paid really well to do my job. I would like you to work for free and add this feature that we really, really need to make even more money. And you're like, no, this is not my priority.
KELSEY: Number one, you're not paying me anything. And then two, you know what? You're going to have to prioritize that itself and maybe step up and do some contributions. And so when you think about it that way, and as someone who's also contributed code to HashiCorp products in the past, I did those contributions to scratch my own itch. And I understand that once I deliver those changes, it's on the HashiCorp team to maintain them forever. And so I understand the relationship here is me contributing code is not the end of the story. And so when they make that licensing change, I put myself in their shoes of trying to run a business and remember, they're a public traded company. So a lot of these decisions are not in fully their control anymore. The market wants to see profit growth.
KELSEY: I don't know if you've ever worked at a profitable company, people listening to this. But having stagnant revenue year over year is a fast way to get shareholders to leave investing in your stock. So now they have this added pressure of no longer just making the open source community happy. The people that they kind of started their careers off of, now they have to try to make the market happy. And there you get into different behaviors. So now you got to figure out where to get revenue from. And if you ask someone, Where do you get revenue from something that is given away 100% for free? Last I checked, most people do not pay for things unless they have to pay for things. And so you got to draw the line somewhere.
KELSEY: And I think the big controversy is, where do you draw the line? Do you draw the line on the core of the product? NGINX tried to do things like that. It didn't work out well over time, do we draw the line on, like, enterprise features and Web UIs? Right? That could be a fair place to draw the line. And so I think for a lot of people, HashiCorp decided to draw the line at commercial competition. If you take our software and start competing against us, using our name, likeness, whatever we say now in our new license, the business source license, that you can't do that. And so if you're being honest, as a user, don't really care. Like, I don't plan to start a business competing against terror. If you're being honest, I literally don't care.
KELSEY: And most people don't really exercise all their open source freedoms anyway. I'm not saying that's not a good reason not to have them, but a lot of these licenses like Apache 2 to me to fully realize the benefits of them. I think you do need to become a contributor to really understand what the code base does, be willing to step up to fork a project when the time comes and having the skills to maintain it. A lot of people don't understand that's the other part of this deal. And so when they change that license, I think people got a wake up call. They own that project. It is not our project. Even those with that HashiCorp logo somewhere tattooed on their body, it's not your project.
KELSEY: It belongs to HashiCorp. And so now I think there's a rethink. And a lot of people forget HashiCorp predates the CNCF, right? So they're not a part of a foundation, even though a lot of their technologies are foundational, TerraForm, Vault, those things belong to HashiCorp, a private company doing what they have to do. And so for me, I look at that business license change and says, great, they made their stake in the sand. From a business perspective, this will be good for HashiCorp. Now they can say no. And now their terms are a bit clear and no longer vague. Now, for the community that is upset,
KELSEY: now it's time to exercise those open source rights we've all been talking about for so long. You get to fork the project, you get to maintain the project, bug, fixes security, fixes new features and then ask the question how compatible should you remain going forward with the thing in which you branch from? That's what's on the table. So those are my thoughts on it. It's very pragmatic. I think it's one of ownership and responsibility and no matter how you feel about it, you're going to have to take on ownership and responsibility going forward.
ADRIANA: Yeah, absolutely. It makes so much sense and I think you hit on a very important thing when it comes to maintaining an open source project, which is maintaining it. It is a lot of freaking work and especially if it's something that you do on the side for funsies. You can only expect so much, especially if you're the solo maintainer. So also hats off to anyone who is a solo maintainer of an open source project or works with a very small team because it's a lot of work. It's a labor of love at that point, right?
KELSEY: I want to make sure people understand. A lot of people may have an ops background. That's definitely where I come from. And people think dev is easy and there's the same stress that you have in operations, right. For example, if you replace a hard drive in a server with a bad hard drive, you worry the first couple of days like, is that RAID configuration going to actually rebuild on time and the hard drive is going to stop being slow before traffic comes. You worry about these things and this is why we started doing things like on-call. And when you are maintained of open source project, you know that anything you merge in will make its way to someone's production, someone you probably don't know and you're going to feel responsible and accountable for doing that. And so there's a lot of this added pressure of like, hey, I got to be able to say no and make the right decisions to make sure that no one is going to be negatively impacted by these projects.
KELSEY: I think a lot of people forget that when we start to ask and I don't like the way this person runs this open source project, there is so much pressure that goes into it. So just know that there's humans behind these projects. There's a lot at stake. So if they say no to your new feature or they have to make a business license change or stop accepting pull requests for a while while they go tend to other matters, you just have to understand that just what comes with the territory.
ADRIANA: Yeah, absolutely. There are humans behind those repos right at the end of the day. Well, we are coming up on time, but before we wrap up, I was wondering if there are any parting words of wisdom that you would like to share with our audience?
KELSEY: I don't know if there's any parting words of wisdom, but I do think we're at this next cycle of new technology on its way, whether it's AI or LLMs, some people only know that stuff as chat GPT. And the question that I'm hearing a lot around is, like, is this thing going to take my job? And I always ask those folks, what is your job? And they say, "Well, for the last ten years, I've just been running scripts and automating things, and I'm like the same things for ten years in a row." I was like, "Listen, if that's how you would describe your job, then yes, you might have a problem when a new set of tooling comes around that reduces the need to do that." And that's always happened throughout tech. And I think what most people should probably think about is take these moments of insecurity and just do some self reflection and say, "Hey, my tools"...and I think we started the conversation this way. People tend to confuse automation to abstraction, and a lot of times, people get so comfortable automating the same things over and over, almost like a Westworld Loop, that they forget that we should rethink the thing that we're automating and ask ourselves if we should replace it with better abstractions. So I would say this this may be your very moment to pause for a second look at the work you do, and ask yourself, "Is it time for a new abstraction?" And if it is, I think that's the perfect opportunity to either go find a project that's attacking that problem or maybe even start your own that introduces the new abstraction based on all of that experience that you have.
ADRIANA: Awesome. I really love that. Well, thank you so much, Kelsey, for geeking out with me today. Y'all don't forget to subscribe, and be sure to check the show notes for additional resources and to connect with us and our guests on social media. Until next time...
KELSEY: All right, everyone, don't forget to Peace Out and Geek Out.
ADRIANA: Geeking Out is hosted and produced by me, Adriana Villela. I also compose and perform the theme music on my trusty clarinet. Geeking out is also produced by my daughter, Hannah Maxwell, who, incidentally, designed all of the cool graphics. Be sure to follow us on all the socials by going to bento.me/geekingout. Hey, hey Geeking Out fans! We're taking a little break for the holidays, so this will be the last episode of 2023. Be sure to catch us again in January as we Geek Out with a fabulous lineup of guests.
ADRIANA: See you in 2024. And Peace Out, and Geek Out. Bye!
ADRIANA: Hey, y'all, welcome to Geeking Out, the podcast about all geeky aspects of software delivery, DevOps, Observability, reliability, and everything in between. I'm your host, Adriana Villela, coming to you from Toronto, Canada. And today I have the pleasure of geeking out with me, Kelsey Hightower. Welcome, Kelsey.
KELSEY: Happy to be here.
ADRIANA: And where are you calling in from today?
KELSEY: I'm in Washington state, so on the border of Portland, Oregon, and Washington.
ADRIANA: Awesome. Well, let us get to it with the warm up questions. Are you ready?
KELSEY: I am.
ADRIANA: Okay, first question. Are you a lefty or a righty?
KELSEY: Right handed.
ADRIANA: All right. iPhone or Android?
KELSEY: iPhone forever. And I've tried android. Given that I've worked at Google for almost eight years, I've tried, but I'm an iPhone person.
ADRIANA: Yeah, I'm an iPhone person too. I never tried android. I went straight from BlackBerry to iPhone.
KELSEY: I think BlackBerry was definitely...I was a BlackBerry person. I was also a Nokia person. But I think once iPhone really dialed in the ability to have third party apps in the App Store, iPhone all day.
ADRIANA: Yeah, I'm the same way. That was like one big sticking point. And for us in Canada, when the iPhone first came out, we didn't even have access to the App Store. So if you wanted any apps, you had to jailbreak your iPhone until it finally became available...because we get everything a little bit late here.
KELSEY: Awesome.
ADRIANA: Okay, next question. What's your favorite programming language?
KELSEY: The one that I can get things done in. So, at one point it was Bash, then it was Python, then it was Ruby when I worked at Puppet Labs, and then it's been Goblin, probably for the last ten years.
ADRIANA: Cool. Awesome. And Mac, Linux, or Windows?
KELSEY: Mac on my desktop. Linux on the server.
ADRIANA: All right, next question. Dev or Ops?
KELSEY: They're one and the same.
ADRIANA: I love it. Okay. JSON or YAML?
KELSEY: JSON. If I had to program against it, YAML if I had to write it.
ADRIANA: By oh, yeah, I definitely agree. I do find, like, manipulating JSON in Python is nicer, but YAML is more readable.
KELSEY: Yeah. To all the people that are like, JSON over YAML, let me watch you write it and see how fast you change your opinion.
ADRIANA: Yes, I totally agree with you there. Okay, this one's a little more controversial, and you can thank one of my previous guests for hinting at it. Spaces or tabs?
KELSEY: I don't care. I actually don't care if Python makes me uses Spaces and my IDE does the right thing. I'm totally fine, actually.
ADRIANA: I'm down for that. Okay, two more questions. Do you prefer to consume video or text when you're consuming content?
KELSEY: It depends. If I'm trying to learn, I need to read it, I need to see it, I need to be able to kind of backscan read it twice. But I do like video in terms of when people are really good at the human side of it. Right? Like, if they're expressing or showing me something, like, I want to see the code run. I want to see where they click. I want to see how they start. But when it's like learning something in the programming world, I need text. People are pretty bad at video and programming lessons.
KELSEY: Like, oh, just write these three lines of code. I'm like, can you please scroll up so I can see what you imported to make this work? So when it comes to seeing code, I want to see no snippets. I want to see as much as possible, but if I'm just going through for the first time to get the flow, video.
ADRIANA: I totally agree with you. And you landed on one of my big pet peeves. When consuming content for learning stuff, which is the code snippets, because I have been and I'm sorry, Hashi people, but this is a crime on the Hashi docs that I see all the time is that I get code snippets, and I don't get to see a full example on the site, and it drives me bananas. And I'm like, what does this apply to? Give me a full fledged code example? Link me to a GitHub repo at some point.
KELSEY: I'm always asking, why are people writing docs out there giving me hints to a murder mystery?
ADRIANA: Yes.
KELSEY: Show me the whole thing. I don't need it to be cute. I don't need it to fit perfectly in your style guide. I just need to see the whole thing and what's going on. So I think people do it out of style. There's really no substance when I'm trying to learn.
ADRIANA: Yeah, I completely agree. I do find it very frustrating. That's why, for me personally, whenever I do technical docs, I give excruciating detail. All right, final question. What is your superpower?
KELSEY: My superpower? I think one thing that I've learned over the years when it comes to mentoring, specifically, I used to be all about sharing my expertise, my background, my learning. And I've noticed that I changed my approach to holding up a mirror in front of other people and convincing them to like what they see and the number of people who actually like what they end up seeing and follow up with me. I really felt like that is a superpower, that you can actually have that impact on people. So that would be my superpower.
ADRIANA: That is such an incredible superpower. And I think it's so relevant to our industry, too, because we have a lot of smart people who suffer from impostor syndrome. And I think showing people that you are actually as good as you think you are is such a huge thing. Right? I mean, we've got some amazing stuff happening. I have some coworkers who are brilliant, and they're like, oh, my God, I feel like I'm just a hack. I'm like, Are you kidding me? I can't even keep up with some of the stuff that you're telling me right now.
KELSEY: Yeah. And I try to get people to understand that sometimes you aren't as good as you want to be. And that's okay too, right? I think there's okay with making progress, entering to new domains, and just helping people just relieve the pressure. Ideally, if you're any good at this thing, you're going to always feel this way forever because you're humble enough to keep learning, so you shouldn't feel so bad about it.
ADRIANA: Yeah, that's true. That's a very excellent point. So let's get into the meaty bits. One of the things that I wanted to share with our audience was how you came to be on the podcast. We met at KubeCon North America in Chicago this year, and you were doing a book signing. And I came, stood in line, the long line. It was totally worth it. And I was wearing this mask that had the sticker for the podcast, Geeking Out, and you said, "Oh, what is that?" And I said, "Oh, that's my podcast."
ADRIANA: And you said, "Oh, I could be on your podcast." So I am so stoked that you were able to join. And yeah, I mean, I've admired your work from afar for many years. I find your approach to Kubernetes very accessible, especially because it's such a complex subject matter. So I wanted to start off with how did you get into this field in the first place? Where did you find your calling to make things technical things, gnarly technical things so accessible to folks?
KELSEY: I want to answer that question, but I want to address this advice that I give to my former self and to people that I run into all the time. And they say, how do you put in the effort to make sure good things happen to you in your career and in your life in general? And so you at that book signing with the podcast on your mask. You're advertising to the world, this is my podcast, this is what I'm doing, and you're advertising what needs to happen next. And so for someone like me, I can see that clearly. I understand in that limited interaction that there is this opportunity that I could actually be on your podcast, because now I know you have one. I think a lot of people really confuse luck and that kind of effort, right? When you put that kind of effort forward, you tend to make things happen. And so I just want to highlight that part of you having that as part of your strategy of going to KubeCon, making the best use of your time and every human interaction. So kudos to you for doing that.
KELSEY: But it's a perfect example of how people kind of design their own careers and create the world that they want. So that's perfect. Now to your question about this whole idea of explaining things simply to other people. When I was getting into tech, a lot of people come from various backgrounds I come from the...fast food was my only job background, and I didn't go to college. And so for me, learning technology was like a pivotal life-making decision. I need to get into this field. I admire people that are in this field.
KELSEY: I don't know anyone that's in this field. And so I would go get all the books and just flip through them. I remember the first book I think I bought was the A+ certification guide. I was like, I'll start there. And you just go through all of this stuff and you look at all your notes, right? You're trying to simplify all the information to truly demonstrate that you understand it. And everyone knows that feeling of the A-ha! moment where you take something that is complex to you and you finally understand it, and your confidence level just goes up. It immediately goes up. And so that feeling, I've always enjoyed having that feeling because it felt so empowering.
KELSEY: So whenever I had the opportunity to speak at a meetup, I've noticed that some people at meetups or conferences, they speak, and it's just like, overwhelming. Hey, here's this computer science diagram. Here's this map that you cannot understand what's happening, and they are happy with just leaving it as a mystery to everyone. And you're like, what the hell was that? You had this opportunity to let me have my light bulb moment, but you chose not to. You chose to try to overwhelm me with your vast understanding of things that I don't. And so I've tried to say, what if I can make people feel like I felt whenever I learned a new subject? So this is why I've always said, hey, now that I understand this thing, I want to show it to you as well. But before I can, I have to give you context where I came from, my understanding beforehand, and then what led me to that understanding. And then let me show it to you.
KELSEY: And I try to use analogies and simple terms, and you can see the light bulb moment go off for people in the audience, and then it becomes a game changer for their own career. So for me, I think I got addicted to that. Like, hey, I don't want to talk. I don't want to write a tutorial if it doesn't have that impact on people.
ADRIANA: I absolutely love that because I can completely relate to that feeling, the euphoria, the high that you get from solving a problem, especially something where you've had to really put on the detective skills hat and try extra, extra hard to solve it. So that is so wonderful. I love that so much, and I think it's so important because making learning accessible to people, I think, makes it fun too, because I agree with you. Like those gnarly architecture diagrams that just look overly complicated, and then your brain starts to wander, and then you miss some important thing, and then that's it your opportunity for learning. That thing is gone if you're watching that lecture, because it's just, like, way over your head. So I think that's so great. Such an awesome approach to really disseminating information across the industry, especially these are not easy topics to unravel, right? So, Kubernetes, for example, how did you come upon doing your work with Kubernetes?
KELSEY: You know how you walk in on someone watching some hit TV show, they're on season six, right? And you ask them, what's going on? Why is this person not like this other person? They're like, I got to recap season three for you to understand what's going on on the screen right now. And so I think for a lot of people, Kubernetes was my season six, right? I had always been in tech trying to share information. If you would have caught me 15 years ago, you would have saw me at a Python conference teaching people about packaging Python applications. If you saw me maybe six years after that, I was at Puppet Labs trying to contribute to configuration management tools using Ruby. And so when I get to Kubernetes, there's a whole career behind me of trying to build similar systems without the terminology or the experience. You just know that there has to be a better way of doing things. So when I saw Kubernetes for the first time and really got hands on time with it, there was an a-ha! moment. I was like, you know what? All the scripts, tools, philosophies, techniques, it has now been serialized into this one checkpoint, and the industry has finally given it a name.
KELSEY: And so when I got that feeling, you know what was next, right? It was like, hey, I can't wait to go to a meetup to show people this thing. And I think the reason why I was able to resonate with so many people is because I had that previous background of doing things manually, trying different automation tools. And so I was just so excited. Like, I think we finally found the thing we've been all trying to build, and it looks like this. And so I think a lot of people got to see that season. It was like, oh, he's the Kubernetes guy. But there's so much historical context that goes into why I was ready to have that conversation, make those contributions at that time.
ADRIANA: That's basically the classic case of, like, everything you've done up until that point has led you to that moment, and now you're ready to take on that thing. Right?
KELSEY: I became a better speaker than I had ever been prior. I became a better engineer than I had ever been prior. And I've gone through all of that experience, and I was able to really articulate what was important. And I think for a lot of people who have been on this DevOps journey for a decade, nothing is working. We're doing all of the things: CI/CD pipeline, infrastructure-as-code. We're missing something here. And I think the industry had overly focused on automation and not abstraction. And Kubernetes was that final thing that you could touch to say, there is a difference between automation and abstraction.
KELSEY: And I think when people saw those new APIs, in many ways, I told people Kubernetes was like this type system to infrastructure. It was like a standard library that we'd never had. It's not like a thing that if you just install, it solves all your problems. But it's definitely a much better checkpoint than what people were doing before.
ADRIANA: Yeah, absolutely. And it's one of those things where I feel like it's a bit of a love hate relationship with Kubernetes. Right. Because in some ways, it makes life so much easier, and then in other ways, it's like, oh, my God, this thing is so complex to try to unravel in your mind. Right.
KELSEY: I want to address that a lot, because there are some people that think I am the biggest Kubernetes fan in the world, and I am not. I actually spent the last four years working on replacements. I spent so much time at Google Cloud working on serverless just to make Kubernetes go away. I learned everything about it because I think the best people that will replace it are the people who understand it the most. And the way I look at Kubernetes is very different. People look at it as a tool that is competing with their other favorite tool or some alternative ways of doing things. To me, Kubernetes is just another word in the dictionary, and my focus has always been, what does it mean? And as a contributor, what should it mean? And when I think about it as an aggregation of the previous ten year set of techniques, and you push them all together, you get this thing. And I study that thing for, like, wow, we've come a long way since those days.
KELSEY: Also, you can see what's missing. And I think that part is where, for me, that's inspiring. Oh, this is what's missing. So this is where the opportunity space is. Go work there and solve that problem. But I think a lot of people get into, oh, this thing is too complex. And I always ask them, but do you understand it? If you don't understand it and you say it's complex, then I think that's a mislabeling of the situation. You can just say, I don't understand it, therefore, I don't know why I would use it.
KELSEY: And I think that's a fair way to start the conversation. I think a lot of people are just dismissing it because it's complex, and I can do something much simpler, and then they tell me what they're doing. I'm like, that sounds like a Rube Goldberg machine. You just named 25 pieces of custom tooling so you can avoid using Kubernetes. I don't know if that adds up.
ADRIANA: Yeah, that makes a lot of sense. I think it makes sense too, like what you said earlier about looking for something that could potentially replace Kubernetes, because I also think that we tend to get into this sort of rut where we think, well, it all ends with Kubernetes. But we all know that software has evolved so much in the last 20 years. Even everyone was talking about Heroku is this awesome thing, and now, yeah, Heroku is still in the picture, but other things have come and kind of taken our attention. So where are we moving towards then in this space?
KELSEY: I think in some ways things haven't changed very much in 20 years. You write code, you build the code, and you try to do some process to get it on the server so people can use the code. About 20 years, people have been doing exactly that thing. Now, how people have gone about doing that thing, that's changed at different speeds. Some people are still writing KornShell scripts right now as we speak, deploying apps at their company, and it probably works well. Then you have some people that are still using tools like Capistrano because they want to use something that's written in their favorite programming language, in that case, Ruby. And so they just want to keep everything within their problem domain. And then you have some people who prefer platforms like Heroku, Cloud Foundry, you name it.
KELSEY: I think the challenge has been is lots of people have been looking for that one solution for everything. I remember when Cloud Foundry, like the Heroku competitor that you could run yourself, it was like, look, twelve factor apps are the way to go, and you can write everything as long as you use Java and Spring Boot. You do that, you're done, you're great. And then it's like, okay, that's fine. What about my batch jobs? Where do I run those? Not there. What about my databases? Where do I run those? Not there? And then what happens is you end up having to bring in a second or third platform. And that's where the harsh reality of all of this stuff is, is that whenever we don't have one solution to solve everything, you end up having to complicate your infrastructure. And I think complicated infrastructure just the actual norm at this point.
KELSEY: What the world wants in terms of if you have a public facing website, you're probably going to have a cache, you're going to have Cloudflare DDoS protection. Various security concerns that Kubernetes versus Heroku is such a small part of the decision making process that even if you got that layer right, it is such a small part of the equation that thinking that's where the complexity is, ignores the big picture, where I think things like Kubernetes are 1/100th of the equation.
ADRIANA: Right. That makes a lot of sense. Now, on a similar vein of Kubernetes like product, you've also done some work with HashiCorp Nomad, right? How would you compare Kubernetes to Nomad for folks who aren't familiar with both?
KELSEY: Respect to everyone that contributes. Because I've written lots of code myself, and you do the best you can. So we just got to make sure we get that out of the way. We're not attacking people here. So if you have a HashiCorp logo tattooed on your body or Kubernetes logo tattooed on your body, this is not about you at all. When I first saw Nomad, I remember, is when they announced it in Portland at one of the smaller first HashiConfs. And I was scheduled to give a talk about Kubernetes, and I changed the talk the night before to do Nomad versus Kubernetes. And I remember Mitchell, Armon and so many people from HashiCorp standing there watching the talk. Everyone's crowded in to watch the talk.
KELSEY: And look, I'm not a mean person, so I'm not someone that's naively attack a project that I'm not working on. Doesn't make any sense. But I did learn it, got it installed. And the things I liked about Nomad, you got this single binary written in Golang. You just put it on the server and it's almost immediately ready to go, starting getting value, right? That part around, just go get a binary and just have it run on the server. It really, really made that easy. The part that wasn't great, though, is the API. You look at it and it's like, what is this thing? Right? I think I get it.
KELSEY: And it felt like, oh, you're trying to copy the Borg white paper about what a task is, but you haven't used Borg enough to know that this is not what you want to copy. And so it was a good serialization of that knowledge that was out there. They built a very high performance fast scheduler. They optimized for scheduling, speed, and performance. But the thing I think that they missed was the ecosystem. This space now is about collaboration. So you have lots of people who want to build infrastructure, automation, tools. And the one problem we've had over time, in my opinion, is that you have to glue them all back together.
KELSEY: And scripting only gets you so far when you have to glue together all these various APIs. So Kubernetes takes a different approach. Kubernetes says these things are all related. Your load balancer and your app and your IPs, and your storage, your secrets, all of it is related. And they depend on each other. And so Kubernetes felt like it lived a life where the maintainers or the people of that project had been using Borg for a decade or two and said, what would we fix? And they come into a popular ecosystem like Docker and all these pieces, and they aggregate them. And when you look at the API, you can see the experience peek through. Right here is a pod.
KELSEY: A pod has to have multiple containers because most apps that people deploy in reality, need things like NGINX or sidecars or logging daemons. And so I felt like Kubernetes had so much more experience baked into it than just being a faster, easier to manage system for deploying things. So given that, it was really nice to see over time that both communities kind of learn from each other. I remember when Nomad started adding things like volumes, sidecars, or other things that you would typically see in Kubernetes. So I think some people like Nomad because of its simplicity. I kind of lean towards the simplicity side of the house, so I kind of resonate with the whole Nomad thing. But watching people kind of glue together, like vault console, and all these other pieces to try to get a whole system, I'm like, man, at this point, now Kubernetes starts to look a little better.
ADRIANA: Yeah, I definitely agree. I worked at a job where so I had come from a Kubernetes background and worked at a job where it was a Hashi shop, and they're like, oh, we're using Nomad. So I'm like, oh, my God. How do I translate this? And when I learned that Nomad is not fully equal to Kubernetes, that you have to still stitch these other pieces together, I'm like, oh, okay, that complicates things. But I definitely agree with you. One of the things that I do appreciate about Nomad is that certain things seem a little bit simpler. And I did find the learning curve not too bad. Maybe it was because I also knew Kubernetes at the time, so maybe that helped and it allowed me to translate.
ADRIANA: But there's definitely a lot of stuff that I appreciate about Nomad, and I'm glad that I've had exposure to both ways of doing things, because I think that's really cool. And like what you were saying, both communities learning from each other rather than, like, let's hoard our secrets, because that way you can end up with better products overall, right?
KELSEY: 100%.
ADRIANA: Now, one thing that I wanted to ask you about was your famous Hashinetes tutorial. What motivated you to put this together? And also, if you can just share with folks what this Hashinetes thing is.
KELSEY: I remember the Hashinetes talk, because that was the year I was like, okay, all of these tools have been out for a while. Vault is out. Consul is out. Nomad is out. Kubernetes is out. Now what? How do you think about all these things? What do you even do with them? And I remember that year I wanted to have fun, right? Previous years, it's more about, what are these things? And then maybe years after that, it's like, it's in production. But I was like, you know what? I want to have a irresponsible talk. I remember starting to talk off: "Today we're going to be irresponsible."
KELSEY: "Do not do this in production." "Do not go to work and say Kelsey said anything." This is just having fun. Okay, and so I remember having a Kubernetes cluster or maybe even Nomad, and said, all right, we're going to install Nomad as an app to see how it works. And I just started adding different layers and components one by one. Number one, teaching people how all of these things actually fit together and how another scheduler could actually arrange them and put them into place. And then I think people had so much fun with the talk. It's like, wow, look how powerful these tools are that they can actually deploy and manage each other if you really wanted to.
KELSEY: And look how they're similar in some ways. And I think a lot of people were like, oh, these are just you need to pick one or the other. And at that time, there was a blog post of a company using Kubernetes for some stuff and then using Nomad for some of their batch jobs that would benefit from the Nomad way of doing things. I thought that was just, like, the right way to think about it. So that talk Hashinetes is like, what happens if you push Kubernetes and all the HashiCorp tools together, like using Vault for secrets instead of the thing that was built into Kubernetes, because I think Vault was a far superior secrets management product and API. And then what if you were to use Consul instead of Kubernetes built-in service-discovery? What would you get? And then let's just say you really do like Nomad. What if you were to run that inside of Nubernetes, too, and let that become the scheduler instead of Kubernetes doing the scheduling? And I think when people kind of saw that talk, they understood how to really fairly evaluate those tools. So we just had a bunch of fun.
ADRIANA: What do you think was the biggest learning from putting this talk together for yourself?
KELSEY: I think, honestly, if you just live 100% in Kubernetes land, all you know is config, maps, secrets, and you have an idea in your mind that there's no other way of thinking about these problems. Right? Everything must be a CRD. Kubernetes, Kubernetes, Kubernetes. But I think people forget I was a contributor to Kubernetes. I knew how some of the inner workings worked. And so it's like, how do you get Vault to work nicely inside of Kubernetes? Then you have to rethink the APIs, and you start, oh, the Kubernetes secret management API isn't that great at all? And so when you bring in Vault and you have to stitch it in and bake it into the whole process, you really do gain empathy for gluing all of these parts together yourself. So I think the biggest learning for me is that, number one, you can do it. There are situations where it does make sense.
KELSEY: Think about it. If you have multiple clusters and you want to have multi cluster service discovery, you cannot do that with Kubernetes alone. When you add something like Consul, you can have Consul be the place that takes over DNS. And guess what? Voilà, you can now address multiple clusters using one service discovery tool. And so it's like, oh, okay. So even though Kubernetes hasn't solved all the problems, it doesn't mean that you can't bring in all these alternative tools to step in and fill that gap.
ADRIANA: And it's nice to see that everything plays nice in that little ecosystem and that you can, I guess, take advantage of each tool superpowers, right, to sort of give that boost to Kubernetes Awesome. Now, on the Hashi front, I also wanted to talk to you briefly about a talk that you gave at KubeCon EU, "From Community to Customers". And I attended that talk, and I really enjoyed the talk. I thought it was very interesting how you were talking about this fine line of what to keep open source versus what not to keep open source. One example that you cited was HashiCorp, and then shortly thereafter, HashiCorp changed their licensing. So what are your thoughts around that?
KELSEY: Yeah, I actually had this question come up a few times, and I always tell people from a place of empathy, I had a project, Confd. It became a little popular. I remember going to FOSSDEM on the other side of the world in Europe, and watching someone give a talk about using Confd, this miniature configuration management tool, and how they were using it and why they thought it was one of the greatest projects ever. Like, as a maintainer of an open source project, you'd love to see a community form around the thing you've built. But as a solo maintainer, you also know how hard it is to say no. And you wake up on, like, a Saturday morning and it's like, hey, I work at a huge company that makes tons of profit, and I get paid really well to do my job. I would like you to work for free and add this feature that we really, really need to make even more money. And you're like, no, this is not my priority.
KELSEY: Number one, you're not paying me anything. And then two, you know what? You're going to have to prioritize that itself and maybe step up and do some contributions. And so when you think about it that way, and as someone who's also contributed code to HashiCorp products in the past, I did those contributions to scratch my own itch. And I understand that once I deliver those changes, it's on the HashiCorp team to maintain them forever. And so I understand the relationship here is me contributing code is not the end of the story. And so when they make that licensing change, I put myself in their shoes of trying to run a business and remember, they're a public traded company. So a lot of these decisions are not in fully their control anymore. The market wants to see profit growth.
KELSEY: I don't know if you've ever worked at a profitable company, people listening to this. But having stagnant revenue year over year is a fast way to get shareholders to leave investing in your stock. So now they have this added pressure of no longer just making the open source community happy. The people that they kind of started their careers off of, now they have to try to make the market happy. And there you get into different behaviors. So now you got to figure out where to get revenue from. And if you ask someone, Where do you get revenue from something that is given away 100% for free? Last I checked, most people do not pay for things unless they have to pay for things. And so you got to draw the line somewhere.
KELSEY: And I think the big controversy is, where do you draw the line? Do you draw the line on the core of the product? NGINX tried to do things like that. It didn't work out well over time, do we draw the line on, like, enterprise features and Web UIs? Right? That could be a fair place to draw the line. And so I think for a lot of people, HashiCorp decided to draw the line at commercial competition. If you take our software and start competing against us, using our name, likeness, whatever we say now in our new license, the business source license, that you can't do that. And so if you're being honest, as a user, don't really care. Like, I don't plan to start a business competing against terror. If you're being honest, I literally don't care.
KELSEY: And most people don't really exercise all their open source freedoms anyway. I'm not saying that's not a good reason not to have them, but a lot of these licenses like Apache 2 to me to fully realize the benefits of them. I think you do need to become a contributor to really understand what the code base does, be willing to step up to fork a project when the time comes and having the skills to maintain it. A lot of people don't understand that's the other part of this deal. And so when they change that license, I think people got a wake up call. They own that project. It is not our project. Even those with that HashiCorp logo somewhere tattooed on their body, it's not your project.
KELSEY: It belongs to HashiCorp. And so now I think there's a rethink. And a lot of people forget HashiCorp predates the CNCF, right? So they're not a part of a foundation, even though a lot of their technologies are foundational, TerraForm, Vault, those things belong to HashiCorp, a private company doing what they have to do. And so for me, I look at that business license change and says, great, they made their stake in the sand. From a business perspective, this will be good for HashiCorp. Now they can say no. And now their terms are a bit clear and no longer vague. Now, for the community that is upset,
KELSEY: now it's time to exercise those open source rights we've all been talking about for so long. You get to fork the project, you get to maintain the project, bug, fixes security, fixes new features and then ask the question how compatible should you remain going forward with the thing in which you branch from? That's what's on the table. So those are my thoughts on it. It's very pragmatic. I think it's one of ownership and responsibility and no matter how you feel about it, you're going to have to take on ownership and responsibility going forward.
ADRIANA: Yeah, absolutely. It makes so much sense and I think you hit on a very important thing when it comes to maintaining an open source project, which is maintaining it. It is a lot of freaking work and especially if it's something that you do on the side for funsies. You can only expect so much, especially if you're the solo maintainer. So also hats off to anyone who is a solo maintainer of an open source project or works with a very small team because it's a lot of work. It's a labor of love at that point, right?
KELSEY: I want to make sure people understand. A lot of people may have an ops background. That's definitely where I come from. And people think dev is easy and there's the same stress that you have in operations, right. For example, if you replace a hard drive in a server with a bad hard drive, you worry the first couple of days like, is that RAID configuration going to actually rebuild on time and the hard drive is going to stop being slow before traffic comes. You worry about these things and this is why we started doing things like on-call. And when you are maintained of open source project, you know that anything you merge in will make its way to someone's production, someone you probably don't know and you're going to feel responsible and accountable for doing that. And so there's a lot of this added pressure of like, hey, I got to be able to say no and make the right decisions to make sure that no one is going to be negatively impacted by these projects.
KELSEY: I think a lot of people forget that when we start to ask and I don't like the way this person runs this open source project, there is so much pressure that goes into it. So just know that there's humans behind these projects. There's a lot at stake. So if they say no to your new feature or they have to make a business license change or stop accepting pull requests for a while while they go tend to other matters, you just have to understand that just what comes with the territory.
ADRIANA: Yeah, absolutely. There are humans behind those repos right at the end of the day. Well, we are coming up on time, but before we wrap up, I was wondering if there are any parting words of wisdom that you would like to share with our audience?
KELSEY: I don't know if there's any parting words of wisdom, but I do think we're at this next cycle of new technology on its way, whether it's AI or LLMs, some people only know that stuff as chat GPT. And the question that I'm hearing a lot around is, like, is this thing going to take my job? And I always ask those folks, what is your job? And they say, "Well, for the last ten years, I've just been running scripts and automating things, and I'm like the same things for ten years in a row." I was like, "Listen, if that's how you would describe your job, then yes, you might have a problem when a new set of tooling comes around that reduces the need to do that." And that's always happened throughout tech. And I think what most people should probably think about is take these moments of insecurity and just do some self reflection and say, "Hey, my tools"...and I think we started the conversation this way. People tend to confuse automation to abstraction, and a lot of times, people get so comfortable automating the same things over and over, almost like a Westworld Loop, that they forget that we should rethink the thing that we're automating and ask ourselves if we should replace it with better abstractions. So I would say this this may be your very moment to pause for a second look at the work you do, and ask yourself, "Is it time for a new abstraction?" And if it is, I think that's the perfect opportunity to either go find a project that's attacking that problem or maybe even start your own that introduces the new abstraction based on all of that experience that you have.
ADRIANA: Awesome. I really love that. Well, thank you so much, Kelsey, for geeking out with me today. Y'all don't forget to subscribe, and be sure to check the show notes for additional resources and to connect with us and our guests on social media. Until next time...
KELSEY: All right, everyone, don't forget to Peace Out and Geek Out.
ADRIANA: Geeking Out is hosted and produced by me, Adriana Villela. I also compose and perform the theme music on my trusty clarinet. Geeking out is also produced by my daughter, Hannah Maxwell, who, incidentally, designed all of the cool graphics. Be sure to follow us on all the socials by going to bento.me/geekingout. Hey, hey Geeking Out fans! We're taking a little break for the holidays, so this will be the last episode of 2023. Be sure to catch us again in January as we Geek Out with a fabulous lineup of guests.
ADRIANA: See you in 2024. And Peace Out, and Geek Out. Bye!