Enterprise software is, in general, garbage. I can write that line without fact-checking it and Republicans and Democrats alike will nod their heads. But why? Why is it so bad? According to open source veteran Matt Wilson, “Because the people building the software are not using it to solve problems.” They’re vendors, not end users.
In a hopeful sign, however, the long-awaited end-user driven open source future might finally be arriving. Take Envoy, for example, an open source edge and service proxy developed at Lyft by Matt Klein. In my recent conversation with Klein, he argued that unlike open source projects spawned by VC-backed startups, Envoy came with a built-in, “captive customer,” which “forces you to really think about things like operational quality, making sure that you’re building features that people actually care about.”
In other words, the people building the software were the same ones using it to solve problems, resulting in better software. Indeed, Klein notes, “One of the major reasons Envoy became so popular is because it was built for an end-user case.”
So why don’t we see more successful end-user driven open source? According to Klein, it comes down to a failure to appreciate that open source is a “f---ing lot of work,” and a failure to think through the costs and benefits clearly.
When open sourcing is a ‘net negative’
Fortunately, Klein had no idea how hard it was going to be to successfully launch Envoy outside of Lyft as an open source project. He and his team spent 2015 building and using Envoy to help Lyft shift from a monolith-based infrastructure to a microservices-based infrastructure. By the spring of 2016, Envoy was an unquestionable success within Lyft.
That’s when the open source itch began. “Towards the middle of 2016,” Klein relates, “we started to have serious conversations about open sourcing Envoy and at the time I was fairly naive about what was about to happen.”
Klein had puttered around with some open source projects, but he’d never engaged deeply with one, and certainly had never launched a project. Yet Klein was curious and wanted to try. Lyft’s management initially demurred (“This is not our business”), but Klein persisted. As Klein put it, “I was motivated to go do all of the work and I thought I understood what this work would involve.” He soon learned he did not, in fact, understand what the work would involve.
Between Lyft management’s willingness to give Klein a shot, and Klein’s ignorance as to what that “shot” would entail, they decided to proceed. This is fortunate to all of us who today benefit from the impressive Envoy project, but it seriously drained Klein for at least a year:
If you look at what I did in 2016 and early 2017 to [introduce] and grow the project, it was not technical. It was all leadership, public relations, marketing, documentation, etc., and I did it all myself and I nearly killed myself [doing it]. My work-life balance in the late 2016 and early 2017 time frame was not good. I was basically doing two jobs but I was super motivated to do it. Still, there were a lot of issues in trying to balance the Lyft work with the open source work.
Starting an open source project is like starting a company, Klein says. Much of the work is the same. “There’s the PR, marketing, and there’s what I call hiring,” he says, “which is getting contributors and maintainers.” There’s also the work entailed in documenting the project and genericizing it, such that it will be useful to those outside the initial developer.
It’s a lot of work, leading Klein to say something that might seem controversial:
For most people who open source something, it’s actually a net negative. By net negative, I mean, if they don’t invest in it, if they don’t do all of these things [like PR, marketing, and “hiring”], they’re just going to throw something over the wall.
Which is useless. Worse than useless, really, “because you give out more than you get in,” he continues.
It’s easy to look at Envoy now and say, “Of course it was worth it!” But Klein and Lyft couldn’t see this in 2016. Today Lyft benefits from having dozens of people actively contributing to make Envoy better. “Lyft benefits from features that we never knew that we were going to need, but now they’re all implemented by other people and then we just use them,” Klein notes. Lyft pays nothing for all that additional help. But in 2016 Klein was having to figure out how to justify the investment he was making in Envoy, sometimes building features that Lyft didn’t even need, but that made the project more attractive to its budding community.
“So you’re asking why don’t people do this?” he says, responding to a question I asked earlier in our interview. “It’s because it’s a f---ing lot of work. And the benefits are not super clear. It’s not a slam dunk. You don’t know if you’re going to win. And if you don’t win, it’s a net negative.”
So why would anyone do this, much less an enterprise that isn’t in the business of selling software?
What’s in it for Lyft?
What’s in it for a Lyft? Why would they make this “massive amount of effort” to open source internal technology that is already valuable within the company? Well, in addition to the potential for significant outside investment, there’s immense recruiting value. As the home of Envoy, Lyft looks like a great place for developers to build industry-defining code. And for Klein, successfully launching Envoy helps his career, too. Such motivations aren’t the same, Klein says, but they’re aligned: “I tell people that they don’t have to be aligned in the same way, they just have to be aligned in the same direction.”
The problem, he continues, is that we can be aligned in the same direction while still completely, utterly failing to gauge the “soft costs” involved. We can evaluate the cost of servers or engineer salaries but it’s much harder to evaluate the TCO (total cost of ownership) of a successful open source project:
You’re speculating, you’re saying, “Well, if it’s successful, I’m going to get a recruiting benefit.” What does that mean in monetary terms? No one has modeled that. Or, “If we’re successful, instead of the team of three that we have working on this, we’re going to get a team of 10 and engineers cost half a million dollars a year and that’s worth this much money.” No one actually thinks about it in this kind of rigorous way. No matter what anyone says, it’s not true.
Because it’s so hard to model the financial implications of open source success, and the potentially daunting costs of getting there, it really comes down to “gut instinct” when companies determine to proceed. But that same nebulous calculus dissuades many from even trying.
Knowing all this, would Klein do another Envoy? Would he subject himself (and his employer) to the rigors of building an open source project? The answer is... it depends.
The go/no-go decision comes down to a “subjective assessment of whether we can win the market.” If this sounds like business talk, not developer talk, that’s intentional. “I’m not an open source purist,” Klein says. “I’m a capitalist.” As such, Klein approaches open source as “an optimization exercise” to determine whether the open source project can “win the market” and therefore “get effort from outside that will be greater than what we put in.”
Which, in its way, re-introduces optimism into the open source calculus with which many companies may be struggling. Every company takes certain risks in an effort to acquire customers. If we think of open source as the same sort of exercise, that may make it easier to understand and countenance the costs of doing it well.