At Hologram, I loved getting to sign into work Slack and share knowledge with fellow software engineers of all experience levels. There were about 40 engineers in a company closing in on 200 employees.

Meanwhile, the biggest change in my career as a software engineer recently has been learning Rust, which I’ve been calling a “compiler-guided tour to systems programming.” For me, it has been that and more – I accidentally fell in love with Rust’s well-crafted APIs, its high assurance qualities, and its community of like-minded and diverse Rustaceans.

I still kept quiet about it because I thought it was irrelevant to our current stack. I also thought that my job as a software engineer was primarily to write code. I hadn’t considered that organizational initiatives were a natural extension of a software engineer’s – or more to the point, any employee’s – role.

That lasted until my coworker Doug unleashed the floodgates by inviting me to pitch Rust in one of the company’s weekly “Backend Guild” video calls (thanks, Doug! 👋). I can’t remember what I said at the moment, but I tried to quickly and concisely sum up the aspects of Rust that appealed to me. It must have sounded at least somewhat compelling, though, because someone encouraged us to work on a spike and some proof of concepts, to which I responded by impulsively creating a Slack channel dedicated for what would become the Rust spike initiative.

Hologram had “Flex Fridays” in which no meetings were scheduled and you could work in a more self-directed way. Flex Fridays became a flurry of Rust-based initiatives for me, including onboarding people interested in Rust onto the ecosystem and walking them through some of the highlights, writing proof of concept code, and writing documentation – so much documentation. Everything became documentation – the merge requests (MRs are PRs in GitLab parlance), Backend Guild presentations, and one-on-one screensharing sessions were all captured and labeled in Notion to build what I hoped would become a library of Hologram-specific resources for onboarding on to the Rust ecosystem.

Alas, it was not to be. So here are my hopes and dreams, as well as some of the obstacles we met along the way.

I will preface the rest of this post by saying that it is not quantitative by any means. I would have loved to show a histogram of lines of Rust code in production or numbers of developers onboarded, but I never got that far before the company laid off 40% of staff, myself included.

“Why not Go?”

The issue of whether or not to use Go instead of Rust seems inevitable in any organization. I personally have not used Go in any professional capacity, so my authority on the topic is limited. I will say that, from what I’ve heard of it, I would likely be more excited to learn and use Go than, for example, Python.

(Sorry Python folks, it’s close enough to Ruby to give me an uncanny valley effect and I also don’t want to evaluate which of the many dependency management systems that have sprung up will work the best for me.)

Here are the good things I’ve heard about Go:

• Straightforward language and syntax
• Fast compile times
• “Batteries included” for web development
• Concurrency primitives included
• Extremely good performance out of the box
• Cross-platform compilation
• Statically typed
• Compiles down to binaries

I absolutely did not want this to turn into a language flame war, but unfortunately there’s something about either the closeness of the release dates of the two languages or their mutual reputations as “systems” programming languages that brings them into contrast with each other. I still don’t want to talk about either language in the framework of “Go vs. Rust,” so I’ll talk about the context that directly lead to that comparison, instead.

Shortly before I had joined Hologram, there was an official decision made in the engineering department to adopt two more languages: Go and Typescript. Naturally, when we started talking about adding yet another language, comparisons with Go came up, though in retrospect, it’s a bit puzzling why Typescript was largely left out of that conversation.

When I started to join Rust advocacy efforts, Doug and I were counseled to know our “opponent” (Go) inside and out, to position Rust as being better than Go, and to really make a business savvy case for Rust. I took this advice to heart, but I also felt that something was off.

I didn’t want to talk about Go because it’s simply not what I was passionate about. I also didn’t want to put down Go users in the company. I just wanted to extol the benefits of Rust and see if anyone was interested in trying it out.

Which brings me to my next point. I thought at first that I had to convince The Company to add it to The Stack™. In actual fact, I had two broad categories of stakeholders to convince, not one Company.

Stakeholder #1: The Business

The Business wants to know why we should invest engineering time into another technology. The Business is about Return on Investment (ROI), and The Business will make do with what we have in order to get things done on time and under budget. The Business will make nuanced arguments about balancing tech debt versus expansion.

I’m not disparaging business interests – it’s important to understand, and it’s part of why any of us are even employed. In fact, one of the biggest lessons I learned from Hologram is how every engineer can focus on leverage in order to have the biggest positive impacts on business.

I had a few ideas on how to convince The Business that Rust was worth investing in.

Bug case studies

I kept an eye out fixes for bugs that would have been entirely prevented or greatly mitigated in a Rust ecosystem. All of the ones I can remember (invalid array accesses, null return value, empty set checks) would have been caught by returning Rust’s Option variants for operations.

Then I estimated how much time it took each developer to find and fix these bugs. Developer time translates into money, and these types of bugs were costing a pretty penny. There were two particularly bad weeks where using Rust would have saved a combined 3 weeks worth of developer time. Whether or not they caused any customer-facing issues is also worth noting.

I made sure to write the case studies up, including what happened, how it impacted engineering time, and how writing such bugs would have been impossible or unidiomatic in Rust.

I presented the case studies during a Backend Guild video call and filed the recording away for later reference.

The case against Go

We were challenged to provide a gap analysis for using Rust over Go. I’ll go into more detail about why I didn’t like that framing in a separate section, but to sum up in bullet points, I said:

• Error handling in Go is optional and not ergonomic
• Nils and other issues can slip by Go’s type system
• Rust’s platform support means that it can be deployed to anywhere from embedded systems to backend web servers

I read a lot about Go experiences, like these:

• Experience Report: 6 months of Go
• Data Race Patterns in Go
• Lies we tell ourselves to keep using Golang

I also made sure to connect Rust’s benefits over Go with the Engineering department’s started charter values, like how we generally end up going all in for the difficult, more widely applicable technologies, and how we needed to be business savvy – and squashing bugs before they reach customers is business savvy, especially for an infrastructure company like Hologram!

Yet, it made me sad to look up articles on the language just to, in effect, bash it. I’m not generally in the business of discouraging people from coding, regardless of what their favorite programming language was. If I were, I don’t think I would have been working on a legacy PHP code base, myself.

Which brings me to the second category of stakeholders who we needed to convince.

Stakeholder #2: The Engineers

In a perfectly rational world, we would all make perfectly rational decisions for ourselves and for the business. These decisions would be carefully weighed until the optimal time and then executed.

But that’s not how humans work. For one, we all have different priorities. There was the engineering manager who was constantly pushing us to write more documentation. There was devops, who wrote a Go CLI tool to orchestrate pipeline tasks. There was the billing department, which I was on, which put out fires necessary to earning the trust of our customers and their wallets. And there were plenty of engineers who just wanted to go along their days and get things done.

Ultimately, it was these folks who we had to convince. In retrospect, while the business had been officially convinced of the use of Go and Typescript through our decision making processes, the proof, as they say, is in the pudding, and engineers were still writing Javascript and PHP. Their minds may have been made, but their hearts hadn’t been won.

This was also where, in my opinion, the real difference was made. If we won people’s hearts, the business case for the language would be made, as all the engineers would want to write Rust, anyway.

Here’s how I had been planning to win my fellow colleagues over to the dark side:

Personalized onboarding

One of the first things I offered to do was to use our Flex Fridays to onboard people to the new language. It entailed getting a working Rust toolchain up and running and maybe a brief tour of high level language features.

I tried to remember to ask if we could record each session, both for later publication and to see if there were common aspects of onboarding that were problematic. I viewed it as an exercise in customer development, Lean Startup style, as much as an active tutorial.

We did a few of these with people from different backgrounds. One principal engineer with Swift experience compared it to guard lets and other features from Swift. Another engineer found the tooling and ecosystem quite fascinating. And a QA team lead had a lot of fun in our introduction to Rust’s built-in testing capabilities.

Goals:

• Get people over the “getting started” hump
• Give high level overview of Rust toolchain features
• Bolster documentation with videos

Metrics:

• Number of engineers onboarded: 4
• Videos added to documentation: 2

Proof of concepts

In our use of the term proof of concept, it was less about feasibility studies and more about small code spikes to disseminate learning. To that end, I wrote one proof of concept myself, a cross-platform (Unixes + Windows!) hosts file editor for local development, and contributed to another proof of concept that Doug had started involving DNS.

For both of these, I posted MRs to the Rust slack channel so that everyone could get context. And in the MRs themselves, I went into great detail on the Rust features I was using and why, like Struct Update Syntax, or doc comments. Of course, in a production code base with shared context around the language and ecosystem, these explanations would be overly long winded and superfluous. But since our deliverable for these proof of concepts was education, it was appropriate to bloviate a little.

Goals:

• Write proof of concepts
• Submit code changes for review by non-Rust users
• Assess feasibility of using Rust

Metrics:

• Proof of concepts written: 2
• Merge requests: 6
• Merge requests with extra details: 3

Presentations

As I mentioned before, we had a Backend Guild video call every week. I like public speaking, so I used the opportunity to talk about Rust when I had something to talk about.

One of the engineering managers mentioned that she needed better documentation, so I whipped up a quick demonstration of Rust doc comments and doc tests. It helps that documentation is a first class citizen in the Rust ecosystem, so it was easy to do so on the fly. It was a demonstration of the built in documentation, how doc comments get included in the by the doc builder, and how the doc tests are executed and run as code.

I also presented an argument for Rust, the language as a whole, with the main points being the documentation, safety (in terms of impact of bugs on the business), and community. I think Rust’s diverse and inclusive community (and therefore steering parties via teams and work groups) is perhaps one of its biggest strengths, which manifests itself in a very nuanced, safe, and comprehensive language.

Goals:

• Introduce the general engineering department to Rust and specific Rust features
• Keep engineering updated on Rust spike initiatives
• Add videos to documentation

Metrics:

• Rust talks given: 3
• Recordings made: 3

Documentation

Because communication and education were key to the Rust spike initiative, documentation was key. I wrote a homepage on our Notion documents entitled “Rust Spike 🦀” which housed all of the above:

• Simple getting started guides
• Links to proof of concepts
• Onboarding and presentation videos

Additionally, I started an #eng_spike_rust Slack channel which we casually posted news and MRs to.

On metrics

I would have loved to have better metrics of engagement rather than just results, and I hope that someone takes up the mantle and introduces those metrics in my absence. But in an ideal scenario where Rust is officially added to the stack, we would have been able to track better metrics for Rust uptake (depending on goals):

• Proportion of Rust code to legacy code
• Proportion of new code bases written in Rust
• Performance and maintenance in Rust vs non-Rust code
  • Bug rates
  • Uptime
  • Incident rates
  • Performance
• Usability
  • Number of authors of Rust code
  • Time to first contribution
  • Satisfaction surveys

…and these are just a first pass at what would hopefully be useful metrics. We would likely need to redo based on what ends up being useful to measure.

Conclusion

I think company-wide goals and metrics were one place where, if the Go initiative at the company had been stronger, there wouldn’t have been as much of a vacuum for another language to be considered for general use. Several teams thriving on using Go for their purposes would have made it harder for an arguably marginally better language to shine. Instead, the language stayed siloed in a single repository owned by one team, and the rest of the backend engineers were either stuck on PHP or starting out on Python. In this environment, Rust had a chance, so we made a case for it.

The success of a new language at a company is about the uptake by the individual contributors, and that is a campaign that’s best made one engineer at a time. That was my starting point, but it would be foolish to leave the business out of the calculus, so I’m grateful to have been pushed to make a business case for the language as well.

In the end, my time at the company came to a close before I could see the Rust initiative to any sort of fruition, but I’m grateful that Hologram had that kind of openness to engineer initiatives. It allowed me to learn a lot about advocacy and organizational change.

…even so, I hope to land somewhere that already uses Rust for my next job!