“What’s the biggest thing you’ve learned while building Segment?”
Over the last five years, we’ve made plenty of mistakes. And while many of them fade into the background, there’s one thing I wish we’d done above all else.
I wish we’d written everything down.
Sooner. In more detail. Describing all the inputs to our decision-making.
This seems obvious because writing itself is so ancient and fundamental. Of course writing is important! It was the revolutionary technology that allowed us to really start advancing.
But it’s easy to skip writing when you only have a few months of runway. Or when you’re rushing to make a product deadline. Or when the site is going down. Or if your entire development team is so close, they share the same tube of toothpaste.
Yet looking back on it now, I wish everything we’d said, wasn’t said at all. But written down instead.
In a sense, writing something down is the most lightweight process imaginable. It at least lets another person retrace your steps and begin to follow your thought process instead of being lost to the hazy record of memory.
And as it turns out, writing crops up all over the place. It’s not just code, but business processes, financial models, and product decisions.
In every part of the business, you want to create reproducible systems. And without any written record, you have zero chance of building upon the common knowledge that your team has already fought to gain.
Here’s a few examples that where we’ve failed during Segment’s history
- lack of READMEs or tests – we worked for about 18 months as four people in a room, before making our first hires. When they came on, there was nothing to get them started short of in-person code walkthroughs.
- ops people with tribal knowledge – for the first few years of existence, all of our devops work was handled by a very small on-call team. We didn’t have ops playbooks or recorded solutions to common problems, and we didn’t share common approaches to fixing issues.
- one-time analysis – in the past we’ve based some of our most important product decisions on the outputs of detailed synthesis and research. That analysis would usually have some one-off graphs or results, but no repeatable means of verifying that data.
It pretty much goes without saying that these all slowed us down in the long-term, even if they accomplished the immediate goal in the short term.
Granted, there is a definite trade-off here. Many of these decisions actually make sense in the early days, when there is a 90% chance that your startup will die within the next year. But it’s important to recognize when that assumption no longer applies. And at that point, it’s time to shift your culture to invest more heavily in written documentation.
As your business gets bigger, writing only becomes more important. The more people working on a problem, the more essential it is to make sure they aren’t doing duplicate work. That they are leveraging one another’s tooling. That they can find what they need quickly without searching for hours or scheduling a meeting.
Writing as a competitive advantage
If you buy the idea that writing can save so much time and lead to greater efficiency–then it makes sense that the tools and culture around writing should make a significant difference in how well a business performs.
Tools accomplishing this task well, are massively valuable. To the point where it allows your teammates to save hours every week that they would have spent explaining systems to one another.
There are three primary aspects to a highly functioning system of record. In order of importance (and lifecycle), they are editing experience, discoverability, and understandability.
The editing experience governs how easy it is to create and update written documentation. If documentation isn’t easy to update and edit, it won’t stay current. And in many cases, won’t be written in the first place.
Discoverability answers “How quickly can I find the information I’m looking for?” If no one can find it, then simply having the written record is useless.
The last aspect is understandability. As a reader, once you’ve found the ‘right’ document–it doesn’t help you until you understand it. And if the barrier there is too high or the material too complex, you’ll look elsewhere.
There’s been businesses built around various aspects of this problem. Google tackles discoverability. Wikipedia puts a premium on the understandability of solutions. Medium has invested in the editing experience.
But even with all those systems, we haven’t yet seen the great business system of record anywhere yet. There is no ‘de facto’ standard for teams working together.
Dropbox Paper nails the editing experience, but its unstructured approach makes it difficult to find and organize information. Atlassian’s Confluence focuses on a building well-structured wiki, but has more of a barrier when it comes to the editing experience.
All tools make different sets of trade-offs, balancing between structure and editing experience. But there’s no clear winner.
And it’s tempting to continually be searching for that holy grail. The one tool that effortlessly captures your every thought, and then indexes them perfectly.
But as enticing as that temptation is, in practice, chasing after the perfect tool ends up being the wrong approach. It’s a case where “perfect is the enemy of good.” The problem isn’t the tool at all–it’s you.
Instead, the businesses that scale effectively are the ones where documentation is a part of the culture. Tools can facilitate documentation, but they can’t create it. Only people can do that (for now, anyway).
If tribal knowledge is lauded or even tolerated within your culture–some aspect of your company will be bottlenecked by an individual person. It’s why the fastest growing companies are the ones that find ways to quickly share knowledge with as much of the team as possible.
Building a writing culture
I won’t sugarcoat it, building the discipline to write is hard. And transitioning from speaking as the default form of communication is even harder.
To be really effective, your entire team has to see the value in writing. It’s just hard to really recognize that value until you’ve experienced it firsthand.
This is definitely a case where a little perspective has gone a long ways. Some of the earliest examples of this we had were just emails sent on our founders email list.
One of us would have an idea, ruminate on it, and then send out a reasonably well-developed version of it. Not to the point of a polished essay, but with enough substance to get the rest of us thinking about the idea as well.
Because these ideas started in email, that’s where the surrounding follow-up discussion happened too. We still have the original emails that launched Analytics.js, Segment, and our data warehousing products. Commentary, alternative ideas, and feedback all perfectly preserved alongside the original messages.
It’s an incredible archive of our thoughts, hopes, fears, and general perceptions of the market. Everything from detailed product brainstorming to briefs on infrastructure. Every so often, I’ll re-visit them to see where our assumptions were right, and where we were wrong.
As we’ve grown, we’ve had to scale the ‘think pieces’ beyond just an email list that goes to the whole team. Though we still have product briefs that get emailed around, writing at Segment today takes many more varied forms.
For example, we run the same set of SQL queries against our data to produce ‘golden metrics’ on a weekly basis. We spent a lot of time verifying that the queries are correct, and that we can always reproduce the analysis and audit the logic behind them at any time. Because all the SQL pulls from raw data in our warehouse, we can tweak the analysis and re-run it against historical data if there’s ever a discrepancy.
On the infrastructure side, we started running a formal outage management process which examines what went wrong during outages in the data pipeline. If it’s preventable, the systemic fixes are made. And all outages are recorded with a timeline, root cause, and follow-up, so that we can notice broader ‘classes’ of issues that might be lost in the heat of the moment.
Lastly, engineering as a whole puts in a lot of effort to create open source libraries and tools. Oddly enough, documentation is one of the main reasons we’ve really embraced open sourcing many parts of our stack.
Early on, building open source software wasn’t even a conscious decision. But, by building code for developers all over the world, we’re forced to create cleaner abstractions and better reference documentation. This code ends up looking almost exactly like any of the other open source libraries we use, and has a much longer ‘shelf life’.
So the next time you’re building a process, be it code, a set of instructions, or even an in-person transfer of knowledge, take a step back. And ask yourself, “If tomorrow, I fall off the face of the earth, can someone else re-trace my steps?”
There are many ways to solve that problem–but the simplest is always the same:
Write it down.