We rebuilt our docs from scratch. It was worth it.

Documentation is a core part of any developer-focused product. Docs help developers learn how to use your product, find inspiration for new use cases and solutions, and debug problems they may encounter. 

Documentation can also be a powerful conversion tool for a modern company with a product-led growth strategy. Good documentation solves problems for users, and great documentation inspires non-users by showing them the possibilities your product creates. In our web analytics, it's not uncommon to see our site visitors land on the homepage, check out the pricing, and then immediately click into docs.

In short, docs matter.

Recently, I worked with my colleague (and all-around Docs wizard) Lucy Mitchell to revamp Tinybird's docs. We ditched our old docs tech stack and rebuilt something new, from scratch. We chose not to use a pre-packaged docs framework, instead opting to piece together parts and get exactly what we needed.

It was totally worth it.

We recently ditched our old docs stack and built something from scratch. It was totally worth it.

What got us here won’t get us to the next stop

Our old docs tech stack was based on Sphinx. It was slow and, for lack of a better term, "monochromatic"; just text and images that weren't very inspiring. The stack served its purpose as Tinybird grew up, but we needed more for where Tinybird was going.

For example, we didn't have an easy way to generate “related” documentation or personalize pages based on a user’s interactions with Tinybird’s website. If you viewed a live coding session on building real-time dashboards over Kafka, the chances are high that you would be interested in Tinybird’s Kafka connector. We think this personalized information should be presented to you as you explore the docs.

We want to create a personalized docs experience that syncs with your product usage in Tinybird.

Our prior stack also didn’t allow for easy integration with Auth0, which meant we couldn't create the “logged-in experience” that we hoped to build so we could personalize content, add easter eggs, show code snippets using your data and schemas, etc.

Perhaps most importantly, our old docs tech stack didn’t gel with the workflows of the people who could and should be our most prolific tech writers: our product and engineering team.

Features we needed to support

When we set out to build new docs, we listed the features we wanted and surveyed our customers asking the same. From there, we built a list of features we felt should be supported in our docs. Here's the feature list that became the foundation from which we began to explore our new tech stack:

Initial (MVP) Features

• Code snippets
• Search
• Auto-generated API documentation
• Simple image management
• Analytics
• Dark mode (and more)

Long-term Features

• Personalized, logged-in docs experience that persists between the product and docs
• Dynamic code snippets that adapt based on your Tinybird data
• Copy-to-clipboard functions that auto-populate code snippets with your API keys and tokens
• Personalization (“read this next,” “related documents”, "bookmarks", etc.)
• AI search
• Image optimization
• Collapsible nav elements
• Content feedback tools
• Guides that adapt based on your preferences (e.g. UI vs. CLI workflow)

Operational requirements

In addition to our feature list, we asked ourselves a series of questions about how we would maintain our docs:

• Who is going to maintain this stack? Will there be a specific team, or will everyone be able (or expected) to maintain it? Do those people have the skills to do the job? Me (Julia), a frontend web developer.
• Will the site require frequent maintenance? Like, disk resizing, restarting of subsystems, upgrading? Not really.
• Is this critical functionality? If so, will it be in on-call alerts? Will it be easy to maintain? Will it fail often? Will it be included in the status pages? Yes, it is critical because it is customer-facing. Failed docs mean bad UX. But, it is a static site and is unlikely to fail, so on-call alerts are not needed.
• Will there be migrations? Will the same team handle them? Will the system scale gracefully? No migrations. 
• Does scalability matter? Yes. We need to be able to scale to upwards of 10x what we already have.

Checking out the developer docs landscape

We also took a look at all the docs that inspire us and the tech stacks they were using:

They all use different tools, but the same approach: static site generator. Some use a specific CMS (such as Readme.com), but in general, the trend is to use Markdown for documentation.

Most of the docs sites we love use a static site generator with Markdown.

Monorepo or separate repo?

We also needed to make some decisions about where our docs would live (both the underlying docs platform and the docs content). We’ve always wanted to open source our docs so that our community can be more involved, raising (or even fixing) issues directly against the content.

We could…

We decided to use a dedicated repository.

On balance, the productivity gains we'd get from having a dedicated repo far outweighed the slight duplication it would require. A dedicated repo also removes almost all of the technical friction to make the docs open source, which gives us one less excuse to put it off! 😅

Which docs language?

Our old stack used reStructuredText (rST). There are some nice things about rST, and it made sense given that our docs started life as comments in Python files back in 2019. But there are also a lot of not-so-nice things about rST. The ecosystem is small, and rST has significantly less adoption in the broader developer community than Markdown. It’s pretty easy to find devs who have never even heard of rST, but almost every dev has written at least a few lines of Markdown.

We didn’t need to research it, we knew Markdown was the only way to go…

But…

We knew we would want to extend just plain Markdown and add additional interactive elements, so that wasn’t the end of the story.

To enrich Markdown with custom code we considered two options: MDX and Markdoc.

Ultimately, MDX and Markdoc are both great, and there was no objectively wrong answer. We chose Markdoc because we like the philosophy of keeping code and documentation separate.

Which framework?

We were originally leaning towards Docusaurus, but then we decided to investigate the pros and cons of the different frameworks we could use. 

Framework is a critical decision, perhaps the most important in this process. Once we made the decision and built the site, there was no turning back without a lot of work. The further we advanced the docs, the more dependent we'd be on our chosen framework.

Framework was probably our most important decision. Once you build with a framework, it's hard to change.

Ultimately, we'd need to choose between a general-purpose framework capable of building any website (e.g. Next.js) and a docs-specific static site generator with built-in capabilities specifically for technical documentation (e.g. Docusaurus or Starlight).

Our current website is built with Next.js. We're comfortable with it, we know its capabilities, and it has a massive community. Plus a bunch of our most loved docs sites use it (Segment and Plaid).

Docusaurus is very well-known for being an awesome framework for building docs. It is based on React and supports MDX. 

Starlight is built as a library on top of Astro, so technically it would be an intermediate solution between both: a general-purpose tool + docs specific.

We laid out the pros and cons of each:

And we created a side-by-side feature comparison matrix:

Docusaurus did not meet some of our core requirements. So it was out.

Next.js and Starlight would both be good options to build the documentation of our dreams.

We decided on Next.js, ultimately prioritizing the consistency with our current stack, our comfort with it, the flexibility it provides when building anything, and the supportive community.

Next.js gave us a lot of flexibility and we were already comfortable with it, so it seemed like the best choice.

Infrastructure and styles

We use Vercel to host and Tailwind CSS for the CSS framework. There’s nothing interesting to say about these decisions. It's what we use for everything, and there was no reason to consider anything else 🤷‍♀️

Features

Now we get to the really fun decisions: the main features we want to keep or incorporate into this documentation site and the technologies we value using. These are not the only features, rather they are the ones we consider worth sharing or investigating to make a more thoughtful and shared decision.

Authentication

We use Auth0 in the product, and soon, we will provide a logged-in experience in our docs based on Auth0. If you have a Tinybird account and are logged in, you will continue to be logged in when you visit our docs.

Analytics

We use Google Tag Manager, to which we have added GA4. This is the spine of our marketing analytics stack, and we will continue to use it in our docs. Eventually, this will enable us to personalize (using Tinybird!) our docs based on your Tinybird website and product activity.

Images

Vercel and Next.js have built-in image optimization features. It’s still important to keep images small for the sake of the repository. In the long term, we will explore more sophisticated image processing tools.

Search

Pagefind is a perfect match for us. It’s a library that does not require a dedicated server, it runs after your static generator, and it outputs a static search bundle to your generated site. The index is generated for you from your generated site.

We considered other options:

• Meilisearch: A nice, modern alternative to ElasticSearch, it’s an attractive option for building a custom search experience. We decided it was too much for us right now as we have a relatively small amount of content, and it would take a bit longer to ship. If we decide that Pagefind isn’t working out, Meilisearch is our top contender.
• Algolia DocSearch: We've used this before & it was free only for FOSS, and we didn't get the memo that this had changed. If we had, we'd probably have used it here and not given it a second thought.
• Fuse.js: This is a nice lightweight JavaScript library, but it's unclear if the project is still maintained 😬, and it would require more effort to adapt for Markdown.

I’m particularly proud of our search implementation. It’s fast and it’s a great developer experience.

API docs in-code

We currently generate our REST API documentation automatically from comments inside our product Python code. This was one of the reasons we had originally selected a Sphinx-based stack for our prior documentation platform. Unfortunately, it will be difficult to keep this integration as easy or smooth as it is currently. It is one of the things we will “sacrifice.”

Our solution here is a bit of a hack. Sphinx compiles this documentation from the comments into HTML fragments, which are then pushed to a known location. The new docs pull them in to build the full page. It’s not ideal, but it works and doesn’t impact the user.

As is typical with small companies, there are many things to do, and migrating our REST API to OpenAPI just hasn’t happened yet. We’ll get there, and then we can revisit this.

Code snippets

Because of the limitations of Sphinx, our old documentation used iframes for code snippets. We had built a little "snippet generator" where we could submit any code and it would return a little HTML <iframe> tag to paste into the rST document.

That iframe took care of the important basics for code snippets:

• Copy to clipboard button
• Highlighting & formatting for different languages: Tinybird SQL, bash, JSON, etc.

This was a nice little hack for a while. It didn’t take long to build, and it solved an annoying problem, but it wasn't ideal moving forward. You couldn’t read snippets in the doc file, and to modify the snippet you just had to create a new iframe and replace it. It also wouldn’t work if the docs were open sourced.

Tinybird SQL is a bit unique because it combines ClickHouse® SQL with our dynamic templating library, so we needed a custom code snippet generator for our documentation.

Fortunately, the migration away from this was quite simple. We’d already built a custom React component for the snippet generator that did everything we needed, so we just took that and used it to render the Markdown code blocks denoted by triple backticks (“```”). The most time-consuming part was just copying all the code from inside all the existing snippets and pasting it into the new Markdown.

Accessibility

Our new documentation ensures that accessibility is taken into account. At the moment, we comply with level AA.

Some of the initial measures we have taken are:

• Adding a dark mode option
• Supporting keyboard navigation
• Including some standard shortcuts, such as Ctrl/⌘ + k to open the search.

As part of our documentation style guide, we ensure that images and videos always have alt text or captions describing what is seen. For custom components, we rely on React Aria Components.

In the future, we are aiming for level AAA support.

What’s next?

We have a huge wish list for features for our docs, including:

• AI-powered help: chatbots, GPT-powered prompts, and more.
• Image optimization for resizing, compression, and hosting 
• Personalization across our site based on interactions a user has had on our various web properties as well as within the product
• Feature flag support. As customers are added to pre-release features using feature flags, that content is automatically shown for them in our docs
• Versioning for different versions of our API and CLI

Check out our docs!

Want to see the fruits of our labor? Check out our documentation and let us know what you think.

And if, while you're reading, you realize that Tinybird is exactly what you've been missing in your life, then you should sign up. It's free to start, with no credit card required and no time limit. If you have questions or feedback, you can join our public Slack community and ask away.