Mixpanel: You shouldn't roll your own billing system, but we did it anyway. Why?

4 hours ago 1

Nowadays, it’s widely understood that building your own billing system is a bad idea — like rolling your own encryption, taking up smoking, or buying a timeshare.

Ask Hacker News;

or your favorite LLM;

the resounding answer is simple: don’t do it!

Thankfully, we live in a world where Hacker News and LLMs aren’t always right. At Mixpanel, we made the decision to invest in our own billing system, instead of purchasing one off the shelf. In this article, I’ll not only explain our decision, but also provide some questions that you (the reader) need to answer before making a decision (in either direction).

Today, I’ll play “devil’s advocate”, and explore many of the problems that can arise when integrating an off-the-shelf billing system into your B2B SaaS business. My goal is not to spread FUD about a particular vendor, or to argue that it is never correct to purchase an off-the-shelf billing system. Rather, my goal is to ensure that my reader is asking the right questions, so that they come to the best possible decision.

The argument against building your own billing system generally goes like this:

  1. Billing has numerous complexities and edge cases. These may not seem like huge problems now, especially from an engineer’s perspective, but they will cause immeasurable pain and suffering in the future.

  2. Solutions to many of these problems can be purchased from vendors that specialize in solving them; open-source solutions are also available.

  3. Billing is rarely core to your product, and working on it requires resources that could be spent on your product instead… especially at a startup.

Therefore, you should not build your own billing system, or build as little of it as possible; rather, you should leverage these out-of-the-box solutions as far as practicable.

Where’s the lie? I agree with all three premises in this argument. For many companies — especially early-stage startups and B2C companies — I also agree with the conclusion. However, the devil is in the details: for some companies, building and maintaining your own billing system can make sense.

At Mixpanel, we’re keenly aware of the complexities and edge cases inherent in billing. We operated for over a decade with a janky homegrown billing system; it caused us endless pain, required constant engineering effort to maintain, and greatly limited our ability to experiment and iterate on monetization (a key growth lever for any B2B SaaS company!)

These struggles came to a head about a year ago, when it became clear that our homegrown system was no longer fit for purpose. It was time to invest in a solution that would actually work. What did we do? We built another billing system ourselves, rather than going with an off-the-shelf provider.

One note before we get started: if you’re happy with your off-the-shelf billing system, great! This article isn’t for you. Likewise, startups should likely purchase an off-the-shelf product. This article is written from the perspective of an established B2B SaaS company that either maintains their own system today, or is unhappy with their off-the-shelf system.

As software engineers, we’ve known for a while that all abstractions are leaky. In other words: whenever something complex is simplified for the benefit of the engineer (the abstraction), that engineer will eventually feel at least some of the complexity that the abstraction tried to shield them from (the abstraction leaks).

Billing is no different. While a vendor might have a way (or several!) to deal with each of the numerous complexities in and around billing, this doesn’t absolve you, as an engineer, from having to consider these complexities.

There are many leaky abstractions in the world of billing. Here are some of the most critical to consider:

They owe you money, and you need to work with your customer to collect the payment(s) that you are due. This process is called “dunning”, and it is both challenging and important to get right:

  • Payments can fail for transient reasons, so they should be retried. Choosing the right cadence is important — too frequent, and you may trigger fraud detection; too slowly, and you might delay much-needed revenue. Should your communications shift their tone after two failed payments? Three? Eight? Should you try another payment method, if available? How does your accounts receivable team stay abreast of the situation?

  • You should communicate the failed payment to your customer — but to whom, and how? Perhaps the contact that signed the contract has left your customer, or your emails are sent to an unmonitored inbox. Exposing the failed payment within the product may increase visibility, making it more likely that the customer will pay, but it could also alarm rank-and-file users who have no power over the purse. Large customers might have an account manager; perhaps they should reach out, with a phone call or a post in a shared Slack channel? What messaging should they use; what note should they strike?

  • At some point, you might realize that your customer can’t — or won’t — pay their bill. What if your customer has been acquired, or has become insolvent? Do you block their access to the product? If so, when, and how? Do you escalate the issue to a collections agency? What laws govern debt collection in your jurisdiction? In your customer’s jurisdiction?

It will eventually happen (perhaps more frequently than you expect) — a support agent or billing administrator makes a data entry error while provisioning a customer. Invoices might be generated incorrectly; your customer could be undercharged, or (much worse!) overcharged.

  • How do you catch this issue before the customer notices? If the customer notices first, it would not only hurt your relationship with the impacted customer, but it could also increase the fraud risk of all of the other payments you process (in the event that your customer disputes the charge).

  • How can these issues be prevented at the data entry step? Does the UI provided by the vendor make the implications of admin changes — on entitlements within the product, on current and future invoices, on payment schedules and deadlines — clear? In any case, what level of control do you have over these manual workflows? Can you implement opinionated processes that minimize the potential risk of data entry errors?

  • Once caught, how can these issues be addressed? Will backdated changes reflect correctly in your financial reporting? How will your support and/or accounting staff know the “right” state for your customer? How — and when — should you communicate these issues to your customers?

It’s a common scenario: in order to secure a strategic customer and sign a deal worth e.g. 10% of company-wide ARR, a contract is signed with novel billing logic that only applies to that one customer.

  • Do the abstractions provided by your vendor allow the customer to be billed according to this bespoke contract? If they don’t, and the customer has to be billed manually, how do you ensure this customer’s wonky billing state is reflected clearly to your admins and support staff? Can this customer’s unique terms be made visible within the product? Do they conflict with any of the other leaky abstractions provided by your vendor?

  • What happens if the “novel billing logic” applied to this one customer, as an exception, turns out to be surprisingly effective at monetizing and retaining that customer? Can the workarounds implemented for one customer scale to dozens or hundreds more?

  • Can you even predict what edge cases like this will come up over the service life of the off-the-shelf system you purchase? Do you feel confident that your marketers, your growth hackers, and your ever-hungry salespeople will be satisfied coloring within the lines that your vendor drew for you? Will they even know what those lines are?

I could go on, for thousands of words, about monetization experiments, resellers and partners, sales made on third-party platforms, deprecating old pricing plans, and migrating customers to new ones. At the end of the day, nobody knows your business’ particular complexities better than you!

While the limits imposed by vendor abstractions could be welcome constraints for startups or simple B2C businesses, they can also become real obstacles to growth and scale, especially for maturing B2B businesses. Thinking about these inevitable leaks in your vendor’s abstractions, and how you’d handle them, is an exercise best done before these welcome constraints become growth-stopping barriers.

Speaking of these abstractions, and their limits:

Even if you are willing to accept the limitations, and color within the lines, of your chosen provider, there will still be plenty of billing-adjacent work for you as an engineer. These aren’t problems with the billing vendor per se; rather, they’re problems that will arise regardless of your billing strategy. Not coincidentally, these are problems that become much easier to solve with a deep understanding of the problem (and solution) space.

For example, it’s common to tie some sort of in-product entitlements to the billing state of a customer. If they purchase a plan, they get access to the product; if they upgrade, they get more features. Your vendor might provide APIs to access a given customer’s entitlements, but what if that API is unreachable? You should cache these entitlements, leveraging webhooks to listen to changes in the customer’s plan state, but a bug in your webhook receiver — or a simple misconfigured firewall policy — could cause nasty state mismatches between your billing system and your product’s entitlement system. If you have customers around the world, you might need a globally-replicated cache, to avoid 100ms+ overhead; how do you plan on keeping those consistent?

From time to time, customers have problems with your product; when they do, they’ll submit a support ticket. Can your ticketing solution identify high-priority tickets using a customer’s billing state, for immediate escalation? You might sell an add-on for “priority support”, but you may also wish to (”secretly”) give the same treatment to all customers that meet some ARR threshold. There could be SLAs tied to these categories as well. How do you implement this prioritization? How do you track your service level? If and when a strategic customer churns, do you have the logging and visibility needed to rule out bugs (in whatever ticketing prioritization system you wind up building) as a potential cause?

Your billing system will also need to integrate with several other back-of-house systems, especially as your business scales. You probably will have some sort of CRM, like Salesforce or Hubspot; how do you expose your customers’ current billing state in those systems, so that your go-to-market teams don’t burn cycles selling to existing customers? Your marketing team will also use tools that need to know which customers you already have — whether to include them in customer roundtables, or exclude them from new-business campaigns. Many technologies are considered “dual-use” nowadays; if your product is subject to export controls, your legal team may have or desire export compliance tools that shield your business from potentially backbreaking legal issues.

Each of these examples may seem straightforward — but in the context of the constraints imposed by both your billing vendor and the tool you’re integrating, the complexity can balloon. Your CRM, marketing tools, ticket management systems, etc are often stateful; subtle bugs in these integrations might pollute that state, causing issues that can smolder, out of sight and mind, until the fire, catastrophically, makes itself known.

Of course, buying a billing system also has consequences that are much easier to foresee:

For a comprehensive package, including usage tracking and invoice generation, most major vendors charge somewhere between 0.5% and 1%. This is on top of your payment processing fees, of course! There can be other costs, as well; for example, with Stripe Billing, you will also be charged for usage-based billing beyond 100 million events per month (which, in Mixpanel’s case, would equate to one mid-size customer). Add on extra charges for revenue reporting, fraud protection, or other ancillary expenses, and the total bill might have more commas than you were expecting. Other providers might have better pricing, or your procurement team could negotiate your bill down substantially, but what guarantees do you have that the pricing won’t change in the future?

There is also an opportunity cost to consider; when purchasing an off-the-shelf tool, you limit your potential monetization strategies to those that work well within the constraints of your chosen vendor. Monetization strategy is one of the most underutilized levers for B2B SaaS growth, and truly novel strategies can have a massive impact on your business. Take the example of Kagi, a paid search engine. Last year, they launched “fair pricing”, which boils down to a simple promise: if you don’t use your subscription, you aren’t charged for it. This initiative made the front page of Hacker News, where this author saw it, subscribed, and became a loyal user. Sure, n=1, but it was effective for me — and it would have been very difficult to implement using an off-the-shelf billing product. How many other groundbreaking pricing strategies are left unexplored due to incompatibility with a billing vendor’s abstractions?

Even if those costs are acceptable, you will still be burdened by a sort of “tech debt”; your business will be aligned with the abstractions and billing logic enabled by your vendor. These aren’t just technical constraints; they’re going to be written into your contracts with your customers! Your support agents and billing admins will be trained to use this proprietary platform, and the integrations you’ve built will align with your vendor’s abstractions (as well as those abstractions’ leaks). The switching cost can be substantial w.r.t. enablement alone. Besides, if and when the time comes to switch, your company will have less institutional capacity to reason about billing than it would have otherwise had (in a world where billing was kept in-house).

Of course, this doesn’t mean that you should never integrate any billing-adjacent vendors! After all,

Similar arguments could be made for any other abstraction or vendor, and outsourcing parts of your billing stack certainly make sense for most companies.

At Mixpanel, we outsource several aspects of our billing stack, including:

  • Payment processing; we use Stripe for this, but only as a “dumb pipe” to process cards. We treat Stripe as a commodity product that can be easily swapped out if the need arises. We also genericize our payment handling, so that we can easily support other payment systems in the future if needed.

  • Tax compliance is another challenge that we leave to a vendor; keeping track of tax obligations in jurisdictions worldwide is extremely difficult, and there is a robust and competitive market of commodity vendors in this space. Similarly, we outsource export compliance to a tool that integrates with both our billing system and our CRM, to prevent export control violations before the sales cycle begins.

  • We use a commercial CPQ (”configure, price, quote”) tool to allow reps to generate quotes for enterprise customers. This allows our sales systems engineers to own the complex approval, pricing, and discounting rules for our many SKUs, allowing that frequently-changing pricing logic to live outside of our billing system (for enterprise customers, at least).

We feel confident in these “buy, not build” decisions, in large part, due to our extensive experience working with billing. Because we are intimately familiar with the problem and solution spaces, we can make deeply informed decisions surrounding how, and when, to integrate commercial products. This familiarity makes us more confident architects, just as it would with any other software-adjacent problem space.

At the end of the day, the design, implementation, and maintenance of your software systems are all your responsibility as an engineer.

  • Ensure you’re asking these questions before making a “build vs buy” decision. Signing a contract does not sign away your responsibility or ownership as an engineer.

  • As many of the pain points surrounding commercial billing products are only truly felt with scale, startups should usually buy a commercial billing system — so long as they don’t abdicate their responsibilities to their vendor entirely.

  • On the other hand, mature and established companies — especially B2B companies — should seriously consider building and maintaining their own systems.

  • In summary: make this decision consciously; don’t just cargo-cult HN or your favorite LLM!

PS: That #1 post on HN, about billing systems being a nightmare for engineers? It’s a marketing post from a company selling a billing system. Of course it is.

Discussion about this post

Read Entire Article