You know what’s in your code and you don’t want to admit it to yourself, your board or your investors. Your product was developed fast to get to MVP, to get the first and the second versions out which should have been named V1.1 but you called it V2.0 to make it look like it’s been developed for longer, and to pretend that you are not just a woman and a computer sitting in a shed.
You know your coverage of unit tests is in the low digit %, and lots of your code is held together with sticky plaster, and only one or two people know how it all works. But you are winning customers and proving your market fit so you keep going at a fast speed, your team keeps adding features to satisfy those early golden customers, and you compromise on documentation, proper design, scalability, test cases, unit tests and just patch it up when it fails. And it’s all going ok, you are delivering fast, the early customers love you and your revenue is growing, so you think you walk on water. Now you just need to raise a B round to add more sales and marketing and get even more customers and grow even faster.
Your B round investor comes on board with the promise that their money is spent on getting more customers and growing your market share. But the reality starts hitting you. With increased sales and number of customers, suddenly your product is creaking, support issues are coming in at a rate that engineers can’t fix because the new software engineers you’ve hired can’t find their way round the code and there is not a proper design or documentation. So it all falls on your early team who really know how it all works but they are now overworked. It’s a vicious circle and to make matters worse, your new engineers are doing the best they can but fixing a bug in one place causes five other bugs to appear somewhere else.
You also notice that these new customers are far more demanding than your early adopters. They actually expect a working product from a company that deals with them professionally, responds immediately to issues and resolves them well within the contracted maintenance times. On top of that, your new customers want to know what’s on your roadmap and expect it to be delivered against the promised timelines. And to put the icing on the cake, in every meeting you have with them, they don’t miss the opportunity to mention they had your closest competitor in their office showing them their latest products – and they are half your price.
You are suffering from “technical debt”, which every software company CEO suffers from when moving into the scale-up phase. If not brought under control, technical debt could end your business momentum by destroying your reputation in the market and delivering you very unhappy customers who will ultimately churn and go to your competitors. In this blog I want to address the questions that every scale up CEO will face. How do I sort out my technical debt whilst keeping my competitive edge? How can I bring out features that will keep my existing customers happy? And how will I increase my market share?
First you need to find out how bad it really is. I suggest having an independent, experienced CTO/VP Development conduct a proper assessment. The assessment should be end-to-end: this will include dev process, design, documentation, architecture, scalability, unit test coverage of the code, QA including regression, integration, performance and backwards compatibility tests. Prioritise and set targets for where you need to be and put together a plan with the dev team to get there. Some CEOs panic and throw all their development resource to fixing technical debt which grinds innovation and roadmap to a halt, and results in a demoralised software engineering team, and loss of momentum in the market. No software engineer likes to fix issues and write documentation all day, every day, for months, and no new customer will be thrilled if you have nothing extra to offer them for months or years.
In my experience, the only way to address this is to own up to the fact that you have to address technical debt in order to increase quality and that this will take resources. If you properly planned for tackling technical debt when you were asking for your B round, things will go a lot smoother.
As a rule of thumb, I’d follow the one third rule. Assign one third of your developers’ time to internal activities that address technical debt, such as re-architecting, re-design, addressing performance, scalability, unit test coverage, re-writes/refactoring, and documentation. Assign one third of their time to customer support and maintenance releases (i.e. fixing customer issues and bugs), and dealing with any “Oh my God, we need to get this done now or we will lose this contract!” emergencies. Then spend only one third on new planned futures in your roadmap. This means if you have scaled your dev team to 15 developers, you will actually only have the equivalent of five full-time developers that can build new features and push the frontiers of your product. I have found the one third rule also takes care of the inefficiencies you will encounter when you increase your software developers, so it’s a fairly accurate rule of thumb. Of course, you can start with this rule, measure and adjust.
Additionally, you need to set aside cost to hire QA staff that can inject testing rigour into your release process. Regression testing, backward compatibility, UI, performance testing should all be part of your QA process before a product is signed off for release. If the version you are releasing is a major one, then I also advocate a controlled customer rollout (CCR). This is where you go through a gradual rollout of the product and allow it to be used in the field for a period of time before all customers are upgraded. So, what ratio do you need in terms of QA people to developers? Depends how buggy or complex the software is and how much effort you have put into automating your testing process. QA is another major subject really worth exploring further.
Monitor KPIs such as % coverage of unit test in new code and old code, as well as defect escape rate and make sure they are going in the right direction. Monitor the severity of bugs found during the full software life cycle and tag accordingly. As a rule, no software release to a customer should have any bug that is severe, i.e. a bug that stops the product from fully or partially working. If this happens, your QA process is in trouble. According to IBM, it costs 100 times more to fix a bug found in production rather than at the design/requirement stage before any code is written. Clearly the reputational cost is far more than the engineering cost to fix these defects, hence why paying attention to the quality of your software and sorting out your technical debt is so important if you want to scale your business and be a serious player in the market.
Technical debt is the reality of every scale-up CEO. Ignoring it is not an option, nor is stopping innovation. The only way to tackle this is head on by including the time and resource cost of this in your plans. In short, get on top of your technical debt before it gets on top of your business.
