30 Apr 2021
Data Insights: don't contribute to technical debt
Glenn Exton, Head of Data and Analytics at RBS International, discusses the causes of technical debt and how to prevent it.
Technical debt – or technology debt – relates to the cost, in both time and money, of additional maintenance required on a technical ‘quick fix’ to a problem, when a better solution that might have taken longer to develop would ultimately be easier to evolve, modify, repair and sustain in the long term. These ‘quick fixes’ could relate to any combination of software, infrastructure or hardware.
However, not all technical debt is bad. A team may spot a market opportunity where a quick fix results in significant revenue capture. If the eventual cost to pay back the technical debt is relatively low, this approach could lead to a profitable operating margin position.
The important first step is to understand what the technical debt is and the impact it may have. Letting what was a temporary fix become a permanent or long-term solution ultimately results in other major problems and expenses down the road. Much like regular debt in our lives, technical debt must be paid off at some point.
What are the major factors that contribute to technical debt?
You will typically find the following factors contributing to technical debt in any organisation:
Needing to pivot: Where the organisation needs to pivot fast – because of a competitive situation, market niche opportunity or natural disaster, for instance – Microsoft Excel may be used to quickly bring vast amounts of data together manually and, if manual-based practices are left to sustain and not addressed appropriately, this could increase future costs, time and inflexibility.
Servicing for ‘own needs’: Organisation-wide technical debt typically arises when multiple departments or teams commission work to produce data insights for themselves. This typically leads to duplicated datasets and disconnected databases and dashboards, making obtaining a single view of the data very challenging. Solutions or development teams are also part of the technical debt chain if they start developing solutions in isolation, without taking a wider view to see if something already exists within the organisation. Senior leaders need to take a holistic view across the organisation and beyond specific projects to avoid creating technical debt.
Project scoping and definition: Too often, technical debt arises at the inception stage of a project, when the scope is being finalised. Teams might be under time pressure so they commit too early to a direction of travel without looking at alternative scenarios.
The further away the change, data analytics or solutions teams are from the business area that is looking to solve a particular problem, the higher the chance of technical debt occurring
Solution design and development: The further away the change, data analytics or solutions teams are from the business area that is looking to solve a particular problem, the higher the chance of technical debt occurring. For instance, the team required to produce the data insights will be looking at the processes involved in sourcing, preparing, moving and storing the data for mining, analysis and presentation purposes; if the team tasked with creating the solution isn’t the one that will be using it, and takes shortcuts in any of these stages, it can lead to the accumulation of technical debt. Involving all the key people in development options early on will typically lead to more informed decisions.
Data-source quality issues: Continually producing new data insights based on sources that unknowingly have data-quality issues ultimately creates a complex chain to unravel and high costs to do so. The further away the team are from knowing the quality and lineage of the data sources, the higher the risk that the outcome will be fragile and of poor quality. This problem is compounded when other teams elsewhere in the organisation build their insights off that fragile base. The problem is further exacerbated for data scientists who rely on uniquely knowing the stability, provenance and frequency of change that may occur with data sources, as it can have a significant impact on the quality of data models.
Spreadsheet-based analytics: Spreadsheets are a wonderful tool for any organisation to manage data and produce insights. However, technical debt will most likely increase where an organisation primarily relies on spreadsheet-based analytics to yield insights from ever-growing data volumes, data types and data structure changes.
What are some obvious signs that technical debt is occurring?
We can all see financial debt on an account statement, but technical debt is not that obvious to spot. Here are some indicators to look out for:
- Strategists working on future operating models without having data architects or analysts embedded within the mixed discipline teams
- Service and business process design models are not being based on data insights or input from user research
- Customer journey design teams not including data architecture, data dictionaries and data glossaries in their approach
- Business intelligence reports or dashboards are produced with incorrect, missing or duplicated data or are not used, sitting idle without being shut down
- Solutions delivery teams are using unsupported and antiquated data sourcing, cleansing and analysis tools
- Data pipelines used to automatically source, load and transform data continually fail
- Data science models start to drift away from expectations or become erratic
The impact of technical debt
The impact to any organisation of not addressing or understanding technical debt is typically:
Poor agility: The ability to react to the pace of the business is handicapped when you’re forced to deal with repaying debt on a continual basis
Lost opportunities: Data is the backbone of any organisation, and if there is a lack of well-planned or integrated infrastructure to access data to make clear decisions in a timely fashion, everything downstream gets affected – from employees, to customer-facing activities, to revenue
Poor performance: Lots of one-off, short-term solutions make for fragmented systems that don’t integrate well, creating additional work and affecting both employee and system performance
Increased complexity: Temporary solutions that were devised as a means of convenience become increasingly more inconvenient as they pile on top of each other. Frustration rises, for example, when one part of the organisation can see a better way to serve customers, only to find that things can’t be done the way they should be due to technical debt not being resolved
Increased costs: The cost to pay back technical debt accumulates as data and analytical systems become more complex (increased data sources, data volumes, and added capabilities) as time goes on.
Inaccurate financial reporting: Working with siloed data sets, as can often happen in situations of technical debt, could result in data inaccuracies on a balance sheet where, for example, an asset is incorrectly classified and therefore incorrectly reported on. This could result in great costs to the business through impacted stakeholder confidence and inaccurate financial forecasts.
Less time to innovate: The more technical debt is allowed to accumulate, the less time and money is available to innovate for the benefits of the customers and stakeholders.
Technical debt is a tax on any organisation’s operating model, and leaving it to accumulate impacts morale, agility, flexibility, competitiveness and costs. Therefore, it is important to ensure a temporary fix does not become a permanent long-term solution. Think through the decisions you are about to take and ask yourself if you are about to create additional technical debt. If so, stop and see if there is another way to approach the problem you are trying to fix.
The UK Budget due out on 17 November is expected to bring significant fiscal tightening in a bid to restore the government’s financial credibility
As consolidation in the alternative fund market continues, general partners are looking for other means to fund their commitments.
In this edition the FCA’s new Consumer Duty, an update on LTAF, ESG investing, ‘side pocket’ proposals and regulatory horizon scanning.