Why Software Projects In Finance Fail; Or, I Really Do Hate Your Guts

Written by Jamie Monday, 02 January 2012 10:47

Software professionals usually can’t stand financial engineers. It’s not just that their code is bad and their manner strange, they are also pompous and often arrogant. The feeling of course is utterly mutual but in return the financial engineer throws derision into his hatred: he knows it’s harder to become a financial engineer than a software engineer and he knows he earns more.

The problem with financial and software engineers is that, like warring cousins, they are too close to each other. The Scottish might hate the English – and the feeling may be mutual - but they’d find it harder to muster up any hatred for a tribe deep in the Amazon basin. It’s the English’s proximity – and their long history of colonisation – that does it for the Scots.

Software and financial engineers might be close enough to each other to fuel mutual contempt, but they are far enough apart to not be able to do each other’s jobs. This, I am now contemplating, is one root of the seemingly endless stream of failed software projects in the finance sector. 

Most companies I’ve worked for realise that they are now software companies. They don’t think they are car rental companies, they don’t think they are airlines, nor do they think they are travel agents. They know they are software companies that happen to rent cars, happen to sell plane tickets or happen to sell package holidays. Finance companies, however, have not quite figured out that they are software companies who happen to price derivatives or sell mortgages. Because of this, they see software as an annoyance, a necessary evil, a thing they don’t want to cede any more ground to.  And because the financial engineer can write SQL scripts and solve problems in VBA, he never has a clear and present need for the software engineer or real software engineering. Ah! the software engineer thinks, but we could make the financial engineer’s life so much easier by taking the leg work out of their analysis. Ha! the financial engineer thinks, like the last time you totally screwed up my commodity pricer? Now fuck off.

It’s true. We - software engineers - always make a mess of software projects in the finance sector. Why? because we are too close to financial engineers and we actually think that we understand what they are doing. But we don’t. And so we fuck things up.

Coming back to the examples from earlier, most of us know or can quickly learn the rules for car rental or air travel. The business processes are hardly more sophisticated than high-school accounts: we add VAT, we sum extras, such as the hiring of a GPS system, and we do the occasional sanity check, such as making sure the customer hasn’t tried to hire a car in the past. Each day, when developing these systems, the business user or analyst learns a little bit about software development, usually testing and specifying problems clearly. And each day the developers learn a little bit about their domain.  This never happens on a project with a financial element. Understanding the time-value of money is hard.  Understanding the time until the next interest payment and its effects on a calculation, the compounding of interest, the pricing of commodities, interest rate fluxes, etc., are all so hard that many financial engineers don’t even understand them. The poor programmer has no chance. So where other teams succeed because the people on them progressively close the gap between their skills, on a project with a large amount of finance in it, the gap remains large.

What this means, and this is why I absolutely love building software systems in the finance sector, is that you have to rely on the good old fashioned techniques. You really do have to manage your projects, else the team will pull itself apart. You really do have to use object-technology (the financial engineer’s implementation, because he can’t program, is never more than a prototype). You really do have to test, and use test rigs, and oracles, and do code reviews.

Mutual hatred, the difficulty of the domain and the finance industry’s dedication to the most dysfunctional organisational structures and management techniques known to man are all reasons that projects fail. But the software engineers have to take responsibility, too. Their inability to navigate the intellectual as well as the organizational landscape leads to a hatred of the financial sector. This hatred comes from a lack of understanding. Until we all get over this, financial and software engineers will continue to be uncomfortable bed-fellows.