How Not To Migrate Customer Data

So when The Bank acquired the Other Bank, they hired a bunch of contractors to migrate the data of Other Bank into The Bank's systems. Which they did. Sort of. Just how "sort of" you can judge by the way they treated customer ID's. A customer, Joe Bloggs, of Other Bank had an unique system ID number, say 1245678. When Joe's details were migrated, he was given a new unique ID in The Bank's systems (they had to do that, you can figure out why), say 7654287. 

Here's the thing. They didn't make up a table that said Joe Bloggs ID's: Other Bank = 1245678, The Bank = 7654287. So I can't link Joe's data before migration with his data after migration. In other words, he may as well have become a brand new customer a few months ago. Re-read those sentences until you realise how monumentally stupid that was, and keep going until you understand that to make that mistake, those contractors and the people guiding them must have almost no understanding of how data works. Now understand that almost everyone who works in bank IT and management is that, errrr, challenged, and you will not be surprised when the next disaster happens. Those people actually don't understand the industry they are working in.

By the way, if you don't get what the issue is, and you work with data, please choose another career. Now. Before you do any serious damage somewhere.

By searching around the wilderness that is The Bank's database set-up, and by exporting 6.5m records from the mainframe to my work laptop where I could perform some simple string manipulations, and then re-import that data, I got round the problem, at least for my product. Yes, you read me telling you I had to export the data to my company laptop. WTF? Because the mainframe DBMS doesn't have any kind of serious string editing, and if it has a companion programming language - as Access has VBA, which was what I used - The Bank doesn't have it. And do not ask how painful it is to process 6.5m strings in Access - if you have done it, you will know, and if you haven't you wouldn't even begin to believe what happens.

There are days when I get scared that these problems get solved by me. I'm not supposed to be the smart guy. This is a HUGE FREAKING CLEARING BANK. It is as important to the functioning of the economy as Thames Water or British Gas. It's not a nice-to-have company - like Apple - that could vanish overnight and we could all go on having hospital operations and getting to work. An organisation like that is supposed to have serious engineers and designers running it, not a bunch of people who make mistakes so dumb even I wouldn't make them.

I'm supposed to be helping our creative types with data-driven insights and what I really do for a living is make up for the deficiencies of a string of IT and DBA contractors, and by extension, the organisation's managers who don't understand why this stuff matters and how to manage the functions that should deal with it.

I do know one thing: it's even worse in the USA. Way, way, way worse.