How Not To Migrate Customer Data

So when The Bank acquired the Other Bank, they hired a bunch of contractors to migrate the data of Other Bank into The Bank's systems. Which they did. Sort of. Just how "sort of" you can judge by the way they treated customer ID's. A customer, Joe Bloggs, of Other Bank had an unique system ID number, say 1245678. When Joe's details were migrated, he was given a new unique ID in The Bank's systems (they had to do that, you can figure out why), say 7654287. 

Here's the thing. They didn't make up a table that said Joe Bloggs ID's: Other Bank = 1245678, The Bank = 7654287. So I can't link Joe's data before migration with his data after migration. In other words, he may as well have become a brand new customer a few months ago. Re-read those sentences until you realise how monumentally stupid that was, and keep going until you understand that to make that mistake, those contractors and the people guiding them must have almost no understanding of how data works. Now understand that almost everyone who works in bank IT and management is that, errrr, challenged, and you will not be surprised when the next disaster happens. Those people actually don't understand the industry they are working in.

By the way, if you don't get what the issue is, and you work with data, please choose another career. Now. Before you do any serious damage somewhere.

By searching around the wilderness that is The Bank's database set-up, and by exporting 6.5m records from the mainframe to my work laptop where I could perform some simple string manipulations, and then re-import that data, I got round the problem, at least for my product. Yes, you read me telling you I had to export the data to my company laptop. WTF? Because the mainframe DBMS doesn't have any kind of serious string editing, and if it has a companion programming language - as Access has VBA, which was what I used - The Bank doesn't have it. And do not ask how painful it is to process 6.5m strings in Access - if you have done it, you will know, and if you haven't you wouldn't even begin to believe what happens.

There are days when I get scared that these problems get solved by me. I'm not supposed to be the smart guy. This is a HUGE FREAKING CLEARING BANK. It is as important to the functioning of the economy as Thames Water or British Gas. It's not a nice-to-have company - like Apple - that could vanish overnight and we could all go on having hospital operations and getting to work. An organisation like that is supposed to have serious engineers and designers running it, not a bunch of people who make mistakes so dumb even I wouldn't make them.

I'm supposed to be helping our creative types with data-driven insights and what I really do for a living is make up for the deficiencies of a string of IT and DBA contractors, and by extension, the organisation's managers who don't understand why this stuff matters and how to manage the functions that should deal with it.

I do know one thing: it's even worse in the USA. Way, way, way worse.

How Was Your January 2012?

 January was horrible. It was so bad I even went to the Quickstep meeting in Islington a couple of times – actually it’s a nice group and I will be going back. It was grey, it was cold, it was wet, it was scuttling through the back streets to get to Blackfriars Bridge and hoping I could get a bus to Liverpool Street rather than have to travel on the ram-jammed Central Line.

I spent a lot of the month wading through some philosophy of science in a book called Cosmopolitics, by Isabelle Stengers, a Belgian chemist-turned-philosopher of science who was a pupil of Iliya Prigogine. It’s written in that fluent post-modern French style that’s as slippery as a fish – every time I think I get what she’s saying, it slips away from me. It’s in two volumes, and it feels like she discusses a lot, but when I try to summarise what I think she’s said… nothing happens. This is not a satisfying reading experience. Sartre’s Being and Nothingness is by contrast snappy and to the point.

Another chunk went on trying to understand the Riemann-Roch theorem. Not the modern proofs, which aren’t that difficult, or the ideas that go into the proofs, which I suspect are simpler than all the formal frippery make out. I want to understand why it should be true – because when you take a really naive approach to estimating the dimension of a function with n poles and m zeros of varying degrees, you don’t wind up subtracting the degrees of the poles from the degrees of the zeros. How on earth did Riemann think of it? Nowhere, and I mean nowhere, will you find an intuitive explanation of this theorem, let alone why the original proof used Dirichelt’s principle, which is about solutions to the Poisson equation! Index theorems – which R-R is an example of – are deeply tied to PDE’s – the Atayah-Singer theorem uses the Dirac equation (!) – but you’d never guess it from the usual literature. This is frustrating, especially as the project of the series is to provide just such an explanation.

Another chunk went on being not-quite-right in the body and mind. One Wednesday morning in mid-January, I swear I nearly turned round, went home and called in with ME. I had practically no will to tackle the world. Let alone the daily journey to Liverpool  Street. The previous evening I had actually dismounted from the bike twenty minutes early in my spin class – I didn’t do that when I first started. I had just Lost It. I took lots of early evenings and did little gym. I took a long weekend at the end of the month, and started to feel stronger and more able to take on the world.

A lot of time went on two hefty – by my standards – projects at work. One of them was a “despite” project – one you get done despite the tool set and data you have, not because of it (if you don’t know what that means, get a job in a retail bank). The other was / is a report where the presentation is as important as the data – long story. Day after day went past with no visible progress and a lot of effort. I solved one of the problems recently and can concentrate on the next one.

Whatever it was I got before Christmas was more than just the cold and fever that laid me out over the holiday week. It’s made my breathing rough – I can barely breathe well enough to run on the treadmill for more than a mile. Some of it was cold and fever, and some of it is something else.

Soon after I wake up, I think “I can’t go on doing this”. It passes, but it’s real. I don’t like the City – like all west Londoners, I get hives if I have to travel East of the Kingsway for any length of time. I read some research that found that rooms with low ceilings are good for doing transactional, routine tasks, but to do any thinking you need higher ceilings. The ceilings in our new office feel about as high as the galleys of the Cutty Sark.  Some of me is prepared to get used to it, but the rest of me wants nothing to do with it. There’s the real conflict. I haven't felt entirely fit and well since we moved there.

Accounting Ratios Are For Accountants, Not Insight

One of the many variables our accountants track is the average weighted selling price (AWSP): the loan-amount-weighted-APR. This varies slightly each week even in the most stable of times, especially with a pricing engine as sensitive to as many customer-centric variables as The Bank's, but no-one has a good idea of how much variation is just "BAU" ('business as usual") and when to start wondering if there's something going on on the outside, Mr Jones.

I'm wondering if we should even be tracking these kinds of accounting ratios at all. The AWSP and the ALA (average loan amount) are more-or-less inversely related: put up the APR and the amount borrowed goes down. (Who knew?) But when you look at what happens, it's not what you think that happens. You think that when we put the prices up by 10%, almost everyone knocks 10% off the amount they borrow. Not so much. In fact, not at all. Very few people take loans for some random amount like £5,237: they take loans of £4,000 or £5,000. The distribution of loans by amount is not normal, but has spikes at the round thousands. Almost no-one takes a loan for £18,000 and a fair proportion take loans of £2,000. And this is true no matter what the prices are.

I'll just say that again, because I can guarantee you didn't get it. You can change the prices all you want, and the peaks will always be at round thousands and multiples of six months in the loan terms. What really happens when we put prices up? A 10% in increase in APR from 10% to 11% on a £10,000 loan over 60 months makes a difference of £5 a month to the repayments. Most people can afford that, if they can afford the original amount in the first place. That's one reason why things don't change much for a building society with a fairly simple rate structure when it bumps up its APR's. However, the pricing models The Bank uses don't usually impose a constant increase on everyone. The changes are obtained by changing propensity curves which can make some hefty differences to some people and very little to others. You might have thought you were going to pay £220 a month, but actually it turns out that now you're going to pay £260. That's a week's petrol for the car for a lot of people. Decision time.

Many other people will be unaffected. I think (because we don't have research - yes, I said: "we don't have research") that some people make a big adjustment - instead of buying a second-hand latest-model car, they buy a second-hand previous-model car that's perhaps only six months older, because that knocks at least £1,000 off the price of even small cars (and that's my Top Tip for saving money when buying second-hand cars) - while some people abandon the project altogether because it's not the sort of project you can save £500 or £1,000 on.

Over half the personal loans made every day are to re-finance other debt, and maybe some people, faced with a higher price for the whole lot, go back one more time to the Bank of Mom and Dad to ask for help on the last £1,000. There's undoubtedly a rounding effect in the intial amount. The project (new TV, new kitchen, wedding, new floors) might cost, say, £8,250 and many people will round up to £9,000 so they can get the TV / holiday / new sheets / some other bunch of stuff they've been putting off for ages, as well. When we put the price up on them, they shrug, borrow £8,000 and fund the last £250 from a credit card or cash flow.

Take 2,000 of these random decisions a week and it smooths out. So the accounting ratios paint one picture, while the detailed MI shows something else. Accounting figures are accurate, but don't provide insight: MI is often slightly inaccurate, but provides insight. (Inaccuracy is fine, as long as it isn't misleading.)

I've been careful to call the AWSP and ALA "accounting ratios" rather than "statistics". Technically, accounting ratios are statistics (symmetric functions of observed values of random variables). Emotionally, they aren't. I have a prejudice that a "statistic" ought to do two things: it should tell me about the distribution, and it should be about something physically or commercially meaningful. AWSP sounds as if it should be commercially meaningful, but I'm no so sure. Because loans are about money, and you think all money is the same, I'll use a different example.

A record label could report the "average price per CD" (APCD). Under certain circumstances, it seems that might be a meaningful indicator. When it changed, you would still need to find out why. Did less people buy from "new release premium" label, or more people from the "new release budget" label, and why? No hot new releases this month? Was one of the budget releases was favourably reviewed on Radio Three? Perhaps the month's hot new release was on a competing label, so no-one bought much from yours. Or maybe the online server went down for two days, and since that's where your premium sales are, that's why the APCD dropped. Or maybe Amazon are making a point to you, which you've now understood. Or maybe a hundred other things because that's what business is: a thousand details that don't make a complete picture.

The accountants won't stop reporting the ratios, but the MI team, if there is one, should start reporting something more useful: the details of the product / price mix and how that changes. Or whatever other mix variables you need to look at to explain the make-up of some critical line.

Resolutions For My Participation In Social Media

I joined Facebook a while ago and haven't been able to rest easy about the whole business of social networks since. Okay, I'm exaggerating. But I don't really get Facebook - maybe because I don't get that sharing stuff you're doing with others is a pleasure for you and them. My life is so based on work-gym-sleep routine and cultural consumption all before a ten o'clock bedtime that I can guarantee it's not interesting to other people. I hope what I think is, but I'm pretty sure what I do isn't.  So the whole "what am I doing on Facebook" thing turned into "am I making use of and contributing to the right sites on the Internet?"

There are two questions. Where do I need to be? And where do I want to be? "Need" is easy. I need to be on whatever industry networking site is actually used by people to hire people who do what I do. There isn't a site specifically for Insight / Pricing Analyst / Managers, though there are agencies that do a certain amount of specialising, but if there was, I would join that. Instead I'm on the general-purpose site for People In Suits, LinkedIn.

Where do I want to be? Put that way, the answer starts to be obvious. I want to be where there's stuff I'm interested in. That's music, attractive women, philosophy, mathematics, holidays, photography and other such. A lot of that is in blog-land and my contribution is by reading. I know the etiquette says I should leave comments, but I can rarely think of anything to say at the time.

Away from the blogs, we have... Amazon. This counts as a social media site: it's about the reviews. I review under my "Real Name". Books are one of my things, so this makes sense.

Other Review Sites: I'm thinking of TripAdvisor here. Google an holiday destination and TripAdvisor comes up. I do feel I should contribute, but can't get round to it. I'm very good at enjoying stuff, and taking photographs of it, but not so good at describing it.

Wikipedia. Perhaps the ultimate social content site: I have two articles from my time in telecommunications on it, under my real name. That's all. Maybe when I have loads of unpaid time on my hands, I'll do more.

Blogging. Yes. Here already. Long form, however. I don't, as you'll have noticed by now, micro-blog, so Tumblr is out. I'd like to make cool, laconic comments under equally cool photographs, but I'm a philosopher, not a fine arts grad. 

Photography. I'm not a pro, I don't do those polished Photoshop-ed pro-am photos that Flickr and deviantArt, amongst many others, are full of. It's not my style. And it strikes me that getting work up to that look, cataloging and tagging it, is a ton of work my diary doesn't have the space for. 

Guilty Feelings. No, that's not a site. I use You Tube so much, I feel I should give something back, but I'm not sure what. Or even how. I feel the same way about TripAdvisor. Google somewhere you're thinking of for a holiday and up this pops. I love the idea of 8Tracks, but the people there seem to have scarily awesome taste in music and accompanying art, and I'm suitably scared.

Forums. Like everyone else I use various forums, mostly for technical computing problems. I'm not good enough to give back in that area, and there are no forums for in-house business analysts (though there is for quants, though it's more of a career board than a how-to). This is because most in-house business analysts have a narrow and shallow transferable skill-set - what they are about is the use of in-house data, and discussing the family secrets with strangers is frowned upon.

Almost lastly, the Facebook Question. (Substitute your local social networking site for China / South America etc). I don't have a personal life - I commute, work, go to the gym, watch movies and read, write, think and sleep. What's my status update going to look like? "seven dials is... trying to calculate the first cohomology of a circle using only short exact sequences"? Yeah. Right. I'm on Facebook as so many are, to see who else is. In the spirit of reciprocity I should have some minimal, PR-friendly information about myself in case some HR department or recruitment agent goes snooping. I have the odd glimpse as to how one's page could be turned into some kind of ironic artwork, though I'm guessing someone else has already got there.

So here are my immediate resolutions. First, I will write at least one Amazon review a month from now on. Second, I will write a couple of reviews on Trip Advisor. Third, I will make up three playlists and attempt to put them on 8Tracks. Fourth, I will investigate what's involved in posting something to You Tube - if I can find something that no-one has already posted. That should get me started.

Derek Parfitt's "On What Matters"

Derek Parfitt has at last published a two-volume meisterwerk called On What Matters. It's been over a decade in the making and discussed with almost everyone on the planet who cares about what he thinks matters. I was one of the many who beat their heads against the prose of Reasons and Persons, all the while wondering why, and I'm not going to do it again with On What Matters. And here's why...

He's asking and answering the wrong questions.

When I was young and knew little of the world, I thought of moral philosophy in abstract terms. I wanted principles, if not a super-principle, by which to judge and act. Well, judge, anyway. I didn't want to make decisions, which imply responsibility and the possibility of error, with all the attendent apologies, amends, revisions and starting over, and the possibility that I'd get it wrong again. What I wanted was apriori principles on which I could act, and if it didn't work out, it would be the world's fault, not mine. More people than you might think still want something like this. Anyone looking for fundamental principles surely does.

The issue isn't whether there is such a fundamental principle. The issue is what difference it would make if there was. The answer is that it makes no difference at all. Because it is high-level, and presumably can be stated in twenty-five words or less, you will need to interpret it in any situation. Unless it is the only general-purpose principle with an utterly unambiguous interpetation in any situation, other people may disagree with your interpretation and consequent actions. They will have and equally strong conviction that their interpretation is correct. You are both agreeing on the principle, but you disagree on what it means in a specific circumstance. Which means you may as well adopt a bunch of less ambiguous lower-level principles, rather than one necessarily ambiguous high-level one. This ambiguity is not something unique to high-level moral principles: there's more than way of applying Newton's Third Law to the orbit of a satellite.

While all this sort of thing is very interesting, at least to philosophers, it's not much use to anyone interested in the question "how should I live", let alone "how do I run this freaking country"? (It's worth noticing that the more freedom we give to answering the first question, the harder it is to answer the second.)

What's often overlooked is that the first question is always asked in a context: by someone who is already placed in an existant society with laws, rules, expectations, manners and etiquette. The first draft answer to "how should I live?" is "as much of an exemplary a member of the society in which I find myself as I can manage". The problems start when I am not a native of that society and have, perhaps from being raised somewhere else, perhaps from having that personality disorder known as an "independent mind", my own ideas about what is acceptable, or find myself having an unexplained but forceful emotional reaction to some practice of that society. "I can't do that, I don't care if I'm supposed to, I can't." We are not born moral tabula rasa, but with, perhaps never-to-be-triggered, ideas of what they will and won't tolerate or do, which appear not as intellectual theories, but as deep-seated emotional reactions. (Not many people are like this: the majority will go along with whatever they find themselves born into.)

We have by now left the realm of detached ethical shoulds and oughts, and entered the prudential and pragmatic world of contracts, argeements, deals, expectations, manipulations, promises, compromises, pay-offs and other assorted rewards and punishments. The question "how should I live" now becomes "how much of this stuff can I accept, how much can I organise my life so it doesn't affect me, and how much am I going to have to take some kind of stand, or accept that there are some jobs I won't get, some parties I won't be invited to, some people who won't be in my circle, and some people who I will sincerly wish were elsewhere but my vicinity." I'm not sure about you, but that sounds more like my real life.

Parfitt knows that a steller reputation as an abstract thinker is not gained by getting down and dirty in the world of prudence and pragmatism. He's arguing abstruse points about high-level moral theories - utilitarianism, Kantian universalism and a social contract theory - that no practical person who makes decisions (judges, jurors, politicians, managers and directors, doctors, administrators) has ever used. If you need a high-level abstract principle to help solve a problem, you're solving the wrong problem.

Moral philosophy has long been politics by other means. Kant wanted to set up a universal moral standard so that there would be no authoritative role for the Church. Utilitarianism was a doctrine that fitted well with the mood and aims of Victorian social reformers. Consequentialism goes well with pragmatic people who deal with the world case-by-case and don't want grand principles and political programmes. As a fallibalist, I don't believe we can know what's right, but I'm pretty sure we can spot what needs to stop happening and shouldn't spend our tax revenues on. We don't need a theory of right and wrong to stop bankers hyping another bunch of useless products, or the pharamceutical industry from corrupting every third doctor and medical researcher into endorsing expensive, barely-effective drugs with more side-effects than e-coli.

If I had to pick one question that I think "matters" above all, it's this: at what point must we abandon the pursuit of our personal goals and be prepared to sacrifice our advancement so that the rest of the community can benefit? The obvious questions here are: how much profit is enough? How big a bonus is too much? Is it really acceptable to collect expensive vintage cars: isn't there something better to do with the money? Do we draw the line at sending jobs to poor countries? Should we donate to charities if we believe that they are ineffective and spend too little of their income on the cause? Notice that our personal goals may well already include charitable and welfare aims, and we may also live a modest life with no great riches or prestige. We may even be mired in debts. Does poverty, or the prospect of sustained unemployment, excuse us for not blowing the whistle on our employer's mal-practices? Does the continuous need for a salary mean that we should be complicit in selling useless and meritricious products and services?

None of these questions, you will notice, have anything to do with runaway trolleys. There are plenty of simple but moral problems in the daily world - should we give money to a beggar if we suspect that they are going to spend it on drugs? - without using toy examples. So maybe if I think there is one thing that "matters" it's the examination of the moral world created by post-modern capitalism and sprawling States that take forty per cent of our income in direct and indirect taxation and still can't keep our hospitals clean and teach children to read, write and show up on time. With a smile. That's what matters.

The Proust Questionnaire

1. What is your idea of perfect happiness?

 Blue sky, clear air, sunshine, beaches, sea, never having to worry about money or organising anything

2. What is your greatest fear?

 That my body goes on living longer than I can afford.

3. What is the trait you most deplore in yourself?

 Low energy levels and crippling self-doubt.

4. What is the trait you most deplore in others?

 When they eat smelly food on trains.

5. Which living person do you most admire?

 Terence Tao.

6. What is your greatest extravagance?

 I don't have enough money to be extravagant.

7. What is your current state of mind?

 Tired. Always.

8. What do you consider the most overrated virtue?

 Whose list of virtues did you have in mind?

9. On what occasion do you lie?

 If I don't trust you.

10. What do you most dislike about your appearance?

 The bit where I don't look like me.

11. Which living person do you most despise?

  Anyone who thinks they can leave their morals behind when they swipe in at work.

12. What is the quality you most like in a man?

 I don't really "like" men.

13. What is the quality you most like in a woman?

 That they are polite enough to flirt with me.

14. Which words or phrases do you most overuse?

 'I think'

15. What or who is the greatest love of your life?

 I don't do oxytocin

16. When and where were you happiest?

  You're making an assumption there...

17. Which talent would you most like to have?

 Energy and lack of doubt

18. If you could change one thing about yourself, what would it be?

 More energy, no self-doubt

19. What do you consider your greatest achievement?

 For a couple of years, I was possibly the best in the world at the job I did. Maybe only a couple of hundred people in the world did it, but that's enough. 

20. If you were to die and come back as a person or a thing, what would it be?

 Oh shit. I have to do this again?

21. Where would you most like to live? 

 See 1.

22. What is your most treasured possession?

 My health. When it's around.

23. What do you regard as the lowest depth of misery?

 Drunk self-pity.

24. What is your favorite occupation?

 Wait. I can do things I like doing? When did that become an option?

25. What is your most marked characteristic?

 I don't drink, drug or use mood-altering chemicals.

26. What do you most value in your friends?

 That they're still here.

27. Who are your favorite writers?

 I steal from them all.

28. Who is your hero of fiction?

Philip Marlowe

29. Which historical figure do you most identify with?

 Socrates

30. Who are your heroes in real life?

 Socrates, Paul Feyerabend, Craig Murray and Groucho Marx

31. What are your favorite names?

 Girls' names that sound like thick black hair and luscious lips: Julia, Rachel, Sarah

32. What is it that you most dislike?

 All the seats being taken already.

33. What is your greatest regret?

 All the women I never made love to.

34. How would you like to die?

 Quickly, and soon after I next lose my day job.

35. What is your motto?

 One day at a time.

In A Hotel Garden, Somerset

When I have the name of this hotel, I'll put it in. I stopped here for lunch on my way down to Somerset a couple of years ago. The food was good, the hotel plush and the grounds quite interesting. The weather was bad. Arriving on the Quantocks, I went for a walk in thick mist, because, well I had to stretch my legs.