What Brainstorming Really Was

There's a fascinating article in The New Yorker  about how valuable being able meet other random smart people is for generating good ideas - that's my take on the description of Building 20. Explains a lot about why no good ideas ever come out of government or institutional British companies (neither smart, nor random nor able to meet).

What I didn't know was that "brainstorming" - the most bogus technique for idea generation ever - was invented, or described, by a Mad Man, Alex Osborn, a partner in the advertising agency B.B.D.O in the 1940's. The notion took off and has been blighting creative effort ever since.

He made the story up. Of course he did. It's what Mad Men do. He couldn't tell his readers the various truths about having good ideas. Who wants to know that good ideas come to you out of nowhere after you've been producing utter rubbish for weeks on end? Or from a random conversation with someone you met on a train, or from a song lyric, which meant something to you because you were soaked in the problem, but nothing to the person sitting next to you, because they weren't? And was he going to tell you what his "brainstorming sessions" really were?

Picture it. A bunch of the guys sitting around, relaxed enough not to be worried about sounding silly, throwing out any old bullshit that occurs to them. What does that sound like? Yep. I missed out the cigarette smoke, the drinks and the leatherette seating. They were in the bar after work, sending up each other's ideas, parodying the clients, making silly and probably obscene remarks about how to use the client's products, and suddenly... the copywriter scribbles something down and vows to use it next morning just as soon as he's had enough aspirin to ease the hangover.

Alex Osborn couldn't describe that. So he re-located it, took away the booze, cleaned up the dialogue and called it "brainstorming". How  else could it have been?

Why I Can Stand On Your Toes If I Want To

A few years ago now, I was walking across a bridge in Amsterdam with my friend. A boat passed underneath, carrying what looked like celebrating medical students. I thought: "it isn't my world any more, it's theirs". 

It took me a while to understand that feeling properly, and some of it isn't pretty.

When I watched those medical students waving their drinks around, what I really felt was that the deserved self-centredness of older folk: you young people can look after the frigging planet now, we have enough on our hands just staying employed, right-weight and fed. You need to have been through the Wasteland Years from forty to sixty (I'm not quite out yet) to understand this: but when you have, you will know that you've paid all the dues you ever needed to.

Twenty years of showing up, standing your ground, fighting for your space and light, dealing with unemployment, finding jobs, gaining and losing relationships, all when the hormonal fires are going out is an endurance test enough. It breaks some people, either financially (divorce, bankruptcy, sustained unemployment), spiritually (too many compromises on the way up the slippery pole or for the sake of the marriage and kids) or physically (just look around you). Of course, for some people it's a wonderful time, but they live somewhere else I've never been.

So when I left the world to those students, I was also reserving the right to be as selfish, or self-interested, as I wanted or needed to be, to get through the remaining time. It's what a lot of older people do.

We get focused on our issues. We don't care about education, because our kids have grown up. We don't care about the planet, because we're not going to be around a lot longer. We do care about pensions and paying bills, because we're shortly to be out of work and faced with maybe twenty years of living. Those pensions and allowances cost younger generations money. Our lives contract as our energy levels decrease and our capacity for finding stuff exciting and rewarding diminishes (except for addicts and nerds, for whom The High from drugs, women, booze, music, movies, solving tough problems and putting one over on the enemy never goes away. This explains me).

My generation will cancel, or cause to be cancelled, the retirement age, because lots of us have worthless pensions wrecked by bouts of unemployment, and we will establish, or cause to be established, a de facto right-to-die based on the current idea of an advance decision. We are going to be short of money, and are threatened with a long and decaying old age of poverty. Not going to be popular with a generation who were used to being fit, present and able to spend.

Pete Townsend wanted to die before he got old because he didn't want to be helpless, compromised, poor, patronised by nurses and social workers, and a nuisance to his children. He's none of those things because the world has changed, and he's a millionaire. I'm none of those things either, and I'm not a millionaire. I still want to die before I get old, and can't find paying work.

And in the meantime, I've paid my dues and I don't owe anybody anything except good manners, settling my debts,  and a day's work for a day's pay, which are the courtesies of everyday life. It's just that now, you have to be more thoughtful to me than I do to you, because I'm the old guy with grey hair.

And if you're not, I'm going to stand on your toes.

First Thoughts On Insight Analysis

Look at this year's job title and it says something about "Insight Analyst". If you've never run across one of those, it's not surprising: they only roam in companies with huge, and I mean millions and even billions of records, databases of customer data. It's what used to be called "desk research" but with mainframe computers: the buzz-name is "big data".

The insight agencies - such as Dunhumby, Tesco's in-house analyst - like to claim the value of their contributions. What they contribute is usually tweaks to the exact mix of coupons sent to a more refined mailing list. Given the costs of direct mail, that may be worth doing. Google and Facebook are basically selling their insight technology: they are offering the world's best targeted advertising. Tesco, Sainsburys and other supermarkets with loyaly cards can tie purchases to people, and banks can look at your current account transactions. Hint: if you want to hide your patterns of consumption from financial services companies, pay by cash. 

As ever, I like to contrast Amazon with the The Bank. Amazon use an insight approach when they make suggestions based on "people who bought this also bought these". It works well with genres - a textbook on Galois Theory, post-rock music, romantic novels or cookbooks - and it may work with accessories for bigger-ticket items, but outside well-defined genres, where customer volumes are small, it can get bizarre. With volume comes consistency and reliability, but also blandness: large numbers of people who bought Katy Perry also bought Ke$ha. I'd never have guessed. A mathematics publisher won't need telling that people  who bought Hartshorne also bought Hatcher, and Mumford's Red Book, but it's only news to outsiders. Which is exactly who it's aimed at: consumers like you and me who are browsing. I think the Amazon algorithms are fine, and the occasional bizarre suggestions just confirm I'm in a minority. What makes Amazon so effective aren't clever algorithms - though that helps. Amazon's real advantage is that it controls its data: you tell it what you bought, using the references that it supplies. Retailers do the same with bar codes. 

Not so much in retail banking. Retail bankers and their IT people aren't as aware of the advantages of standards as their counterparts in telecoms. As a result, a bank can identify the method of payment, but not what it was for. Reliably. That £254.32 you give to Sainsburys each month? We don't know if it's for food, a loan repayment, savings, a credit card payment or whatever else. Banking developed in the days when retailers generally did one thing - except the Co-op and Woolworth, and everyone paid them with cash. The industry never recognised a need for standards in transaction description.

As a result, when I took over a report that claimed to tell us how many of our customers were taking loans with other lenders, I found more ambiguities and estimates than had been advertised. And I can't resolve them. Nor can a room full of analysts, because we don't have the data. Big Brother may be watching you, but his glasses are steamed up, and he doesn't know what he's looking at half the time. (Are you convinced by those grainy CCTV photos they show on the news? Not me.) 

Messy data in, sloppy conclusions out. I discovered recently that people were using an earlier an even messier version of this data to calibrate pricing models. I have no shares in The Bank for  reason.

Insight analysis isn't management information, which is itself a step down from financial reporting. As long as the segment my analysis tells me should be large and spread over the country doesn't turn out to consist of two hundred people in Guernsey, I'm okay. Does it matter if it's really 4.5m people instead of 2.5m or 6.5m? As long as we start small with the ability to scale up quickly? The point is to spot a genuine segment, and it may not even be that. The point is to produce a product or service that makes money and people come back to buy. Does it really matter how precise the process is? Hell, I'd be happy if I designed the product for one group of poeple who ignored it completely but it was taken up by another group to the volumes we were hoping for. The onyl issue there is that I may not be able to repeat the fluke.

But if the success is big enough, I don't need to.

How Not To Migrate Customer Data

So when The Bank acquired the Other Bank, they hired a bunch of contractors to migrate the data of Other Bank into The Bank's systems. Which they did. Sort of. Just how "sort of" you can judge by the way they treated customer ID's. A customer, Joe Bloggs, of Other Bank had an unique system ID number, say 1245678. When Joe's details were migrated, he was given a new unique ID in The Bank's systems (they had to do that, you can figure out why), say 7654287. 

Here's the thing. They didn't make up a table that said Joe Bloggs ID's: Other Bank = 1245678, The Bank = 7654287. So I can't link Joe's data before migration with his data after migration. In other words, he may as well have become a brand new customer a few months ago. Re-read those sentences until you realise how monumentally stupid that was, and keep going until you understand that to make that mistake, those contractors and the people guiding them must have almost no understanding of how data works. Now understand that almost everyone who works in bank IT and management is that, errrr, challenged, and you will not be surprised when the next disaster happens. Those people actually don't understand the industry they are working in.

By the way, if you don't get what the issue is, and you work with data, please choose another career. Now. Before you do any serious damage somewhere.

By searching around the wilderness that is The Bank's database set-up, and by exporting 6.5m records from the mainframe to my work laptop where I could perform some simple string manipulations, and then re-import that data, I got round the problem, at least for my product. Yes, you read me telling you I had to export the data to my company laptop. WTF? Because the mainframe DBMS doesn't have any kind of serious string editing, and if it has a companion programming language - as Access has VBA, which was what I used - The Bank doesn't have it. And do not ask how painful it is to process 6.5m strings in Access - if you have done it, you will know, and if you haven't you wouldn't even begin to believe what happens.

There are days when I get scared that these problems get solved by me. I'm not supposed to be the smart guy. This is a HUGE FREAKING CLEARING BANK. It is as important to the functioning of the economy as Thames Water or British Gas. It's not a nice-to-have company - like Apple - that could vanish overnight and we could all go on having hospital operations and getting to work. An organisation like that is supposed to have serious engineers and designers running it, not a bunch of people who make mistakes so dumb even I wouldn't make them.

I'm supposed to be helping our creative types with data-driven insights and what I really do for a living is make up for the deficiencies of a string of IT and DBA contractors, and by extension, the organisation's managers who don't understand why this stuff matters and how to manage the functions that should deal with it.

I do know one thing: it's even worse in the USA. Way, way, way worse.

How Was Your January 2012?

 January was horrible. It was so bad I even went to the Quickstep meeting in Islington a couple of times – actually it’s a nice group and I will be going back. It was grey, it was cold, it was wet, it was scuttling through the back streets to get to Blackfriars Bridge and hoping I could get a bus to Liverpool Street rather than have to travel on the ram-jammed Central Line.

I spent a lot of the month wading through some philosophy of science in a book called Cosmopolitics, by Isabelle Stengers, a Belgian chemist-turned-philosopher of science who was a pupil of Iliya Prigogine. It’s written in that fluent post-modern French style that’s as slippery as a fish – every time I think I get what she’s saying, it slips away from me. It’s in two volumes, and it feels like she discusses a lot, but when I try to summarise what I think she’s said… nothing happens. This is not a satisfying reading experience. Sartre’s Being and Nothingness is by contrast snappy and to the point.

Another chunk went on trying to understand the Riemann-Roch theorem. Not the modern proofs, which aren’t that difficult, or the ideas that go into the proofs, which I suspect are simpler than all the formal frippery make out. I want to understand why it should be true – because when you take a really naive approach to estimating the dimension of a function with n poles and m zeros of varying degrees, you don’t wind up subtracting the degrees of the poles from the degrees of the zeros. How on earth did Riemann think of it? Nowhere, and I mean nowhere, will you find an intuitive explanation of this theorem, let alone why the original proof used Dirichelt’s principle, which is about solutions to the Poisson equation! Index theorems – which R-R is an example of – are deeply tied to PDE’s – the Atayah-Singer theorem uses the Dirac equation (!) – but you’d never guess it from the usual literature. This is frustrating, especially as the project of the series is to provide just such an explanation.

Another chunk went on being not-quite-right in the body and mind. One Wednesday morning in mid-January, I swear I nearly turned round, went home and called in with ME. I had practically no will to tackle the world. Let alone the daily journey to Liverpool  Street. The previous evening I had actually dismounted from the bike twenty minutes early in my spin class – I didn’t do that when I first started. I had just Lost It. I took lots of early evenings and did little gym. I took a long weekend at the end of the month, and started to feel stronger and more able to take on the world.

A lot of time went on two hefty – by my standards – projects at work. One of them was a “despite” project – one you get done despite the tool set and data you have, not because of it (if you don’t know what that means, get a job in a retail bank). The other was / is a report where the presentation is as important as the data – long story. Day after day went past with no visible progress and a lot of effort. I solved one of the problems recently and can concentrate on the next one.

Whatever it was I got before Christmas was more than just the cold and fever that laid me out over the holiday week. It’s made my breathing rough – I can barely breathe well enough to run on the treadmill for more than a mile. Some of it was cold and fever, and some of it is something else.

Soon after I wake up, I think “I can’t go on doing this”. It passes, but it’s real. I don’t like the City – like all west Londoners, I get hives if I have to travel East of the Kingsway for any length of time. I read some research that found that rooms with low ceilings are good for doing transactional, routine tasks, but to do any thinking you need higher ceilings. The ceilings in our new office feel about as high as the galleys of the Cutty Sark.  Some of me is prepared to get used to it, but the rest of me wants nothing to do with it. There’s the real conflict. I haven't felt entirely fit and well since we moved there.

Accounting Ratios Are For Accountants, Not Insight

One of the many variables our accountants track is the average weighted selling price (AWSP): the loan-amount-weighted-APR. This varies slightly each week even in the most stable of times, especially with a pricing engine as sensitive to as many customer-centric variables as The Bank's, but no-one has a good idea of how much variation is just "BAU" ('business as usual") and when to start wondering if there's something going on on the outside, Mr Jones.

I'm wondering if we should even be tracking these kinds of accounting ratios at all. The AWSP and the ALA (average loan amount) are more-or-less inversely related: put up the APR and the amount borrowed goes down. (Who knew?) But when you look at what happens, it's not what you think that happens. You think that when we put the prices up by 10%, almost everyone knocks 10% off the amount they borrow. Not so much. In fact, not at all. Very few people take loans for some random amount like £5,237: they take loans of £4,000 or £5,000. The distribution of loans by amount is not normal, but has spikes at the round thousands. Almost no-one takes a loan for £18,000 and a fair proportion take loans of £2,000. And this is true no matter what the prices are.

I'll just say that again, because I can guarantee you didn't get it. You can change the prices all you want, and the peaks will always be at round thousands and multiples of six months in the loan terms. What really happens when we put prices up? A 10% in increase in APR from 10% to 11% on a £10,000 loan over 60 months makes a difference of £5 a month to the repayments. Most people can afford that, if they can afford the original amount in the first place. That's one reason why things don't change much for a building society with a fairly simple rate structure when it bumps up its APR's. However, the pricing models The Bank uses don't usually impose a constant increase on everyone. The changes are obtained by changing propensity curves which can make some hefty differences to some people and very little to others. You might have thought you were going to pay £220 a month, but actually it turns out that now you're going to pay £260. That's a week's petrol for the car for a lot of people. Decision time.

Many other people will be unaffected. I think (because we don't have research - yes, I said: "we don't have research") that some people make a big adjustment - instead of buying a second-hand latest-model car, they buy a second-hand previous-model car that's perhaps only six months older, because that knocks at least £1,000 off the price of even small cars (and that's my Top Tip for saving money when buying second-hand cars) - while some people abandon the project altogether because it's not the sort of project you can save £500 or £1,000 on.

Over half the personal loans made every day are to re-finance other debt, and maybe some people, faced with a higher price for the whole lot, go back one more time to the Bank of Mom and Dad to ask for help on the last £1,000. There's undoubtedly a rounding effect in the intial amount. The project (new TV, new kitchen, wedding, new floors) might cost, say, £8,250 and many people will round up to £9,000 so they can get the TV / holiday / new sheets / some other bunch of stuff they've been putting off for ages, as well. When we put the price up on them, they shrug, borrow £8,000 and fund the last £250 from a credit card or cash flow.

Take 2,000 of these random decisions a week and it smooths out. So the accounting ratios paint one picture, while the detailed MI shows something else. Accounting figures are accurate, but don't provide insight: MI is often slightly inaccurate, but provides insight. (Inaccuracy is fine, as long as it isn't misleading.)

I've been careful to call the AWSP and ALA "accounting ratios" rather than "statistics". Technically, accounting ratios are statistics (symmetric functions of observed values of random variables). Emotionally, they aren't. I have a prejudice that a "statistic" ought to do two things: it should tell me about the distribution, and it should be about something physically or commercially meaningful. AWSP sounds as if it should be commercially meaningful, but I'm no so sure. Because loans are about money, and you think all money is the same, I'll use a different example.

A record label could report the "average price per CD" (APCD). Under certain circumstances, it seems that might be a meaningful indicator. When it changed, you would still need to find out why. Did less people buy from "new release premium" label, or more people from the "new release budget" label, and why? No hot new releases this month? Was one of the budget releases was favourably reviewed on Radio Three? Perhaps the month's hot new release was on a competing label, so no-one bought much from yours. Or maybe the online server went down for two days, and since that's where your premium sales are, that's why the APCD dropped. Or maybe Amazon are making a point to you, which you've now understood. Or maybe a hundred other things because that's what business is: a thousand details that don't make a complete picture.

The accountants won't stop reporting the ratios, but the MI team, if there is one, should start reporting something more useful: the details of the product / price mix and how that changes. Or whatever other mix variables you need to look at to explain the make-up of some critical line.

Resolutions For My Participation In Social Media

I joined Facebook a while ago and haven't been able to rest easy about the whole business of social networks since. Okay, I'm exaggerating. But I don't really get Facebook - maybe because I don't get that sharing stuff you're doing with others is a pleasure for you and them. My life is so based on work-gym-sleep routine and cultural consumption all before a ten o'clock bedtime that I can guarantee it's not interesting to other people. I hope what I think is, but I'm pretty sure what I do isn't.  So the whole "what am I doing on Facebook" thing turned into "am I making use of and contributing to the right sites on the Internet?"

There are two questions. Where do I need to be? And where do I want to be? "Need" is easy. I need to be on whatever industry networking site is actually used by people to hire people who do what I do. There isn't a site specifically for Insight / Pricing Analyst / Managers, though there are agencies that do a certain amount of specialising, but if there was, I would join that. Instead I'm on the general-purpose site for People In Suits, LinkedIn.

Where do I want to be? Put that way, the answer starts to be obvious. I want to be where there's stuff I'm interested in. That's music, attractive women, philosophy, mathematics, holidays, photography and other such. A lot of that is in blog-land and my contribution is by reading. I know the etiquette says I should leave comments, but I can rarely think of anything to say at the time.

Away from the blogs, we have... Amazon. This counts as a social media site: it's about the reviews. I review under my "Real Name". Books are one of my things, so this makes sense.

Other Review Sites: I'm thinking of TripAdvisor here. Google an holiday destination and TripAdvisor comes up. I do feel I should contribute, but can't get round to it. I'm very good at enjoying stuff, and taking photographs of it, but not so good at describing it.

Wikipedia. Perhaps the ultimate social content site: I have two articles from my time in telecommunications on it, under my real name. That's all. Maybe when I have loads of unpaid time on my hands, I'll do more.

Blogging. Yes. Here already. Long form, however. I don't, as you'll have noticed by now, micro-blog, so Tumblr is out. I'd like to make cool, laconic comments under equally cool photographs, but I'm a philosopher, not a fine arts grad. 

Photography. I'm not a pro, I don't do those polished Photoshop-ed pro-am photos that Flickr and deviantArt, amongst many others, are full of. It's not my style. And it strikes me that getting work up to that look, cataloging and tagging it, is a ton of work my diary doesn't have the space for. 

Guilty Feelings. No, that's not a site. I use You Tube so much, I feel I should give something back, but I'm not sure what. Or even how. I feel the same way about TripAdvisor. Google somewhere you're thinking of for a holiday and up this pops. I love the idea of 8Tracks, but the people there seem to have scarily awesome taste in music and accompanying art, and I'm suitably scared.

Forums. Like everyone else I use various forums, mostly for technical computing problems. I'm not good enough to give back in that area, and there are no forums for in-house business analysts (though there is for quants, though it's more of a career board than a how-to). This is because most in-house business analysts have a narrow and shallow transferable skill-set - what they are about is the use of in-house data, and discussing the family secrets with strangers is frowned upon.

Almost lastly, the Facebook Question. (Substitute your local social networking site for China / South America etc). I don't have a personal life - I commute, work, go to the gym, watch movies and read, write, think and sleep. What's my status update going to look like? "seven dials is... trying to calculate the first cohomology of a circle using only short exact sequences"? Yeah. Right. I'm on Facebook as so many are, to see who else is. In the spirit of reciprocity I should have some minimal, PR-friendly information about myself in case some HR department or recruitment agent goes snooping. I have the odd glimpse as to how one's page could be turned into some kind of ironic artwork, though I'm guessing someone else has already got there.

So here are my immediate resolutions. First, I will write at least one Amazon review a month from now on. Second, I will write a couple of reviews on Trip Advisor. Third, I will make up three playlists and attempt to put them on 8Tracks. Fourth, I will investigate what's involved in posting something to You Tube - if I can find something that no-one has already posted. That should get me started.