The Illusion of Marketing Segmentation

One of the things that Insight Analysts are expected to produce from the data are useful customer segments. The conventional wisdom is this:

Customer segments are differentiated by the customers’ different requirements for your product. The value proposition for any product or service is different in different market segments, and the price strategy must reflect that difference. Your price realization strategy should include options that tailor your product, packaging, delivery options, marketing message and your pricing structure to specific customer segments, in order to capture the additional value created for these segments.

Publishers have been doing this since Gutenberg. It makes sense for cultural production: books, music, paintings, wallpaper, paint colours, rugs, furniture, plate design, perfume, clothes, pens, watches and so on. But for domestic water? How do you vary “clean water”? The logistics of water supply mean that we all get the same water.

And just how different can a telephone service really be? It’s all the same copper and fibre, and no matter what price plan you’re on, you’re going through the same switches. Those price plans are there to smooth the usage out. Data services can be made faster or slower, but not telephone calls. Are Hertz “segmenting” when they offer you a range of cars from a Ford Fiesta to a Mercedes E-class? Insofar as a car is a cultural object, yes they are, but the service you get, from online reservation to them checking it for damage and petrol tank fullness on return is the same.

Airlines don’t segment their customers when they provide First, Business and Economy: they segment their cabins and hope there are enough people willing to pay those prices. If they researched their customers, they would hear overwhelmingly that we all want 36 inches or more leg-room, no middle seats, no reclining seats, and either no children under about 18 outside a special sound-proofed part of the cabin. Like hell airlines are going to do any of that. 

The truth is that in most cases you can’t "tailor your product, packaging, delivery options, marketing message and your pricing structure to specific customer segments”. A supermarket can’t identify who will take home delivery of groceries until it offers it, and offer it at a sensible price and under sensible terms. The most it can do is conduct some interviews and surveys, and everyone knows the problems with those. 

An airline can’t identify who will pay for flexible return or more legroom: it can only offer it and see. Or conduct those same dubious surveys. A bank, however, can make a good guess about who it is prepared to lend money to. A supermarket can make a good guess about who might be tempted by an offer of a discount on Pampers. Because unlike the airline, who don’t have a clue about their passengers' disposable income, a bank does. And so do supermarkets, at least for some of their customers.

But companies with access to really good MI are few and far between. Banks. Insurance companies. Supermarkets (includes Amazon). Two out of three of those industries are basically commodities. And strictly, so are retailers. (Think carefully: the shopping experience is the same, whether you buy the branded or the Value Range, fresh baked or sliced white - you’re pushing the same trolley round the same aisles and standing in the same queues. Supermarkets provide a commodity service: what they sell sometimes isn’t.)

It’s not customers who drive segments, it’s producers and providers who take the risk of creating them. A company can create a product no-one wants (I’m looking at you, Sinclair C5), and it can create a product everyone thought they wanted until they got it and then they realised it wasn’t that great (hello, Kindle), and there will be those (hiya Steve Jobs) who say it can create a product that no-one knew they wanted until it was there, but I’m not so sure. I think every product has a fore-runner of some kind. The theatre has spun off the cinema, radio, TV, videotape and DVD. All come from the same desire to have dramatic entertainment.

And those companies will have absolutely no idea who will take up the products - unless the company has a strong cultural image (take a bow Apple). That’s how Burberry got themselves into a mess a while back.

Jazz and Whiplash: Review

A couple of months ago a friend of mine told me about a movie he'd seen in which a young man drives himself crazy trying to play drums like Buddy Rich. I saw it last weekend. it was a well-made, beautifully-shot and dressed movie, nice editing, the script, the acting, all top-notch. But. They didn't push the envelope. On the level of content, it is entirely nonsensical, on an emotional level it works rather well.

What this film is really about is how Academia has wrecked jazz as a music form. Let’s get this more or less straight: jazz ended with On The Corner.


Miles was to jazz what Jean-Luc Godard was to movies: On The Corner was his fin du cinema moment from Weekend. Oh sure, plenty of people went on playing stuff they were playing back in the 1960’s, and Wynton Marsalis came along and re-cycled time-no-changes while Miles was in hiding during the 1970’s, but nobody did anything new. Miles, Trane and the avant-garde guys did it all in the 1960’s.

The point of jazz and blues is that it’s freedom within a genre: they played blues, ballads, hard bop, cool, swing, modal, time no changes, or free, and they did it in their own manner and with their own voice. That’s why even a newcomer can identify Miles, Bill Evans, John Coltrane, Charlie Parker, Eric Dolphy, Ornette Coleman, Ron Carter, Wynton Kelly, Tony Williams and about fifty other players after just a few notes. Those musicians had voices as distinctive as any composer. They learned as much technique as they needed to make the music they wanted to make.

That’s not what happens now. Listen to Giant Steps.


It took Coltrane three months to figure out how to solo over those fast-changing chords. Let me say that another way: it took the one of the most creative jazz musicians that ever lived three months to figure out how to solo over those chords.Yet young saxophonists are expected to be able to play a Coltrane-like solo over those chords before they graduate. They have amazing techniques, which are entirely beside the point. If Giant Steps has to be used at all, and not treated as a musical dead-end, they should be expected to figure out their solo over those chords. Coltrane’s solo was a solution to a problem: how do you play a hard bop solo over so many chords? Solved. Next problem: how do you play a different solo over those chords? It may be the Koybayoshi Maru of music problems, but it’s better than learning to play a bunch of changes quickly.

And Giant Steps lead to the chord-scale system. It’s the reason all contemporary jazz sounds cold, identical, soul-less and un-musical. Because the musicians are so darn occupied trying to remember what scale they play over E maj 9-5 that they forget to play a tune and a feeling. A real musician can play one note and make you feel an emotion. With chord-scale, the musicians don’t get time to get to the emotions.

Chord-scale, and polished technique, is what gets taught in the colleges. So there’s an analogical, emotional truth in Whiplash: the tutors might not actually shout at the students - and they would lose their jobs if they talked as JK Simmons does - but what they teach crushes them into moulds just as surely.

The ending of Whiplash was yet another bully-and-victim reconciliation scene. Those don't happen in real life and everyone over the age of about 12 knows it. I wanted to believe it, even though I knew it was hokum.

The "good job” speech that Simmons character gets is utter twaddle, both as an author's message and from that character. Which brings me to what I really didn't like about the film. I lost count of the number of times the JK Simmons character told the Joe-Jones-threw-a-cymbal-at-the-young-Charlie-Parker story. (And as my friend said: he did it once, not eighteen times a day for a year.) The teacher's excuse was that he was trying to find, or make, the next Bird. And that's arrogant beyond all measure. It's not a teacher's job to produce another Charlie Parker, it's a teacher's job to teach the frickin' trade skills and knowledge at the speed required. It's the artist's job to become an artist. Teachers are there to set a pace, but not to push. If the student can’t keep up, you let them ring out. You don’t throw chairs at them.

Here's my idea. Talented young man who can ace the technical stuff but wants to play his own music. Get girls, has friends, but those are just entertainment. Fifty minutes of all the different kinds of music in New York. Some glimpses of how the various scenes (jazz, Latin, etc) work, what the economics are. He's trying to find what Miles called a "direction". His friends join orchestras to play other people's music, or bands to play genre music. His girlfriends do what they do - office jobs, whatever. And then he hears it. Five bars. Boom! And we're off. Because that's how creativity works: we build on what others do. And when he finds his musical direction, he finds The Girl as well, because that's the kind of happy ending the audiences like. Everybody behaves well, and nobody gets pregnant or shouts. The most we get is puzzlement: why would anyone want to miss out on jobs and careers just to find their own music? And that is what the story explains.

Just a thought. I mean, it's not within the realm of conventional cinema... but what if?

December 2014 Review

Am I the only person who realises that Dickens was commissioned by the retailers of central London to write a story that promoted the true, consumerist, meaning of Christmas? A Christmas Carol is as blatant a piece of PR as anyone should ever recognise. Scrooge is quite right: Humbug! The whole damn country shuts down for about a week to ten days.

The best thing that happened in December is that I finally threw off the colds and food poisoning I’d had for the last two months, and then had to address the extra poundage I was carrying as a result of all that carb-heavy comfort eating. It took while for the guys at what’s now my regular lunchtime cafe to get that I just wanted meat-and-salad, but that’s what I eat now. Do it consistently and it works.

I read FUSE: The Russia Shift by Johnston and Greenwood; Rupert Smith’s eye-opening The Utility of Force: The Art of War in the Modern World; Neville Shute’s On The Beach; Richard Bentall’s Doctoring The Mind; Mat Ruff’s Bad Monkeys; Unziker and Jones’ Bankrupting Physics; and Philip Kerr’s March Violets, and The Pale Criminal.

I watched all of Elementary S1, Sons of Anarchy S5 and True Detective S1; we had the annual Peter Jackson family outing to Cineworld Feltham to see The Hobbit: The Battle of Five Armies; and The Grandmaster and Electricity at the Curzon Soho.

Sis and I had the worst supper we’ve ever had at the Golden Dragon in Chinatown, and I made it up to her with supper at Hix. Four of us from work filled a table at Tay Do on the Kingsland Road with starters and had an excellent meal; I had lunch at Randall and Auben, and at Balans as a friend knocked on the window and waved me in as I was passing by.

I worked the days between Christmas and New Years', using them to revise my SAS Base and document exactly how much I don't like using it. And I made sure I went training so I didn't make the mistake of taking a fortnight out and needing a month to get back to normal.

Coral and Boots: The City

Yep, more photographs with silly captions.

I've been on a streak with the Riemman-Roch essay and a couple of other things, and haven't had the time to write about anything else. Besides which, I've said a lot of the things I want to say - even if I have to go back and remind myself I said them.

Je Suis Charlie: Pas Peur

What's really sickening is the way that the security service and police use such random tragedies to pitch for bigger budgets and more powers.

The thugs who shot those journalists were not terrorists in the way that the members of an IRA Active Service Unit were terrorists. The IRA had political demands, objectives and plans, the "Islamist" terrorists have nothing: they have no political demands, and no State or organisational backing. They are a-political and un-aligned. They aren't terrorists because the last thing they want is to get rid of the State that pays their welfare cheques. They are young men with what will soon become a recognised psychiatric condition.

This is the right sentiment:



The sooner these random killings are seen as the work of emotionally-disturbed people, and not as some kind of religious or political protest, the sooner the various States will get the proper perspective on what's happening.

Citizen Four (2): Big Data Is Even More Useless To The NSA Than To Tesco

Security consultants have a vested interest in scaring you, your employer's IT department and the politicians. Mo' fear means mo' money. There are a few good guys out there, but none of them are working for McAfee, Norton, the Big Consultancies or the big software support companies. Those people are in it for the money. Here's a quick test for anyone who claims to be a computer security consultant. Ask them if you need McAfee or Norton running all the time on your computer. If they say YES, thank them for their time, show them the door and check you still have your watch and all your fingers when they've left. (Why you should is the subject of another post.)

Though the security consultants often seem to be against the Sigint community, their interests are more or less exactly aligned. The Signint guys want the Bad Guys off the Internet because the Bad Guys are lost in the noise there, so they spread stories about how the Internet and phone service is their bitch. The security consultants want to sell you their stuff, so they spread stories about how the Internet and phone service is anyone's bitch, but especially the sigint guys'.

(Don't get me wrong. Banks, medical companies and government departments that deal in personal data need to have secure communications and computers and data. They should vet their staff and make it difficult for even employees to sign on to their networks. You need to practice safe computing at home and in cafes, and run your OS firewalls. But like all security, this is to deter amateurs and up the cost of hacking you as against the next person. If the pros want access to your computers, they will get it.)

The hype says that the sigint agencies can search amongst all this data to find "patterns". There are two kinds of patterns. First those obtained by looking at who is contacting who, and who visits what websites, sometimes called "traffic analysis". The idea is that the agencies have certain kinds of pattern-archetypes they prepared earlier, and go looking for those in new traffic records, thus finding terrorists, drug dealers, illegal gambling lines and all sorts of other illegal activity. Because terrorists and drug dealers don't learn and are creatures of habit. This is more-or-less nonsense. Traffic analysis works when listening in on radio traffic between armed forces engaged in industrial-scale warfare (which is where it came from), but unless it's used in conjunction with a list of "numbers (or URL's) of interest", it's more or less useless on a retail scale. The second kind of patterns are about content: word use, photographs and the like. In business, this is known as predictive modelling, and there's a huge problem with it.

Predictive modelling is used to identify people who have a higher probability of doing whatever it is you're selling or supplying: using certain kinds of social services, taking insurance or loans, making insurance claims, defaulting on payments (that's a huge industry in the financial sector called "credit risk"), committing crimes, or redeeming coupons for Pampers. These are almost always events with a very low incidence - very few people do them each month - and a fairly low prevalence - the stock of people who have done them is less than 10%.

A bliding glimpse of the obvious is that if you want to predict a rare event with high probability, it must be with a bunch of indicators which line up just right almost equally rarely. In business, it can be acceptable to use a method that over-predicts wildly, as long as it over-predicts less wildly than the previous method. If you can send only half as many leaflets and get twice the response rate from those letters, you've halved your marketing costs and kept the same revenues. In business that counts as a result. In espionage, that's awful: you have far too many false positives.

The other blinding glimpse of the obvious is that you need enough examples of people doing whatever it is to find and prove patterns with statistical techniques. There just aren't enough terrorists in the UK, and there haven't been enough bombings, to gather that amount of data.

The holy grail of predictive modelling is that the private process has a public choke-point: everyone who does X, must do Y or Z and almost the only reason for doing Y or Z is X, and that Y and Z are both easily observable. Seeing someone come out of a branch of William Hill is pretty good evidence that they laid a bet. As far as anyone knows, there’s no equivalent of William Hill’s for terrorists and other nasties. And even if there was, it wouldn’t last for long, as they will change methods on an erratic basis. This is basic tradecraft that’s been practiced since Sun Tzu ran spies, and it’s not rocket science. You think that bit in The Wire where the bad guys sent each other photographs of clocks wasn’t based on a real example?

No. Nobody is using Big Data techiniques to spot malfeasors and terrorists. They might be trying, but you can rest easy that they will fail. The benefits of Big (commercial) Data are mostly hype, and the benefits of Big (Intelligence) Data are total vapourware. Except, and this is crucial, when the agencies have a bona fide target and can get that target’s phone numbers and other comms identities. That takes humint, not Big Data. Business has had Big Data for a long time, and the best it can do is improve the efficiency of its mail order shots from, oh, 0.2% to 0.6%.

Collecting data on “everyone” is so obviously pointless, un-economic and silly that if the NSA and GCHQ are doing it, or heading that way, the people in charge should be fired. I don’t think the people who run these agencies are stupid. I don’t think they are really doing what the FUD-meisters in the security business suggest they are doing. But I do think they don’t mind that the security FUD-meisters are saying that they can and are.

So was Edward Snowden actually planted on us by the NSA to spread the fear? I don’t think so. Though it would explain why his location wasn’t found within an hour by an operator looking at hotel security footage from across the world, and why he wasn’t shot the next evening by a special forces sniper flown out to Hong Kong on a Gulfstream and guided by imagery of the hotel bedroom taken from one of the smartphone cameras that was turned on automatically from half-way across the world. Because that’s what the NSA and CIA can really do. Right?

Oh. And the scene in Citizen Four where the bullies from GCHQ make the Guardian journalists grind and drill holes in the hard drives to destroy the data? Pure hype. On a modern terabyte-storing 3.5" platter, a single write 0's pass will eradicate the data past all restoring, just as securely as some fancy 7-pass US DoD wiping algorithm. The forensic guys can deal with lightly damaged discs, discs that have lost their controllers and stuff like that, but once you do a standard disk wipe, it's gone. Hit it with a hammer a few times afterwards if you like. But the guys from GCHQ would prefer you believed that they can see past a data shred, so that you didn't bother in the first place. Then they could "recover" the data.

Citizen Four (1): The Logistics of Tapping

Citizen Four is an excellent documentary about the first days of the Edward Snowden revelations. There’s a lot of him in the movie, and he seems to be an intelligent, savvy young man. This post isn’t about him or the rights and wrongs of indiscriminate surveillance, but about the feasibility of the claims being made about the recent activities of the NSA and GCHQ. It's therefore also about how worried you should be by all those revelations.

Right now the sigint (signals intelligence, as opposed to “humint” which is actual people) community are sending out some very mixed messages. On the one hand, they want to get content-level access to e-mails, websites, Facebook, Twitter and everything else, and they want ISPs to keep it all for a few months. On the other, seemingly they can tap and de-crypt anything, anywhere and in real time, they can turn on the microphone of your smartphone and listen in to your regular conversations of your smartphone, while using its GPS to track you.

Personally, I find the idea that, should I ever get lost or kidnapped, all anyone has to do is call Fort Meade and ask them where my phone is.

As if. The sigint community are, and have been for a good few years, drowning in digital noise. Let's do a little history.

The heyday of sigint was up to the mid 1990’s when most of the world’s telecoms traffic went through copper cable or by radio to satellites. That’s what the GPO Tower was built for: maser trunk transmission.

They took the masers away a couple of years ago. That’s what all those domes at Menwith Park and other places are for. It’s all still useful, as a lot of traffic to Africa, North Asia, parts of the Middle East and other assorted hot spots still goes over satellite. (The Sea-Me-We and FLAG cables go to the major towns in their destination countries, not to places like Syria or Kurdistan. Don't even think about trunk landlines in Syria or Kyrgyzstan.) All you have to do with copper was wrap some wire round it to pick up the magnietic fields created by the changes in current that is the signal, attach it to some headphones or a tape recorder and you're in the bugging business. It's much the same with radio waves. Point an ariel at the sky, tune your reciever to what you know is the satellite's frequency and wander around until you get a good signal. A few technical details aside, that's more or less it.

And then came fire-optic cable and digital. The Signint community hates fibre-optics and digital communication, because:

It makes effective encryption easy;
There’s no regulation of the technology;
It allows humungous amounts of traffic: they aren’t just looking for a needle in a haystack but a salt crystal in a ocean;
It’s horribly difficult clandestinely to monitor communications

That last bit contradicts what you will find on interwebz, which will have you believing that you too can tap into a fibre-optic cable for a tiny cost. Well, first you have to find it. Then you’ve got to dig it up. Then you have to put in your tapping device - and since that involves physically manipulating the fibre, it’s impossible to do without setting off alarms back at the carrier’s NOC - but let’s assume the operators were watching football at the time, and then you re-bury the cable.

Here’s the first question: how are you going to get all that data back to base? A main trunk line will pour out data at around 2T bits/second. Lucky for you that you just happen to have a similar quality fibre-optic cable laid right up to where you did the intercept? Because that doesn’t cost anything to do and isn’t a bureaucratic nightmare anywhere except the City of London. Ah, I see, you have a submarine - the USS Jimmy Carter - that specialises in doing this. And also happens to carry and be able to lay enough cable to get from your tapping point back to some secure naval base, because the commercial cable-layers are just kidding with those big specialised ships and nine-figure costs. Having got the data back to your secure naval base, you then send it down a secure high-capacity line that comes free with every big ol’ shed you build in Utah.

Here’s the real joke. When you’ve done all that, all you’re getting is a light show. Billions and billions of different-coloured photons. You have no idea which photons belong where and do what. The telcos and ISPs have expensive multiplexers at each end of the cable to send and receive all those photons. Those multiplexers have to be set up and synchronised, and can be changed quite easily and without telling the NSA. Without knowing how the sending multiplexer is set up, all you’re getting is a very fast sparkler. So it’s a good thing you have an inside source at the ISP or carrier. You do, right? And no, you can’t use some fancy algorithm to find the order in the light show. Just in case you were thinking that.

No. Nobody’s doing any large-scale tapping of modern fibre-optic cable. The logistics are impossible. What the sigint services do is connect some kit to the telco's switch (for TDM / SS7 voice traffic) or router (for data traffic) so they get a feed that's been neatly structured. They still have to de-crypt it, maybe, and search it, but it's a manageable amount of traffic. They are supposed to have a Court Order when they do that, and I'm sure they do, but... I'm guessing that what's in their kit these days is an array of multi-terabyte drives, and they copy more data than they have permission for. Every week they pop in and swap out the storage arrays. Hence their desire to make legal what they are doing now anyway. But this is a guess.

(To be continued)

Happy New Year - With Some 70s’ Songs

Happy New Year. Three 70's tunes for you.

"As complete a portrait of total alienation as I've ever heard in music" (Charles Shaar Murray, NME)
 

It starts in the middle of a phrase, has a downbeat swing, stunning vocals and I get lost in it after a couple of bars

Is this where the Bee Gees got that Saturday Night Fever feel from?