Data is Expensive, Conclusions Are Cheap: How To Fix Research Fraud

It's probably just my echo chamber, but I've seen a number of YT's on scientific fraud recently. This does not shake my faith in Quantum Mechanics, because this isn't happening in real science. It's happening in psychology (evolutionary or otherwise), behavioural economics and other such pseudo-subjects with lousy replicability, and a tendency to pass off small samples of undergraduates as sufficient data. I've read my share of pop-science from these people, and while I've been amused and intrigued, I've never been convinced. The samples are too small. The conclusions are too darn cute, and fit way too well into the current academic Goodthink. Also a lot of it is just plain wrong.

What does one do about all this nonsense research?

Realise that statistical analyses, summaries, graphics, and conclusions are cheap.

It's the data that matters.

Any research project funded by the taxpayer must make its raw data publicly available, along with a detailed description of how the data was obtained.

With no controls over access. In CSV format so we don't have to write complicated scripts to read it.

And at no charge. We already paid with our taxes.

Give us the data, and we will draw our own conclusions, thank you. Research will become valuable because it produces data that people use.

Not because some publicity-savvy academic produces an eye-catching claim.

The infamous thirty-undergraduate sample will simply vanish.

Researchers who provide lots of dimensions of analysis that can be correlated with ONS data will get readers, those who use a few that maybe can't be matched against anything else will be passed by.

It works like this.

Hypothesis: children from single-parent families do better at school than children from two-parent families. 'Do better' means more and better grades at GCSE. So get a sample of single-parent households with kids who just did their GCSE's and another of dual-parent households with kids who just did their GCSEs. Same size, as there are plenty of both.

Recognise that the initial question is attractive but silly. It's the kind of question a single-purpose charity might ask, and if it liked the answer, would use in their next fund-raising round.

"Single-parent homes" are not all the same. Neither are "dual-parent homes". Families are all different. And they are an effect, not a cause. Parental behaviour, sibling examples, household economics, the location, the religion and the culture are causes.

Here's your chance to get some data-kudos.

Get a decent sample size. 10,000 or so of each.

Get the results for the kids. Grade by subject. With the exam board. No summarising or grouping. I've got a computer to do that if I want it.

And get the number of GCSEs the kids were entered for, because Head Teachers game the stats like crazy. While you're doing that, find out how else the Heads game the stats.

Get the details about those households. Age, religion, nationality, gender, political allegiance if any, car owner, rent / mortgage, highest level of education reached by parent(s), subject of degree, employed / self-employed / unemployed / retired / not able to work, occupation if working, postcode (all of it), place of work, large or small employer, private or public sector. Income and sources, expenses and spending patterns. Savings. Help from relatives. Drug use. Exercise regimes.

How long had the parents been divorced before the GCSE exams? How long had they been co-habiting or married? What are the childcare arrangements? What are the visitation rights? How often are these denied? Has the divorced partner lost touch with their children? Are the divorced parents still co-operating with each other over raising the children? Has the custodial parent moved home? How far away are the parents from each other? Was a family member in jail when the kids were taking the exams? Is the father in the dual-household away a lot? Do any of the parents work unsociable hours? Do they use daycare?

See how that data could be interesting to certain groups? Even if they weren't interested in GCSE results?

Did the parents arrange private tutoring? Help their children with their homework? Do the children have long-term health problems? Did they have health problems at the time of the exams? Were they able to revise? What is the school's record in the league tables?

You get the idea. Ask a wide range of detailed questions to cover the vast complexity of human life. Notice when a colleague demurs at something that allows the data to show the influence of (enter taboo subject here). Find somewhere else they can be useful and send them there. Do the same to yourself. The question you resist the most is the one everyone wants answered.

Test the questions. Test the interview process and the online questionnaire (if you must). Do A/B layout and question-order tests. Learn and make adjustments.

Now go out and ask the questions. Tabulate the answers. No leaving anyone out because they missed a bunch of answers. I can deal with that in my analysis. No corrections for this or that. No leaving out the answers to some questions because of "sensitivity" or "mis-interpretation".

That's where you put in the effort. If too many people give incomplete answers, go recruit some more people. Comparing those who gave complet(er) answers to those who didn't to see if there's a pattern.

Put the raw results up on Github or wherever. Along with the questionnaire, the times and dates of each interview, and a video of the whole thing if possible. I want to see their body language to judge which questions are likely to have, uh, aspirational answers. (Okay, that's asking a lot.)

I'll do my own analyses.

The researchers can publish a summary and conclusion if they want. With a keep-it-simple press release for the science journalists. The rest of us will dig into the data and draw our own conclusions.

The people who don't do data analysis can get some popcorn and follow the disputes.

Data financed by private money? Make it public or we get to treat it as self-serving.

Faced with some conclusion about medicines or human behaviour, ask if the raw data and research protocols are publicly available. If the answer is NO, or "you have to pay", dismiss the conclusion, because there is no evidence that you can judge for yourself. Without the data, we have to take their word for it or not, which means we need to judge their competence, honesty and career pressures. That makes it about the researchers, and it isn't. They may be insightful and honest, or they may be academic hacks. You can't judge that either. What you can judge is that they are hiding their data. If they are, it fails the smell test.

37 Great Portland Street

   (This is what's known in the trade as a "deadpan photo") 

 I did look at the chairs in IKEA, at Wembley, just for the pleasure of driving on the slowest parts of the North Circular. I have one of their Stradmon wing-back chairs, and since then they seem to have been through one cost-cutting round too many. Everything is one of four slightly muddy shades of grey (except the yellow and red Stradies), the material feels cheap, and the cushioning feels like it might give up too quickly. So I went back to the semi-fancy furniture shop, because looking for value is one thing, but being cheap is another, and I had one more round of sitting on all the chairs and then ordering two of the same swivel chair, one navy, one black. How much did they cost? Less than my hi-fi amplifier, more than the TV and DVD player.  

Southend Pier

(Southend Pier is the longest in the UK, because the Thames Estuary is very wide and very flat, and the pier was to disembark passengers from the London boats. There's a train from the land to the end of the Pier. It's all pretty neat and way more chill than Brighton Pier.)

The Big Day was calling the council to come collect the bed. I have to get it into the front garden, so it's a good thing the mattress is a bit flimsy (though "orthopaedic") and the base is two boxes made out of wood that takes the weight but not if you stood on it directly. The decline in the quality of our lives is hidden in such details. I had to clear the hallway to get the bits down - because I was Thinking Like A Designer and planning ahead. I moved the bases and mattress into the hallway the previous evening, and woke up at 06:00 the next day to wrangle them into the garden, covering them with some plastic against a light shower. They collect between 07:00 and 15:00, and showed up at 11:00. I spent the afternoon clearing the back bedroom and laying plastic sheeting over the fitted carpet, before laying sheets over the plastic. Because I was going to put a second coat of white over the walls and ceilings. I'm not really a pro, but I have learned some of the tricks.

Regent Sound Studio, Denmark Street

 Also I want to read more. At the moment I read in three places: the train, in bed, or in my garden chair. If I try to read on the couch, I fall asleep within ten minutes. I read paperbacks and on the iPad, and I also read hulking art books that need to be on my lap, a table, or better still, a bookstand. This means I need to find chairs I find comfortable to read in that don't come from garden centres. Try it. I wound up Googling "reading posture" but all I got was the usual posture Nazi stuff about sitting upright with a straight back and the book held at head height. The same posture they recommend for using a keyboard. These people have never in their lives cut a line of code, written a paragraph of fiction, edited a photograph or anything else requiring thought. That advice is for people who spend all day typing other people's words. The moment the brain focuses on a difficult problem, it takes resources from un-important activities like, yep, maintaining posture. So I added in some trips to a local decent furniture store, as well as IKEA.

Regents Canal: Islington Tunnel to Kingsland Bridge

(This is a very pretty part of the canal, until about the A1200 bridge when the Council must change, because it's a bit scruffier.)

Think Like A Decorator wants you to think about what you want your house / room to say about you, and about what you're going to be doing in it. Socialising? Escaping the human race with a good book? Mostly Leslie Banker is socialising New York style - as we might expect - there's not a hi-fi listening room, a TV, or a computer gaming rig and chair in sight. Never mind, I can adapt the principles. I could also figure out what I thought was ridiculous styling, and thus what I might find acceptable. (You can't escape styling: you have so much as one plant in the house, that's an attempt at styling.) I realised I was trying to use one room to do too many things, and that a music listening room should be pretty much just that. It should not also try to be the room where I eat and do all my creative activities. That's why the back bedroom suddenly stopped being a bedroom. The guitars, piano keyboard and breakfast table are going there

Russell Square and Around

 

(Russell Square is right by a large chunk of Academic London: SOAS, Birkbeck, a Law Institute I didn't even know existed, Senate House, the UCL School of Pharmacy and Royal Free Hospital, with the LSE and Kings College down the road, and bits of UCL scattered all over the place.)

I have no innate talent for interior design, decoration and styling, but there are people who aren't professional designers who do. I have to read a book. The one that made sense to me this time around was Leslie Banker's Think Like A Decorator The pictures are pure decorator pornography, as in all those books, but the text is actually quite practical. At one point she explains that those photos are of rooms that have been re-arranged, styled to within an inch of their lives, and then had the power points, light switches and other intrusions Photoshopped out. But then there was the remark about thinking through the exact order you do things. For instance, when do you wallpaper the narrow corridor to a bedroom that needs a new bed moving into it?

Little Venice

(Little Venice is less than a mile from Paddington, and off Warwick Avenue. There are boat trips between it and Camden Town, which is why we were there.)

Another round of gratuitous catch-up. I've been deep in thought about re-jigging the back rooms in my house. I spend way too much time in one room, in one position, because hi-fi soundstage. The upstairs back bedroom dates from when I had lodgers, and I haven't had those for the longest time, but neither have I had anything else to do with the room, so it just stayed that way. Now I have an idea for what I could do in there, so it's time to move this, throw out that, buy a couple of those, re-paint the other and all that jazz.