Why Cloud Drives Should Be Your Daily Drivers

(Edited 20/9/2025: In the original version, I was too focussed on device loss, and underplayed the necessity of conventional backups. The two work complement each other.)

Think of a Cloud Drive as an external hard drive attached to your computer by wi-fi + internet rather than wire. That's all it is. It behaves exactly like an eternal drive. You can treat it like an external drive: copy files to it, delete files on it, create sub-directories, even back it up. It has one huge advantage over a conventional external drive: it is not on your premises or in your possession, so it can't get taken away by anyone with or without a warrant, or soaked if the house floods, or melts in a fire, or lost if someone's kid spills their orange juice on it, or (enter description of disaster here).

We need to protect against four things: confiscation of our devices; loss of one or more devices; loss of specific data on our devices; loss of access to our Cloud account, and other Cloud-based services. We also need to handle backup of data across multiple devices, without creating a complicated and time-consuming backup routine. For some people, especially recording artists, photographers and videographers, handling large-scale data backup is part of the job, and I’m addressing the needs of people who generate smaller amounts of data.

A Cloud drive protects against losing devices, or having devices confiscated. It allows sharing between multiple devices, and usually offer a browser-based interface, allowing access from any device, as long as you remember the password. Most provide a thirty-day undelete facility to deal with stumble-thumb deletes.

Conventional hard disk backups protect against losing access to your Cloud drive account, or losing internet access. Restores or data-reading can be done to any device with a suitable data port, though if you used special software (e.g. to encrypt and compress) you will need that on the device.

Online backup services protect against loss of external hard drives and the Cloud drive.

(Professionals will usually use a Cloud drive, keeping local copies as storage allows, and use an online backup service - since they aren’t going to walk around making backups of individual employee’s laptops.)

You're already on the Cloud in so many ways. All the e-mail services keep your archives on their servers - in the Cloud. Your password manager keeps your vault in the Cloud. Apple Notes / Evernote and the others all store in the Cloud. All your social media is in the Cloud. Any product that offers multi-device access and syncing is cloud-based.

If you use music streaming, you are accessing a cloud-based music library - far bigger than anything the most obsessed audiophile could make. If you stream video, same thing.

It's the final step to put the documents, photographs, graphics and other files you create onto a drive in the Cloud. The Cloud application will create a directory on your device for its use, and anything you put in that directory will get copied up to the Cloud drive. You can tell the Cloud application you want to keep local copies, or that you are happy downloading from the Cloud drive as needed.

What gets stored in that local directory? The data I need to carry on my life, in case I lose everything in the same day (lose phone on the train, laptop stolen in the pub, dog eats the iPad... you know the kind of thing). Contact lists, schedules, programming code, drafts of novels / poems / textbooks, DAW files, reference books, my favourite photos, music tracks, videos, diary / journal, Notes app, and so on.

E-mails, contacts, appointments. On Gmail. Since 1891.

Text documents / Notes-style apps. Pffft. Doesn't matter how much I have, it likely is no more than 1 GB. Nobody can write that much in one lifetime.

PDFs and similar: Not really an issue. 3-4 GB at most for even the most avid reader. Stored in Books.

Music. No more ripping CDs into Apple Music and synching playlists to the phone. CD-quality streaming services are now affordable and allow downloading for off-line access. If I am actually going to be off-line: 5G and wireless coverage has expanded beyond anyone's wildest dreams in 2015. I keep my CDs (unless I really regret buying it), while Amazon and other downloaders keep track of what I bought from them, so the files are always there. I only really need to keep music files I downloaded or ripped, which can't be found elsewhere. That is remarkably few. (**)

(Qobuz subscribers: downloaded files are added to your Library, with a tick next to them.)

Photo / Video backup. A glance at Reddit suggests this is deeply personal: some people are hoarders and remind you that storage is cheap (which at £150 for at 5TB external SSD it is); while others are minimalists and do not want to spend time managing backups. Then there's a whole bunch of people between these extremes. I tend to the minimalist side. My intention is that Favourites and recent haven't-made-my-mind-up-yet shots will be in iCloud Photos, and everything else I don't throw out will be in directory that gets an external backup from time to time. There's a lot of prep required for this, which I'll talk about this in another post.

With my minimalist approach, I don't have a lot of data, so I get by on iCloud's 50GB plan. FLAC-ripping, RAW-shooting, Lightroom-editing, keep-everything maximalists, will be looking at least at 2TB or more. After making use of the 1TB Adobe throw in with Lightroom subscriptions. That starts to cost.

Bonus Topic: sharing across devices. One benefit of Cloud drives is that all your Windows / iOS / OS X devices can access the drives, but sharing is not compulsory. Apple devices can be customised app by app. How much you do depends on how much you trust the other people who can use your devices, and I'll let you think through how much you trust the kids not to accidentally delete photos or mail stuff to their friends. I let Files access everything it can, and share my Photos on the phone, iPad and laptop. My music apps (Sonos, Qobuz, Amazon, Apple Music) are on all my iOS devices, and I can access those services through a browser on the laptop. I don't have Books on the phone, because I can't read on it. I can get Mail and Contacts via any browser, and share the Calendar. 

Bonus bonus topic: internet privacy. I'm all for someone from Apple, Microsoft, the NSA, Five Eyes (or Five Guys, for that matter) reading my thoughts and looking at my snapshots. They might learn something. You're welcome.

(*) Having all your equipment and backup drives taken away because you made a mean tweet, or one of your darling children downloaded instructions to make an atomic bomb, or a dodgy mate sent you a dodgy image, or whatever happens to be illegal these days. They can take your devices, but they can't stop you looking at your data if you can get at it. 

(**) A while ago there was a fad for ripping all one's CDs and then throwing them away. Because shelf space or something. I ripped everything to AAC / MP3 because it was the only way to download it to the music player / phone. Hence the need to backup music libraries. To become really useful, streaming services needed a) large libraries, b) CD-quality at £10 / month, c) reliable hi-speed internet and wireless, d) high-quality DAC chipsets at commodity prices. This only really came together in 2020.

Online OPSEC Made Simple-ish

(Are you sitting comfortably? Because you will need to be...)

The Online Safety Act got me thinking about VPNs and other gadgets, that got me thinking about online security, which pointed me to the ideas of OPSEC, and that changed my thinking on some of these matters.

Online security is about reducing the chances of financial and reputational loss by identity theft, unauthorised third-party use of your accounts and other means, that is brought about by using the Internet. Privacy is a by-product of doing OPSEC well. This approach leads to some interesting conclusions, for instance...

Adult sites carry a reputational risk (with almost everyone in your domestic and professional life). It hits when people who want to pick a fight with you, find out you visit adult sites. They find out because they catch you in the act, or because you leave traces. Those risks will not be reduced one jot by using a 15-character randomly-generated password stored in an encrypted vault. That password protects literally nothing, since the site advertises its content. The reputational risk will be defrayed by you leaving no trace of visiting the site or being a member, which is a whole bunch of measures discussed later. The password is the least of your concerns.

Let's talk about passwords.

Device, e-mail and Cloud storage passwords must never be stored on-line. In your memory, or on paper hidden in (your choice of unlikely place here).

First, financial / identity risk. Official document numbers - National Insurance number, NHS number, passport number, driving license number - as well as bank account details, credit card numbers and the like, must never be stored in password managers or anywhere else online. Where are you going to store them? Well, gee, how about on the document or card itself? Which you keep somewhere as safe as it needs to be (the room safe in a hotel, for instance). Only take such cards and documents as you really need when you leave the house / hotel room.

Passwords and challenge responses for banking sites, payment processors (e.g. PayPal), and retail sites where you store payment details (Amazon, for most of us), must never be stored in password managers or anywhere else online.

Do not store your credit card or other payment details on any retail website. (Okay, maybe Amazon and PayPal.) A commercial / charity / academic site gets your name, address and e-mail, and maybe some relevant preferences. (When they send the first marketing e-mail, click the "unsubscribe" link to keep down the spam.)

If you have anything valuable - don't post a photograph of it, or post about it.

Second, reputational risk. If someone gets your social media password, they can post scurrilous content that will land you in jail (these days in the UK, that's a low bar). With that in mind, you may not want to put those passwords in a password manager or similar. This is the first of the convenience trade-offs, and it's your decision. The same applies to passwords for your favourite online forum.

While it's nice to flex about your fabulous life, every week there's a story about someone being caught out by Welfare or HR or divorce lawyers, because of a social media post. The more reputation-sensitive the industry you work in (financial services and the Arts especially) the more your social media content becomes a performative PR exercise. This is a whole other can of worms.

Third, work-in-progress. Documents, photos, files, projects, recordings and any other of your work-in-progress, finished product and records, should be in Cloud storage (Instagram and the like also counts) - that way, you can recover from the loss of your devices. By all means keep local drive copies and take external drive backups as well, if you like, but anyone who takes the computers will take the external drives as well. Choosing suitable Cloud storage is a separate subject.

Next some good news.

The hardware and software industry knows you are not going to use the Internet if you think everything you do can be seen by anyone who can download the right program. So they work hard at providing encryption and security. They are actually so good at it, Governments keep asking them to provide "back doors", which the industry actively resists.

Wi-fi these days comes with WPA2 encryption by default, but if you have older equipment, you should check.

Your computer and phone (these days) have built-in firewalls, virus-checkers, and other such. These are good enough that you never see security hype about having anti-virus programs anymore.

HTTPS is the dominant standard for Internet transmission. Your internet traffic is encrypted from your device to the final destination server, and cannot be snooped by anyone in the middle. Your ISP can see the main page address, but nothing more.

So let's get to the counter-measures. As far as possible, these are setup-and-forget. The best security measures are affordable, invisible, do not require constant maintenance, and discourage all but the best of the pros and the worst of the crazy amateurs. "Eternal vigilance" is not a technique. These counter-measures are for you - how much you trust your partner and children to be sensible and respect everyone's privacy and security is up to you.

Your devices must have a password and / or fingerprint or facial recognition. While it may be possible to do without a password on an account, the OS may prevent other security-dependent features from working with that account.

Your Web Browser should have something along the lines of "Block trackers and third-party cookies" in its Settings menu. For Safari, it's Preferences -> Privacy -> Prevent cross-site tracking. Turn that on if it isn't already. You may need to ask Google for help finding it. If you can't find such a setting, look for an Extension that does the same and install that. This will take care of a lot of the "they are selling your data" issues. (Warning, Google disable your ability to upload images into Blogger if you disable this. Tut tut guys.)

Only visit sites that are HTTPS (or "secure"). (This is almost all of them now.) Your browser should have a setting like "warn when visiting insecure sites" or "force HTTPS" or something similar. Use that.

Use your 5G service rather than coffee-shop / airport / wherever wi-fi's. 4G and earlier are less secure, but better than a spoof wi-fi provider.

Use phone apps in preference to websites wherever possible. Aside from anything else, the app is often easier to use than the website.

Taking your work computer or phone home is a convenience / risk trade-off. Let your employer decide. Anyone who takes your personal devices from home will take the work devices as well. If you must take work devices home, go straight home. Having your laptop stolen in the pub is not a good look.

Open Banking is a convenience vs risk decision. Once someone gets one account, they get as many as you have linked.

Password managers are not a security tool, but a convenient way to log on to low-risk sites that require passwords (typically anywhere that doesn't have payment or official document numbers stored, nor is reputation-threatening: retailers, charities, museums, music streamers, online newspapers, and the like). Especially if you are logging in and out of even a handful of sites every day. Choosing and managing one is a separate subject.

VPNs are a tool to bypass geo-restrictions rather than a security solution. Don't use free ones - since how else are they making money except by selling traffic data? VPNs hide the ultimate destination from your ISP, but the VPN still knows it. Who do you trust more? Choosing one is a separate subject.

Apple's OS X and iOS are terrific operating systems. NEVER use either to do anything remotely shady, because you will never be able to remove all the traces. OS X and iOS are not designed to allow that level of access.

Windows is designed to allow that level of access. Even if you are an Apple fanboy, use a cheap Windows machine for... errr... private purposes, get a decent File Shredder / Disk Wiper, set up a routine to cleanse all the temporary files that get generated by web browsing, and run it every time you finish a session. You may need help with that - this is where you find out who you trust.

What about the (digital) stash? (if you don't know, I'm not telling you) There are online storage services providing end-to-end encryption, a solution that fits well with the rest of the advice here.

If you want some security theatre over-kill, try this guy...

If you like some of his hacks - use 'em.

We end with the harsh truth. A father's biggest OPSEC problem is not that he isn't using a VPN. It's allowing his son access to money so that the boy can lose nearly $6,000 in an online game.  A husband's biggest OPSEC problem is not the length of his logon password, but the deteriorating relationship with his wife that leads her to snoop on his computer and phone looking for divorce-fuel.

Most people wind up in trouble over something digital because someone snitches on them. Someone at work reports something to HR; one of your kids says something that a teacher over-hears and reports; a "good person" whose precious Liberal conscience won't let them not report it; management looking to stitch someone up. ISPs run scans on un-encrypted data to compare file signatures, and report matches. Teachers, therapists, social workers and other functionaries have been turned into informants. And never forget that un-answerable question "Darling, why do you use Private Browsing?" Wives and children are entitled to privacy from you - just ask them - but you are expected to let them see everything you do.

Hackers and "government surveillance" are "stranger danger". Hackers are after entire databases, crypto-currency, and corporates, not random individuals. The security services can barely keep track of the Bad Guys they know about, and don't need to add to the list with mass surveillance - and they have said so. The people who will spend time going after you are people who know you and want bad things to happen to you. We do not want to know that about the people we deal with every day, and so the industry pushes "stranger danger".

A few days after I finished and polished these thoughts, The Algorithm threw this up for me, and his views are very close to mine.

CrowdStrike and Other Security Nonsense

I don’t have CrowdStrike on my laptop.

So it never stopped working.

*smug*

Computer security software is generally a waste of processing cycles, RAM and storage space. In a corporate, all the security should be on the corporate wall and whitelist maintenance. Inside the corporate wall, no security except firewalls which are more or less unavoidable now. On a personal machine, let the OS take care of security.

If the NSA, GCHQ, the Chinese, Israelis, French, or very probably the Finnish security services want to get at your computer, they’re going to do so. Private or corporate. Does anyone not think the Russians and Chineses had Crowdstrike reverse-engineered (or were just sent a copy of the code) within about a couple of months?

The criminals are pretty obvious, though anyone can be caught off-guard, as I was a few months ago until the voice told me they could not cancel a transaction unless I was at my machine to do this and that. My guard came right back.

Personally, I think the companies who let CrowdStrike - or anyone - push an update out onto their live systems without testing it on an isolated guinea pig system first, deserve anything that happens to them.

And it is way past time that all these software companies, from Microsoft onwards, carried insurance to compensate the consumers who were affected. Not the freaking airlines, but the suffering passengers. The corporates can sue each other, but the consumers should be compensated.

Artificial Intelligence and Aluminium Tennis Racquets: Same Opportunity, Different Sport

The PR flacks for ChatGPT probably deserve a bonus.

They have managed to convince a lot of people that anyone who currently has a desk job will be out of work by the end of the year.

ChatGPT and anything like it dips into the existing corpus of pictures / words / facts / academic papers / novels / poems / political speeches / whatever and produces something to your specifications from that corpus.

In response to "Produce a Theory of Everything that does not look like String Theory or Quantum Gravity" it will fail. Unless it tires pulling a Sokal Hoax on us. Likewise "Prove or refute the Collatz Conjecture" will draw a blank, though an icon of a blowing fuse might be appreciated.

"Paint a picture in a style nobody has seen yet" is logically impossible for it to do, since it has to draw on existing pictures. "Write a novel in the style of no previous writer about a subject nobody has written about before" would be equally impossible. A Real Person could attempt both, even if they failed through lack of checking for uniqueness - the point is that a Real Person could produce a painting or a novel that they had never seen or read before, but for ChatGPT it is a logical impossibility.(*)

Anyway, that isn't the point.

The point is that AI is just another tool. Like all new tools it will split a given community into those who can make it work to their advantage and those who can't get on with it. This was also true about spreadsheets, the metal tennis racquet, oil paint, sous vide and a thousand other new techniques and gadgets. AI in medicine would be a tremendous support tool, especially to older doctors who start to forget things or don't keep up (yes, I know Dreyfus used doctors as examples of Experts who could not be reproduced by AI, but after many years I've come to the conclusion that he wouldn't say those things now. He was writing in a medically-simpler age.)

How much of a difference is there between a writer turning out novels within a genre and an AI turning out a novel within a genre? The genre fans want the genre content, not the writer's unique personality (even if the genre is the writer, as in Chandler or Simenon).

There is probably an audience who would appreciate AI-generated genre novels, and one that won't. Publishers will produce both.

For decades, the British television has been producing TV that looks as if it was written to a set of rules and conventions that feel fake to me.

As for AI-Fake paintings, what else would you call a Thomas Kincaid? Or a Jeff Koons, Damien Hirst or Takashi Murakami? Just because these are Human Beings, doesn't mean to say they aren't using AI techniques, or perhaps, that those are the techniques that can be incorporated into AI.

Which actually helps to make a point I'm exploring in a long essay on the Philosophy of Mathematics, that technique rather than specific content is where originality really lies.



(*) This may be debatable, depending on how one thinks of the difference between "originality" and "variations", and at what point a "variation" becomes a "novelty" becomes "original". Those concepts are related but also have differences.

Data Backups and External Drives

It's been nearly two years since I over-thought how I do backups. Technology changes, prices change, and use cases change: reviews are appropriate.

The cause was finding that one of my external hard drives (EHD) would disconnect and then re-connect from the laptop at random. Diskfix couldn't find any faults on the drive, so I have to blame the USB ports. Everyone thinks that the worst that can happen is dropping an EHD onto a hard surface. Also bad is having the EHD fall while still attached to the computer and so putting stress on the computer's USB port. Those things aren't welded to the card.

The only way to avoid plugging and unplugging an EHD is to use a NAS (either dedicated or a headless Raspberry Pi with EHD's plugged into it), or to use another computer with a more reliable USB port. Either way, we're still using EHDs of some quality or other.

Network-attached storage is the first thing one tends to think about. The original use-case for this was a) it was nerdier than using external drives, b) it was not a lot more expensive per MB than external drives, c) people used to download video files from e.g. iTunes before watching them. The nerds have now migrated to using Raspberry Pi's with EHDs attached, which are computers, not NAS. The cost of EHDs has dropped considerably, while the cost of NAS-related hardware and software has gone up. Finally, what is "downloading movies"? Because streaming.

I don't do daily backups. You should, because you are creating GBs of media every day. Or making irreplaceable music in your DAW, with a bank of plug-ins that cumulates to mucho dinero. If I get to that stage, I too will get a NAS and set up automated differential backup jobs. Because I'd be a pro, and that's what pros do. Most of what I do is writing, and I'm old enough to remember when a writer's nightmare was leaving his work-in-progress on the train. Our idea of backup was carbon paper.

Everything I write is in iCloud, which will do for the short-term, as long as I NEVER delete files. iCloud is really for making a file accessible from a number of devices and keeping them in sync: delete a file by mistake on your laptop, and it will vanish from iCloud.

My ripped music is currently on an external LaCie HDD. I made that decision because I only ever used it as a source to load tracks on my iPhone. Then it was switched off. I do have albums on it that I downloaded from Amazon expressly for train music, and sometimes I want to listen to that. I have the LaCie plugged into a ASUS netbook from 2010 or so. Music does not need high-spec as video does. So I would switch the Asus on, count to twenty and then fire up Music Streamer on my iPad to feed it to my amp through the USB port. Then I would have to remember to switch the Asus off before going to bed. Yes, I know, and I've set the Asus up to hibernate(*) at 22:00 and wake up again at 09:00. Its hard drive spins down even on mains power, but I'm not sure the LaCie does. External drive spin-down seems to be hit-and-miss on Win 7, and there's no provision in the LaCie driver. I looked. So that drive could be on for thirteen hours a day, and I think that disc is spinning all the time. It's warm enough. Not what we want.

So I should use an SSD for the music. No moving parts. No activity, less heat generation. I can buy an internal 1TB SSD drive for about £90 and an enclosure for about £10, which is cheaper than the non-sale price of a 1TB SSD on Amazon. For the added speed, reliability and lifetime, that feels worth it.

(When you get the SSD, you will need to initialise and format it with the Windows disk management facility. Format it as EXFAT as b****y Apple have only just added NTFS-reading with iOS 15. If you don't, you will get strange "access forbidden" errors, which will point you in the direction of permissions, and it has nothing to do with permissions.)

Then I can go back to using cheap HDDs from reputable brands (only because I assume they have better quality control in their Chinese factories than the Chinese brands do), and accept that after 3 - 5 years they will need replacing. A 1TB Seagate HDD is currently £38.

I am going to make sure that each of my laptops can communicate with EHDs connected to any other of my laptops. That way, if a laptop USB port goes squiffy, I can backup over Wi-Fi to another machine.

I adopted Free File Sync - to which I have donated - a while ago, and I'm still liking it.

(*) Hibernate, not shutdown. A computer can't wake from shutdown.

That M1 Mac Air Though

My trusty early 2015 Mac Air, has begun doing odd things every three or so weeks. A complete screen glitch in the middle of a YT video felt like an advanced warning. So I ordered an M1 Air.

Then my bank sent me a message saying that it had cancelled the payment instruction because fraud.

I called the number I had for these occasions, and the agent told me that this happened a lot with Apple, John Lewis, Curries and a couple of other retailers. If someone gets your card and orders from those guys, they can do a lot of damage in a short time. They choose those guys because they can resell (on eBay presumably) readily. If they bought an X-E4 from Fuji, it's going to be a while before that shifts. Fair enough, if inconvenient. The bank's agent asked me to make another order so he could make sure this payment went through. Not to worry about the first order, Apple would cancel that when they saw the payment was cancelled.

You can guess what happened. I got one Air delivered and the money taken from my account. I noticed that there was still another payment outstanding. I called the bank. Not to worry, they said, it's just a delay because New Year.

Nope. A couple of days later I got another message from DHL saying they would be delivering my Air. I called Apple (in fact, their outsourced customer handling company) explained the situation, and their agent was totally calm and competent, told me to refuse delivery when the courier arrived, and set a return and credit in motion. All of which went through within a couple of days.

So when the bank cancels your next payment to a retailer, cancel the order as well. Do not assume the retailer will notice.

Anyway.

Everything they say about these M1 Airs is true. It's even smaller than the previous one. The keyboard is terrific, and only Apple make trackpads this good. The screen is just pin-point wow!

The stereo sound system is using some psycho-acoustics that puts voices in the middle of the screen, but then ventriloquises background sound as coming from the left and right walls. Music videos have the music in the middle, though it feels like they are using a `spatial' effect that I last heard on a ghetto-blaster in *cough, splutter*.

After making up an old-school what-I-need-to-transfer-and-install list, I found out about the Migration Assistant. See, that's why we all pay Apple prices. It's not for the kit, it's for the eco-system. It's because they put a tonne of work into developing a program that we would only use twice in the life of the machine, once at the start to import everything from the old machine, and once at the end to export it to the even newer one. Everything came over, and everything works, except for a program called LatextIt!, which is probably searching for some files somewhere they aren't any more.

On the hardware side, I got an Anker multi-adapter for USB-A and card readers, plus additional USB-C ports.

Then I tried ripping some CDs.

My Air told me the USB-A CD drive needed power (the Anker supplies power, the USB-A HDD was running) and I should plug my kit into a USB-C port on the laptop. But. But.

OK. Order a USB-A to C adapter so I can plug the USB-A CD drive into the other USB-C port. Arrives via Amazon Prime in less than 24 hours. Plug in. Yea! Works.

This Air turns into a desktop sprawl if you want to do rip CDs, or read an SD card. The rumours are that the next iteration will have extra slots and an SD card reader, and the Magsafe (which I have not felt the need for so far) port.

I am, however, a happy customer.

Mac Pro 14" vs Mac Air 13.3" - Overthinking the Choice

I thought I'd call it "overthinking" before you did.

My trusty Mac Air has been with me for nearly six years. Intel i5, 4GB RAM, 256 SDD. I mostly use it for writing and media consumption, so it's vastly over-powered for my "use case" (as it used to be called).

But one day, it's going to pack up.

And I want to do other things. Like record some of the music I play on guitar and piano. Which means running Garageband and maybe some plug-ins. I'm not sure about making YT videos, or even why I might do it. Also photography, if I have good enough photographs to do things with. The current Air can handle that, except at industrial levels of production, which I'm not going to reach.

But the Intel Airs are no more. Only the M1 machines. Which are:

The minimum 256x8, 8-7-16 cores (CPU-GPU-Neural) Air for £999.
Trade up to 512x8 and it's £1,199.
Add another graphics core to 8-8-16 and it's £1,249.
Add another 8GB RAM and it's £1,449 / £1,399.

The basic Mac Pro is 512x16 and 8-14-16 cores for £1,899. It comes with Magsafe ("it's not a Mac if it's not Magsafe"), an HDMI port and a SDXC card reader (which my Mac Air has and I have NEVER used). Add adapters for all that to the cost of the Air, and it's another £45.

Whichever, I'm going to need USB A -> C adapters for my external CD drive, and Lightning to USB C adapters for my other iDevices. This is not going to be horribly expensive.

I think 512GB SSD is needed for the extra things I want to do. Plug-ins gobble up space. So £1,199 is the baseline.

The extra £700 to the Mac Pro gets me: a slightly bigger, but undeniably better screen; 8GB more RAM, 6 more video cores, plus better speakers, microphone array, and Face Time camera. Also the Mac Pro has fans, and the Air doesn't. Would I really use any of that?

Don't forget that the real upgrade is from my 4GB Intel i5 Air to an 8GB M1 chip. That's the WOW factor right there. The reviewers of the M1 Air said its video editing was easily fast enough unless you were doing really big files in 4K. If I ever do anything, it will be smaller and in 1080p. So I'm covered.

If I ever do need that extra processing power, I will probably buy an iMac. Heaven knows what I'll be doing though.

Is there any improvement in performance in going to 16GB of RAM? The reviewers say there isn't, except at insane loads I would never reach, because the M1 chip accesses the SSD so fast it's almost RAM. Other voices suggest 'future-proofing' with 16GB. Well, when I got my Intel Air, I wondered about getting 8GB because future-proofing, but 4GB has turned out to be just fine - especially when I dumped out Evernote and Dropbox, which had bloated beyond all reasonableness. Both Apple and Microsoft are paying more attention to making their operating systems use less RAM and work faster. Apple even more so. So I'm taking it that 8GB in the new Air will be as future-proof as well.

So is the luxury stuff (speakers, screen, microphones and iSight camera) and the additional ports worth £700?

Nah. The guy (*) did say that if you don't know that you need a Mac Pro, then you don't. And I've convinced myself the baseline Air will have all the oomph my modest needs will need.

Baseline Air with 512GB plus a Magsafe adapter, and the USB A -> C converters for my heritage USB A and Lightning gear.

OMG did I just reach a conclusion?



(*) You know, the guy whose YT video you watched and thought was good, but now you can't remember who it was. He makes a lot of videos.

Bye-bye Evernote and Dropbox

I got Dropbox and Evernote because I read about them on Rands in Repose. That was a long time ago in Internet Years. Something like (looks up account on Evernote) 8th October 2010. (Stone me guv'nor, 10 years.)

Now I use iCloud instead of Dropbox and Apple Notes instead of Evernote. Why?

Bloat.

About a year ago I looked at what was using my RAM. I have 4GB in my early 2015 Mac Air (blimey guv, I've had that for five years) so this is something I need to do, even though OS X handles memory way better than Windows 7 as disfigured by an FTSE 100 company.

Fire up Activity Monitor, look at the Memory tab, but order by the Process Name column, not the Memory column. This is because apps are now split into a number of processes. For instance, my Brave Browser with one tab open has one main process, two Helper processes, a GPU process, and six Renderer processes going. The main process is 240MB, and all the others take 504MB. That browser is taking 0.75GB of memory.

(I just closed Brave and opened Firefox at my Amazon Wish List. Main process 401MB, supporting processes 337MB, so still 0.7GB of memory. Re-opening Brave on my Wish List gets me 135MB in the main process and 270MB of supporting processes. Which is better. I bet if I open You Tube it bloats out bigly.)

When I did that exercise for Dropbox, it was using over 0.5GB of RAM. Just to sync some files in a Cloud. WTF? Since the iCloud process was already running - because Apple - this was a ridiculous use of resources. It took a few minutes to put the files I wanted backing in the iCloud in the iCloud drive directory, and then I could shut down Dropbox. I even uninstalled it on all my machines.

Evernote updated itself a couple of weeks ago. One of those OMG-they-have-so-changed-everything-WTF-did-they-do-that? changes. Every time Evernote does a major change its users lose control over more and more things. This one changed the fonts and graphicalised everything - like Office 2016 did and Apple Notes is. The feel of how the type appears on the screen in response to the keyboard is different. It was when I navigated to a large note which choked the viewer I began to think something was wrong. Yep, Evernote, which used to be whip-cracking smart and fast, now had a Renderer process that took around 400MB. The whole lot took up around 0.7GB, and Evernote has never been great at memory management.

Notes is running in less than 84MB, and even Pages takes only 153MB. How the **** does it need a 400MB renderer process to do what Apple does in 84MB? Makes me wonder what other dumb things they did.

Export all my Evernote Notebooks. Load them into Apple Notes. Close Evernote. I had cancelled my Premium membership a few months previously. I haven't uninstalled it, and I still have an account I can access via a browser. Moving to Apple Notes is one-way: it only lets you export to pdf files. Because Apple.

I blame SSDs. I see this at work, where people who don't have problematical unknown programs like SAS Base (oh, wait) are getting Windows 10 machines with SSDs and 8GB of RAM. Half of that is taken by Windows and security bloatware - McAfee and Tanium Endpoint plus all the supporting Windows services - so users are still trying to run their real programs in under 4GB. Everyone with one of those machines is agog at how fast they are - what do they have at home? - but that speed comes from the SSD, not efficient and effective use of the RAM. The fast disk swapping SSDs allow seems to make people lazy about RAM use and program bloat - who cares how big it is, it'll swap real fast on an SSD, and the clock speed will cover up the lousy coding.

So deeper into the Apple eco-system I go. Because third-party suppliers got lazy with their coding.

My Western Digital Network Drive Has The Green Flashing Light of Death!

Western Digital My Book Live drives can get the flashing green light of death. They boot up, start talking to themselves and don't stop, locked in an endless read-write loop. Heaven knows what they are reading and writing, but that's what they are doing. Hold the device and feel the disk rumbling away to itself. A reset does not help. The drive becomes invisible to the outside IP world, so I assume it's failing to load some of its software. Good thing I did a backup of it in March. It was pretty darn old: I must have bought it in 2013 or so. (You do not know how long I just spent with my Calendar app trying to work that out.)

The My Book Live was a 2TB network drive: it had ethernet connectivity. Its main use was to hold my iTunes files so I could stream on my other iDevices, and load up my iPhone with music now and again.

They don't make ethernet drives like that anymore. Now the choice is between USB3 portable drives, USB3 desktop drives, and NAS units. Or Cloud storage.

Cloud services reserve the right to scan your content for copyright infringement and other naughties, and delete at will. Not that I have any content like that, but there is no appeal to Apple, so why take a chance? As well as that, I would be at the mercy of Talk-Talk. It so much as rains for twenty minutes and my alleged high-speed broadband drops to half-pace, putting them in breach of contract. I can live with the 20 / 7 Mbps that results, and it does seem to sort itself out after a few days. On the other hand, HDD's have mechanical failures and I bet NAS's get OS problems as well.

Nevertheless, it's worth having the costs.

5GB free at iCloud 2TB at £6.99
iCloud = £83.88 / year
2TB external HDD approx £70 = 10 months of iCloud
4TB Synology DS218Play £400 (2TB storage with Raid 1) = 57 months of iCloud = 5.5 external drives
2TB external SSD approx £320 = 4xHDD or 46 months iCloud
1TB external SSD approx £150 = 2xHDD or 21 months iCloud
512GB on Air £200
2TB on Air £800

(Note that Apple internal SSD storage is more than twice the price of external SSD storage.)

We text-people don't need lots of storage. It's photographers, video-makers and serious musicians who need serious storage, fast graphics cards and 8-core processors. If I was one of those, I would already have a NAS. Not all NAS are made by Synology, but it's the only name you hear in consumers circles. I could in theory get away with 500GB of storage: my music is around 120GB, my photos including Photos Library are about 15GB, and Calibre back-up is about 5GB.

Of course, using drives to share music is just so 2014. So audiophile. I mean, do I even Roon? https://roonlabs.com I dream about Roon-ing. Not so much having a subscription, but leading a life that would make Roon necessary. If you dropped a lifetime subscription in my lap, I would thank you very much. But I just can't justify it.

I listen to music on CDs. I watch movies and foreign cop shows on DVD. I stream music from Spotify, and I have been hitting the Above and Beyond live concerts on You Tube recently. I stream movies from MUBI and Curzon Home Cinema via my Apple TV. I have no desire to download anything.

If I want to stream music in other rooms, I can. It's called Spotify + Sonos. Spotify has all the music I do and a bit more. Without that I would have needed a NAS.

Which is cross-product substitution for you. £400 of Synology is 40 months of Spotify. This is a no-brainer of a decision. Unless you think you can tell the difference between a 320kps stream and a 4Mps uncompressed stream.

A NAS can provide back-up especially if it's RAID 1 or above, but one should still backup the NAS now and again, and that means portable drives. Which I have already.

So this is what I did. I re-set my Music directory to my Air and imported the files from the last backup. That leaves me with 60GB spare after clean-up, which is enough. Then I copied all my content directories onto a portable drive. After that fiddling around, I decided to find a proper backup / sync program, and found FreeFileSync. It's pretty easy to work out how to use it. I trialled it, it worked nicely, so I gave them a donation. It's now a whole lot easier to do backups.

Let's face it, I barely used the My Book Live. So replacing it with a more expensive NAS would be silly. Next time around I will spend money on additional storage in my iDevices instead.

Mac OS Big Sur - Stop Futzing Already

I have seen at least three You Tube reviews of macOS Big Sur. Which is supposed to be an operating system. 85 new features later, nothing about the operating system. Emojis, searching in messages, some pointless new options in Music, face recognition in the Home App, and a bunch of other stuff in the Apps.

What about changes to the actual operating system? Does it use less RAM than Catalina? Does it load apps faster? How about the firewall? Improvements to the music playing software? Does it sandbox like iOS does? Any new features I might like to know about as a developer?

When the **** is Apple going to give us proper window management, because Windows still does that better than anyone. When are they going to detect all the hard drives and other equipment on my network, and re-connect to them, like Windows does, even if the devices aren't in Apple's ecosystem? And WHEN will they get the album art in Music sorted out? It works if I have all my files on my Air, but on the NAS? Half the images would go missing and the loading time was awful. My Sonos and File Manager apps on the iPod Touch did a better job. And please can we have the same wonderfully functional File Open / Save dialogs that Windows does so much better.

I get the feeling that Apple do a lot of hard-core OS-stuff to improve performance. They should crow about it some more. But maybe they don't, because they can fix it on the chip, in a way that Microsoft can't.

It may also be the reviewers, who assume their audience want to know about the awful battery icon more than the way Apple have (maybe) improved battery life by doing something with the battery management algorithm. I'd rather know about the algorithms. I can see the awful design for myself.

What makes a computer operating system is flawless multi-tasking, powerful peripheral management, and simple multi-user management. If I can't run Nora En Pure on my browser on one screen, while typing in Evernote and downloading some Amazon music, while rendering a movie clip, then it's not a real computer. So I would expect to hear about improvements to those things as well.

But OMG the desktop art on Big Sur! I suspect it's going to be a while before this late adopter adopts it.

Cloud Drives, Backups and Time Machines

The things I over-think! How difficult is to store your digital files: photos, books, music, movies and documents?

In the old days we put books on shelves or in boxes; photographs in albums, or in boxes; records and CDs on shelves, or in boxes; and any documents we generated in files, or in boxes. How easy was that? Then computers came along and made everything simpler by making it more complicated. Everything was stored on the hard drive - passing over the days of floppy drives and diskettes - but what happened if the hard drive packed up? Hence backup which meant taking a copy of everything on another disk and storing that somewhere safe. Real Men did backups daily, weekly, and monthly. The daily and weekly ones were cycled, the monthly ones kept forever. On tape. Lots of people didn’t do backups, and every now and then they wrote to Jack Schofield at the Guardian about the disaster, and Uncle Jack scolded them lightly.

Backup is copies you make on a storage device and then leave alone unless disaster strikes, when you copy them back to wherever you copied them from. You never edit backup copies. Ever. You should keep at least two generations of backups.

Cloud drives in particular and synching services in general, are not backup. Do something dumb on one synched device, like delete everything, and all the other synched devices will lose everything as well. iCloud, One Drive, Google Drive and all the other services have this feature. To make up for it, some of them add a versioning, recovery or history feature that gets over this. When I tried using the versioning on Pages on my 4GB Mac Air, it choked so bad I had to turn it off.

A Cloud drive is what you use when you want to access, edit and save back your work-in-progress files from a number of different devices. Put your photos up in the iCloud and you can regale everyone with your holiday snaps on your iPhone, iPad, Macbook or even Windows device. Ditto music, though why you would when you can get it from a streaming service is another question. Unless it’s a recording of your band, or a track you’re working on. Same for videos.

Cloud drives offer you the assurance that even if your device gets lost, goes crazy or gets broken, you won’t lose your files, which will be there waiting for you to re-connect to them after you have bought a new device. But that’s not backup.

What to backup on? Storage is the physical medium on which you place the files. Internal hard drives, pen drives, external SSD’s, Network drives and NAS, cloud drives, tape, CD’s. You can also store files on a disk in some company’s cloud service, effectively renting a hard drive from them. You have to backup to a physically different device, which you should then keep somewhere else. (So if someone drops something on the computer, the backup is in another room.)

How to backup?

Both Apple and Microsoft offer a continuous backup service: Time Machine for Apple and File History for Windows 10. Setting these up requires an external drive that can be configured for Time Machine / File History: that’s not always a given. Note also that both those are complex pieces of software. If you take your Time Machine disk to someone else’s computer, good luck if that other computer is a Windows 7 device. Time Machine and File History are good insurance against something stupid happening to your files, but not against something stupid happening to your computer.

A good backup sits on a drive that any computer can read and copy from. Since every operating system can read Windows file system disks, put your backup on a NTFS drive, in other words, get an off-the-shelf external drive from Amazon, PC World or any other major retailer. I am not going to get involved in USB types at this point. Roughly, make sure the USB port on the drive looks like one of the USB ports on your computer. Or use a NAS, which plugs into the router so you connect over Wi-Fi.

How to get the files over? There are back-up programs available that have nice user interfaces, but there are also much simpler options. On Windows 10 (and any other version) look up the xcopy command. This is a thing of wonder. There seems to be an OS X version called rsync that does the same thing. Both these tools only copy new or updated files relative to the destination folder. So the first one is painful, but the following ones can be quite fast. At work I use a Powershell script to sync my local drive to a personal network drive at the end of each week, or whenever I’m taking the laptop to and from work.

What to backup? Your ‘data’ of course. You may want to backup the software you’ve downloaded (the files in your Download directory) so you can re-install it if need be: this may be sensible if the latest versions are not compatible with your machine or operating system (OS X users will understand).

So let’s take a work-in-progress document. It’s in my iCloud drive, and hence on my laptop, maybe another device, and on Apple’s servers. iCloud is a synching service, not a backup. So I need a backup in case I do something stupid on iCloud. The laptop drive is the first stop: I save from the iCloud drive to my Documents folder daily. I backup my Documents folder up to my NAS at least once a week. And I backup my NAS to an external drive every month, or straight after I add significant amounts of data to the NAS (by ripping a bunch of CDs). I have two external drives and rotate them.

A professional photographer, video-maker or musician may want to take archive copies of the original files before even thinking about importing to the editing software. In the future it will be those archives that scholars want, not the finished product, which will reflect the fashions of the moment.

Having reviewed all this, my shortcomings are not so bad. I’ve understood the correct role of cloud drives, and the limitation of the Time Machine / File History technology.

I need to backup my Mac Air Documents and Photos folders to the NAS more frequently than I do. I’ll do that with rsync. I need an external drive to backup my NAS, and I’ll do that with xcopy. I could do with tidying up the NAS and taking the plunge of doing a soft reset because I forgot its admin password (!).

(Later). I sorted out the NAS. The filesystem works fine, but the bit that runs diagnostic tests and sets up a Time Machine volume seems to have gone astray. Or it isn’t compatible with Catalina (I updated to Catalina after finding enough people who said it worked fine on their Early 2015 Air. It does.)

Bye-bye Tidal, Hello FTTC, Bye-Bye Yolt

The Highly Significant Birthday is approaching. I have booked the week before and after off. In case I get some uncontrollable emotions, or just don't want to get out of bed. Sometimes, though, I'm not sure I can tell what is actual emotion and what is pollen and too little sleep. So my posting is going to be a bit erratic.

In other news, I dropped Tidal. Listen, I searched for "Shoegaze" and it came back with about three entries. A colleague at work searched on Spotify, and it returned pages of the stuff. So I cancelled Tidal and signed up for Spotify.

The king of shoegaze compilations on You Tube seems to be the unlikely-moinkered Tabitha Mustang. If you've never heard any shoegaze, try this



I finally gave up on my old-school copper broadband service from Talk-Talk, and upgraded to their FTTC (fibre to the cabinet) service. Which has churned out 40Mbs down and 10Mbps up so far since. I even speed-tested it, and got pretty darn close, even though the route included the wireless-ac to my laptop. 40Mbs down by the way, is slightly more than what we old telecom folk used to call a DS3 (34Mbs). Back in the mid-1990's I am told, selling a DS3 across the Atlantic meant celebrations involving champagne and nightclubs. Now every home can have one.

And then one day my Yolt app demanded my passport number and other KYC (as we in the retail banking trade call it) details. And it would not let me get to the control panel without it. I was upset by this, because I don't like software strong-arming me, found the Contact Us email on their site, and asked them to DELETE MY ACCOUNT several times in all caps. Which they did without any fuss. My suspicion that they were about to launch actual banking services via the app was confirmed a day or so ago in the news.

And over Easter, I listened to Parsifal and Gotterdammerung on Spotify. Probably not quite CD quality, but it confirmed to me why I'm not rushing to get Wagner in my collection. The first movement of Parsifal is musically astonishing, whether you understand German or not. But the second act is a lot of singing, and the music probably means a lot more if you know what Kundry just tried to suggest to Parsifal.

Fixing That Missing Artwork in iTunes

Macs make a lot more sense when you think of them as designed to be stand-alone. OS X can do networking, but it doesn’t really get it. That’s partly because of the UNIX everything-is-a-file philosophy: Unix doesn’t see networks, it only sees external drives. Windows sees networks and drives on the networks, which sounds as if it should be the same, but really is not.

Windows exposes the filesystem to the user in its programmes, and asks the user to make the filesystem their friend. In Windows you open a file by navigating through the filesystem in the File Open dialog or by clicking on a name in the Recently Used Files list. The only time you see icons representing your files in Windows is a) when you’re looking at the directory in Explorer, or b) when you’re using a programme that’s modelled on something from OSX, such as Calibre or most music management programs.

By design, OS X hides all the nasty OS stuff - such as the filesystem - from the user. This is usually done by adding an intermediate layer between the user interface and the filesystem. In an OS X programme the user sees a friendly icon and clicks on that: the programme looks at a database file - usually in XML format - that tells it where the file is stored. This works wonderfully until the user does something with their music files directly via the filesystem, say, with Finder and then everything goes to pieces. Because the database behind the cool interface has not been updated and indeed can’t be updated with changes made directly via Finder or with Unix commands on the Terminal.

OS X wants you to deal with your music collection ONLY through iTunes or a similar programme. That is the only way the database is kept up to date. It’s also the only way that the very quirky file permissions that iTunes uses get handled correctly. It’s possible to alter the file and directory permissions to 777 (all access to everyone) in Terminal but not have that show up in Finder, or have iTunes behave as if the files and directory have permission 777. (Oh yes, I have done this.)

(Windows programs assume that you’re going to do all sorts of things, so they have a ‘Watch this directory’ function that scans and updates the cool interface when you start the program. That can be kinda annoying at times. But it’s swapping one annoyance for another.)

Suppose you copied Volunteers by the Jefferson Airplane from your Macbook iTunes directory to a backup directory on your NAS. When you later build an iTunes library from this NAS directory, it will add the Volunteers album and files it finds, but you won’t be able to modify the album info, and the artwork will get lost. What’s happened is that the file and directory permissions have become mis-aligned - not in Terminal world, but in the parallel universe of iTunes / Finder permissions.

So how do you get the artwork back, and become able to change the album info?

Here’s the fix that occurred to me…

1. Create a directory in the Music section on the computer that has the iTunes you are using. This will typically be the SDD of your Mac Book / Air / Pro. I called mine Temp. It doesn’t matter.

2. Copy the files from Volunteers (or whatever) into this Temp directory.

3. Use iTunes to delete the original Songs and the Files.

4. Check that the directory for Volunteers (or whatever) is also deleted, as it might not be if it has artwork files or other stuff in it

5. in iTunes, do File -> Add to Library and choose Temp. iTunes will now do its stuff.

6. Now you can add the album artwork and make whatever other changes you want, and it will stick.

7. Delete the files in Temp.

8. Wash. Rinse. Repeat.

9. Works On My Machine and NAS. YMMV.

It sounds clunky, but it soon become muscle memory. Do it as a background job when you’re writing blog entries or something, and you’ll be all caught up before you know it.

Ripping The CD Collection

I’m not an audiophile, but I want the music to sound right. What comes out of my iPhone through the Bose QC20’s noise-cancelling earphones is right for the purpose, but what comes out of the headphone socket through my hifi is muffled and tiring. Same signal, different context. Hence all my fussing around with Dragonflys and Jitterbugs earlier this year. While not quite up to the DAC in my Marantz CD player, the combination is close, and the sound isn’t tiring. And I fall asleep to music from the Bose Colour Sound fed via Bluetooth from my iPhone streaming via WiFi from the NAS.

I was not ripping the CD collection for backup. That would have meant copying the discs. And then buying another NAS to backup the backup, because NAS can fail more frequently than a CD. Anyway, there is a CD backup service: it’s called Amazon, or Foyles, or any number of other online retailers. This won’t work for compilations of obscure composers by performers whose careers never lived up to their early promise (British Piano Music of the 1980’s where art though now?) but then, well, how often did you listen to it when you had it? Classic performances that you really want tend to be re-released. Or surpassed.

I ripped the collection because I stream more than I do, and may stream more, especially with the Sonos in the front room, and because I do put some of it on the iPhone and the Nano for portable use from time to time. And I’ve only got 128GB on my Air SDD. Which is my way of explaining why I used the iTunes default of 192kps M4a.

Why did I use iTunes? For one thing, Hans Beekhuysen mentions it as one of three which are decent rippers. Since one of the others is Roon, that’s a pretty good recommendation. iTunes is good at getting song titles and parsing the artist / composer, no worse than any of the others at getting album art (I tried Clementine: it won some, lost others) and once you accept its quirky little ways is pretty good at managing the library. It’s not Roon. But neither is the price.

So the workflow looks like this:

0. Create an Album Artwork directory on your Mac. Open a new music library in iTunes and point it at your NAS. Save.

1. Put CD into drive

2. Choose the album title that a) isn’t in Japanese, and b) doesn’t say it’s Disk 3 of some ‘Best of Bach’ collection when your CD is a stand-alone

3. Let iTunes do its thing

4. While it is, if you ripping a jazz or classical CD, copy the CD title, paste it into the Amazon search box and see if they have a decent copy of the artwork. Almost always they will, but if not, use Google. I did so on less than one in thirty CDs. Paste the CD title into the the ‘Save As’ name and save it to your Album Artwork directory (*).

5. When the CD is done, eject it, and right-click for Album Info. Here I put in the artwork, change the Album Artist to the composer for classical music, and get rid of the [Disc 1] that often appears in titles. Sometimes, as with the 22-CD Stravinsky set, the disk number is useful, but not for double-albums.

6. Untick that damn Album is compilation of songs by various artists box. Otherwise it winds up in a Compilations directory. And you won’t find it when browsing with File Explorer.

7. Press OK and find something to do while iTunes does its thing with the music files over the WiFi to the NAS.

It’s tedious. It’s best done while doing something else, pausing to deal with the album info, eject the disc, put another one in. I did it in batches of about twenty, one day at a a time, until it was over.

8. Review the results and edit. How much work you put into changing what iTunes (or any other organiser) found is up to you. Filling in the missing artwork, un-compiling compilations that aren’t really, making sure that J S Bach is spelled and spaced like that in all the albums so it’s easier to find when browsing outside of iTunes… just how anally-retentive are you? (Turns away as if this doesn’t concern him…) The day you see me changing genres, I really will have nothing to do.

9. Accept that the music catalogue is a case of progress not perfection. I’m going to make tweaks every now and then when I notice something.

Because some of the files I have were ripped earlier, under different versions of iTunes, there were permission issues, and I needed to refresh the library in the way described in a future post. It was worth it. A library with every bit of cover art and all the double-albums put together is a thing of delight.

(*) WHY THE FRACK DON’T JAZZ and CLASSICAL CD’S HAVE ARTWORK? I load a progressive house CD, it has artwork. I load a Mahler box-set, I have to get the artwork myself. Digital music libraries and organiser programs have been with us for over a decade, and iTunes, Roon and all the others aren't going to disappear. How difficult can it be for a record company to package all its artwork up and send the zip to Apple? And how difficult would it be for Apple not to charge the record company, in the name of giving us all a better experience? Not even Amazon restrict the number of times we can find and download artwork. (Because they’re smart: every time you get some artwork from them, they get some more goodwill, and you might buy something.) Music industry, get your freaking act together on this.

Writing a Python Program: Tools

Real Developers use vi at the command line and have memorised the every single Python library. Actually, Read Developers probably don't use Python. This is for the rest of us. There’s no question of anyone knowing all the Python libraries by heart, let alone by muscle memory. It’s impossible for any one mere mortal to know the Excel VBA object model and all the constants. That’s why VBA has Intellisense, which Microsoft patented, forcing everyone else to provide a slightly cut back version called ‘auto-complete’. PyCharm has autocomplete, but not the full-fledged Intellisense. Auto-complete doesn’t walk you through all the parameters for a function, nor offer values when the number of options is less than about five. Microsoft spent all that money developing Intellisense because they knew if would pay back in productivity and user loyalty. The minimalist Python IDE is IDLE, which doesn’t have a visual form editor. To find the typos in your code, you have to run the program and then deal with the messages from the python debugger. That’s what makes it minimalist. PyCharm Community Edition has a neat feature where it puts red line in the right of the code screen against the lines it thinks are wrong in some way, thus sparing you doing debugging runs to find the bits where you forgot to put ‘:’ at the end of a ‘def’. This helps, but if the interpreter spots something else wrong while your code is running, you have to correct it and start over. This is when one realises that the VBA editor / debugger is a thing of utter wonder. Being able to do on-the-fly code correction while debugging is like having a superpower. No other IDE provides it. A search for the reason brought up a comment from someone on the original VBA project to the effect that the trick was keeping track of all the threads and re-aligning them when the user had re-coded and started from some earlier point in the program that they halted. That is nowhere near as easy to do as it is to say. While it doesn’t take much time to stop, edit, and re-run code from the start, as PyCharm and all the others force us to do, compared with the Microsoft Way, it just feels clunky. To me. I'm going to try Visual Studio Code from Microsoft next time I do something. The demo looks interesting. Anything to get on-the-fly debugging again. But then Real Developers don't need to debug. (PS: It's too hot for me to keep up my usual schedule. Normal service will be resumed when this hot spell breaks.)

How To Run a Python program? And Why Is It A Good Thing?

Isn’t that a question with an obvious answer? Not so much. Answering it takes us to the heart of how Python works.

Write a Windows program in a language such as C#, and the IDE compiles to a self-contained .exe file (as much as any .exe is ‘self-contained’ when it makes calls to all those Windows DLL’s). My reflexes have been developed on languages like that: compile and run.

So my first thought, having got code that worked as it was supposed to, was to make an .exe file. It was then I found out that PyCharm doesn’t compile Python scripts as, say, a C# IDE does. If you really want an .exe file, you have to find a third-party application to compile the scripts and wrap in all the dependencies. The most popular way of doing that is to use Py2Exe. That only works on Python 2.7, and hasn’t been upgraded to handle the 3.x versions of Python. Lesson in there somewhere.

The lesson is this. There are two Pythons. One is the scripting language. The other is a program that runs those scripts. That’s why the command line prompt is ‘python mystuff.py’. It’s telling the Python program to run the Python script in ‘mystuff.py’.

The Python program on Windows converts a .py file into something Windows understands. The Python program on OS X converts the same .py file into something OS X understands. So .py files are portable. mystuff.py will run on any computer with a Python program. (Given Python version compatibility.)

Browsers work in the same way. You download a web page full of all sorts of HTML and other programming junk, and the browser interprets it all in terms the operating system understands and throws the result at the computer to run.

Which means that ‘everyone’ has to have a Python install to use a .py script. Well, okay. Would you expect to be able to ‘run’ an Excel workbook without Excel? You need Excel to be installed.

So how do I run the program? Set up a .bat file with the command line in it, and put that in the Start Menu. Click on it, and it will start up like a ‘real’ .exe program. Which is all I need.

Learning Python by Writing An Application (2)

I haven’t written high-security web applications that link to secure back-end databases, so I’m not talking about those. I’m talking about regular, runs-on-your-computer applications with a UI.

Real programmers know that their command-line utility never hangs and has a try...else to make it stop at the first sign of trouble, so they don’t need it to talk to them. I hate it when a program sits there seemingly doing nothing, I want some feedback. There’s a trade-off between speed and user dialogue. No, not progress bars. Progress bars require sizing the job first, so that the progress can be shown as a percentage of the total. Ever noticed that if the number of files you’re copying is large enough, Windows suddenly takes an age doing some mysterious preparation? That’s what I want to avoid. The only way round that is a running commentary. I don’t want much, maybe something like

Scanned N directories, copied M files to (destination)

updated every 10 files. When it’s syncing, I want to see

Scanned N directories, deleted M files from (destination)

(Real Java Programmers would write a log file of every action, along with the time it happened. I’m so glad I’m not a Real Java Programmer.)

Tkinter is notorious for updating at random. I don’t think this is Python’s fault, but the way the OS and hardware parcel out the tasks. So by the time Tkinter gets a moment to display the first confirmation, the whole task might have been done. To get round this, I called my first thread! I’m a VBA-basher in my day job so threading is something exotic and mysterious, like (insert inappropriate analogy here). It solved the Tkinter delay problem nicely.

My program has a button to choose the source directory, another to choose the target, one to Start and one to say Stop. Think about the UI logic. The Stop button should not be active unless there’s copying or syncing going on. The Run button should not be active unless there are valid source and destination directories.

What happens if I copy a file and there isn’t enough spare space on the target disc? At the very least I’m going to get an error message, and so I’d need to handle that. Or I could test for spare space and stop copying when I reach some kind of limit. That’s what I did: read in the spare space at the start, store that in a variable, subtract the copied file size from that value, and stop if the file to be copied is within some limit of the calculated space. Using the constant cuts down on filesystem calls.

What happens when there’s already a file with the name of the being copied BUT the size is different? The program needs to add a suffix to the filename, but then it has to test that that filename doesn’t exist, which sounds like a loop. And don’t forget to put a numerical limit on the number of attempts just in case someone used a silly file-naming convention.

A lot of programming is testing for and resolving edge-cases. For instance, the destination directory should not be the source directory, nor a subdirectory of the source directory. I got the first condition on the first pass, but only thought of the second when writing this. (Documentation reveals many improvements.)

This is all exception and edge-case handling.

In my experience, the actual business functionality, the code that does what you want done, is about ten per cent of the total code. Variable declarations and syntactic seasoning is about another ten per cent. Setting up, controlling and making the internal logic of the UI consistent, will be about twenty-five percent, depending on how complex it is.

All the rest is handling edge cases, data weirdness, potential errors, null values, and other stuff that might trip up the program.

I’m not kidding.

If half your code isn’t testing that things that need to be there for the next step to work are actually there, or for missing values, forcing stuff to be strings when the function expects strings, putting limits on loops in case something you have never thought could happen does happen and the terminating condition is never reached... if you’re not doing all that, and you’re not writing in sensible defaults for when the user has to make a choice, and you’re not testing for when the user does something really dumb because they’re distracted, or walks away for five minutes, or all those other things you don’t want users to do because then your life gets difficult... then you’re going to produce a program that people are not going to use twice.

The difference between a professional and an amateur is that the professional writes all the edge-case and exception handling they can think of. An amateur thinks ‘Well, the user will just have to...’.

Don’t forget to put in a default drive for the source and destination directories, or the getdirectory() function will dump the user deep down a file tree it will take a boring time to get out of and get to where the user should have been in the first place.

Programming isn’t about writing code to do stuff. That’s the easy part. It’s about writing code that makes sure the user and the computer don’t get in the way of the stuff getting done.

Learning Python by Writing An Application (1)

I don’t read Slashdot as much as I used to, and so I have no idea if people still ask “What’s the best way to learn a new (programming) language”? The answer was and ever more shall be: read a decent introduction to it, then write an application with it. Not toy examples, but something you will use. Why? Because then you will have to get to grips with the file-handling and string-processing functions, as well as some basic UI functionality, as well as the specialist libraries you need for the task.

That’s what I’m doing for Python.

Assume you have a bunch of files in a high-level directory called MyFiles, on the drive of your working computer. You want to make a backup of this from time to time. If it’s small, the simplest thing to do is copy MyFiles onto an external drive. With a lot of files that could get time-consuming, so maybe a backup program will do. Except that backup programs often create their own special file structures, databases, or at the least, zip everything up. I don’t want to use a special program to manage the backup: I want to use Windows Explorer.

What I need is a clever copying program. I tell it the source directory and the backup directory, and it recursively scans down the source tree, copying only the newfiles and the new directories it finds. Then, if I’ve asked it to, it removes any files from the backup directory that I deleted in the source directory since the last backup.

That’s the sort of thing that Python is good for. So I downloaded JetBrains’ PyCharm Community edition (because free) and started. I know how to program, and I understand basic Python syntax, but what I don’t have is a lot of familiarity with all the libraries, and especially Tkinter, which is what I needed for the user interface.

How does one cut code without knowing all the libraries? Slowly, with various reference websites open in the browser. That’s what manuals are for.

This illustrates the difference between being a programmer, and being a productive (insert name of language here) programmer. Because I’m a programmer, I can break a task into activities the computer or user needs to do (‘get the spare space on the destination drive’), and it usually turns out that the tasks I identify correspond pretty closely with the functions available in the system libraries. Programmers tend to think alike.

I understand control flow, arrays, variable types and declaration, threading, and use of functions to make code readable and maintainable. (I know, you want to hire me already.) I learned that writing FORTRAN, C, and VBA. Writing long chains of macros and the queries for them in Access is programming of a kind as well, as is constructing chains of raw SQL. It’s all about organising the resources of the language to get done what you need to get done.

Faced with a new language, I look for how those things are done, as well as any other cool tricks or idiosyncrasies it has. And, because I’m a programmer, I Read A Freaking Manual first. (This approach will not work with LISP and other such languages. Those really do need to be learned from zero, or you will miss the point.) On a meta-level, I’m Dreyfus-Proficient. That’s the engineering training. I’m just not slick.

All that said, I wouldn’t go for a Python job because one of the requirements is muscle-memory of the main system libraries, and familiarity with whatever the job-specific libraries are. I don’t want to be one of those guys who bluffed about their skills.

I have the sneaking suspicion that I’ve written VBA in Python, and that a Truly Pythonic Programmer would do the whole thing in two lines of code, one of which is a recursively self-referential array of filenames, and the other of which would implement a beautiful wxPython interface in less than twenty characters.

‘Pythonic Programming’ is a Thing. It’s a style. TPPs can do in a line what RJPs need six hundred lines to do. But becoming a TPP takes serious study, not to mention the right examples.

Next post, I’ll talk about the program.

Upgrading the Windows Computer

Every now and then I think about upgrading my Windows laptop. At the moment I have a Samsung that may be seven or eight years old. It has a 17.5 inch screen, a 2GHz Pentium, a 500GB HDD, 4GB of RAM and a basic video card. The body is plastic and the keyboard is only for light use, and I mean maybe ten minutes at a time. It does the job I want it to do, and I suspect the video card is a part of that.

So I’ve been looking at laptops and mini-PC’s.

At the bottom end of the price range, say about £300 or so, are Celerons and Pentium with 2GB of RAM. At the other end are 4k video editing machines for photographers. These have GeForce 1040 or above video cards, 8GB+ of RAM, an i7 HQ, some SSD and a 1TB drive or a lot more SSD, and USB 3. / 3.1 to connect external HDD’s with decent transfer rates. MacBook Pros and Dell XPS 15’s are get a lot of mentions. (High-end gaming laptops are way over what I need.) These machines have good keyboards and aluminium unibodies (the Dell is almost an aluminium unibody).

In between is a mass of i5 / i7 U-series machines, with or without graphics cards, random amounts of SSD, rarely an HDD, random combinations of USB / HDMI / VGA ports and build quality that ranges from awful to okay-I-guess. These can vary in price between about £600 - £1,000. The only way to make sense of these component salads is to assume that Wintel manufacturers design a high-end model, a low-end model and a mid-range model, and all the others are put together from excess parts and left-overs.

The decisions are much simpler than all those fancy specs and combinations make it look.

Want to browse the net, do text-based work, basic photographic editing and adjustment, and send e-mails? But don’t do 1080p and upwards video-editing or scientific computing? Take a serious look at an iPad and an external keyboard.

Get a Macbook Pro if you want to do iOS, Mac or UNIX / Linux development.

Get the high-end £1,100+ video-editing capable machines if you want: the aluminium body, high-quality keyboard and sharp screen; a large HDD; to use Adobe Creative Cloud or the equivalents on 1080p and 4k movies.

Want to do lots of calculations but not much graphics? (Very rare). Then you can get one of the component-salads with an i7-HQ, 256GB of SSD and integrated graphics.

Sustained daily typing and use - because you’re an author, or journalist or other content-producer? Consider a Mac Air or one of the top-end machines. (The price difference is not that large, especially if it’s how you make your money.)

Anything else? Get the mid-range Wintel.

It’s the build quality. Once you’ve had Mac, you can’t go back. At work, they hand out POS HP’s with Win 7, VGA adapters, and a keyboard with the Page Up / Down / End / Home keys in the wrong place. But that’s institutional companies for you. Cheap. The Adobe Creative Cloud Suite user in the family does so on an MSI gaming machine with an i7 HQ and 16GB of RAM with a 17-inch screen. It’s wicked fast but it’s got that corporate cheap build feel. And he doesn’t work for a big company.

I don’t want to come home and use something similar to the junk they give us at work. It’s my home, not an office. Computers are one of the things I’m willing to Pay Good Money for. (Not Silly Money, but Good Money.) As a tool to do a job. And the video-editing performance laptop is a tool for a job I’m not going to do.

The outsider for my needs is an Asus mini-PC, which has the 1TB drive, an i5-7U series, a mid-range graphics card, some SSD, Wireless-AC, and ports out the wazoo. It can drive two external screens, which is a nice-to-have I’ve wanted for a while. It will be an ace media centre, but would need hi-fi to make nice sound, but then, so does a laptop. It costs £650 and already I have an external screen (the TV), a mouse and keyboard. At a pinch I could get a 21-inch monitor for about £150 and work on a table if I wanted to use it as a computer. Back to the future.

PS: I didn’t do any of this. I did something else instead that didn’t involve spending money. I’ll talk about that later.

Toy Examples: How os.walk Works

One of the many things a programmer needs to do is walk a directory tree and do stuff to the files and folders in it, even if it’s just list them. This is a recursive exercise and those can be mind-bending to code, and if done badly can mess up all sorts of low-level things. It’s best left to the kind of people who have actually read Knuth. Fortunately someone on the Python project did, and they gave us os.walk. (It’s in the os module, and is called walk().)

Unfortunately, the almost identical explanations of how to use os.walk mostly miss the point. All of them - that I’ve found - print out a directory and file listing. Which is not what I wanted to do with it.

When you execute os.walk(starting_point), for a directory called ‘starting_point’, it returns a triple consisting of: the path for starting_point a list of the subdirectories of starting_point a list of the files in starting_point.

os.walk works in a loop. Outside a loop, it doesn’t do much. Here’s how to use it in Python:

for current_directory, subdirectories, files in os.walk(starting_point): (do stuff)

What happens? The first time execution hits os.walk, it returns a triple like this:

current_directory = starting_point
subdirectories .... of starting_point
files ... in starting point

If you want to do something to all the files in the starting_point directory, you loop like this

for file in files: do stuff to file

If you want to do something to the subdirectories, unless it’s to list them, don’t. Wait for a moment, because...

The second time execution hits os.walk, it steps one directory down the tree, like this:

current_directory = first_subdirectory_in_starting_point
subdirectories .... of first_subdirectory_in_starting_point
files ... in first_subdirectory_in_starting_point

Now you can ‘do stuff to file’ for the files in first_subdirectory_in_starting_point.

What happens if there’s a subdirectory in first_subdirectory? The next time os.walk is executed it will return

current_directory = first_subdirectory_of_first_subdirectory_in_starting_point
subdirectories .... of first_subdirectory_of_first_subdirectory_in_starting_point
files ... in first_subdirectory_of_first_subdirectory_in_starting_point

Why don’t you do anything to the directories? Because os.walk is using that list to walk through them, so if you change names or permissions or something, before you have walked to the directory, os.walk (probably) won’t work.

If you want to mess with the subdirectories themselves, the chances are you need to run os_walk in the reverse direction (look that up).

The toy example of a directory listing just doesn’t expose the inner workings clearly enough. It can leave you thinking you have to do stuff with the directories as well as the files, but you don’t, of course.

Someone who has worked a lot with recursive Python functions will, should they have got all that experience before needing os.walk, grok os.walk fairly quickly. They will read the description and look at the examples and match that against the way they know Python array-returning programs have worked in the past, and say after a moment ‘Oh, sure, it does this and that, and you always have to use it in a loop’.

Catch is, walking a directory tree is one of the first things a programmer wants to do. And grokking recursive-return functions like os.walk is not simple. Being able to picture a recursive process is one of those big-jump differences between code-bashers and actual programmers. A toy example isn’t going to cut it.