I don’t have CrowdStrike on my laptop.
So it never stopped working.
*smug*
Computer security software is generally a waste of processing cycles, RAM and storage space. In a corporate, all the security should be on the corporate wall and whitelist maintenance. Inside the corporate wall, no security except firewalls which are more or less unavoidable now. On a personal machine, let the OS take care of security.
If the NSA, GCHQ, the Chinese, Israelis, French, or very probably the Finnish security services want to get at your computer, they’re going to do so. Private or corporate. Does anyone not think the Russians and Chineses had Crowdstrike reverse-engineered (or were just sent a copy of the code) within about a couple of months?
The criminals are pretty obvious, though anyone can be caught off-guard, as I was a few months ago until the voice told me they could not cancel a transaction unless I was at my machine to do this and that. My guard came right back.
Personally, I think the companies who let CrowdStrike - or anyone - push an update out onto their live systems without testing it on an isolated guinea pig system first, deserve anything that happens to them.
And it is way past time that all these software companies, from Microsoft onwards, carried insurance to compensate the consumers who were affected. Not the freaking airlines, but the suffering passengers. The corporates can sue each other, but the consumers should be compensated.
Showing posts with label Computing. Show all posts
Showing posts with label Computing. Show all posts
Tuesday 23 July 2024
Friday 17 February 2023
Artificial Intelligence and Aluminium Tennis Racquets: Same Opportunity, Different Sport
The PR flacks for ChatGPT probably deserve a bonus.
They have managed to convince a lot of people that anyone who currently has a desk job will be out of work by the end of the year.
ChatGPT and anything like it dips into the existing corpus of pictures / words / facts / academic papers / novels / poems / political speeches / whatever and produces something to your specifications from that corpus.
In response to "Produce a Theory of Everything that does not look like String Theory or Quantum Gravity" it will fail. Unless it tires pulling a Sokal Hoax on us. Likewise "Prove or refute the Collatz Conjecture" will draw a blank, though an icon of a blowing fuse might be appreciated.
"Paint a picture in a style nobody has seen yet" is logically impossible for it to do, since it has to draw on existing pictures. "Write a novel in the style of no previous writer about a subject nobody has written about before" would be equally impossible. A Real Person could attempt both, even if they failed through lack of checking for uniqueness - the point is that a Real Person could produce a painting or a novel that they had never seen or read before, but for ChatGPT it is a logical impossibility.(*)
Anyway, that isn't the point.
The point is that AI is just another tool. Like all new tools it will split a given community into those who can make it work to their advantage and those who can't get on with it. This was also true about spreadsheets, the metal tennis racquet, oil paint, sous vide and a thousand other new techniques and gadgets. AI in medicine would be a tremendous support tool, especially to older doctors who start to forget things or don't keep up (yes, I know Dreyfus used doctors as examples of Experts who could not be reproduced by AI, but after many years I've come to the conclusion that he wouldn't say those things now. He was writing in a medically-simpler age.)
How much of a difference is there between a writer turning out novels within a genre and an AI turning out a novel within a genre? The genre fans want the genre content, not the writer's unique personality (even if the genre is the writer, as in Chandler or Simenon).
There is probably an audience who would appreciate AI-generated genre novels, and one that won't. Publishers will produce both.
For decades, the British television has been producing TV that looks as if it was written to a set of rules and conventions that feel fake to me.
As for AI-Fake paintings, what else would you call a Thomas Kincaid? Or a Jeff Koons, Damien Hirst or Takashi Murakami? Just because these are Human Beings, doesn't mean to say they aren't using AI techniques, or perhaps, that those are the techniques that can be incorporated into AI.
Which actually helps to make a point I'm exploring in a long essay on the Philosophy of Mathematics, that technique rather than specific content is where originality really lies.
(*) This may be debatable, depending on how one thinks of the difference between "originality" and "variations", and at what point a "variation" becomes a "novelty" becomes "original". Those concepts are related but also have differences.
They have managed to convince a lot of people that anyone who currently has a desk job will be out of work by the end of the year.
ChatGPT and anything like it dips into the existing corpus of pictures / words / facts / academic papers / novels / poems / political speeches / whatever and produces something to your specifications from that corpus.
In response to "Produce a Theory of Everything that does not look like String Theory or Quantum Gravity" it will fail. Unless it tires pulling a Sokal Hoax on us. Likewise "Prove or refute the Collatz Conjecture" will draw a blank, though an icon of a blowing fuse might be appreciated.
"Paint a picture in a style nobody has seen yet" is logically impossible for it to do, since it has to draw on existing pictures. "Write a novel in the style of no previous writer about a subject nobody has written about before" would be equally impossible. A Real Person could attempt both, even if they failed through lack of checking for uniqueness - the point is that a Real Person could produce a painting or a novel that they had never seen or read before, but for ChatGPT it is a logical impossibility.(*)
Anyway, that isn't the point.
The point is that AI is just another tool. Like all new tools it will split a given community into those who can make it work to their advantage and those who can't get on with it. This was also true about spreadsheets, the metal tennis racquet, oil paint, sous vide and a thousand other new techniques and gadgets. AI in medicine would be a tremendous support tool, especially to older doctors who start to forget things or don't keep up (yes, I know Dreyfus used doctors as examples of Experts who could not be reproduced by AI, but after many years I've come to the conclusion that he wouldn't say those things now. He was writing in a medically-simpler age.)
How much of a difference is there between a writer turning out novels within a genre and an AI turning out a novel within a genre? The genre fans want the genre content, not the writer's unique personality (even if the genre is the writer, as in Chandler or Simenon).
There is probably an audience who would appreciate AI-generated genre novels, and one that won't. Publishers will produce both.
For decades, the British television has been producing TV that looks as if it was written to a set of rules and conventions that feel fake to me.
As for AI-Fake paintings, what else would you call a Thomas Kincaid? Or a Jeff Koons, Damien Hirst or Takashi Murakami? Just because these are Human Beings, doesn't mean to say they aren't using AI techniques, or perhaps, that those are the techniques that can be incorporated into AI.
Which actually helps to make a point I'm exploring in a long essay on the Philosophy of Mathematics, that technique rather than specific content is where originality really lies.
(*) This may be debatable, depending on how one thinks of the difference between "originality" and "variations", and at what point a "variation" becomes a "novelty" becomes "original". Those concepts are related but also have differences.
Tuesday 8 February 2022
Data Backups and External Drives
It's been nearly two years since I over-thought how I do backups. Technology changes, prices change, and use cases change: reviews are appropriate.
The cause was finding that one of my external hard drives (EHD) would disconnect and then re-connect from the laptop at random. Diskfix couldn't find any faults on the drive, so I have to blame the USB ports. Everyone thinks that the worst that can happen is dropping an EHD onto a hard surface. Also bad is having the EHD fall while still attached to the computer and so putting stress on the computer's USB port. Those things aren't welded to the card.
The only way to avoid plugging and unplugging an EHD is to use a NAS (either dedicated or a headless Raspberry Pi with EHD's plugged into it), or to use another computer with a more reliable USB port. Either way, we're still using EHDs of some quality or other.
Network-attached storage is the first thing one tends to think about. The original use-case for this was a) it was nerdier than using external drives, b) it was not a lot more expensive per MB than external drives, c) people used to download video files from e.g. iTunes before watching them. The nerds have now migrated to using Raspberry Pi's with EHDs attached, which are computers, not NAS. The cost of EHDs has dropped considerably, while the cost of NAS-related hardware and software has gone up. Finally, what is "downloading movies"? Because streaming.
I don't do daily backups. You should, because you are creating GBs of media every day. Or making irreplaceable music in your DAW, with a bank of plug-ins that cumulates to mucho dinero. If I get to that stage, I too will get a NAS and set up automated differential backup jobs. Because I'd be a pro, and that's what pros do. Most of what I do is writing, and I'm old enough to remember when a writer's nightmare was leaving his work-in-progress on the train. Our idea of backup was carbon paper.
Everything I write is in iCloud, which will do for the short-term, as long as I NEVER delete files. iCloud is really for making a file accessible from a number of devices and keeping them in sync: delete a file by mistake on your laptop, and it will vanish from iCloud.
My ripped music is currently on an external LaCie HDD. I made that decision because I only ever used it as a source to load tracks on my iPhone. Then it was switched off. I do have albums on it that I downloaded from Amazon expressly for train music, and sometimes I want to listen to that. I have the LaCie plugged into a ASUS netbook from 2010 or so. Music does not need high-spec as video does. So I would switch the Asus on, count to twenty and then fire up Music Streamer on my iPad to feed it to my amp through the USB port. Then I would have to remember to switch the Asus off before going to bed. Yes, I know, and I've set the Asus up to hibernate(*) at 22:00 and wake up again at 09:00. Its hard drive spins down even on mains power, but I'm not sure the LaCie does. External drive spin-down seems to be hit-and-miss on Win 7, and there's no provision in the LaCie driver. I looked. So that drive could be on for thirteen hours a day, and I think that disc is spinning all the time. It's warm enough. Not what we want.
So I should use an SSD for the music. No moving parts. No activity, less heat generation. I can buy an internal 1TB SSD drive for about £90 and an enclosure for about £10, which is cheaper than the non-sale price of a 1TB SSD on Amazon. For the added speed, reliability and lifetime, that feels worth it.
(When you get the SSD, you will need to initialise and format it with the Windows disk management facility. Format it as EXFAT as b****y Apple have only just added NTFS-reading with iOS 15. If you don't, you will get strange "access forbidden" errors, which will point you in the direction of permissions, and it has nothing to do with permissions.)
Then I can go back to using cheap HDDs from reputable brands (only because I assume they have better quality control in their Chinese factories than the Chinese brands do), and accept that after 3 - 5 years they will need replacing. A 1TB Seagate HDD is currently £38.
I am going to make sure that each of my laptops can communicate with EHDs connected to any other of my laptops. That way, if a laptop USB port goes squiffy, I can backup over Wi-Fi to another machine.
I adopted Free File Sync - to which I have donated - a while ago, and I'm still liking it.
(*) Hibernate, not shutdown. A computer can't wake from shutdown.
The cause was finding that one of my external hard drives (EHD) would disconnect and then re-connect from the laptop at random. Diskfix couldn't find any faults on the drive, so I have to blame the USB ports. Everyone thinks that the worst that can happen is dropping an EHD onto a hard surface. Also bad is having the EHD fall while still attached to the computer and so putting stress on the computer's USB port. Those things aren't welded to the card.
The only way to avoid plugging and unplugging an EHD is to use a NAS (either dedicated or a headless Raspberry Pi with EHD's plugged into it), or to use another computer with a more reliable USB port. Either way, we're still using EHDs of some quality or other.
Network-attached storage is the first thing one tends to think about. The original use-case for this was a) it was nerdier than using external drives, b) it was not a lot more expensive per MB than external drives, c) people used to download video files from e.g. iTunes before watching them. The nerds have now migrated to using Raspberry Pi's with EHDs attached, which are computers, not NAS. The cost of EHDs has dropped considerably, while the cost of NAS-related hardware and software has gone up. Finally, what is "downloading movies"? Because streaming.
I don't do daily backups. You should, because you are creating GBs of media every day. Or making irreplaceable music in your DAW, with a bank of plug-ins that cumulates to mucho dinero. If I get to that stage, I too will get a NAS and set up automated differential backup jobs. Because I'd be a pro, and that's what pros do. Most of what I do is writing, and I'm old enough to remember when a writer's nightmare was leaving his work-in-progress on the train. Our idea of backup was carbon paper.
Everything I write is in iCloud, which will do for the short-term, as long as I NEVER delete files. iCloud is really for making a file accessible from a number of devices and keeping them in sync: delete a file by mistake on your laptop, and it will vanish from iCloud.
My ripped music is currently on an external LaCie HDD. I made that decision because I only ever used it as a source to load tracks on my iPhone. Then it was switched off. I do have albums on it that I downloaded from Amazon expressly for train music, and sometimes I want to listen to that. I have the LaCie plugged into a ASUS netbook from 2010 or so. Music does not need high-spec as video does. So I would switch the Asus on, count to twenty and then fire up Music Streamer on my iPad to feed it to my amp through the USB port. Then I would have to remember to switch the Asus off before going to bed. Yes, I know, and I've set the Asus up to hibernate(*) at 22:00 and wake up again at 09:00. Its hard drive spins down even on mains power, but I'm not sure the LaCie does. External drive spin-down seems to be hit-and-miss on Win 7, and there's no provision in the LaCie driver. I looked. So that drive could be on for thirteen hours a day, and I think that disc is spinning all the time. It's warm enough. Not what we want.
So I should use an SSD for the music. No moving parts. No activity, less heat generation. I can buy an internal 1TB SSD drive for about £90 and an enclosure for about £10, which is cheaper than the non-sale price of a 1TB SSD on Amazon. For the added speed, reliability and lifetime, that feels worth it.
(When you get the SSD, you will need to initialise and format it with the Windows disk management facility. Format it as EXFAT as b****y Apple have only just added NTFS-reading with iOS 15. If you don't, you will get strange "access forbidden" errors, which will point you in the direction of permissions, and it has nothing to do with permissions.)
Then I can go back to using cheap HDDs from reputable brands (only because I assume they have better quality control in their Chinese factories than the Chinese brands do), and accept that after 3 - 5 years they will need replacing. A 1TB Seagate HDD is currently £38.
I am going to make sure that each of my laptops can communicate with EHDs connected to any other of my laptops. That way, if a laptop USB port goes squiffy, I can backup over Wi-Fi to another machine.
I adopted Free File Sync - to which I have donated - a while ago, and I'm still liking it.
(*) Hibernate, not shutdown. A computer can't wake from shutdown.
Friday 28 January 2022
That M1 Mac Air Though
My trusty early 2015 Mac Air, has begun doing odd things every three or so weeks. A complete screen glitch in the middle of a YT video felt like an advanced warning. So I ordered an M1 Air.
Then my bank sent me a message saying that it had cancelled the payment instruction because fraud.
I called the number I had for these occasions, and the agent told me that this happened a lot with Apple, John Lewis, Curries and a couple of other retailers. If someone gets your card and orders from those guys, they can do a lot of damage in a short time. They choose those guys because they can resell (on eBay presumably) readily. If they bought an X-E4 from Fuji, it's going to be a while before that shifts. Fair enough, if inconvenient. The bank's agent asked me to make another order so he could make sure this payment went through. Not to worry about the first order, Apple would cancel that when they saw the payment was cancelled.
You can guess what happened. I got one Air delivered and the money taken from my account. I noticed that there was still another payment outstanding. I called the bank. Not to worry, they said, it's just a delay because New Year.
Nope. A couple of days later I got another message from DHL saying they would be delivering my Air. I called Apple (in fact, their outsourced customer handling company) explained the situation, and their agent was totally calm and competent, told me to refuse delivery when the courier arrived, and set a return and credit in motion. All of which went through within a couple of days.
So when the bank cancels your next payment to a retailer, cancel the order as well. Do not assume the retailer will notice.
Anyway.
Everything they say about these M1 Airs is true. It's even smaller than the previous one. The keyboard is terrific, and only Apple make trackpads this good. The screen is just pin-point wow!
The stereo sound system is using some psycho-acoustics that puts voices in the middle of the screen, but then ventriloquises background sound as coming from the left and right walls. Music videos have the music in the middle, though it feels like they are using a `spatial' effect that I last heard on a ghetto-blaster in *cough, splutter*.
After making up an old-school what-I-need-to-transfer-and-install list, I found out about the Migration Assistant. See, that's why we all pay Apple prices. It's not for the kit, it's for the eco-system. It's because they put a tonne of work into developing a program that we would only use twice in the life of the machine, once at the start to import everything from the old machine, and once at the end to export it to the even newer one. Everything came over, and everything works, except for a program called LatextIt!, which is probably searching for some files somewhere they aren't any more.
On the hardware side, I got an Anker multi-adapter for USB-A and card readers, plus additional USB-C ports.
Then I tried ripping some CDs.
My Air told me the USB-A CD drive needed power (the Anker supplies power, the USB-A HDD was running) and I should plug my kit into a USB-C port on the laptop. But. But.
OK. Order a USB-A to C adapter so I can plug the USB-A CD drive into the other USB-C port. Arrives via Amazon Prime in less than 24 hours. Plug in. Yea! Works.
This Air turns into a desktop sprawl if you want to do rip CDs, or read an SD card. The rumours are that the next iteration will have extra slots and an SD card reader, and the Magsafe (which I have not felt the need for so far) port.
I am, however, a happy customer.
Then my bank sent me a message saying that it had cancelled the payment instruction because fraud.
I called the number I had for these occasions, and the agent told me that this happened a lot with Apple, John Lewis, Curries and a couple of other retailers. If someone gets your card and orders from those guys, they can do a lot of damage in a short time. They choose those guys because they can resell (on eBay presumably) readily. If they bought an X-E4 from Fuji, it's going to be a while before that shifts. Fair enough, if inconvenient. The bank's agent asked me to make another order so he could make sure this payment went through. Not to worry about the first order, Apple would cancel that when they saw the payment was cancelled.
You can guess what happened. I got one Air delivered and the money taken from my account. I noticed that there was still another payment outstanding. I called the bank. Not to worry, they said, it's just a delay because New Year.
Nope. A couple of days later I got another message from DHL saying they would be delivering my Air. I called Apple (in fact, their outsourced customer handling company) explained the situation, and their agent was totally calm and competent, told me to refuse delivery when the courier arrived, and set a return and credit in motion. All of which went through within a couple of days.
So when the bank cancels your next payment to a retailer, cancel the order as well. Do not assume the retailer will notice.
Anyway.
Everything they say about these M1 Airs is true. It's even smaller than the previous one. The keyboard is terrific, and only Apple make trackpads this good. The screen is just pin-point wow!
The stereo sound system is using some psycho-acoustics that puts voices in the middle of the screen, but then ventriloquises background sound as coming from the left and right walls. Music videos have the music in the middle, though it feels like they are using a `spatial' effect that I last heard on a ghetto-blaster in *cough, splutter*.
After making up an old-school what-I-need-to-transfer-and-install list, I found out about the Migration Assistant. See, that's why we all pay Apple prices. It's not for the kit, it's for the eco-system. It's because they put a tonne of work into developing a program that we would only use twice in the life of the machine, once at the start to import everything from the old machine, and once at the end to export it to the even newer one. Everything came over, and everything works, except for a program called LatextIt!, which is probably searching for some files somewhere they aren't any more.
On the hardware side, I got an Anker multi-adapter for USB-A and card readers, plus additional USB-C ports.
Then I tried ripping some CDs.
My Air told me the USB-A CD drive needed power (the Anker supplies power, the USB-A HDD was running) and I should plug my kit into a USB-C port on the laptop. But. But.
OK. Order a USB-A to C adapter so I can plug the USB-A CD drive into the other USB-C port. Arrives via Amazon Prime in less than 24 hours. Plug in. Yea! Works.
This Air turns into a desktop sprawl if you want to do rip CDs, or read an SD card. The rumours are that the next iteration will have extra slots and an SD card reader, and the Magsafe (which I have not felt the need for so far) port.
I am, however, a happy customer.
Friday 3 December 2021
Mac Pro 14" vs Mac Air 13.3" - Overthinking the Choice
I thought I'd call it "overthinking" before you did.
My trusty Mac Air has been with me for nearly six years. Intel i5, 4GB RAM, 256 SDD. I mostly use it for writing and media consumption, so it's vastly over-powered for my "use case" (as it used to be called).
But one day, it's going to pack up.
And I want to do other things. Like record some of the music I play on guitar and piano. Which means running Garageband and maybe some plug-ins. I'm not sure about making YT videos, or even why I might do it. Also photography, if I have good enough photographs to do things with. The current Air can handle that, except at industrial levels of production, which I'm not going to reach.
But the Intel Airs are no more. Only the M1 machines. Which are:
The minimum 256x8, 8-7-16 cores (CPU-GPU-Neural) Air for £999.
Trade up to 512x8 and it's £1,199.
Add another graphics core to 8-8-16 and it's £1,249.
Add another 8GB RAM and it's £1,449 / £1,399.
The basic Mac Pro is 512x16 and 8-14-16 cores for £1,899. It comes with Magsafe ("it's not a Mac if it's not Magsafe"), an HDMI port and a SDXC card reader (which my Mac Air has and I have NEVER used). Add adapters for all that to the cost of the Air, and it's another £45.
Whichever, I'm going to need USB A -> C adapters for my external CD drive, and Lightning to USB C adapters for my other iDevices. This is not going to be horribly expensive.
I think 512GB SSD is needed for the extra things I want to do. Plug-ins gobble up space. So £1,199 is the baseline.
The extra £700 to the Mac Pro gets me: a slightly bigger, but undeniably better screen; 8GB more RAM, 6 more video cores, plus better speakers, microphone array, and Face Time camera. Also the Mac Pro has fans, and the Air doesn't. Would I really use any of that?
Don't forget that the real upgrade is from my 4GB Intel i5 Air to an 8GB M1 chip. That's the WOW factor right there. The reviewers of the M1 Air said its video editing was easily fast enough unless you were doing really big files in 4K. If I ever do anything, it will be smaller and in 1080p. So I'm covered.
If I ever do need that extra processing power, I will probably buy an iMac. Heaven knows what I'll be doing though.
Is there any improvement in performance in going to 16GB of RAM? The reviewers say there isn't, except at insane loads I would never reach, because the M1 chip accesses the SSD so fast it's almost RAM. Other voices suggest 'future-proofing' with 16GB. Well, when I got my Intel Air, I wondered about getting 8GB because future-proofing, but 4GB has turned out to be just fine - especially when I dumped out Evernote and Dropbox, which had bloated beyond all reasonableness. Both Apple and Microsoft are paying more attention to making their operating systems use less RAM and work faster. Apple even more so. So I'm taking it that 8GB in the new Air will be as future-proof as well.
So is the luxury stuff (speakers, screen, microphones and iSight camera) and the additional ports worth £700?
Nah. The guy (*) did say that if you don't know that you need a Mac Pro, then you don't. And I've convinced myself the baseline Air will have all the oomph my modest needs will need.
Baseline Air with 512GB plus a Magsafe adapter, and the USB A -> C converters for my heritage USB A and Lightning gear.
OMG did I just reach a conclusion?
(*) You know, the guy whose YT video you watched and thought was good, but now you can't remember who it was. He makes a lot of videos.
My trusty Mac Air has been with me for nearly six years. Intel i5, 4GB RAM, 256 SDD. I mostly use it for writing and media consumption, so it's vastly over-powered for my "use case" (as it used to be called).
But one day, it's going to pack up.
And I want to do other things. Like record some of the music I play on guitar and piano. Which means running Garageband and maybe some plug-ins. I'm not sure about making YT videos, or even why I might do it. Also photography, if I have good enough photographs to do things with. The current Air can handle that, except at industrial levels of production, which I'm not going to reach.
But the Intel Airs are no more. Only the M1 machines. Which are:
The minimum 256x8, 8-7-16 cores (CPU-GPU-Neural) Air for £999.
Trade up to 512x8 and it's £1,199.
Add another graphics core to 8-8-16 and it's £1,249.
Add another 8GB RAM and it's £1,449 / £1,399.
The basic Mac Pro is 512x16 and 8-14-16 cores for £1,899. It comes with Magsafe ("it's not a Mac if it's not Magsafe"), an HDMI port and a SDXC card reader (which my Mac Air has and I have NEVER used). Add adapters for all that to the cost of the Air, and it's another £45.
Whichever, I'm going to need USB A -> C adapters for my external CD drive, and Lightning to USB C adapters for my other iDevices. This is not going to be horribly expensive.
I think 512GB SSD is needed for the extra things I want to do. Plug-ins gobble up space. So £1,199 is the baseline.
The extra £700 to the Mac Pro gets me: a slightly bigger, but undeniably better screen; 8GB more RAM, 6 more video cores, plus better speakers, microphone array, and Face Time camera. Also the Mac Pro has fans, and the Air doesn't. Would I really use any of that?
Don't forget that the real upgrade is from my 4GB Intel i5 Air to an 8GB M1 chip. That's the WOW factor right there. The reviewers of the M1 Air said its video editing was easily fast enough unless you were doing really big files in 4K. If I ever do anything, it will be smaller and in 1080p. So I'm covered.
If I ever do need that extra processing power, I will probably buy an iMac. Heaven knows what I'll be doing though.
Is there any improvement in performance in going to 16GB of RAM? The reviewers say there isn't, except at insane loads I would never reach, because the M1 chip accesses the SSD so fast it's almost RAM. Other voices suggest 'future-proofing' with 16GB. Well, when I got my Intel Air, I wondered about getting 8GB because future-proofing, but 4GB has turned out to be just fine - especially when I dumped out Evernote and Dropbox, which had bloated beyond all reasonableness. Both Apple and Microsoft are paying more attention to making their operating systems use less RAM and work faster. Apple even more so. So I'm taking it that 8GB in the new Air will be as future-proof as well.
So is the luxury stuff (speakers, screen, microphones and iSight camera) and the additional ports worth £700?
Nah. The guy (*) did say that if you don't know that you need a Mac Pro, then you don't. And I've convinced myself the baseline Air will have all the oomph my modest needs will need.
Baseline Air with 512GB plus a Magsafe adapter, and the USB A -> C converters for my heritage USB A and Lightning gear.
OMG did I just reach a conclusion?
(*) You know, the guy whose YT video you watched and thought was good, but now you can't remember who it was. He makes a lot of videos.
Thursday 10 December 2020
Bye-bye Evernote and Dropbox
I got Dropbox and Evernote because I read about them on Rands in Repose. That was a long time ago in Internet Years. Something like (looks up account on Evernote) 8th October 2010. (Stone me guv'nor, 10 years.)
Now I use iCloud instead of Dropbox and Apple Notes instead of Evernote. Why?
Bloat.
About a year ago I looked at what was using my RAM. I have 4GB in my early 2015 Mac Air (blimey guv, I've had that for five years) so this is something I need to do, even though OS X handles memory way better than Windows 7 as disfigured by an FTSE 100 company.
Fire up Activity Monitor, look at the Memory tab, but order by the Process Name column, not the Memory column. This is because apps are now split into a number of processes. For instance, my Brave Browser with one tab open has one main process, two Helper processes, a GPU process, and six Renderer processes going. The main process is 240MB, and all the others take 504MB. That browser is taking 0.75GB of memory.
(I just closed Brave and opened Firefox at my Amazon Wish List. Main process 401MB, supporting processes 337MB, so still 0.7GB of memory. Re-opening Brave on my Wish List gets me 135MB in the main process and 270MB of supporting processes. Which is better. I bet if I open You Tube it bloats out bigly.)
When I did that exercise for Dropbox, it was using over 0.5GB of RAM. Just to sync some files in a Cloud. WTF? Since the iCloud process was already running - because Apple - this was a ridiculous use of resources. It took a few minutes to put the files I wanted backing in the iCloud in the iCloud drive directory, and then I could shut down Dropbox. I even uninstalled it on all my machines.
Evernote updated itself a couple of weeks ago. One of those OMG-they-have-so-changed-everything-WTF-did-they-do-that? changes. Every time Evernote does a major change its users lose control over more and more things. This one changed the fonts and graphicalised everything - like Office 2016 did and Apple Notes is. The feel of how the type appears on the screen in response to the keyboard is different. It was when I navigated to a large note which choked the viewer I began to think something was wrong. Yep, Evernote, which used to be whip-cracking smart and fast, now had a Renderer process that took around 400MB. The whole lot took up around 0.7GB, and Evernote has never been great at memory management.
Notes is running in less than 84MB, and even Pages takes only 153MB. How the **** does it need a 400MB renderer process to do what Apple does in 84MB? Makes me wonder what other dumb things they did.
Export all my Evernote Notebooks. Load them into Apple Notes. Close Evernote. I had cancelled my Premium membership a few months previously. I haven't uninstalled it, and I still have an account I can access via a browser. Moving to Apple Notes is one-way: it only lets you export to pdf files. Because Apple.
I blame SSDs. I see this at work, where people who don't have problematical unknown programs like SAS Base (oh, wait) are getting Windows 10 machines with SSDs and 8GB of RAM. Half of that is taken by Windows and security bloatware - McAfee and Tanium Endpoint plus all the supporting Windows services - so users are still trying to run their real programs in under 4GB. Everyone with one of those machines is agog at how fast they are - what do they have at home? - but that speed comes from the SSD, not efficient and effective use of the RAM. The fast disk swapping SSDs allow seems to make people lazy about RAM use and program bloat - who cares how big it is, it'll swap real fast on an SSD, and the clock speed will cover up the lousy coding.
So deeper into the Apple eco-system I go. Because third-party suppliers got lazy with their coding.
Now I use iCloud instead of Dropbox and Apple Notes instead of Evernote. Why?
Bloat.
About a year ago I looked at what was using my RAM. I have 4GB in my early 2015 Mac Air (blimey guv, I've had that for five years) so this is something I need to do, even though OS X handles memory way better than Windows 7 as disfigured by an FTSE 100 company.
Fire up Activity Monitor, look at the Memory tab, but order by the Process Name column, not the Memory column. This is because apps are now split into a number of processes. For instance, my Brave Browser with one tab open has one main process, two Helper processes, a GPU process, and six Renderer processes going. The main process is 240MB, and all the others take 504MB. That browser is taking 0.75GB of memory.
(I just closed Brave and opened Firefox at my Amazon Wish List. Main process 401MB, supporting processes 337MB, so still 0.7GB of memory. Re-opening Brave on my Wish List gets me 135MB in the main process and 270MB of supporting processes. Which is better. I bet if I open You Tube it bloats out bigly.)
When I did that exercise for Dropbox, it was using over 0.5GB of RAM. Just to sync some files in a Cloud. WTF? Since the iCloud process was already running - because Apple - this was a ridiculous use of resources. It took a few minutes to put the files I wanted backing in the iCloud in the iCloud drive directory, and then I could shut down Dropbox. I even uninstalled it on all my machines.
Evernote updated itself a couple of weeks ago. One of those OMG-they-have-so-changed-everything-WTF-did-they-do-that? changes. Every time Evernote does a major change its users lose control over more and more things. This one changed the fonts and graphicalised everything - like Office 2016 did and Apple Notes is. The feel of how the type appears on the screen in response to the keyboard is different. It was when I navigated to a large note which choked the viewer I began to think something was wrong. Yep, Evernote, which used to be whip-cracking smart and fast, now had a Renderer process that took around 400MB. The whole lot took up around 0.7GB, and Evernote has never been great at memory management.
Notes is running in less than 84MB, and even Pages takes only 153MB. How the **** does it need a 400MB renderer process to do what Apple does in 84MB? Makes me wonder what other dumb things they did.
Export all my Evernote Notebooks. Load them into Apple Notes. Close Evernote. I had cancelled my Premium membership a few months previously. I haven't uninstalled it, and I still have an account I can access via a browser. Moving to Apple Notes is one-way: it only lets you export to pdf files. Because Apple.
I blame SSDs. I see this at work, where people who don't have problematical unknown programs like SAS Base (oh, wait) are getting Windows 10 machines with SSDs and 8GB of RAM. Half of that is taken by Windows and security bloatware - McAfee and Tanium Endpoint plus all the supporting Windows services - so users are still trying to run their real programs in under 4GB. Everyone with one of those machines is agog at how fast they are - what do they have at home? - but that speed comes from the SSD, not efficient and effective use of the RAM. The fast disk swapping SSDs allow seems to make people lazy about RAM use and program bloat - who cares how big it is, it'll swap real fast on an SSD, and the clock speed will cover up the lousy coding.
So deeper into the Apple eco-system I go. Because third-party suppliers got lazy with their coding.
Thursday 23 July 2020
My Western Digital Network Drive Has The Green Flashing Light of Death!
Western Digital My Book Live drives can get the flashing green light of death. They boot up, start talking to themselves and don't stop, locked in an endless read-write loop. Heaven knows what they are reading and writing, but that's what they are doing. Hold the device and feel the disk rumbling away to itself. A reset does not help. The drive becomes invisible to the outside IP world, so I assume it's failing to load some of its software. Good thing I did a backup of it in March. It was pretty darn old: I must have bought it in 2013 or so. (You do not know how long I just spent with my Calendar app trying to work that out.)
The My Book Live was a 2TB network drive: it had ethernet connectivity. Its main use was to hold my iTunes files so I could stream on my other iDevices, and load up my iPhone with music now and again.
They don't make ethernet drives like that anymore. Now the choice is between USB3 portable drives, USB3 desktop drives, and NAS units. Or Cloud storage.
Cloud services reserve the right to scan your content for copyright infringement and other naughties, and delete at will. Not that I have any content like that, but there is no appeal to Apple, so why take a chance? As well as that, I would be at the mercy of Talk-Talk. It so much as rains for twenty minutes and my alleged high-speed broadband drops to half-pace, putting them in breach of contract. I can live with the 20 / 7 Mbps that results, and it does seem to sort itself out after a few days. On the other hand, HDD's have mechanical failures and I bet NAS's get OS problems as well.
Nevertheless, it's worth having the costs.
5GB free at iCloud 2TB at £6.99
iCloud = £83.88 / year
2TB external HDD approx £70 = 10 months of iCloud
4TB Synology DS218Play £400 (2TB storage with Raid 1) = 57 months of iCloud = 5.5 external drives
2TB external SSD approx £320 = 4xHDD or 46 months iCloud
1TB external SSD approx £150 = 2xHDD or 21 months iCloud
512GB on Air £200
2TB on Air £800
(Note that Apple internal SSD storage is more than twice the price of external SSD storage.)
We text-people don't need lots of storage. It's photographers, video-makers and serious musicians who need serious storage, fast graphics cards and 8-core processors. If I was one of those, I would already have a NAS. Not all NAS are made by Synology, but it's the only name you hear in consumers circles. I could in theory get away with 500GB of storage: my music is around 120GB, my photos including Photos Library are about 15GB, and Calibre back-up is about 5GB.
Of course, using drives to share music is just so 2014. So audiophile. I mean, do I even Roon? https://roonlabs.com I dream about Roon-ing. Not so much having a subscription, but leading a life that would make Roon necessary. If you dropped a lifetime subscription in my lap, I would thank you very much. But I just can't justify it.
I listen to music on CDs. I watch movies and foreign cop shows on DVD. I stream music from Spotify, and I have been hitting the Above and Beyond live concerts on You Tube recently. I stream movies from MUBI and Curzon Home Cinema via my Apple TV. I have no desire to download anything.
If I want to stream music in other rooms, I can. It's called Spotify + Sonos. Spotify has all the music I do and a bit more. Without that I would have needed a NAS.
Which is cross-product substitution for you. £400 of Synology is 40 months of Spotify. This is a no-brainer of a decision. Unless you think you can tell the difference between a 320kps stream and a 4Mps uncompressed stream.
A NAS can provide back-up especially if it's RAID 1 or above, but one should still backup the NAS now and again, and that means portable drives. Which I have already.
So this is what I did. I re-set my Music directory to my Air and imported the files from the last backup. That leaves me with 60GB spare after clean-up, which is enough. Then I copied all my content directories onto a portable drive. After that fiddling around, I decided to find a proper backup / sync program, and found FreeFileSync. It's pretty easy to work out how to use it. I trialled it, it worked nicely, so I gave them a donation. It's now a whole lot easier to do backups.
Let's face it, I barely used the My Book Live. So replacing it with a more expensive NAS would be silly. Next time around I will spend money on additional storage in my iDevices instead.
The My Book Live was a 2TB network drive: it had ethernet connectivity. Its main use was to hold my iTunes files so I could stream on my other iDevices, and load up my iPhone with music now and again.
They don't make ethernet drives like that anymore. Now the choice is between USB3 portable drives, USB3 desktop drives, and NAS units. Or Cloud storage.
Cloud services reserve the right to scan your content for copyright infringement and other naughties, and delete at will. Not that I have any content like that, but there is no appeal to Apple, so why take a chance? As well as that, I would be at the mercy of Talk-Talk. It so much as rains for twenty minutes and my alleged high-speed broadband drops to half-pace, putting them in breach of contract. I can live with the 20 / 7 Mbps that results, and it does seem to sort itself out after a few days. On the other hand, HDD's have mechanical failures and I bet NAS's get OS problems as well.
Nevertheless, it's worth having the costs.
5GB free at iCloud 2TB at £6.99
iCloud = £83.88 / year
2TB external HDD approx £70 = 10 months of iCloud
4TB Synology DS218Play £400 (2TB storage with Raid 1) = 57 months of iCloud = 5.5 external drives
2TB external SSD approx £320 = 4xHDD or 46 months iCloud
1TB external SSD approx £150 = 2xHDD or 21 months iCloud
512GB on Air £200
2TB on Air £800
(Note that Apple internal SSD storage is more than twice the price of external SSD storage.)
We text-people don't need lots of storage. It's photographers, video-makers and serious musicians who need serious storage, fast graphics cards and 8-core processors. If I was one of those, I would already have a NAS. Not all NAS are made by Synology, but it's the only name you hear in consumers circles. I could in theory get away with 500GB of storage: my music is around 120GB, my photos including Photos Library are about 15GB, and Calibre back-up is about 5GB.
Of course, using drives to share music is just so 2014. So audiophile. I mean, do I even Roon? https://roonlabs.com I dream about Roon-ing. Not so much having a subscription, but leading a life that would make Roon necessary. If you dropped a lifetime subscription in my lap, I would thank you very much. But I just can't justify it.
I listen to music on CDs. I watch movies and foreign cop shows on DVD. I stream music from Spotify, and I have been hitting the Above and Beyond live concerts on You Tube recently. I stream movies from MUBI and Curzon Home Cinema via my Apple TV. I have no desire to download anything.
If I want to stream music in other rooms, I can. It's called Spotify + Sonos. Spotify has all the music I do and a bit more. Without that I would have needed a NAS.
Which is cross-product substitution for you. £400 of Synology is 40 months of Spotify. This is a no-brainer of a decision. Unless you think you can tell the difference between a 320kps stream and a 4Mps uncompressed stream.
A NAS can provide back-up especially if it's RAID 1 or above, but one should still backup the NAS now and again, and that means portable drives. Which I have already.
So this is what I did. I re-set my Music directory to my Air and imported the files from the last backup. That leaves me with 60GB spare after clean-up, which is enough. Then I copied all my content directories onto a portable drive. After that fiddling around, I decided to find a proper backup / sync program, and found FreeFileSync. It's pretty easy to work out how to use it. I trialled it, it worked nicely, so I gave them a donation. It's now a whole lot easier to do backups.
Let's face it, I barely used the My Book Live. So replacing it with a more expensive NAS would be silly. Next time around I will spend money on additional storage in my iDevices instead.
Thursday 9 July 2020
Mac OS Big Sur - Stop Futzing Already
I have seen at least three You Tube reviews of macOS Big Sur. Which is supposed to be an operating system. 85 new features later, nothing about the operating system. Emojis, searching in messages, some pointless new options in Music, face recognition in the Home App, and a bunch of other stuff in the Apps.
What about changes to the actual operating system? Does it use less RAM than Catalina? Does it load apps faster? How about the firewall? Improvements to the music playing software? Does it sandbox like iOS does? Any new features I might like to know about as a developer?
When the **** is Apple going to give us proper window management, because Windows still does that better than anyone. When are they going to detect all the hard drives and other equipment on my network, and re-connect to them, like Windows does, even if the devices aren't in Apple's ecosystem? And WHEN will they get the album art in Music sorted out? It works if I have all my files on my Air, but on the NAS? Half the images would go missing and the loading time was awful. My Sonos and File Manager apps on the iPod Touch did a better job. And please can we have the same wonderfully functional File Open / Save dialogs that Windows does so much better.
I get the feeling that Apple do a lot of hard-core OS-stuff to improve performance. They should crow about it some more. But maybe they don't, because they can fix it on the chip, in a way that Microsoft can't.
It may also be the reviewers, who assume their audience want to know about the awful battery icon more than the way Apple have (maybe) improved battery life by doing something with the battery management algorithm. I'd rather know about the algorithms. I can see the awful design for myself.
What makes a computer operating system is flawless multi-tasking, powerful peripheral management, and simple multi-user management. If I can't run Nora En Pure on my browser on one screen, while typing in Evernote and downloading some Amazon music, while rendering a movie clip, then it's not a real computer. So I would expect to hear about improvements to those things as well.
But OMG the desktop art on Big Sur! I suspect it's going to be a while before this late adopter adopts it.
What about changes to the actual operating system? Does it use less RAM than Catalina? Does it load apps faster? How about the firewall? Improvements to the music playing software? Does it sandbox like iOS does? Any new features I might like to know about as a developer?
When the **** is Apple going to give us proper window management, because Windows still does that better than anyone. When are they going to detect all the hard drives and other equipment on my network, and re-connect to them, like Windows does, even if the devices aren't in Apple's ecosystem? And WHEN will they get the album art in Music sorted out? It works if I have all my files on my Air, but on the NAS? Half the images would go missing and the loading time was awful. My Sonos and File Manager apps on the iPod Touch did a better job. And please can we have the same wonderfully functional File Open / Save dialogs that Windows does so much better.
I get the feeling that Apple do a lot of hard-core OS-stuff to improve performance. They should crow about it some more. But maybe they don't, because they can fix it on the chip, in a way that Microsoft can't.
It may also be the reviewers, who assume their audience want to know about the awful battery icon more than the way Apple have (maybe) improved battery life by doing something with the battery management algorithm. I'd rather know about the algorithms. I can see the awful design for myself.
What makes a computer operating system is flawless multi-tasking, powerful peripheral management, and simple multi-user management. If I can't run Nora En Pure on my browser on one screen, while typing in Evernote and downloading some Amazon music, while rendering a movie clip, then it's not a real computer. So I would expect to hear about improvements to those things as well.
But OMG the desktop art on Big Sur! I suspect it's going to be a while before this late adopter adopts it.
Monday 16 March 2020
Cloud Drives, Backups and Time Machines
The things I over-think! How difficult is to store your digital files: photos, books, music, movies and documents?
In the old days we put books on shelves or in boxes; photographs in albums, or in boxes; records and CDs on shelves, or in boxes; and any documents we generated in files, or in boxes. How easy was that? Then computers came along and made everything simpler by making it more complicated. Everything was stored on the hard drive - passing over the days of floppy drives and diskettes - but what happened if the hard drive packed up? Hence backup which meant taking a copy of everything on another disk and storing that somewhere safe. Real Men did backups daily, weekly, and monthly. The daily and weekly ones were cycled, the monthly ones kept forever. On tape. Lots of people didn’t do backups, and every now and then they wrote to Jack Schofield at the Guardian about the disaster, and Uncle Jack scolded them lightly.
Backup is copies you make on a storage device and then leave alone unless disaster strikes, when you copy them back to wherever you copied them from. You never edit backup copies. Ever. You should keep at least two generations of backups.
Cloud drives in particular and synching services in general, are not backup. Do something dumb on one synched device, like delete everything, and all the other synched devices will lose everything as well. iCloud, One Drive, Google Drive and all the other services have this feature. To make up for it, some of them add a versioning, recovery or history feature that gets over this. When I tried using the versioning on Pages on my 4GB Mac Air, it choked so bad I had to turn it off.
A Cloud drive is what you use when you want to access, edit and save back your work-in-progress files from a number of different devices. Put your photos up in the iCloud and you can regale everyone with your holiday snaps on your iPhone, iPad, Macbook or even Windows device. Ditto music, though why you would when you can get it from a streaming service is another question. Unless it’s a recording of your band, or a track you’re working on. Same for videos.
Cloud drives offer you the assurance that even if your device gets lost, goes crazy or gets broken, you won’t lose your files, which will be there waiting for you to re-connect to them after you have bought a new device. But that’s not backup.
What to backup on? Storage is the physical medium on which you place the files. Internal hard drives, pen drives, external SSD’s, Network drives and NAS, cloud drives, tape, CD’s. You can also store files on a disk in some company’s cloud service, effectively renting a hard drive from them. You have to backup to a physically different device, which you should then keep somewhere else. (So if someone drops something on the computer, the backup is in another room.)
How to backup?
Both Apple and Microsoft offer a continuous backup service: Time Machine for Apple and File History for Windows 10. Setting these up requires an external drive that can be configured for Time Machine / File History: that’s not always a given. Note also that both those are complex pieces of software. If you take your Time Machine disk to someone else’s computer, good luck if that other computer is a Windows 7 device. Time Machine and File History are good insurance against something stupid happening to your files, but not against something stupid happening to your computer.
A good backup sits on a drive that any computer can read and copy from. Since every operating system can read Windows file system disks, put your backup on a NTFS drive, in other words, get an off-the-shelf external drive from Amazon, PC World or any other major retailer. I am not going to get involved in USB types at this point. Roughly, make sure the USB port on the drive looks like one of the USB ports on your computer. Or use a NAS, which plugs into the router so you connect over Wi-Fi.
How to get the files over? There are back-up programs available that have nice user interfaces, but there are also much simpler options. On Windows 10 (and any other version) look up the xcopy command. This is a thing of wonder. There seems to be an OS X version called rsync that does the same thing. Both these tools only copy new or updated files relative to the destination folder. So the first one is painful, but the following ones can be quite fast. At work I use a Powershell script to sync my local drive to a personal network drive at the end of each week, or whenever I’m taking the laptop to and from work.
What to backup? Your ‘data’ of course. You may want to backup the software you’ve downloaded (the files in your Download directory) so you can re-install it if need be: this may be sensible if the latest versions are not compatible with your machine or operating system (OS X users will understand).
So let’s take a work-in-progress document. It’s in my iCloud drive, and hence on my laptop, maybe another device, and on Apple’s servers. iCloud is a synching service, not a backup. So I need a backup in case I do something stupid on iCloud. The laptop drive is the first stop: I save from the iCloud drive to my Documents folder daily. I backup my Documents folder up to my NAS at least once a week. And I backup my NAS to an external drive every month, or straight after I add significant amounts of data to the NAS (by ripping a bunch of CDs). I have two external drives and rotate them.
A professional photographer, video-maker or musician may want to take archive copies of the original files before even thinking about importing to the editing software. In the future it will be those archives that scholars want, not the finished product, which will reflect the fashions of the moment.
Having reviewed all this, my shortcomings are not so bad. I’ve understood the correct role of cloud drives, and the limitation of the Time Machine / File History technology.
I need to backup my Mac Air Documents and Photos folders to the NAS more frequently than I do. I’ll do that with rsync. I need an external drive to backup my NAS, and I’ll do that with xcopy. I could do with tidying up the NAS and taking the plunge of doing a soft reset because I forgot its admin password (!).
(Later). I sorted out the NAS. The filesystem works fine, but the bit that runs diagnostic tests and sets up a Time Machine volume seems to have gone astray. Or it isn’t compatible with Catalina (I updated to Catalina after finding enough people who said it worked fine on their Early 2015 Air. It does.)
In the old days we put books on shelves or in boxes; photographs in albums, or in boxes; records and CDs on shelves, or in boxes; and any documents we generated in files, or in boxes. How easy was that? Then computers came along and made everything simpler by making it more complicated. Everything was stored on the hard drive - passing over the days of floppy drives and diskettes - but what happened if the hard drive packed up? Hence backup which meant taking a copy of everything on another disk and storing that somewhere safe. Real Men did backups daily, weekly, and monthly. The daily and weekly ones were cycled, the monthly ones kept forever. On tape. Lots of people didn’t do backups, and every now and then they wrote to Jack Schofield at the Guardian about the disaster, and Uncle Jack scolded them lightly.
Backup is copies you make on a storage device and then leave alone unless disaster strikes, when you copy them back to wherever you copied them from. You never edit backup copies. Ever. You should keep at least two generations of backups.
Cloud drives in particular and synching services in general, are not backup. Do something dumb on one synched device, like delete everything, and all the other synched devices will lose everything as well. iCloud, One Drive, Google Drive and all the other services have this feature. To make up for it, some of them add a versioning, recovery or history feature that gets over this. When I tried using the versioning on Pages on my 4GB Mac Air, it choked so bad I had to turn it off.
A Cloud drive is what you use when you want to access, edit and save back your work-in-progress files from a number of different devices. Put your photos up in the iCloud and you can regale everyone with your holiday snaps on your iPhone, iPad, Macbook or even Windows device. Ditto music, though why you would when you can get it from a streaming service is another question. Unless it’s a recording of your band, or a track you’re working on. Same for videos.
Cloud drives offer you the assurance that even if your device gets lost, goes crazy or gets broken, you won’t lose your files, which will be there waiting for you to re-connect to them after you have bought a new device. But that’s not backup.
What to backup on? Storage is the physical medium on which you place the files. Internal hard drives, pen drives, external SSD’s, Network drives and NAS, cloud drives, tape, CD’s. You can also store files on a disk in some company’s cloud service, effectively renting a hard drive from them. You have to backup to a physically different device, which you should then keep somewhere else. (So if someone drops something on the computer, the backup is in another room.)
How to backup?
Both Apple and Microsoft offer a continuous backup service: Time Machine for Apple and File History for Windows 10. Setting these up requires an external drive that can be configured for Time Machine / File History: that’s not always a given. Note also that both those are complex pieces of software. If you take your Time Machine disk to someone else’s computer, good luck if that other computer is a Windows 7 device. Time Machine and File History are good insurance against something stupid happening to your files, but not against something stupid happening to your computer.
A good backup sits on a drive that any computer can read and copy from. Since every operating system can read Windows file system disks, put your backup on a NTFS drive, in other words, get an off-the-shelf external drive from Amazon, PC World or any other major retailer. I am not going to get involved in USB types at this point. Roughly, make sure the USB port on the drive looks like one of the USB ports on your computer. Or use a NAS, which plugs into the router so you connect over Wi-Fi.
How to get the files over? There are back-up programs available that have nice user interfaces, but there are also much simpler options. On Windows 10 (and any other version) look up the xcopy command. This is a thing of wonder. There seems to be an OS X version called rsync that does the same thing. Both these tools only copy new or updated files relative to the destination folder. So the first one is painful, but the following ones can be quite fast. At work I use a Powershell script to sync my local drive to a personal network drive at the end of each week, or whenever I’m taking the laptop to and from work.
What to backup? Your ‘data’ of course. You may want to backup the software you’ve downloaded (the files in your Download directory) so you can re-install it if need be: this may be sensible if the latest versions are not compatible with your machine or operating system (OS X users will understand).
So let’s take a work-in-progress document. It’s in my iCloud drive, and hence on my laptop, maybe another device, and on Apple’s servers. iCloud is a synching service, not a backup. So I need a backup in case I do something stupid on iCloud. The laptop drive is the first stop: I save from the iCloud drive to my Documents folder daily. I backup my Documents folder up to my NAS at least once a week. And I backup my NAS to an external drive every month, or straight after I add significant amounts of data to the NAS (by ripping a bunch of CDs). I have two external drives and rotate them.
A professional photographer, video-maker or musician may want to take archive copies of the original files before even thinking about importing to the editing software. In the future it will be those archives that scholars want, not the finished product, which will reflect the fashions of the moment.
Having reviewed all this, my shortcomings are not so bad. I’ve understood the correct role of cloud drives, and the limitation of the Time Machine / File History technology.
I need to backup my Mac Air Documents and Photos folders to the NAS more frequently than I do. I’ll do that with rsync. I need an external drive to backup my NAS, and I’ll do that with xcopy. I could do with tidying up the NAS and taking the plunge of doing a soft reset because I forgot its admin password (!).
(Later). I sorted out the NAS. The filesystem works fine, but the bit that runs diagnostic tests and sets up a Time Machine volume seems to have gone astray. Or it isn’t compatible with Catalina (I updated to Catalina after finding enough people who said it worked fine on their Early 2015 Air. It does.)
Thursday 25 April 2019
Bye-bye Tidal, Hello FTTC, Bye-Bye Yolt
The Highly Significant Birthday is approaching. I have booked the week before and after off. In case I get some uncontrollable emotions, or just don't want to get out of bed. Sometimes, though, I'm not sure I can tell what is actual emotion and what is pollen and too little sleep. So my posting is going to be a bit erratic.
In other news, I dropped Tidal. Listen, I searched for "Shoegaze" and it came back with about three entries. A colleague at work searched on Spotify, and it returned pages of the stuff. So I cancelled Tidal and signed up for Spotify.
The king of shoegaze compilations on You Tube seems to be the unlikely-moinkered Tabitha Mustang. If you've never heard any shoegaze, try this
I finally gave up on my old-school copper broadband service from Talk-Talk, and upgraded to their FTTC (fibre to the cabinet) service. Which has churned out 40Mbs down and 10Mbps up so far since. I even speed-tested it, and got pretty darn close, even though the route included the wireless-ac to my laptop. 40Mbs down by the way, is slightly more than what we old telecom folk used to call a DS3 (34Mbs). Back in the mid-1990's I am told, selling a DS3 across the Atlantic meant celebrations involving champagne and nightclubs. Now every home can have one.
And then one day my Yolt app demanded my passport number and other KYC (as we in the retail banking trade call it) details. And it would not let me get to the control panel without it. I was upset by this, because I don't like software strong-arming me, found the Contact Us email on their site, and asked them to DELETE MY ACCOUNT several times in all caps. Which they did without any fuss. My suspicion that they were about to launch actual banking services via the app was confirmed a day or so ago in the news.
And over Easter, I listened to Parsifal and Gotterdammerung on Spotify. Probably not quite CD quality, but it confirmed to me why I'm not rushing to get Wagner in my collection. The first movement of Parsifal is musically astonishing, whether you understand German or not. But the second act is a lot of singing, and the music probably means a lot more if you know what Kundry just tried to suggest to Parsifal.
In other news, I dropped Tidal. Listen, I searched for "Shoegaze" and it came back with about three entries. A colleague at work searched on Spotify, and it returned pages of the stuff. So I cancelled Tidal and signed up for Spotify.
The king of shoegaze compilations on You Tube seems to be the unlikely-moinkered Tabitha Mustang. If you've never heard any shoegaze, try this
I finally gave up on my old-school copper broadband service from Talk-Talk, and upgraded to their FTTC (fibre to the cabinet) service. Which has churned out 40Mbs down and 10Mbps up so far since. I even speed-tested it, and got pretty darn close, even though the route included the wireless-ac to my laptop. 40Mbs down by the way, is slightly more than what we old telecom folk used to call a DS3 (34Mbs). Back in the mid-1990's I am told, selling a DS3 across the Atlantic meant celebrations involving champagne and nightclubs. Now every home can have one.
And then one day my Yolt app demanded my passport number and other KYC (as we in the retail banking trade call it) details. And it would not let me get to the control panel without it. I was upset by this, because I don't like software strong-arming me, found the Contact Us email on their site, and asked them to DELETE MY ACCOUNT several times in all caps. Which they did without any fuss. My suspicion that they were about to launch actual banking services via the app was confirmed a day or so ago in the news.
And over Easter, I listened to Parsifal and Gotterdammerung on Spotify. Probably not quite CD quality, but it confirmed to me why I'm not rushing to get Wagner in my collection. The first movement of Parsifal is musically astonishing, whether you understand German or not. But the second act is a lot of singing, and the music probably means a lot more if you know what Kundry just tried to suggest to Parsifal.
Monday 3 September 2018
Fixing That Missing Artwork in iTunes
Macs make a lot more sense when you think of them as designed to be stand-alone. OS X can do networking, but it doesn’t really get it. That’s partly because of the UNIX everything-is-a-file philosophy: Unix doesn’t see networks, it only sees external drives. Windows sees networks and drives on the networks, which sounds as if it should be the same, but really is not.
Windows exposes the filesystem to the user in its programmes, and asks the user to make the filesystem their friend. In Windows you open a file by navigating through the filesystem in the File Open dialog or by clicking on a name in the Recently Used Files list. The only time you see icons representing your files in Windows is a) when you’re looking at the directory in Explorer, or b) when you’re using a programme that’s modelled on something from OSX, such as Calibre or most music management programs.
By design, OS X hides all the nasty OS stuff - such as the filesystem - from the user. This is usually done by adding an intermediate layer between the user interface and the filesystem. In an OS X programme the user sees a friendly icon and clicks on that: the programme looks at a database file - usually in XML format - that tells it where the file is stored. This works wonderfully until the user does something with their music files directly via the filesystem, say, with Finder and then everything goes to pieces. Because the database behind the cool interface has not been updated and indeed can’t be updated with changes made directly via Finder or with Unix commands on the Terminal.
OS X wants you to deal with your music collection ONLY through iTunes or a similar programme. That is the only way the database is kept up to date. It’s also the only way that the very quirky file permissions that iTunes uses get handled correctly. It’s possible to alter the file and directory permissions to 777 (all access to everyone) in Terminal but not have that show up in Finder, or have iTunes behave as if the files and directory have permission 777. (Oh yes, I have done this.)
(Windows programs assume that you’re going to do all sorts of things, so they have a ‘Watch this directory’ function that scans and updates the cool interface when you start the program. That can be kinda annoying at times. But it’s swapping one annoyance for another.)
Suppose you copied Volunteers by the Jefferson Airplane from your Macbook iTunes directory to a backup directory on your NAS. When you later build an iTunes library from this NAS directory, it will add the Volunteers album and files it finds, but you won’t be able to modify the album info, and the artwork will get lost. What’s happened is that the file and directory permissions have become mis-aligned - not in Terminal world, but in the parallel universe of iTunes / Finder permissions.
So how do you get the artwork back, and become able to change the album info?
Here’s the fix that occurred to me…
1. Create a directory in the Music section on the computer that has the iTunes you are using. This will typically be the SDD of your Mac Book / Air / Pro. I called mine Temp. It doesn’t matter.
2. Copy the files from Volunteers (or whatever) into this Temp directory.
3. Use iTunes to delete the original Songs and the Files.
4. Check that the directory for Volunteers (or whatever) is also deleted, as it might not be if it has artwork files or other stuff in it
5. in iTunes, do File -> Add to Library and choose Temp. iTunes will now do its stuff.
6. Now you can add the album artwork and make whatever other changes you want, and it will stick.
7. Delete the files in Temp.
8. Wash. Rinse. Repeat.
9. Works On My Machine and NAS. YMMV.
It sounds clunky, but it soon become muscle memory. Do it as a background job when you’re writing blog entries or something, and you’ll be all caught up before you know it.
Windows exposes the filesystem to the user in its programmes, and asks the user to make the filesystem their friend. In Windows you open a file by navigating through the filesystem in the File Open dialog or by clicking on a name in the Recently Used Files list. The only time you see icons representing your files in Windows is a) when you’re looking at the directory in Explorer, or b) when you’re using a programme that’s modelled on something from OSX, such as Calibre or most music management programs.
By design, OS X hides all the nasty OS stuff - such as the filesystem - from the user. This is usually done by adding an intermediate layer between the user interface and the filesystem. In an OS X programme the user sees a friendly icon and clicks on that: the programme looks at a database file - usually in XML format - that tells it where the file is stored. This works wonderfully until the user does something with their music files directly via the filesystem, say, with Finder and then everything goes to pieces. Because the database behind the cool interface has not been updated and indeed can’t be updated with changes made directly via Finder or with Unix commands on the Terminal.
OS X wants you to deal with your music collection ONLY through iTunes or a similar programme. That is the only way the database is kept up to date. It’s also the only way that the very quirky file permissions that iTunes uses get handled correctly. It’s possible to alter the file and directory permissions to 777 (all access to everyone) in Terminal but not have that show up in Finder, or have iTunes behave as if the files and directory have permission 777. (Oh yes, I have done this.)
(Windows programs assume that you’re going to do all sorts of things, so they have a ‘Watch this directory’ function that scans and updates the cool interface when you start the program. That can be kinda annoying at times. But it’s swapping one annoyance for another.)
Suppose you copied Volunteers by the Jefferson Airplane from your Macbook iTunes directory to a backup directory on your NAS. When you later build an iTunes library from this NAS directory, it will add the Volunteers album and files it finds, but you won’t be able to modify the album info, and the artwork will get lost. What’s happened is that the file and directory permissions have become mis-aligned - not in Terminal world, but in the parallel universe of iTunes / Finder permissions.
So how do you get the artwork back, and become able to change the album info?
Here’s the fix that occurred to me…
1. Create a directory in the Music section on the computer that has the iTunes you are using. This will typically be the SDD of your Mac Book / Air / Pro. I called mine Temp. It doesn’t matter.
2. Copy the files from Volunteers (or whatever) into this Temp directory.
3. Use iTunes to delete the original Songs and the Files.
4. Check that the directory for Volunteers (or whatever) is also deleted, as it might not be if it has artwork files or other stuff in it
5. in iTunes, do File -> Add to Library and choose Temp. iTunes will now do its stuff.
6. Now you can add the album artwork and make whatever other changes you want, and it will stick.
7. Delete the files in Temp.
8. Wash. Rinse. Repeat.
9. Works On My Machine and NAS. YMMV.
It sounds clunky, but it soon become muscle memory. Do it as a background job when you’re writing blog entries or something, and you’ll be all caught up before you know it.
Monday 27 August 2018
Ripping The CD Collection
I’m not an audiophile, but I want the music to sound right. What comes out of my iPhone through the Bose QC20’s noise-cancelling earphones is right for the purpose, but what comes out of the headphone socket through my hifi is muffled and tiring. Same signal, different context. Hence all my fussing around with Dragonflys and Jitterbugs earlier this year. While not quite up to the DAC in my Marantz CD player, the combination is close, and the sound isn’t tiring. And I fall asleep to music from the Bose Colour Sound fed via Bluetooth from my iPhone streaming via WiFi from the NAS.
I was not ripping the CD collection for backup. That would have meant copying the discs. And then buying another NAS to backup the backup, because NAS can fail more frequently than a CD. Anyway, there is a CD backup service: it’s called Amazon, or Foyles, or any number of other online retailers. This won’t work for compilations of obscure composers by performers whose careers never lived up to their early promise (British Piano Music of the 1980’s where art though now?) but then, well, how often did you listen to it when you had it? Classic performances that you really want tend to be re-released. Or surpassed.
I ripped the collection because I stream more than I do, and may stream more, especially with the Sonos in the front room, and because I do put some of it on the iPhone and the Nano for portable use from time to time. And I’ve only got 128GB on my Air SDD. Which is my way of explaining why I used the iTunes default of 192kps M4a.
Why did I use iTunes? For one thing, Hans Beekhuysen mentions it as one of three which are decent rippers. Since one of the others is Roon, that’s a pretty good recommendation. iTunes is good at getting song titles and parsing the artist / composer, no worse than any of the others at getting album art (I tried Clementine: it won some, lost others) and once you accept its quirky little ways is pretty good at managing the library. It’s not Roon. But neither is the price.
So the workflow looks like this:
0. Create an Album Artwork directory on your Mac. Open a new music library in iTunes and point it at your NAS. Save.
1. Put CD into drive
2. Choose the album title that a) isn’t in Japanese, and b) doesn’t say it’s Disk 3 of some ‘Best of Bach’ collection when your CD is a stand-alone
3. Let iTunes do its thing
4. While it is, if you ripping a jazz or classical CD, copy the CD title, paste it into the Amazon search box and see if they have a decent copy of the artwork. Almost always they will, but if not, use Google. I did so on less than one in thirty CDs. Paste the CD title into the the ‘Save As’ name and save it to your Album Artwork directory (*).
5. When the CD is done, eject it, and right-click for Album Info. Here I put in the artwork, change the Album Artist to the composer for classical music, and get rid of the [Disc 1] that often appears in titles. Sometimes, as with the 22-CD Stravinsky set, the disk number is useful, but not for double-albums.
6. Untick that damn Album is compilation of songs by various artists box. Otherwise it winds up in a Compilations directory. And you won’t find it when browsing with File Explorer.
7. Press OK and find something to do while iTunes does its thing with the music files over the WiFi to the NAS.
It’s tedious. It’s best done while doing something else, pausing to deal with the album info, eject the disc, put another one in. I did it in batches of about twenty, one day at a a time, until it was over.
8. Review the results and edit. How much work you put into changing what iTunes (or any other organiser) found is up to you. Filling in the missing artwork, un-compiling compilations that aren’t really, making sure that J S Bach is spelled and spaced like that in all the albums so it’s easier to find when browsing outside of iTunes… just how anally-retentive are you? (Turns away as if this doesn’t concern him…) The day you see me changing genres, I really will have nothing to do.
9. Accept that the music catalogue is a case of progress not perfection. I’m going to make tweaks every now and then when I notice something.
Because some of the files I have were ripped earlier, under different versions of iTunes, there were permission issues, and I needed to refresh the library in the way described in a future post. It was worth it. A library with every bit of cover art and all the double-albums put together is a thing of delight.
(*) WHY THE FRACK DON’T JAZZ and CLASSICAL CD’S HAVE ARTWORK? I load a progressive house CD, it has artwork. I load a Mahler box-set, I have to get the artwork myself. Digital music libraries and organiser programs have been with us for over a decade, and iTunes, Roon and all the others aren't going to disappear. How difficult can it be for a record company to package all its artwork up and send the zip to Apple? And how difficult would it be for Apple not to charge the record company, in the name of giving us all a better experience? Not even Amazon restrict the number of times we can find and download artwork. (Because they’re smart: every time you get some artwork from them, they get some more goodwill, and you might buy something.) Music industry, get your freaking act together on this.
I was not ripping the CD collection for backup. That would have meant copying the discs. And then buying another NAS to backup the backup, because NAS can fail more frequently than a CD. Anyway, there is a CD backup service: it’s called Amazon, or Foyles, or any number of other online retailers. This won’t work for compilations of obscure composers by performers whose careers never lived up to their early promise (British Piano Music of the 1980’s where art though now?) but then, well, how often did you listen to it when you had it? Classic performances that you really want tend to be re-released. Or surpassed.
I ripped the collection because I stream more than I do, and may stream more, especially with the Sonos in the front room, and because I do put some of it on the iPhone and the Nano for portable use from time to time. And I’ve only got 128GB on my Air SDD. Which is my way of explaining why I used the iTunes default of 192kps M4a.
Why did I use iTunes? For one thing, Hans Beekhuysen mentions it as one of three which are decent rippers. Since one of the others is Roon, that’s a pretty good recommendation. iTunes is good at getting song titles and parsing the artist / composer, no worse than any of the others at getting album art (I tried Clementine: it won some, lost others) and once you accept its quirky little ways is pretty good at managing the library. It’s not Roon. But neither is the price.
So the workflow looks like this:
0. Create an Album Artwork directory on your Mac. Open a new music library in iTunes and point it at your NAS. Save.
1. Put CD into drive
2. Choose the album title that a) isn’t in Japanese, and b) doesn’t say it’s Disk 3 of some ‘Best of Bach’ collection when your CD is a stand-alone
3. Let iTunes do its thing
4. While it is, if you ripping a jazz or classical CD, copy the CD title, paste it into the Amazon search box and see if they have a decent copy of the artwork. Almost always they will, but if not, use Google. I did so on less than one in thirty CDs. Paste the CD title into the the ‘Save As’ name and save it to your Album Artwork directory (*).
5. When the CD is done, eject it, and right-click for Album Info. Here I put in the artwork, change the Album Artist to the composer for classical music, and get rid of the [Disc 1] that often appears in titles. Sometimes, as with the 22-CD Stravinsky set, the disk number is useful, but not for double-albums.
6. Untick that damn Album is compilation of songs by various artists box. Otherwise it winds up in a Compilations directory. And you won’t find it when browsing with File Explorer.
7. Press OK and find something to do while iTunes does its thing with the music files over the WiFi to the NAS.
It’s tedious. It’s best done while doing something else, pausing to deal with the album info, eject the disc, put another one in. I did it in batches of about twenty, one day at a a time, until it was over.
8. Review the results and edit. How much work you put into changing what iTunes (or any other organiser) found is up to you. Filling in the missing artwork, un-compiling compilations that aren’t really, making sure that J S Bach is spelled and spaced like that in all the albums so it’s easier to find when browsing outside of iTunes… just how anally-retentive are you? (Turns away as if this doesn’t concern him…) The day you see me changing genres, I really will have nothing to do.
9. Accept that the music catalogue is a case of progress not perfection. I’m going to make tweaks every now and then when I notice something.
Because some of the files I have were ripped earlier, under different versions of iTunes, there were permission issues, and I needed to refresh the library in the way described in a future post. It was worth it. A library with every bit of cover art and all the double-albums put together is a thing of delight.
(*) WHY THE FRACK DON’T JAZZ and CLASSICAL CD’S HAVE ARTWORK? I load a progressive house CD, it has artwork. I load a Mahler box-set, I have to get the artwork myself. Digital music libraries and organiser programs have been with us for over a decade, and iTunes, Roon and all the others aren't going to disappear. How difficult can it be for a record company to package all its artwork up and send the zip to Apple? And how difficult would it be for Apple not to charge the record company, in the name of giving us all a better experience? Not even Amazon restrict the number of times we can find and download artwork. (Because they’re smart: every time you get some artwork from them, they get some more goodwill, and you might buy something.) Music industry, get your freaking act together on this.
Monday 23 July 2018
Writing a Python Program: Tools
Real Developers use vi at the command line and have memorised the every single Python library. Actually, Read Developers probably don't use Python. This is for the rest of us.
There’s no question of anyone knowing all the Python libraries by heart, let alone by muscle memory. It’s impossible for any one mere mortal to know the Excel VBA object model and all the constants. That’s why VBA has Intellisense, which Microsoft patented, forcing everyone else to provide a slightly cut back version called ‘auto-complete’. PyCharm has autocomplete, but not the full-fledged Intellisense. Auto-complete doesn’t walk you through all the parameters for a function, nor offer values when the number of options is less than about five. Microsoft spent all that money developing Intellisense because they knew if would pay back in productivity and user loyalty.
The minimalist Python IDE is IDLE, which doesn’t have a visual form editor. To find the typos in your code, you have to run the program and then deal with the messages from the python debugger. That’s what makes it minimalist.
PyCharm Community Edition has a neat feature where it puts red line in the right of the code screen against the lines it thinks are wrong in some way, thus sparing you doing debugging runs to find the bits where you forgot to put ‘:’ at the end of a ‘def’. This helps, but if the interpreter spots something else wrong while your code is running, you have to correct it and start over.
This is when one realises that the VBA editor / debugger is a thing of utter wonder. Being able to do on-the-fly code correction while debugging is like having a superpower. No other IDE provides it. A search for the reason brought up a comment from someone on the original VBA project to the effect that the trick was keeping track of all the threads and re-aligning them when the user had re-coded and started from some earlier point in the program that they halted. That is nowhere near as easy to do as it is to say.
While it doesn’t take much time to stop, edit, and re-run code from the start, as PyCharm and all the others force us to do, compared with the Microsoft Way, it just feels clunky. To me.
I'm going to try Visual Studio Code from Microsoft next time I do something. The demo looks interesting. Anything to get on-the-fly debugging again.
But then Real Developers don't need to debug.
(PS: It's too hot for me to keep up my usual schedule. Normal service will be resumed when this hot spell breaks.)
Monday 18 June 2018
How To Run a Python program? And Why Is It A Good Thing?
Isn’t that a question with an obvious answer? Not so much. Answering it takes us to the heart of how Python works.
Write a Windows program in a language such as C#, and the IDE compiles to a self-contained .exe file (as much as any .exe is ‘self-contained’ when it makes calls to all those Windows DLL’s). My reflexes have been developed on languages like that: compile and run.
So my first thought, having got code that worked as it was supposed to, was to make an .exe file. It was then I found out that PyCharm doesn’t compile Python scripts as, say, a C# IDE does. If you really want an .exe file, you have to find a third-party application to compile the scripts and wrap in all the dependencies. The most popular way of doing that is to use Py2Exe. That only works on Python 2.7, and hasn’t been upgraded to handle the 3.x versions of Python. Lesson in there somewhere.
The lesson is this. There are two Pythons. One is the scripting language. The other is a program that runs those scripts. That’s why the command line prompt is ‘python mystuff.py’. It’s telling the Python program to run the Python script in ‘mystuff.py’.
The Python program on Windows converts a .py file into something Windows understands. The Python program on OS X converts the same .py file into something OS X understands. So .py files are portable. mystuff.py will run on any computer with a Python program. (Given Python version compatibility.)
Browsers work in the same way. You download a web page full of all sorts of HTML and other programming junk, and the browser interprets it all in terms the operating system understands and throws the result at the computer to run.
Which means that ‘everyone’ has to have a Python install to use a .py script. Well, okay. Would you expect to be able to ‘run’ an Excel workbook without Excel? You need Excel to be installed.
So how do I run the program? Set up a .bat file with the command line in it, and put that in the Start Menu. Click on it, and it will start up like a ‘real’ .exe program. Which is all I need.
Write a Windows program in a language such as C#, and the IDE compiles to a self-contained .exe file (as much as any .exe is ‘self-contained’ when it makes calls to all those Windows DLL’s). My reflexes have been developed on languages like that: compile and run.
So my first thought, having got code that worked as it was supposed to, was to make an .exe file. It was then I found out that PyCharm doesn’t compile Python scripts as, say, a C# IDE does. If you really want an .exe file, you have to find a third-party application to compile the scripts and wrap in all the dependencies. The most popular way of doing that is to use Py2Exe. That only works on Python 2.7, and hasn’t been upgraded to handle the 3.x versions of Python. Lesson in there somewhere.
The lesson is this. There are two Pythons. One is the scripting language. The other is a program that runs those scripts. That’s why the command line prompt is ‘python mystuff.py’. It’s telling the Python program to run the Python script in ‘mystuff.py’.
The Python program on Windows converts a .py file into something Windows understands. The Python program on OS X converts the same .py file into something OS X understands. So .py files are portable. mystuff.py will run on any computer with a Python program. (Given Python version compatibility.)
Browsers work in the same way. You download a web page full of all sorts of HTML and other programming junk, and the browser interprets it all in terms the operating system understands and throws the result at the computer to run.
Which means that ‘everyone’ has to have a Python install to use a .py script. Well, okay. Would you expect to be able to ‘run’ an Excel workbook without Excel? You need Excel to be installed.
So how do I run the program? Set up a .bat file with the command line in it, and put that in the Start Menu. Click on it, and it will start up like a ‘real’ .exe program. Which is all I need.
Monday 11 June 2018
Learning Python by Writing An Application (2)
I haven’t written high-security web applications that link to secure back-end databases, so I’m not talking about those. I’m talking about regular, runs-on-your-computer applications with a UI.
Real programmers know that their command-line utility never hangs and has a try...else to make it stop at the first sign of trouble, so they don’t need it to talk to them. I hate it when a program sits there seemingly doing nothing, I want some feedback. There’s a trade-off between speed and user dialogue. No, not progress bars. Progress bars require sizing the job first, so that the progress can be shown as a percentage of the total. Ever noticed that if the number of files you’re copying is large enough, Windows suddenly takes an age doing some mysterious preparation? That’s what I want to avoid. The only way round that is a running commentary. I don’t want much, maybe something like
Scanned N directories, copied M files to (destination)
updated every 10 files. When it’s syncing, I want to see
Scanned N directories, deleted M files from (destination)
(Real Java Programmers would write a log file of every action, along with the time it happened. I’m so glad I’m not a Real Java Programmer.)
Tkinter is notorious for updating at random. I don’t think this is Python’s fault, but the way the OS and hardware parcel out the tasks. So by the time Tkinter gets a moment to display the first confirmation, the whole task might have been done. To get round this, I called my first thread! I’m a VBA-basher in my day job so threading is something exotic and mysterious, like (insert inappropriate analogy here). It solved the Tkinter delay problem nicely.
My program has a button to choose the source directory, another to choose the target, one to Start and one to say Stop. Think about the UI logic. The Stop button should not be active unless there’s copying or syncing going on. The Run button should not be active unless there are valid source and destination directories.
What happens if I copy a file and there isn’t enough spare space on the target disc? At the very least I’m going to get an error message, and so I’d need to handle that. Or I could test for spare space and stop copying when I reach some kind of limit. That’s what I did: read in the spare space at the start, store that in a variable, subtract the copied file size from that value, and stop if the file to be copied is within some limit of the calculated space. Using the constant cuts down on filesystem calls.
What happens when there’s already a file with the name of the being copied BUT the size is different? The program needs to add a suffix to the filename, but then it has to test that that filename doesn’t exist, which sounds like a loop. And don’t forget to put a numerical limit on the number of attempts just in case someone used a silly file-naming convention.
A lot of programming is testing for and resolving edge-cases. For instance, the destination directory should not be the source directory, nor a subdirectory of the source directory. I got the first condition on the first pass, but only thought of the second when writing this. (Documentation reveals many improvements.)
This is all exception and edge-case handling.
In my experience, the actual business functionality, the code that does what you want done, is about ten per cent of the total code. Variable declarations and syntactic seasoning is about another ten per cent. Setting up, controlling and making the internal logic of the UI consistent, will be about twenty-five percent, depending on how complex it is.
All the rest is handling edge cases, data weirdness, potential errors, null values, and other stuff that might trip up the program.
I’m not kidding.
If half your code isn’t testing that things that need to be there for the next step to work are actually there, or for missing values, forcing stuff to be strings when the function expects strings, putting limits on loops in case something you have never thought could happen does happen and the terminating condition is never reached... if you’re not doing all that, and you’re not writing in sensible defaults for when the user has to make a choice, and you’re not testing for when the user does something really dumb because they’re distracted, or walks away for five minutes, or all those other things you don’t want users to do because then your life gets difficult... then you’re going to produce a program that people are not going to use twice.
The difference between a professional and an amateur is that the professional writes all the edge-case and exception handling they can think of. An amateur thinks ‘Well, the user will just have to...’.
Don’t forget to put in a default drive for the source and destination directories, or the getdirectory() function will dump the user deep down a file tree it will take a boring time to get out of and get to where the user should have been in the first place.
Programming isn’t about writing code to do stuff. That’s the easy part. It’s about writing code that makes sure the user and the computer don’t get in the way of the stuff getting done.
Real programmers know that their command-line utility never hangs and has a try...else to make it stop at the first sign of trouble, so they don’t need it to talk to them. I hate it when a program sits there seemingly doing nothing, I want some feedback. There’s a trade-off between speed and user dialogue. No, not progress bars. Progress bars require sizing the job first, so that the progress can be shown as a percentage of the total. Ever noticed that if the number of files you’re copying is large enough, Windows suddenly takes an age doing some mysterious preparation? That’s what I want to avoid. The only way round that is a running commentary. I don’t want much, maybe something like
Scanned N directories, copied M files to (destination)
updated every 10 files. When it’s syncing, I want to see
Scanned N directories, deleted M files from (destination)
(Real Java Programmers would write a log file of every action, along with the time it happened. I’m so glad I’m not a Real Java Programmer.)
Tkinter is notorious for updating at random. I don’t think this is Python’s fault, but the way the OS and hardware parcel out the tasks. So by the time Tkinter gets a moment to display the first confirmation, the whole task might have been done. To get round this, I called my first thread! I’m a VBA-basher in my day job so threading is something exotic and mysterious, like (insert inappropriate analogy here). It solved the Tkinter delay problem nicely.
My program has a button to choose the source directory, another to choose the target, one to Start and one to say Stop. Think about the UI logic. The Stop button should not be active unless there’s copying or syncing going on. The Run button should not be active unless there are valid source and destination directories.
What happens if I copy a file and there isn’t enough spare space on the target disc? At the very least I’m going to get an error message, and so I’d need to handle that. Or I could test for spare space and stop copying when I reach some kind of limit. That’s what I did: read in the spare space at the start, store that in a variable, subtract the copied file size from that value, and stop if the file to be copied is within some limit of the calculated space. Using the constant cuts down on filesystem calls.
What happens when there’s already a file with the name of the being copied BUT the size is different? The program needs to add a suffix to the filename, but then it has to test that that filename doesn’t exist, which sounds like a loop. And don’t forget to put a numerical limit on the number of attempts just in case someone used a silly file-naming convention.
A lot of programming is testing for and resolving edge-cases. For instance, the destination directory should not be the source directory, nor a subdirectory of the source directory. I got the first condition on the first pass, but only thought of the second when writing this. (Documentation reveals many improvements.)
This is all exception and edge-case handling.
In my experience, the actual business functionality, the code that does what you want done, is about ten per cent of the total code. Variable declarations and syntactic seasoning is about another ten per cent. Setting up, controlling and making the internal logic of the UI consistent, will be about twenty-five percent, depending on how complex it is.
All the rest is handling edge cases, data weirdness, potential errors, null values, and other stuff that might trip up the program.
I’m not kidding.
If half your code isn’t testing that things that need to be there for the next step to work are actually there, or for missing values, forcing stuff to be strings when the function expects strings, putting limits on loops in case something you have never thought could happen does happen and the terminating condition is never reached... if you’re not doing all that, and you’re not writing in sensible defaults for when the user has to make a choice, and you’re not testing for when the user does something really dumb because they’re distracted, or walks away for five minutes, or all those other things you don’t want users to do because then your life gets difficult... then you’re going to produce a program that people are not going to use twice.
The difference between a professional and an amateur is that the professional writes all the edge-case and exception handling they can think of. An amateur thinks ‘Well, the user will just have to...’.
Don’t forget to put in a default drive for the source and destination directories, or the getdirectory() function will dump the user deep down a file tree it will take a boring time to get out of and get to where the user should have been in the first place.
Programming isn’t about writing code to do stuff. That’s the easy part. It’s about writing code that makes sure the user and the computer don’t get in the way of the stuff getting done.
Thursday 7 June 2018
Learning Python by Writing An Application (1)
I don’t read Slashdot as much as I used to, and so I have no idea if people still ask “What’s the best way to learn a new (programming) language”? The answer was and ever more shall be: read a decent introduction to it, then write an application with it. Not toy examples, but something you will use. Why? Because then you will have to get to grips with the file-handling and string-processing functions, as well as some basic UI functionality, as well as the specialist libraries you need for the task.
That’s what I’m doing for Python.
Assume you have a bunch of files in a high-level directory called MyFiles, on the drive of your working computer. You want to make a backup of this from time to time. If it’s small, the simplest thing to do is copy MyFiles onto an external drive. With a lot of files that could get time-consuming, so maybe a backup program will do. Except that backup programs often create their own special file structures, databases, or at the least, zip everything up. I don’t want to use a special program to manage the backup: I want to use Windows Explorer.
What I need is a clever copying program. I tell it the source directory and the backup directory, and it recursively scans down the source tree, copying only the newfiles and the new directories it finds. Then, if I’ve asked it to, it removes any files from the backup directory that I deleted in the source directory since the last backup.
That’s the sort of thing that Python is good for. So I downloaded JetBrains’ PyCharm Community edition (because free) and started. I know how to program, and I understand basic Python syntax, but what I don’t have is a lot of familiarity with all the libraries, and especially Tkinter, which is what I needed for the user interface.
How does one cut code without knowing all the libraries? Slowly, with various reference websites open in the browser. That’s what manuals are for.
This illustrates the difference between being a programmer, and being a productive (insert name of language here) programmer. Because I’m a programmer, I can break a task into activities the computer or user needs to do (‘get the spare space on the destination drive’), and it usually turns out that the tasks I identify correspond pretty closely with the functions available in the system libraries. Programmers tend to think alike.
I understand control flow, arrays, variable types and declaration, threading, and use of functions to make code readable and maintainable. (I know, you want to hire me already.) I learned that writing FORTRAN, C, and VBA. Writing long chains of macros and the queries for them in Access is programming of a kind as well, as is constructing chains of raw SQL. It’s all about organising the resources of the language to get done what you need to get done.
Faced with a new language, I look for how those things are done, as well as any other cool tricks or idiosyncrasies it has. And, because I’m a programmer, I Read A Freaking Manual first. (This approach will not work with LISP and other such languages. Those really do need to be learned from zero, or you will miss the point.) On a meta-level, I’m Dreyfus-Proficient. That’s the engineering training. I’m just not slick.
All that said, I wouldn’t go for a Python job because one of the requirements is muscle-memory of the main system libraries, and familiarity with whatever the job-specific libraries are. I don’t want to be one of those guys who bluffed about their skills.
I have the sneaking suspicion that I’ve written VBA in Python, and that a Truly Pythonic Programmer would do the whole thing in two lines of code, one of which is a recursively self-referential array of filenames, and the other of which would implement a beautiful wxPython interface in less than twenty characters.
‘Pythonic Programming’ is a Thing. It’s a style. TPPs can do in a line what RJPs need six hundred lines to do. But becoming a TPP takes serious study, not to mention the right examples.
Next post, I’ll talk about the program.
That’s what I’m doing for Python.
Assume you have a bunch of files in a high-level directory called MyFiles, on the drive of your working computer. You want to make a backup of this from time to time. If it’s small, the simplest thing to do is copy MyFiles onto an external drive. With a lot of files that could get time-consuming, so maybe a backup program will do. Except that backup programs often create their own special file structures, databases, or at the least, zip everything up. I don’t want to use a special program to manage the backup: I want to use Windows Explorer.
What I need is a clever copying program. I tell it the source directory and the backup directory, and it recursively scans down the source tree, copying only the newfiles and the new directories it finds. Then, if I’ve asked it to, it removes any files from the backup directory that I deleted in the source directory since the last backup.
That’s the sort of thing that Python is good for. So I downloaded JetBrains’ PyCharm Community edition (because free) and started. I know how to program, and I understand basic Python syntax, but what I don’t have is a lot of familiarity with all the libraries, and especially Tkinter, which is what I needed for the user interface.
How does one cut code without knowing all the libraries? Slowly, with various reference websites open in the browser. That’s what manuals are for.
This illustrates the difference between being a programmer, and being a productive (insert name of language here) programmer. Because I’m a programmer, I can break a task into activities the computer or user needs to do (‘get the spare space on the destination drive’), and it usually turns out that the tasks I identify correspond pretty closely with the functions available in the system libraries. Programmers tend to think alike.
I understand control flow, arrays, variable types and declaration, threading, and use of functions to make code readable and maintainable. (I know, you want to hire me already.) I learned that writing FORTRAN, C, and VBA. Writing long chains of macros and the queries for them in Access is programming of a kind as well, as is constructing chains of raw SQL. It’s all about organising the resources of the language to get done what you need to get done.
Faced with a new language, I look for how those things are done, as well as any other cool tricks or idiosyncrasies it has. And, because I’m a programmer, I Read A Freaking Manual first. (This approach will not work with LISP and other such languages. Those really do need to be learned from zero, or you will miss the point.) On a meta-level, I’m Dreyfus-Proficient. That’s the engineering training. I’m just not slick.
All that said, I wouldn’t go for a Python job because one of the requirements is muscle-memory of the main system libraries, and familiarity with whatever the job-specific libraries are. I don’t want to be one of those guys who bluffed about their skills.
I have the sneaking suspicion that I’ve written VBA in Python, and that a Truly Pythonic Programmer would do the whole thing in two lines of code, one of which is a recursively self-referential array of filenames, and the other of which would implement a beautiful wxPython interface in less than twenty characters.
‘Pythonic Programming’ is a Thing. It’s a style. TPPs can do in a line what RJPs need six hundred lines to do. But becoming a TPP takes serious study, not to mention the right examples.
Next post, I’ll talk about the program.
Monday 2 April 2018
Upgrading the Windows Computer
Every now and then I think about upgrading my Windows laptop. At the moment I have a Samsung that may be seven or eight years old. It has a 17.5 inch screen, a 2GHz Pentium, a 500GB HDD, 4GB of RAM and a basic video card. The body is plastic and the keyboard is only for light use, and I mean maybe ten minutes at a time. It does the job I want it to do, and I suspect the video card is a part of that.
So I’ve been looking at laptops and mini-PC’s.
At the bottom end of the price range, say about £300 or so, are Celerons and Pentium with 2GB of RAM. At the other end are 4k video editing machines for photographers. These have GeForce 1040 or above video cards, 8GB+ of RAM, an i7 HQ, some SSD and a 1TB drive or a lot more SSD, and USB 3. / 3.1 to connect external HDD’s with decent transfer rates. MacBook Pros and Dell XPS 15’s are get a lot of mentions. (High-end gaming laptops are way over what I need.) These machines have good keyboards and aluminium unibodies (the Dell is almost an aluminium unibody).
In between is a mass of i5 / i7 U-series machines, with or without graphics cards, random amounts of SSD, rarely an HDD, random combinations of USB / HDMI / VGA ports and build quality that ranges from awful to okay-I-guess. These can vary in price between about £600 - £1,000. The only way to make sense of these component salads is to assume that Wintel manufacturers design a high-end model, a low-end model and a mid-range model, and all the others are put together from excess parts and left-overs.
The decisions are much simpler than all those fancy specs and combinations make it look.
Want to browse the net, do text-based work, basic photographic editing and adjustment, and send e-mails? But don’t do 1080p and upwards video-editing or scientific computing? Take a serious look at an iPad and an external keyboard.
Get a Macbook Pro if you want to do iOS, Mac or UNIX / Linux development.
Get the high-end £1,100+ video-editing capable machines if you want: the aluminium body, high-quality keyboard and sharp screen; a large HDD; to use Adobe Creative Cloud or the equivalents on 1080p and 4k movies.
Want to do lots of calculations but not much graphics? (Very rare). Then you can get one of the component-salads with an i7-HQ, 256GB of SSD and integrated graphics.
Sustained daily typing and use - because you’re an author, or journalist or other content-producer? Consider a Mac Air or one of the top-end machines. (The price difference is not that large, especially if it’s how you make your money.)
Anything else? Get the mid-range Wintel.
It’s the build quality. Once you’ve had Mac, you can’t go back. At work, they hand out POS HP’s with Win 7, VGA adapters, and a keyboard with the Page Up / Down / End / Home keys in the wrong place. But that’s institutional companies for you. Cheap. The Adobe Creative Cloud Suite user in the family does so on an MSI gaming machine with an i7 HQ and 16GB of RAM with a 17-inch screen. It’s wicked fast but it’s got that corporate cheap build feel. And he doesn’t work for a big company.
I don’t want to come home and use something similar to the junk they give us at work. It’s my home, not an office. Computers are one of the things I’m willing to Pay Good Money for. (Not Silly Money, but Good Money.) As a tool to do a job. And the video-editing performance laptop is a tool for a job I’m not going to do.
The outsider for my needs is an Asus mini-PC, which has the 1TB drive, an i5-7U series, a mid-range graphics card, some SSD, Wireless-AC, and ports out the wazoo. It can drive two external screens, which is a nice-to-have I’ve wanted for a while. It will be an ace media centre, but would need hi-fi to make nice sound, but then, so does a laptop. It costs £650 and already I have an external screen (the TV), a mouse and keyboard. At a pinch I could get a 21-inch monitor for about £150 and work on a table if I wanted to use it as a computer. Back to the future.
PS: I didn’t do any of this. I did something else instead that didn’t involve spending money. I’ll talk about that later.
So I’ve been looking at laptops and mini-PC’s.
At the bottom end of the price range, say about £300 or so, are Celerons and Pentium with 2GB of RAM. At the other end are 4k video editing machines for photographers. These have GeForce 1040 or above video cards, 8GB+ of RAM, an i7 HQ, some SSD and a 1TB drive or a lot more SSD, and USB 3. / 3.1 to connect external HDD’s with decent transfer rates. MacBook Pros and Dell XPS 15’s are get a lot of mentions. (High-end gaming laptops are way over what I need.) These machines have good keyboards and aluminium unibodies (the Dell is almost an aluminium unibody).
In between is a mass of i5 / i7 U-series machines, with or without graphics cards, random amounts of SSD, rarely an HDD, random combinations of USB / HDMI / VGA ports and build quality that ranges from awful to okay-I-guess. These can vary in price between about £600 - £1,000. The only way to make sense of these component salads is to assume that Wintel manufacturers design a high-end model, a low-end model and a mid-range model, and all the others are put together from excess parts and left-overs.
The decisions are much simpler than all those fancy specs and combinations make it look.
Want to browse the net, do text-based work, basic photographic editing and adjustment, and send e-mails? But don’t do 1080p and upwards video-editing or scientific computing? Take a serious look at an iPad and an external keyboard.
Get a Macbook Pro if you want to do iOS, Mac or UNIX / Linux development.
Get the high-end £1,100+ video-editing capable machines if you want: the aluminium body, high-quality keyboard and sharp screen; a large HDD; to use Adobe Creative Cloud or the equivalents on 1080p and 4k movies.
Want to do lots of calculations but not much graphics? (Very rare). Then you can get one of the component-salads with an i7-HQ, 256GB of SSD and integrated graphics.
Sustained daily typing and use - because you’re an author, or journalist or other content-producer? Consider a Mac Air or one of the top-end machines. (The price difference is not that large, especially if it’s how you make your money.)
Anything else? Get the mid-range Wintel.
It’s the build quality. Once you’ve had Mac, you can’t go back. At work, they hand out POS HP’s with Win 7, VGA adapters, and a keyboard with the Page Up / Down / End / Home keys in the wrong place. But that’s institutional companies for you. Cheap. The Adobe Creative Cloud Suite user in the family does so on an MSI gaming machine with an i7 HQ and 16GB of RAM with a 17-inch screen. It’s wicked fast but it’s got that corporate cheap build feel. And he doesn’t work for a big company.
I don’t want to come home and use something similar to the junk they give us at work. It’s my home, not an office. Computers are one of the things I’m willing to Pay Good Money for. (Not Silly Money, but Good Money.) As a tool to do a job. And the video-editing performance laptop is a tool for a job I’m not going to do.
The outsider for my needs is an Asus mini-PC, which has the 1TB drive, an i5-7U series, a mid-range graphics card, some SSD, Wireless-AC, and ports out the wazoo. It can drive two external screens, which is a nice-to-have I’ve wanted for a while. It will be an ace media centre, but would need hi-fi to make nice sound, but then, so does a laptop. It costs £650 and already I have an external screen (the TV), a mouse and keyboard. At a pinch I could get a 21-inch monitor for about £150 and work on a table if I wanted to use it as a computer. Back to the future.
PS: I didn’t do any of this. I did something else instead that didn’t involve spending money. I’ll talk about that later.
Thursday 22 March 2018
Toy Examples: How os.walk Works
One of the many things a programmer needs to do is walk a directory tree and do stuff to the files and folders in it, even if it’s just list them. This is a recursive exercise and those can be mind-bending to code, and if done badly can mess up all sorts of low-level things. It’s best left to the kind of people who have actually read Knuth. Fortunately someone on the Python project did, and they gave us os.walk. (It’s in the os module, and is called walk().)
Unfortunately, the almost identical explanations of how to use os.walk mostly miss the point. All of them - that I’ve found - print out a directory and file listing. Which is not what I wanted to do with it.
When you execute os.walk(starting_point), for a directory called ‘starting_point’, it returns a triple consisting of: the path for starting_point a list of the subdirectories of starting_point a list of the files in starting_point.
os.walk works in a loop. Outside a loop, it doesn’t do much. Here’s how to use it in Python:
for current_directory, subdirectories, files in os.walk(starting_point): (do stuff)
What happens? The first time execution hits os.walk, it returns a triple like this:
current_directory = starting_point
subdirectories .... of starting_point
files ... in starting point
If you want to do something to all the files in the starting_point directory, you loop like this
for file in files: do stuff to file
If you want to do something to the subdirectories, unless it’s to list them, don’t. Wait for a moment, because...
The second time execution hits os.walk, it steps one directory down the tree, like this:
current_directory = first_subdirectory_in_starting_point
subdirectories .... of first_subdirectory_in_starting_point
files ... in first_subdirectory_in_starting_point
Now you can ‘do stuff to file’ for the files in first_subdirectory_in_starting_point.
What happens if there’s a subdirectory in first_subdirectory? The next time os.walk is executed it will return
current_directory = first_subdirectory_of_first_subdirectory_in_starting_point
subdirectories .... of first_subdirectory_of_first_subdirectory_in_starting_point
files ... in first_subdirectory_of_first_subdirectory_in_starting_point
Why don’t you do anything to the directories? Because os.walk is using that list to walk through them, so if you change names or permissions or something, before you have walked to the directory, os.walk (probably) won’t work.
If you want to mess with the subdirectories themselves, the chances are you need to run os_walk in the reverse direction (look that up).
The toy example of a directory listing just doesn’t expose the inner workings clearly enough. It can leave you thinking you have to do stuff with the directories as well as the files, but you don’t, of course.
Someone who has worked a lot with recursive Python functions will, should they have got all that experience before needing os.walk, grok os.walk fairly quickly. They will read the description and look at the examples and match that against the way they know Python array-returning programs have worked in the past, and say after a moment ‘Oh, sure, it does this and that, and you always have to use it in a loop’.
Catch is, walking a directory tree is one of the first things a programmer wants to do. And grokking recursive-return functions like os.walk is not simple. Being able to picture a recursive process is one of those big-jump differences between code-bashers and actual programmers. A toy example isn’t going to cut it.
Unfortunately, the almost identical explanations of how to use os.walk mostly miss the point. All of them - that I’ve found - print out a directory and file listing. Which is not what I wanted to do with it.
When you execute os.walk(starting_point), for a directory called ‘starting_point’, it returns a triple consisting of: the path for starting_point a list of the subdirectories of starting_point a list of the files in starting_point.
os.walk works in a loop. Outside a loop, it doesn’t do much. Here’s how to use it in Python:
for current_directory, subdirectories, files in os.walk(starting_point): (do stuff)
What happens? The first time execution hits os.walk, it returns a triple like this:
current_directory = starting_point
subdirectories .... of starting_point
files ... in starting point
If you want to do something to all the files in the starting_point directory, you loop like this
for file in files: do stuff to file
If you want to do something to the subdirectories, unless it’s to list them, don’t. Wait for a moment, because...
The second time execution hits os.walk, it steps one directory down the tree, like this:
current_directory = first_subdirectory_in_starting_point
subdirectories .... of first_subdirectory_in_starting_point
files ... in first_subdirectory_in_starting_point
Now you can ‘do stuff to file’ for the files in first_subdirectory_in_starting_point.
What happens if there’s a subdirectory in first_subdirectory? The next time os.walk is executed it will return
current_directory = first_subdirectory_of_first_subdirectory_in_starting_point
subdirectories .... of first_subdirectory_of_first_subdirectory_in_starting_point
files ... in first_subdirectory_of_first_subdirectory_in_starting_point
Why don’t you do anything to the directories? Because os.walk is using that list to walk through them, so if you change names or permissions or something, before you have walked to the directory, os.walk (probably) won’t work.
If you want to mess with the subdirectories themselves, the chances are you need to run os_walk in the reverse direction (look that up).
The toy example of a directory listing just doesn’t expose the inner workings clearly enough. It can leave you thinking you have to do stuff with the directories as well as the files, but you don’t, of course.
Someone who has worked a lot with recursive Python functions will, should they have got all that experience before needing os.walk, grok os.walk fairly quickly. They will read the description and look at the examples and match that against the way they know Python array-returning programs have worked in the past, and say after a moment ‘Oh, sure, it does this and that, and you always have to use it in a loop’.
Catch is, walking a directory tree is one of the first things a programmer wants to do. And grokking recursive-return functions like os.walk is not simple. Being able to picture a recursive process is one of those big-jump differences between code-bashers and actual programmers. A toy example isn’t going to cut it.
Monday 5 March 2018
Why You Should Replace Your ISP's Modem-Router
Last year I had a long-running problem with Talk-Talk, who could not find out why my broadband speeds had been so slow. They were in fact in breach of the contract, that’s how slow it was, but I wasn’t hammering the bandwidth at the time, so didn’t notice. I had a Bright Sparks engineer visit, I had BT replace the copper into the house from the distribution pole. Still nothing. So I retired the modem they had sent me a few years ago, and used the modem in the Netgear D6400 I was using as a router. It has wireless-AC and Talk-Talk’s router only had wireless-N.
Hey presto! Instant speed and reliability increase. The problem must have been with the Talk-Talk router: one of those faults that hides in the shadows when an engineer pokes around with a stick. That lasted for about 120 days, and then the connection started to go. I could watch the Netgear sign on, and see the downstream signal/noise sink down, go below zero and drop the connection. By now, I had learned not to call the helplines, who start talking about freaking micro-filters, and diagnosed a faulty modem chip: the rest of the router was just fine.
So I swapped in the Huawei modem-router Talk-Talk had finally managed to send me while they were not finding out what was wrong with my connection. I turned off the wireless and hooked it into the Netgear. Line reliability restored, with a speed of 4600 mbps. Um yeah. Let’s see if it improves after a week or so.
Nope.
So after having read many reviews about the Billion routers, I bought an 8800NL R2. £55 plus change from Amazon, with delivery via Doodle. It needs to be connected to a computer via LAN cable to be configured at start-up, but otherwise, plug it into the ADSL socket, hook it up to the Netgear. Take a look at the line speed.
8190 mbps.
Out of the box.
Boom. For real.
It’s currently at 9120 mbs. Never had that before.
This proves what I’ve often suspected. It’s in the ISP’s interests to send you something half-way reliable with speeds towards the lower end of what they promise. That way it naturally restricts the amount of their network bandwidth you can take. If it drops out fairly frequently, that’s a plus as well because they can always restrict your line speed even more while they "test for quality of connection". Oddly, you’ll never go above about half of their quoted maximum.
There are reasons Talk-Talk is cheap. A call centre in Guam is one of them. Second-rate Huawei modem-routers is another.
You can’t do anything about the call centre. You can buy a decent modem.
(Caveat: the Billion has wireless-N. That didn’t matter because I use the Netgear as a router. If you need wireless-AC, you will need a different model.)
Hey presto! Instant speed and reliability increase. The problem must have been with the Talk-Talk router: one of those faults that hides in the shadows when an engineer pokes around with a stick. That lasted for about 120 days, and then the connection started to go. I could watch the Netgear sign on, and see the downstream signal/noise sink down, go below zero and drop the connection. By now, I had learned not to call the helplines, who start talking about freaking micro-filters, and diagnosed a faulty modem chip: the rest of the router was just fine.
So I swapped in the Huawei modem-router Talk-Talk had finally managed to send me while they were not finding out what was wrong with my connection. I turned off the wireless and hooked it into the Netgear. Line reliability restored, with a speed of 4600 mbps. Um yeah. Let’s see if it improves after a week or so.
Nope.
So after having read many reviews about the Billion routers, I bought an 8800NL R2. £55 plus change from Amazon, with delivery via Doodle. It needs to be connected to a computer via LAN cable to be configured at start-up, but otherwise, plug it into the ADSL socket, hook it up to the Netgear. Take a look at the line speed.
8190 mbps.
Out of the box.
Boom. For real.
It’s currently at 9120 mbs. Never had that before.
This proves what I’ve often suspected. It’s in the ISP’s interests to send you something half-way reliable with speeds towards the lower end of what they promise. That way it naturally restricts the amount of their network bandwidth you can take. If it drops out fairly frequently, that’s a plus as well because they can always restrict your line speed even more while they "test for quality of connection". Oddly, you’ll never go above about half of their quoted maximum.
There are reasons Talk-Talk is cheap. A call centre in Guam is one of them. Second-rate Huawei modem-routers is another.
You can’t do anything about the call centre. You can buy a decent modem.
(Caveat: the Billion has wireless-N. That didn’t matter because I use the Netgear as a router. If you need wireless-AC, you will need a different model.)
Monday 24 July 2017
Revising My Computer Security
A few weeks ago one of our in-house digital security people gave a presentation. He was not singing the usual tunes and had some interesting things to say, so I looked again at my security and privacy arrangements.
The public discussion about privacy is about keeping the prying eyes of the government and advertisers away from what you're up to. That's because no-one wants to say that the privacy you really need is from your wife, children, extended family, friends, and housemates. That doesn't sound sharey-carey-trusting-loving, but until the day the last person who likes to embarrass their mates is swinging from a tree, we're going to need that privacy.
I'm a single-occupancy household, so I don't need to lock my computers against my fellow trusted dwellers. On the other hand, I take two devices, the phone and the iPad, out with me most days, so those should have security enabled.
Also, I should do my bit to maintain herd immunity. Herd immunity happens when a high enough proportion of a group of animals has immunity from a disease that it can't spread. Maintaining herd immunity is why mothers who refuse to get MMR jabs for their darling ones are not exercising personal choice, but being irresponsible. If the word goes round the amateur villain chat boards that they have to steal twenty phones to find one that has no security and can be exploited, they will decide the odds aren't worth it.
For a long time I didn't do my part. There was nothing on any device I took out of the house that could be used to steal from me. Then along came PayPal, banking apps, Apple Pay and password managers.
I lock my work laptop every time I step away from my desk, and that's in a reasonably secure corporate environment. However, that's what my employer insists I do, and there are folk whose job it is to wander around spotting unlocked, unattended computers: it's part of my job, and I'm being paid to do it.
But then, I don't mind being locked out of my work thoughts. I do mind about being locked out of my personal thoughts. If that makes sense.
Anyway, as a result of the guru's advice, I made a few changes.
Apparently, advertisers put all sorts of tracking gizmos and other crapware on our machines. Some of it for people who have postcodes in Kaliningrad. I want to avoid that, so I put Adblock Plus on both my iOS devices, which improved the browsing experience as well. I have it on all my laptop browsers already.
I put my serious passwords into LastPass and have that on the devices I use to run my life. Caveat: LastPass doesn't sign you out after N minutes of inactivity. Signing out is manual. This is a mistake on their part. If you don't sign out, anyone who can get into your phone has access to the password manager that's still open because you forgot to sign out. As soon as you put a password manager on a device, you must activate the physical access security on that device. And sign out of the password manager anyway.
So I trained my phone to recognise my thumbprint, giving me a HTF (Do They Do That) moment. Folklore says it can tell if the Mafia cut off your finger and are using that. I'd like to know how that's done.
The guru has F-Secure on his phone. I met Mikko Hypponen, on a flight to Helsinki back in the day. He's a great ambassador for his company, but I still don't like active scanners. I use the default Windows Defender and the default Windows or OS X firewalls. I don't run McAffee, Norton or F-Secure. On iOS there's no point because of the way iOS sandboxes apps. On Windows or OS X, scanners are an operational overhead with little benefit. I read somewhere that the pros don't do use any. Instead they practice safe computing:
Just because I've cleared the cache or deleted the file, doesn't mean it's gone. Deleting is one thing, shredding is another. Here's the thing: file shredding and free / slack space wiping works on conventional hard drives (HDDs) but is iffy, if not discouraged on modern SSDs. It's not even clear what 'secure delete' in means on an SSD. There are encrypted drives that use a key which gets wiped, and unless the NSA or the Chinese are after you, guessing at the key is going to be computationally unfeasible. Most SSDs are not encrypted.
If you want to store large amounts of personal or private data, do it on a conventional hard drive. The you can shred-and-wipe, and it's gone. As soon as an SSD gets involved, you can't be sure the data won't still be there.
On Windows I use CC Cleaner to shred files in the Wastebin after deletion. Every now and then, I over-write the spare space on the drive as well. A three-pass wipe will do fine. The disk recovery people can work wonders with a physically damaged drive. The stuff they have works at bit-level. If you have, however, written random bits all over the drive, all they will get back are random bits. And no, on a modern 2.5-inch multi-gigabyte drive, all those tricks invented in the 1980's don't work.
My work laptop encrypts my Documents folder, but leaves the rest alone, which is sensible. On my personal computers, I'm not so sure. I might forget the password.
Encrypted files on personal computers are a red rag to anyone who wants to pick a fight. Encrypted files will be assumed to be the worst thing the person finding them wants them to be. Why else you you encrypt the stuff if it wasn't stolen company data / classified government documents / illegally-downloaded movies / whatever. Anyway, unless you are a journalist, very rich or have high-profile lawyers, you can be compelled to de-crypt it all by US Immigration, the Police, an Anton Pillar order, your wife, anyone with a gun... you get the idea.
(It occurs to me that the most secure personal laptop is one of those Lenovos or Dells that only corporates buy, dressed up with a corporate logon and two layers of passwords. Create at least two other user profiles and fill them with encrypted junk, suggesting that you are the third person to be using this computer. Make sure none of the software is within two releases of the latest version. Put on an old VT terminal emulator, McAffee, and make IE9 the default browser. Add a sticky label declaring that the Asset ID is BG788453TD, remove one of the keys (say Z) on the keyboard, and everybody will assume you work for a large financial services company and this is your work computer.)
While we're talking about encryptions, the guru suggested using Signal to communicate, or WhatsApp, which uses the Signal protocols. Use any end-to-end encrypted communication, as long as it is well-known. The quickest way to get GCHQ interested in you is to use fancy e-mail encryption, or a program known only to people who have attended Black Hat more than twice. I have WhatsApp, use the regular message app on the iPhone, and have a totally boring life.
All this stuff is free, by the way.
The public discussion about privacy is about keeping the prying eyes of the government and advertisers away from what you're up to. That's because no-one wants to say that the privacy you really need is from your wife, children, extended family, friends, and housemates. That doesn't sound sharey-carey-trusting-loving, but until the day the last person who likes to embarrass their mates is swinging from a tree, we're going to need that privacy.
I'm a single-occupancy household, so I don't need to lock my computers against my fellow trusted dwellers. On the other hand, I take two devices, the phone and the iPad, out with me most days, so those should have security enabled.
Also, I should do my bit to maintain herd immunity. Herd immunity happens when a high enough proportion of a group of animals has immunity from a disease that it can't spread. Maintaining herd immunity is why mothers who refuse to get MMR jabs for their darling ones are not exercising personal choice, but being irresponsible. If the word goes round the amateur villain chat boards that they have to steal twenty phones to find one that has no security and can be exploited, they will decide the odds aren't worth it.
For a long time I didn't do my part. There was nothing on any device I took out of the house that could be used to steal from me. Then along came PayPal, banking apps, Apple Pay and password managers.
I lock my work laptop every time I step away from my desk, and that's in a reasonably secure corporate environment. However, that's what my employer insists I do, and there are folk whose job it is to wander around spotting unlocked, unattended computers: it's part of my job, and I'm being paid to do it.
But then, I don't mind being locked out of my work thoughts. I do mind about being locked out of my personal thoughts. If that makes sense.
Anyway, as a result of the guru's advice, I made a few changes.
Apparently, advertisers put all sorts of tracking gizmos and other crapware on our machines. Some of it for people who have postcodes in Kaliningrad. I want to avoid that, so I put Adblock Plus on both my iOS devices, which improved the browsing experience as well. I have it on all my laptop browsers already.
I put my serious passwords into LastPass and have that on the devices I use to run my life. Caveat: LastPass doesn't sign you out after N minutes of inactivity. Signing out is manual. This is a mistake on their part. If you don't sign out, anyone who can get into your phone has access to the password manager that's still open because you forgot to sign out. As soon as you put a password manager on a device, you must activate the physical access security on that device. And sign out of the password manager anyway.
So I trained my phone to recognise my thumbprint, giving me a HTF (Do They Do That) moment. Folklore says it can tell if the Mafia cut off your finger and are using that. I'd like to know how that's done.
The guru has F-Secure on his phone. I met Mikko Hypponen, on a flight to Helsinki back in the day. He's a great ambassador for his company, but I still don't like active scanners. I use the default Windows Defender and the default Windows or OS X firewalls. I don't run McAffee, Norton or F-Secure. On iOS there's no point because of the way iOS sandboxes apps. On Windows or OS X, scanners are an operational overhead with little benefit. I read somewhere that the pros don't do use any. Instead they practice safe computing:
Don't visit dodgy websites, ignore any website that tells you your computer has viruses or your files are corrupt, and anybody who wants your passwords. Don't open e-mails from people or companies you don't know, and only download from the original supplier. Here I will tell you nothing have to do on sites which English not best used.I clean out browsing history, caches and other stuff with CC Cleaner on Windows, and Clean My Mac for OS X. Cache cleaners for iOS are still lacking in functionality.
Just because I've cleared the cache or deleted the file, doesn't mean it's gone. Deleting is one thing, shredding is another. Here's the thing: file shredding and free / slack space wiping works on conventional hard drives (HDDs) but is iffy, if not discouraged on modern SSDs. It's not even clear what 'secure delete' in means on an SSD. There are encrypted drives that use a key which gets wiped, and unless the NSA or the Chinese are after you, guessing at the key is going to be computationally unfeasible. Most SSDs are not encrypted.
If you want to store large amounts of personal or private data, do it on a conventional hard drive. The you can shred-and-wipe, and it's gone. As soon as an SSD gets involved, you can't be sure the data won't still be there.
On Windows I use CC Cleaner to shred files in the Wastebin after deletion. Every now and then, I over-write the spare space on the drive as well. A three-pass wipe will do fine. The disk recovery people can work wonders with a physically damaged drive. The stuff they have works at bit-level. If you have, however, written random bits all over the drive, all they will get back are random bits. And no, on a modern 2.5-inch multi-gigabyte drive, all those tricks invented in the 1980's don't work.
My work laptop encrypts my Documents folder, but leaves the rest alone, which is sensible. On my personal computers, I'm not so sure. I might forget the password.
Encrypted files on personal computers are a red rag to anyone who wants to pick a fight. Encrypted files will be assumed to be the worst thing the person finding them wants them to be. Why else you you encrypt the stuff if it wasn't stolen company data / classified government documents / illegally-downloaded movies / whatever. Anyway, unless you are a journalist, very rich or have high-profile lawyers, you can be compelled to de-crypt it all by US Immigration, the Police, an Anton Pillar order, your wife, anyone with a gun... you get the idea.
(It occurs to me that the most secure personal laptop is one of those Lenovos or Dells that only corporates buy, dressed up with a corporate logon and two layers of passwords. Create at least two other user profiles and fill them with encrypted junk, suggesting that you are the third person to be using this computer. Make sure none of the software is within two releases of the latest version. Put on an old VT terminal emulator, McAffee, and make IE9 the default browser. Add a sticky label declaring that the Asset ID is BG788453TD, remove one of the keys (say Z) on the keyboard, and everybody will assume you work for a large financial services company and this is your work computer.)
While we're talking about encryptions, the guru suggested using Signal to communicate, or WhatsApp, which uses the Signal protocols. Use any end-to-end encrypted communication, as long as it is well-known. The quickest way to get GCHQ interested in you is to use fancy e-mail encryption, or a program known only to people who have attended Black Hat more than twice. I have WhatsApp, use the regular message app on the iPhone, and have a totally boring life.
All this stuff is free, by the way.
Subscribe to:
Posts (Atom)