Canada Was Offline Today

Today Roger's Telecom of Canada was down again with an 18 hour outage. In addition to Internet and mobile phones, a huge number of services were affected such as ATMs, transportation, payment terminals, etc. So 10 million wireless subscribers and 2 million retail subscribers were dead in the water, annoyed, and disappointed. I am surprised it took so long to find root cause of the problem and resolve it.

This episode shows how fragile these eggs-in-one-basket do-everything-for-you systems are in the modern world.

I have Internet and TV from Roger's but wireless phone from Bell. So when the Internet died, I could verify on my phone ... yup Rogers is broke again.
 
Cash was king up here yesterday.

What were they saying about a conversion to crypto currency or a cashless society? I think we need a far more robust network before that happens widespread
Credit worked everywhere I was; I assume that Rogers is the network provider for Interac here in Canada. Also apparently at some places if your Debit card was RBC it would work. That’s unsubstantiated but was what heard.
 
What's amazing is the lack of redundancy. The fact Interac doesn't have multiple network connections for this very reason is terrible.

In other news, Bell works like it's supposed to because Bell isn't incompetant (in their technical department, billing is another issue).
 
  • Like
Reactions: soundchaser
https://www.theregister.com/2022/07/08/canada_rogers_outage/
cloudflare-radar-asn-change-in-traffic-default-2022-07-09T22 32 54.116Z.png
 
What's amazing is the lack of redundancy.
Depends where you want the redundancy at telco level** that can be an expensive pain*. By what little has leaked out there was a BGP snafu which took out the whole rogers IP range. As everything is IP now basically everything in their network stopped being able to talk. Cloudflare went out a couple of weeks ago (and took this website off the air for a while) with a BGP issue. BIG difference is they fixed it quickly. And its the ability to quickly resolve that sorts the men from the boys.

*I'm sure everyone on here who's worked in or around datacentres has had to deal with $$$ redundant links where it turns out both pairs of fibre go out the building in the same conduit so one badly placed back hoe knocks both out at once

** disclaimer I do currently work with stuff that IS redundant partially at cellular level so can handle a full radio network failure. It sometimes works but still ends up backhauling out of the primary telco so would not survive this sort of outage.
 
  • Like
Reactions: kodabmx