Why is that surfcam offline?
C'mon Ben, banks all-time at caba ATM? ;)
Ha!
Actually, just got a call from Telstra then (8pm!). Told me to ignore the last message, they can’t get there until Tuesday April 2nd.
Eh? Message received last night: now Caba is due to be fixed by 7pm Wednesday. What's another day between friends?
And.. Cabarita is back.
Manly surfcam has been offline all day today - according to our network provider, "an NBN fibre cable has been damaged by third party boring near the Chatswood exchange. We are unable to provide us with an estimated time of resolution at this stage."
Manly update: "The supplier has confirmed that the new cable has arrived at the site of the damage. The supplier may need traffic management engagement since the site is located on a major road. The supplier is unable to provide us with an estimated time of resolution at this stage."
Manly is back!
Argh! Cabarita is offline again.
As you can see from our monitoring trace (that's a two-hour snapshop below, from about 1:10am last night), everything was fine until about 1:36am, when it went offline for eight minutes. It then came back/dropped out another ten times until 3:06am, which is the last we've heard of it.
I suspect it's the same issue as before - Telstra related (and not our equipment) - but unfortunately the process to ascertain the problem is arduous (see above!). I'll try my best to get it back up and running ASAP.
And.. Caba is back again (for now!). Must have been a Telstra exchange issue.. though I haven't yet had time to lodge a fault.
So, I'm expecting it to go down again.
Standby in 3, 2, 1....
Caba is offline again. Let’s wait and see if it comes back before dawn.
And Caba is back.
Hey Ben. Any word on the Bondi cam? It's the only south-facing cam between Cronulla and Avoca and a useful indicator for a number of breaks. Been down for yonks.
Yep, it's taken way longer than expected but we've got verbal agreement for a new location, so we're just finalising the paperwork before we hook up the data lines. It's a massive improvement from our old Bondi surfcam too.
good news - thanks mate
Caba went offline at 5:12am.
Here we go again!
And... Caba is back.
We're still experiencing occasional surfcam outages at Cabarita, all of which are Telstra related and therefore out of our control.
Specifically, we seem to lose connectivity most mornings for a couple of hours, and then the service returns and runs fine for the rest of the day. I've got no idea what it is but I'll continue to follow up with Telstra to see if they can pinpoint the problem.
As for NBN - the estimated date for the entire suburb to upgrade keeps getting pushed back - the NBN website now says that it's "Planned to be available from Jul-Sep 2019". However, it was meant to be live 18 months ago, so there's no way to know if this date will be pushed back again once we reach July.
Thanks for your patience, and apologies for the inconvenience.
Hmmm, the surfcam came back just a few minutes after I posted above.
Very odd.
Speeds are very low too, so surfcam stream quality isn't great, but I'll give the equipment a reboot to see if that helps.
OK, that's a little better. Fingers crossed it remains stable.
Surprised to see Caba still online this morning (it's gone down just before dawn, every morning for the last week), but it's barely hanging in there - internet speeds are terrible.
Thought I'd share a graph showing how the streaming speeds were fine until about 5:25am, when they started to drop (this is normally about the time it switches off completely).
I managed to log into the router - only just - to reboot it (took three minutes to authenticate!) but it hasn't fixed the problem. Therefore the vision is slow and jerky this morning.
As it does every day, speeds will probably improve around 8-9am onwards. Again, this is a Telstra problem - the Cabarita region is about to roll over to NBN soon, so I get the feeling it'll be hard to get Telstra to fix their ageing ADSL infrastructure in a timely manner.
Bugger. Now it's gone completely offline.
We're back up to an acceptable surfcam stream at Caba now. As per the graph below, internet speeds came back up on their own accord around 6:50am (after being offline since about 6am).
Looks to me that it could be local network congestion - everyone in the suburb waking up around that time and using the internet for a few minutes each, but bringing everyone's speeds to a crawl because of the lack of bandwidth to the exchange. The stream is otherwise fine throughout the day because everyone is off at work. And by the time everyone gets home, the sun's down so the surfcam isn't being watched.
Hey guys just putting an idea out there, what about users potentially gaining access to surf cam footage of a particular day and timeframe?
We already offer access to replays for some surfcams (all remaining cams to come online soon) but we only hold five days of footage (because storage costs are considerable).
For example, Burleigh this AM: https://www.swellnet.com/surfcams/burleigh-heads/replays#/2019-05-08/444420
Was there something else you had in mind?
Caba's offline again this morning.
Though curiously, it went down in the early hours of the morning (speeds gradually slowed from about 1:20am) - which goes against the theory that it's related to congestion around sunrise when everyone wakes up and logs into Facebook and Netflix. Graph below is a 2-hour snapshot of speeds between 11:50pm and 1:50am last night.
I've already remotely power-cycled the equipment (and it hasn't come back up) so we'll just have to wait until Telstra's infrastructure fixes itself again (usually 7:30am onwards).
Online again, bang on 7:30 am.
Middleton cam been down since Saturday.
Haven't been able to get anyone to check it out (though have tried multiple channels).. sorry mate. Will have someone there ASAP.
any more word on the bondi cam ben?
Bondi'll be done soon as we knock off our Illawarra cams. NBN activations have been a nightmare for the last few months (including some upgrades of existing ADSL services).
No problem Ben....probably not much to see anyway lol. Just wanted to let u guys know.
Thanks MDM. We actually have a custom alert system that notifies us within seconds of there being a power outage, internet drop out etc. So we are always aware of what's going on - what we don't know is whether it's temporary, or a major issue.
Found out today there's an electrical issue at Middleton so we're gonna have to get the sparky around to fix it up.
Middleton went offline at 4:45am, Goolwa went offline five minutes ago. Both seem to be power related issues (not ISP) though SA Power Networks are not reporting any local problems (yet).
Middleton back, waiting for Goolwa. Still nothing on SA Power Networks website.
Goolwa is now back online.
Any chance the middleton cam can be pointed towards the point instead of the bay? There's a lot of days where the bay is flat but its surfable off the point.
Margaret river cam has not been working for me for a few weeks now. Is it just my end or what? Had a just a grey screen for a while now I get a still image but vid wont play.
Someone else had the same problem recently, turned it it was a slow internet connection at their end. Can you check your speed via Speedtest.net and let me know what you’re getting.
Portsea is back!
Excellent now you can watch close outs in real time again!!
Sorry bone
Nice clear picture though
Finally found out why Lorne is offline... the surf club undergoing some major renovations. Should be back late Oct or early Nov.
Both Thirroul cams are back.
Wollongong is back!
Argh! Just spent a couple of weeks redoing the Byron surfcam(s) - had to fabricate a new mounting pole, because the house was renovated with a brand new roof - finally got everything back up and running, and yesterday's thunderstorms across the Northern Rivers seems to have knocked out some of the equipment. Not sure what the damage is (whether it's our gear, or the ISP's network) but we'll look into as a priority.
Here's a WTF moment - during yesterday's thunderstorms, one of our surfcam modems was bricked. Normally, they just stop working. Occasionally I've seen the data port fried. But... check out the plastic stuff that came out of the modem following yesterday's lightning strike! Never seen this before.
Couple of weeks ago a friends neighbour's rendered block wall was hit by lightning, showering the area with fence fragments up to 30 m away.
Both homes lost many electrical appliances.
Crazy.
Hey Ben, spewing about the Byron cams causing you headaches. I’ve got experience with POE cams (electrician) let me know if I can help getting them going for you.
Thanks mate - issue with Byron is the NBN connection. They haven't been able to fix it yet. Insanely frustrating. I've been down there maybe five times in the last 2 weeks to "try another modem" etc.. useless!
There is an intermittent fault at the Byron fibre exchange causing signal to drop in and out rapidly, multiple faults per second which won’t get picked up using your average ping test. They have been trying to fix it for months. not sure if NBN have let that slip to you. I’ve got clients tearing their hair out over it. It only affects uploads like streaming etc, download speeds have been ok. Im just around the corner from the cams if you need a hand
I installed Swellnet’s first surfcam sometime in 2005. Since then - more than thirteen years! - technology has advanced rapidly, and the quality and reliability of our surfcam streams has improved considerably.
However, the last few weeks have seen quite a few surfcams go offline for a wide variety of reasons - most of which are completely outside of of our control. So, whilst we work our hardest to get them back up and running, I thought it was worth explaining a few of the problems that can go wrong, and why it can take quite some time to rectify these kinds of issues.
Our surfcams are an automated network. That is, once the cameras are installed, in a perfect world they never have be attended to again unless they go offline. Of course, being right on the beach in highly corrosive marine environments, facing the brunt of every single weather system that sweeps the coast, this means that our hardware is frequently put to the test.
But, by and large, our surfcams will often run for many years without any problems. We only notice when they’re down.
So I thought it'd be interesting to share our troubleshooting regime for when our surfcams do go offline, and an example of the check list we might run through.
Is it a local power issue?
Whilst each surfcam has an on-site uninterruptible power supply (UPS), it’s only small and offers just twenty minutes of backup power. They also protect our equipement from brown-outs, surges, spikes etc - though not all the time - I've seen crazy instances where a surge has gone through one device without affecting it, but then destroyed something on the other side (for example, killing a surfcam but not the power supply that the surge had to pass through to reach the surfcam).
Anyway, the UPS is fine for most minor glitches in the electrical grid network. However, if the power is down across the suburb, then the surfcam will also be offline. Most power companies share this information online, but sometimes it can be slow to be made publicly available.
Is it a local electrical issue at the property?
This happens less frequently, but is the source of our current surfcam outage at 13th Beach. Let me explain in more detail…
After many years of uninterrupted service, our 13th Beach surfcam went offline a few weeks ago. Access to the surfcam is difficult (it’s at the Surf Club, and we’re out of patrolling season), but finally after our local sparky got into the building, we found that the surfcam shared an electrical circuit with some SLSC equipment, which had shorted.
The problem was temporarily fixed, though installing a dedicated circuit was recommended - but couldn’t be done there and then (as parts needed to be sourced, and an installation time booked). Because we’re upgrading that site to NBN in the coming months, we thought we’d do it all in the one go.
But of course, it didn’t last that long… it appears that the circuit has shorted again, just a few days ago (it'll be fixed ASAP).
Is the problem related to our Internet Service Provider?
This issue is even more complex than you can imagine, especially with the gradual switch from ADSL to NBN around the country (around 85% of our network is now on NBN).
Aside from the physical points of failure - all of which are out of our control (the first connection point at the property, the pit in the street, the pillar down the road, the local exchange and the network data centre), sometimes we get thrown a crazy curveball. Allow me to digress…
We’ve had a surfcam at Lorne for many, many years, inside the observation tower at the Surf Lifesaving Club. Initially on an ADSL connection, it was upgraded to ADSL2 a while back, and then last year, finally upgraded to NBN. All was working fine.
However, a couple of weeks ago our ISP advised that they’d mistakenly received notification that the service was to be ‘churned’ to another ISP - something we hadn’t instigated. So, my ISP stopped the process (at a cost to them), and we tried to figure out what went on.
Another week later, the same ‘churn’ request came through again, and my ISP went through the motions to stop it happening (incurring another cost).
We were finally advised that - despite our internet connection being in place, and physically unchanged for well over a decade - it was actually using the ‘wrong’ connection somewhere along the network path (external from the Surf Club, on the street), and was mapped to someone else’s property up the road. The owners of this unknown property were now applying for their own NBN connection, and hence the problem - Swellnet's new problem - was uncovered.
For some unknown reason, NBN were unable to re-route our existing connection, so they simply disconnected Swellnet's internet service last Friday (without notification, either - we are still yet to receive confirmation). We've simply been told that the fastest way to get the Lorne surfcam back is to “re-apply for a new NBN connection”. Which not only means the surfcam will be offline for a few weeks, but now requires two specialist technician visits - a qualified sparky (ACMA approved with open/restricted cabling license) to switch the internal cable at the Surf Club to an unused line, and another technician to reconfigure our modem with the new log in details. And then we’ll need to make some changes within our surfcam streaming infrastructure to get the vision live on the website again.
Fun and games, eh? Add up all of the associated time and cost to fix this problem, including replying to disgruntled Swellnet subscribers who can't view the Lorne surfcam, and it's a royal pain the arse.
This is just one of many examples of what goes on behind the scenes on a weekly basis, just to keep the surfcam internet connections up and running. I'm gonna write a book one day.
Anyway, back to troubleshooting.
Is the problem related to our surfcam hardware?
Each surfcam location has several pieces of equipment, and although we utilise industrial/business-grade hardware, every device still has the capacity to freeze, or break down at some point. Usually when the swell of the year is bearing down on the coast.
Some of our equipment has lasted five or six years before needing to be replaced. On rare occasions, certain pieces of hardware have been DOA (dead on arrival) which is a pain in the arse when you've freighted all of the equipment down to some remote location for a new install, teed up an electrician to install it all, and then the bloody camera needs to be returned to the supplier under warranty for a replacement (resulting in another costly technician visit and a couple of weeks delay on the install).
Have you ever had your ISP ask you to “turn the modem off and on again”? Well, it’s not as stupid as it sounds - that simple procedure often fixes most problems. But, back in 2006 it only took one four-hour drive from Sydney to Ulladulla to press the ‘restart’ button on our old Mollymook surfcam (remember that?), for me to realise that there had to be a better way to remotely power cycle my remote surfcams. So, ever since then we’ve installed expensive remote rebooting equipment at each surfcam that allows us to turn all of the devices off and on again, without needing anyone to visit.
Another example: two weeks ago, our Newcastle surfcam went down.
It took almost a week to arrange a visit (cam went down on a Friday night, so couldn't arrange anything until the Monday; bloke got there Friday). All equipment in the cabinet was working fine, but the technician reported the power source to the surfcam was dead as the power lights were completely off. Now, this was unusual - the power supply rarely dies outright, more often, when there's a fault it'll change the colour of the indicator LEDs from green to orange. So, I asked the technician to look a little more closely. Lo and behold, the power cord into the back of the power supply had come loose, and it just needed to be plugged back in again and the surfcam came instantaneously back. Lucky I didn't arrange for spare parts to be sent down, as that would have delayed its return by up to another week.
But sometimes it's not that easy to source the problem. It could be one or more of the following issues:
Ordinarily, we need someone to visit the surfcam site to check everything out and determine what the problem might be. We have a lot of spare equipment on standby in the Swellnet office, so once we know what the problem is, we can usually replace broken hardware relatively quickly - assuming our local technician is available, and can gain access to the property - I am not kidding when I say that there’s been more than a couple of occasions when a surfcam outage has happened the day after the person with the keys to the property goes overseas on holidays for a couple of weeks.
So, in order to cover all bases, when a surfcam goes offline and we suspect it’s hardware related (i.e. we're confident it's not an electrical problem, nor an ISP/network issue), we often freight a single Pelican case with one replacement of everything (camera, midspan, router, cleaning hardware, rebooter, UPS) to our local technician. This ultimately saves a second technician visit, which could take another week or two once availability and freight times are factored in. However the actual surfcams themselves are very expensive, and we use a couple of different models that each have different mounting brackets, so we can't keep spares of everything - which means purchasing new equipment, some of which is only available from overseas suppliers, which therefore takes time to source and freight. It's not like a supermarket where you can walk in and buy stuff off the shelf - some of our equipment is specialised and may not be available to purchase for periods of time, if the manufacturer runs out of stock.
Anyway, that's just a basic overview of some of the issues that affect the uptime of a surfcam. We're always developing new tools and techniques to improve the reliability, so I hope this answers some of the question you may have been pondering.