Submitted by thermalben on Thu, 10/04/2018 - 09:11
I installed Swellnet’s first surfcam sometime in 2005. Since then - more than thirteen years! - technology has advanced rapidly, and the quality and reliability of our surfcam streams has improved considerably.
However, the last few weeks have seen quite a few surfcams go offline for a wide variety of reasons - most of which are completely outside of of our control. So, whilst we work our hardest to get them back up and running, I thought it was worth explaining a few of the problems that can go wrong, and why it can take quite some time to rectify these kinds of issues.
Our surfcams are an automated network. That is, once the cameras are installed, in a perfect world they never have be attended to again unless they go offline. Of course, being right on the beach in highly corrosive marine environments, facing the brunt of every single weather system that sweeps the coast, this means that our hardware is frequently put to the test.
But, by and large, our surfcams will often run for many years without any problems. We only notice when they’re down.
So I thought it'd be interesting to share our troubleshooting regime for when our surfcams do go offline, and an example of the check list we might run through.
Is it a local power issue?
Whilst each surfcam has an on-site uninterruptible power supply (UPS), it’s only small and offers just twenty minutes of backup power. They also protect our equipement from brown-outs, surges, spikes etc - though not all the time - I've seen crazy instances where a surge has gone through one device without affecting it, but then destroyed something on the other side (for example, killing a surfcam but not the power supply that the surge had to pass through to reach the surfcam).
Anyway, the UPS is fine for most minor glitches in the electrical grid network. However, if the power is down across the suburb, then the surfcam will also be offline. Most power companies share this information online, but sometimes it can be slow to be made publicly available.
Is it a local electrical issue at the property?
This happens less frequently, but is the source of our current surfcam outage at 13th Beach. Let me explain in more detail…
After many years of uninterrupted service, our 13th Beach surfcam went offline a few weeks ago. Access to the surfcam is difficult (it’s at the Surf Club, and we’re out of patrolling season), but finally after our local sparky got into the building, we found that the surfcam shared an electrical circuit with some SLSC equipment, which had shorted.
The problem was temporarily fixed, though installing a dedicated circuit was recommended - but couldn’t be done there and then (as parts needed to be sourced, and an installation time booked). Because we’re upgrading that site to NBN in the coming months, we thought we’d do it all in the one go.
But of course, it didn’t last that long… it appears that the circuit has shorted again, just a few days ago (it'll be fixed ASAP).
Is the problem related to our Internet Service Provider?
This issue is even more complex than you can imagine, especially with the gradual switch from ADSL to NBN around the country (around 85% of our network is now on NBN).
Aside from the physical points of failure - all of which are out of our control (the first connection point at the property, the pit in the street, the pillar down the road, the local exchange and the network data centre), sometimes we get thrown a crazy curveball. Allow me to digress…
We’ve had a surfcam at Lorne for many, many years, inside the observation tower at the Surf Lifesaving Club. Initially on an ADSL connection, it was upgraded to ADSL2 a while back, and then last year, finally upgraded to NBN. All was working fine.
However, a couple of weeks ago our ISP advised that they’d mistakenly received notification that the service was to be ‘churned’ to another ISP - something we hadn’t instigated. So, my ISP stopped the process (at a cost to them), and we tried to figure out what went on.
Another week later, the same ‘churn’ request came through again, and my ISP went through the motions to stop it happening (incurring another cost).
We were finally advised that - despite our internet connection being in place, and physically unchanged for well over a decade - it was actually using the ‘wrong’ connection somewhere along the network path (external from the Surf Club, on the street), and was mapped to someone else’s property up the road. The owners of this unknown property were now applying for their own NBN connection, and hence the problem - Swellnet's new problem - was uncovered.
For some unknown reason, NBN were unable to re-route our existing connection, so they simply disconnected Swellnet's internet service last Friday (without notification, either - we are still yet to receive confirmation). We've simply been told that the fastest way to get the Lorne surfcam back is to “re-apply for a new NBN connection”. Which not only means the surfcam will be offline for a few weeks, but now requires two specialist technician visits - a qualified sparky (ACMA approved with open/restricted cabling license) to switch the internal cable at the Surf Club to an unused line, and another technician to reconfigure our modem with the new log in details. And then we’ll need to make some changes within our surfcam streaming infrastructure to get the vision live on the website again.
Fun and games, eh? Add up all of the associated time and cost to fix this problem, including replying to disgruntled Swellnet subscribers who can't view the Lorne surfcam, and it's a royal pain the arse.
This is just one of many examples of what goes on behind the scenes on a weekly basis, just to keep the surfcam internet connections up and running. I'm gonna write a book one day.
Anyway, back to troubleshooting.
Is the problem related to our surfcam hardware?
Each surfcam location has several pieces of equipment, and although we utilise industrial/business-grade hardware, every device still has the capacity to freeze, or break down at some point. Usually when the swell of the year is bearing down on the coast.
Some of our equipment has lasted five or six years before needing to be replaced. On rare occasions, certain pieces of hardware have been DOA (dead on arrival) which is a pain in the arse when you've freighted all of the equipment down to some remote location for a new install, teed up an electrician to install it all, and then the bloody camera needs to be returned to the supplier under warranty for a replacement (resulting in another costly technician visit and a couple of weeks delay on the install).
Have you ever had your ISP ask you to “turn the modem off and on again”? Well, it’s not as stupid as it sounds - that simple procedure often fixes most problems. But, back in 2006 it only took one four-hour drive from Sydney to Ulladulla to press the ‘restart’ button on our old Mollymook surfcam (remember that?), for me to realise that there had to be a better way to remotely power cycle my remote surfcams. So, ever since then we’ve installed expensive remote rebooting equipment at each surfcam that allows us to turn all of the devices off and on again, without needing anyone to visit.
Another example: two weeks ago, our Newcastle surfcam went down.
It took almost a week to arrange a visit (cam went down on a Friday night, so couldn't arrange anything until the Monday; bloke got there Friday). All equipment in the cabinet was working fine, but the technician reported the power source to the surfcam was dead as the power lights were completely off. Now, this was unusual - the power supply rarely dies outright, more often, when there's a fault it'll change the colour of the indicator LEDs from green to orange. So, I asked the technician to look a little more closely. Lo and behold, the power cord into the back of the power supply had come loose, and it just needed to be plugged back in again and the surfcam came instantaneously back. Lucky I didn't arrange for spare parts to be sent down, as that would have delayed its return by up to another week.
But sometimes it's not that easy to source the problem. It could be one or more of the following issues:
Ordinarily, we need someone to visit the surfcam site to check everything out and determine what the problem might be. We have a lot of spare equipment on standby in the Swellnet office, so once we know what the problem is, we can usually replace broken hardware relatively quickly - assuming our local technician is available, and can gain access to the property - I am not kidding when I say that there’s been more than a couple of occasions when a surfcam outage has happened the day after the person with the keys to the property goes overseas on holidays for a couple of weeks.
So, in order to cover all bases, when a surfcam goes offline and we suspect it’s hardware related (i.e. we're confident it's not an electrical problem, nor an ISP/network issue), we often freight a single Pelican case with one replacement of everything (camera, midspan, router, cleaning hardware, rebooter, UPS) to our local technician. This ultimately saves a second technician visit, which could take another week or two once availability and freight times are factored in. However the actual surfcams themselves are very expensive, and we use a couple of different models that each have different mounting brackets, so we can't keep spares of everything - which means purchasing new equipment, some of which is only available from overseas suppliers, which therefore takes time to source and freight. It's not like a supermarket where you can walk in and buy stuff off the shelf - some of our equipment is specialised and may not be available to purchase for periods of time, if the manufacturer runs out of stock.
Anyway, that's just a basic overview of some of the issues that affect the uptime of a surfcam. We're always developing new tools and techniques to improve the reliability, so I hope this answers some of the question you may have been pondering.
Yeah we had major throughoutput issues beforehand, but the lightning strike seemed to knock everything out completely. We had intermittent line sync for a few days, then it all went blank.
NBN said they visited the site and "got line sync" - except my newly replaced modem still didn't work (after multiple power-cycling too). And they won't send someone down again until I "try (yet) another modem".
So I treked down yesterday with yet another modem, and guess what! Nothing changed.. both still can't get sync. I reckon they never visited in the first place - wouldn't be the first time this has happened either.
So now we're back on the waiting list.
Surf Cam replays don't seem to be working for Qld cams.
Not sure why they're offline.. but we're looking into it (most of our tech support contacts are away for the Xmas break.. always a bad time for things to go down).
Scarborough (Perth) is back online. Thanks for your patience everywhere.. what a mission! Massive thanks to Tully, Howard & Josh at Engie Services for their amazing assistance.
Any update with the Bondi cam? Last update was almost a year ago saying it was getting sorted soon
Unfortunately, Bondi has taken way longer than expected - for a bunch of reasons I won't go into here - but we're almost there. Getting technicians sorted now to finalise the install ('tis a tricky one, requiring custom build mounting frames).
Kings Beach NBN is slated to be activated mid-late next week. Fingers crossed (as we've lost the previous ADSL service.. came back last week for a day, then it went again).
G'day Ben, any reason the Cronulla Point cam looks like its covered in vaseline?
It's had focus issues for a while. I've got brand new cams for the Point, Island and Beaches here, just trying to schedule a time to get 'em done.
Lorne is back! And we've got a second cam too, looking at the beach. Thanks for your patience everyone.
any word on how long the narra cam will be out of action? (another surf website’s closely located cam seemed to go down at the same time).
also, any update on the bondi cam? last word on 31 may 2019 was ‘soon’
Narra is down for a few weeks while they do some maintenance on the building. Bondi was meant to be installed a few months ago (along with an upgrade of many other sites) but we've had travel/access issues. Hoping to get down there in the next month or so.
thanks for the fast response ben. especially on a sunday morning
The crowds immediately dropped here when the Narra cams on both sites went down and that was during school holidays too.
Find that very hard to believe knowing the diminutive traffic the cam saw - also the angle of the camera in relation to NN - compared to the overwhelming numbers in the water since COVID.
Might this hiatus in east coast activity be used to install the mythical Bondi cam?
Actually, Covid-19 is making our planned surfcam upgrades/installs very tricky, as my guys are unable to travel to Covid-19 "hotspots" (we're based in Northern NSW, and are doing quite a bit of work on the Gold Coast at the moment - so travelling outside of the bubble here means a 14 day isolation etc).
But, I'm working on an alternative solution.
Hi Ben. Fairhaven reportedly had whale sightings early this morning - 6.30 - 7.00 am. Checked the cam around 8.00, but it was down for a while. Any chance it was functioning early and there are recordings?
No replays for Fairhaven at the moment, sorry.
hey ben. 3 months now for narra and years for bondi. any actual time lines? (i understand covid but still ...)
re the crowds at narra, there have been two cams on that for years (3 including southy) and almost never any crowds. as soon as the banks got good earlier this year it's been crowded whenever it's working regardless of the cams
Major NBN issues at Narrabeen. Despite having an ADSL service here for more than 15 years, NBN won't recognise the location nor allow me to create a new location ID (been through this many times before, just takes a long time to sort).
Both of our Coffs surfcams have been offline since 1pm yesterday - turns out someone cut the main backhaul fibre link near Mayfield (i.e. shutting down the internet across vast areas of the Coffs region, apparently). No ETA from our wholesaler at this point in time.
Argh. Lots of network issues this morning.
Coffs - both cams came back yesterday arvo once the regional fibre service was restored, but Park Beach went back offline around 12:20am. Not sure why, we are investigating.
Also, almost all of our Gold Coast surfcam network is offline this morning.
Unfortunately, all of our Goldy cams - except Narrowneck and Kirra, which are still streaming - are on a different ISP to the rest of our national network, and this ISP seems to have experienced a major outage around 4am (all of our gear is fine). There wasn't any storm related activity overnight, so I'm not sure what the issue is. Currently looking into it as a prriority.
Good news on D'Bah though, the scaffolding came down yesterday so once the network is restored, we'll have a clear view of the southern end of the beach. Apparently they have a little more work to do in late Jan but until then it'll be clear.
All back now.
And now they're down again. Argh.
Over two years to put a camera up at Bondi now
Ain't as easy as it might appear. Fortunately, we now have access to Sydney (tis no longer a Covid hotpot) so the backlog of upgrades/installations are about to kick off.
any actual dates on when these cams will be back up? Bondi cam (down for years). Narra cam (down for months). Avoca cam (getting on to a month now). of the four cams that I actually use regularly, 3 are down - 2 seemingly permanently which doesn't give me much optimism for the 3rd. I know plenty of people in IT and AV companies in the greater sydney region and they're all working and have been all through covid
Avoca has only been down since Friday (not "a month"). NBN visit scheduled for tomorrow.
Other Sydney sites will be reactivated once we I can schedule our new IT team (based in northern NSW) for a Sydney trip. The on-again, off-again Covid situation makes it very difficult, because we have an enormous amount of Qld based work, and can't afford to have to be banned from entering the state from 14 days because we've been to Sydney.
But rest assured, we're trying as hard as we can to find solutions for this.
thanks for the update ben. the avoca cam went down right around xmas day and came back up briefly a few days back before going back down again