Why is that surfcam offline?

thermalben's picture
thermalben started the topic in Thursday, 4 Oct 2018 at 9:11am

I installed Swellnet’s first surfcam sometime in 2005. Since then - more than thirteen years! - technology has advanced rapidly, and the quality and reliability of our surfcam streams has improved considerably. 

However, the last few weeks have seen quite a few surfcams go offline for a wide variety of reasons - most of which are completely outside of of our control. So, whilst we work our hardest to get them back up and running, I thought it was worth explaining a few of the problems that can go wrong, and why it can take quite some time to rectify these kinds of issues.

Our surfcams are an automated network. That is, once the cameras are installed, in a perfect world they never have be attended to again unless they go offline. Of course, being right on the beach in highly corrosive marine environments, facing the brunt of every single weather system that sweeps the coast, this means that our hardware is frequently put to the test. 

But, by and large, our surfcams will often run for many years without any problems. We only notice when they’re down. 

So I thought it'd be interesting to share our troubleshooting regime for when our surfcams do go offline, and an example of the check list we might run through. 

Is it a local power issue?

Whilst each surfcam has an on-site uninterruptible power supply (UPS), it’s only small and offers just twenty minutes of backup power. They also protect our equipement from brown-outs, surges, spikes etc - though not all the time - I've seen crazy instances where a surge has gone through one device without affecting it, but then destroyed something on the other side (for example, killing a surfcam but not the power supply that the surge had to pass through to reach the surfcam).

Anyway, the UPS is fine for most minor glitches in the electrical grid network. However, if the power is down across the suburb, then the surfcam will also be offline. Most power companies share this information online, but sometimes it can be slow to be made publicly available. 

Is it a local electrical issue at the property?

This happens less frequently, but is the source of our current surfcam outage at 13th Beach. Let me explain in more detail…

After many years of uninterrupted service, our 13th Beach surfcam went offline a few weeks ago. Access to the surfcam is difficult (it’s at the Surf Club, and we’re out of patrolling season), but finally after our local sparky got into the building, we found that the surfcam shared an electrical circuit with some SLSC equipment, which had shorted.

The problem was temporarily fixed, though installing a dedicated circuit was recommended - but couldn’t be done there and then (as parts needed to be sourced, and an installation time booked). Because we’re upgrading that site to NBN in the coming months, we thought we’d do it all in the one go.

But of course, it didn’t last that long… it appears that the circuit has shorted again, just a few days ago (it'll be fixed ASAP).

Is the problem related to our Internet Service Provider?

This issue is even more complex than you can imagine, especially with the gradual switch from ADSL to NBN around the country (around 85% of our network is now on NBN).

Aside from the physical points of failure - all of which are out of our control (the first connection point at the property, the pit in the street, the pillar down the road, the local exchange and the network data centre), sometimes we get thrown a crazy curveball. Allow me to digress…

We’ve had a surfcam at Lorne for many, many years, inside the observation tower at the Surf Lifesaving Club. Initially on an ADSL connection, it was upgraded to ADSL2 a while back, and then last year, finally upgraded to NBN. All was working fine.

However, a couple of weeks ago our ISP advised that they’d mistakenly received notification that the service was to be ‘churned’ to another ISP - something we hadn’t instigated. So, my ISP stopped the process (at a cost to them), and we tried to figure out what went on. 

Another week later, the same ‘churn’ request came through again, and my ISP went through the motions to stop it happening (incurring another cost). 

We were finally advised that - despite our internet connection being in place, and physically unchanged for well over a decade - it was actually using the ‘wrong’ connection somewhere along the network path (external from the Surf Club, on the street), and was mapped to someone else’s property up the road. The owners of this unknown property were now applying for their own NBN connection, and hence the problem - Swellnet's new problem - was uncovered. 

For some unknown reason, NBN were unable to re-route our existing connection, so they simply disconnected Swellnet's internet service last Friday (without notification, either - we are still yet to receive confirmation). We've simply been told that the fastest way to get the Lorne surfcam back is to “re-apply for a new NBN connection”. Which not only means the surfcam will be offline for a few weeks, but now requires two specialist technician visits - a qualified sparky (ACMA approved with open/restricted cabling license) to switch the internal cable at the Surf Club to an unused line, and another technician to reconfigure our modem with the new log in details. And then we’ll need to make some changes within our surfcam streaming infrastructure to get the vision live on the website again. 

Fun and games, eh? Add up all of the associated time and cost to fix this problem, including replying to disgruntled Swellnet subscribers who can't view the Lorne surfcam, and it's a royal pain the arse.

This is just one of many examples of what goes on behind the scenes on a weekly basis, just to keep the surfcam internet connections up and running. I'm gonna write a book one day. 

Anyway, back to troubleshooting.

Is the problem related to our surfcam hardware? 

Each surfcam location has several pieces of equipment, and although we utilise industrial/business-grade hardware, every device still has the capacity to freeze, or break down at some point. Usually when the swell of the year is bearing down on the coast.

Some of our equipment has lasted five or six years before needing to be replaced. On rare occasions, certain pieces of hardware have been DOA (dead on arrival) which is a pain in the arse when you've freighted all of the equipment down to some remote location for a new install, teed up an electrician to install it all, and then the bloody camera needs to be returned to the supplier under warranty for a replacement (resulting in another costly technician visit and a couple of weeks delay on the install). 

Have you ever had your ISP ask you to “turn the modem off and on again”? Well, it’s not as stupid as it sounds - that simple procedure often fixes most problems. But, back in 2006 it only took one four-hour drive from Sydney to Ulladulla to press the ‘restart’ button on our old Mollymook surfcam (remember that?), for me to realise that there had to be a better way to remotely power cycle my remote surfcams. So, ever since then we’ve installed expensive remote rebooting equipment at each surfcam that allows us to turn all of the devices off and on again, without needing anyone to visit.

Another example: two weeks ago, our Newcastle surfcam went down.

It took almost a week to arrange a visit (cam went down on a Friday night, so couldn't arrange anything until the Monday; bloke got there Friday). All equipment in the cabinet was working fine, but the technician reported the power source to the surfcam was dead as the power lights were completely off. Now, this was unusual - the power supply rarely dies outright, more often, when there's a fault it'll change the colour of the indicator LEDs from green to orange. So, I asked the technician to look a little more closely. Lo and behold, the power cord into the back of the power supply had come loose, and it just needed to be plugged back in again and the surfcam came instantaneously back. Lucky I didn't arrange for spare parts to be sent down, as that would have delayed its return by up to another week. 

But sometimes it's not that easy to source the problem. It could be one or more of the following issues:

  • Modem is dead and needs to be replaced
  • Surfcam is dead and needs to be replaced
  • Surfcam power source (also called a midspan) is dead and needs to be replaced
  • UPS is dead (or its battery is dead, as per Snapper Rocks last weekend) and needs to be replaced
  • Custom cleaning hardware is dead and needs to be replaced
  • Remote rebooting hardware is dead and needs to be replaced (the most ironic of all!)
  • Surfcam and/or modem firmware is playing up and needs to be factory reset (requiring a specialist technician visit)
  • Cabling has been affected and needs to be replaced (once, we had rats chew through the data cable to the surfcam!). Also worth pointing out that this is usually a not-insignificant project; laying cable is often the most time consuming part of any surfcam installation
  • Lightning strike and an associated power surge may have damaged cabling through the property (couple of years back, we had four locations affected by lightning in one week) - again, a difficult problem to source, and can only be fixed by an electrician

Ordinarily, we need someone to visit the surfcam site to check everything out and determine what the problem might be. We have a lot of spare equipment on standby in the Swellnet office, so once we know what the problem is, we can usually replace broken hardware relatively quickly - assuming our local technician is available, and can gain access to the property - I am not kidding when I say that there’s been more than a couple of occasions when a surfcam outage has happened the day after the person with the keys to the property goes overseas on holidays for a couple of weeks. 

So, in order to cover all bases, when a surfcam goes offline and we suspect it’s hardware related (i.e. we're confident it's not an electrical problem, nor an ISP/network issue), we often freight a single Pelican case with one replacement of everything (camera, midspan, router, cleaning hardware, rebooter, UPS) to our local technician. This ultimately saves a second technician visit, which could take another week or two once availability and freight times are factored in. However the actual surfcams themselves are very expensive, and we use a couple of different models that each have different mounting brackets, so we can't keep spares of everything - which means purchasing new equipment, some of which is only available from overseas suppliers, which therefore takes time to source and freight. It's not like a supermarket where you can walk in and buy stuff off the shelf - some of our equipment is specialised and may not be available to purchase for periods of time, if the manufacturer runs out of stock.

Anyway, that's just a basic overview of some of the issues that affect the uptime of a surfcam. We're always developing new tools and techniques to improve the reliability, so I hope this answers some of the question you may have been pondering.

thermalben's picture
thermalben's picture
thermalben Tuesday, 26 Apr 2022 at 2:31pm

Next week. Been difficult to schedule a trip to Sydney; two of my technicians were sick last week (one with Covid), so all of our work got pushed back accordingly.

Sprout's picture
Sprout's picture
Sprout Tuesday, 26 Apr 2022 at 2:54pm

No worries, thanks mate.

marcus's picture
marcus's picture
marcus Thursday, 28 Apr 2022 at 12:32pm

Hi Ben
is there any way we could get stu to stand on the point under the norfolk island palm overlooking the island and do morse code when a set rolls though?

just kidding, hope you are well mate

thermalben's picture
thermalben's picture
thermalben Thursday, 28 Apr 2022 at 12:35pm

The other technician caught Covid yesterday, sent home from a job site to isolate for 7 days. Argh.

dannyz's picture
dannyz's picture
dannyz Monday, 2 May 2022 at 10:02am

hey mate any chance we can get the Avoca cam panning?

eel's picture
eel's picture
eel Sunday, 22 May 2022 at 1:38pm
thermalben wrote:

The other technician caught Covid yesterday, sent home from a job site to isolate for 7 days. Argh.

How's that Narrabeen cam going that's been down for 2 and a bit years that you KEEP saying is getting fixed in the next 'few weeks'. Pretty shit as a paying customer.

DAW's picture
DAW's picture
DAW Sunday, 22 May 2022 at 2:52pm

Deceptive advertising

Marc_09's picture
Marc_09's picture
Marc_09 Tuesday, 24 May 2022 at 4:06pm

I second the sentiment of eel. I love Swellnet but two years or more of no reliable surf cam at Narrabeen is making it very difficult to justify not moving to the competitor.

eel's picture
eel's picture
eel Tuesday, 24 May 2022 at 7:39pm

Yep. The fact that they aren't even replying (even though they are online replying to other posts) probably indicates that the Narrabeen cam is still a long way off. Very deceptive how they said several times 'the next few weeks' about Sydney cams. I'm pissed off that I paid yearly and not month to month. When my subscription runs out I'll be moving to Surfline.

eel's picture
eel's picture
eel Monday, 30 May 2022 at 3:11pm

Sooo that Narrabeen cam?....Another week gone by and still no response?

AlfredWallace's picture
AlfredWallace's picture
AlfredWallace Monday, 30 May 2022 at 3:45pm
eel wrote:

Yep. The fact that they aren't even replying (even though they are online replying to other posts) probably indicates that the Narrabeen cam is still a long way off. Very deceptive how they said several times 'the next few weeks' about Sydney cams. I'm pissed off that I paid yearly and not month to month. When my subscription runs out I'll be moving to Surfline.

Eel, Swellnet folk like us will applaud your migration to whatever other platform you decide to join, you’d probably whinge there as well. So fuck right off then as you please. Do you understand the enormity of operating a company like this one that involves, electronics, high end tech, cameras, satellites, purchasing data from other scientific organisations etc,, hardware that’s on the frontline of meteorological events, vandalism and so on, obviously not you clown. Please remind me how much your annual subscription is ?
You probably count your money on a daily basis and spend most of your time checking your bank balance. Fair dinkum. Do you even surf or are you a keyboard warrior ?

eel's picture
eel's picture
eel Monday, 30 May 2022 at 8:58pm
AlfredWallace wrote:
eel wrote:

Yep. The fact that they aren't even replying (even though they are online replying to other posts) probably indicates that the Narrabeen cam is still a long way off. Very deceptive how they said several times 'the next few weeks' about Sydney cams. I'm pissed off that I paid yearly and not month to month. When my subscription runs out I'll be moving to Surfline.

Eel, Swellnet folk like us will applaud your migration to whatever other platform you decide to join, you’d probably whinge there as well. So fuck right off then as you please. Do you understand the enormity of operating a company like this one that involves, electronics, high end tech, cameras, satellites, purchasing data from other scientific organisations etc,, hardware that’s on the frontline of meteorological events, vandalism and so on, obviously not you clown. Please remind me how much your annual subscription is ?
You probably count your money on a daily basis and spend most of your time checking your bank balance. Fair dinkum. Do you even surf or are you a keyboard warrior ?

The only keyboard warrior is the internet tough guy calling me a clown and suggesting I should be donating freely to Swellnet. I have been a paid subscriber to this site for a number of years. The gall of me expecting something that I paid money for. Or at the least, expecting the staff to answer a simple question about when a cam is going to be online?
Yes I surf and have done so for many years at a lot of the bucket list waves around the world. Thanks for asking champion.

Rabbits68's picture
Rabbits68's picture
Rabbits68 Thursday, 2 Jun 2022 at 12:18pm

"Jeez, looks like someone's not going to be on the SN Xmas card list!"

I'm ok with that. We (not SN & I) stopped doing Xmas cards years ago. Save the trees n all....

Do you care to comment on the subject matter? Why would surfcam feedback comments all of a sudden be getting deleted without explanation? It's pretty rare for any comments on any subject matter to get deleted on SN. First world issue I know. I am curious though.

Stok's picture
Stok's picture
Stok Thursday, 2 Jun 2022 at 12:31pm

You know, theyd get deleted because of that balance between being a business and also have attaching a soapbox to your business (this forum) allowing anyone to say whatever they want. It's fairly unique.

I'm all for more cams tbh. The horse had bolted, genie is out of the bottle, past is the past, golden years are gone. Get busy living or get busy dying. Anyone against cams is living in a fantasy world of yesteryear when phones only worked when plugged into copper, cars weighed several tonnes and women weren't allowed in offices.

DudeSweetDudeSweet's picture
DudeSweetDudeSweet's picture
DudeSweetDudeSweet Thursday, 2 Jun 2022 at 12:43pm
Stok wrote:

You know, theyd get deleted because of that balance between being a business and also have attaching a soapbox to your business (this forum) allowing anyone to say whatever they want. It's fairly unique.

I'm all for more cams tbh. The horse had bolted, genie gis out of the bottle, past is the past, golden years are gone. Get busy living or get busy dying. Anyone against cams is living in a fantasy world of yesteryear when phones only worked when plugged into copper, cars weighed several tonnes and women weren't allowed in offices.

Do you live in a densely populated area and predominantly surf a city beach?

Rabbits68's picture
Rabbits68's picture
Rabbits68 Thursday, 2 Jun 2022 at 1:08pm

"SN have not explained (that I have seen) why the obvious change in attitude or tolerance to the subject matter, that's all."

Thanks for the explanation Ben. Appreciate it. Enjoy your holiday.

Stok's picture
Stok's picture
Stok Thursday, 2 Jun 2022 at 1:21pm
DudeSweetDudeSweet wrote:

Do you live in a densely populated area and predominantly surf a city beach?

I grew up in cities, but now live essentially in coastal rural suburbia.
I'm essentially public enemy number 1! I used to be a blowin, now I'm the one who's moved in from the city and bought a house.

Look, I'm not for cams literally everywhere, but I do feel the track has been well beaten here in Aus. Doesn't matter if a surfer lives in a city or not, if they're serious about getting waves (which so, so many people are) then they'll happily travel 1-3 hours from home to score. Drop a pin at any surfbreak in Aus, then measure out a 2-3 hour radius drive - the result is heaps of surfers. And they're coming for a surf, when the tide, wind and swell line up, regardless of whether a cam is installed or not.

sypkan's picture
sypkan's picture
sypkan Thursday, 2 Jun 2022 at 1:31pm
DudeSweetDudeSweet wrote:
Stok wrote:

You know, theyd get deleted because of that balance between being a business and also have attaching a soapbox to your business (this forum) allowing anyone to say whatever they want. It's fairly unique.

I'm all for more cams tbh. The horse had bolted, genie gis out of the bottle, past is the past, golden years are gone. Get busy living or get busy dying. Anyone against cams is living in a fantasy world of yesteryear when phones only worked when plugged into copper, cars weighed several tonnes and women weren't allowed in offices.

Do you live in a densely populated area and predominantly surf a city beach?

the dude lives in a bubble, that's where he lives!

sypkan's picture
sypkan's picture
sypkan Thursday, 2 Jun 2022 at 1:33pm

"...Drop a pin at any surfbreak in Aus, then measure out a 2-3 hour radius drive - the result is heaps of surfers. And they're coming for a surf, when the tide, wind and swell line up, regardless of whether a cam is installed or not..."

yeh nah bubble boy

AndyM's picture
AndyM's picture
AndyM Thursday, 2 Jun 2022 at 1:45pm
Stok wrote:

You know, theyd get deleted because of that balance between being a business and also have attaching a soapbox to your business (this forum) allowing anyone to say whatever they want. It's fairly unique.

I'm all for more cams tbh. The horse had bolted, genie is out of the bottle, past is the past, golden years are gone. Get busy living or get busy dying. Anyone against cams is living in a fantasy world of yesteryear when phones only worked when plugged into copper, cars weighed several tonnes and women weren't allowed in offices.

What a bizarre little comment.

DudeSweetDudeSweet's picture
DudeSweetDudeSweet's picture
DudeSweetDudeSweet Thursday, 2 Jun 2022 at 1:55pm
Stok wrote:
DudeSweetDudeSweet wrote:

Do you live in a densely populated area and predominantly surf a city beach?

I grew up in cities, but now live essentially in coastal rural suburbia.
I'm essentially public enemy number 1! I used to be a blowin, now I'm the one who's moved in from the city and bought a house.

Look, I'm not for cams literally everywhere, but I do feel the track has been well beaten here in Aus. Doesn't matter if a surfer lives in a city or not, if they're serious about getting waves (which so, so many people are) then they'll happily travel 1-3 hours from home to score. Drop a pin at any surfbreak in Aus, then measure out a 2-3 hour radius drive - the result is heaps of surfers. And they're coming for a surf, when the tide, wind and swell line up, regardless of whether a cam is installed or not.

Unless you’re talking about a point or reef break this is patently untrue. Even then crowds ebb and build on word of good sessions getting around. You only have to witness a crowd growing at a break over a sustained period of good waves to see evidence of people’s demand for return on travel investment, even if it’s a destination within their usual travel bubble.

There’s hundreds of breaks within a couple of hours of crowd centres which are usually empty but which get very busy once word of recent good waves spreads. Surf cams ensure that any periods of semi-rideable waves are always crowded due to the fact that they are never given a chance to slip under the radar.

Just this morning I was watching a super fun little peak at a beach not regularly blessed with decent waves . Three crew on it. If this beach had a cam there would have been a couple of dozen people on it no doubt.

That’s why the expansion of surf cam numbers is always contentious.

You do t have to be a Luddite to grasp this truth and oppose the incursion of business profit onto our beautiful coast.

Besides….who wants every private beach moment broadcast into the universe? Not me! I feel sorry for crew who have to go to beaches surrounded by houses , shops and busy carparks, let alone having a friggen camera pointed at you all day.

Stok's picture
Stok's picture
Stok Thursday, 2 Jun 2022 at 2:08pm

Fair enough, you've got me there - if a cam is exposing the once in a blue moon epic bank on a beachie then my views differ. The context for me was the cam at Winki (expecting this post to be deleted shortly).

I don't really see a surfcam on private land as an overstep into our coastlines. That ship left when the council zoned that area for private land.

I also don't see surfcams popping up in areas without erected buildings, infrastructure, suburbs etc. Although maybe I'm wrong?

velocityjohnno's picture
velocityjohnno's picture
velocityjohnno Thursday, 2 Jun 2022 at 2:13pm

I did notice in the local papery thing, there was a headline article about the Winki and Wye cams, suggesting disapproval and 'is it above board?'...
In the meantime, ermagerrd, Fishos!

velocityjohnno's picture
velocityjohnno's picture
velocityjohnno Thursday, 2 Jun 2022 at 2:15pm

That cam is going to be so handy, we will be able to refer you all to it and say 'see? what a circus!'

DudeSweetDudeSweet's picture
DudeSweetDudeSweet's picture
DudeSweetDudeSweet Thursday, 2 Jun 2022 at 2:19pm

It’s not just once in a blue moon sand banks at peril though is it? I can think of dozens of often fun beachies which are usually uncrowded because people can’t check them at their convenience, they don’t have contacts to tell them what local conditions are like and the local crew know better than to broadcast the quality of their waves to the general public.

Put a cam at any of these places and the regular ctrowd numbers would swell alarmingly.

I can even think of quite a few beaches adjacent to very crowded waves which remain uncrowded because crew are too lazy/ too stupid to check them. Often great waves breaking for a few crew whilst the nearby name spot is half as good and ridiculously crowded.

A surf cam would change all that quick smart.

Also the idea that crew understand the required conditions to make somewhere pump has caveats attached. Often conditions will punch above their weight and far exceed how they should be relative to spots nearby or based on predictions and computer model. Localised weather patterns throw a big spinner into that machine.

Plenty of reasons to cheer everyday a beach goes without a Big Brother surf cam on it.

DY's picture
DY's picture
DY Friday, 10 Jun 2022 at 12:44pm

I think many of us in Sydney find it frustrating that a business we want badly to keep supporting has acted shabbily with cameras for a long time (well before Covid). It seems SN can't compete on cameras and, rather than investing in trying to, my suggestion would be to offer different subscription levels that appeals to its competitive advantages, eg, a Forecast Notes subscription. I reluctantly have a SL subscription for cams (literally 10x more on the NBs) and keep my SN subscription for the notes, but its a close call and I reckon your average surfer would just pick SL for the app and cams.

thermalben's picture
thermalben's picture
thermalben Friday, 10 Jun 2022 at 1:21pm

hey DY (and others), I'm on leave at the moment, but will give an update on all of this later next week when I'm back in the office. Appreciate your points - they're all valid - but there's a lot of stuff happening behind the scenes. Just not online much at the moment due to travel and associated commitments, so don't currently have the time to run through everything.

Fifty_Plus's picture
Fifty_Plus's picture
Fifty_Plus Wednesday, 22 Jun 2022 at 3:28pm

Any updates on the Bondi Camera? Been a few years now...

Sprout's picture
Sprout's picture
Sprout Wednesday, 22 Jun 2022 at 4:09pm

The other joint's got 2 Bondi and 2 Narrabeen cams, just join it too. $160/year for here and there combined. If that's a stretch just point your DNS to Google's servers, cancel all your streaming services, put a parrot on your shoulder and DL your shows from the bay.

eel's picture
eel's picture
eel Wednesday, 20 Jul 2022 at 7:13pm
thermalben wrote:

hey DY (and others), I'm on leave at the moment, but will give an update on all of this later next week when I'm back in the office. Appreciate your points - they're all valid - but there's a lot of stuff happening behind the scenes. Just not online much at the moment due to travel and associated commitments, so don't currently have the time to run through everything.

So how are those surf cams coming along?

eel's picture
eel's picture
eel Wednesday, 3 Aug 2022 at 6:31pm

So it's August and still no word from Swellnet about those Sydney cams. On June 10 you said "will give an update on all of this later next week when I'm back in the office. " August 3 and still no word and cams still offline - for over 2 years.