Radios stop responding every day at same time for 1 hour


#21

I am going to reboot this discussion since it is once again driving me insane.

Every single B5 or B5 Lite (we don’t have any B5c) we put on our network stops responding to our monitor machine 24 hours after it was installed and it stops for exactly 1 hour - every day. You install it at 3:00pm and the next day at 3:00pm, it stops responding to pings and SNMP for exactly 60 minutes - and every single day after that. You reboot the Mimosa at 11:30am and the next day, instead of 3:00pm, it stops responding at 11:30am for exactly 60 minutes. It is 100% related to the Mimosa uptime.

Our monitor machine is a Windows 7 machine running PRTG. We use both SNMP and Ping to monitor our radios.

When the radio stops responding to PRTG, if I log into the Windows machine and try to ping a Mimosa radio from command line, it does not respond. If I try to ping that radio from our switches or another random device on the management VLAN, it works. So, the Windows 7 machine can’t see every single B5 and B5 Lite for exactly 1 hour every day at the same time.

Thinking this was a Windows issue, we built a Windows 10 PRTG server on a different piece of hardware and on a different IP address. Guess what? Same exact thing happens to that server - it can’t see the Mimosa radios for exactly 1 hour nor can the computer ping them during that hour.

If I reboot the Windows PRTG server during this “outage”, it still can’t access them until that hour is up. So, this really has me believing this is a Mimosa thing, not a PRTG or Windows issue. We monitor over 200 devices with this server. The only devices that have an issue are the Mimosa B5’s and B5-Lites (A5 and C5’s are fine).

I am not able to confidently monitor these radios since we get alarms every day for every radio. We end up turning off alerting so if it really does go down, we won’t know until devices behind it start squawking.

Would love some help from Mimosa on this… We are running the latest release firmware as of 5/2017. It has been happening on at least the previous firmware as well.


#22

The radios may be blacklisting the router IP for 1 hour because they detect what appears to be a Denial of Service (DoS), or ping flood attack. It could also be because of a port scan. Reducing the ping frequency from your NMS to the Mimosa management interface (radio IP address) should be the first troubleshooting step.

Security was added with iptables in firmware 1.3.0. It is configured to limit ICMP to 3 pings per second, and block IPs that attempt port scans for 1 hour.

Firmware 1.4.5 shows blacklisted IPs in the Support log, so this should tell us if that is the root cause. Please send your logs to Support.


#23

OK, I have reached out to support. We are only using SNMP on almost all of the Mimosa radios (no pings). I don’t think PRTG is doing anything else (since I haven’t told it to) but we’ll triple check that.

Can I whitelist IPs or subnets?


#24

No, the Backhaul radios do not have any additional firewall features exposed. This was for basic self-protection, and not meant to be a full firewall solution. I can submit the trusted IP/subnet as a feature request.


#25

Yes, please do submit that request as I have spend several hours along with my team trying to figure out what was going on. Furthermore, your immature product not only does not let us modify firewall settings, there is no documentation expressing this “feature” or service nor does it report this info to our syslog server. So, is Mimosa forcing us to use its cloud monitoring services and if so, why has it taken almost a year for Mimosa to figure out the issue and yet to expose the device firewall services?

I would love to continue to using Mimosa as your radios do work very well but this continued attitude of you know best what your clients want or need is a bit arrogant and may force me to migrate away from Mimosa.


#26

We have not confirmed that the radio’s self-protection mechanisms are the cause of the issues reported above.

Our view is that the ideal place for a firewall is on an upstream router (not on the radio), but some users reported that the management interfaces on publicly accessible radios became slow or inaccessible when their networks were under denial of service attacks. We added basic protection to stop this from happening.

We don’t require the use of the Mimosa cloud monitoring service. It is completely optional.


#27

I must have totally missed the information about a built-in firewall. While I can see (maybe) a reason to have your radios on the Internet, none of ours are and I’d prefer to not have the radios doing any sort of black listing or firewall. How does that get turned off??


#28

There are no controls for the management interface firewall at this time. I have submitted the request for a white list feature and toggle for the firewall. Everyone operates their networks differently.

Last year when we were troubleshooting your issue, firmware 1.3.1 was running for several months before the incidents occurred. We theorized that TDWR exclusions were to blame since you were operating on a TDWR channel near TDWR and we saw radar detection messages in the logs. After you moved channels, the problem went away.

The radios are not on the cloud so I can’t retrieve any logs. I checked our support tickets and did not find any current ones. Would you please PM me one of them so that I can look at it? We really need that to say if this is indeed the problem.


#29

PM has been sent to @Chris Once we learned there was a firewall built in, we made some modifications to our PRTG install and I think have finally solved this. We used to ping with 5 packets every 30 seconds and our PRTG install defaulted to do a port scan on the hardware once every 24 hours. We turned that off and set the pink for 1 packet every 30 seconds - since both of those appear to be firewall blacklist triggers.


#30

The daily port scan is likely the cause. Pings can be as much as 3 per second per IP address, so the earlier ping settings were not the cause.

The 1.4.5 firmware is what reports the blacklisted IP address(es) as an event, but you have older firmware versions where those message are not available.


#31

When can we expect the ability to either turn off this “feature” or the ability to exclude IP from blacklisting? Also, is there a list somewhere that shows all of the features added that are not otherwise made clear to your customers?


#32

Hi Alex,

I have submitted the request for a toggle control to turn off the firewall that would normally protect the radio’s management interface. I’m trying to get this scheduled for inclusion in the next backhaul firmware release. We don’t yet have a date for it.

Product features are documented on the Help Site within Datasheets, FAQs, User Guides and Troubleshooting documents. We continually add to these resources and endeavor to make them as complete as practical.

The problem described above is covered within our troubleshooting guides, but we do not regularly publish details about internal security mechanisms that could provide hackers with specific attack vectors.


#33

Hi Alex,

I have units with the latest firmware installed 1.4.6 and still have this problem. Suddenly we are not able to reach them by web or by icmp or even snmp but they keep passing traffic. This is becoming a problem since there is no way to manage the units or commit any configuration changes. Is there any workaround to overcome this issue?. Thanks a lot


#34

Hi,

We have been seeing this behavior as well with our monitoring system. At first, we were only pinging each backhaul radio once per minute for basic uptime notifications and we would get blocked from connecting to each of our Mimosa backhaul radios about once per week. After 1 hour, we were able to connect again. Eventually, we stopped the ping test from our monitoring server and started pulling the Streams and Chains table via SNMP instead. We are pulling this data every minute from each radio. Yesterday our monitoring system was blocked again by all of our backhaul radios for 1 hour. This does get confusing because it appears we have links down when we really don’t. I would second a request to have the ability to turn this feature off for the backhaul radios.
Thanks!


#35

We are planning on a 1.4.7 release for the Backhaul products. It is primarily a bug fix release, but does include a switch to allow a user to disable internal rules which limit pings to 3 per second. We are looking for users who would be willing to test this feature.


#36

We are running a mix of 1.4.5 and 1.4.7 beta3 in our network. We just started using Spiceworks to map and monitor our network. The DOS protection kicks in on all the B5 lites we have on the network and we start getting down alarms for the network. This really needs to be able to be turned off or allow me to whitelist my management subnet.

It does not look like we have the ability in 1.4.7beta3 to turn off this 3 pings per second feature. Where are we with this?


#37

Hi @Chadwick,

In 1.4.7 beta (and soon GA 1.4.7) you can turn off the “DOS protection” in the radio. Log in and go to Management. Under Miscellaneous put the “Firewall” slider to Off.