C5s stop passing traffic


#1

I have a couple of A5’s with a couple dozen C5’s connected to them, and I have seen this issue 4 times in the recent weeks, and probably a dozen times in the past 6 months.

For legacy reasons I use PPPOE on the client side so in this case the client’s router is PPPOE client through the C5 to the Mikrotik NAS sitting behind a Netnoix switch which is tagging/untagging VLANs and powering the A5 at the tower site.

The A5 is on one VLAN for management (100) and there is another VLAN trunked through the A5 to the C5 (101) for CPE management.

What is happening is that randomly a C5 remains manageable via the wireless interface but it won’t pass PPPOE and it cannot be managed over the Ethernet interface using 169.254.200.20. The Ethernet Interface is up and not flapping.

When the C5 is taken out of service and brought back to the office where it’s not in range of the A5 it can then be managed over the Ethernet interface on the 169 IP (directly plugged into the same POE and Laptop that it wouldn’t talk to when deployed) and after performing a Reset Defaults it can be re-programmed and successfully redeployed. I’ve only just tried this on the 2 most recent cases and they are both back in production now - I wrote the others off as borked :frowning:

The C5’s are properly grounded and the issue randomly affects different C5’s. Rebooting the A5 does not resolve the problem. Replacing the POE does not resolve the problem. All are running the current firmware.


#2

I have not seen this, but I am a heretic who doesn’t run PPPOE or DHCP, and we are only doing a few VLAN deals.

What do the logs say when it goes into Dead PPPOE? (can I coin that for this problem?) Does the C5 become manageable if you un-align it from the sector? (with or without reboot?) Have you tried downgrading firmware? (I guess I should first ask what firmware you are on, but I like skipping steps. But if you are not on the latest firmware what happens when you go to it?)


#3

Heretic - lol I WISH I could not use PPPOE.

Nothing in the logs. This seems to coincide with a power cycle of the C5. Shows Ethernet up 100Full and no traffic.

We’re talking less than 200m to the A5. So the C5’s are still connected when they are in the backseat of the truck.

But the hint on the reset was that I remotely reset one to Factory when it was still deployed and the tech who was plugged in on the test cable to the C5 could then ping it. He configured it up and as soon as it associated to the A5 he could PPPOE. Put it back on the customer’s cable and PPPOE Client and it’s still working. Since that one we’ve brought one back to the office and reset it here and sent it out on a new install and it’s working too.

Yes running the latest firmware on Both A5 anc C5. Haven’t tried downgrading… afraid of bricking it, mind you they seem to be bricking themselves so maybe worth a try…


#4

Promise, the grass is greener in Static IP Land. :wink:

From what you are saying, this should happen for any link and not just ones utilizing PPPOE. Are the routers/computers you are using gig Ethernet ports or just 100M? I am reminded of a known issue that happens if the Ethernet Port on a C5 comes online under heavy traffic: http://client.help.mimosa.co/client-ptmp-firmware-release-notes-c5

Do you use DHCP for the antenna IP or are you static?

This is what I am understanding your network layout to be:
C5 -> A5 -> Netonix -> Mikrotik NAS -> (Internet or close enough)

I am guessing the “Mikrotik NAS” is also your PPPOE server, if not, what and where is it in your network layout?

Also, what is your path to the wireless management interface?

If you are already using Mikrotiks, what happens if you try to “ARP ping” the antenna from each side? (Wireless and Wired)


#5

Some customer routers are gig others are 100Mbps. It’s not under heavy load when it’s just the testers laptop plugged in and still they cannot even ping the stateless IP (169.x.x.x)

Yes the CPE’s get their IP addresses in the Management network from a DHCP server. The DHCP server logs are read by the Helpdesk/CSR’s Management console so that they can single click to get to a customer’s radio.

You have the lay out right. Every tower is the same : AP <> Netonix <> Mikrotik NAS <> Backhaul <> Internet POP

Wireless management interface is on the Management Network which is logically separate from the User
network traffic but uses the same physical transport.

Only the NAS is Mikrotik - the customer Router is whatever they buy. Mostly DLink because that is what we recommend but also Netgear, ASUS and Linksys. It doesn’t seem to matter what the PPPOE client is because even the installer’s Laptops which run the PPPOE Connectoid on them won’t connect.

The lack of any type of Ethernet stats on the C5 makes it very difficult to see traffic but even when I span the A5 switch port into a Packet sniffer I DON’T see the PADI packets to initiate PPPOE session coming through AND the installer can’t even ping the C5 loop back

It’s like the C5 goes into a spanning tree fit and disables traffic on the Ethernet Interface.


#6

Have you tried to ping through the C5 while it is booting up, maybe see if there is a short period of time where it will pass traffic before it freaks out?

Do you have flow control enabled?


#7

Without PPPOE running nothing passes through. I tried setting up a DHCP server on that VLAN and had the Tech setup a DHCP client on his Laptop and there were no requests received at the DHCP server.

I tried with and without Flow Control and we tried hard coding both the C5 Interface speed and duplex. Nothing.

Turns out that a number of the Other C5’s that I thought went to e-cycling in fact got shelved in storage waiting for a enough to make a load. I’m going to pull them out and see if they can be recovered as well.


#8

Have you tried ARP Pinging through? I can ARP ping from both sides of a C5 with a Mikrotik, but it should be something doable with Windows.

Yay! Thank goodness for saving money and a hit of laziness.