Broadcast packets from vlan clients - odd results

Hey Guys,

I have my clients on vlans, APs on management vlan. My router behind my AP is getting odd Mikrotik Discovery Protocol messages coming in from the Mikrotik clients located behind my Mimosa C5x CPEs.

It almost seems like maybe an issue of MTU. As if the packet is getting stripped of some info when the vlan tag is added. Any thoughts, suggestions?

tried enabling jumbo mtu - does not seem to help!

system mtu jumbo 2474 in ios. show int giX/X verifies MTU is at 2474

Are you terminating the VLANs on that router? I mean, if the router is on the VLANs then it’s going to see discovery broadcasts from other routers on the same VLAN…

What is odd about the Discovery Protocol messages?

Yes they should be seeing the broadcasts, the issue is that the IP address field is coming through “jumbled up” so say if it was 192.168.0.1 it will show up as 192.1.0.0 or something very similar. Not causing any major issues as this seems to only be these broadcast messages, but obviously something is not right!

I could pull pcap’s from originating device and receiving device to compare?

That’s probably your best option. If the packet was getting screwed up by the link, you would be seeing a lot of malformed, CRC or FCS errors on the link.

Source or Destination IP address? For a broadcast address, assuming a /24 network and using your IP example, it would be 192.168.0.255 for the Destination IP. If it’s a DHCP Discover or Request, then the Source IP will be 0.0.0.0 and the Destination IP will be 255.255.255.255. For a DHCP Offer or ACK message, the Source IP will be that of the DHCP server and the Destination IP will be 255.255.255.255. If you have a DHCP Relay (Helper) in the mix, that will change the messages. The above assumes the computer and DHCP server are on the same VLAN.

These start as layer 2 broadcast messages going to FF:FF:FF:FF:FF:FF which eventually gets translated to 255.255.255.255 if that helps!

I have some other links using Mikrotik PtMP and the data is coming through fine. Just on my Mimosa PtMP clients.

No errors on ports, all other info is coming through OK. Just the data holding the IP Address being broadcasted in these packets. Not to be confused with the source IP - this is the IP Address data field located in the data payload of the packet, not the source IP of the packet itself.

These start as layer 2 broadcast messages going to FF:FF:FF:FF:FF:FF which eventually gets translated to 255.255.255.255<<

That’s what I would expect to see for a broadcast message.

Was there a problem to begin with that caused you to look or did you just happen to find these when you used Wireshark? If there wasn’t a problem, then I would suspect that there’s something on that link sending the messages you mention. If you find out what that is, it may answer your question about the messages themselves. If there was a problem showing up, then knowing what the problem was may help with the diagnosis.

We know the source, it is a Mikrotik routerboard. We are expecting to see these packets. The issue is that the packets contain invalid data - the data is valid leaving the device and invalid when received so the perceived issue is somewhere between the C5X and A5C.

MT1 = Mikrotik Router 1 (where broadcast packet is originating)
MT2 = Mikrotik Router 2 (where broadcast packet is received)

MT1 <-ethernet-> C5X <-wireless-> A5C <-ethernet-> Cisco Switch <-ethernet-> MT2

Show CDP Neighbors detail on Cisco Switch shows:
Device ID: [John Doe]
Entry address(es):
IP address: 192.168.1.162 - [this is showing correct here, modified for obv reasons]
Platform: MikroTik, Capabilities: Router
Interface: GigabitEthernet0/24, Port ID (outgoing port): ether1
Holdtime : 117 sec

Version :
6.47.9 (long-term)

advertisement version: 1
Power Available TLV:

Power request id: 0, Power management id: 0, Power available: 0, Power management level: 0

Management address(es):

But on MT2 it is showing up malformed. Not sure if CDP uses the source IP vs Mikrotik Discover Protocol using IP from data in packet. Will do some more homework and post any results I find!

If I’m understanding you correctly, the packet from MT1 is OK at the Cisco switch, but bad by the time it reaches MT2. For starters, CDP will only work on adjacent devices. So, the MT1 CDP packet shouldn’t be getting to MT2. It should stop at the Cisco Switch. So, I don’t know what’s getting to MT2, but it should NOT be CDP from MT1. If it is, it makes me wonder if there’s a problem with the Mikrotik implementation of CDP. Have you tried LLDP instead? That is usually more multi-vendor compatible.

https://learningnetwork.cisco.com/s/article/cisco-discovery-protocol-cdp-x

So digging here it looks like MNDP just spans layer 2 broadcast domains. Mikrotik’s also blast out discovery messages on CDP, LLDP, and MNDP.

So I may be seeing CDP messages in one spot and MNDP in another. I will trim this out by protocol and see if that can shed some more light here!

https://help.mikrotik.com/docs/display/ROS/Neighbor+discovery

OK guys an update here! I have just now realized after doing a bit more digging here that there is in fact some info being stripped from the UDP packet as suspected.

When I look at a known good pcap of this MNDP (Mikrotik Network Discovery Protocol) it is received with a length of 170 bytes.
When I take a look at the pcap of the MNDP device over vlan through an A5c and C5x, it is received with a length of 166 bytes.

So somehow 32 bits (4 bytes) are getting stripped or fragmented somewhere. Wish I could take pcaps from straight from the A5c! Will do some more digging. Maybe mirror the A5c port and pull a pcap from it?

Any ideas welcome!!

Quick side note, going the other way (gateway to customer router) MNDP message is being received OK.

Any idea on what is getting stripped out?

Well, making an assumption here, I can only imagine it is the data payload containing the IP address. It seems the IP is taken from the data payload itself as opposed to the source IP.

Those extra 4 bytes likely have the last two octets in them somewhere!

The 4 bytes missing are from the MNDP data payload, as well

MT has also released a Off/On switch for lldp, cdp, and mndp. The cdp and lldp messages are only sent to neighbors where the MNDP protocol sends a global broadcoast.

So in essence I can say CDP and LLDP seem to work from the Cisco switch reading the client on the vlan, just not MNDP.

I think the bigger issue here is that data is being lost or stripped. I am receiving some RX drops on my interface here as well.

Could you post some of your packet captures?

It seems to me that if the packet were being messed with either more packets would have been messed with or it would be effected in a different way. (You would be getting FCS error because the CRC checksum would be erroring out)

I have multiple APs where I am doing stuff with VLANs, and none of them are having issues with packet drops or missing source IP address. Here is one of my routers getting correct data from one of my customers who’s behind a VLAN on one of my Mimosa APs…

any way to PM them? I hate to post a pcap containing customer info