A5 -> Mikrotik -> Sonar


#1

We are a small startup WISP and have come to a road block that seems to very troublesome. We are running Mikrotik Routers, Mimosa Equipment and Sonar Software. When we initially setup our network with Sonar everything worked perfectly. Then… we upgraded the firmware on our A5 from 2.4.1.1 to 2.5.1 and everything crashed. Now, when I mean crashed, our DHCP server on the mikrotiks are no longer firing the lease script to Sonar. I was only able to get the leasing script to work by downgrading the Firmware from 2.5.1 on the A5 and the C5 to 2.3.3. Then through testing firmware versions we noticed that it would randomly work on 2.4.1.1 with a C5 at 2.3.3. Then do some more testing and it die. We have been working on this issue for over 2 weeks an unable to come up with an answer. We know it has to do something with Mimosa because any equipment pulled directly into the router works flawlessly. It’s just the leasing going through the Mimosa equipment that is causing issues.
Has anyone had issues with integration with Sonar with Mimosa firmware updates?


#2

Someone else was complaining about and issue with DHCP over C5s, but I have not heard of anyone having issues with Sonar. I have run DHCP from a Mikrotik and had it serve both a C5c and a router behind the C5c with addresses. If you describe you setup a bit more maybe I can duplicate it and get an idea of what’s happening.

Do you have any of the traffic prioritization going on on your network?

Traffic Shaping Plans (Application Prioritization)?

Management VLANs?

DHCP Option 82?

Anything in the logs of the Mikrotik, Mimosa Equipment?


#3

Hi @Byron_S,

I have sent you an email asking for additional information. I am hopeful we can help you resolve this quickly.


#5

William, thanks for the quick reply. In response;
Physical setup:
Currently we have one core router (Mikrotik CCR1009). From there we have an RB3011 that serves as our “tower router” for our AP’s. Now currently since we are just starting out, we only have the one RB3011 that interfaces with our A5/N5-360 then services 2 C5’s for testing. So… very basic at this point. But we are hesitant about putting in our other equipment till we get this “issue” resolved.

Traffic prioritization: Not at this point. We will soon when we start on-boarding business customers and utilizing VoIP services.

Traffic Shaping Plans: We do packet marking on the inline router (RB3011) which then allows us to set our bandwidth controls for the customer side via Sonar

Management VLANs: Yes, we have a management VLAN on bride (same switch chip) where the AP is plugged into.

DHCP Option 82: It is enabled although we have yet to test it’s functionality or success

Logs:
(Mikrotik) - When I delete the lease and reboot the radio, I get VLAN XX assigned “IP” to “MAC Address”, along with the the API users logging in and out that come from Sonar.
(Mimosa) - I see nothing out of the normal from the AP logs or C5 logs. Normal interface up/down and reboots from us testing

More details:
A Mikrotik HaP plugged into the same bride as the AP gets the dhcp assignment just like the AP and C5 and end-user equipment. But the HaP ALWAYS fires the DHCP Script (below) and the “soft assignment” works flawlessly into Sonar every-time. We only figured this out through testing.

DHCP Script that fires leasing info to DHCP Batcher;

:if ($leaseBound = 0) do={
/tool fetch url=“http://x.x.x.x/api/dhcp_assignments?ip_address=$leaseActIP&leased_mac_address=$leaseActMAC&expired=1” mode=http keep-result=no user=YyYy password=ZzZz
} else={
{ :delay 1 };
:local remoteID
:set remoteID [/ip dhcp-server lease get [find where address=$leaseActIP] agent-remote-id]
/tool fetch url=“http://x.x.x.x/api/dhcp_assignments?ip_address=$leaseActIP&leased_mac_address=$leaseActMAC&remote_id=$remoteID&expired=0” mode=http keep-result=no user=YyYy password=ZzZz
};

Strange findings: (I might ramble, see if you can follow)
When I finally got everything to work, I had the A5/C5 on 2.3.3 (on SRS btw). But I knew we orginally had it on 2.4.1.1 because we wanted to utilize the DATA VLAN through the AP and C5. So did the following…

  1. Delete the soft assignment in Sonar (worked and deleted the dhcp assignment on the router); confirms Sonar and Mikrotik are talking.
  2. Reboot the radio to get dhcp. DHCP on both MGMT VLAN and Data VLAN work just fine. Leases are given, network connectivity works but now the dhcp soft assignment isn’t “fired” to the dhcp batcher. Nothing shows up in Sonar and I can confirm the assignment never even reaches the DHCP Batcher, no pending assignments.
  3. (just for testing) I then switch it to WiFi Interop, reboot A5, then C5 and soft assignment works!! Yay! But then I again for testing purposes delete the assignment out of Sonar, which then propagates to Mikrotik. I reboot the radio to see if it pulls soft assignment; no go!
  4. Then I notice WiFi interop Mode with Traffic Optimization is turned on; so I turn it off, reboot the AP, soft assignment works! But then again delete assignment out of Sonar, and nothing works again.
  5. Then I figure; WTH, I’ll see if the firmware will work on the AP, so I upgrade it to 2.4.1.1 and the C5 on 2.3.3. Soft assignment works! “Racking my brain against a wall!!!” So I delete the assignment and test again, no go!
  6. So now I reduce firmware to 2.3.3 on AP, and nothing works. Switch to SRS, nothing works. Switch back to Wifi Interop; nope. Turn on Traffic Optimazation, nothing!
  7. So finally today I just the static assignment in Sonar and everything seems to work now. I let the DHCP server hand out the addresses, take that address and put it as the manual assignment in Sonar and everything is working fine.

So it’s just my “soft assignments” through the AP that are driving us crazy. Sonar tech support has been awesome and really tried to help us. They have confirmed Sonar connection to Mikrotik is working when soft assignment IS WORKING, everything is fine. And even with static assignments, if I change the IP in Sonar, it propagates to Mikrotik. Everything is talking; just not the DHCP script that needs to fire the following into to the DHCP Batcher on our network.

{
    "leased_mac_address": "00:AA:BB:CC:DD:EE",
    "ip_address": "192.168.100.2",
    "remote_id": "BB:CC:DD:11:22:33", #Can be null
    "expired": false
}

#6

So, what you are saying is that:

Depending on weird circumstances, the Mikrotik will notify Sonar about the DHCP stuff. But Occasionally you reboot the radio and the router and the Mikrotik gives out a DHCP lease, but the Mikrotik doesn’t run the script to notify sonar about the DHCP lease renewal/new lease?

Honestly, you have listed enough different firmwares and settings where initially the deal worked, but then the system decided to not work.

Just to be absolutely certain, the radio (C5) and the router (behind the C5) are both getting a DHCP lease from the Mikrotik?

As an aside, I don’t do any of the traffic prioritization stuff on any of the radio equipment we run. (Mimosa doesn’t even give you the ability to prioritize VoIP traffic anyways). As my father always said, “moving parts break, the more you have the more that can break on you.” If you really want to do traffic prioritization, test the hell out of it. When things break it can be really hard to figure out where the break happened

I like doing the bandwidth limiting in the router, Mikrotiks give you lots of tools to control and see what is happening. Doing it as close as you can to the customer is recommended, in the tower router works well enough for us.


#7

Yeah we do bandwidth limiting on the Mikrotik and it works great! Even if we plug the addresses in static, everything works, we just can’t get soft assignments to work.


#8

Every device is getting the appropriate lease. Network connectivity is fine. Internet is fine. The problem is that when going through the mimosa equipment, our dhcp server (RB3011) will not “fire” the script to the DHCP Batcher (this sends the dhcp lease information to Sonar). So know for sure the following:

  • Sonar talks to their DHCP Batcher (on our network) and vise versa.
  • Mikrotik dhcp server talks to Batcher (just not when using mimosa)
  • Mimosa gets correct leases and connectivity from Mikrotik on correct VLANs
  • And only have changing firmware versions or certain options in the AP, have I gotten the soft assignment to work.

Yes I know, not very specific and very strange. I have a talk with Mimosa today which we are excited about in hopes we can get some clarification on where the bottleneck is; hopefully something simple. If we can’t resolve the issue we’re just going to use static assignments as when we put them in, everything then talks without issues; mikrotik, AP, Sonar, Batcher, etc.


#9

That reminds me, I was also going to say that we do almost universal static assigned IP addresses. No moving parts. We have a DHCP server running in case someone resets or replaces their router, but as long as you are not trying to do anything crazy it works pretty well.

Good luck, you defiantly are out of my realm of expertise. If you get an answer I would appreciate it if you reported back here. So I can point to this for the next person with a similar problem…


#10

Mimosa fixed it! Absolutely amazing group of people there. We were on the phone today with a DustinS and the engineering team and after about an hour of wireshark packet analysis, they determined it was the RemoteID field in Option 82 that was causing the issue. When we disabled Option 82, everything worked. So they went back and looked at the packets and noticed it appeared the Sonar Batcher was getting the information it needed being due to the wrong format. So they switched our DHCP Option 82 Code from ASCII to Binary, and left REMOTEID to CPE MAC, and now everything is working like a charm!!!

I’m telling you, between Sonar and Mimosa Support; I’ll use no one else. They are both great and care about their customers. Just simply amazing!!!


#11

Good to hear.

I should have caught that you had Option 82 on and something there was throwing a fit… I remember asking as I see you said it was on now. Sometimes my eyes get crossed looking at walls of information…

Thank you for reporting back.


#12

There are many schools of thought on bandwidth limiting and traffic prioritization. There is an important truth about it, though: it only works where there is a limitation. In our networks, the possibility of limitation is always over wireless connections.

For VoIP traffic to remain functional on a saturated link… it is critical to have that traffic somehow prioritized by the link itself. I have not yet found a tool that prioritizes traffic over a variable wireless link that functions outside of the link itself.

Mimosa needs to implement some sort of prioritization capability. At this point, I rely on good capacity, but when something goes wrong, it takes out everything. With my Cambium hardware, if something goes wrong (sudden interference, for example), my VoIP traffic continues to run just fine while the issue is sorted out.

Speed controls can really happen anywhere, as long as there are considerations for priority wherever there is a bottleneck.


#13

@Christopher1 I am aware that there are lots of thoughts on bandwidth limiting, lot’s of ways to skin a cat and all that, do whatever you want on your network. I am making suggestions that have proven to work well for myself when I am using Mimosa (and other) equipment to someone who is entering the WISP space. I see lots of people on these and other forums who have problems with “extra features” breaking things all the time, I don’t have many of those problems because I don’t turn on those “extra features”.

I am presented my thoughts. If you want to have unlicensed backhauls that don’t have backups or want to run links near capacity. Good for you, your ROI will be way better then mine ever will be. But I don’t like diagnosing stupid problems that occur with proprietary systems messing around with traffic in potentially unpredictable ways. (No offense to Mimosa, but we can only guess how they do the traffic prioritization). I have found that selling plain jane bandwidth with minimal stuff being done to it makes for far fewer phone calls and much less diagnostics time/money spent.

As I have said before. When you are using cheap equipment (not that Mimosa makes junk, just that it is relatively cheap compared to lot’s of other options, though not all) don’t rely on it to do lot’s of stuff. Minimize the moving parts and keep things simple. At the end of the day we are dealing with little computers strapped to radios bolted to someone’s roof, they can only do so much so it’s my opinion that it is best to rely on them doing as little as possible…


#14

I actually specifically stated, “I rely on good capacity, but when something goes wrong, it takes out everything.” What this means, is, I do not run the network at or near capacity as you’ve implied… but if a radio becomes significantly misaligned for some reason, or local interference goes crazy, the lack of good prioritization makes that bad situation worse.

You’re right, the fewer Mimosa features you use, the less likely you are to have a Mimosa based problem. [You can insert any manufacturer in there.]