B5 Rebooting "ethernet link emac rx dma stuck"

Upgraded this link to 1.5.1 a few days ago. There was a power last night and when the power came back this B5 rebooted every three minutes for three hours with the following in the log file over and over again.

BTW: Startup reason says ‘user reboot’ but we didn’t touch it until this AM when I logged in and saw the alarms.

2018-08-06 09:27:40 (UTC +0000) root: MIMO_EVENT binaryversion=90afb59714e83b80041e9bf58c3ba9765029cc29 build_time=2018-06-06 23:01:55 (UTC +0000) bamboo_build_time_stamp=2018-06-06T23:02:19.832Z bamboo_build_number=19 project=FWREL-REL5 branch=release/1.5.1
2018-08-06 09:27:47 (UTC +0000) root: MIMO_EVENT ethernet vendor id: 0x243d91.
2018-08-06 09:27:47 (UTC +0000) root: MIMO_EVENT Shutdown reason: Reboot by a SW User Process - Ethernet inactivity
2018-08-06 09:27:49 (UTC +0000) root: MIMO_EVENT Startup reason: User Reboot
2018-08-06 09:28:41 (UTC +0000) root: MIMO_EVENT ethernet link emac rx dma stuck
2017-01-01 00:00:02 (UTC +0000) root: MIMO_EVENT no GPS signal for 3 seconds
2018-08-06 09:29:34 (UTC +0000) root: MIMO_EVENT GPS signal recovered
2018-08-06 09:29:46 (UTC +0000) root: MIMO_EVENT Wireless associated
2018-08-06 09:30:01 (UTC +0000) root: MIMO_EVENT ethernet link becomes ready
2018-08-06 09:30:04 (UTC +0000) root: MIMO_EVENT ethernet link down
2018-08-06 09:30:06 (UTC +0000) root: MIMO_EVENT ethernet link up (100/Full)
2018-08-06 09:30:09 (UTC +0000) root: MIMO_EVENT binaryversion=90afb59714e83b80041e9bf58c3ba9765029cc29 build_time=2018-06-06 23:01:55 (UTC +0000) bamboo_build_time_stamp=2018-06-06T23:02:19.832Z bamboo_build_number=19 project=FWREL-REL5 branch=release/1.5.1
2018-08-06 09:30:17 (UTC +0000) root: MIMO_EVENT ethernet vendor id: 0x243d91.
2018-08-06 09:30:17 (UTC +0000) root: MIMO_EVENT Shutdown reason: Reboot by a SW User Process - Ethernet inactivity
2018-08-06 09:30:19 (UTC +0000) root: MIMO_EVENT Startup reason: User Reboot
2018-08-06 09:30:58 (UTC +0000) root: MIMO_EVENT ethernet link emac rx dma stuck
2017-01-01 00:00:03 (UTC +0000) root: MIMO_EVENT no GPS signal for 3 seconds
2018-08-06 09:31:51 (UTC +0000) root: MIMO_EVENT GPS signal recovered
2018-08-06 09:32:04 (UTC +0000) root: MIMO_EVENT Wireless associated
2018-08-06 09:32:17 (UTC +0000) root: MIMO_EVENT ethernet link is not ready
2018-08-06 09:32:20 (UTC +0000) root: MIMO_EVENT ethernet link up (100/Full)
2018-08-06 09:32:23 (UTC +0000) root: MIMO_EVENT ethernet link down
2018-08-06 09:32:23 (UTC +0000) root: MIMO_EVENT binaryversion=90afb59714e83b80041e9bf58c3ba9765029cc29 build_time=2018-06-06 23:01:55 (UTC +0000) bamboo_build_time_stamp=2018-06-06T23:02:19.832Z bamboo_build_number=19 project=FWREL-REL5 branch=release/1.5.1
2018-08-06 09:32:25 (UTC +0000) root: MIMO_EVENT ethernet link up (100/Full)
2018-08-06 09:32:30 (UTC +0000) root: MIMO_EVENT ethernet vendor id: 0x243d91.
2018-08-06 09:32:30 (UTC +0000) root: MIMO_EVENT Shutdown reason: Reboot by a SW User Process - Ethernet inactivity
2018-08-06 09:32:32 (UTC +0000) root: MIMO_EVENT Startup reason: User Reboot
2018-08-06 09:33:21 (UTC +0000) root: MIMO_EVENT ethernet link emac rx dma stuck
2017-01-01 00:00:01 (UTC +0000) root: MIMO_EVENT no GPS signal for 3 seconds
2018-08-06 09:34:14 (UTC +0000) root: MIMO_EVENT GPS signal recovered
2018-08-06 09:34:38 (UTC +0000) root: MIMO_EVENT ethernet link is not ready
2018-08-06 09:34:42 (UTC +0000) root: MIMO_EVENT Wireless associated
2018-08-06 09:34:46 (UTC +0000) root: MIMO_EVENT ethernet link up (100/Full)
2018-08-06 09:34:49 (UTC +0000) root: MIMO_EVENT binaryversion=90afb59714e83b80041e9bf58c3ba9765029cc29 build_time=2018-06-06 23:01:55 (UTC +0000) bamboo_build_time_stamp=2018-06-06T23:02:19.832Z bamboo_build_number=19 project=FWREL-REL5 branch=release/1.5.1

Ouch, can’t say I remember seeing something like this. Have you contacted Mimosa Support? If they find the issue, please let the rest of us know what the solution is!

Hi @ian1,

This is very interesting and also very bad. I haven’t seen anything like this, but I am creating you a case right now. Hang tight and I’ll send you an email with more info!

2 Likes

Thanks Dustin - sent the info in email as requested.

Ian

Please do let the rest of us know what you all find.

I do know they are looking at it. I’m behind on this and a few other things because I’m out on a customer site. :frowning: Will let you guys know more soon.

1 Like

Very grateful if someone can provide an update on this.

Almost the exact situation for me; problem came around right after power loss/return.

“ethernet link emac rx dma stuck”

Production link… B5 has only been in production for two months; this is not good :frowning:

Hi @Renaldo,

I’m contacting you momentarily via email about this.

1 Like

@DustinS thank you for the quick response via email.

Writing to keep others updated on this issue.


The link is stable again. Here are the series of events from the top.

  • Power Loss

  • UPS kicked in.

  • Generator should have powered up and taken the load but did not (this is being looked into).

  • UPS died.

  • Power was restored and the Mimosa link started giving errors and rebooting every 3-5 minutes.

  • I visited the site and everything seemed to be fine except the Ethernet port would occasionally flap (possibly during the mimosa boot sequences?).

  • I disconnected the mimosa PoE block (original unit) and waited for the power to drain… seems it took longer than usual for the blue light to die; 5 minutes?

  • Reconnected PoE but only to the B5; not to the router. The device stayed up without rebooting this time.

  • I eventually connected the router again maybe 15 minutes after recognizing the long session without reboot.

  • Router is now connected but only negotiated at 100mb.

If you had a power surge, it might have tripped something in the PoE. Do you have another you could try?

1 Like

Will be taking another PoE on location tomorrow.

Also, note that the PoE was on a surge protector/battery backup.

Standby for an update.

No luck.

Same thing happens with a PoE unit replacement.

While the Mimosa radio is going through the infinite boot loop, here is what the Mikrotik logs look like (after each radio reboot):

Jan 27 19:54:24 interface: ,info [ROUTER_NAME]: ether2 link up (speed 1G, full duplex)
Jan 27 19:54:25 interface: ,info [ROUTER_NAME]: ether2 link down
Jan 27 19:54:27 interface: ,info [ROUTER_NAME]: ether2 link up (speed 1G, full duplex)
Jan 27 19:54:28 interface: ,info [ROUTER_NAME]: ether2 link down
Jan 27 19:54:30 interface: ,info [ROUTER_NAME]: ether2 link up (speed 1G, full duplex)
Jan 27 19:54:31 interface: ,info [ROUTER_NAME]: ether2 link down
Jan 27 19:54:33 interface: ,info [ROUTER_NAME]: ether2 link up (speed 1G, full duplex)
Jan 27 19:54:34 interface: ,info [ROUTER_NAME]: ether2 link down
Jan 27 19:54:36 interface: ,info [ROUTER_NAME]: ether2 link up (speed 1G, full duplex)
Jan 27 19:54:37 interface: ,info [ROUTER_NAME]: ether2 link down
Jan 27 19:54:39 interface: ,info [ROUTER_NAME]: ether2 link up (speed 100M, full duplex)

Even though that last line states that the Ethernet links is up at 100M, router is still unable to ping the Mimosa over the ethernet link unless I reboot the PoE without the router connected (powering up the radio only), wait a minute or two for the link to come up THEN connect the Mikrotik router.

Is there a solution to this problem? Yesterday a B5 went down due to power outage, when the power was restored the radio was stuck in a reboot loop. I removed the radio from service and left it powered off all night. This morning I plugged it in and was able to get to the log to find this “ethernet link emac rx dma stuck”.

Hi @Jeffrey,

We have 1.5.2 beta 2 which addresses some of these issues. If you’d like the firmware, please email me directly at dustin@mimosa.co.