Scary incident with Mikrotik CCR

A few days ago, I encountered a frightening incident. When I opened the routing > BGP > advertisements menu and selected just one of the peers to display, all our peers reset their sessions.

On another day, I attempted to access it via the terminal using the command /routing BGP advertisements print, and upon entering it, the system froze, all our BGP peers reset their sessions again. It took approximately 31 minutes for BGP route propagation to stable. There were no logs displayed, just the unsettling feeling that our NOC team should avoid that menu in the future, despite its importance to us.

Our CCR1036-12G-4G specifications are currently operating on a dual stack ipv4/ipv6 with 4 overseas peers (+- 900k x 3) & 4 domestic peers (prefix: -+ 10-20k x 5 peers). Since 2022 (version 6.49.7) until this period in 2023, there have been no changes in BGP patches according to the historical changelogs on the Mikrotik Website.

There is no need to elaborate on client complaints following this brief incident. I attempted to email Mikrotik support, hoping they could provide some advice and guidance on these specifications for our convenience and that of our customers. I received the following brief response:

"It may happen on ROSv6 when other tasks like fetching large tables for monitoring (e.g. open routing table in Winbox or advertisements in Winbox) router will not be able to process keepalives while fetching and drop the sessions."

Now, we are taking the safer approach by using SSH to execute changes instead of Winbox, although this is only a temporary solution. This reflects the ongoing challenges we face with Mikrotik.

The issues related to BGP that emerge in this type of Mikrotik CCR are, in fact, nothing new.

"Problem with ccr is slow single core performance. With BGP one cpu is 100% all the time. You will see slow routing table updates. I see seldom watchdog reboots causing a big network interuption. MT is working on this but it may take a long time until this is stable. Forwarding performance is fine as this is done on more cores. I consider going VMware with CHR or vyos." source, forum.mikrotik

The story of the experience written in the Mikrotik forum is the same as my experience since dealing with the Mikrotik CCR type.

Personal blog by A Rahman.
Menulis untuk mengingatnya.