These are my notes from implementing, CAKE in my network. I will be doing some updating to this page. Anything you find here should be taken with a grain of salt and verified with your own testing. I am NOT an expert.
FYI, if you just want to do fq_codel, it seems that a default simple queue is fine. Until some more questions are answered about CAKE and proper setup, I would recommend just going this route.
/queue/type/add type=fq_codel name=fq_codel
/queue/simple/add target=192.168.1.1 queue=fq_codel/fq_codel
CPU seems like it is way better then “ethernet-default”, RAM usage is significantly (25-40%) higher on a CCR1009, but that’s going from 390MB of usage to 550MB of usage and ~120 queues.
On to the CAKE stuff
If you want to use Cake, then there are a couple steps to doing so, first off you need to decide which route you can/want to go with implementation: Simple Queues or Mangle/Address Lists. There a benefits to either direction and will mostly depend on your network layout and how you deliver bandwidth to your customers. Also there is some sway depending on the hardware that you are using.
I implemented Cake with Simple Queues because that is what I will be using in my network, I point a Simple Queue at a VLAN interface and don’t care about what the IP or anything about the customer’s traffic beyond the bandwidth they are passing. This only works if you are running a VLAN to each customer or if you are doing some sort of tunneling through your network (PPPoE or whatever). There are negatives with the Simple Queue route, but that is more completely hashed out elsewhere on the internet. I am running CCR 1009s at all of my towers and even with several hundred customers they handle Simple Queues just fine, though I have not tested a couple hundred Cake Queues on a 1009 yet… If you would prefer to go the Mangle/Address List route, but don’t know how let me know and I can probably get something written up here about that.
This basically duplicates the Layer 2 Connectivity of my customers to the internet. The reality is much more complex, but I didn’t want to play around with setting up a whole wireless network and the rest of that silliness in figuring out how to play with Cake.
Note: if you are unfamiliar with Mikrotik CLI and setting up new Queues let me define each statement to give you a more clear idea of what’s happening:
/queue/type/ = this is a movement command in Mikrotik CLI which gives you access to the queue type menu.
add = we are creating a new queue type to use later on
cake-bandwidth = This is a Cake specific command for setting what bandwidth we want to limit to, default is bits per second, if you add “k” it turns it to kilobits per second or “M” will change to Mega bits per second.
cake-diffserve = Another Cake specific command, lower down I talk about the options
cake-nat = Turns on NAT Table lookups
cake-overhead-scheme = another cake specific command, lower down I talk about this
cake-rtt-scheme = again lower down I explain rtt-scheme
kind = This defines the type of queue we will be using
name = This will be the name we will use to reference this Cake Queue we are creating
/queue/simple/ = another movement command to take us to the Simple Queues menu.
name = Just like above, we put a name to our Simple Queue if you don’t give a name RouterOS will give a default name.
queue = This is where we set the Simple Queue to use our Cake queue, by default Simple Queues will bandwidth limit both packets going to and from our Target. Upload (or packets coming from our target) is set in the name we use preceding the ‘/’ and Download is set by what we put after the ‘/’.
target = Is where we define what we will be bandwidth limiting to and from. Here I am using a VLAN interface, but you can use an IP address or Subnet or any other interface in the router. Technically you can set multiple targets, but that’s another writeup.
I tested a lot of different combinations of Cake settings and Simple Queue settings, I wouldn’t say this is perfect and I will be updating this in particular when I have things more fleshed out.
/queue/type/
add cake-bandwidth=10.0Mbps cake-diffserv=besteffort cake-flowmode=triple-isolate cake-nat=yes cake-overhead-scheme=raw cake-rtt-scheme=internet kind=cake name=10Mcake cake-memlimit=30.5MiB
/queue/simple/
add name=queue1 queue=10Mcake/10Mcake target=vlan1593.ether2
Note, if you are feeling really frisky you can directly set a Queue Type to a physical interface. This could be useful if you are just looking to bandwidth everything going across a specific interface:
/queue/interface/set ether1 type=10Mcake
Keep in mind though, there is no HTB infront of CAKE so every single packet will be going through CAKE Queuing. (Which is pretty CPU intensive.)
Bandwidth Limit (Number in bps)
Pretty self-explanatory, like PCQ Queues you have to setup the speeds of the queues before you can use them. This follows the normal Mikrotik Queues speed numbers (straight number is bits per second, number followed by a ‘k’ is kilo bits per second, number followed by ‘M’ is Mbps). Leaving the box blank set’s Cake to “unlimited bandwidth”.
Autorate Ingress (On/Off)
Not really applicable to many WISPs, we are doing the bandwidth limiting. This allows for the user to automagically set their Bandwidth Limit to what it is detecting in real time. It will override Bandwidth Limit so don’t use this unless you are just trying to divvy up customers going across an unreliable backhaul link.
Automatic capacity estimation based on traffic arriving at this qdisc. This is most likely to be useful with cellular links, which tend to change quality randomly. A bandwidth parameter can be used in conjunction to specify an initial estimate. The shaper will periodically be set to a bandwidth slightly below the estimated rate. This estimator cannot estimate the bandwidth of links downstream of itself.
https://man7.org/linux/man-pages/man8/tc-cake.8.html (this is actually a really good page where I got a lot of my starting info)
This seems to be oriented towards a system like Cell phone providers who don’t really limit based on speed but usage and the speed of the link is the biggest limiting factor for the connection. Infrequent in the WISP world and replaces bandwidth.
Note: In testing with a Mikrotik RB2011 this seemed to break things and speed tests were erratic at best. In 7.1beta4
cake-atm (atm/none/ptm)
Focused on low bandwidth ADSL links, not terribly important unless you are looking at really limited traffic that needs some sort of framing on top of it. If you are using an ADSL line check out this forum thread for some tweaking.
Overhead (Number in Bytes)
Adds/Removes bytes to the packets actual calculated size. So if you are using PPPoE or VLANs you can utilize this to bandwidth limit as close to line rate as possible
MPU (Number in Bytes)
Rounds small packets up to a minimum packet size (Minimum Packet Unit?). There are some interesting technical reasons for this, mostly so that bandwidth limiting doesn’t get goofed up too much with lot’s of little packets.
This can be set with Overhead Scheme. Most Ethernet networks will use an MPU 82.
Overhead Scheme (Lots of options and you can select multiple to represent your network.)
Adds/Removes Bytes to the calculated size of each packet, this is to help with the fact that sometimes Cake can’t see the packet headers (Like VLANS or other stuff) so this way it is actually limiting on the correct bandwidth.
From my reading this is focused towards the end user. As the ISP you control the link, but I feel like this might be useful for situations where you want to account for headers and such that might limit the customer negatively, that your router won’t see. Possibly to setup your system so they see their full speeds on speed tests?
You may be able to optimize your settings a bit by playing around with the settings in here, but I didn’t find many optimizations that were much better then what I posted above.
RTT (time in ms)
How long should Cake expect packets to take to go out and come back. What is the latency of the network to the rest of the internet? This is most easily set by RTT Scheme. I would play around with this one, if you see a lot of “jumping” on your speedtest’s speeds then lowering this might help Cake limit things in situations where your latency is much lower then normal.
This adjusts how quickly Cake will act upon various flows to prevent buffers from overflowing. If you have a lot of connections that are low latency then lowering this number may help, but if you have too many connections that have more latency then it will signal to Cake that the upstream pipe is fuller then it really is and it will not do it’s job properly.
RTT Scheme: (Lots of options, the ones you care about are datacenter, internet, Metro and regional)
Datacenter sets the RTT to .1 ms (Not recommended unless you have a custom Kernal, which we have in Mikrotik)
Internet = 100 ms
metro = 10 ms
regional = 30 ms
none = allows you to manually set the RTT above.
Most users will be just fine with “internet” or maybe manually setting to something a bit lower like 50-80 ms might give you a bit of optimization if you have a little bit better than normal connectivity. If you have latencies above 100 ms (like after a lot of backhauls or a poorly performing AP then you might want to adjust this higher)
Note: if you are in a super remote place or somehow have a lot of latency on your link then you should look at the oceanic (300 ms) or satellite (1000ms) settings.
Diffserv: (besteffort, diffserv3, diffserv4, diffserv8, precedence)
https://lists.bufferbloat.net/pipermail/cake/2020-July/005352.html
Sets the number of “Tins” that Cake uses for sorting traffic. Tins hold Queues so we can’t say they are Queues, but they are bandwidth categories. “besteffort” disables this and treats all traffic as the same, diffserv3 is the default and seems to work alright, I used diffserve4 and it seemed to work alright, but then again I don’t trust that traffic is being properly marked across my network although I will probably implement QoS tagging if we ever have issues with VOIP again.
Diffserve3 and diffserve4 are how many types of traffic Cake respects, 4 somewhat correlates to WiFi’s priority types, BK, BE, VI and VO (Background, Best Effort, Video and Voice.) and 3 drops the video (Lots of packets that can have a few dropped without issue) to just have background, Best Effort and Voice.
3 is default because it “seems to work well enough” and 4 doesn’t seem to provide many more benefits because most applications don’t properly tag their traffic anyways. (Though there is a bit of push for 5 levels of prioritization, or actually slight prioritization and de-prioritization)
If your applications are actually marking their traffic then you might have a use for diffserv4, otherwise, diffserv3 should be fine.
Diffserve8 is supposed to have 8 levels of prioritization, but I wasn’t able to find much documentation about it and I am unsure if this is a Mikrotik specific or something to do with the Kernel that Mikrotik is using.
Precedence is in there for legacy reasons and the Cake guys recommend against using it. I don’t know why.
Flow Mode: (lots of options, I recommend triple isolate, but other options are flowblind, flows, dual-dsthost)
This is how Cake splits up traffic and tries to apply “fairness” to different applications that are in the same “tin” (see above with diffserv). Most traffic you will be seeing will be in the same “tin” as an ISP or at least might be incorrectly marked. So to minimize the number of phone calls you get from your customers I would recommend triple isolate, this splits up flows according to just about everything that can make them unique.
flowblind turns all flow tracking off. All traffic goes through a single queue in each tin.
flows: tracks the individual flows, but doesn’t attempt to do fairness between the flows.
dual-dsthost: Like triple isolate, but only applies fairness on the dst-host. Helps to prevent bandwidth from being monopolized based on the destination address/port.
NAT (On/Off)
https://lists.bufferbloat.net/pipermail/cake/2018-July/004009.html
Basically this allows Cake to be aware of NAT translation, if shaping is happening on the same router as there is NAT being done there isn’t much of a hit to performance, but if NAT isn’t being done then there can be significant performance hits to looking up an empty NAT table. Don’t turn this on unless you are doing NAT on the same connections you are bandwidth limiting on the same router.
Note, if you are doing NAT and don’t have this turned on you probably will have issues with Cake.
Wash
Removes most of the QoS bits from a packet when it comes into the network and leaves. If you don’t trust your Customer’s QoS settings and you probably don’t trust the QoS settings coming in from the Internet then this might not be a bad idea to try out. Probably a good idea to turn on “besteffort” on diffserv as well if you are going to be wiping out QoS bits.
Note, CAKE does not pay attention to VLAN QOS settings only packet layer QOS.
Cake-ack-filter
https://blog.cerowrt.org/post/ack_filtering/
https://lists.bufferbloat.net/pipermail/cake/2018-July/004012.html
https://lists.bufferbloat.net/pipermail/cake/2020-May/005224.html
It seems to slim down the number of “requests for ACKs from the client” useful for scenarios where there is very asymmetrical traffic (like 10-50 to 1). The idea is to not starve the upstream traffic with a bunch of ACK packets for stuff that doesn’t care all that much about lost data. (This is seriously against the TCP protocol and can break things, but in certain situations can really improve bandwidth.)
In Conservative mode Cake keeps 2 ACK packets per TCP stream in it’s queue at a time and drops redundant and extra ACKs, in Aggressive, it will drop all but 1 ACK packet per stream.
Note: Mikrotik uses a different naming scheme it seems, I assume that Mikrotik’s “filter” option matches to “Conservative” and “aggressive” is the same on both, but I do not have the resources to actually test this.
Memory Limit (Number in Bytes)
David Taht on the Mikrotik forums recommeded a pretty large number (+512KB of memory) in my testing the more memory I could throw at CAKE seemed to help things. 32MB (32M) seemed to be where I stopped seeing issues,
This is to limit the amount of memory that Cake uses for it’s queues. Cake can get pretty large pretty quick, limiting just one computer’s queues I saw 1-2 MB of memory drop when the queue was getting full. So if you have issues with your router running out of Memory then this can help.
If you used the terminal you can use a shorthand of what seems to be the normal “k,M” shorthand. More testing is going to be needed here to optimize.
Ultra High Bandwidth (+20 Gbps) Notes:
There have been multiple people who have tested Cake +20 Gbps. That said there are some limitations, it seems that Cake is limited to single threaded processing, hopefully each Queue can be it’s own thread in which case 1009 and especially 1072’s will crush this system easily.
With a CCR1009 I was seeing different Simple Queues were being taken care of by separate cores, with ~300 different flows I was able to push ~1.2 Gbps through the 1009, but that was in a pretty artificial environment, again YMMV.
Further reading:
I may not have directly reference some stuff, but here are some other useful links if you feel like really diving into Cake:
https://arxiv.org/pdf/1804.07617.pdf Piece of CAKE: A Comprehensive QueueManagement Solution for Home Gateways
https://lwn.net/Articles/758353/ Let them run CAKE (LWN.net)
https://gettys.wordpress.com/2018/02/11/the-blind-men-and-the-elephant/ (More about FQ_CoDel, but Cake is based on that so… good to know parentage.)
https://forum.mikrotik.com/viewtopic.php?t=179307 David Taht giving some directions and comments about Cake and fq_codel and testing advise. VERY LONG forum thread.