Garden does have (currently not very well documented) support for setting hard cpu maximums as well as the fair share limits described above. This is off by default but can be enabled via the 'garden.cpu_quota_per_share_in_us' bosh property (more details at https://github.com/cloudfoundry/garden-runc-release/releases/tag/v1.3.0). This is a first pass at solving this so we're very interested in feedback for it and any use cases it still doesn't solve.
toggle quoted message
Show quoted text
On Tue, 18 Jul 2017 at 14:07, Daniel Mikusa <dmikusa(a)pivotal.io> wrote: John,
Further down the stack, Garden uses CPU cgroups to achieve CPU resource management. CPU intensive applications can not affect other applications in the same space, org or cell.
In the case of apps on the same cell, I would say that one CPU intensive application can affect other applications on the same cell, but it cannot completely starve out the other applications on the cell.
Each application on CF is given a number of CPU shares (I believe this is proportional to the memory limit for the application, i.e. more memory, more CPU shares) and, while this might be an over simplification, the shares essentially guarantee you'll receive a minimum amount of CPU time. They don't prevent you from getting more CPU time, but it prevents other processes from starving out a process.
Ex 1: You have one Cell with two apps running on it. The apps have the same memory limit, thus the same CPU shares. - If App A is doing nothing, App B can make *full* use of the CPUs - If App B is doing nothing, App A can make *full* use of the CPUs - if App A & App B are running very CPU intensive workloads, they should each roughly get half of the CPU resources.
It is technically possible for App A to affect App B's CPU time because both are guaranteed their shares, however, it's not possible for App A to starve App B and prevent it from getting any CPU time. The cgroups configuration guarantees this an app a minimum amount of time.
Where you have to be careful is when you have applications of varying memory limits (and thus varying CPU shares). The example above is simple because they both have the same limits. If you have two applications with wildly different memory limits, you might see issues.
Ex 2: You have one Cell with two apps running on it. App A is very small and has a 64M memory limit, App B is very large and has a 8G memory limit. - Again, if App A is doing nothing, App B can make *full* use of the CPUs - Again, if App B is doing nothing, App A can make *full* use of the CPUs - If App A & App B are running very CPU intensive workloads, they will *not* receive equal time on the CPU. App A will basically get 1/128th the CPU time of App B because of the way CF configures the shares.
Despite this being a fair (i.e. App A is getting what it asked), App A will undoubtedly be perceived as running slow by its users since it's getting significantly less access to the CPU while App B is running its intensive workload. Again, App B cannot completely starve App A, but in this case because of the CPU shares, it can definitely impact App A a lot more.
If you find yourself in this situation, you can try restarting your app, which will perhaps cause it to be run on a different Cell where CPU is not in contention or you can bump up your memory limit, which will give you more CPU shares.
Hope that helps!
Dan
PS. George please feel free to correct me if I've misstated anything.
On Tue, Jul 18, 2017 at 5:26 AM, John jerrby <jjerrby(a)gmail.com> wrote:
Hi George,
Thanks! 1. So who "pay" :) when the application in CPU intensive ? if not other app in the cell ... 2. if there any documentation which can shad some additional lights on this topic it it will be great if you can share it...
Thanks a lot and have a nice day,
John
|