Re: ERR Failed to stage application: insufficient resources


Matthew Sykes <matthew.sykes@...>
 

There are three resources that you can run out of that would result in that
error: memory, disk, or containers. [1] These resources are validated
during the auction that determines where a task or LRP will land.

Advertised memory and disk resources are configurable [2] in diego and
container limits are defined in garden [3].

If you don't specify memory or disk for diego, the actual system capacity
is advertised for placement without any overcommit. That means if you're
running 8 containers with a 2048M limit on a 16GB cell, you're out of
memory capacity - regardless of how much memory is actually used.

Many deployments can get away with 2 to 3x overcommit on memory and disk
but it really depends on the kinds of apps that are being deployed.

[1]:
https://github.com/cloudfoundry-incubator/rep/blob/320f5e9ff0ba2a7bb1294927cbe31ce5af40c987/resources.go#L63-L68
[2]:
https://github.com/cloudfoundry-incubator/diego-release/blob/develop/jobs/rep/spec#L55-L60
[3]:
https://github.com/cloudfoundry-incubator/garden-linux-release/blob/develop/jobs/garden/spec#L65-L67

On Thu, Jan 28, 2016 at 11:52 AM, Stanley Shen <meteorping(a)gmail.com> wrote:

when I push an application to CF with diego deployed, the app failed for
error message

ERR Failed to stage application: insufficient resources

The app asks for:
disk_quota: 2048M
memory: 4072M
instances: 1

The Runner VM is c3.2xlarge, which has 8vCPU and 15G memory.
The resource usage of runner VM is:

Filesystem Size Used Avail Use% Mounted on
udev 7.4G 4.0K 7.4G 1% /dev
tmpfs 1.5G 352K 1.5G 1% /run
/dev/xvda1 2.9G 1.3G 1.6G 45% /
none 4.0K 0 4.0K 0% /sys/fs/cgroup
none 5.0M 0 5.0M 0% /run/lock
none 7.4G 0 7.4G 0% /run/shm
none 100M 0 100M 0% /run/user
/dev/xvdb2 65G 3.0G 58G 5% /var/vcap/data
tmpfs 1.0M 20K 1004K 2% /var/vcap/data/sys/run
/dev/loop0 120M 1.6M 115M 2% /tmp
none 7.4G 0 7.4G 0% /tmp/warden/cgroup

%Cpu(s): 0.0 us, 0.0 sy, 0.0 ni,100.0 id, 0.0 wa, 0.0 hi, 0.0 si,
0.0 st
KiB Mem: 15399972 total, 3539764 used, 11860208 free, 19664 buffers
KiB Swap: 15406328 total, 0 used, 15406328 free. 3106884 cached Mem

I didn't find useful information message related to this issue under
/var/vcap/sys/log.

What could be the reason, and what should I change for fixing it?
How CF determine the resource?
--
Matthew Sykes
matthew.sykes(a)gmail.com

Join cf-dev@lists.cloudfoundry.org to automatically receive all group messages.