Hi John,

I had been thinking about this a bit more, and I think it would be
reasonable to consider adding an end point that is similar to the /stats
end point, that queries Diego, to get actual state on environment variables
and as part of that VCAP_SERVICES for the running app instances.
Thoughts on that approach? Would that help address the problem?

-Dieu

On Wed, Nov 4, 2015 at 9:03 AM, john mcteague <john.mcteague(a)gmail.com
<javascript:_e(%7B%7D,'cvml','john.mcteague(a)gmail.com');>> wrote:

I had this conversation with a few different people during the berlin
summit and promised one of them I would repeat it on the mailing list today
to get further feedback.

Today, once we push an application, the droplet is immutable. It doesnt
change until you push the application again or restage. I believe the
entire container could change without a new push if you upgrade the rootfs
and restart all the apps (which the CF operator would do).

However, the environment vars and service bindings can be changed on an
application but they would not take affect until the next restart.The CF
API would report these changes as active when you run *cf env *or *cf
services. *There is no distinction between desired state and current
state when using the API.

To me this is a significant gap as we cannot necessarily get a true view
of the world (i call cf set-env but dont restart the app, how do I know
from the API what value of that env var my app is using).

How are people addressing this in their own environments and is it
something that the core API team should be considering (I ask the latter
publicly even though I asked Dieu during the summit :) ).

John

John,

I'd like to understand more about what you are hoping to accomplish.

Are you trying to detect gaps where a new env var has been pushed, but the
app has not been restarted? Is this for monitoring/compliance or debugging
purposes?

It seems like Apps these days get configuration from a pretty wide variety
of sources, env variables being only one. Apps also ship with defaults,
UNIX defaults (max filedes, semaphores, etc), and apps sometimes adjust
their settings based on values in SQL or Redis*, etc.

Each app cares about these settings differently, some are sensitive to RAM
available, but not number of open files, etc.

I only ask because it seems like asking for "effective environment" vs
"actual environment" is only a subset of configurations.

The infrastructure can only express what has been provided to the app, but
for any number of reasons, may not represent the current actual running
state of the app.

I understand that it means more work for the app dev, but I often
implement a quick "/config" endpoint in my apps. This allows me to verify
what configuration the App is actually using at a given moment.

It's a more comprehensive way of validating that all "intended
configuration," is in fact actual configuration.

--
Marco Nicosia
Product Manager
Pivotal Cloud Foundry

* A quick example: the NOC can use an admin endpoint to toggle on/off an
experimental feature, which is detected via Redis, etc.

On Monday, November 9, 2015, Dieu Cao <dcao(a)pivotal.io> wrote:

Hi John,

I had been thinking about this a bit more, and I think it would be
reasonable to consider adding an end point that is similar to the /stats
end point, that queries Diego, to get actual state on environment variables
and as part of that VCAP_SERVICES for the running app instances.
Thoughts on that approach? Would that help address the problem?

-Dieu

On Wed, Nov 4, 2015 at 9:03 AM, john mcteague <john.mcteague(a)gmail.com>
wrote:

I had this conversation with a few different people during the berlin
summit and promised one of them I would repeat it on the mailing list today
to get further feedback.

Today, once we push an application, the droplet is immutable. It doesnt
change until you push the application again or restage. I believe the
entire container could change without a new push if you upgrade the rootfs
and restart all the apps (which the CF operator would do).

However, the environment vars and service bindings can be changed on an
application but they would not take affect until the next restart.The CF
API would report these changes as active when you run *cf env *or *cf
services. *There is no distinction between desired state and current
state when using the API.

To me this is a significant gap as we cannot necessarily get a true view
of the world (i call cf set-env but dont restart the app, how do I know
from the API what value of that env var my app is using).

How are people addressing this in their own environments and is it
something that the core API team should be considering (I ask the latter
publicly even though I asked Dieu during the summit :) ).

John

--
--
Marco Nicosia
Product Manager
Pivotal Software, Inc.
mnicosia(a)pivotal.io
c: 650-796-2948

Hi John, thanks for getting back to me.

It seems inconvenient for you to have to monitor divergence of
CF-initiated configuration changes from the Apps' runtime environment.

I'm not PM for these projects, so I get to ask leading questions, but
obviously I can't make any commitments. If you have a question about MySQL,
I'm your guy. :)

It seems like the two areas of exposure are `cf set-env` and `cf
bind-service`. Would it make more sense that, in your environment, any time
either of those are executed the app is _automatically_ restarted (if and
only if the app is already running)? If it turns out to be the case that we
make several, simultaneous changes to environment and bindings, we could
look into batching them together, so there isn't a serial string of rolling
restarts.

That way you wouldn't have to build any additional auditing capabilities;
it theoretically wouldn't be possible for the environment, buildpacks
settings, and service configuration to diverge from what the app is
actively leveraging?

--
Marco Nicosia
Product Manager
Pivotal Cloud Foundry



On Tue, Nov 10, 2015 at 2:13 PM, john mcteague <john.mcteague(a)gmail.com>
wrote:

Marco, So I understand that configuration for an app is mutli-faceted and
not the complete view of what is happening. My goal is to understand what
is running at any moment in time from the information the API provides me.

Buildpack's can be heavily influenced by env var changes. Developers can
affect the version of java or the JVM args via env vars, but it is not
possible via the API to be sure from the variables presented whether they
are those that actually took effect at startup/restage. This causes an
audit concern.

Where an app adjusts its settings based on Redis being bound for example,
that is auditable by the fact we wrote the app and know what is due to
happen upon binding. But because binding can occur on a running app and
that is not guaranteed to be effective immediately, from the API output,
the developer may assume the application is running with that Redis
specific configuration. But that is only true if the app was restarted.

CF cant control every aspect of the apps configuration, but where it is
the method of configuration (bindings, env vars), I feel its important that
appropriate distinctions between actual and desired state are made.

On Tue, Nov 10, 2015 at 5:08 PM, Marco Nicosia <mnicosia(a)pivotal.io>
wrote:

John,

I'd like to understand more about what you are hoping to accomplish.

Are you trying to detect gaps where a new env var has been pushed, but
the app has not been restarted? Is this for monitoring/compliance or
debugging purposes?

It seems like Apps these days get configuration from a pretty wide
variety of sources, env variables being only one. Apps also ship with
defaults, UNIX defaults (max filedes, semaphores, etc), and apps sometimes
adjust their settings based on values in SQL or Redis*, etc.

Each app cares about these settings differently, some are sensitive to
RAM available, but not number of open files, etc.

I only ask because it seems like asking for "effective environment" vs
"actual environment" is only a subset of configurations.

The infrastructure can only express what has been provided to the app,
but for any number of reasons, may not represent the current actual running
state of the app.

I understand that it means more work for the app dev, but I often
implement a quick "/config" endpoint in my apps. This allows me to verify
what configuration the App is actually using at a given moment.

It's a more comprehensive way of validating that all "intended
configuration," is in fact actual configuration.

--
Marco Nicosia
Product Manager
Pivotal Cloud Foundry

* A quick example: the NOC can use an admin endpoint to toggle on/off an
experimental feature, which is detected via Redis, etc.

On Monday, November 9, 2015, Dieu Cao <dcao(a)pivotal.io> wrote:

Hi John,

I had been thinking about this a bit more, and I think it would be
reasonable to consider adding an end point that is similar to the /stats
end point, that queries Diego, to get actual state on environment variables
and as part of that VCAP_SERVICES for the running app instances.
Thoughts on that approach? Would that help address the problem?

-Dieu

On Wed, Nov 4, 2015 at 9:03 AM, john mcteague <john.mcteague(a)gmail.com>
wrote:

I had this conversation with a few different people during the berlin
summit and promised one of them I would repeat it on the mailing list today
to get further feedback.

Today, once we push an application, the droplet is immutable. It
doesnt change until you push the application again or restage. I believe
the entire container could change without a new push if you upgrade the
rootfs and restart all the apps (which the CF operator would do).

However, the environment vars and service bindings can be changed on
an application but they would not take affect until the next restart.The
CF API would report these changes as active when you run *cf env *or *cf
services. *There is no distinction between desired state and current
state when using the API.

To me this is a significant gap as we cannot necessarily get a true
view of the world (i call cf set-env but dont restart the app, how do I
know from the API what value of that env var my app is using).

How are people addressing this in their own environments and is it
something that the core API team should be considering (I ask the latter
publicly even though I asked Dieu during the summit :) ).

John

--
--
Marco Nicosia
Product Manager
Pivotal Software, Inc.
mnicosia(a)pivotal.io
c: 650-796-2948