Hi,

We are using the `syslog_deamon_config` proprty to stream all of our CF infrastructure logs to an external stack for later processing.

We have noticed that operations like `cf create-user-provided-service`, `cf bind-service`, and others are logged by multiple components in CF. That would normally not be a problem, except that these commands often involve passwords and those passwords get logged as well, ending up in the log files on the VM and the target log processing stack, which allows operators of the system to view end-user passwords.

We have noticed that the following jobs are responsible for the logs:

* api
* runner
* nats

Increasing the log level from the default `debug` / `debug2` to `info` solves the problem for the first two, at the cost of making troubleshooting tasks more difficult on the system.
The last one can only be solved by removing the `nats_stream_forwarder` component from the `nats` job, again making troubleshooting more difficult.

I believe the ideal solution is to have those components not log the payload of commands holding confidential information. Maybe they could replace it with some pattern.
This would help for the first two but might not help for nats, where some other means would be needed (encryption of the private content?).

How are you solving this issue on your productive system? What are your thoughts on this matter?

Thanks in advance!

Regards,
Momchil Atanassov

Hey Momchil,

Do you know whether it's the DEA or Warden that's logging that sensitive
data when you say "runner"?

I would recommended opening issues against the relevant projects:

API: https://github.com/cloudfoundry/cloud_controller_ng/issues
DEA or Warden: https://github.com/cloudfoundry/dea_ng/issues or
https://github.com/cloudfoundry/warden/issues

As for NATS, you may be able to change the logging level? Alternatively,
NATS is not a Cloud Foundry project but you could ask over there about
encrypting log output: https://github.com/nats-io/gnatsd

In Pivotal's production environments, we run 100% on Diego, so we are not
concerned with DEA/Warden logging, and this move also removes NATS from the
flows like create-user-provided-service. CC is likely still an issue, so
it would be a good one to raise against their GitHub project.

Best,
Amit

Hi Amit,

Thanks for the quick reply!

We get logs from both DEA and Warden.

As for NATS, it's not NATS itself that is logging but rather the `nats_steam_forwarder` ( https://github.com/cloudfoundry/cf-release/tree/master/jobs/nats_stream_forwarder ) job running on the `nats` VMs, as can be seen here: https://github.com/cloudfoundry/cf-release/blob/master/jobs/nats_stream_forwarder/templates/nats_stream_forwarder.rb
It always logs in `info` level so the only way to disable it is to remove it from the deployment configuration so that it is never located on the NATS VM and never runs.

You say that you are running 100% Diego on your productive environment. Doesn't Garden also do some type of logging that would contain the container configuraiton (including environment variables) or are you not running `debug` on your systems?

Regards,
Momchil

Hi, Momchil,

We at Pivotal generally don't run with debug logging on for Diego and
Garden in our production systems, instead opting for the default 'info'
log-level. It is possible to toggle that log-level dynamically on
garden-linux via its debug server (which defaults to running on port 17013)
by doing a PUT to its `/log-level` endpoint with the payload 'debug',
though. Even so, the Garden team has scrubbed a lot of user-specified
information such as environment variables and command arguments out of
their logs, even at the debug level. The most relevant stories appear to be
https://www.pivotaltracker.com/story/show/101689730,
https://www.pivotaltracker.com/story/show/101874066, and
https://www.pivotaltracker.com/story/show/102666020, but if you need more
details I'm sure the Garden team can direct you to other stories as well.

Thanks,
Eric

On Tue, Dec 8, 2015 at 12:45 AM, Momchil Atanassov <
momchil.atanassov(a)sap.com> wrote:

Hi Amit,

Thanks for the quick reply!

We get logs from both DEA and Warden.

As for NATS, it's not NATS itself that is logging but rather the
`nats_steam_forwarder` (
https://github.com/cloudfoundry/cf-release/tree/master/jobs/nats_stream_forwarder
) job running on the `nats` VMs, as can be seen here:
https://github.com/cloudfoundry/cf-release/blob/master/jobs/nats_stream_forwarder/templates/nats_stream_forwarder.rb
It always logs in `info` level so the only way to disable it is to remove
it from the deployment configuration so that it is never located on the
NATS VM and never runs.

You say that you are running 100% Diego on your productive environment.
Doesn't Garden also do some type of logging that would contain the
container configuraiton (including environment variables) or are you not
running `debug` on your systems?

Regards,
Momchil