Proposed BOSH logging interface

Jesse T. Alford

Hello! We're the CF Platform Logging team. We maintain `syslog-release` and have been working to improve and regularize platform logging behavior.

This is a proposal intended to establish reasonable expectations about what should be logged and what should be forwarded in bosh-deployed cloud systems.

Historically, it has been up to each release to provide for their log forwarding, if any. We intend `syslog-release` to provide a consistent interface useful enough to replace all other provisions for the forwarding of logs from bosh jobs.

## Proposed Interface
If log forwarding is enabled, some files in `/var/vcap/sys/log` (and its subdirectories, recursively), will have any line written to them forwarded as the MSG portion of an RFC5424 compliant syslog message. Which files are forwarded is governed first by file extension, and secondarily by file permissions.

`syslog-release` attempts to read any file ending in `.log`.
(This allows us to avoid forwarding rotated logs, swapfiles, etc.)
It will forward from such files if either of the following are true:
- it is world-readable
- it is readable to the `vcap` group

In particular, this means that logs will not be forwarded from files where:
- user and group are root:root
- user and group are vcap:root or vcap:none
- user and group are vcap:vcap, but it is not group-readable

…unless they are world-readable.

We think that this interface will allow us to avoid running a log forwarder with elevated permissions, while also allowing jobs to, for instance, write DEBUG or similar logs to a file that is not group-readable, thus improving their security and reducing the load on the logging system while still making them available on the ephemeral disk for debugging purposes.

## Questions
There are a couple of things around this interface we're especially interested in feedback on, in addition to the obvious "will this be a problem for you" overall question.

We may have to have a proviso that the depth of this is not unlimited. This depends somewhat on what is inexpensive to implement and maintain, and is an area we'd appreciate feedback on. Is three levels deep from `/var/vcap/sys/log` (i.e. `/var/vcap/sys/log/jobname/processname/*`) enough? Would four be?

In the old way of doing things, more control over the PRI information and other syslog fields was available to release authors. Logs forwarded from files currently all come out as PRI 14, which translates to Facility: User, Severity: Info. Additionally, the appname/tag field is set to the name of the directory containing the log file. Is this enough/good info? If we were to include the filename, too, would that be useful? Sufficient?

## Testing with the Proposed Interface
We have recently implemented a feature to help release authors evaluate the proposed interface. If you set `syslog.respect_file_permissions: true`, blackbox will not be run with elevated capabilities, and you'll be able to see what is and isn't forwarded under the proposed interface.

Join to automatically receive all group messages.