Re: [CAUTION] Re: [cf-dev] Proposed BOSH logging interface
|
Marco Voelz
Dear Jesse,
did anything come out of this proposal? Did you end up picking up this track of work?
Warm regards Marco
From: <cf-dev@...> on behalf of Marco Voelz <marco.voelz@...>
Dear Jesse,
Thanks for putting this proposal out there. We would be happy to see an automated logfile forwarding mechanism. Here's a couple of comments on your initial points: * Including the filename in the syslog metadata is very useful and something we'd really like to have. Currently it is something we're working around a bit. * The appname/tag field should probably contain the release's name as well as a prefix. My proposal here is `<deployment name>.<instance group name>.<job name>`. wdyt? * We haven't made any particular use of the priority field, so losing control over this field wouldn't matter for out use-cases. Severity is usually something that the actual log message needs to contain, as the logger's severity can only be set on its initial creation, afaik. * Restricting the depth of recursion seems reasonable. So far, I don't think we're using bosh releases which have more than 1 folder below their /var/vcap/sys/log/<job name>/ folder.
Concerning the requirements about permissions on the logfiles you'd want to forward: Did you talk to Dmitriy/the BOSH team about this? With stemcell series 3541.x the permissions on the standard folders below /var/vcap were tightened a bit, so just wanted to make sure that your assumptions are in line with the upcoming changes in the stemcells.
Warm regards Marco From: cf-dev@... <cf-dev@...> on behalf of Jesse T. Alford <jalford@...>
Sent: Tuesday, April 3, 2018 12:55:38 AM To: cf-dev@... Subject: [cf-dev] Proposed BOSH logging interface
Hello! We're the CF Platform Logging team. We maintain `syslog-release` and have been working to improve and regularize platform logging behavior.
This is a proposal intended to establish reasonable expectations about what should be logged and what should be forwarded in bosh-deployed cloud systems.
Historically, it has been up to each release to provide for their log forwarding, if any. We intend `syslog-release` to provide a consistent interface useful enough to replace all other provisions for the forwarding of logs from bosh jobs.
## Proposed Interface If log forwarding is enabled, some files in `/var/vcap/sys/log` (and its subdirectories, recursively), will have any line written to them forwarded as the MSG portion of an RFC5424 compliant syslog message. Which files are forwarded is governed first by file extension, and secondarily by file permissions.
`syslog-release` attempts to read any file ending in `.log`. (This allows us to avoid forwarding rotated logs, swapfiles, etc.) It will forward from such files if either of the following are true: - it is world-readable - it is readable to the `vcap` group
In particular, this means that logs will not be forwarded from files where: - user and group are root:root - user and group are vcap:root or vcap:none - user and group are vcap:vcap, but it is not group-readable
…unless they are world-readable.
We think that this interface will allow us to avoid running a log forwarder with elevated permissions, while also allowing jobs to, for instance, write DEBUG or similar logs to a file that is not group-readable, thus improving their security and reducing the load on the logging system while still making them available on the ephemeral disk for debugging purposes.
## Questions There are a couple of things around this interface we're especially interested in feedback on, in addition to the obvious "will this be a problem for you" overall question.
We may have to have a proviso that the depth of this is not unlimited. This depends somewhat on what is inexpensive to implement and maintain, and is an area we'd appreciate feedback on. Is three levels deep from `/var/vcap/sys/log` (i.e. `/var/vcap/sys/log/jobname/processname/*`) enough? Would four be?
In the old way of doing things, more control over the PRI information and other syslog fields was available to release authors. Logs forwarded from files currently all come out as PRI 14, which translates to Facility: User, Severity: Info. Additionally, the appname/tag field is set to the name of the directory containing the log file. Is this enough/good info? If we were to include the filename, too, would that be useful? Sufficient?
## Testing with the Proposed Interface We have recently implemented a feature to help release authors evaluate the proposed interface. If you set `syslog.respect_file_permissions: true`, blackbox will not be run with elevated capabilities, and you'll be able to see what is and isn't forwarded under the proposed interface. |
|
|