We're currently using ansible to execute a number of BOSH operations for
deploy and manage of cloud foundry. Is there any better option than using
shell scripting to interact with BOSH? Being able to look at BOSH data such
as deployment details and vm information would be much easier if we didn't
have to try to get at the data with cut grep and awk or trying to parse the
generated deployment manifest.

Can you provide a bit more detail what kind of operations you are running?

We're using Ansible to execute BOSH commands such as uploading stemcells
and releases, doing deployments of manifests for fresh deploys or updates
etc. Basically orchestrating the commands a human might run when dealing
with BOSH managed environments using the bosh_cli.

I'm specifically interested in being able introspect the data BOSH knows
about the environment such as getting the detailed information available
for virtual machines in a deployment, or releases and stemcells managed by
the director. For example I may want to find out what machines nats is
running on in my CF deployment and determine their internal ip address, or
correlate a virtual machine belonging to a job to the physical resource it
lives on in OpenStack to understand impact of doing a maintenance. Being
able to interact with the data available in the directory via API calls
instead of trying to parse it out of BOSH commands or reverse engineer it
in some other way would be quite useful.

You could use jq to cleanly parse and pull out data from bosh API in shell scripts.




To inspect bosh yaml manifests, we like to first pipe the yaml into https://github.com/bronze1man/yaml2json and then pipe the json into jq




For changing yaml dynamically, perhaps generate small yaml stubs and merge them into main yaml using http://spruce.cf

Is there an API? I can see the CLI makes https calls against the director
and I could reverse engineer it from the source, but as it's not documented
anywhere it seems like it could end up as an exercise in chasing the tail
if changes are made.

On Tue, Nov 3, 2015 at 1:24 AM, Dr Nic Williams <drnicwilliams(a)gmail.com>
wrote:

You could use jq to cleanly parse and pull out data from bosh API in shell
scripts.

To inspect bosh yaml manifests, we like to first pipe the yaml into
https://github.com/bronze1man/yaml2json and then pipe the json into jq

For changing yaml dynamically, perhaps generate small yaml stubs and merge
them into main yaml using http://spruce.cf




On Tue, Nov 3, 2015 at 12:49 AM, Jeff Weber <jweber(a)cofront.net> wrote:

We're using Ansible to execute BOSH commands such as uploading stemcells
and releases, doing deployments of manifests for fresh deploys or updates
etc. Basically orchestrating the commands a human might run when dealing
with BOSH managed environments using the bosh_cli.

I'm specifically interested in being able introspect the data BOSH knows
about the environment such as getting the detailed information available
for virtual machines in a deployment, or releases and stemcells managed by
the director. For example I may want to find out what machines nats is
running on in my CF deployment and determine their internal ip address, or
correlate a virtual machine belonging to a job to the physical resource it
lives on in OpenStack to understand impact of doing a maintenance. Being
able to interact with the data available in the directory via API calls
instead of trying to parse it out of BOSH commands or reverse engineer it
in some other way would be quite useful.



On Mon, Nov 2, 2015 at 6:09 PM, Dmitriy Kalinin <dkalinin(a)pivotal.io>
wrote:

Can you provide a bit more detail what kind of operations you are
running?

On Mon, Nov 2, 2015 at 1:51 PM, Jeff Weber <jweber(a)cofront.net> wrote:

We're currently using ansible to execute a number of BOSH operations
for deploy and manage of cloud foundry. Is there any better option than
using shell scripting to interact with BOSH? Being able to look at BOSH
data such as deployment details and vm information would be much easier if
we didn't have to try to get at the data with cut grep and awk or trying to
parse the generated deployment manifest.

Director API has been stable for last 3 years.




Look at gogobosh repo to see some go structs for a subset of the API.




In the bosh_cli sub folder of bosh repo look for the director.rb for another view of available urls on the director.

I've documented basics of the Director API:
https://bosh.io/docs/director-api-v1.html

On Tue, Nov 3, 2015 at 5:10 AM, Dr Nic Williams <drnicwilliams(a)gmail.com>
wrote:

Director API has been stable for last 3 years.

Look at gogobosh repo to see some go structs for a subset of the API.

In the bosh_cli sub folder of bosh repo look for the director.rb for
another view of available urls on the director.




On Tue, Nov 3, 2015 at 2:07 PM, Jeff Weber <jweber(a)cofront.net> wrote:

Is there an API? I can see the CLI makes https calls against the director
and I could reverse engineer it from the source, but as it's not documented
anywhere it seems like it could end up as an exercise in chasing the tail
if changes are made.

On Tue, Nov 3, 2015 at 1:24 AM, Dr Nic Williams <drnicwilliams(a)gmail.com>
wrote:

You could use jq to cleanly parse and pull out data from bosh API in
shell scripts.

To inspect bosh yaml manifests, we like to first pipe the yaml into
https://github.com/bronze1man/yaml2json and then pipe the json into jq

For changing yaml dynamically, perhaps generate small yaml stubs and
merge them into main yaml using http://spruce.cf




On Tue, Nov 3, 2015 at 12:49 AM, Jeff Weber <jweber(a)cofront.net> wrote:

We're using Ansible to execute BOSH commands such as uploading
stemcells and releases, doing deployments of manifests for fresh deploys or
updates etc. Basically orchestrating the commands a human might run when
dealing with BOSH managed environments using the bosh_cli.

I'm specifically interested in being able introspect the data BOSH
knows about the environment such as getting the detailed information
available for virtual machines in a deployment, or releases and stemcells
managed by the director. For example I may want to find out what machines
nats is running on in my CF deployment and determine their internal ip
address, or correlate a virtual machine belonging to a job to the physical
resource it lives on in OpenStack to understand impact of doing a
maintenance. Being able to interact with the data available in the
directory via API calls instead of trying to parse it out of BOSH commands
or reverse engineer it in some other way would be quite useful.



On Mon, Nov 2, 2015 at 6:09 PM, Dmitriy Kalinin <dkalinin(a)pivotal.io>
wrote:

Can you provide a bit more detail what kind of operations you are
running?

On Mon, Nov 2, 2015 at 1:51 PM, Jeff Weber <jweber(a)cofront.net> wrote:

We're currently using ansible to execute a number of BOSH operations
for deploy and manage of cloud foundry. Is there any better option than
using shell scripting to interact with BOSH? Being able to look at BOSH
data such as deployment details and vm information would be much easier if
we didn't have to try to get at the data with cut grep and awk or trying to
parse the generated deployment manifest.

This may be the most important thing you’ve done this year.

спасибо!

Thanks Dmitriy.




Just the other day a client who has been using bosh in product for 6 months said "but bosh doesn't have an API". Crazy but true.

On Thu, Nov 12, 2015 at 10:14 AM, Joshua McKenty <jmckenty(a)pivotal.io>
wrote:

This may be the most important thing you’ve done this year.
спасибо!

On Nov 12, 2015, at 10:00 AM, Dmitriy Kalinin <dkalinin(a)pivotal.io> wrote:

I've documented basics of the Director API: https://bosh.io/docs/director-api-v1.html <https://bosh.io/docs/director-api-v1.html>

On Tue, Nov 3, 2015 at 5:10 AM, Dr Nic Williams <drnicwilliams(a)gmail.com <mailto:drnicwilliams(a)gmail.com>> wrote:
Director API has been stable for last 3 years.

Look at gogobosh repo to see some go structs for a subset of the API.

In the bosh_cli sub folder of bosh repo look for the director.rb for another view of available urls on the director.




On Tue, Nov 3, 2015 at 2:07 PM, Jeff Weber <jweber(a)cofront.net <mailto:jweber(a)cofront.net>> wrote:

Is there an API? I can see the CLI makes https calls against the director and I could reverse engineer it from the source, but as it's not documented anywhere it seems like it could end up as an exercise in chasing the tail if changes are made.

On Tue, Nov 3, 2015 at 1:24 AM, Dr Nic Williams <drnicwilliams(a)gmail.com <mailto:drnicwilliams(a)gmail.com>> wrote:
You could use jq to cleanly parse and pull out data from bosh API in shell scripts.

To inspect bosh yaml manifests, we like to first pipe the yaml into https://github.com/bronze1man/yaml2json <https://github.com/bronze1man/yaml2json> and then pipe the json into jq

For changing yaml dynamically, perhaps generate small yaml stubs and merge them into main yaml using http://spruce.cf <http://spruce.cf/>




On Tue, Nov 3, 2015 at 12:49 AM, Jeff Weber <jweber(a)cofront.net <mailto:jweber(a)cofront.net>> wrote:

We're using Ansible to execute BOSH commands such as uploading stemcells and releases, doing deployments of manifests for fresh deploys or updates etc. Basically orchestrating the commands a human might run when dealing with BOSH managed environments using the bosh_cli.

I'm specifically interested in being able introspect the data BOSH knows about the environment such as getting the detailed information available for virtual machines in a deployment, or releases and stemcells managed by the director. For example I may want to find out what machines nats is running on in my CF deployment and determine their internal ip address, or correlate a virtual machine belonging to a job to the physical resource it lives on in OpenStack to understand impact of doing a maintenance. Being able to interact with the data available in the directory via API calls instead of trying to parse it out of BOSH commands or reverse engineer it in some other way would be quite useful.



On Mon, Nov 2, 2015 at 6:09 PM, Dmitriy Kalinin <dkalinin(a)pivotal.io <mailto:dkalinin(a)pivotal.io>> wrote:
Can you provide a bit more detail what kind of operations you are running?

On Mon, Nov 2, 2015 at 1:51 PM, Jeff Weber <jweber(a)cofront.net <mailto:jweber(a)cofront.net>> wrote:
We're currently using ansible to execute a number of BOSH operations for deploy and manage of cloud foundry. Is there any better option than using shell scripting to interact with BOSH? Being able to look at BOSH data such as deployment details and vm information would be much easier if we didn't have to try to get at the data with cut grep and awk or trying to parse the generated deployment manifest.

Great work!
---
Ken Ojiri <ozzozz(a)gmail.com>
Mitaka, Tokyo Japan


On Fri, Nov 13, 2015 at 3:21 AM, Dr Nic Williams
<drnicwilliams(a)gmail.com> wrote:

Thanks Dmitriy.

Just the other day a client who has been using bosh in product for 6 months
said "but bosh doesn't have an API". Crazy but true.




On Thu, Nov 12, 2015 at 10:14 AM, Joshua McKenty <jmckenty(a)pivotal.io>
wrote:

This may be the most important thing you’ve done this year.

спасибо!



On Nov 12, 2015, at 10:00 AM, Dmitriy Kalinin <dkalinin(a)pivotal.io> wrote:

I've documented basics of the Director API:
https://bosh.io/docs/director-api-v1.html

On Tue, Nov 3, 2015 at 5:10 AM, Dr Nic Williams <drnicwilliams(a)gmail.com>
wrote:

Director API has been stable for last 3 years.

Look at gogobosh repo to see some go structs for a subset of the API.

In the bosh_cli sub folder of bosh repo look for the director.rb for
another view of available urls on the director.




On Tue, Nov 3, 2015 at 2:07 PM, Jeff Weber <jweber(a)cofront.net> wrote:

Is there an API? I can see the CLI makes https calls against the
director and I could reverse engineer it from the source, but as it's not
documented anywhere it seems like it could end up as an exercise in chasing
the tail if changes are made.

On Tue, Nov 3, 2015 at 1:24 AM, Dr Nic Williams
<drnicwilliams(a)gmail.com> wrote:

You could use jq to cleanly parse and pull out data from bosh API in
shell scripts.

To inspect bosh yaml manifests, we like to first pipe the yaml into
https://github.com/bronze1man/yaml2json and then pipe the json into jq

For changing yaml dynamically, perhaps generate small yaml stubs and
merge them into main yaml using http://spruce.cf




On Tue, Nov 3, 2015 at 12:49 AM, Jeff Weber <jweber(a)cofront.net> wrote:

We're using Ansible to execute BOSH commands such as uploading
stemcells and releases, doing deployments of manifests for fresh deploys or
updates etc. Basically orchestrating the commands a human might run when
dealing with BOSH managed environments using the bosh_cli.

I'm specifically interested in being able introspect the data BOSH
knows about the environment such as getting the detailed information
available for virtual machines in a deployment, or releases and stemcells
managed by the director. For example I may want to find out what machines
nats is running on in my CF deployment and determine their internal ip
address, or correlate a virtual machine belonging to a job to the physical
resource it lives on in OpenStack to understand impact of doing a
maintenance. Being able to interact with the data available in the directory
via API calls instead of trying to parse it out of BOSH commands or reverse
engineer it in some other way would be quite useful.



On Mon, Nov 2, 2015 at 6:09 PM, Dmitriy Kalinin <dkalinin(a)pivotal.io>
wrote:

Can you provide a bit more detail what kind of operations you are
running?

On Mon, Nov 2, 2015 at 1:51 PM, Jeff Weber <jweber(a)cofront.net>
wrote:

We're currently using ansible to execute a number of BOSH operations
for deploy and manage of cloud foundry. Is there any better option than
using shell scripting to interact with BOSH? Being able to look at BOSH data
such as deployment details and vm information would be much easier if we
didn't have to try to get at the data with cut grep and awk or trying to
parse the generated deployment manifest.

API docs are very much appreciated!

As requested in the docs about sharing usage of the APIs, the
https://github.com/davidehringer/bosh-java-client is currently using the
API directly.

On Thu, Nov 12, 2015 at 1:21 PM, Dr Nic Williams <drnicwilliams(a)gmail.com>
wrote:

Thanks Dmitriy.

Just the other day a client who has been using bosh in product for 6
months said "but bosh doesn't have an API". Crazy but true.




On Thu, Nov 12, 2015 at 10:14 AM, Joshua McKenty <jmckenty(a)pivotal.io>
wrote:

This may be the most important thing you’ve done this year.

спасибо!



On Nov 12, 2015, at 10:00 AM, Dmitriy Kalinin <dkalinin(a)pivotal.io>
wrote:

I've documented basics of the Director API:
https://bosh.io/docs/director-api-v1.html

On Tue, Nov 3, 2015 at 5:10 AM, Dr Nic Williams <drnicwilliams(a)gmail.com>
wrote:

Director API has been stable for last 3 years.

Look at gogobosh repo to see some go structs for a subset of the API.

In the bosh_cli sub folder of bosh repo look for the director.rb for
another view of available urls on the director.




On Tue, Nov 3, 2015 at 2:07 PM, Jeff Weber <jweber(a)cofront.net> wrote:

Is there an API? I can see the CLI makes https calls against the
director and I could reverse engineer it from the source, but as it's not
documented anywhere it seems like it could end up as an exercise in chasing
the tail if changes are made.

On Tue, Nov 3, 2015 at 1:24 AM, Dr Nic Williams <
drnicwilliams(a)gmail.com> wrote:

You could use jq to cleanly parse and pull out data from bosh API in
shell scripts.

To inspect bosh yaml manifests, we like to first pipe the yaml into
https://github.com/bronze1man/yaml2json and then pipe the json into jq

For changing yaml dynamically, perhaps generate small yaml stubs and
merge them into main yaml using http://spruce.cf




On Tue, Nov 3, 2015 at 12:49 AM, Jeff Weber <jweber(a)cofront.net>
wrote:

We're using Ansible to execute BOSH commands such as uploading
stemcells and releases, doing deployments of manifests for fresh deploys or
updates etc. Basically orchestrating the commands a human might run when
dealing with BOSH managed environments using the bosh_cli.

I'm specifically interested in being able introspect the data BOSH
knows about the environment such as getting the detailed information
available for virtual machines in a deployment, or releases and stemcells
managed by the director. For example I may want to find out what machines
nats is running on in my CF deployment and determine their internal ip
address, or correlate a virtual machine belonging to a job to the physical
resource it lives on in OpenStack to understand impact of doing a
maintenance. Being able to interact with the data available in the
directory via API calls instead of trying to parse it out of BOSH commands
or reverse engineer it in some other way would be quite useful.



On Mon, Nov 2, 2015 at 6:09 PM, Dmitriy Kalinin <dkalinin(a)pivotal.io>
wrote:

Can you provide a bit more detail what kind of operations you are
running?

On Mon, Nov 2, 2015 at 1:51 PM, Jeff Weber <jweber(a)cofront.net>
wrote:

We're currently using ansible to execute a number of BOSH
operations for deploy and manage of cloud foundry. Is there any better
option than using shell scripting to interact with BOSH? Being able to look
at BOSH data such as deployment details and vm information would be much
easier if we didn't have to try to get at the data with cut grep and awk or
trying to parse the generated deployment manifest.

Thanks Dmitriy, this is going to help us out immediately.

On Thu, Nov 12, 2015 at 7:53 PM, David Ehringer <david.ehringer(a)gmail.com>
wrote:

API docs are very much appreciated!

As requested in the docs about sharing usage of the APIs, the
https://github.com/davidehringer/bosh-java-client is currently using the
API directly.

On Thu, Nov 12, 2015 at 1:21 PM, Dr Nic Williams <drnicwilliams(a)gmail.com>
wrote:

Thanks Dmitriy.

Just the other day a client who has been using bosh in product for 6
months said "but bosh doesn't have an API". Crazy but true.




On Thu, Nov 12, 2015 at 10:14 AM, Joshua McKenty <jmckenty(a)pivotal.io>
wrote:

This may be the most important thing you’ve done this year.

спасибо!



On Nov 12, 2015, at 10:00 AM, Dmitriy Kalinin <dkalinin(a)pivotal.io>
wrote:

I've documented basics of the Director API:
https://bosh.io/docs/director-api-v1.html

On Tue, Nov 3, 2015 at 5:10 AM, Dr Nic Williams <drnicwilliams(a)gmail.com

wrote:

Director API has been stable for last 3 years.

Look at gogobosh repo to see some go structs for a subset of the API.

In the bosh_cli sub folder of bosh repo look for the director.rb for
another view of available urls on the director.




On Tue, Nov 3, 2015 at 2:07 PM, Jeff Weber <jweber(a)cofront.net> wrote:

Is there an API? I can see the CLI makes https calls against the
director and I could reverse engineer it from the source, but as it's not
documented anywhere it seems like it could end up as an exercise in chasing
the tail if changes are made.

On Tue, Nov 3, 2015 at 1:24 AM, Dr Nic Williams <
drnicwilliams(a)gmail.com> wrote:

You could use jq to cleanly parse and pull out data from bosh API in
shell scripts.

To inspect bosh yaml manifests, we like to first pipe the yaml into
https://github.com/bronze1man/yaml2json and then pipe the json into
jq

For changing yaml dynamically, perhaps generate small yaml stubs and
merge them into main yaml using http://spruce.cf




On Tue, Nov 3, 2015 at 12:49 AM, Jeff Weber <jweber(a)cofront.net>
wrote:

We're using Ansible to execute BOSH commands such as uploading
stemcells and releases, doing deployments of manifests for fresh deploys or
updates etc. Basically orchestrating the commands a human might run when
dealing with BOSH managed environments using the bosh_cli.

I'm specifically interested in being able introspect the data BOSH
knows about the environment such as getting the detailed information
available for virtual machines in a deployment, or releases and stemcells
managed by the director. For example I may want to find out what machines
nats is running on in my CF deployment and determine their internal ip
address, or correlate a virtual machine belonging to a job to the physical
resource it lives on in OpenStack to understand impact of doing a
maintenance. Being able to interact with the data available in the
directory via API calls instead of trying to parse it out of BOSH commands
or reverse engineer it in some other way would be quite useful.



On Mon, Nov 2, 2015 at 6:09 PM, Dmitriy Kalinin <dkalinin(a)pivotal.io

wrote:

Can you provide a bit more detail what kind of operations you are
running?

On Mon, Nov 2, 2015 at 1:51 PM, Jeff Weber <jweber(a)cofront.net>
wrote:

We're currently using ansible to execute a number of BOSH
operations for deploy and manage of cloud foundry. Is there any better
option than using shell scripting to interact with BOSH? Being able to look
at BOSH data such as deployment details and vm information would be much
easier if we didn't have to try to get at the data with cut grep and awk or
trying to parse the generated deployment manifest.

Thanks Dmitry this is very helpful!

As a follow up to the note in the documentation about use cases, I think
with some minor changes the existing cli tools could be modified to allow
us to use them from ansible directly.

Documenting the return codes for commands and their meaning would simplify
things quite a bit.

Adding flags to allow commands to output json or yaml instead of human
readable format would also be useful. A use case is introspecting
information about BOSH managed vms/jobs to use during maintenance
activities and then further running BOSH commands to stop / start certain
jobs etc.



On Thu, Nov 12, 2015 at 3:54 PM, john mcteague <john.mcteague(a)gmail.com>
wrote:

Thanks Dmitriy, this is going to help us out immediately.

On Thu, Nov 12, 2015 at 7:53 PM, David Ehringer <david.ehringer(a)gmail.com>
wrote:

API docs are very much appreciated!

As requested in the docs about sharing usage of the APIs, the
https://github.com/davidehringer/bosh-java-client is currently using the
API directly.

On Thu, Nov 12, 2015 at 1:21 PM, Dr Nic Williams <drnicwilliams(a)gmail.com

wrote:

Thanks Dmitriy.

Just the other day a client who has been using bosh in product for 6
months said "but bosh doesn't have an API". Crazy but true.




On Thu, Nov 12, 2015 at 10:14 AM, Joshua McKenty <jmckenty(a)pivotal.io>
wrote:

This may be the most important thing you’ve done this year.

спасибо!



On Nov 12, 2015, at 10:00 AM, Dmitriy Kalinin <dkalinin(a)pivotal.io>
wrote:

I've documented basics of the Director API:
https://bosh.io/docs/director-api-v1.html

On Tue, Nov 3, 2015 at 5:10 AM, Dr Nic Williams <
drnicwilliams(a)gmail.com> wrote:

Director API has been stable for last 3 years.

Look at gogobosh repo to see some go structs for a subset of the API.

In the bosh_cli sub folder of bosh repo look for the director.rb for
another view of available urls on the director.




On Tue, Nov 3, 2015 at 2:07 PM, Jeff Weber <jweber(a)cofront.net> wrote:

Is there an API? I can see the CLI makes https calls against the
director and I could reverse engineer it from the source, but as it's not
documented anywhere it seems like it could end up as an exercise in chasing
the tail if changes are made.

On Tue, Nov 3, 2015 at 1:24 AM, Dr Nic Williams <
drnicwilliams(a)gmail.com> wrote:

You could use jq to cleanly parse and pull out data from bosh API in
shell scripts.

To inspect bosh yaml manifests, we like to first pipe the yaml into
https://github.com/bronze1man/yaml2json and then pipe the json into
jq

For changing yaml dynamically, perhaps generate small yaml stubs and
merge them into main yaml using http://spruce.cf




On Tue, Nov 3, 2015 at 12:49 AM, Jeff Weber <jweber(a)cofront.net>
wrote:

We're using Ansible to execute BOSH commands such as uploading
stemcells and releases, doing deployments of manifests for fresh deploys or
updates etc. Basically orchestrating the commands a human might run when
dealing with BOSH managed environments using the bosh_cli.

I'm specifically interested in being able introspect the data BOSH
knows about the environment such as getting the detailed information
available for virtual machines in a deployment, or releases and stemcells
managed by the director. For example I may want to find out what machines
nats is running on in my CF deployment and determine their internal ip
address, or correlate a virtual machine belonging to a job to the physical
resource it lives on in OpenStack to understand impact of doing a
maintenance. Being able to interact with the data available in the
directory via API calls instead of trying to parse it out of BOSH commands
or reverse engineer it in some other way would be quite useful.



On Mon, Nov 2, 2015 at 6:09 PM, Dmitriy Kalinin <
dkalinin(a)pivotal.io> wrote:

Can you provide a bit more detail what kind of operations you are
running?

On Mon, Nov 2, 2015 at 1:51 PM, Jeff Weber <jweber(a)cofront.net>
wrote:

We're currently using ansible to execute a number of BOSH
operations for deploy and manage of cloud foundry. Is there any better
option than using shell scripting to interact with BOSH? Being able to look
at BOSH data such as deployment details and vm information would be much
easier if we didn't have to try to get at the data with cut grep and awk or
trying to parse the generated deployment manifest.