|
Re: Service Broker API - updating instances
Hello Mike,
Quite reasonable. I'll put a story in the backlog.
Thank you,
Shannon Coen
Product Manager, Cloud Foundry
Pivotal, Inc.
toggle quoted messageShow quoted text
On Thu, Jun 11, 2015 at 2:15 PM, Mike Heath <elcapo(a)gmail.com> wrote: One of the really nice features of the V2 Service Broker API is the
ability to broker multiple services from a single process. Each service
provides its unique id in the broker catalog. The Cloud Controller then
includes that unique id with requests to the broker making it easy for the
broker to route the request to the correct service.
The problem I'm currently having is that the API added in version 2.4 for
updating a service instances does not include the service's unique id [1].
It only provides the service instance's id and the new plan id. This makes
it very difficult to route the update request to the correct service
provider within a process brokering multiple services as it would require
querying each service provider to determine if it brokered the provided
instance.
I would like to propose adding a service_id field to the update request,
the same as every other request in the service broker API. I believe that
the main issue with this is that it implies that an instance's service
provider could be changed by the broker. I think proper documentation would
alleviate that issue though.
Thoughts?
-Mike
1 -
http://docs.cloudfoundry.org/services/api.html#updating_service_instance
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
|
|
|
Re: How random is Metron's Doppler selection?
Actually, this might explain why some of our customers are so frustrated
trying to read their stack traces in Splunk. :\
So each line of a stack trace could go to a different Doppler. That means
each line of the stack trace goes out to a different syslog drain making it
impossible to consolidate that stack trace into a single logging event when
passed off to a third-party logging system like Splunk. This sucks. To be
fair, Splunk has never been any good at dealing with stack traces.
What are the possibilities of getting some kind of optionally enabled
application instance affinity put into Metron? (I know. I know. I can
submit a PR.)
-Mike
toggle quoted messageShow quoted text
On Thu, Jun 11, 2015 at 3:54 PM, John Tuley <jtuley(a)pivotal.io> wrote: Oops, wrong link. Should be
https://github.com/cloudfoundry/loggregator/blob/develop/src/metron/main.go#L188-L197
.
Sorry about that!
– John Tuley
On Thu, Jun 11, 2015 at 3:36 PM, John Tuley <jtuley(a)pivotal.io> wrote:
Mike,
Metron chooses a randomly-available Doppler for each message
<https://www.pivotaltracker.com/story/show/96801752>. Availability
prefers same-zone Doppler servers:
- If a Metron instance knows about any same-zone Dopplers, it chooses
one at random for each message.
- If no same-zone Dopplers are present, the random choice is made
from the list of all known servers.
In fact, the behavior you describe is the behavior of DEA Logging Agent
before Metron existed. What we discovered with that approach is that it
balances load very unfairly, as a single high-volume app is stuck on one
server. While the "new" mechanism does not guarantee consistency, it does
enable the Doppler pool to more-evenly share load.
If you're seeing that a single app instance is routed to the same Doppler
server every time, then (without further information) I would guess that
you're either running a single Doppler instance in each availability zone,
or your deck is stacked. :-) If neither of those is true and you're still
observing that Metron routes messages from an app instance to a single
Doppler, I'd love to investigate how that is happening.
– John Tuley
On Thu, Jun 11, 2015 at 2:45 PM, Mike Heath <elcapo(a)gmail.com> wrote:
Metron's documentation [1] says "All Metron traffic is randomly
distributed across available Dopplers." How random is this? Based on
observation, it appears that logs for an individual application instance
are consistently sent to the same Doppler instance. The consistency aspect
is very important for us so that our Syslog forwarder can consolidate stack
traces into a single logging event.
How random is this distribution really for an application instance's
logs?
-Mike
1 - https://github.com/cloudfoundry/loggregator/tree/develop/src/metron
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
|
|
|
Re: How random is Metron's Doppler selection?
toggle quoted messageShow quoted text
On Thu, Jun 11, 2015 at 3:36 PM, John Tuley <jtuley(a)pivotal.io> wrote: Mike,
Metron chooses a randomly-available Doppler for each message
<https://www.pivotaltracker.com/story/show/96801752>. Availability
prefers same-zone Doppler servers:
- If a Metron instance knows about any same-zone Dopplers, it chooses
one at random for each message.
- If no same-zone Dopplers are present, the random choice is made from
the list of all known servers.
In fact, the behavior you describe is the behavior of DEA Logging Agent
before Metron existed. What we discovered with that approach is that it
balances load very unfairly, as a single high-volume app is stuck on one
server. While the "new" mechanism does not guarantee consistency, it does
enable the Doppler pool to more-evenly share load.
If you're seeing that a single app instance is routed to the same Doppler
server every time, then (without further information) I would guess that
you're either running a single Doppler instance in each availability zone,
or your deck is stacked. :-) If neither of those is true and you're still
observing that Metron routes messages from an app instance to a single
Doppler, I'd love to investigate how that is happening.
– John Tuley
On Thu, Jun 11, 2015 at 2:45 PM, Mike Heath <elcapo(a)gmail.com> wrote:
Metron's documentation [1] says "All Metron traffic is randomly
distributed across available Dopplers." How random is this? Based on
observation, it appears that logs for an individual application instance
are consistently sent to the same Doppler instance. The consistency aspect
is very important for us so that our Syslog forwarder can consolidate stack
traces into a single logging event.
How random is this distribution really for an application instance's logs?
-Mike
1 - https://github.com/cloudfoundry/loggregator/tree/develop/src/metron
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
|
|
|
Re: How random is Metron's Doppler selection?
Mike, Metron chooses a randomly-available Doppler for each message < https://www.pivotaltracker.com/story/show/96801752>. Availability prefers same-zone Doppler servers: - If a Metron instance knows about any same-zone Dopplers, it chooses one at random for each message. - If no same-zone Dopplers are present, the random choice is made from the list of all known servers. In fact, the behavior you describe is the behavior of DEA Logging Agent before Metron existed. What we discovered with that approach is that it balances load very unfairly, as a single high-volume app is stuck on one server. While the "new" mechanism does not guarantee consistency, it does enable the Doppler pool to more-evenly share load. If you're seeing that a single app instance is routed to the same Doppler server every time, then (without further information) I would guess that you're either running a single Doppler instance in each availability zone, or your deck is stacked. :-) If neither of those is true and you're still observing that Metron routes messages from an app instance to a single Doppler, I'd love to investigate how that is happening. – John Tuley
On Thu, Jun 11, 2015 at 2:45 PM, Mike Heath <elcapo(a)gmail.com> wrote: Metron's documentation [1] says "All Metron traffic is randomly
distributed across available Dopplers." How random is this? Based on
observation, it appears that logs for an individual application instance
are consistently sent to the same Doppler instance. The consistency aspect
is very important for us so that our Syslog forwarder can consolidate stack
traces into a single logging event.
How random is this distribution really for an application instance's logs?
-Mike
1 - https://github.com/cloudfoundry/loggregator/tree/develop/src/metron
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
|
|
|
Service Broker API - updating instances
One of the really nice features of the V2 Service Broker API is the ability to broker multiple services from a single process. Each service provides its unique id in the broker catalog. The Cloud Controller then includes that unique id with requests to the broker making it easy for the broker to route the request to the correct service. The problem I'm currently having is that the API added in version 2.4 for updating a service instances does not include the service's unique id [1]. It only provides the service instance's id and the new plan id. This makes it very difficult to route the update request to the correct service provider within a process brokering multiple services as it would require querying each service provider to determine if it brokered the provided instance. I would like to propose adding a service_id field to the update request, the same as every other request in the service broker API. I believe that the main issue with this is that it implies that an instance's service provider could be changed by the broker. I think proper documentation would alleviate that issue though. Thoughts? -Mike 1 - http://docs.cloudfoundry.org/services/api.html#updating_service_instance
|
|
|
How random is Metron's Doppler selection?
Metron's documentation [1] says "All Metron traffic is randomly distributed across available Dopplers." How random is this? Based on observation, it appears that logs for an individual application instance are consistently sent to the same Doppler instance. The consistency aspect is very important for us so that our Syslog forwarder can consolidate stack traces into a single logging event. How random is this distribution really for an application instance's logs? -Mike 1 - https://github.com/cloudfoundry/loggregator/tree/develop/src/metron
|
|
|
Re: TCP Router VS NoRouter
Mike Youngstrom <youngm@...>
Thanks for your interest. We are close to supporting all current routing experimental features but we recently experienced some personnel issues which will caused us to lower NoRouter priority for a few months. :( We'll update the list when we think it is production ready. Mike On Thu, Jun 11, 2015 at 1:22 PM, Mohamed, Owais <Owais.Mohamed(a)covisint.com> wrote: Makes perfect sense. I really like the NoRouter because of its simplicity.
I really hope CloudFoundry adopts it and makes it part of its official
release so that NoRouter stays in step with GoRouter updates.
Thanks,
Owais
Thinking a little more about your question Mohamed. In his presentation
Mike might have made reference to people making adhoc changes to F5. In
our organization today we have a handfull of engineers that have LTM
access. And some of our existing applications that we're moving to Cloud
Foundry do require some specific LTM config or behaviour since they
required that behaviour previously. One of the benfits of NoRouter is it
simplifies the porting of these applications to Cloud Foundry since they
can make the same or similar changes to the LTM that they needed before.
We definitly view these use cases as a hacks though and are using them to
compile a list of official Service Brokers we need to eventually provide
to
formalize this functionality in NoRouter. Other orgs may not have the
requirement and would prefer that nobody touches the LTM. That type of
org
would work well with NoRouter as well.
Hope that helps.
Mike
On Thu, Jun 11, 2015 at 9:50 AM, Mike Youngstrom <youngm(a)gmail.com>
wrote:
Thanks for responding Mohamed.
Help me understand what you mean when you refer to opening up the LTM to
developers. NoRouter itself must have access to the LTM to update app
pool
members and such but end users/developers themselves will not actually
have
direct access to the LTM.
Some of the features Mike mentioned in his presentation were more a
comment on what is possible if we enable a feature to do so. When/If
NoRouter were to expose some of that functionality it would most likely
by
done using a Service Broker. For example, if an application wishes to
provide a custom OOS page we wouldn't give the developer direct access
to
the LTM to configure an OOS page, instead, I think we'd provide an OOS
Page
Service Broker. When created and bound to an application it would
prompt
NoRouter to configure the virtual server to use that Custom OOS page for
all of the routers mapped to that application. Keeping the details
sufficiently removed from the developer.
Does that help answer your question?
Mike
On Thu, Jun 11, 2015 at 9:04 AM, Mohamed, Owais <
Owais.Mohamed(a)covisint.com> wrote:
Apologize for restarting this thread after a few weeks.
That?s right Mike. Once the LTM opens up to the developers they might
start putting in too much functionality into the Load Balancer like
filters, interceptors. By using NoRouter the LTM will have to be
opened up
to the developer team.
I know that there has to be discipline and access control. But was
just
putting it out as con in the NoRouter Section.
Hoping that Chip does not have concerns with discussing NoRouter here.
Let me know and I can take this discussion offline.
From: Mike Youngstrom <youngm(a)gmail.com>
Date: Monday, May 18, 2015 at 3:22 PM
To: Chip Childers <cchilders(a)cloudfoundry.org>
Cc: Owais Mohamed <owais.mohamed(a)covisint.com>, "
cf-dev(a)lists.cloudfoundry.org" <cf-dev(a)lists.cloudfoundry.org>
Subject: Re: [cf-dev] TCP Router VS NoRouter
Hi Owais,
What are you referring to when you say you're concerned about the LTM
becoming a Monolith? Too much functionality in one component? Too
much of
the system depending on one component?
Chip,
Hopefully it's ok to discuss NoRouter here even though it isn't an
official CF project. Let us know if it is not.
Mike
On Fri, May 15, 2015 at 10:55 AM, Chip Childers <
cchilders(a)cloudfoundry.org> wrote:
The "norouter", while interesting, isn't the official CF project
approach to HTTP traffic routing. The TCP router is being built to
support
TCP routing as a general solution, with the goal of it becoming an
official
part of the CF release when ready.
-chip
Chip Childers | Technology Chief of Staff | Cloud Foundry Foundation
On Fri, May 15, 2015 at 8:58 AM, Mohamed, Owais <
Owais.Mohamed(a)covisint.com> wrote:
Hi,
I attended sessions on both TCP Router (Cloud Foundry and IOT
protocol support by Atul Kshirsagar) and NoRouter (Norouter: Running
Cloud Foundry without the Gorouter by Mike Heath).
I just wanted to start a discussion on the pros and cons of each
approach.
As personal opinion I think NoRouter is a simpler approach and can
definitely be made to support IOT protocols. The main drawback I see
with
the NoRouter is the danger of the LTM becoming a Monolith.
Any suggestions\ideas?
Regards,
Owais
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
*************************************
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
|
|
|
Re: Key Rotation Strategies
Mike Youngstrom <youngm@...>
On Thu, Jun 11, 2015 at 12:33 PM, CF Runtime <cfruntime(a)gmail.com> wrote: as well as info on other keys that need attention.
That is a good question. Looking through a manifest it is difficult to pick out which keys/passwords should be rotated. A quick perusal found these candidates: * UAA/CC DB passwords can probably be rotated without issue. * Might initially think Nats should be rotated. However, since etcd isn't even password protected we're probably ok relying on network firewall for nats and etcd. * cc.bulk_api_password should probably be rotated and could cause downtime if the components that rely upon it don't have their config changed the exact same time the CC is changed * doppler_endpoint.shared_secret will probably cause the loss of messages if rotated * uaa.cc.token_secret would probably cause everyone to get logged out right? Probably not optimal. * uaa.cc.client_secret might cause connectivity issues between UAA and CC while rotating could cause some downtime Any others? Mike
|
|
|
Re: TCP Router VS NoRouter
Makes perfect sense. I really like the NoRouter because of its simplicity.
I really hope CloudFoundry adopts it and makes it part of its official
release so that NoRouter stays in step with GoRouter updates.
Thanks,
Owais
toggle quoted messageShow quoted text
Thinking a little more about your question Mohamed. In his presentation
Mike might have made reference to people making adhoc changes to F5. In
our organization today we have a handfull of engineers that have LTM
access. And some of our existing applications that we're moving to Cloud
Foundry do require some specific LTM config or behaviour since they
required that behaviour previously. One of the benfits of NoRouter is it
simplifies the porting of these applications to Cloud Foundry since they
can make the same or similar changes to the LTM that they needed before.
We definitly view these use cases as a hacks though and are using them to
compile a list of official Service Brokers we need to eventually provide
to
formalize this functionality in NoRouter. Other orgs may not have the
requirement and would prefer that nobody touches the LTM. That type of
org
would work well with NoRouter as well.
Hope that helps.
Mike
On Thu, Jun 11, 2015 at 9:50 AM, Mike Youngstrom <youngm(a)gmail.com> wrote:
Thanks for responding Mohamed.
Help me understand what you mean when you refer to opening up the LTM to
developers. NoRouter itself must have access to the LTM to update app
pool
members and such but end users/developers themselves will not actually
have
direct access to the LTM.
Some of the features Mike mentioned in his presentation were more a
comment on what is possible if we enable a feature to do so. When/If
NoRouter were to expose some of that functionality it would most likely
by
done using a Service Broker. For example, if an application wishes to
provide a custom OOS page we wouldn't give the developer direct access
to
the LTM to configure an OOS page, instead, I think we'd provide an OOS
Page
Service Broker. When created and bound to an application it would
prompt
NoRouter to configure the virtual server to use that Custom OOS page for
all of the routers mapped to that application. Keeping the details
sufficiently removed from the developer.
Does that help answer your question?
Mike
On Thu, Jun 11, 2015 at 9:04 AM, Mohamed, Owais <
Owais.Mohamed(a)covisint.com> wrote:
Apologize for restarting this thread after a few weeks.
That?s right Mike. Once the LTM opens up to the developers they might
start putting in too much functionality into the Load Balancer like
filters, interceptors. By using NoRouter the LTM will have to be
opened up
to the developer team.
I know that there has to be discipline and access control. But was
just
putting it out as con in the NoRouter Section.
Hoping that Chip does not have concerns with discussing NoRouter here.
Let me know and I can take this discussion offline.
From: Mike Youngstrom <youngm(a)gmail.com>
Date: Monday, May 18, 2015 at 3:22 PM
To: Chip Childers <cchilders(a)cloudfoundry.org>
Cc: Owais Mohamed <owais.mohamed(a)covisint.com>, "
cf-dev(a)lists.cloudfoundry.org" <cf-dev(a)lists.cloudfoundry.org>
Subject: Re: [cf-dev] TCP Router VS NoRouter
Hi Owais,
What are you referring to when you say you're concerned about the LTM
becoming a Monolith? Too much functionality in one component? Too
much of
the system depending on one component?
Chip,
Hopefully it's ok to discuss NoRouter here even though it isn't an
official CF project. Let us know if it is not.
Mike
On Fri, May 15, 2015 at 10:55 AM, Chip Childers <
cchilders(a)cloudfoundry.org> wrote:
The "norouter", while interesting, isn't the official CF project
approach to HTTP traffic routing. The TCP router is being built to
support
TCP routing as a general solution, with the goal of it becoming an
official
part of the CF release when ready.
-chip
Chip Childers | Technology Chief of Staff | Cloud Foundry Foundation
On Fri, May 15, 2015 at 8:58 AM, Mohamed, Owais <
Owais.Mohamed(a)covisint.com> wrote:
Hi,
I attended sessions on both TCP Router (Cloud Foundry and IOT
protocol support by Atul Kshirsagar) and NoRouter (Norouter: Running
Cloud Foundry without the Gorouter by Mike Heath).
I just wanted to start a discussion on the pros and cons of each
approach.
As personal opinion I think NoRouter is a simpler approach and can
definitely be made to support IOT protocols. The main drawback I see
with
the NoRouter is the danger of the LTM becoming a Monolith.
Any suggestions\ideas?
Regards,
Owais
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
*************************************
|
|
|
Re: Staging error: no available stagers (status code: 400, error code: 170001)
The issue was likely the disk. The staging_disk_limit_mb tells the cloud controller how much disk space is needed for staging an app (unless the app asks for more). This is normally much higher than the size of the app itself because you have to accommodate all of the buildpacks on the DEA in addition to the app itself. So the staging_disk_limit_mb was 6144 mb, but the DEA was set to advertise 2048 mb. Joseph Palermo CF Runtime Team On Thu, Jun 4, 2015 at 9:12 PM, Guangcai Wang <guangcai.wang(a)gmail.com> wrote: Finally, it works after I changed the configuration related to memory and
disk for dea.
< disk_mb: 2048
---
disk_mb: 10000 1025c1029
< memory_overcommit_factor: 3
---
memory_overcommit_factor: 8 1028c1032
< staging_disk_limit_mb: 6144
---
staging_disk_limit_mb: 4096 [#1] Received on [staging.advertise] :
'{"id":"0-2b2f83b4755749aba3c31cc58a69a306","stacks":["lucid64","cflinuxfs2"],"available_memory":8192,"available_disk":20000}'
[#2] Received on [staging.advertise] :
'{"id":"0-2b2f83b4755749aba3c31cc58a69a306","stacks":["lucid64","cflinuxfs2"],"available_memory":8192,"available_disk":20000}'
After I deployed a simple php-demo application, they are changed to
[#8] Received on [staging.advertise] :
'{"id":"0-2b2f83b4755749aba3c31cc58a69a306","stacks":["lucid64","cflinuxfs2"],"available_memory":8064,"available_disk":18976}'
[#9] Received on [staging.advertise] :
'{"id":"0-2b2f83b4755749aba3c31cc58a69a306","stacks":["lucid64","cflinuxfs2"],"available_memory":8064,"available_disk":18976}'
However, I still cannot understand why my previous configuration led to
"Staging error: no available stagers" as the nats messages below said it
has enough resource. My php application is only consuming 128M memory and
1G disk. Who can share some insights?
[#41] Received on [staging.advertise] :
'{"id":"0-05b732df21c54f9cab3ac42869b4be64","stacks":["lucid64","cflinuxfs2"],"available_memory":3072,"available_disk":4096}'
ubuntu(a)boshclivm:~/apps/cf-php-demo$ cat manifest.yml
---
applications:
- name: cf-php-demo
memory: 128M
instances: 1
host: cf-php-demo
path: .
ubuntu(a)boshclivm:~/apps/cf-php-demo$ cf apps
name requested state instances memory disk urls
cf-php-demo started 1/1 128M 1G
cf-php-demo.runmyapp.io
On Thu, Jun 4, 2015 at 5:08 PM, Guangcai Wang <guangcai.wang(a)gmail.com>
wrote:
I got the nats message on 'staging.advertise'. It has the enough
resource. but it seems not correct. And it also cannot explain the error -
Server error, status code: 400, error code: 170001, message: Staging error:
no available stagers.
[#41] Received on [staging.advertise] :
'{"id":"0-05b732df21c54f9cab3ac42869b4be64","stacks":["lucid64","cflinuxfs2"],"available_memory":3072,"available_disk":4096}'
[#42] Received on [staging.advertise] :
'{"id":"0-05b732df21c54f9cab3ac42869b4be64","stacks":["lucid64","cflinuxfs2"],"available_memory":3072,"available_disk":4096}'
[#43] Received on [staging.advertise] :
'{"id":"0-05b732df21c54f9cab3ac42869b4be64","stacks":["lucid64","cflinuxfs2"],"available_memory":3072,"available_disk":4096}'
[#44] Received on [staging.advertise] :
'{"id":"0-05b732df21c54f9cab3ac42869b4be64","stacks":["lucid64","cflinuxfs2"],"available_memory":3072,"available_disk":4096}'
+------------------------------------+---------+---------------+---------------+
| Job/index | State | Resource Pool |
IPs |
+------------------------------------+---------+---------------+---------------+
| api_worker_z1/0 | running | small_z1 |
100.64.0.23 |
| api_z1/0 | running | medium_z1 |
100.64.0.21 |
| clock_global/0 | running | medium_z1 |
100.64.0.22 |
| etcd_z1/0 | running | medium_z1 |
100.64.1.8 |
| ha_proxy_z1/0 | running | router_z1 |
100.64.1.0 |
| | | |
137.172.74.90 |
| hm9000_z1/0 | running | medium_z1 |
100.64.0.24 |
| loggregator_trafficcontroller_z1/0 | running | small_z1 |
100.64.0.27 |
| loggregator_z1/0 | running | medium_z1 |
100.64.0.26 |
| login_z1/0 | running | medium_z1 |
100.64.0.20 |
| nats_z1/0 | running | medium_z1 |
100.64.1.2 |
| nfs_z1/0 | running | medium_z1 |
100.64.1.3 |
| postgres_z1/0 | running | medium_z1 |
100.64.1.4 |
| router_z1/0 | running | router_z1 |
100.64.1.5 |
| runner_z1/0 | running | runner_z1 |
100.64.0.25 |
| stats_z1/0 | running | small_z1 |
100.64.0.18 |
| uaa_z1/0 | running | medium_z1 |
100.64.0.19 |
+------------------------------------+---------+---------------+---------------+
- 100.64.0.25
m1.large | 8GB RAM | 4 VCPU | 20.0GB Disk
92cf66ec-f2e1-4505-bd25-28c02e991535 | m1.large | 8192 |
20 | 20 | | 4 | 1.0 | True
On Thu, Jun 4, 2015 at 11:57 AM, Guangcai Wang <guangcai.wang(a)gmail.com>
wrote:
From the source code
/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/dea/app_stager_task.rb:26,
it seems there is no enough for memory or disk.
def stage(&completion_callback)
@stager_id = @stager_pool.find_stager(@app.stack.name,
staging_task_memory_mb, staging_task_disk_mb)
raise Errors::ApiError.new_from_details('StagingError', 'no
available stagers') unless @stager_id
However, this is my first app. It should be light. The DEA is using
m1.large which is
m1.large | 4096 | 20
Anyone has the same error? and any suggestion on manifest or debug tips?
Another question, I want to add more debug information in
cloud_controller_ng.log. I tried to add some code in
/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/lib/cloud_controller/dea/app_stager_task.rb,
but it did not show in the log. How to do?
On Thu, Jun 4, 2015 at 10:14 AM, Guangcai Wang <guangcai.wang(a)gmail.com>
wrote:
attached the deployment manifest. This is generated by spiff and then I
modified it.
On Thu, Jun 4, 2015 at 12:47 AM, Takeshi Morikawa <moog0814(a)gmail.com>
wrote:
Please check the 'staging.advertise' of nats message
https://github.com/cloudfoundry/dea_ng#staging
sample command:
bundle exec nats-sub -s
nats://[nats.user]:[nats.password]@[nats_ipaddress]:[nats.port]
'staging.advertise'
I have one additional request
Can you share your bosh deployment manifest?
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
|
|
|
Re: Key Rotation Strategies
There is currently no way for users to rotate the cc.db_encryption_key. We're going to schedule some work to look into ways to solve the problem without downtime. Any input would be great, as well as info on other keys that need attention. Joseph Palermo CF Runtime Team On Thu, Jun 11, 2015 at 10:44 AM, Christopher B Ferris <chrisfer(a)us.ibm.com> wrote: We are also very interested in pursuing this capability.
Cheers,
Christopher Ferris
IBM Distinguished Engineer, CTO Open Cloud
IBM Software Group, Open Technologies
email: chrisfer(a)us.ibm.com
twitter: @christo4ferris
blog: http://thoughtsoncloud.com/index.php/author/cferris/
phone: +1 508 667 0402
[image: Inactive hide details for Mike Youngstrom ---06/11/2015 01:31:45
PM---There are a lot of Keys in my CF deployment manifest. I']Mike
Youngstrom ---06/11/2015 01:31:45 PM---There are a lot of Keys in my CF
deployment manifest. I'd like to be able to rotate them. Most of
From: Mike Youngstrom <youngm(a)gmail.com>
To: CF Developers Mailing List <cf-dev(a)lists.cloudfoundry.org>
Date: 06/11/2015 01:31 PM
Subject: [cf-dev] Key Rotation Strategies
Sent by: cf-dev-bounces(a)lists.cloudfoundry.org
------------------------------
There are a lot of Keys in my CF deployment manifest. I'd like to be able
to rotate them. Most of the keys I could probably just change in a
deployment but would cause some downtime or a service disruption. Others
like "cc.db_encryption_key" I have no idea how I'd rotate.
Any thoughts on key rotation for a CF deployment?
Mike
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
|
|
|
Re: Key Rotation Strategies
Christopher B Ferris <chrisfer@...>
We are also very interested in pursuing this capability. Cheers, Christopher Ferris IBM Distinguished Engineer, CTO Open Cloud IBM Software Group, Open Technologies email: chrisfer(a)us.ibm.com twitter: @christo4ferris blog: http://thoughtsoncloud.com/index.php/author/cferris/phone: +1 508 667 0402 From: Mike Youngstrom <youngm(a)gmail.com> To: CF Developers Mailing List <cf-dev(a)lists.cloudfoundry.org> Date: 06/11/2015 01:31 PM Subject: [cf-dev] Key Rotation Strategies Sent by: cf-dev-bounces(a)lists.cloudfoundry.org There are a lot of Keys in my CF deployment manifest. I'd like to be able to rotate them. Most of the keys I could probably just change in a deployment but would cause some downtime or a service disruption. Others like "cc.db_encryption_key" I have no idea how I'd rotate. Any thoughts on key rotation for a CF deployment? Mike _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
|
|
|
Mike Youngstrom <youngm@...>
There are a lot of Keys in my CF deployment manifest. I'd like to be able
to rotate them. Most of the keys I could probably just change in a
deployment but would cause some downtime or a service disruption. Others
like "cc.db_encryption_key" I have no idea how I'd rotate.
Any thoughts on key rotation for a CF deployment?
Mike
|
|
|
I believe the full crashed Warden container is kept around for an hour.
The DEA keeps the Warden handle to the container. The Warden grace time
only applies after all handles have been released.
Joseph Palermo
CF Runtime Team
toggle quoted messageShow quoted text
On Thu, Jun 11, 2015 at 8:55 AM, Pablo Alonso Rodriguez <palonsoro(a)gmail.com wrote: Ok. Just a question: When you say "the DEA will keep its container carcass
around for an hour", you mean that the DEA does not remove the container
files. However, if Warden grace time is configured at 300 seconds (5
minutes), the container is actually destroyed after that time (although its
files remain). Is this right?
Thank you very much.
2015-06-11 17:27 GMT+02:00 Dieu Cao <dcao(a)pivotal.io>:
Hi Pablo,
Ops Metrics is a PCF product and questions about that should be directed
to Pivotal customer support.
Regarding your second question, about the difference between crashed
indices and crashed indexes.
The NumberOfCrashedInstances metric is usually about 4 times the
NumberOfCrashedIndices metric. First, NumberOfCrashedInstances is the total
number of crashed containers that remain on the DEAs, while
NumberOfCrashedIndices is the number of app-index pairs which have only
crashed instances.
If an app has a droplet that crashes on startup, HM9000 will eventually
settle on restarting an instance at each of its indices every 16
minutes. When the instance crashes, the DEA will keep its container
carcass around for an hour (to allow the space developers to inspect its
files via the files API if they have the instance guid). So on average,
there will be 60/16 = 3.75 crashed instances in the system per crashed
index. That should account for most of the indices and instances that
are crashed in the system.
Hope that helps.
-Dieu
CF Runtime PM
On Thu, Jun 11, 2015 at 4:48 AM, Pablo Alonso Rodriguez <
palonsoro(a)gmail.com> wrote:
Good morning.
Recently, I have been revising metrics emitted by CF components. In
order to understand HM9000 metrics, I have been reading the metrics
documentation (at https://github.com/cloudfoundry/hm9000#metricsserver)
I post this message because I have two questions.
First question:
Not all the metrics retrieved via Ops Metrics are documented there. Is
there any additional documentation? If not, could you please explain my
what do the following metrics mean?
- StartEvacuating, StartCrashed, StartMissing
- StopDuplicate, StopEvacuationComplete, StopExtra
I have some guesses about some of them, but I am not completely sure
about them.
Second question:
I do not fully understand the difference between the concepts of
"instances" and "indices" at metrics like "NumberOfCrashedIndices" and
"NumberOfCrashedInstances".
For example, I have one crashed app in my CF instance, and
"NumberOfCrashedIndices" reports '1' and "NumberOfCrashedInstances" reports
'3'. If I have a look at `cf app myapp`, I see one single crashed instance
(this was expected). If I have a look at hm9000 dump, I see the following
about my crashed app (UUIDs have been replaced by false ones):
Guid: 7ef08c44-102d-11e5-9c0d-0fb30c2610f7 | Version:
8e16b09a-102d-11e5-b6ce-27f9445313f8
Desired: [1] instances, (STARTED, STAGED)
Heartbeats:
[0 CRASHED] a42a7236102d11e5813abfab583ad850 on 1-abc
[0 CRASHED] b35b9f1e102d11e5ad29cfc4c2c4e3ea on 2-ac3
[0 CRASHED] bbd37658102d11e5ba8e2b98d1fd1793 on 4-a67
CrashCounts: [0]:7499
Pending Starts:
[0] priority:1.00 send:2m34.628437793s
So, what does all this mean? I do not understand why do I get 3
heartbeats while
I only was trying to start a single instance.
Thank you in advance
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
|
|
|
Re: TCP Router VS NoRouter
Mike Youngstrom <youngm@...>
Thinking a little more about your question Mohamed. In his presentation
Mike might have made reference to people making adhoc changes to F5. In
our organization today we have a handfull of engineers that have LTM
access. And some of our existing applications that we're moving to Cloud
Foundry do require some specific LTM config or behaviour since they
required that behaviour previously. One of the benfits of NoRouter is it
simplifies the porting of these applications to Cloud Foundry since they
can make the same or similar changes to the LTM that they needed before.
We definitly view these use cases as a hacks though and are using them to
compile a list of official Service Brokers we need to eventually provide to
formalize this functionality in NoRouter. Other orgs may not have the
requirement and would prefer that nobody touches the LTM. That type of org
would work well with NoRouter as well.
Hope that helps.
Mike
toggle quoted messageShow quoted text
On Thu, Jun 11, 2015 at 9:50 AM, Mike Youngstrom <youngm(a)gmail.com> wrote: Thanks for responding Mohamed.
Help me understand what you mean when you refer to opening up the LTM to
developers. NoRouter itself must have access to the LTM to update app pool
members and such but end users/developers themselves will not actually have
direct access to the LTM.
Some of the features Mike mentioned in his presentation were more a
comment on what is possible if we enable a feature to do so. When/If
NoRouter were to expose some of that functionality it would most likely by
done using a Service Broker. For example, if an application wishes to
provide a custom OOS page we wouldn't give the developer direct access to
the LTM to configure an OOS page, instead, I think we'd provide an OOS Page
Service Broker. When created and bound to an application it would prompt
NoRouter to configure the virtual server to use that Custom OOS page for
all of the routers mapped to that application. Keeping the details
sufficiently removed from the developer.
Does that help answer your question?
Mike
On Thu, Jun 11, 2015 at 9:04 AM, Mohamed, Owais <
Owais.Mohamed(a)covisint.com> wrote:
Apologize for restarting this thread after a few weeks.
That’s right Mike. Once the LTM opens up to the developers they might
start putting in too much functionality into the Load Balancer like
filters, interceptors. By using NoRouter the LTM will have to be opened up
to the developer team.
I know that there has to be discipline and access control. But was just
putting it out as con in the NoRouter Section.
Hoping that Chip does not have concerns with discussing NoRouter here.
Let me know and I can take this discussion offline.
From: Mike Youngstrom <youngm(a)gmail.com>
Date: Monday, May 18, 2015 at 3:22 PM
To: Chip Childers <cchilders(a)cloudfoundry.org>
Cc: Owais Mohamed <owais.mohamed(a)covisint.com>, "
cf-dev(a)lists.cloudfoundry.org" <cf-dev(a)lists.cloudfoundry.org>
Subject: Re: [cf-dev] TCP Router VS NoRouter
Hi Owais,
What are you referring to when you say you're concerned about the LTM
becoming a Monolith? Too much functionality in one component? Too much of
the system depending on one component?
Chip,
Hopefully it's ok to discuss NoRouter here even though it isn't an
official CF project. Let us know if it is not.
Mike
On Fri, May 15, 2015 at 10:55 AM, Chip Childers <
cchilders(a)cloudfoundry.org> wrote:
The "norouter", while interesting, isn't the official CF project
approach to HTTP traffic routing. The TCP router is being built to support
TCP routing as a general solution, with the goal of it becoming an official
part of the CF release when ready.
-chip
Chip Childers | Technology Chief of Staff | Cloud Foundry Foundation
On Fri, May 15, 2015 at 8:58 AM, Mohamed, Owais <
Owais.Mohamed(a)covisint.com> wrote:
Hi,
I attended sessions on both TCP Router (Cloud Foundry and IOT
protocol support by Atul Kshirsagar) and NoRouter (Norouter: Running
Cloud Foundry without the Gorouter by Mike Heath).
I just wanted to start a discussion on the pros and cons of each
approach.
As personal opinion I think NoRouter is a simpler approach and can
definitely be made to support IOT protocols. The main drawback I see with
the NoRouter is the danger of the LTM becoming a Monolith.
Any suggestions\ideas?
Regards,
Owais
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
|
|
|
Re: Syslog Drain to Logstash Problems
toggle quoted messageShow quoted text
On Thu, Jun 11, 2015 at 10:22 AM, Mike Jacobi <jacobi(a)adobe.com> wrote: We had the same problem due to missing templates in our manifest.
We initially used the example manifest snippet shown at
https://github.com/cloudfoundry/loggregator which mentions only the
doppler template. Looking at
https://github.com/cloudfoundry/loggregator/blob/develop/manifest-templates/cf-lamb.yml we
later determined that we also needed the syslog_drain_binder
and metron_agent templates for a complete loggregator deployment.
-Mike
On Jun 10, 2015, at 9:35 AM, Steve Wall <steve.wall(a)primetimesoftware.com>
wrote:
I was able submit a log entry from the loggregator VM using -
nc -w0 10.xx.xx.xx 5000 <<< "logging from loggregator"
and to test UDP
nc -u -w0 10.xx.xx.xx 5000 <<< "logging from loggregator"
Which leads me to believe the networking is working properly. Any other
thoughts?
Thanks!
Steve
On Wed, Jun 3, 2015 at 6:14 PM, Josh Ghiloni <jghiloni(a)ecsteam.com> wrote:
We’ll check that, thanks!
Josh Ghiloni
Senior Consultant
303.932.2202 o | 303.590.5427 m | 303.565.2794 f
jghiloni(a)ecsteam.com
ECS Team
Technology Solutions Delivered
ECSTeam.com <http://ecsteam.com/>
On Jun 3, 2015, at 15:41, John Tuley <jtuley(a)pivotal.io> wrote:
Steve,
Until recently (cf-release v198), binding a syslog service required
restarting the app. If you're post-v198, it *should* Just Work.
However, one of the things that could be in your way is network
security. In order to forward logs to your drain, your loggregator servers
must be able to access that server. This is the most common cause we see of
systems failing to forward to syslog drains.
Please let us know if you have more questions.
– John Tuley
On Wed, Jun 3, 2015 at 12:37 PM, Steve Wall <
steve.wall(a)primetimesoftware.com> wrote:
Hello,
We are having problems draining log messages to Logstash. The drain is
setup as a user provided service.
cf cups logstash-drain -l syslog://xx.xx.xx.xx:5000
And then bound to the service.
cf bind-service myapp logstash-drain
But no log messages are coming through to Logstash. Or more
specifically, we are using ELK and the messages aren't seen through Kibana.
We were able to log into the DEA and using netcat (nc), messages were
successfully submitted to the ELK stack.
nc -w0 -u xx.xx.xx.xx 5000 <<< "logging from remote"
Any suggestions on how to debug this further?
-Steve
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
|
|
|
Re: Syslog Drain to Logstash Problems
I don't have an answer, so here are some diagnostic questions to help you debug: Have you set doppler.blacklisted_syslog_ranges to something that includes your destination? (This would be bad.) Do you have syslog_drain_binder running? (It should be colocated with doppler on the loggregator VMs.) Does it seem healthy? Can you use the HTTP API to etcd to look at the keys in /loggregator/services. You should see an entry per (app ID, drain URL) pair. If those aren't present, then the syslog drain binder isn't working, and doppler won't get bindings. If you have the collector enabled, look in your metrics for one containing "DopplerServer.messageRouter.numberOfSyslogSinks". If that number seems accurate, then it's likely networking (which you have already ruled out) or something really weird in the code. – John Tuley On Wed, Jun 10, 2015 at 10:35 AM, Steve Wall < steve.wall(a)primetimesoftware.com> wrote: I was able submit a log entry from the loggregator VM using -
nc -w0 10.xx.xx.xx 5000 <<< "logging from loggregator"
and to test UDP
nc -u -w0 10.xx.xx.xx 5000 <<< "logging from loggregator"
Which leads me to believe the networking is working properly. Any other
thoughts?
Thanks!
Steve
On Wed, Jun 3, 2015 at 6:14 PM, Josh Ghiloni <jghiloni(a)ecsteam.com> wrote:
We’ll check that, thanks!
Josh Ghiloni
Senior Consultant
303.932.2202 o | 303.590.5427 m | 303.565.2794 f
jghiloni(a)ecsteam.com
ECS Team
Technology Solutions Delivered
ECSTeam.com
On Jun 3, 2015, at 15:41, John Tuley <jtuley(a)pivotal.io> wrote:
Steve,
Until recently (cf-release v198), binding a syslog service required
restarting the app. If you're post-v198, it *should* Just Work.
However, one of the things that could be in your way is network
security. In order to forward logs to your drain, your loggregator servers
must be able to access that server. This is the most common cause we see of
systems failing to forward to syslog drains.
Please let us know if you have more questions.
– John Tuley
On Wed, Jun 3, 2015 at 12:37 PM, Steve Wall <
steve.wall(a)primetimesoftware.com> wrote:
Hello,
We are having problems draining log messages to Logstash. The drain is
setup as a user provided service.
cf cups logstash-drain -l syslog://xx.xx.xx.xx:5000
And then bound to the service.
cf bind-service myapp logstash-drain
But no log messages are coming through to Logstash. Or more
specifically, we are using ELK and the messages aren't seen through Kibana.
We were able to log into the DEA and using netcat (nc), messages were
successfully submitted to the ELK stack.
nc -w0 -u xx.xx.xx.xx 5000 <<< "logging from remote"
Any suggestions on how to debug this further?
-Steve
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
|
|
|
Re: Syslog Drain to Logstash Problems
toggle quoted messageShow quoted text
On Jun 10, 2015, at 9:35 AM, Steve Wall <steve.wall(a)primetimesoftware.com<mailto:steve.wall(a)primetimesoftware.com>> wrote: I was able submit a log entry from the loggregator VM using - nc -w0 10.xx.xx.xx 5000 <<< "logging from loggregator" and to test UDP nc -u -w0 10.xx.xx.xx 5000 <<< "logging from loggregator" Which leads me to believe the networking is working properly. Any other thoughts? Thanks! Steve On Wed, Jun 3, 2015 at 6:14 PM, Josh Ghiloni <jghiloni(a)ecsteam.com<mailto:jghiloni(a)ecsteam.com>> wrote: We’ll check that, thanks! Josh Ghiloni Senior Consultant 303.932.2202<tel:303.932.2202> o | 303.590.5427<tel:303.590.5427> m | 303.565.2794<tel:303.565.2794> f jghiloni(a)ecsteam.com<mailto:jghiloni(a)ecsteam.com> ECS Team Technology Solutions Delivered ECSTeam.com< http://ecsteam.com/> On Jun 3, 2015, at 15:41, John Tuley <jtuley(a)pivotal.io<mailto:jtuley(a)pivotal.io>> wrote: Steve, Until recently (cf-release v198), binding a syslog service required restarting the app. If you're post-v198, it should Just Work. However, one of the things that could be in your way is network security. In order to forward logs to your drain, your loggregator servers must be able to access that server. This is the most common cause we see of systems failing to forward to syslog drains. Please let us know if you have more questions. – John Tuley On Wed, Jun 3, 2015 at 12:37 PM, Steve Wall <steve.wall(a)primetimesoftware.com<mailto:steve.wall(a)primetimesoftware.com>> wrote: Hello, We are having problems draining log messages to Logstash. The drain is setup as a user provided service. cf cups logstash-drain -l syslog://xx.xx.xx.xx:5000 And then bound to the service. cf bind-service myapp logstash-drain But no log messages are coming through to Logstash. Or more specifically, we are using ELK and the messages aren't seen through Kibana. We were able to log into the DEA and using netcat (nc), messages were successfully submitted to the ELK stack. nc -w0 -u xx.xx.xx.xx 5000 <<< "logging from remote" Any suggestions on how to debug this further? -Steve _______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev_______________________________________________ cf-dev mailing list cf-dev(a)lists.cloudfoundry.org<mailto:cf-dev(a)lists.cloudfoundry.org> https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
|
|
|
Pablo Alonso Rodriguez <palonsoro@...>
Ok. Just a question: When you say "the DEA will keep its container carcass
around for an hour", you mean that the DEA does not remove the container
files. However, if Warden grace time is configured at 300 seconds (5
minutes), the container is actually destroyed after that time (although its
files remain). Is this right?
Thank you very much.
2015-06-11 17:27 GMT+02:00 Dieu Cao <dcao(a)pivotal.io>:
toggle quoted messageShow quoted text
Hi Pablo,
Ops Metrics is a PCF product and questions about that should be directed
to Pivotal customer support.
Regarding your second question, about the difference between crashed
indices and crashed indexes.
The NumberOfCrashedInstances metric is usually about 4 times the
NumberOfCrashedIndices metric. First, NumberOfCrashedInstances is the total
number of crashed containers that remain on the DEAs, while
NumberOfCrashedIndices is the number of app-index pairs which have only
crashed instances.
If an app has a droplet that crashes on startup, HM9000 will eventually
settle on restarting an instance at each of its indices every 16 minutes.
When the instance crashes, the DEA will keep its container carcass around
for an hour (to allow the space developers to inspect its files via the
files API if they have the instance guid). So on average, there will be
60/16 = 3.75 crashed instances in the system per crashed index. That
should account for most of the indices and instances that are crashed in
the system.
Hope that helps.
-Dieu
CF Runtime PM
On Thu, Jun 11, 2015 at 4:48 AM, Pablo Alonso Rodriguez <
palonsoro(a)gmail.com> wrote:
Good morning.
Recently, I have been revising metrics emitted by CF components. In order
to understand HM9000 metrics, I have been reading the metrics documentation
(at https://github.com/cloudfoundry/hm9000#metricsserver)
I post this message because I have two questions.
First question:
Not all the metrics retrieved via Ops Metrics are documented there. Is
there any additional documentation? If not, could you please explain my
what do the following metrics mean?
- StartEvacuating, StartCrashed, StartMissing
- StopDuplicate, StopEvacuationComplete, StopExtra
I have some guesses about some of them, but I am not completely sure
about them.
Second question:
I do not fully understand the difference between the concepts of
"instances" and "indices" at metrics like "NumberOfCrashedIndices" and
"NumberOfCrashedInstances".
For example, I have one crashed app in my CF instance, and
"NumberOfCrashedIndices" reports '1' and "NumberOfCrashedInstances" reports
'3'. If I have a look at `cf app myapp`, I see one single crashed instance
(this was expected). If I have a look at hm9000 dump, I see the following
about my crashed app (UUIDs have been replaced by false ones):
Guid: 7ef08c44-102d-11e5-9c0d-0fb30c2610f7 | Version:
8e16b09a-102d-11e5-b6ce-27f9445313f8
Desired: [1] instances, (STARTED, STAGED)
Heartbeats:
[0 CRASHED] a42a7236102d11e5813abfab583ad850 on 1-abc
[0 CRASHED] b35b9f1e102d11e5ad29cfc4c2c4e3ea on 2-ac3
[0 CRASHED] bbd37658102d11e5ba8e2b98d1fd1793 on 4-a67
CrashCounts: [0]:7499
Pending Starts:
[0] priority:1.00 send:2m34.628437793s
So, what does all this mean? I do not understand why do I get 3
heartbeats while
I only was trying to start a single instance.
Thank you in advance
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
|
|
|
Re: TCP Router VS NoRouter
Mike Youngstrom <youngm@...>
Thanks for responding Mohamed. Help me understand what you mean when you refer to opening up the LTM to developers. NoRouter itself must have access to the LTM to update app pool members and such but end users/developers themselves will not actually have direct access to the LTM. Some of the features Mike mentioned in his presentation were more a comment on what is possible if we enable a feature to do so. When/If NoRouter were to expose some of that functionality it would most likely by done using a Service Broker. For example, if an application wishes to provide a custom OOS page we wouldn't give the developer direct access to the LTM to configure an OOS page, instead, I think we'd provide an OOS Page Service Broker. When created and bound to an application it would prompt NoRouter to configure the virtual server to use that Custom OOS page for all of the routers mapped to that application. Keeping the details sufficiently removed from the developer. Does that help answer your question? Mike On Thu, Jun 11, 2015 at 9:04 AM, Mohamed, Owais <Owais.Mohamed(a)covisint.com> wrote: Apologize for restarting this thread after a few weeks.
That’s right Mike. Once the LTM opens up to the developers they might
start putting in too much functionality into the Load Balancer like
filters, interceptors. By using NoRouter the LTM will have to be opened up
to the developer team.
I know that there has to be discipline and access control. But was just
putting it out as con in the NoRouter Section.
Hoping that Chip does not have concerns with discussing NoRouter here.
Let me know and I can take this discussion offline.
From: Mike Youngstrom <youngm(a)gmail.com>
Date: Monday, May 18, 2015 at 3:22 PM
To: Chip Childers <cchilders(a)cloudfoundry.org>
Cc: Owais Mohamed <owais.mohamed(a)covisint.com>, "
cf-dev(a)lists.cloudfoundry.org" <cf-dev(a)lists.cloudfoundry.org>
Subject: Re: [cf-dev] TCP Router VS NoRouter
Hi Owais,
What are you referring to when you say you're concerned about the LTM
becoming a Monolith? Too much functionality in one component? Too much of
the system depending on one component?
Chip,
Hopefully it's ok to discuss NoRouter here even though it isn't an
official CF project. Let us know if it is not.
Mike
On Fri, May 15, 2015 at 10:55 AM, Chip Childers <
cchilders(a)cloudfoundry.org> wrote:
The "norouter", while interesting, isn't the official CF project approach
to HTTP traffic routing. The TCP router is being built to support TCP
routing as a general solution, with the goal of it becoming an official
part of the CF release when ready.
-chip
Chip Childers | Technology Chief of Staff | Cloud Foundry Foundation
On Fri, May 15, 2015 at 8:58 AM, Mohamed, Owais <
Owais.Mohamed(a)covisint.com> wrote:
Hi,
I attended sessions on both TCP Router (Cloud Foundry and IOT
protocol support by Atul Kshirsagar) and NoRouter (Norouter: Running
Cloud Foundry without the Gorouter by Mike Heath).
I just wanted to start a discussion on the pros and cons of each
approach.
As personal opinion I think NoRouter is a simpler approach and can
definitely be made to support IOT protocols. The main drawback I see with
the NoRouter is the danger of the LTM becoming a Monolith.
Any suggestions\ideas?
Regards,
Owais
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
_______________________________________________
cf-dev mailing list
cf-dev(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-dev
|
|
|
