Resource Conflict error in RabbitMQ broker.


Guruprakash S <prakash.guru4@...>
 

Hi,

We are seeing this error while creating a service instance against a latest
RabbitMQ deployment which we just deployed:

Server error, status code: 409, error code: 10001, message: Resource
conflict:

We checked the CC database table service_instances to see if there are any
service_instances related to RMQ and we don't see any there.

Request your help.

Thanks,
Guru.


William Martin
 

You can find the relevant code for this at:

https://github.com/pivotal-cf/cf-rabbitmq-release/blob/master/src/rabbitmq-broker/src/clojure/io/pivotal/pcf/rabbitmq/server.clj#L140-L165

Is it possible you have an orphaned service instance (a vhost on your
rabbit cluster). I'd be surprised to see this error from a `cf
create-service`, how are you trying to provision?

On Sat, May 7, 2016 at 10:44 PM, Guruprakash S <prakash.guru4(a)gmail.com>
wrote:

Hi,

We are seeing this error while creating a service instance against a
latest RabbitMQ deployment which we just deployed:

Server error, status code: 409, error code: 10001, message: Resource
conflict:

We checked the CC database table service_instances to see if there are any
service_instances related to RMQ and we don't see any there.

Request your help.

Thanks,
Guru.


Guruprakash S <prakash.guru4@...>
 

When trying to register a service against the RabbitMQ broker using cf
create-service



Example:

*cloudfoundry(a)micro-deploy-uspl:~/persis-store/deploy/initial$ cf
create-service p-rabbitmq standard ngis-rabbit-platformngis(a)emc.com
<ngis-rabbit-platformngis(a)emc.com>*

*Creating service ngis-rabbit-platformngis(a)emc.com
<ngis-rabbit-platformngis(a)emc.com> in org EMC / space ngisspace as admin...*

*FAILED*

*Server error, status code: 409, error code: 10001, message: Resource
conflict:
http://192.168.225.200:4567/v2/service_instances/8e038a12-a409-4f04-96ce-6948dd009ee7?accepts_incomplete=true
<http://192.168.225.200:4567/v2/service_instances/8e038a12-a409-4f04-96ce-6948dd009ee7?accepts_incomplete=true>*



We narrowed down the problem by circumventing the Cloud Controller and
trying to register against the RabbitMQ Broker directly.



The broker seems to be the culprit – we tried registering the service
against it directly using the following on the broker VM itself:

*curl http://administrator:password(a)192.168.225.200:4567/v2/service_instances/rabbitmq-test
<http://administrator:password(a)192.168.225.200:4567/v2/service_instances/rabbitmq-test>
-d '{"organization_guid":
"**50c0e92e-0ab7-4888-904e-cd043597972d","plan_id":
"178083d7-17fe-494b-b9ed-8f4dec16b11d","service_id":
"25a2f684-b1b6-4ff4-9082-7afdc38d4f86","space_guid”:
"7e27e716-24da-44cb-8f60-65506efb7df2"}' -X PUT -H
"X-Broker-API-Version: 2.6" -H "Content-Type: application/json**”*

It still resulted in throwing errors about the service already existing:

*2016-May-07 22:15:30 +0000 localhost INFO
[io.pivotal.pcf.rabbitmq.server] - Asked to provision a service:
rabbitmq-test*

*2016-May-07 22:15:30 +0000 localhost WARN
[io.pivotal.pcf.rabbitmq.server] - Vhost rabbitmq-test already exists*

*2016-May-07 22:15:30 +0000 localhost INFO
[io.pivotal.pcf.rabbitmq.server] - PUT
/v2/service_instances/rabbitmq-test 409 2 (in 32 ms)*

Essentially we need to understand why we can’t register any services
against the RabbitMQ broker.

On Sun, May 8, 2016 at 7:20 AM, William Martin <wmartin(a)pivotal.io> wrote:

You can find the relevant code for this at:


https://github.com/pivotal-cf/cf-rabbitmq-release/blob/master/src/rabbitmq-broker/src/clojure/io/pivotal/pcf/rabbitmq/server.clj#L140-L165

Is it possible you have an orphaned service instance (a vhost on your
rabbit cluster). I'd be surprised to see this error from a `cf
create-service`, how are you trying to provision?

On Sat, May 7, 2016 at 10:44 PM, Guruprakash S <prakash.guru4(a)gmail.com>
wrote:

Hi,

We are seeing this error while creating a service instance against a
latest RabbitMQ deployment which we just deployed:

Server error, status code: 409, error code: 10001, message: Resource
conflict:

We checked the CC database table service_instances to see if there are
any service_instances related to RMQ and we don't see any there.

Request your help.

Thanks,
Guru.


Guruprakash S <prakash.guru4@...>
 

William - Thanks for looking at this.

We deleted the old rabbit cluster and deployed this new release of Pivotal.
So there would not be any orphan instances. The below command was tried as
well and database tables were checked to see if any service instances, but
we found nothing there.

*cf purge-service-offering -f p-rabbitmq*

On Sun, May 8, 2016 at 1:27 PM, Guruprakash S <prakash.guru4(a)gmail.com>
wrote:

When trying to register a service against the RabbitMQ broker using cf
create-service



Example:

*cloudfoundry(a)micro-deploy-uspl:~/persis-store/deploy/initial$ cf
create-service p-rabbitmq standard ngis-rabbit-platformngis(a)emc.com
<ngis-rabbit-platformngis(a)emc.com>*

*Creating service ngis-rabbit-platformngis(a)emc.com
<ngis-rabbit-platformngis(a)emc.com> in org EMC / space ngisspace as admin...*

*FAILED*

*Server error, status code: 409, error code: 10001, message: Resource
conflict:
http://192.168.225.200:4567/v2/service_instances/8e038a12-a409-4f04-96ce-6948dd009ee7?accepts_incomplete=true
<http://192.168.225.200:4567/v2/service_instances/8e038a12-a409-4f04-96ce-6948dd009ee7?accepts_incomplete=true>*



We narrowed down the problem by circumventing the Cloud Controller and
trying to register against the RabbitMQ Broker directly.



The broker seems to be the culprit – we tried registering the service
against it directly using the following on the broker VM itself:

*curl http://administrator:password(a)192.168.225.200:4567/v2/service_instances/rabbitmq-test <http://administrator:password(a)192.168.225.200:4567/v2/service_instances/rabbitmq-test> -d '{"organization_guid": "**50c0e92e-0ab7-4888-904e-cd043597972d","plan_id": "178083d7-17fe-494b-b9ed-8f4dec16b11d","service_id": "25a2f684-b1b6-4ff4-9082-7afdc38d4f86","space_guid”: "7e27e716-24da-44cb-8f60-65506efb7df2"}' -X PUT -H "X-Broker-API-Version: 2.6" -H "Content-Type: application/json**”*

It still resulted in throwing errors about the service already existing:

*2016-May-07 22:15:30 +0000 localhost INFO [io.pivotal.pcf.rabbitmq.server] - Asked to provision a service: rabbitmq-test*

*2016-May-07 22:15:30 +0000 localhost WARN [io.pivotal.pcf.rabbitmq.server] - Vhost rabbitmq-test already exists*

*2016-May-07 22:15:30 +0000 localhost INFO [io.pivotal.pcf.rabbitmq.server] - PUT /v2/service_instances/rabbitmq-test 409 2 (in 32 ms)*

Essentially we need to understand why we can’t register any services
against the RabbitMQ broker.





On Sun, May 8, 2016 at 7:20 AM, William Martin <wmartin(a)pivotal.io> wrote:

You can find the relevant code for this at:


https://github.com/pivotal-cf/cf-rabbitmq-release/blob/master/src/rabbitmq-broker/src/clojure/io/pivotal/pcf/rabbitmq/server.clj#L140-L165

Is it possible you have an orphaned service instance (a vhost on your
rabbit cluster). I'd be surprised to see this error from a `cf
create-service`, how are you trying to provision?

On Sat, May 7, 2016 at 10:44 PM, Guruprakash S <prakash.guru4(a)gmail.com>
wrote:

Hi,

We are seeing this error while creating a service instance against a
latest RabbitMQ deployment which we just deployed:

Server error, status code: 409, error code: 10001, message: Resource
conflict:

We checked the CC database table service_instances to see if there are
any service_instances related to RMQ and we don't see any there.

Request your help.

Thanks,
Guru.


MaggieMeng
 

Hi,

We had found that it was due to a mistake in rabbitmq deployment manifest. The administrator user names are not consistent between server and broker.

Thank everyone who helped to take a look at this problem.

Regards,
Maggie

From: Guruprakash S [mailto:prakash.guru4(a)gmail.com]
Sent: 2016年5月9日 5:03
To: Discussions about Cloud Foundry projects and the system overall. <cf-dev(a)lists.cloudfoundry.org>
Subject: [cf-dev] Re: Re: Resource Conflict error in RabbitMQ broker.

William - Thanks for looking at this.

We deleted the old rabbit cluster and deployed this new release of Pivotal. So there would not be any orphan instances. The below command was tried as well and database tables were checked to see if any service instances, but we found nothing there.

cf purge-service-offering -f p-rabbitmq

On Sun, May 8, 2016 at 1:27 PM, Guruprakash S <prakash.guru4(a)gmail.com<mailto:prakash.guru4(a)gmail.com>> wrote:
When trying to register a service against the RabbitMQ broker using cf create-service

Example:
cloudfoundry(a)micro-deploy-uspl:~/persis-store/deploy/initial$ cf create-service p-rabbitmq standard ngis-rabbit-platformngis(a)emc.com<mailto:ngis-rabbit-platformngis(a)emc.com>
Creating service ngis-rabbit-platformngis(a)emc.com<mailto:ngis-rabbit-platformngis(a)emc.com> in org EMC / space ngisspace as admin...
FAILED
Server error, status code: 409, error code: 10001, message: Resource conflict: http://192.168.225.200:4567/v2/service_instances/8e038a12-a409-4f04-96ce-6948dd009ee7?accepts_incomplete=true

We narrowed down the problem by circumventing the Cloud Controller and trying to register against the RabbitMQ Broker directly.

The broker seems to be the culprit – we tried registering the service against it directly using the following on the broker VM itself:

curl http://administrator:password(a)192.168.225.200:4567/v2/service_instances/rabbitmq-test -d '{"organization_guid": "50c0e92e-0ab7-4888-904e-cd043597972d","plan_id": "178083d7-17fe-494b-b9ed-8f4dec16b11d","service_id": "25a2f684-b1b6-4ff4-9082-7afdc38d4f86","space_guid”: "7e27e716-24da-44cb-8f60-65506efb7df2"}' -X PUT -H "X-Broker-API-Version: 2.6" -H "Content-Type: application/json”

It still resulted in throwing errors about the service already existing:

2016-May-07 22:15:30 +0000 localhost INFO [io.pivotal.pcf.rabbitmq.server] - Asked to provision a service: rabbitmq-test

2016-May-07 22:15:30 +0000 localhost WARN [io.pivotal.pcf.rabbitmq.server] - Vhost rabbitmq-test already exists

2016-May-07 22:15:30 +0000 localhost INFO [io.pivotal.pcf.rabbitmq.server] - PUT /v2/service_instances/rabbitmq-test 409 2 (in 32 ms)
Essentially we need to understand why we can’t register any services against the RabbitMQ broker.



On Sun, May 8, 2016 at 7:20 AM, William Martin <wmartin(a)pivotal.io<mailto:wmartin(a)pivotal.io>> wrote:
You can find the relevant code for this at:

https://github.com/pivotal-cf/cf-rabbitmq-release/blob/master/src/rabbitmq-broker/src/clojure/io/pivotal/pcf/rabbitmq/server.clj#L140-L165

Is it possible you have an orphaned service instance (a vhost on your rabbit cluster). I'd be surprised to see this error from a `cf create-service`, how are you trying to provision?

On Sat, May 7, 2016 at 10:44 PM, Guruprakash S <prakash.guru4(a)gmail.com<mailto:prakash.guru4(a)gmail.com>> wrote:
Hi,

We are seeing this error while creating a service instance against a latest RabbitMQ deployment which we just deployed:

Server error, status code: 409, error code: 10001, message: Resource conflict:

We checked the CC database table service_instances to see if there are any service_instances related to RMQ and we don't see any there.

Request your help.

Thanks,
Guru.