Dear All,

I am deploying bosh direct server on Openstack Newton(RDO Project/Packstack). When I am creating environment, vm creation with stemcell have failed. I know this error is related to openstack environment. At openstack level it is showing no valid host found and Filter ImagePropertiesFilter returned 0 hosts. I have tried directly to deploy VM from bosh-openstack-kvm-ubuntu-trusty-go_agent/3421.9 image.. But it is also failed. I have installed openstack on machine with 4VCPU,24GB RAM and 295GB local storage... Is this resources are enough for launching of VM from bosh-openstack-kvm-ubuntu-trusty-go_agent/3421.9 image? Is there any issue with this image?

On the other side when I am creating instances on Openstack.. they are working ok. That means openstack is working ok. But why it is stucking at director deployment. For your response, here I am copying error.


Started validating
Downloading release 'bosh'... Skipped [Found in local cache] (00:00:00)
Validating release 'bosh'... Finished (00:00:00)
Downloading release 'bosh-openstack-cpi'... Skipped [Found in local cache] (00:00:00)
Validating release 'bosh-openstack-cpi'... Finished (00:00:00)
Validating cpi release... Finished (00:00:00)
Validating deployment manifest... Finished (00:00:00)
Downloading stemcell... Skipped [Found in local cache] (00:00:00)
Validating stemcell... Finished (00:00:01)
Finished validating (00:00:03)

Started installing CPI
Compiling package 'ruby_openstack_cpi/6576c0d52231e773f4ad53f5c5a0785c4247696a'... Finished (00:01:24)
Compiling package 'bosh_openstack_cpi/918abecbb3015ee383d5cb2af23e8dbfed6392d1'... Finished (00:00:01)
Installing packages... Finished (00:00:00)
Rendering job templates... Finished (00:00:00)
Installing job 'openstack_cpi'... Finished (00:00:00)
Finished installing CPI (00:01:26)

Starting registry... Finished (00:00:00)
Uploading stemcell 'bosh-openstack-kvm-ubuntu-trusty-go_agent/3421.9'... Finished (00:00:05)

Started deploying
Creating VM for instance 'bosh/0' from stemcell '55f5c6b5-1f8c-4f82-acd1-92084374a92c'... Failed (00:00:08)
Failed deploying (00:00:08)

Stopping registry... Finished (00:00:00)
Cleaning up rendered CPI jobs... Finished (00:00:00)

Deploying:
Creating instance 'bosh/0':
Creating VM:
Creating vm with stemcell cid '55f5c6b5-1f8c-4f82-acd1-92084374a92c':
CPI 'create_vm' method responded with error: CmdError{"type":"Bosh::Clouds::VMCreationFailed","message":"Server `75fd53e2-0ec4-4089-a952-92f4703f2ccd' state is error, expected active\nNo valid host was found. There are not enough hosts available. File \"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\", line 496, in build_instances\n context, request_spec, filter_properties)\n File \"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\", line 567, in _schedule_instances\n hosts = self.scheduler_client.select_destinations(context, spec_obj)\n File \"/usr/lib/python2.7/site-packages/nova/scheduler/utils.py\", line 370, in wrapped\n return func(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py\", line 51, in select_destinations\n return self.queryclient.select_destinations(context, spec_obj)\n File \"/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py\", line 37, in __run_method\n ret
urn getattr(self.instance, __name)(*args, **kwargs)\n File \"/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py\", line 32, in select_destinations\n return self.scheduler_rpcapi.select_destinations(context, spec_obj)\n File \"/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py\", line 126, in select_destinations\n return cctxt.call(ctxt, 'select_destinations', **msg_args)\n File \"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py\", line 169, in call\n retry=self.retry)\n File \"/usr/lib/python2.7/site-packages/oslo_messaging/transport.py\", line 97, in _send\n timeout=timeout, retry=retry)\n File \"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\", line 464, in send\n retry=retry)\n File \"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\", line 455, in _send\n raise result\n","ok_to_retry":true}

Exit code 1

Arpit,
for debugging purpose: please try to download a stemcells and upload it by
hand into OS. See if it is working, if not you might have the wrong
stemcell. (there is a difference between the format: default and raw).

On Tue, 11 Jul 2017 at 14:44 Arpit Sharma <arpitvipulsharma(a)gmail.com>
wrote:

Dear All,

I am deploying bosh direct server on Openstack Newton(RDO
Project/Packstack). When I am creating environment, vm creation with
stemcell have failed. I know this error is related to openstack
environment. At openstack level it is showing no valid host found and
Filter ImagePropertiesFilter returned 0 hosts. I have tried directly to
deploy VM from bosh-openstack-kvm-ubuntu-trusty-go_agent/3421.9 image.. But
it is also failed. I have installed openstack on machine with 4VCPU,24GB
RAM and 295GB local storage... Is this resources are enough for launching
of VM from bosh-openstack-kvm-ubuntu-trusty-go_agent/3421.9 image? Is there
any issue with this image?

On the other side when I am creating instances on Openstack.. they are
working ok. That means openstack is working ok. But why it is stucking at
director deployment. For your response, here I am copying error.


Started validating
Downloading release 'bosh'... Skipped [Found in local cache] (00:00:00)
Validating release 'bosh'... Finished (00:00:00)
Downloading release 'bosh-openstack-cpi'... Skipped [Found in local
cache] (00:00:00)
Validating release 'bosh-openstack-cpi'... Finished (00:00:00)
Validating cpi release... Finished (00:00:00)
Validating deployment manifest... Finished (00:00:00)
Downloading stemcell... Skipped [Found in local cache] (00:00:00)
Validating stemcell... Finished (00:00:01)
Finished validating (00:00:03)

Started installing CPI
Compiling package
'ruby_openstack_cpi/6576c0d52231e773f4ad53f5c5a0785c4247696a'... Finished
(00:01:24)
Compiling package
'bosh_openstack_cpi/918abecbb3015ee383d5cb2af23e8dbfed6392d1'... Finished
(00:00:01)
Installing packages... Finished (00:00:00)
Rendering job templates... Finished (00:00:00)
Installing job 'openstack_cpi'... Finished (00:00:00)
Finished installing CPI (00:01:26)

Starting registry... Finished (00:00:00)
Uploading stemcell 'bosh-openstack-kvm-ubuntu-trusty-go_agent/3421.9'...
Finished (00:00:05)

Started deploying
Creating VM for instance 'bosh/0' from stemcell
'55f5c6b5-1f8c-4f82-acd1-92084374a92c'... Failed (00:00:08)
Failed deploying (00:00:08)

Stopping registry... Finished (00:00:00)
Cleaning up rendered CPI jobs... Finished (00:00:00)

Deploying:
Creating instance 'bosh/0':
Creating VM:
Creating vm with stemcell cid '55f5c6b5-1f8c-4f82-acd1-92084374a92c':
CPI 'create_vm' method responded with error:
CmdError{"type":"Bosh::Clouds::VMCreationFailed","message":"Server
`75fd53e2-0ec4-4089-a952-92f4703f2ccd' state is error, expected active\nNo
valid host was found. There are not enough hosts available. File
\"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\", line 496,
in build_instances\n context, request_spec, filter_properties)\n File
\"/usr/lib/python2.7/site-packages/nova/conductor/manager.py\", line 567,
in _schedule_instances\n hosts =
self.scheduler_client.select_destinations(context, spec_obj)\n File
\"/usr/lib/python2.7/site-packages/nova/scheduler/utils.py\", line 370, in
wrapped\n return func(*args, **kwargs)\n File
\"/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py\",
line 51, in select_destinations\n return
self.queryclient.select_destinations(context, spec_obj)\n File
\"/usr/lib/python2.7/site-packages/nova/scheduler/client/__init__.py\",
line 37, in __run_method\
n ret
urn getattr(self.instance, __name)(*args, **kwargs)\n File
\"/usr/lib/python2.7/site-packages/nova/scheduler/client/query.py\", line
32, in select_destinations\n return
self.scheduler_rpcapi.select_destinations(context, spec_obj)\n File
\"/usr/lib/python2.7/site-packages/nova/scheduler/rpcapi.py\", line 126, in
select_destinations\n return cctxt.call(ctxt, 'select_destinations',
**msg_args)\n File
\"/usr/lib/python2.7/site-packages/oslo_messaging/rpc/client.py\", line
169, in call\n retry=self.retry)\n File
\"/usr/lib/python2.7/site-packages/oslo_messaging/transport.py\", line 97,
in _send\n timeout=timeout, retry=retry)\n File
\"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\",
line 464, in send\n retry=retry)\n File
\"/usr/lib/python2.7/site-packages/oslo_messaging/_drivers/amqpdriver.py\",
line 455, in _send\n raise result\n","ok_to_retry":true}

Exit code 1

Yes Johannes... Was thinking same response from your side.... Anyways I am trying to download openstack-kvm-centos image.. I will upload it and will check the same. Thanks for your kind support.

Hi Johannes,

I have tried with some Images and found that bosh cli is not able to create instance with these images. I am unable to deploy bosh director. when I am trying to create bosh environment, "Filter ImagePropertiesFilter returned 0 hosts" error appears with nova schedular. But when I am trying to create instance with these images manually, yes I am able to do it. Why it is happening?

Perhaps you have different AvailabilityZones for Nova and Cinder. Please go
to the configuration file for OS V2 and add the following entry:

ignore_server_availability_zone: true

On Tue, 11 Jul 2017 at 17:11 Arpit Sharma <arpitvipulsharma(a)gmail.com>
wrote:

Hi Johannes,

I have tried with some Images and found that bosh cli is not able to
create instance with these images. I am unable to deploy bosh director.
when I am trying to create bosh environment, "Filter ImagePropertiesFilter
returned 0 hosts" error appears with nova schedular. But when I am trying
to create instance with these images manually, yes I am able to do it. Why
it is happening?

Hi Johannes,

Thanks for your response. I have checked and found that nova compute and cinder services are running in nova zone. while nova schedular, nova-cert,nova-consoleauth,nova-conductor are working in internal zone. I have inserted "ignore_server_availability_zone: true" in /root/bosh-1/bosh-deployment/openstack/cpi.yml file. but still there is issue. I dont know why it is showing "No valid host was found. There are not enough hosts available" for this images. Is there any other configuration required?

Ok hmpf that's interesting. To unterstand it again Arpit, are you able to
spawn an instance from Horizon or the CLI with the uploaded Stemcell?

On Wed, 12 Jul 2017 at 09:23 Arpit Sharma <arpitvipulsharma(a)gmail.com>
wrote:

Hi Johannes,

Thanks for your response. I have checked and found that nova compute and
cinder services are running in nova zone. while nova schedular,
nova-cert,nova-consoleauth,nova-conductor are working in internal zone. I
have inserted "ignore_server_availability_zone: true" in
/root/bosh-1/bosh-deployment/openstack/cpi.yml file. but still there is
issue. I dont know why it is showing "No valid host was found. There are
not enough hosts available" for this images. Is there any other
configuration required?

I have done it with horizon... let me try with cli also.

Hey! I found some keypoints....
1) when I am trying to upload that stemcell manually from horizon as image, size is around 535 MB and I am able to launch instance also from that image via horizon.
2) But when I am trying to create environment and executing bosh create-env , it uploads image, but size is 3 GB. I am not able to launch instance from that via horizon.

see these output

openstack image list
+--------------------------------------+--------------------------------------------------+--------+
| ID | Name | Status |
+--------------------------------------+--------------------------------------------------+--------+
| 3e8eadc1-1171-49a7-affb-76b819105c9c | samestemcellmanually | active |
| 103cb62d-c40b-481b-b821-74ec7fb9e9a9 | bosh-openstack-kvm-centos-7-go_agent-raw/3421.11 | active |

openstack image show 3e8eadc1-1171-49a7-affb-76b819105c9c
+------------------+------------------------------------------------------+
| Field | Value |
+------------------+------------------------------------------------------+
| checksum | a98478d63957cafab6c06eb88d26e795 |
| container_format | bare |
| created_at | 2017-07-12T07:47:45Z |
| disk_format | raw |
| file | /v2/images/3e8eadc1-1171-49a7-affb-76b819105c9c/file |
| id | 3e8eadc1-1171-49a7-affb-76b819105c9c |
| min_disk | 0 |
| min_ram | 0 |
| name | samestemcellmanually |
| owner | 0b57c21403bb46d2a77de147c506ae89 |
| protected | False |
| schema | /v2/schemas/image |
| size | 561621381 |
| status | active |
| tags | |
| updated_at | 2017-07-12T07:55:19Z |
| virtual_size | None |
| visibility | public



openstack image show 103cb62d-c40b-481b-b821-74ec7fb9e9a9
+------------------+-------------------------------------------------------------------------------------------------------------------------------+
| Field | Value |
+------------------+-------------------------------------------------------------------------------------------------------------------------------+
| checksum | 980d07ff5a1c5649173bb02c2d34707a |
| container_format | bare |
| created_at | 2017-07-12T07:42:29Z |
| disk_format | raw |
| file | /v2/images/103cb62d-c40b-481b-b821-74ec7fb9e9a9/file |
| id | 103cb62d-c40b-481b-b821-74ec7fb9e9a9 |
| min_disk | 0 |
| min_ram | 0 |
| name | bosh-openstack-kvm-centos-7-go_agent-raw/3421.11 |
| owner | 0b57c21403bb46d2a77de147c506ae89 |
| properties | architecture='x86_64', auto_disk_config='true', hypervisor_type='kvm', os_distro='centos', os_type='linux', version='3421.11' |
| protected | False |
| schema | /v2/schemas/image |
| size | 3221225472 |
| status | active |
| tags | |
| updated_at | 2017-07-12T07:42:37Z |
| virtual_size | None |
| visibility | private |


3) same things are happening with CLI. I am able to launch instance from manually uploaded image.

Hi Arpit,
did you check: architecture='x86_64', auto_disk_config='true',
hypervisor_type='kvm', os_distro='centos', os_type='linux',
version='3421.11'

Maybe no Hypervisor of type KVM?

On Wed, 12 Jul 2017 at 10:05 Arpit Sharma <arpitvipulsharma(a)gmail.com>
wrote:

Hey! I found some keypoints....
1) when I am trying to upload that stemcell manually from horizon as
image, size is around 535 MB and I am able to launch instance also from
that image via horizon.
2) But when I am trying to create environment and executing bosh
create-env , it uploads image, but size is 3 GB. I am not able to launch
instance from that via horizon.

see these output

openstack image list

+--------------------------------------+--------------------------------------------------+--------+
| ID | Name
| Status |

+--------------------------------------+--------------------------------------------------+--------+
| 3e8eadc1-1171-49a7-affb-76b819105c9c | samestemcellmanually
| active |
| 103cb62d-c40b-481b-b821-74ec7fb9e9a9 |
bosh-openstack-kvm-centos-7-go_agent-raw/3421.11 | active |

openstack image show 3e8eadc1-1171-49a7-affb-76b819105c9c
+------------------+------------------------------------------------------+
| Field | Value |
+------------------+------------------------------------------------------+
| checksum | a98478d63957cafab6c06eb88d26e795 |
| container_format | bare |
| created_at | 2017-07-12T07:47:45Z |
| disk_format | raw |
| file | /v2/images/3e8eadc1-1171-49a7-affb-76b819105c9c/file |
| id | 3e8eadc1-1171-49a7-affb-76b819105c9c |
| min_disk | 0 |
| min_ram | 0 |
| name | samestemcellmanually |
| owner | 0b57c21403bb46d2a77de147c506ae89 |
| protected | False |
| schema | /v2/schemas/image |
| size | 561621381 |
| status | active |
| tags | |
| updated_at | 2017-07-12T07:55:19Z |
| virtual_size | None |
| visibility | public



openstack image show 103cb62d-c40b-481b-b821-74ec7fb9e9a9

+------------------+-------------------------------------------------------------------------------------------------------------------------------+
| Field | Value
|

+------------------+-------------------------------------------------------------------------------------------------------------------------------+
| checksum | 980d07ff5a1c5649173bb02c2d34707a
|
| container_format | bare
|
| created_at | 2017-07-12T07:42:29Z
|
| disk_format | raw
|
| file | /v2/images/103cb62d-c40b-481b-b821-74ec7fb9e9a9/file
|
| id | 103cb62d-c40b-481b-b821-74ec7fb9e9a9
|
| min_disk | 0
|
| min_ram | 0
|
| name | bosh-openstack-kvm-centos-7-go_agent-raw/3421.11
|
| owner | 0b57c21403bb46d2a77de147c506ae89
|
| properties | architecture='x86_64', auto_disk_config='true',
hypervisor_type='kvm', os_distro='centos', os_type='linux',
version='3421.11' |
| protected | False
|
| schema | /v2/schemas/image
|
| size | 3221225472
|
| status | active
|
| tags |
|
| updated_at | 2017-07-12T07:42:37Z
|
| virtual_size | None
|
| visibility | private
|


3) same things are happening with CLI. I am able to launch instance from
manually uploaded image.

Johannes!

Custom Properties for bosh-openstack-kvm-centos-7-go_agent-raw/3421.11

os_distro: centos
Tags
hypervisor_type:kvm
version: 3421.11
file: /v2/images/103cb62d-c40b-481b-b821-74ec7fb9e9a9/file
auto_disk_config: true
os_type: linux
Virtual Size
schema: /v2/schemas/image

Custom Properties for samestemcellmanually
schema : /v2/schemas/image
Virtual Size
file: /v2/images/3e8eadc1-1171-49a7-affb-76b819105c9c/file
Tags


I think we are going in right direction. Here I am using qemu as a hypervisor. In nova.conf , I defined virt_type = qemu. I am doing all these on my personal workstation.

Then take a look here:
https://github.com/cloudfoundry-incubator/bosh-openstack-cpi-release/issues/39

On Wed, 12 Jul 2017 at 10:34 Arpit Sharma <arpitvipulsharma(a)gmail.com>
wrote:

Johannes!

Custom Properties for bosh-openstack-kvm-centos-7-go_agent-raw/3421.11

os_distro: centos
Tags
hypervisor_type:kvm
version: 3421.11
file: /v2/images/103cb62d-c40b-481b-b821-74ec7fb9e9a9/file
auto_disk_config: true
os_type: linux
Virtual Size
schema: /v2/schemas/image

Custom Properties for samestemcellmanually
schema : /v2/schemas/image
Virtual Size
file: /v2/images/3e8eadc1-1171-49a7-affb-76b819105c9c/file
Tags


I think we are going in right direction. Here I am using qemu as a
hypervisor. In nova.conf , I defined virt_type = qemu. I am doing all these
on my personal workstation.

Hey Johannes,

I stucked here. I dont have any developer background. I am unable to find solution on this. stemcells are using kvm as hypervisor type while I am using qemu as a hypervisor in openstack. I want to change hypervisor_type to qemu for these stemcells. I have tested and found that after changing, these stemcells are working ok.

Hi Arpit,
that means it works?

On Wed, 12 Jul 2017 at 14:03 Arpit Sharma <arpitvipulsharma(a)gmail.com>
wrote:

Hey Johannes,

I stucked here. I dont have any developer background. I am unable to find
solution on this. stemcells are using kvm as hypervisor type while I am
using qemu as a hypervisor in openstack. I want to change hypervisor_type
to qemu for these stemcells. I have tested and found that after changing,
these stemcells are working ok.

Yes... I have changed hypervisor type from kvm to qemu after it uploaded by bosh env create command. Now image is working fine if i swap an instance. But In bosh env command hypervisor type is still kvm. How we can change with bosh env create command itself.

I want that whenever I upload stemcell in openstack, hypervisor_type should be qemu.

Hi Johannes,

Any suggestion? I am waiting for your valuable suggestions. we caught the problem but still dont have any solution. I can switch to other hardware if you think it will be better.

Hi Arpit,
I would switch to a different hardware and in the meanwhile open a ticket
to request configuration of this element as an option on the
bosh_openstack_cpi.

Best,
Johannes

On Wed, 12 Jul 2017 at 16:18 Arpit Sharma <arpitvipulsharma(a)gmail.com>
wrote:

Hi Johannes,

Any suggestion? I am waiting for your valuable suggestions. we caught the
problem but still dont have any solution. I can switch to other hardware if
you think it will be better.

Hi Johannes,

Thanks for your response. Let me switch to other hardware. Heartly thanks for your support. Hope will be in touch again and again. You have very sound knowledge of Cloud automation and different other tools. Can you share the link for this ticket opening? I think i need to open on pivotal...

Hi Arpit,
thanks for your kind words. This should be the right place, where you can
open a ticket:
https://github.com/cloudfoundry-incubator/bosh-openstack-cpi-release/issues

On Wed, 12 Jul 2017 at 16:45 Arpit Sharma <arpitvipulsharma(a)gmail.com>
wrote:

Hi Johannes,

Thanks for your response. Let me switch to other hardware. Heartly thanks
for your support. Hope will be in touch again and again. You have very
sound knowledge of Cloud automation and different other tools. Can you
share the link for this ticket opening? I think i need to open on pivotal...