Re: Failed updating job api_z1 > api_z1/0 (canary): `api_z1/0' is not running after update


Dulanjalie Dhanapala <dulanjalie@...>
 

Hello Matthias,
The issue i hit was a bug. It cause cos i was using NFS. I removed NFS
server and used S3 bucket (Credit goes to Pablo Alonso Rodriguez). then i
was able to deploy successfully. See the below link. I have added the work
around that Pablo gave me. Or use the build that Pivotal has suggested to
avoid the issue.

https://github.com/cloudfoundry/cf-release/issues/712
https://www.pivotaltracker.com/n/projects/966314/stories/98158086

Sincerely,
DUlanjalie

On Fri, Jul 24, 2015 at 10:11 AM, Johannes Hiemer <jvhiemer(a)gmail.com>
wrote:

Hi Matthias,
did you get into api_z1 0 via ssh and look at the specific error? The
problem here is, that the failure might arise from different sources.

On Fri, Jul 24, 2015 at 6:57 PM, Matthias Ender <Matthias.Ender(a)sas.com>
wrote:

Dulanjalie,

did you ever get this resolved?

I am running into the same issue, with a blank-slate 212 install:



Failed updating job api_z1 > api_z1/0 (canary): `api_z1/0' is not

running after update (00:12:18)



Error 400007: `api_z1/0' is not running after update



thanks,

Matthias



*From:* cf-bosh-bounces(a)lists.cloudfoundry.org [mailto:
cf-bosh-bounces(a)lists.cloudfoundry.org] *On Behalf Of *Dulanjalie
Dhanapala
*Sent:* Wednesday, June 24, 2015 1:14 PM
*To:* Discussions about the Cloud Foundry BOSH project.
*Subject:* Re: [cf-bosh] Failed updating job api_z1 > api_z1/0 (canary):
`api_z1/0' is not running after update



Hi all,

I deleted my "cf" deployment

git cloned the latest https://github.com/cloudfoundry/cf-release.git

I am using the latest stemcells 2999

+-----------------------------------------+---------+--------------------+

| Name | Version | CID |

+-----------------------------------------+---------+--------------------+

| bosh-aws-xen-hvm-ubuntu-trusty-go_agent | 2989 | ami-8bc83be0 light |

| bosh-aws-xen-hvm-ubuntu-trusty-go_agent | 2999* | ami-4dcc3526 light |

+-----------------------------------------+---------+--------------------+

(*) Currently in-use

Finally, i am using 212 release

bosh releases

+------+----------+-------------+

| Name | Versions | Commit Hash |

+------+----------+-------------+

| cf | 193 | 54613ebb+ |

| | 212* | ae2ec7a5+ |

+------+----------+-------------+

(*) Currently deployed

(+) Uncommitted changes

Still fails with

Failed updating job api_z1 > api_z1/0 (canary): `api_z1/0' is not
running after update (00:12:18)

Error 400007: `api_z1/0' is not running after update



But error logs are different:

http://fpaste.org/236236/65314143/

http://fpaste.org/236241/16579314/



Still "Buildpacks installation failed"



Sincerely,

Dulanjalie



On Wed, Jun 24, 2015 at 7:04 AM, Dulanjalie Dhanapala <
dulanjalie(a)gmail.com> wrote:

Thanks a lot for the respond.

Pablo, Lev and Nic also gave me some pointers. I am redeploying with
cf-212.



thanks all.

Sincerely,

Dulanjalie



On Wed, Jun 24, 2015 at 2:03 AM, Gwenn Etourneau <getourneau(a)pivotal.io>
wrote:

Best way delete deployment and redeploy to start from clean env.







On Wed, Jun 24, 2015 at 5:07 PM, James Bayer <jbayer(a)pivotal.io> wrote:

this looks like DB migrations may be messed up.



On Mon, Jun 22, 2015 at 10:32 PM, Dulanjalie Dhanapala <
dulanjalie(a)gmail.com> wrote:

Some more error messages

------------ STARTING cloud_controller_clock_ctl at Tue Jun 23 04:58:50
UTC 2015 --------------

c9/cloud_controller_ng/lib/cloud_controller/background_job_environment.rb:10:in
`setup_environment'

lib/tasks/clock.rake:6:in `block (2 levels) in <top (required)>'

PG::UndefinedTable: ERROR: relation "apps" does not exist

LINE 1: SELECT * FROM "apps" LIMIT 1



------------ STARTING cloud_controller_ng_ctl at Tue Jun 23 04:58:52 UTC
2015 --------------

Preparing local package directory

Preparing local resource_pool directory

Preparing local droplet directory

{"timestamp":1435035533.863138,"message":"PG::UndefinedTable: ERROR:
relation \"schema_migrations\" does not exist\nLINE 1: SELECT NULL AS
\"nil\" FROM \"schema_migrations\" LIMIT 1\n
^: SELECT NULL AS \"nil\" FROM \"schema_migrations\" LIMIT
1","log_level":"error","source":"cc.db.migrations","data":{},"thread_id":69988649218860,"fiber_id":69988678772900,"process_id":2704,"file":"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sequel-4.15.0/lib/sequel/database/logging.rb","lineno":70,"method":"block
in log_each"}




=====================cloud_controller_clock_ctl.log===========================

{"timestamp":1435035531.961396,"message":"PG::UndefinedTable: ERROR:
relation \"billing_events\" does not exist\nLINE 1: SELECT * FROM
\"billing_events\" LIMIT 1\n ^: SELECT * FROM
\"billing_events\" LIMIT
1","log_level":"error","source":"cc.background","data":{},"thread_id":69935043310380,"fiber_id":69935071682700,"process_id":2579,"file":"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sequel-4.15.0/lib/sequel/database/logging.rb","lineno":70,"method":"block
in log_each"}

{"timestamp":1435035531.9634626,"message":"PG::UndefinedTable: ERROR:
relation \"billing_events\" does not exist\nLINE 1: SELECT * FROM
\"billing_events\" LIMIT 1\n ^: SELECT * FROM
\"billing_events\" LIMIT
1","log_level":"error","source":"cc.background","data":{},"thread_id":69935043310380,"fiber_id":69935071682700,"process_id":2579,"file":"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sequel-4.15.0/lib/sequel/database/logging.rb","lineno":70,"method":"block
in log_each"}

{"timestamp":1435035531.9643211,"message":"PG::UndefinedTable: ERROR:
relation \"billing_events\" does not exist\nLINE 1: SELECT * FROM
\"billing_events\" LIMIT 1\n ^: SELECT * FROM
\"billing_events\" LIMIT
1","log_level":"error","source":"cc.background","data":{},"thread_id":69935043310380,"fiber_id":69935071682700,"process_id":2579,"file":"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sequel-4.15.0/lib/sequel/database/logging.rb","lineno":70,"method":"block
in log_each"}



On Mon, Jun 22, 2015 at 10:17 PM, Dulanjalie Dhanapala <
dulanjalie(a)gmail.com> wrote:

I am seeing this error. I am checking other logs


=======================cloud_controller_worker_ctl.err.log==============================

rake aborted!

SignalException: SIGTERM

/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/httpclient-2.5.1/lib/httpclient/session.rb:20:in
`require'

/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/httpclient-2.5.1/lib/httpclient/session.rb:20:in
`<top (required)>'

/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/httpclient-2.5.1/lib/httpclient.rb:17:in
`require'

/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/httpclient-2.5.1/lib/httpclient.rb:17:in
`<top (required)>'

/var/vcap/data/packages/cloud_controller_ng/27a4f1b135a8415d623996c6f3a7c668801d1ea3.1-2c0961678c940361c8069ddcac42449d59c222c9/cloud_controller_ng/app/controllers/runtime/files_controller.rb:1:in
`require'

/var/vcap/data/packages/cloud_controller_ng/27a4f1b135a8415d623996c6f3a7c668801d1ea3.1-2c0961678c940361c8069ddcac42449d59c222c9/cloud_controller_ng/app/controllers/runtime/files_controller.rb:1:in
`<top (required)>'

/var/vcap/data/packages/cloud_controller_ng/27a4f1b135a8415d623996c6f3a7c668801d1ea3.1-2c0961678c940361c8069ddcac42449d59c222c9/cloud_controller_ng/lib/cloud_controller/controllers.rb:4:in
`require'

/var/vcap/data/packages/cloud_controller_ng/27a4f1b135a8415d623996c6f3a7c668801d1ea3.1-2c0961678c940361c8069ddcac42449d59c222c9/cloud_controller_ng/lib/cloud_controller/controllers.rb:4:in
`block in <top (required)>'

/var/vcap/data/packages/cloud_controller_ng/27a4f1b135a8415d623996c6f3a7c668801d1ea3.1-2c0961678c940361c8069ddcac42449d59c222c9/cloud_controller_ng/lib/cloud_controller/controllers.rb:3:in
`each'

/var/vcap/data/packages/cloud_controller_ng/27a4f1b135a8415d623996c6f3a7c668801d1ea3.1-2c0961678c940361c8069ddcac42449d59c222c9/cloud_controller_ng/lib/cloud_controller/controllers.rb:3:in
`<top (required)>'

/var/vcap/data/packages/cloud_controller_ng/27a4f1b135a8415d623996c6f3a7c668801d1ea3.1-2c0961678c940361c8069ddcac42449d59c222c9/cloud_controller_ng/lib/cloud_controller.rb:46:in
`require'

/var/vcap/data/packages/cloud_controller_ng/27a4f1b135a8415d623996c6f3a7c668801d1ea3.1-2c0961678c940361c8069ddcac42449d59c222c9/cloud_controller_ng/lib/cloud_controller.rb:46:in
`<top (required)>'

/var/vcap/data/packages/cloud_controller_ng/27a4f1b135a8415d623996c6f3a7c668801d1ea3.1-2c0961678c940361c8069ddcac42449d59c222c9/cloud_controller_ng/Rakefile:10:in
`require'

/var/vcap/data/packages/cloud_controller_ng/27a4f1b135a8415d623996c6f3a7c668801d1ea3.1-2c0961678c940361c8069ddcac42449d59c222c9/cloud_controller_ng/Rakefile:10:in
`<top (required)>'

(See full trace by running task with --trace)



On Mon, Jun 22, 2015 at 9:39 PM, Gwenn Etourneau <getourneau(a)pivotal.io>
wrote:

Don t think is an error, please check all files.



On Tue, Jun 23, 2015 at 1:15 PM, Dulanjalie Dhanapala <
dulanjalie(a)gmail.com> wrote:


i am hitting an error when i execute bosh deploy in minimal-aws.yml
based cf deployment.



Failed updating job api_z1 > api_z1/0 (canary): `api_z1/0' is not
running after update (00:12:11)

Error 400007: `api_z1/0' is not running after update

Any suggestions to resolve this? I tried in #bosh freenode but no one
responded.



I see this error related to api_z1 in
the ./cloud_controller_worker_ctl.log

./cloud_controller_worker_ctl.log:{"timestamp":1434996874.6546366,"message":"(0.003843s)
UPDATE \"delayed_jobs\" SET \"guid\" =
'a3af8feb-30b7-4b4d-91fd-112045501a0d', \"created_at\" = '2015-06-22
18:14:32.651086+0000', \"updated_at\" = CURRENT_TIMESTAMP, \"priority\" =
0, \"attempts\" = 0, \"handler\" = '---
!ruby/struct:VCAP::CloudController::Jobs::ExceptionCatchingJob\nhandler:
!ruby/struct:VCAP::CloudController::Jobs::RequestJob\n job:
!ruby/struct:VCAP::CloudController::Jobs::TimeoutJob\n job:
!ruby/struct:VCAP::CloudController::Jobs::Runtime::BuildpackInstaller\n
name: java_buildpack\n file:
\"/var/vcap/packages/buildpack_java/java-buildpack-v2.5.zip\"\n opts:
{}\n config: \n request_id: \n', \"last_error\" = NULL, \"run_at\" =
'2015-06-22 18:14:32.660520+0000', \"locked_at\" = '2015-06-22
18:14:34.649651+0000', \"failed_at\" = NULL, \"locked_by\" = 'cc_api_worker.
*api_z1*.0.2', \"queue\" = 'cc-*api_z1*-0', \"cf_api_error\" = NULL
WHERE (\"id\" =
1)","log_level":"debug2","source":"cc.background","data":{},"thread_id":69964119192340,"fiber_id":69964148702360,"process_id":2727,"file":"/var/vcap/packages/cloud_controller_ng/cloud_controller_ng/vendor/bundle/ruby/2.1.0/gems/sequel-4.15.0/lib/sequel/database/logging.rb","lineno":70,"method":"block
in log_each"}

I searched online but it seems like there are some others hitting the
same issue. I could not find a proper solution.



Sincerely,

Dulanjalie



_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh




_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh





--

Dulanjalie





--

Dulanjalie


_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh





--

Thank you,



James Bayer


_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh




_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh





--

Dulanjalie





--

Dulanjalie

_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh


--
Mit freundlichen Grüßen

Johannes Hiemer

_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh


--
Dulanjalie

Join cf-bosh@lists.cloudfoundry.org to automatically receive all group messages.