Deployment lock stuck in CloudFoundry


Monish Kaul
 

I have cloud foundry deployed on azure machines using bosh. For adding another node to the cluster I made changes in the cloud foundry deployment manifest file and redeployed cloud foundry using 'bosh deploy'.

The deployment failed in-between but bosh deployment lock didn't get released. When I do 'bosh locks' I see the lock is acquired by deployment and the expiring time is slightly more than current time and it keeps increasing endlessly.

bosh locks
Acting as user 'admin' on 'bosh'

+------------+-----------------------+-------------------------+
| Type | Resource | Expires at |
+------------+-----------------------+-------------------------+
| deployment | single-vm-cf-on-azure | 2017-05-23 10:27:59 UTC |
+------------+-----------------------+-------------------------+
I tried cancelling the deployment task (bosh cancel task #task-number). The status of deployment task changed to 'cancelling' but it didn't get cancelled.

bosh tasks
Acting as user 'admin' on 'bosh'

+----+------------+-------------------------+-------+-------------------+--------+
| # | State | Timestamp | User | Description | Result |
+----+------------+-------------------------+-------+-------------------+--------+
| 38 | cancelling | 2017-05-23 08:40:12 UTC | admin | create deployment | |
+----+------------+-------------------------+-------+-------------------+--------+
The issue I am facing is bosh has acquired deployment lock and everytime I try to start deployment again or try to delete deployment I get an error saying:

Error 100: Unable to get deployment lock, maybe a deployment is in progress. Try again later.
1. Can I delete the acquired deployment lock information to release the lock? If yes, where is the information stored and how to delete it?

2. If a task (example, bosh deploy) fails does it keep an infinite hold on the bosh locks? Is there a way of handling task failure gracefully?

3. How to do "bosh deploy" with changes in deployment manifest file to avoid getting into infinite deployment lock acquired situation?

Join {cf-dev@lists.cloudfoundry.org to automatically receive all group messages.