Bosh Deploy Says NoSuchKey However Object exists in the blobstore

nitin padalia


I've switched my bosh director blobstore from one s3 endpoint to another s3 endpoint. However now when I doing deploy of a cf-relase is says :
stderr: '2017/07/07 13:31:28 performing operation get: NoSuchKey: The specified key does not exist.
status code: 404, request id: 183c919a-f2cb-10a2-9459-005056a56656

By looking into logs when I see it failed for command:
'bosh-blobstore-s3 -c /var/vcap/bosh/etc/blobstore-s3.json get 04e527da-3594-4661-bf50-6591d7d32ed3 /var/vcap/data/tmp/bosh-blobstore....

When I looked for object 04e527da-3594-4661-bf50-6591d7d32ed3 in the respective s3 bucket, I found that object was there, not sure why it is giving me 404 then.

Ran bosh cck, also tried uploading releases with --fix and then doing bosh deploy

nitin padalia

When I ran the bosh-blobstore-s3 command from director for the same objectId, I was able to download the object.

nitin padalia

agents were pointing to old blobstore, updating it fixed the isseue. However I believe bosh upload release/stemcell --fix should've fix it.