Thanks for replying, but I'm not sure I understand your instructions.
I'm not sure what "check the route registrar merge" means.

Do you mean that I should change the last 2 lines in cf-deployment.yml as such:

- instances: 1
name: uaa_z1
...
properties:
.....
route_registrar:
routes:
........
uris:
- uaa.10.60.18.186.xip.io
- '*.uaa.10.60.18.186.xip.io'
- login.sysdomain.10.60.18.186.xip.io
- '*.login.sysdomain.10.60.18.186.xip.io'

PS with regards to above comment the login.10.60.18.186.xip.io literal appears not only under the route_registrar, but also here (should it be changed as well?):

login:
authorities: oauth.login,scim.write,clients.read,notifications.write,critical_notifications.write,emails.write,scim.userids,password.write
authorized-grant-types: authorization_code,client_credentials,refresh_token
autoapprove: true
override: true
redirect-uri: https://login.10.60.18.186.xip.io

For lack of guidance I went ahead and changed all three occurances.
I still get a 404. But it seems to happen later on:
cf api api.10.60.18.186.xip.io --skip-ssl-validation
Setting api endpoint to api.10.60.18.186.xip.io...
OK


API endpoint: https://api.10.60.18.186.xip.io (API version: 2.56.0)
Not logged in. Use 'cf login' to log in.


cf login -v --skip-ssl-validation
API endpoint: https://api.10.60.18.186.xip.io

REQUEST: [2016-06-28T20:28:17Z]
GET /v2/info HTTP/1.1
Host: api.10.60.18.186.xip.io
Accept: application/json
Content-Type: application/json
User-Agent: go-cli 6.19.0+b29b4e0 / linux



RESPONSE: [2016-06-28T20:28:17Z]
HTTP/1.1 200 OK
Content-Length: 580
Content-Type: application/json;charset=utf-8
Date: Tue, 28 Jun 2016 20:28:23 GMT
Server: nginx
X-Content-Type-Options: nosniff
X-Vcap-Request-Id: 1444e97c-1562-46d4-6820-fe06d920e947
X-Vcap-Request-Id: 1444e97c-1562-46d4-6820-fe06d920e947::b7301932-6078-4334-82ff-46fa76d0032c

{"name":"","build":"","support":"http://support.cloudfoundry.com","version":0,"description":"","authorization_endpoint":"http://login.sysdomain.10.60.18.186.xip.io","token_endpoint":"https://uaa.10.60.18.186.xip.io","min_cli_version":null,"min_recommended_cli_version":null,"api_version":"2.56.0","app_ssh_endpoint":"ssh.sysdomain.10.60.18.186.xip.io:2222","app_ssh_host_key_fingerprint":null,"app_ssh_oauth_client":"ssh-proxy","logging_endpoint":"wss://loggregator.sysdomain.10.60.18.186.xip.io:4443","doppler_logging_endpoint":"wss://doppler.sysdomain.10.60.18.186.xip.io:4443"}

REQUEST: [2016-06-28T20:28:17Z]
GET /login HTTP/1.1
Host: login.sysdomain.10.60.18.186.xip.io
Accept: application/json
Content-Type: application/json
User-Agent: go-cli 6.19.0+b29b4e0 / linux



RESPONSE: [2016-06-28T20:28:17Z]
HTTP/1.1 404 Not Found
Content-Length: 124
Cache-Control: no-store
Content-Language: en-US
Content-Type: application/json;charset=UTF-8
Date: Tue, 28 Jun 2016 20:28:24 GMT
Server: Apache-Coyote/1.1
X-Vcap-Request-Id: a43cdd2a-1c0f-4f8d-7439-8174c88c7fde

{"passwd":"https://console.10.60.18.186.xip.io/password_resets/new","signup":"https://console.10.60.18.186.xip.io/register"}


API endpoint: https://api.10.60.18.186.xip.io (API version: 2.56.0)
Not logged in. Use 'cf login' to log in.
FAILED
Server error, status code: 404, error code: , message:

Hi Eyal,

Some background info on routes, domains, the system domain, and apps
domains.

Cloud Foundry deployments include a component called the gorouter. It
essentially holds a routing table (actually a trie) in memory that maps
routes to IPs and ports. So "foo.mysystemdomain.com" might map to some
collection of IPs and ports, and "bar.myappsdomain.com" can map to other
IPs and ports. All publicly routable things in cloud foundry typically
have a route registered on their behalf with the gorouter. This includes
system components, like cloud controller, as well as all (routable) apps
pushed to the CF platform by developers.

The gorouter doesn't have a notion of domain ownership, but a platform
operator might want to make sure that an app developer doesn't try to claim
the same route as the Cloud Controller. And since CF is designed for
multitenancy, one organization might have their own custom app domain, and
may want to make sure other organizations can't use the same app domain for
their application routes.

A typical pattern to deal with this is to have all system components (CC,
UAA, etc.) that need to register routes to do so using routes that use a
special "system domain" that will not be accessible to user applications.
"domains" are owned by "organizations" in the cloud controller view of the
world, so typically a "dummy" system organization is created to own the
system domain, and this prevents it from being used by any other orgs that
users create. In practice, this "dummy" org is not a dummy, and actually
used for applications, e.g. if your Cloud Foundry installation has a custom
user portal, e.g. https://console.run.pivotal.io.

Separate from system components, users' applications also need routes. By
default, they will be given a route of the form
${app_name}.${default_shared_app_domain}. While it's technically possible
to use the same domain for the apps domain and system domain, it's not
recommended, because then random users could push an app called "api" for
example, and the gorouter would balance traffic intended for the CC between
the CC and this random app.

If you search http://docs.cloudfoundry.org/deploying/aws/cf-stub.html for
"system_domain" you can see editing instructions that recommend how to set
system domain and apps domains. In your case, I would recommend:

system_domain: sys.10.60.18.186.xip.io
app_domains:
- apps.10.60.18.186.xip.io

If you update your stub thusly, you then need to regenerate your manifest
and redeploy to make sure this has all been updated across the board. The
fact that your output shows "login.sysdomain.10.60.18.186.xip.io" and "
api.10.60.18.186.xip.io" suggest there's something inconsistent about how
the system domain is being used throughout your manifest. If you follow
the above recommendations, you would use

cf api api.sys.api.10.60.18.186.xip.io --skip-ssl-validation

Best,
Amit
<http://docs.cloudfoundry.org/deploying/aws/cf-stub.html>

Hello amit,
Regarding your above post, I have followed those instructions exactly in my cluster besides the fact that I called my SYSTEM_DOMAIN "sysdomain" to make it more easily searchable in logs later (I have had to read a lot of log files to debug errors. "sys" is not a good string to grep for as it truns up too many times...)

My stub configuration is as such:
properties:
domain: 10.60.18.186.xip.io
system_domain: sysdomain.10.60.18.186.xip.io
system_domain_organization: sysdomainorg.10.60.18.186.xip.io
app_domains:
- appsdomain.10.60.18.186.xip.io

However, the problem looks like a problem in the instructions.
When I follow your new instructions I immeditaly get a 404 which I did not get beforehand:
ubuntu(a)cf-installer:~/cloudfoundry-stubs$ cf api api.sysdomain.api.10.60.18.186.xip.io --skip-ssl-validation
Setting api endpoint to api.sysdomain.api.10.60.18.186.xip.io...
FAILED
Server error, status code: 404, error code: 0, message:

What more I have read the instructions on using the API which are linked from in your documentation ( http://docs.cloudfoundry.org/cf-cli/getting-started.html and https://github.com/cloudfoundry/cli ). They do not give explicit instructions about which node is the api node, but when I look at the example, it says nothing about accessing it through the system domain.
Also when I used "cf api api.10.60.18.186.xip.io" on the global domain (as in the doc example) I did not get a 404
Please copy-paste above, to see that there is no 404 on the original cli command.

So It does not seem plausible that the problem is as you describe it.