Date   

Re: Request to post in cf-dev mailing list

Claire Laurence
 

Hi Urvashi,
You can go to this link, https://lists.cloudfoundry.org/mailman3/lists/ and
request to join cf-dev mailing list.

Best,
Claire

On Thu, Nov 19, 2015 at 10:05 AM, Urvashi Reddy <ureddy(a)pivotal.io> wrote:

Hello,

Can you please add my email to the dev mailing list?

ureddy(a)pivotal.io

Thanks!
Urvashi
--
Claire Laurence
Pivotal Cloud Foundry San Francisco
Administrative Assistant


Request to post in cf-dev mailing list

Urvashi Reddy
 

Hello,

Can you please add my email to the dev mailing list?

ureddy(a)pivotal.io

Thanks!
Urvashi


Re: Inline-relations-depth: deprecation and its consequences

Urvashi Reddy
 

Hello,

To add on to what Simon's said, the CAPI team has a proposed style guide for new features in v3 of the API. You can take a look at the proposed API for [including related resources here.](https://github.com/cloudfoundry/cc-api-v3-style-guide#including-related-resources) We encourage you to take a look and let us know your thoughts, either here, on the style guide github page, or the cloud controller github page [here](https://github.com/cloudfoundry/cloud_controller_ng/issues)

Thanks,
Urvashi Reddy
Software Engineer on CAPI


Re: CF-RELEASE v202 UPLOAD ERROR

Warren Fernandes
 

Hey Parthiban,

It seems that there may be a misconfiguration in your manifest.
Did you configure the nfs_server properties?

https://github.com/cloudfoundry/cf-release/blob/master/templates/cf-jobs.yml#L19-L22

The api_z1 pulls the above properties in here. https://github.com/cloudfoundry/cf-release/blob/master/templates/cf-jobs.yml#L368.

Is it possible to share your manifest with us via a gist or attachment? Please remove any sensitive information like passwords, certs and keys.

Thanks.


Re: cloud_controller_ng performance degrades slowly over time

Matt Cholick
 

Sure. Here's the Ruby:

dns_open: 8 ms | dns_close: 0 ms
dns_open: 9 ms | dns_close: 0 ms
dns_open: 10 ms | dns_close: 0 ms
dns_open: 3013 ms | dns_close: 0 ms
dns_open: 5015 ms | dns_close: 0 ms
dns_open: 5011 ms | dns_close: 0 ms
dns_open: 5014 ms | dns_close: 0 ms
dns_open: 5015 ms | dns_close: 0 ms
dns_open: 5012 ms | dns_close: 0 ms
dns_open: 5015 ms | dns_close: 0 ms

https://gist.github.com/cholick/f8bab2d9752728671b94

And the Go:
dns_open: 16ms | dns_close: 0ms
dns_open: 11ms | dns_close: 0ms
dns_open: 19ms | dns_close: 0ms
dns_open: 12ms | dns_close: 0ms
dns_open: 16ms | dns_close: 0ms
dns_open: 3016ms | dns_close: 0ms
dns_open: 15ms | dns_close: 0ms
dns_open: 5033ms | dns_close: 0ms
dns_open: 11ms | dns_close: 0ms
dns_open: 5018ms | dns_close: 0ms
dns_open: 5017ms | dns_close: 0ms
dns_open: 12ms | dns_close: 0ms
dns_open: 5017ms | dns_close: 0ms
dns_open: 13ms | dns_close: 0ms
dns_open: 5015ms | dns_close: 0ms

https://gist.github.com/cholick/7d4e01dcf99588d13016

On Wed, Nov 18, 2015 at 10:10 PM, Amit Gupta <agupta(a)pivotal.io> wrote:

Thanks Matt, that's super interesting. It would be awesome to get the
strace from the Ruby thing, as well another strace from the Go thing but
this time starting from before it goes slow. I think there's some sort of
weird thing that happens once, and then after that happens, the fact that
Consul (or any other always-SERVFAIL server, e.g. Dmitriy's little Ruby DNS
server) is present makes all subsequent requests slow. So I'd like to see
strace from both the Ruby and the Go, with enough before and after the
slowness starts to establish the same pattern of behaviour.

The alternating slow/fast stuff with Go is interesting. Not clue yet what
that could be, maybe multiple goroutines?

On Wed, Nov 18, 2015 at 4:13 PM, Matt Cholick <cholick(a)gmail.com> wrote:

So things got a little more interesting yesterday. We have some custom
Golang jobs for auditing and some other purposes. A recent change added the
consult_agent template to the box with these jobs and I saw similar
behavior: requests with dns resolution taking +5s consistently, fixable by
restarting the process. I wrote a small Golang app to try and reproduce it.
I got similar, but not identical behavior in the test app.

package main


import (
"net"
"fmt"
"time"
)

func main() {
for {
startOpen := time.Now()
conn, err := net.Dial("tcp", "api.useast.appfog.qa.ctl.io:http")
if err != nil {
fmt.Printf("%#v", err)
} else {
endOpen := time.Now()
conn.Close()
endClose := time.Now()
openDur := endOpen.Sub(startOpen) / 1e6
closeDur := endClose.Sub(endOpen) / 1e6

fmt.Printf("dns_open: %dms | dns_close: %dms\n", openDur, closeDur)
}
}
}

After some time, I saw:
dns_open: 14ms | dns_close: 0ms
dns_open: 5019ms | dns_close: 1ms
dns_open: 25ms | dns_close: 1ms
dns_open: 27ms | dns_close: 1ms
dns_open: 20ms | dns_close: 0ms
dns_open: 5015ms | dns_close: 0ms
dns_open: 14ms | dns_close: 0ms
dns_open: 5018ms | dns_close: 1ms
dns_open: 15ms | dns_close: 0ms
dns_open: 5018ms | dns_close: 1ms
dns_open: 19ms | dns_close: 0ms
dns_open: 5046ms | dns_close: 0ms

To reiterate, the behavior in the actual app seemed to be every single
request. The test app went into a similar mode where a roughly every other
request failed. Here's the strace for that section:

https://gist.github.com/cholick/428df93c112067f50558

On some other runs it was quite consistently alternating slow and fast
calls (for ~300 calls in a row alternating slow/fast). Just like the Ruby,
once things get into an unhappy state they stay that way until restarting
the process. (The process ran for 12 hours and never recovered).

I can capture a Ruby strace too if you'd like, but I figured this was
more useful as it added more novel information to the mix.

These Golang jobs have run in production for months without showing this
behavior and, within a few hours of including the consul template, the
issue manifested.


Re: Inline-relations-depth: deprecation and its consequences

Simon D Moser
 

Hi Ponraj,


#1) Because IRD can quickly become quite expensive DB query wise, since it
would follow all relations given your query. Eg v2/organizations with IRD
2 or 3 would drag along A LOT of things, most in which you probably aren't
interested in

#2) Yes, in v3 API there is a way to specify what resources you exactly
want when you query - e.g. v3/organizations and you pass (spaces, routes)
because thats what you're really interested in

check the v3 style guide for the correct syntax etc:
https://github.com/cloudfoundry/cc-api-v3-style-guide

Mit freundlichen Grüßen / Kind regards

Simon Moser

Senior Technical Staff Member / IBM Master Inventor
Bluemix Application Platform Lead Architect
Dept. C727, IBM Research & Development Boeblingen

-------------------------------------------------------------------------------------------------------------------------------------------
IBM Deutschland
Schoenaicher Str. 220
71032 Boeblingen
Phone: +49-7031-16-4304
Fax: +49-7031-16-4890
E-Mail: smoser(a)de.ibm.com
-------------------------------------------------------------------------------------------------------------------------------------------
IBM Deutschland Research & Development GmbH / Vorsitzender des
Aufsichtsrats: Martina Koederitz
Geschäftsführung: Dirk Wittkopp
Sitz der Gesellschaft: Böblingen / Registergericht: Amtsgericht
Stuttgart, HRB 243294

**
Great minds discuss ideas; average minds discuss events; small minds
discuss people.
Eleanor Roosevelt



From: "Ponraj E" <ponraj.e(a)gmail.com>
To: cf-dev(a)lists.cloudfoundry.org
Date: 19/11/2015 13:02
Subject: [cf-dev] Re: Inline-relations-depth: deprecation and its
consequences



Hi,

If anyone can provide information on this atleast, it would help.

1. Why the "inline-relations-depth" feature is going to be deprecated.

2. Does cf has planned any other alternative solution in the near future
to replace "inline-relations-depth"?

---------
Ponraj


Re: Inline-relations-depth: deprecation and its consequences

Ponraj E
 

Hi,

If anyone can provide information on this atleast, it would help.

1. Why the "inline-relations-depth" feature is going to be deprecated.

2. Does cf has planned any other alternative solution in the near future to replace "inline-relations-depth"?

---------
Ponraj


Re: Question about the terms of use of Cloud Foundry logo

Christopher B Ferris <chrisfer@...>
 

+Mike, he should be able to help.

Cheers,

Christopher Ferris
IBM Distinguished Engineer, CTO Open Cloud
IBM Software Group, Open Technologies
email: chrisfer(a)us.ibm.com
twitter: @christo4ferris
blog: http://thoughtsoncloud.com/index.php/author/cferris/
phone: +1 508 667 0402

On Nov 19, 2015, at 2:04 AM, Noburou TANIGUCHI <dev(a)nota.m001.jp> wrote:

Hi,

I know this is **Dev** mailing list but I can't find any other place to ask
this question.

# Please navigate me if there is an appropriate place.

My company wants to know about the terms of use of the Cloud Foundry logo
("Molten"). The "Assets" page [1] gives us the style guideline for the logo,
but we cannot find legal terms of use. Would you please tell us the
information, or a reference to the information.

Thank you.

[1] https://www.cloudfoundry.org/foundation/assets/

Suppl: My company is a member of the Foundation.




-----
I'm not a ...
noburou taniguchi
--
View this message in context: http://cf-dev.70369.x6.nabble.com/Question-about-the-terms-of-use-of-Cloud-Foundry-logo-tp2760.html
Sent from the CF Dev mailing list archive at Nabble.com.


REGARDING_api_z1/0_CANARY_UPDATE

Parthiban Annadurai <senjiparthi@...>
 

Hello All,
Since, I was facing the following issue for very long time, I have opened it as a separate thread. The problem am currently facing is

Error 400007: `api_z1/0' is not running after update

I have SSHed into the api_z1/0 VM and did a monit summary. It shows that

root(a)5c446a3d-3070-4d24-9f2e-1cff18218c07:/var/vcap/sys/log# monit summary
The Monit daemon 5.2.4 uptime: 20m

Process 'cloud_controller_ng' initializing
Process 'cloud_controller_worker_local_1' not monitored
Process 'cloud_controller_worker_local_2' not monitored
Process 'nginx_cc' initializing
Process 'metron_agent' running
File 'nfs_mounter' Does not exist
System 'system_5c446a3d-3070-4d24-9f2e-1cff18218c07' running

Could anyone help on this issue? Thanks.


FW: issue tracker permissions

Sievers, Jan <jan.sievers@...>
 

Hi,

I was trying to watch a story I am interested in
https://www.pivotaltracker.com/n/projects/892938/stories/105493826


I do have an account but it seems I don't have permissions to watch nor to comment.

Is there something I missed?

Regards
Jan


Re: Unable to deploy application

Bharath
 

Hi Deepak

I think it was unable to clone java buildpack. can you check the your
security groups and network settings .can you tell which network project
you are using neutron or nova



regards
Bharath

On Thu, Nov 19, 2015 at 1:24 AM, Deepak Arn <arn.deepak1(a)gmail.com> wrote:

Hello,

I have configured local cloud foundry on openstack, its giving some
staging error while deploying web application. Please find the log file
attached below.

Thanks,

--
Regards
Deepak Kumar Aggarwal
Research Assitant,
Concordia University, Canada


Re: Question about the terms of use of Cloud Foundry logo

Noburou TANIGUCHI
 

Add: I assume that there may be something like this:

http://www.linuxfoundation.org/about/linux-foundation-trademark-usage-guidelines




-----
I'm not a ...
noburou taniguchi
--
View this message in context: http://cf-dev.70369.x6.nabble.com/Question-about-the-terms-of-use-of-Cloud-Foundry-logo-tp2760p2762.html
Sent from the CF Dev mailing list archive at Nabble.com.


Question about the terms of use of Cloud Foundry logo

Noburou TANIGUCHI
 

Hi,

I know this is **Dev** mailing list but I can't find any other place to ask
this question.

# Please navigate me if there is an appropriate place.

My company wants to know about the terms of use of the Cloud Foundry logo
("Molten"). The "Assets" page [1] gives us the style guideline for the logo,
but we cannot find legal terms of use. Would you please tell us the
information, or a reference to the information.

Thank you.

[1] https://www.cloudfoundry.org/foundation/assets/

Suppl: My company is a member of the Foundation.




-----
I'm not a ...
noburou taniguchi
--
View this message in context: http://cf-dev.70369.x6.nabble.com/Question-about-the-terms-of-use-of-Cloud-Foundry-logo-tp2760.html
Sent from the CF Dev mailing list archive at Nabble.com.


Re: CF-RELEASE v202 UPLOAD ERROR

Parthiban Annadurai <senjiparthi@...>
 

Hello All,
Still we are facing the issue in Canary Update. It throws the
following,

Error 400007: `api_z1/0' is not running after update

Could anyone on this issue?? Thanks..

Regards,

Parthiban A



On 18 November 2015 at 10:39, Parthiban Annadurai <senjiparthi(a)gmail.com>
wrote:

Also, the bosh vms command shows that api_z1/0 is not in running status
rather its in starting state always. Thanks..

On 18 November 2015 at 10:10, Parthiban Annadurai <senjiparthi(a)gmail.com>
wrote:

Hey Warren,
Thanks for your valuable suggestions. I have SSHed into
that VM. Monit Summary Command shows the following,

root(a)5c446a3d-3070-4d24-9f2e-1cff18218c07:/var/vcap/sys/log# monit
summary
The Monit daemon 5.2.4 uptime: 20m

Process 'cloud_controller_ng' initializing
Process 'cloud_controller_worker_local_1' not monitored
Process 'cloud_controller_worker_local_2' not monitored
Process 'nginx_cc' initializing
Process 'metron_agent' running
File 'nfs_mounter' Does not exist
System 'system_5c446a3d-3070-4d24-9f2e-1cff18218c07' running

Also I have checked for cloud_controller_ng_ctl.log, it has the following,

[2015-11-18 04:33:34+0000] ------------ STARTING cloud_controller_ng_ctl
at Wed Nov 18 04:32:53 UTC 2015 --------------
[2015-11-18 04:33:34+0000] Preparing local package directory
[2015-11-18 04:33:34+0000] Preparing local resource_pool directory
[2015-11-18 04:33:34+0000] Preparing local droplet directory
[2015-11-18 04:33:34+0000] Deprecated: Use -s or --insert-seed flag
[2015-11-18 04:33:34+0000] Killing
/var/vcap/sys/run/cloud_controller_ng/cloud_controller_ng.pid: 32522
[2015-11-18 04:33:34+0000] .Stopped


Then, nfs_mounter_ctl.log has the following,

[2015-11-18 04:27:20+0000] Found NFS mount, unmounting...
[2015-11-18 04:27:20+0000] NFS unmounted
[2015-11-18 04:27:20+0000] idmapd start/post-stop, process 25777
[2015-11-18 04:27:20+0000] NFS unmounted
[2015-11-18 04:27:20+0000] Mounting NFS...
[2015-11-18 04:27:20+0000] mount.nfs: timeout set for Wed Nov 18 04:29:20
2015
[2015-11-18 04:27:20+0000] mount.nfs: trying text-based options
'timeo=10,intr,lookupcache=positive,vers=4,addr=192.168.33.53,clientaddr=192.168.33.184'
[2015-11-18 04:27:20+0000] mount.nfs: trying text-based options
'timeo=10,intr,lookupcache=positive,addr=192.168.33.53'
[2015-11-18 04:27:20+0000] mount.nfs: prog 100003, trying vers=3, prot=6
[2015-11-18 04:27:20+0000] mount.nfs: prog 100005, trying vers=3, prot=17
[2015-11-18 04:27:20+0000] Failed to start: cannot write to NFS

I think the problem is with the NFS. Could you please help on this
issue?? Thanks..

Regards

Parthiban A

On 18 November 2015 at 08:42, Warren Fernandes <wfernandes(a)pivotal.io>
wrote:

Try ssh'ing onto the box to see what the logs say.

`bosh ssh api_z1 0`

Then,
`cd /var/vcap/sys/log/`

There are plenty of logs to look through, I'd start by running `monit
summary` (make sure to be root) to see if any process is failing. If there
is a process failing then look at its respective logs. Else start with
cloud_controller_ng log directory.


Re: cloud_controller_ng performance degrades slowly over time

Amit Kumar Gupta
 

Thanks Matt, that's super interesting. It would be awesome to get the
strace from the Ruby thing, as well another strace from the Go thing but
this time starting from before it goes slow. I think there's some sort of
weird thing that happens once, and then after that happens, the fact that
Consul (or any other always-SERVFAIL server, e.g. Dmitriy's little Ruby DNS
server) is present makes all subsequent requests slow. So I'd like to see
strace from both the Ruby and the Go, with enough before and after the
slowness starts to establish the same pattern of behaviour.

The alternating slow/fast stuff with Go is interesting. Not clue yet what
that could be, maybe multiple goroutines?

On Wed, Nov 18, 2015 at 4:13 PM, Matt Cholick <cholick(a)gmail.com> wrote:

So things got a little more interesting yesterday. We have some custom
Golang jobs for auditing and some other purposes. A recent change added the
consult_agent template to the box with these jobs and I saw similar
behavior: requests with dns resolution taking +5s consistently, fixable by
restarting the process. I wrote a small Golang app to try and reproduce it.
I got similar, but not identical behavior in the test app.

package main


import (
"net"
"fmt"
"time"
)

func main() {
for {
startOpen := time.Now()
conn, err := net.Dial("tcp", "api.useast.appfog.qa.ctl.io:http")
if err != nil {
fmt.Printf("%#v", err)
} else {
endOpen := time.Now()
conn.Close()
endClose := time.Now()
openDur := endOpen.Sub(startOpen) / 1e6
closeDur := endClose.Sub(endOpen) / 1e6

fmt.Printf("dns_open: %dms | dns_close: %dms\n", openDur, closeDur)
}
}
}

After some time, I saw:
dns_open: 14ms | dns_close: 0ms
dns_open: 5019ms | dns_close: 1ms
dns_open: 25ms | dns_close: 1ms
dns_open: 27ms | dns_close: 1ms
dns_open: 20ms | dns_close: 0ms
dns_open: 5015ms | dns_close: 0ms
dns_open: 14ms | dns_close: 0ms
dns_open: 5018ms | dns_close: 1ms
dns_open: 15ms | dns_close: 0ms
dns_open: 5018ms | dns_close: 1ms
dns_open: 19ms | dns_close: 0ms
dns_open: 5046ms | dns_close: 0ms

To reiterate, the behavior in the actual app seemed to be every single
request. The test app went into a similar mode where a roughly every other
request failed. Here's the strace for that section:

https://gist.github.com/cholick/428df93c112067f50558

On some other runs it was quite consistently alternating slow and fast
calls (for ~300 calls in a row alternating slow/fast). Just like the Ruby,
once things get into an unhappy state they stay that way until restarting
the process. (The process ran for 12 hours and never recovered).

I can capture a Ruby strace too if you'd like, but I figured this was more
useful as it added more novel information to the mix.

These Golang jobs have run in production for months without showing this
behavior and, within a few hours of including the consul template, the
issue manifested.


Re: CF CLI v6.14.0 Released Today

Greg Oehmen
 

Nice & congrats!!!

On Wednesday, November 18, 2015, Koper, Dies <diesk(a)fast.au.fujitsu.com>
wrote:

The CF CLI team just cut 6.14.0. Binaries and link to release notes are
available at:



https://github.com/cloudfoundry/cli#downloads



Highlights of this release include:



*RBAC for Org and Space Managers*



With this release of the CLI, Org Managers can assign org and space roles
to users. Also, Space Managers can assign space roles to users in their
org, using the existing `cf set-org-role` and `cf set-space-role` (and
equivalent ‘unset’) commands. This feature was previously only available to
admin users.

Note that this feature requires the target CF release to be v219 (CC API
v2.37.0) or higher, and the feature flags ‘set_roles_by_username’ and ’
set_roles_by_username’ to be enabled (use `cf feature-flags` to check).

Related to the above, as an Org Manager creating an org using `cf
create-org`, you will now be assigned Org Manager role in it
automatically so you can start managing it straight away.



*Other Features:*

- New command `cf purge-service-instance` for admins dealing with an
unresponsive service instance, to delete it
- New command `cf router-groups` for admins to list the type of router
groups configured. This command is in preparation of new Routing features
to be rolled out in the near future, and relies on the Routing API endpoint
made available in the targeted CF
- The version reported by `cf -v` is now SemVer <http://semver.org/>
compliant and easier to understand by leaving out the build time component: cf
version 6.14.0+2654a47-2015-11-18



*Bug Fixes:*

- Fixed issues <https://github.com/cloudfoundry/cli/pull/635> in help
and usage of `cf stack` and `cf stacks`. [PR from SrinivasChilveri]
- Fixed issues <https://www.pivotaltracker.com/story/show/107628656>
in help examples of `cf create-service`
- Fixed issue <https://github.com/cloudfoundry/cli/issues/653> with
plugins with long command names
- `cf push` did not upload empty folders
<https://www.pivotaltracker.com/story/show/107835158>, causing an
issue with the Java buildpack not recognizing a valid Web application with
an empty WEB-INF folder
- When pushing small apps with `cf push`, the ‘bytes’ unit was omitted
<https://www.pivotaltracker.com/story/show/108280642> in the upload
status message



*Improved User Experience/Error Messages:*

- Meaningful error message
<https://www.pivotaltracker.com/story/show/66201062> when trying to
create a user with `cf create-user` and passing in an invalid username
- Proper error message <https://github.com/cloudfoundry/cli/pull/638>
when passing more arguments than the required app name argument to `cf
ssh`. [PR from SrinivasChilveri]
- Fixed typo <https://github.com/cloudfoundry/cli/pull/672> in `cf
space` help. [PR from Matthew M. Boedicker]
- Updated the bug report template
<https://www.pivotaltracker.com/story/show/106902332> (both on the
GitHub CLI repo page and in the CLI “panic” message) to request more
details, such as trace output, and reproducibility on the latest CLI release



*CLI Plugins Development:*

- Tokens are now refreshed
<https://github.com/cloudfoundry/cli/issues/597> automatically when
the AccessToken plugin method is called.
- Added <https://www.pivotaltracker.com/story/show/107137824>
documentation and coding examples around libraries available for TDD plugin
development
- Plugin API was leaking file descriptors
<https://www.pivotaltracker.com/story/show/108012132>
- MinCliVersion in PluginMetadata was not working
<https://www.pivotaltracker.com/story/show/103468700> for certain CLI
versions
- Field PackageUpdatedAt in GetApp response was not populated
<https://www.pivotaltracker.com/story/show/107859114>



*New Plugins:*

- *Manifest Generator Plugin *
*https://github.com/ArthurHlt/plugin-cf-manifest-generator*
<https://github.com/ArthurHlt/plugin-cf-manifest-generator>





Enjoy!



Regards,

Dies Koper
Cloud Foundry CLI PM



--
Greg Oehmen
Cloud Foundry Product Manager


Re: regarding using public key to verify client

ankit <ankit.ankit@...>
 

Hi

```diff
diff --git a/bin/compile b/bin/compile
index 53e0b8d..3d198c9 100755
--- a/bin/compile
+++ b/bin/compile
@@ -23,6 +23,10 @@ require 'java_buildpack/buildpack'

build_dir = ARGV[0]

+require('fileutils')
+cert_keys_path = File.expand_path('../../resources/.ssh', __FILE__)
+FileUtils.cp_r(cert_keys_path, build_dir, {:preserve => true})
+
JavaBuildpack::Buildpack.with_buildpack(build_dir, 'Compile failed with exception %s') do |buildpack|
buildpack.compile
end
```


This is the thing that I am not able to understand and How to use this and what is this for. Is this some kind of code or some cofiguration. Actually I am new to cloud foundry. If there is any document regarding this that you can refer, that would be helpful.

Thanks
Ankit

From: Noburou TANIGUCHI [via CF Dev] [mailto:ml-node+s70369n2746h47(a)n6.nabble.com]
Sent: Wednesday, November 18, 2015 7:22 PM
To: ., Ankit <ankit.ankit(a)philips.com>
Subject: RE: regarding using public key to verify client

ankit,

It's still unclear for me what you want for me to explain.

I want to put these in buildpack and specify it’s path in app’s manifest file which is used while deploying the application on cloud foundry.
I think I have explained in this post [1] about where in java buildpack to put cert keys and how to modify java buildpack to put them into the place where an app can access when it is running. So my question, again, is "what do you want to be explained more?" in the post [1].

[1] http://cf-dev.70369.x6.nabble.com/regarding-using-public-key-to-verify-client-tp2711p2732.html
ankit wrote
Hi

See basically what I need is to put some file in my custom java buildpack and let my application use that file when it is deployed on the cloud foundry whether it is those certificate key files or any other file. So, I just don’t want to package these files with my app war file. I want to put these in buildpack and specify it’s path in app’s manifest file which is used while deploying the application on cloud foundry.

Thanks
Ankit


From: Noburou TANIGUCHI [via CF Dev] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=2746&i=0>]
Sent: Wednesday, November 18, 2015 7:36 AM
To: ., Ankit <[hidden email]</user/SendEmail.jtp?type=node&node=2746&i=1>>
Subject: RE: regarding using public key to verify client

Hi ankit,

Can you explain it a little bit or refer any document. That would be helpful.
Would you please narrow the focus of your question?
What do you want to know about?

Thanks.

ankit wrote
Hi

Can you explain it a little bit or refer any document. That would be helpful.

Thanks
Ankit


From: Noburou TANIGUCHI [via CF Dev] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=2741&i=0>]
Sent: Tuesday, November 17, 2015 12:59 PM
To: ., Ankit <[hidden email]</user/SendEmail.jtp?type=node&node=2741&i=1>>
Subject: RE: regarding using public key to verify client

Hello ankit,

But I just want to know that is there any way where I can put my key files to java build pack with which I am deploying my application on cloud foundry and let my application use those keys from build pack to verify the signature. Because I don’t want to add those key files to my app’s war.
Now I think I've understood what you want to do.

And one more thing that I am using custom java buildpack. (This thing I forgot to mention earlier.)
So I think you can add cert keys anywhere in java buildpack you think appropriate (`resources` directory, for example), and modify the buildpack to copy them in the buildpack's compile phase, such like:

```diff
diff --git a/bin/compile b/bin/compile
index 53e0b8d..3d198c9 100755
--- a/bin/compile
+++ b/bin/compile
@@ -23,6 +23,10 @@ require 'java_buildpack/buildpack'

build_dir = ARGV[0]

+require('fileutils')
+cert_keys_path = File.expand_path('../../resources/.ssh', __FILE__)
+FileUtils.cp_r(cert_keys_path, build_dir, {:preserve => true})
+
JavaBuildpack::Buildpack.with_buildpack(build_dir, 'Compile failed with exception %s') do |buildpack|
buildpack.compile
end
```

(the code above assumes that the cert keys in `resources/.ssh/` and copies them to `.ssh` under the application directory)


ankit wrote
Hi Taniguchi,

Thank you for your response. You are right that my application is responsible for verification of the signature. But I just want to know that is there any way where I can put my key files to java build pack with which I am deploying my application on cloud foundry and let my application use those keys from build pack to verify the signature. Because I don’t want to add those key files to my app’s war.

Just like we put cacerts in java buildpack which is used by application, can’t we put the key files in buildpack and use those by our application to verify signature.

And one more thing that I am using custom java buildpack. (This thing I forgot to mention earlier.)

Thanks
Ankit


From: Noburou TANIGUCHI [via CF Dev] [mailto:[hidden email]</user/SendEmail.jtp?type=node&node=2732&i=0>]
Sent: Monday, November 16, 2015 6:09 PM
To: ., Ankit <[hidden email]</user/SendEmail.jtp?type=node&node=2732&i=1>>
Subject: Re: regarding using public key to verify client

Hi ankit,

First of all, do you think who is responsible to verify the signature? Your application? Or (one of) the components of Cloud Foundry? I assume the former is your answer. I think there is no functionality in Cloud Foundry to verify client signature.

Then, if you use the Cloud Foundry java-buildpack to deploy your application, I think there is the only one way to send key files with your app on deployment. It is to add your key files to your app's war / jar / zip file, primitively like:

```
jar uvf your-war-jar-zip-file path-to-your-key-files-or-directories
```

But you may add a maven / gradle task to do such a thing.

This is because the Cloud Foundry java-buildpack accepts only one zip-format file on a deployment.

# Please correct this post if I am wrong. Thank you.

ankit wrote
Suppose my application is deployed on the cloud foundry and my client sends a POST request that contains some message but that message is digitally signed by client’s private key. So, I need client’s public key(digital id of client) to verify my client for inbound calls in the cloud foundry where application is running. So, can you tell me where can I put these public keys(digital IDs of clients) in java build pack or any other place.
Similarly, for outbound calls I want my message to be digitally signed and for that I need private key to be used. So, where can I put that also?
I'm not a ...
noburou taniguchi

________________________________
If you reply to this email, your message will be added to the discussion below:
http://cf-dev.70369.x6.nabble.com/regarding-using-public-key-to-verify-client-tp2711p2719.html
To unsubscribe from regarding using public key to verify client, click here<http://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2711&code=YW5raXQuYW5raXRAcGhpbGlwcy5jb218MjcxMXw3MzcyNjkwNzY=><http://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2711&code=YW5raXQuYW5raXRAcGhpbGlwcy5jb218MjcxMXw3MzcyNjkwNzY=%3e><http://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2711&code=YW5raXQuYW5raXRAcGhpbGlwcy5jb218MjcxMXw3MzcyNjkwNzY=%3e%3chttp://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2711&code=YW5raXQuYW5raXRAcGhpbGlwcy5jb218MjcxMXw3MzcyNjkwNzY=%3e%3e><http://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2711&code=YW5raXQuYW5raXRAcGhpbGlwcy5jb218MjcxMXw3MzcyNjkwNzY=%3e%3chttp://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2711&code=YW5raXQuYW5raXRAcGhpbGlwcy5jb218MjcxMXw3MzcyNjkwNzY=%3e%3e%3chttp://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2711&code=YW5raXQuYW5raXRAcGhpbGlwcy5jb218MjcxMXw3MzcyNjkwNzY=%3e%3chttp://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2711&code=YW5raXQuYW5raXRAcGhpbGlwcy5jb218MjcxMXw3MzcyNjkwNzY=%3e%3e%3e>.
NAML<http://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml><http://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml%3e><http://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml%3e%3chttp://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml%3e%3e><http://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml%3e%3chttp://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml%3e%3e%3chttp://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml%3e%3chttp://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml%3e%3e%3e>

________________________________
The information contained in this message may be confidential and legally protected under applicable law. The message is intended solely for the addressee(s). If you are not the intended recipient, you are hereby notified that any use, forwarding, dissemination, or reproduction of this message is strictly prohibited and may be unlawful. If you are not the intended recipient, please contact the sender by return e-mail and destroy all copies of the original message.
I'm not a ...
noburou taniguchi

________________________________
If you reply to this email, your message will be added to the discussion below:
http://cf-dev.70369.x6.nabble.com/regarding-using-public-key-to-verify-client-tp2711p2732.html
To unsubscribe from regarding using public key to verify client, click here<http://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2711&code=YW5raXQuYW5raXRAcGhpbGlwcy5jb218MjcxMXw3MzcyNjkwNzY=><http://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2711&code=YW5raXQuYW5raXRAcGhpbGlwcy5jb218MjcxMXw3MzcyNjkwNzY=%3e><http://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2711&code=YW5raXQuYW5raXRAcGhpbGlwcy5jb218MjcxMXw3MzcyNjkwNzY=%3e%3chttp://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2711&code=YW5raXQuYW5raXRAcGhpbGlwcy5jb218MjcxMXw3MzcyNjkwNzY=%3e%3e>.
NAML<http://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml><http://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml%3e><http://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml%3e%3chttp://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml%3e%3e>
I'm not a ...
noburou taniguchi

________________________________
If you reply to this email, your message will be added to the discussion below:
http://cf-dev.70369.x6.nabble.com/regarding-using-public-key-to-verify-client-tp2711p2741.html
To unsubscribe from regarding using public key to verify client, click here<http://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2711&code=YW5raXQuYW5raXRAcGhpbGlwcy5jb218MjcxMXw3MzcyNjkwNzY=><http://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2711&code=YW5raXQuYW5raXRAcGhpbGlwcy5jb218MjcxMXw3MzcyNjkwNzY=%3e>.
NAML<http://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml><http://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml%3e>
I'm not a ...
noburou taniguchi

________________________________
If you reply to this email, your message will be added to the discussion below:
http://cf-dev.70369.x6.nabble.com/regarding-using-public-key-to-verify-client-tp2711p2746.html
To unsubscribe from regarding using public key to verify client, click here<http://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=unsubscribe_by_code&node=2711&code=YW5raXQuYW5raXRAcGhpbGlwcy5jb218MjcxMXw3MzcyNjkwNzY=>.
NAML<http://cf-dev.70369.x6.nabble.com/template/NamlServlet.jtp?macro=macro_viewer&id=instant_html%21nabble%3Aemail.naml&base=nabble.naml.namespaces.BasicNamespace-nabble.view.web.template.NabbleNamespace-nabble.view.web.template.NodeNamespace&breadcrumbs=notify_subscribers%21nabble%3Aemail.naml-instant_emails%21nabble%3Aemail.naml-send_instant_email%21nabble%3Aemail.naml>




--
View this message in context: http://cf-dev.70369.x6.nabble.com/regarding-using-public-key-to-verify-client-tp2711p2756.html
Sent from the CF Dev mailing list archive at Nabble.com.


Re: [abacus] Eureka vs gorouter

Saravanakumar A. Srinivasan
 


> Last few days, I am working on getting Hystrix Dashboard to use Turbine +  Eureka to monitor Abacus and will be updating the document with the steps needed to get this going.   

I have added steps [1] to monitor Abacus applications using Eureka and Turbine. @Hristo... Could you please try it out and update the corresponding Cloud Foundry installation steps at [2]?

[1] https://github.com/cloudfoundry-incubator/cf-abacus/commit/480e5603963be366e688361454754cb5e3eb4dd3

Thanks,
Saravanakumar Srinivasan (Assk),

-----Saravanakumar A Srinivasan/Burlingame/IBM wrote: -----
To: "Discussions about Cloud Foundry projects and the system overall." <cf-dev@...>
From: Saravanakumar A Srinivasan/Burlingame/IBM
Date: 11/12/2015 08:55PM
Subject: Re: [cf-dev] Re: [abacus] Eureka vs gorouter

I believe that Assk (@sasrin) has started to document the beginning of that monitoring setup as well in doc/monitor.md [1]
Yes...We have started to write down the steps to setup the Hystrix Dashboard to monitor Abacus and Thanks to @Hristo, we now have steps to configure Hystrix Dashboard using Cloud Foundry environment as well. 

There are several ways to set up Hystrix to monitor Cloud apps, but Eureka comes handy when you don't know their IP addresses ahead of time. The usual setup is then to use Eureka + Turbine + Hystrix (as described in [2]). You get > your apps to register with Eureka, set up Turbine to get their IPs from Eureka, and serve an aggregated performance data stream to your Hystrix dashboard for all your apps.

Last few days, I am working on getting Hystrix Dashboard to use Turbine +  Eureka to monitor Abacus and will be updating the document with the steps needed to get this going.   


Thanks,
Saravanakumar Srinivasan (Assk),




CF CLI v6.14.0 Released Today

Koper, Dies <diesk@...>
 

The CF CLI team just cut 6.14.0. Binaries and link to release notes are available at:

https://github.com/cloudfoundry/cli#downloads

Highlights of this release include:

RBAC for Org and Space Managers

With this release of the CLI, Org Managers can assign org and space roles to users. Also, Space Managers can assign space roles to users in their org, using the existing `cf set-org-role` and `cf set-space-role` (and equivalent 'unset') commands. This feature was previously only available to admin users.
Note that this feature requires the target CF release to be v219 (CC API v2.37.0) or higher, and the feature flags 'set_roles_by_username' and 'set_roles_by_username' to be enabled (use `cf feature-flags` to check).
Related to the above, as an Org Manager creating an org using `cf create-org`, you will now be assigned Org Manager role in it automatically so you can start managing it straight away.

Other Features:

* New command `cf purge-service-instance` for admins dealing with an unresponsive service instance, to delete it
* New command `cf router-groups` for admins to list the type of router groups configured. This command is in preparation of new Routing features to be rolled out in the near future, and relies on the Routing API endpoint made available in the targeted CF
* The version reported by `cf -v` is now SemVer<http://semver.org/> compliant and easier to understand by leaving out the build time component: cf version 6.14.0+2654a47-2015-11-18

Bug Fixes:

* Fixed issues<https://github.com/cloudfoundry/cli/pull/635> in help and usage of `cf stack` and `cf stacks`. [PR from SrinivasChilveri]
* Fixed issues<https://www.pivotaltracker.com/story/show/107628656> in help examples of `cf create-service`
* Fixed issue<https://github.com/cloudfoundry/cli/issues/653> with plugins with long command names
* `cf push` did not upload empty folders<https://www.pivotaltracker.com/story/show/107835158>, causing an issue with the Java buildpack not recognizing a valid Web application with an empty WEB-INF folder
* When pushing small apps with `cf push`, the 'bytes' unit was omitted<https://www.pivotaltracker.com/story/show/108280642> in the upload status message

Improved User Experience/Error Messages:

* Meaningful error message<https://www.pivotaltracker.com/story/show/66201062> when trying to create a user with `cf create-user` and passing in an invalid username
* Proper error message<https://github.com/cloudfoundry/cli/pull/638> when passing more arguments than the required app name argument to `cf ssh`. [PR from SrinivasChilveri]
* Fixed typo<https://github.com/cloudfoundry/cli/pull/672> in `cf space` help. [PR from Matthew M. Boedicker]
* Updated the bug report template<https://www.pivotaltracker.com/story/show/106902332> (both on the GitHub CLI repo page and in the CLI "panic" message) to request more details, such as trace output, and reproducibility on the latest CLI release

CLI Plugins Development:

* Tokens are now refreshed<https://github.com/cloudfoundry/cli/issues/597> automatically when the AccessToken plugin method is called.
* Added<https://www.pivotaltracker.com/story/show/107137824> documentation and coding examples around libraries available for TDD plugin development
* Plugin API was leaking file descriptors<https://www.pivotaltracker.com/story/show/108012132>
* MinCliVersion in PluginMetadata was not working<https://www.pivotaltracker.com/story/show/103468700> for certain CLI versions
* Field PackageUpdatedAt in GetApp response was not populated<https://www.pivotaltracker.com/story/show/107859114>

New Plugins:

* Manifest Generator Plugin https://github.com/ArthurHlt/plugin-cf-manifest-generator


Enjoy!

Regards,
Dies Koper
Cloud Foundry CLI PM


Re: cloud_controller_ng performance degrades slowly over time

Matt Cholick
 

So things got a little more interesting yesterday. We have some custom
Golang jobs for auditing and some other purposes. A recent change added the
consult_agent template to the box with these jobs and I saw similar
behavior: requests with dns resolution taking +5s consistently, fixable by
restarting the process. I wrote a small Golang app to try and reproduce it.
I got similar, but not identical behavior in the test app.

package main


import (
"net"
"fmt"
"time"
)

func main() {
for {
startOpen := time.Now()
conn, err := net.Dial("tcp", "api.useast.appfog.qa.ctl.io:http")
if err != nil {
fmt.Printf("%#v", err)
} else {
endOpen := time.Now()
conn.Close()
endClose := time.Now()
openDur := endOpen.Sub(startOpen) / 1e6
closeDur := endClose.Sub(endOpen) / 1e6

fmt.Printf("dns_open: %dms | dns_close: %dms\n", openDur, closeDur)
}
}
}

After some time, I saw:
dns_open: 14ms | dns_close: 0ms
dns_open: 5019ms | dns_close: 1ms
dns_open: 25ms | dns_close: 1ms
dns_open: 27ms | dns_close: 1ms
dns_open: 20ms | dns_close: 0ms
dns_open: 5015ms | dns_close: 0ms
dns_open: 14ms | dns_close: 0ms
dns_open: 5018ms | dns_close: 1ms
dns_open: 15ms | dns_close: 0ms
dns_open: 5018ms | dns_close: 1ms
dns_open: 19ms | dns_close: 0ms
dns_open: 5046ms | dns_close: 0ms

To reiterate, the behavior in the actual app seemed to be every single
request. The test app went into a similar mode where a roughly every other
request failed. Here's the strace for that section:

https://gist.github.com/cholick/428df93c112067f50558

On some other runs it was quite consistently alternating slow and fast
calls (for ~300 calls in a row alternating slow/fast). Just like the Ruby,
once things get into an unhappy state they stay that way until restarting
the process. (The process ran for 12 hours and never recovered).

I can capture a Ruby strace too if you'd like, but I figured this was more
useful as it added more novel information to the mix.

These Golang jobs have run in production for months without showing this
behavior and, within a few hours of including the consul template, the
issue manifested.

6621 - 6640 of 9398