|
Re: Loggregator/Doppler Syslog Drain Missing Logs
Erik Jasiak
Hi Michael
toggle quoted messageShow quoted text
First question that springs to mind when I see ~50% - how many zones are you running as part of your setup? ("every other log" sounds like a round-robin to something dead or misconfigured.) Have to run but will follow up more soon, Erik Michael Schwartz wrote:
|
|
|
|
Re: Loggregator/Doppler Syslog Drain Missing Logs
Michael Schwartz
The system is currently running ~200 apps and they all bind to an external syslog drain.
|
|
|
|
Loggregator/Doppler Syslog Drain Missing Logs
Michael Schwartz
Loggregator appears to be dropping logs without notice. I'm noticing about 50% of the logs do not make it to our external log service. If I tail the logs using "cf logs ...", all logs are visible. They just never make it to the external drain.
I have seen the "TB: Output channel too full. Dropped 100 messages for app..." message in the doppler logs, and that makes sense for applications producing many logs. What's confusing is that I'm seeing logs missing from apps that produce very little logs (like 10 per minute). If I send curl requests to a test app very slowly, I notice approx. every other log is missing. We've been using an ELK stack for persisting application logs. All applications bind to a user-provided service containing the syslog drain URL. Logstash does not appear to be the bottleneck here because I've tested other endpoints such as a netcat listener and I see the same issue. Ever after doubling our logstash server count to 6, I see the exact same drop rate. Our current CF (v210) deployment contains 4 loggregator instances each running doppler, syslog_drain_binder, and metron_agent. I tried bumping the loggregator instance count to 8 and noticed very little improvement. Monitoring CPU, memory, and diskspace on loggregator nodes show no abnormalities. CPU is under 5%. Is this expected behavior? Thank you.
|
|
|
|
Re: Introducing CF-Swagger
Deepak Vij
Hi Mohamed and Dr. Max, I fully support this effort. By having Swagger based “Application Interface” capability as part of the overall CF PaaS platform would be very useful for the CF community as a whole. As a matter of fact, I also initiated a similar thread few months ago on cf-dev alias (see email text below). Your work exactly matches up with what our current thinking is.
By having “Swagger” based “Application Interface” is a very good start along those lines. This opens up lots of other possibilities such as building out “Deployment Governance” capabilities not merely for Cloud Foundry API or Services assets but for the whole Application landscape built & deployed within CF PaaS environment and subsequently exposed as APIs to end consumers. As described below in my email I sent out earlier that “Deployment Governance” as part of overall API Management is what we are striving towards in order to expose comprehensive telecom API Management capabilities within the public cloud environment. Dr. Max, as I mentioned to you during our brief discussion few days ago that “Heroku” folks also have a similar initiative ongoing. They have gone lightweight “JSON” schema route versus Swagger/WADL/RAML etc. In any case, I am fully in support of your proposal. Thanks. Regards, Deepak Vij ============================= Hi folks, I would like to start a thread on the need for machine-readable “Application Interface” supported at the platform level. Essentially, this interface describes details such as available methods/operations, inputs/outputs data types (schema), application dependencies etc. Any standard specifications language can be used for this purpose, as long as it clearly describes the schema of the requests and responses – one can use Web Application Description Language (WADL), Swagger, RESTful API Modeling Language (RAML), JSON Schema (something like JSON Schema for Heroku Platform APIs) or any other language that provides similar functionality. These specifications are to be automatically derived from the code and are typically part of the application development process (e.g. generated by the build system). Such functionality can have lots of usage scenarios: 1. First and foremost, Deployment Governance for API Management (our main vested interest) – API Versioning & Backward Compatibility, Dependency Management and many more as part of the comprehensive telecom API Management capabilities which we are currently in the process of building out. 2. Auto-creating client libraries for your favorite programming language. 3. Automatic generation of up-to-date documentation. 4. Writing automatic acceptance and integration tests etc. From historical perspective, in the early 2000s when SOA started out, the mindset was to author the application contract-first (application interface using WSDL at that time) and subsequently generate and author code from the application interface. With the advent of RESTful services, REST community initially took a stand against such metadata for applications. Although, a number of metadata standards have none-the-less emerged over the last couple of years, mainly fueled by the use case scenarios described earlier. Based on my knowledge, none of this currently exists within Cloud Foundry at the platform level. It would be highly desirable to have a standard common “application interface” definition at the platform level, agnostic of the underlying application development frameworks. I hope this all makes sense. I think this is something could be very relevant to the “Utilities” PMC. I will also copy&paste this text under “Utilities” PMC-notes on the github. I would love to hear from the community on this. Thanks. Regards, Deepak Vij From: Michael Maximilien [mailto:maxim(a)us.ibm.com] Sent: Friday, September 18, 2015 4:52 PM To: cf-dev(a)lists.cloudfoundry.org Cc: Heiko Ludwig; Mohamed Mohamed; Alex Tarpinian; Christopher B Ferris Subject: [cf-dev] Introducing CF-Swagger Hi, all, This email serves two purposes: 1) introduce CF-Swagger, and 2) shares the results of the CF service broker compliance survey I sent out a couple of weeks ago. ------ My IBM Research colleague, Mohamed (on cc:), and I have been working on creating Swagger descriptions for some CF APIs. Our main goal was to explore what useful tools or utilities we could build with these Swagger descriptions once created. The initial results of this exploratory research is CF-Swagger which is included in the following: See presentation here: https://goo.gl/Y16plT Video demo here: http://goo.gl/C8Nz5p Temp repo here: https://github.com/maximilien/cf-swagger The gist of of our work and results are: 1. We created a full Swagger description of the CF service broker 2. Using this description you can use the Swagger editor to create a neat API docs that is browsable and even callable 3. Using the description you can create client and server stubs for service brokers in a variety of languages, e.g., JS, Java, Ruby, etc. 4. We've extended go-swagger to generate workable client and server stubs for service brokers in Golang. We plan to submit all changes to go-swagger back to that project 5. We've extended go-swagger to generate prototypes of working Ginkgo tests to service brokers 6. We've extended go-swagger to generate a CF service broker Ginkgo Test Compliance Kit (TCK) that anyone could use to validate their broker's compliance with any Swagger-described version of spec 7. We've created a custom Ginkgo reporter that when ran with TCK will give you a summary of your compliance, e.g., 100% compliant with v2.5 but 90% compliant with v2.6 due to failing test X, Y, Z... (in Ginkgo fashion) 8. The survey results (all included in the presentation) indicate that over 50% of respondants believe TCK tests for service broker would be valuable to them. Many (over 50%) are using custom proprietary tests, and this project maybe a way to get everyone to converge to a common set of tests we could all use and improve... ------ We plan to propose this work to become a CF incubator at the next CAB and PMC calls, especially the TCK part for service brokers. The overall approach and project could be useful for other parts of the CF APIs but we will start with CF Service Brokers. The actual Swagger descriptions should ideally come from the teams who own the APIs. So for service brokers, the CAPI team. We are engaging them as they have also been looking at improving APIs docs and descriptions. Maybe there are potential for synergies and at a minimum making sure what we generate ends up becoming useful to their pipelines. Finally, while the repo is temporary and will change, I welcome you to take a look at presentation and video and code and let us know your thoughts and feedback. Thanks for your time and interest. Mohamed and Max IBM
|
|
|
|
Re: Error 400007: `stats_z1/0' is not running after update
Amit Kumar Gupta
I often take the following approach to debugging issues like this:
* Open two shell sessions to your failing VM using bosh ssh, and switch to superuser * In one session, `watch monit summary`. You might see collector going back and forth between initializing and not monitored, but please report anything else of interest you see here * In the other session, `cd /var/vcap/sys/log` and then `watch --differences=cumulative ls -altr **/*` to see which files are being written to while the startup processes are thrashing. Then `tail -f FILE_1 FILE_2 ...` listing all the files that were being written to, and seem relevant to the thrashing process(es) in monit On Wed, Sep 23, 2015 at 12:21 AM, Guangcai Wang <guangcai.wang(a)gmail.com> wrote: It frequently logs the message below. It seems not helpful.
|
|
|
|
Re: Removing support for v1 service brokers
Mike Youngstrom <youngm@...>
Thanks Dieu, honestly I was just trying to find an angle to bargain for a
toggle quoted messageShow quoted text
bit more time. :) Three months is generous. But six months would be glorious. :) After the CAB call this month we got started converting our brokers over but our migration is more difficult because we use Service instance credentials quite a bit and those don't appear to be handled well when doing "migrate-service-instances". I think we can do 3 months but we'll be putting our users through a bit of a fire drill. That said I'll understand if you stick to 3 months since, we should have started this conversion log ago. Mike
On Wed, Sep 23, 2015 at 1:22 AM, Dieu Cao <dcao(a)pivotal.io> wrote:
We've found NATS to be unstable under certain conditions, temporary
|
|
|
|
Re: DEA/Warden staging error
kyle havlovitz <kylehav@...>
Here's the output from those commands:
https://gist.github.com/MrEnzyme/36592831b1c46d44f007 Soon after running those I noticed that the container loses its IPv4 address shortly after coming up and ifconfig looks like this: root(a)cf-build:/home/cloud-user/test# ifconfig -a docker0 Link encap:Ethernet HWaddr 56:84:7a:fe:97:99 Any idea what would be causing that? On Tue, Sep 22, 2015 at 10:31 PM, Matthew Sykes <matthew.sykes(a)gmail.com> wrote: Based on your description, it doesn't sound like warden networking or the
|
|
|
|
Re: Curious why CF UAA uses DNS
Filip Hanik
hi Anna,
Can you elaborate a little bit about what you are referring to? I'm not quite sure what you are asking. Filip On Wed, Sep 23, 2015 at 8:23 AM, Anna Muravieva <ana-mur21s(a)yandex.ru> wrote: Hello,
|
|
|
|
Curious why CF UAA uses DNS
Anna Muravieva
Hello, We are using cf product in development. The question relates to uaa, if you coordinate in research will be very appreciated. What are the benefits why CF UAA uses DNS in routes management in opposite to checking this identity for instance in request header. Thanks in advance, Anna
|
|
|
|
Curious why CF UAA uses DNS
Anna Muravieva
Hello,
We are using cf product in development. The question relates to uaa, if you coordinate in research will be very appreciated. What are the benefits why CF UAA uses DNS in routes management in opposite to checking this identity for instance in request header. Thanks in advance, Anna
|
|
|
|
RSA Security Analytics Users List
Mary Lopez <mary.lopez@...>
Hi,
Would you be interested in acquiring the list of users using RSA Security Analytics? We also have some authentic data of other Cloud Computing, ERP, PLM, Analytics software users too. Job Titles - CIO, CTO, Data Center Managers, CSO, Director of IT, IT Security Head, Network Engineer etc. Information Fields - Name, Title, Email, Phone Numbers, Company Name and Company Details like Physical Address, Web Address, Revenue Size, Employee Size and Industry. Reach out with your specific requirement and get a set of free samples. If you are not the right person to discuss this, please forward this email to the right person in your organization. I look forward to hearing from you. Kind Regards, Mary Lopez Business Development Coordinator Dynamics IT Solutions 7800 Shoal Creek Blvd. Suite 230-S Austin, TX 78757 If you do not wish to receive an email from us, please reply "Remove" in the subject line.
|
|
|
|
Re: Security Question --- Securely wipe data on warden container removal / destruction???
Will Pragnell <wpragnell@...>
Guillaume, I'm not aware of any plans for secure memory wiping
toggle quoted messageShow quoted text
specifically, but I can say that another track of security work is one of several candidates for the next phase of work on Garden after OCS/runC integration is completed. That said, such a change may fall outside the remit of the Garden team; it may be a platform wide change that involves changes to the stemcell.
On 23 September 2015 at 13:28, Guillaume Berche <bercheg(a)gmail.com> wrote:
Chris, thanks for bringing up this important security topic.
|
|
|
|
Re: Security group rules to allow HTTP communication between 2 apps deployed on CF
Naveen Asapu
I'm using cf version 6.12.1
|
|
|
|
Re: Security Question --- Securely wipe data on warden container removal / destruction???
Chris, thanks for bringing up this important security topic.
toggle quoted messageShow quoted text
In terms of secrets an app is handling and carrying, I'd think its code has generally limited sensitivity (e.g credentials or API key secrets are rather stored in env vars). I'd expect memory to be much more sensitive (e.g. holding user data), as well as state handed over to data services (12 factor apps are unlikely to store much state on their ephemeral file system). So related to your question about securely wipping data upon app instance deletion, it may be interesting to consider secure RAM wiping when an app container exits (sometimes killed by the oomkiller leaving few opportunity for the app itself to wipe out RAM before exit). See related discussions in [1] [2] [3] [4]. Quickly searching the bosh stemcell builder, and bosh tracker I could not find mention of gresec or pax linux kernel packages/patches that could strengthen RAM wiping after an app instance exits. Will, do you know if is there plans to tackle such kernel hardening ? Related to secrets stored on disk in data services (p-mysql, p-redis), the services should be designed to not provide access to previous deleted service instances when normally functionning. The secured data wiping might be useful if ever the data service itself would get compromised so that an attacker would not be able to access data from deleted service instances after hand. Guillaume. [1] http://security.stackexchange.com/questions/42179/is-there-any-linux-distro-or-kernel-patch-that-wipes-a-process-memory-space-afte [2] https://github.com/coreos/bugs/issues/332#issuecomment-109293958 [3] https://en.wikibooks.org/wiki/Grsecurity/Appendix/Grsecurity_and_PaX_Configuration_Options#Sanitize_all_freed_memory [4] https://blog.docker.com/2013/08/containers-docker-how-secure-are-they/#other-kernel-security-features
On Thu, Sep 17, 2015 at 1:38 PM, Will Pragnell <wpragnell(a)pivotal.io> wrote:
In Diego/Garden, container files are stored on btrfs subvolumes. When a
|
|
|
|
Re: How to deploy a Web application using HTTPs
Juan Antonio Breña Moral <bren at juanantonio.info...>
@James,
who add the headers? "x-forwarded-for":"CLIENT_REAL_IP, CLOUD_FOUNDRY_IP", "x-forwarded-proto":"https" the load balancer or the GoRouter?
|
|
|
|
Re: Security group rules to allow HTTP communication between 2 apps deployed on CF
Denilson Nastacio <dnastacio@...>
The message indicates this problem is unrelated to security groups. You
toggle quoted messageShow quoted text
would get something like "host not found" instead of "connection refused". Which version of CF are you using? Can you curl a url from app2 at all?
On Wed, Sep 23, 2015, 3:27 AM Naveen Asapu <asapu.naveen(a)gmail.com> wrote:
Hi Matthew Sykes,
|
|
|
|
Re: How to deploy a Web application using HTTPs
Juan Antonio Breña Moral <bren at juanantonio.info...>
Hi James,
Now, understood your technical explanation: "the standard way to do this is to terminate SSL at a load balancer, which then forwards to the CF routing tier. the hop between the load balancer and the cf router may be done with SSL. the network path from gorouter to the DEA / Diego Cell backend is only supported with http today." "app client ---HTTPS---> LB ---HTTPS---> GoRouter ---HTTP---> DEA/DiegoCell" Cloud foundry supports SSL connections, but currently GoRouter only handle http. I checked the idea and I noticed that when I deploy an application, the platform add the following http headers: "x-forwarded-for":"CLIENT_REAL_IP, CLOUD_FOUNDRY_IP", "x-forwarded-proto":"https" So, if you only want to execute an API for example with https, it is necessary to filter with this header: "x-forwarded-proto":"https" (The idea from Matthew Sykes) I think that it is necessary to create another issue to add the support for http2 I checked, but if fails, the same reason: https://github.com/jabrena/CloudFoundryLab/blob/master/Node_HelloWorld_http2/index.js
|
|
|
|
Re: Avoid some folder or files using the command cf push
Juan Antonio Breña Moral <bren at juanantonio.info...>
Many thanks for the info, I will check the file: .cfignore
http://docs.pivotal.io/pivotalcf/devguide/deploy-apps/prepare-to-deploy.html Juan Antonio
|
|
|
|
Re: Avoid some folder or files using the command cf push
Chunhua Zhang <chzhang@...>
please ref to :
toggle quoted messageShow quoted text
https://docs.cloudfoundry.org/devguide/deploy-apps/manifest.htmlHow cf push Finds the Application By default, cf push recursively pushes the contents of the current working directory. Alternatively, you can provide a path using either a manifest or a command line option. - If the path is to a directory, cf push recursively pushes the contents of that directory instead of the current working directory. - If the path is to a file, cf push pushes only that file. *Note*: If you want to push more than a single file, but not the entire contents of a directory, consider using a .cfignore file to tell cf push what to exclude. 2015-09-23 16:08 GMT+08:00 Juan Antonio Breña Moral <bren(a)juanantonio.info>:
Hi, --
Thanks & Best Regards, chunhua, zhang（张春华） M: ＋86 187 5198 6615 Department: CONSULTING Manager: Leon Cheng IT issue? Mail to: ask(a)pivotal.io
|
|
|
|
Avoid some folder or files using the command cf push
Juan Antonio Breña Moral <bren at juanantonio.info...>
Hi,
sometimes, I deploy applications using CLI with the command cf push. This command uploads the content of a folder and it uses the manifest file. I would like to know if exist some way in the manifest.yml or another file to avoid uploading some folder. For example, if any developer create Node.js Application, the folder node_modules is not necessary to upload because Node.js buildpack is able to read and download the required dependencies described in the file package.json Does exist some way to do it? Many thanks in advance. Juan Antonio
|
|
|