Re: Questions of how to add projects to Cloud Foundry Community repo
Dr Nic Williams <drnicwilliams@...>
Layne, if you want to start with cloudfoundry-community then you can ask
toggle quoted messageShow quoted text
any existing member/owner to add you, or me if you don't know anyone else from the 160+ list. Send me/them your github ID. If you also want to create BOSH releases and would like some AWS credentials for Amazon S3, then let me know. Nic
On Fri, May 13, 2016 at 2:51 AM, Amit Gupta <agupta(a)pivotal.io> wrote:
Hi Layne, --
Dr Nic Williams Stark & Wayne LLC - consultancy for Cloud Foundry users http://drnicwilliams.com http://starkandwayne.com cell +1 (415) 860-2185 twitter @drnic
|
|
Re: Ubuntu Xenial stemcell and rootfs plans
Daniel Mikusa
On Thu, May 12, 2016 at 12:52 PM, Danny Rosen <drosen(a)pivotal.io> wrote:
Fair enough. Though if you divorce the binary from the buildpack youI respectfully disagree. Build pack binaries are nothing more than packages (just a different format than rpm or deb). Any competent Linux administrator should be familiar with package management for their distro of choice, and that includes the ability to have an internal repo (i.e. mirror) for the distro's packages. Essentially the idea here would be to do the same thing but for build pack binaries. Because having internal repos / mirrors is something a lot of large companies do, I suspect many administrators will be familiar with these concepts. I think this change could actually simplify build packs. Currently most of the build packs have ugly hacks in them to intercept requests for external files, translate URLs and load local copies of files instead [1]. The idea that I'm proposing would simply require the build packs to pull files from a repo. It doesn't matter if you are behind a firewall or on the public Internet. You just download the files from a configured repository. Simple and straightforward. [1] - https://github.com/cloudfoundry/compile-extensions/tree/9932bb1d352b88883d76df41e797a6fa556844f0#download_dependency I think we can simplify this conversation a bit though using our *current*Again, I disagree. I don't think you can address them using the current architecture because it's the architecture that's the problem. Bundling binaries with the build packs is at it's core a bad idea. Mike D listed some of these earlier in this email thread. Summarizing the ones that come to mind below. - large build packs are hard to distribute - large build packs increase staging time and in some cases cause staging failures - build packs are tied to the stack of the binaries they include - build packs are tied to specific versions of the binaries they include - supporting multiple sets of binaries requires multiple build packs or really large build packs - customizing build packs becomes more difficult as you now have to wrap up the binaries in your custom build pack - build packs are forced to release to keep up with their binaries, not because the build packs need to change at that pace Separating the binaries and build packs would seem to address these issues. It would also turn binary management into a task that is more similar to what Linux admins do today for their distro's packages. Perhaps we could even piggy back on existing tools in this space to manage the binaries like Artifactory. bundling binaries into the build pack. The original issue on this thread is being able to support the addition of a new stack. Mike D made the point that supporting an additional stack would be difficult because it would cause the size of the build pack to spike. He offered one possible solution, but that looked like it would require work to the cloud controller. I offered the idea of splitting the binaries out of the build pack. It doesn't require any cloud controller work and it would scale nicely as additional stacks are added (assuming you have an HTTP server with a large enough disk). One idea we're throwing around is being able to use a url containing a zipI think that could be helpful. I remember back to the early days of Diego when it could pull in a zip archive and it was nice in certain situations. Having said that, I'm not seeing how this would help with the other issues caused by having build packs and binaries bundled together. In particular, the one of supporting multiple stacks. Dan If you're interested in working with us on this solution, let's talk! We're happy to work with the community.
|
|
Re: Spiff reloaded...
Alexander Lomov <alexander.lomov@...>
Spiff++ looks like a good idea. I've already tried it on my home projects.
Are there any updates on your project? Do you plan to merge changes to commonly used spiff? ------------------------ Alex Lomov *Altoros* — Cloud Foundry deployment, training and integration *Twitter:* @code1n <https://twitter.com/code1n> *GitHub:* @allomov <https://gist.github.com/allomov>
|
|
Re: Ubuntu Xenial stemcell and rootfs plans
Danny Rosen
Fair enough. Though if you divorce the binary from the buildpack you
toggle quoted messageShow quoted text
greatly increase the complexity of the environment. I think we can simplify this conversation a bit though using our *current* architecture rather than creating new paradigms ... and more work for the buildpacks team :) As an operator, I want my users to use custom buildpacks because the official buildpacks (their binaries, their configuration, etc) don't suit my needs. - This can be achieved today! Via: - a proxy - an internet enabled environment - an internal git server One idea we're throwing around is being able to use a url containing a zip file which could enable interesting solutions for operators who prefer the "bring your own buildpacks but not from the internet and don't ask me to upload it as an admin buildpack" solution. If you're interested in working with us on this solution, let's talk! We're happy to work with the community.
On Thu, May 12, 2016 at 12:26 PM, Daniel Mikusa <dmikusa(a)pivotal.io> wrote:
On Thu, May 12, 2016 at 11:59 AM, Danny Rosen <drosen(a)pivotal.io> wrote:Thanks. This is helpful! I'd like get a better understanding of theI don't think it wouldn't allow them to run arbitrary binaries, but it
|
|
Re: Questions of how to add projects to Cloud Foundry Community repo
Amit Kumar Gupta
Hi Layne,
toggle quoted messageShow quoted text
There are three main Cloud Foundry-related organizations in GitHub: cloudfoundry, cloudfoundry-incubator, and cloudfoundry-community. Each organization has many repositories. If there is a specific repository you wish to contribute to, and open a issue on GitHub stating your intent, and hopefully the core maintainers can discuss options with you. Make sure to look for a repository with recent activity. Cheers, Amit
On Thu, May 12, 2016 at 3:15 AM, Peng, Layne <Layne.Peng(a)emc.com> wrote:
Hi,
|
|
Re: Ubuntu Xenial stemcell and rootfs plans
Daniel Mikusa
On Thu, May 12, 2016 at 11:59 AM, Danny Rosen <drosen(a)pivotal.io> wrote:
Thanks. This is helpful! I'd like get a better understanding of theI don't think it wouldn't allow them to run arbitrary binaries, but it would allow them to run arbitrary build packs. If you divorce the binaries from the build pack, you can control the binaries separate in a corporate IT managed, not public repository of binaries. Then users can use any build pack they want so long as it points to the blessed internal repo of trusted binaries. Dan For if an operator wanted to provide users the flexibility of executing
|
|
Re: Ubuntu Xenial stemcell and rootfs plans
Danny Rosen
Thanks. This is helpful! I'd like get a better understanding of the
toggle quoted messageShow quoted text
following: Why would an operator set their environment to be disconnected from the internet if they wanted to enable their users to run arbitrary binaries via a buildpack? For if an operator wanted to provide users the flexibility of executing arbitrary binaries in buildpacks, custom buildpacks can be implemented via an environment with internet *or* by providing a proxy <http://docs.cloudfoundry.org/buildpacks/proxy-usage.html> that would allow custom buildpacks to be deployed <http://docs.cloudfoundry.org/buildpacks/custom.html#deploying-with-custom-buildpacks> with an app.
On Thu, May 12, 2016 at 11:37 AM, Mike Youngstrom <youngm(a)gmail.com> wrote:
See responses inline:
|
|
Re: Ubuntu Xenial stemcell and rootfs plans
Mike Youngstrom
See responses inline:
On Thu, May 12, 2016 at 9:04 AM, Danny Rosen <drosen(a)pivotal.io> wrote: * One of the key value propositions of a buildpack is the lightweightI may be missing something but it was my understanding that buildpacks with binaries included must (unless checking all binaries into git) be added as admin buildpacks which non admin users of CF cannot do. Therefore, if I am a simple user of cloud foundry I cannot customize a buldpack for my one off need without involving an administrator to upload and manage the one off buildpack. If binary dependencies were instead managed in a way like Daniel proposes the process would simply be to fork the buildpack and specifying that git repo when pushing. Completely self service without admin intervention. Making it a lighter weight process. * For some of my customers the binary inclusion policies is too restrictive. -- It's hard for me to understand this point as I do not know yourI've attempted to express that need previously here: https://github.com/cloudfoundry/compile-extensions/issues/7 I don't view this as a major issue but I think it could be something to consider if buildpacks binary management is being reconsidered. Hope those additional details help Mike
|
|
Re: Ubuntu Xenial stemcell and rootfs plans
Danny Rosen
* One of the key value propositions of a buildpack is the lightweight
toggle quoted messageShow quoted text
process to fork and customize a buildpack. *The inclusion of binaries makes buildpack customization a much heavier process and less end user friendly in a number of ways.* -- I'm not sure I agree with this point and would like to understand your reasoning. * For some of my customers the binary inclusion policies is too restrictive. -- It's hard for me to understand this point as I do not know your customers' requirements. Would you mind providing details so we can better understand their needs?
On Wed, May 11, 2016 at 2:01 PM, Mike Youngstrom <youngm(a)gmail.com> wrote:
I really like the idea of finding a way to move away from bundling
|
|
Re: How to install things on specfic APP container
Michal Tekel
Hi,
toggle quoted messageShow quoted text
depending on which buildpack you use it might be easier or more complicated to launch custom scripts at app runtime. In these scripts you can install apps, but only in "userspace" - that is, not as root. This is possible for various ubuntu packages, but it involves manual resolution of dependencies, which all need to be installed in the same userspace. In our case we have run nmap to do port scan from within app container (to verify what everything is reachable by deployed apps). We have used install script [1], which we added into bin/post_compile (python buildpack) - which would run at the end of staging and put installed packages into the final app droplet, making them available inside app container on launch. We then run the scan by using another script [2] where we explicitly define LD_LIBRARY_PATH to point to dependencies that we have also installed in "userspace". This is quite cumbersome, but at least it can be done this way. Some other PAASes support direct installation of package dependencies in their buildpacks [3]. [1] https://github.com/alphagov/paas-cf/blob/c0db1e38a9294112b8ecbfd7e0eee3dea5cf94ac/tests/example-apps/port-scan/nmap_portable.sh [2] https://github.com/alphagov/paas-cf/blob/c0db1e38a9294112b8ecbfd7e0eee3dea5cf94ac/tests/example-apps/port-scan/scan.sh [3] https://docs.tsuru.io/stable/using/python.html - see requirements.apt file description
On 7 May 2016 at 03:52, Stanley Shen <meteorping(a)gmail.com> wrote:
Yes, the file is actually stored in database, we don't rely on the FS of
|
|
[abacus] Separate time-based from discrete usage metrics
Hi,
We're trying to fix Abacus issue 88: Missing aggregated usage for the running application [1]. Background ========= See the jsdelfino comment in the GitHub issue [2]. TL;DR: Resource providers have to send a 'ping' doc per month for time-based metrics. Proposed solution ============== We decided to implement a solution in Abacus that frees the usage providers from sending the 'ping' submission. To fix the issue we decided to: 1. Distinguish between time-based (linux-container) and discrete usage metrics (the rest basically) 2. Store the time-based metrics in a separate DB(s) We already drafted a proposal for adding measurement type in the usage plans with PR #320 [3]. We're about to spike on storing the time-based metrics in their own Database, but we wanted to get the community opinion on the topic. Motivation ======== The discrete usage submitted to Abacus is: * stored in partitioned databases, due to their size/number * like an event log, storing the history of the usage/resources In contrast the current time-based metrics are: * limited number (usually around 2 million on a loaded CF system) * storing just the app resources usage state (GB/h consumed so far, GB/h consuming currently) Therefore it looks like a good idea to separate the two usage metrics types and store the time-based metrics in a separate database. This will allow us not only to solve the issue, but also to store and query the data more effectively. We may still need to maintain 2 databases and swap new/old (irrelevant) metrics to reduce the DB size on the month boundaries. Regards, Hristo & Adriana [1] https://github.com/cloudfoundry-incubator/cf-abacus/issues/88 [2] https://github.com/cloudfoundry-incubator/cf-abacus/issues/88#issuecomment-148498164 [3] https://github.com/cloudfoundry-incubator/cf-abacus/pull/320
|
|
Questions of how to add projects to Cloud Foundry Community repo
Layne Peng
Hi,
I found there are three repos related to Cloud Foundry in Github. And I prefers to contribute some projects to the Cloud Foundry Community one. So, whom I can contact? And what steps I should follow? - Layne
|
|
Garv Sachdeva
i am working as an admin user on a deployment guided by azure mongodb release https://github.com/cloudfoundry-incubator/bosh-azure-cpi-release/tree/master/docs/advanced/deploy-mongodb
Am stuck on a step to create a service auth token. Have successfully deployed mongodb as a deployment on my cf, but it doesnt show up on my cf marketplace. Have included the security group so my mongo gateway network discovery must be doing good. Am attaching the trace log on the failing step, please help! `CF_TRACE=true cf create-service-auth-token mongodb core 12345678 Creating service auth token as admin... REQUEST: [2016-05-05T14:46:03Z] POST /v2/service_auth_tokens HTTP/1.1 Host: api.xx.xx.xx.xx.xip.io Accept: application/json Authorization: [PRIVATE DATA HIDDEN] Content-Type: application/json User-Agent: go-cli 6.14.1+dc6adf6 / linux {"label":"mongodb","provider":"core","token":"[PRIVATE DATA HIDDEN]"} RESPONSE: [2016-05-05T14:46:03Z] HTTP/1.1 404 Not Found Content-Length: 86 Content-Type: application/json;charset=utf-8 Date: Thu, 05 May 2016 14:46:06 GMT Server: nginx X-Cascade: pass X-Content-Type-Options: nosniff X-Vcap-Request-Id: 462b356f-5416-4d75-62cf-858920848ce3 X-Vcap-Request-Id: 0f65fe6a-3f31-46c2-7b90-e05e55b6b47f::bdd528e8-f26a-4516-bc53-7bc848673c12 { "code": 10000, "description": "Unknown request", "error_code": "CF-NotFound" } FAILED Server error, status code: 404, error code: 10000, message: Unknown request` cf --version cf version 6.14.1+dc6adf6-2015-12-22 cf api API endpoint: https://api.xx.xx.xx.xx.xip.io (API version: 2.51.0)
|
|
Re: Loggregator has updated protobufs definitions and compiler for dropsonde
Jim CF Campbell
Protobufs is smart. Given how we added the map, it should just work either
toggle quoted messageShow quoted text
way.
On Wed, May 11, 2016 at 9:54 AM, Mike Youngstrom <youngm(a)gmail.com> wrote:
You mention that this change is non-breaking for metrics users. I don't --
Jim Campbell | Product Manager | Cloud Foundry | Pivotal.io | 303.618.0963
|
|
Re: aligning cf push health-check default value
Nicholas Calugar
Hi Dies,
toggle quoted messageShow quoted text
I spoke with Eric and he explained that this could be the desired UX for a majority of the apps pushed with --no-route. There are more advanced deployment strategies that might set --no-route and bind to routes later, but I think we can expect these users to be explicit with their health check as well. I think my discomfort with this arose when you mentioned to me that we might want to do this in the Cloud Controller. As long as this continues to be explicit from the API perspective, I'm fine with changing the UX of the CLI per your above proposal. Thanks, Nick
On Wed, May 4, 2016 at 1:13 PM, Shannon Coen <scoen(a)pivotal.io> wrote:
Hi Dies,
|
|
Re: Ubuntu Xenial stemcell and rootfs plans
Mike Youngstrom
I really like the idea of finding a way to move away from bundling binaries
toggle quoted messageShow quoted text
with the buildpacks while continuing to not require internet access. My organization actually doesn't even use the binary bundled buildpacks for our 2 main platforms (node and java). Some issues we have with the offline buildpacks in addition to those already mentioned: * One of the key value propositions of a buildpack is the lightweight process to fork and customize a buildpack. The inclusion of binaries makes buildpack customization a much heavier process and less end user friendly in a number of ways. * We require some java-buildpack binaries that are not packaged with the java-buildpack because of licensing issues, etc. * For some of my customers the binary inclusion policies is too restrictive. So, I agree with your 100% Dan. I'd love to see some work more in the direction of not including binaries rather than making admin bulidpack selection more stack specific. Mike
On Wed, May 11, 2016 at 11:09 AM, Daniel Mikusa <dmikusa(a)pivotal.io> wrote:
On Wed, May 11, 2016 at 9:45 AM, Mike Dalessio <mdalessio(a)pivotal.io>
|
|
Re: Announcing support for TCP Routing
Shannon Coen
Hello Ruben,
toggle quoted messageShow quoted text
We're currently adding support for quota management of route ports, as they can be a limited resource in some environments. This is particularly bad on AWS, where an ELB can be configured to listen on a maximum of 100 ports. We're nearly done with quota support for route ports, and the CLI team is adding support for the new quota attribute. Once the API and CLI are delivered, we'll look at offering TCP routing on PWS to a limited audience. Best, Shannon Coen Product Manager, Cloud Foundry Pivotal, Inc.
On Wed, May 11, 2016 at 2:54 AM, Ruben Koster <superruup(a)gmail.com> wrote:
Really nice!! When will we be able to play with this functionality on PWS?
|
|
Re: Ubuntu Xenial stemcell and rootfs plans
Daniel Mikusa
On Wed, May 11, 2016 at 9:45 AM, Mike Dalessio <mdalessio(a)pivotal.io> wrote:
Hi Mike,Could we please, please not call it `cflinuxfs3`? A very common question I get is what is `cflinuxfs2` really? I then have to explain that it is basically Ubuntu Trusty. That invariably results in the follow up question, why it's called `cflinuxfs2` then, to which I have no good answer. Since it would seem that this naming choice has resulted in confused users, can we think of something that is more indicative of what you actually get from the rootfs? I would throw out cfxenialfs as it indicates it's CF, Xenial and a file system. This seems more accurate as the rootfs isn't really about "linux", if you look at linux as being the kernel [1]. It's about user land packages and those are Ubuntu Trusty or Xenial based, so it seems like the name should reflect that. [1] - https://en.wikipedia.org/wiki/GNU/Linux_naming_controversy to CF pretty quickly (and in fact have considered doing exactly this), and could build precompiled Xenial binaries to add to each buildpack prettyHas there been any thought / consideration given to just not shipping binaries with build packs? I know that we ship binaries with the build packs so that they will work in offline environments, but doing so has the obvious drawbacks you mentioned above (plus others). Have we considered other ways to make the build packs work in offline environments? If the build packs were just build pack code, it would make them *way* simpler to manage and they could care much less about the stack. One idea (sorry it's only half-baked) for enabling offline support but not bundling binaries with the build packs would be to instead package binaries into a separate job that runs as an HTTP server inside CF. Build packs could then use that as an offline repo. Populating the repo could be done in a few different ways. You could package binaries with the job, you could have something (an errand maybe?) that uploads binaries to the VM, you could have the HTTP server setup as a caching proxy that would fetch them from some where else (perhaps just the proxy is allowed to access the Internet) or the user could manually populate the files. It would also give the user greater flexibility as to what versions of software are being used in the environment, since build packs would no longer be limited by the binary versions packaged with them, and instead just pull from what is available on the repo. It would also change upgrading build packs to a task that is mostly just pulling down the latest binaries to the HTTP server. You'd only need to upgrade build packs when there is a problem with the build pack itself. Anyway, I like this option so I wanted to through it out there for comment. Curious to hear thoughts from others. Happy to discuss further. Thanks, Dan
|
|
Re: Loggregator has updated protobufs definitions and compiler for dropsonde
Mike Youngstrom
You mention that this change is non-breaking for metrics users. I don't
know much about protobufs backwards compatibility story. Can you detail a little more the implications from a compatibility standpoint with this change? Some questions: * If I compile the new protocol via protobuf3 can I still communicate with an old loggregator deployment? * If I don't use the new protocol can I talk I communicate with a loggregator that is using the new protocol? (absent the new field of course) Thanks, Mike On Wed, May 11, 2016 at 8:58 AM, Jim CF Campbell <jcampbell(a)pivotal.io> wrote: Hopefully it's OK because you're bought into the value of tagged metrics...
|
|
Re: Ubuntu Xenial stemcell and rootfs plans
Mike Youngstrom
Thanks Mike that helps. Hopefully that work will get prioritized in the
toggle quoted messageShow quoted text
next year or so. :) For the record, on the stemcell side I've been battling a non CF issue [0] with Trusty that I'm hoping is fixed in Xenial. I could verify if it is fixed without a stemcell. I'm just being lazy. :) Perhaps I'll verify first so I have a more concrete reason to request a Xenial stemcell. Thanks, Mike [0] https://github.com/hazelcast/hazelcast/issues/5209
On Wed, May 11, 2016 at 7:45 AM, Mike Dalessio <mdalessio(a)pivotal.io> wrote:
Hi Mike,
|
|