thanks for doing the research danny!

please let us know if you try working on this. we'll definitely try and
help if you get stuck.

On Mon, Jun 22, 2015 at 5:04 PM, Dmitriy Kalinin <dkalinin(a)pivotal.io>
wrote:

Thanks for looking into this!

I would be very interested in enabling enhanced networking as a default
for AWS stemcells. It sounds like you already identified needed changes to
make that happen. Here is some more details around the code:

1) modify stemcell builder to install correct version of ixgbevf
- add sriov_net_support => true to
https://github.com/cloudfoundry/bosh/blob/master/bosh-stemcell/lib/bosh/stemcell/infrastructure.rb#L96
to signify that that stemcell supports it
- add a new stage to
https://github.com/cloudfoundry/bosh/tree/master/stemcell_builder/stages
with vendored source for ixgbevf
- include stage name in a list of stages to run when building stemcells:
https://github.com/cloudfoundry/bosh/blob/master/bosh-stemcell/lib/bosh/stemcell/stage_collection.rb
- make sure it happens after system_kernel stage

1.1) see https://github.com/cloudfoundry/bosh/blob/master/bosh-stemcell
how to build your own stemcell

2) modify aws cpi to set sriovNetSupport attributes when creating stemcells
- include the attribute in
https://github.com/cloudfoundry/bosh/blob/master/bosh_aws_cpi/lib/cloud/aws/stemcell_creator.rb#L84
if stemcell's cloud_properties includes sriov_net_support => true
- this code path is used by our CI to create light stemcells but also
when full AWS stemcell is uploaded to the director, so it covers all cases
- make sure that stemcells still boot on instance types that do not
support sriov even if stemcell supports it
- make sure that stemcell booting on instance types that support sriov
actually take advantage of that feature

3) submit PR and we will get that into our pipelines!


On Mon, Jun 22, 2015 at 1:06 PM, Danny Berger <dpb587(a)gmail.com> wrote:

Hi all - I'm interested in the "Enhanced Networking" capabilities on the
BOSH-published AWS HVM light stemcells, with particular interest in the
lower latencies it can bring. Amazon has two stemcell-relevant
requirements
<https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html#enhanced-networking-requirements>,
which are not entirely being met (based on the
bosh-aws-xen-hvm-ubuntu-trusty-go_agent/2992 VM I was looking at):

* Kernel Module ixgbevf, recommended 2.14.2+ (stemcell currently has
2.12.1-k)
* Image Attribute sriovNetSupport (not currently configured)

In terms of changes, I think the first and easiest change would be to
start registering the published AMIs with the additional
`--sriov-net-support simple` option, a la AWS CLI. That would at least
enable the enhanced networking capabilities, with the second possible
change being to upgrade lixgbevf for the latest bug fixes and minor
improvements.

In terms of trying to make this change, I'm not sure what the next step
is. Can anyone offer further guidance on this? Or advise me on how I can
build the stemcells for internal use (which I can then register as
sriov-capable)? Or is there a better way to go about this?

Thanks for your time!

Danny


--
Danny Berger
http://dpb587.me

_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh

_______________________________________________
cf-bosh mailing list
cf-bosh(a)lists.cloudfoundry.org
https://lists.cloudfoundry.org/mailman/listinfo/cf-bosh

--
Thank you,

James Bayer

Thanks for looking into this!

I would be very interested in enabling enhanced networking as a default for
AWS stemcells. It sounds like you already identified needed changes to make
that happen. Here is some more details around the code:

1) modify stemcell builder to install correct version of ixgbevf
- add sriov_net_support => true to
https://github.com/cloudfoundry/bosh/blob/master/bosh-stemcell/lib/bosh/stemcell/infrastructure.rb#L96
to signify that that stemcell supports it
- add a new stage to
https://github.com/cloudfoundry/bosh/tree/master/stemcell_builder/stages
with vendored source for ixgbevf
- include stage name in a list of stages to run when building stemcells:
https://github.com/cloudfoundry/bosh/blob/master/bosh-stemcell/lib/bosh/stemcell/stage_collection.rb
- make sure it happens after system_kernel stage

1.1) see https://github.com/cloudfoundry/bosh/blob/master/bosh-stemcell how
to build your own stemcell

2) modify aws cpi to set sriovNetSupport attributes when creating stemcells
- include the attribute in
https://github.com/cloudfoundry/bosh/blob/master/bosh_aws_cpi/lib/cloud/aws/stemcell_creator.rb#L84
if stemcell's cloud_properties includes sriov_net_support => true
- this code path is used by our CI to create light stemcells but also
when full AWS stemcell is uploaded to the director, so it covers all cases
- make sure that stemcells still boot on instance types that do not
support sriov even if stemcell supports it
- make sure that stemcell booting on instance types that support sriov
actually take advantage of that feature

3) submit PR and we will get that into our pipelines!

Hi all - I'm interested in the "Enhanced Networking" capabilities on the
BOSH-published AWS HVM light stemcells, with particular interest in the
lower latencies it can bring. Amazon has two stemcell-relevant requirements
<https://docs.aws.amazon.com/AWSEC2/latest/UserGuide/enhanced-networking.html#enhanced-networking-requirements>,
which are not entirely being met (based on the
bosh-aws-xen-hvm-ubuntu-trusty-go_agent/2992 VM I was looking at):

* Kernel Module ixgbevf, recommended 2.14.2+ (stemcell currently has
2.12.1-k)
* Image Attribute sriovNetSupport (not currently configured)

In terms of changes, I think the first and easiest change would be to start
registering the published AMIs with the additional `--sriov-net-support
simple` option, a la AWS CLI. That would at least enable the enhanced
networking capabilities, with the second possible change being to upgrade
lixgbevf for the latest bug fixes and minor improvements.

In terms of trying to make this change, I'm not sure what the next step is.
Can anyone offer further guidance on this? Or advise me on how I can build
the stemcells for internal use (which I can then register as
sriov-capable)? Or is there a better way to go about this?

Thanks for your time!

Danny


--
Danny Berger
http://dpb587.me