Monday, October 24, 2011

VMDirectPath not supported on Cisco Palo CNA, except when it is.

Yes, it's confusing, I know. It seems that when you try to enable VMDirectPath for a PCI nic you've created in UCS using Palo (M81kr), it doesn't work. You can select the PCI device for VMDirectPath, and vSphere will tell you to reboot the server to enable the change ...



BUT, you can reboot over and over again, but the PCI device never gets enabled. Looking through the log file on the ESXi server will yield an entry stating "not a ACS capable device".  Thinking that perhaps my M81KR was too old, or too new, I thought that the maybe newer Cisco CNA's like the VIC 1280 would support VMDirectPath. But no. 

UPDATE: CISCO PRODUCT MANAGEMENT INFORMS US THAT NEXUS 1000V OR 1010 IS NOT WHAT YOU NEED. YOU NEED VM-FEX. VM-FEX uses the same VEM as 1000v, but has a different manager. The link below with instruction will get you going. 
But you do need a DVS. That part is correct. 
As it turns out, you need Cisco's Nexus 1000v (or 1010v, if you're into that) in order to make VMDirectPath work. Yes, you read correctly. You need a Distributed Virtual Switch in order to enable VMDirectPath, a technology which completely bypasses Virtual Switches. 

Here's why:

When VMware first came up with vMotion technology back in '03, I.T. people who understood it and it's implications got goosebumps. No more waiting 'till 3 am to do server upgrades, hoping no one was still using the system, taking the system offline, apologizing to whoever was still using the system, upgrading, hoping it worked, and rolling it back when it didn't. Now just evacuate the server, fix it, bring it back on line, and no one was the wiser.

But of course, they also wanted more. In order to make vMotion work, the VM needed consistency within. Continuous connection from the Guest OS to the outside world was mandatory. Changing the nic type inside the VM wasn't an option, since it would take too long for the Guest OS to adapt to the new adapter, if it even could.

But this meant a virtual adapter in the Guest OS. Which meant emulation, not virtualization. Which meant slower networking. Customers wanted speed, so VMware delivered. In conjunction with Intel, they created VMDirectPath, a method of connecting a PCI device directly into the VM. Speed was restored, but at the cost of transportability, a key feature of virtualization. So customers could have one or the other, but not both. Until now. 

Cisco SAVBU (Nuova) and VMware have finally delivered the capability to have VMDirectPath AND vMotion for the same VM!  They're calling it DirectPath IO with vMotion. This is quite an impressive achievement, and people who know me know that I don't impress easily. They've been working on it for over 3 years, that I know of, and the collaboration has been tumultuous. But it yielded I.T. gold. 

So naturally, I had to test it out. I had waited so long. And I found what I documented above. You can't turn it on. At least not the way you would turn on VMDirectPath for any other device.

< to do, put in more accurate instructions, and pictures>

*In order to enable DirectPath IO for a Cisco CNA, you have to:
1) Configure pass through switching for your service profiles.
2) Set up a port profile for DirectPath, and specify "High Performance"

3) Add your servers to the vDS
4) add a VMXNET 3 virtual nic to the VM.
5) connect the VMXNET 3 vnic to the port profile.
6) watch the fun!

*The steps above are not exhaustive, there are a few things left out, but that's the gist of it. For a more complete set of steps to get this going, see this blog entry.

Once the Nexus detects that a VMXNET3 vnic is connected, and "High Performance" is selected as part of the port profile, it begins allowing DirectPath IO. This is a new, special VMDirectPath that only works with Cisco CNA's and VM-FEX. While the VM is sitting on one ESXi server, VMDirectPath is in effect, giving that 30% speed boost that only VMDirectPath can deliver.

When it's time to vMotion the VM to another server, the Nexus 1000v will shift that vnic back into old fashioned Virtual Switch mode, complete the vMotion, then shift back to DirectPath IO once the vMotion is complete. It's awesome! And it's "set and forget", nothing special to be done afterward.

Cisco is the only vendor that can do it, and it's only available with vSphere 5.  It has something to do with Ed Bugnion having been a founder at VMware, then CTO at Nuova, the company that created UCS and Nexus 1000v, and the similarities between the VMXNET 3 design and the Cisco "Palo" virtual nic. Anyway, this feature is the kind of innovation that just can't be done by VMware or Cisco alone. and it brings huge benefits in performance and virtualization. Try it out for yourself!

Here's a good description of the two different modes of operation.

No comments:

Post a Comment