VMworld HOL using VCVA (vCenter Virtual Appliance)

This is the first of a series of HOL posts about "how we did it".

For the primary workload, we used the vCenter Virtual Appliance using the local Postgres database.

Due to the unusually high churn rate of HOL, we need to have a high ratio of vCenters.  These vCenters needed to have a lot of horsepower behind them to survive this churn.

1) Paravirtualized SCSI adaptors for disk controllers for the VCVA vm.
2) Created 2 additional dedicated datastores (Luns) for each of the DB & Logs on the VCVA vm.
3) 4 CPU's x 32GB memory (we might have gone a bit high on memory)
4) Removed all long term logging and rollups, we are doing all stats in vC Ops.
5) Increased heap sizes to large for the SPS, tomcat inventory & vCenter process.

The only downside to the VCVA is the fact that it doesn't support linked mode, but you can get around that with the NGC & SSO.   http://www.virtuallyghetto.com/2012/09/automatically-join-multiple-vcsa-51.html

Deploying a VMware vCloud Director (vCD) 5.1 virtual appliance with MS SQL backend

This is a guide for deploying vCloud in a LAB environment, these settings are not the most secure or performant, but this should get you up and running with vCloud 5.1 so you can test and learn it. The easiest way is with the appliance, you don’t need to use a MS SQL DB, but occasionally I need to crack open the DB and I am the most comfortable with that technology.

1. My Assumptions about what you already have:
   1. One ESXi Host with the following VM’s on it.
      1. Windows with MS SQL DB (I’m using MS SQL 2008 R2)
      2. vShield Manager 5.1 with an IP set (also known as vCloud Networking and Security 5.1)
      3. vSphere 5.1 vCenter (can be the appliance)
      4. Available resources to Deploy vCloud Director Appliance
   2. VCP or equivalent level of knowledge
2. Prepare your Database (same steps as with non-appliance) 
   1. Again, I am assuming you have MS SQL 2008R2 installed, without a local firewall, or ports opened.
   2. This is a great article, follow it, I will paste the highlights from it below, you can copy/paste these commands into SQL Query analyzer!!

   1)    Configure the database server.
   A database server configured with 16GB of memory, 100GB storage, and 4 CPUs should be adequate for most vCloud Director clusters. (this is for production level quality)
   2)    Specify Mixed Mode authentication during SQL Server setup.
   Windows Authentication is not supported when using SQL Server with vCloud Director.
   3)    Create the database instance.
   The following script creates the database and log files, specifying the proper collation sequence.

   USE [master]
   GO
   CREATE DATABASE [vcloud] ON PRIMARY
   (NAME = N'vcloud', FILENAME = N'C:\vcloud.mdf', SIZE = 100MB, FILEGROWTH = 10% )
   LOG ON
   (NAME = N'vcdb_log', FILENAME = N'C:\vcloud.ldf', SIZE = 1MB, FILEGROWTH = 10%)
   COLLATE Latin1_General_CS_AS
   GO

   The values shown for SIZE are suggestions. You might need to use larger values.
   4)    Set the transaction isolation level.
   The following script sets the database isolation level to READ_COMMITTED_SNAPSHOT.

   USE [vcloud]
   GO
   ALTER DATABASE [vcloud] SET SINGLE_USER WITH ROLLBACK IMMEDIATE;
   ALTER DATABASE [vcloud] SET ALLOW_SNAPSHOT_ISOLATION ON;
   ALTER DATABASE [vcloud] SET READ_COMMITTED_SNAPSHOT ON WITH NO_WAIT;
   ALTER DATABASE [vcloud] SET MULTI_USER;
   GO

   For more about transaction isolation, see http://msdn.microsoft.com/en-us/library/ms173763.aspx.

   5)    Create the vCloud Director database user account.
   The following script creates database user name vcloud with password vcloudpass.

   USE [vcloud]
   GO
   CREATE LOGIN [vcloud] WITH PASSWORD = 'vcloudpass', DEFAULT_DATABASE =[vcloud],
      DEFAULT_LANGUAGE =[us_english], CHECK_POLICY=OFF
   GO
   CREATE USER [vcloud] for LOGIN [vcloud]
   GO

   6)    Assign permissions to the vCloud Director database user account.
   The following script assigns the db_owner role to the database user created in Step 5.

   USE [vcloud]
   GO
   sp_addrolemember [db_owner], [vcloud]
   GO

    

3. Deploy and configure vCloud virtual appliance
   1. Login to vCenter with vSphere client
   2. Click on File/Deploy OVF template, choose vCloud-Director-VA-T2-5.1.0.0-817173_OVF10.ova
   3. After you choose the obvious options, you should get a properties page to fill out like below
   4. 
   5. 
   6. Fill in the options
   7. Scroll Down, Fill in Database Name, vcloud if you used my info above
   8. Fill out Networking Properties (or leave blank for DHCP)
   9. After the standard deploy progress bar, you will see a long delay(5-10 minutes) during VM boot up while it sets up the DB, it’s not hung, just give it time.  It will occasionally ask questions, don’t answer them, just let it go..go get coffee..
   10. Eventually you should see a screen like this indicating that installation is finally complete:
   11. 
   12. Login to https://ipaddress:5480 admin/vmware if you need to configure the VM any further
   13. Login to https://ipaddress/cloud/ to begin configuring vCloud, but that will be my next blog post.(link soon)

Good Links:

vCloud Director 5.1 Release Notes

VMware vCloud Director 5.1 Documentation Center

VMware vCloud Directory Documentation

Disable Fibre Channel HBA so I can connect to another Fabric

We are doing a forklift upgrade of our servers, in order to do so, I would like to connect my esx hosts to both fabrics for a while so I can transfer the VM's over that network and not the front end ethernet network.  Because I don't want to make changes to the legacy Fibre, and I don't want to connect the fabrics more than I must, I will disconnect the redundant fibre cable from each ESX host and connect it to the new fibre.  Here are my steps.
First go into vcenter, identify the correct HBA WWN, I just use the last octet, so I want to re-use "86", so that is the HBA I am going to disconnect from the existing Fibre.

Then Go into the Properties of each Datastore and click "Manage Paths", change the path selection policy to "Fixed" (so we can control what path the Host is using to access the current storage), Click "Change", click the status of the current path you wish to keep (not using 86), and click Preferred.  After that, click the paths that 86 is using, and choose Disable.  It should then look something like below, you can see the Adapter listed after the Path is chosen.

Click Close and repeat for all datastores.  You also can verify/do this work from the Configuration/Storage Adapter page, which will look something like below, Note that we are looking at HBA2 (86), Under "Paths" and you can see status is Disabled.

After disabling all of the paths under each datastore, you may still have some paths left, those are just the connections to the array, but not actually a connection to a LUN.  You can disable those if you choose, I do because I try to never leave anything to chance.  At this point this HBA is no longer in use and is ready to be re-used to connect to the new Fibre.

In my case, I am using an HP server with HBA's installed into PCIe slots.  My question is, which HBA is mapped to which PCIe Slot (I don't want to disconnect the wrong one since I just removed redundancy).  I am using ESXi 5.x, so I have enabled SSH for troubleshooting.  I tried logging into iLo, however it did not have PCIe slot card information, so I decided to go right twards the horses mouth so to speak and have SSH'd into my server and ran the following command:
esxcli hardware pci list
but that gave me alot of information I couldn't use, so then I tried:
lspci
which gave me:
000:067:00.0 Serial bus controller: QLogic Corp ISP2432-based 4Gb Fibre Channel to PCI Express HBA [vmhba1]
000:070:00.0 Serial bus controller: QLogic Corp ISP2432-based 4Gb Fibre Channel to PCI Express HBA [vmhba2]
This is perfect information, because the HP Quickspecs for the DL385G2 has the following info:

You can see Expansion Slot #1 is Bus Number 70, which from lspci above is vmhba2, and Expansion Slot #2 is Bus Number 67, which is vmhba1

User’s AD account being locked out by vCenter 5.0 server, how I found our culprit

My vCenter is running Windows 2008R2.  I tried Running the Microsoft sysinternals tools to find out which host is locking out this user and it kept pointing me to vCenter.  Of course to me it didn’t make any sense why vCenter would be locking this user out every 60 minutes.  I also checked the standard stuff, no services running as the user, nothing under the “Credential Manager” on vCenter.  There are also several VMware KB’s for dealing with the issue, they want you to use TCPVIEW, and some other tools, but none of them helped me on this one.

I’ll start my troubleshooting steps below after the normal sysInternals AD “find out which machine is locking the user out” steps.

1) I logged into vCenter and opened “Server_Manager/Diagnostics/Event_Viewer/Windows Logs/Security”

I saw the following 3 Audit Failures happening ever 60:16 seconds (not sure if that interval is important to the issue, but it gave me an ability to predict the next occurrence).  Since the last event was at 9:31:30am, I can predict the next occurrence at 10:31:46am.   You can see the “Account for Which Logon Failed:” “Account Name:” is the usersloginname, this is key information we’ll use later.

2) Logging into vCenter I see this under “Tasks & Events”, the timestamp is an exact match to the Event Log Event above.  I see nothing that correlates to this timestamp under vCenter Tasks, which makes me think this is being initiated from an external (to vCenter Service) source.

3) Before I realized this was happening every 60:16 I grabbed a large packet capture on my vCenter Server.  Now that I could predict the exact second this would happen again, I decided to capture a smaller time window with a script.

I setup a single scheduled wireshark dumpcap by utilizing Windows Task Scheduler.

The dumpcap –a parameter specifies a 4 second duration of packet capture, the –w specifies what to name the output file, lo.pcap in this example.  I dumped the files into a directory called c:\output .  I didn’t use the default location because when I ran the script manually (from the command line to test) I received access denied, and since I’m too lazy to troubleshoot it, but I’m pretty security minded, I setup a temp directory called output that I gave everyone full control to.  I setup the schedule to start the capture 2 seconds before the planned event and it did successfully gather the data I needed.

4) Now to analyze what I captured.  Because the user swore that he was not manually doing anything to the systems, I though if I did a search within the file for the users name, it may give me some good data.  I did a Wireshark Filter of tcp contains usersloginname.  What I saw was 4 packets containing his name, the first going from vCenter to DC, then the response from DC to vCenter, then the same repeated.

By itself, this isn’t overall useful, but IF this problem wasn’t coming FROM virtual center (or a locally installed app), it must be coming over the network.  And if vCenter is talking to AD, most likely something is happening right before this event to cause it.  You see this communication started with packet 316 @ 2.74 seconds after I began capturing.  This is a good correlation since I started the capture 2 seconds early, I was right on time.  I was also able to verify that the issue had happened again  right on time in the Windows Event log.  So what is happening on vCenter’s network RIGHT BEFORE packet 316?  I removed my filter and scrolled up a bit, most of the traffic was from vShield Manager, the SQL DB, misc vCloud items, but one IP stood out, it was not from my vCloud farm of servers.  This Mystery IP seemed like my most likely candidate.  I also did a ping –a 10.x.x.x hoping for a strike on reverse DNS to get a hostname without luck.

To confirm my suspicions that the mystery IP was somehow triggering the lockouts, I switched to analyzing my older bigger packet capture and did a Filter by ip.src==10.x.x.x (the mystery IP). This came up with traffic starting at packet 67701 @ 305.4 seconds into the capture.  Now if my hypothesis is correct and this mystery IP is the culprit, then there should be some talk back and forth with my DC immediately following.  I did another search in this old capture with tcp contains usersloginname and found this traffic on packet 67714 @ 305.5 seconds into the capture!!!  This means the Mystery IP was contacting vCenter, and then locking out my users AD account, but since it was vCenter talking directly to AD, vCenter was getting credit for locking the user out.

5) I located the Mystery IP VM, it was a VMware vCenter Operations VM using the users old password, we fixed it and then waited until 60:16 more seconds had passed, no new logs in vCenter, no new failures in AD, the issue is resolved!

Removing an ESX(i) host from a DvS (distributed virtual Switch)

For some reason the UI makes it difficult to find this option.  It’s not hard to find when you remember where to look.  Go into Home/Inventory/Networking, choose your DvS you want to remove the host from, then choose the “Hosts” tab, then click the host you want to remove and choose “Remove from vSphere Distributed Switch…”

VMware HA policy restricting VM deployment well before ESXi Hosts have reached capacity.

This can happen due to the default mechanic for slot size calculation in VMware HA.  The default slot size is calculated based on the size of your largest VM in the cluster, and in this example I’m talking about memory because that is generally the first HA bottleneck, not CPU.  If your like me and you have a single large 8GB VM and a bunch of small 256MB VM’s, your cluster admission policy will stop deployments even when ESX host memory usage is below 50%.  To increase your slot size, which you can see here:

You can see the Slot size is 8332 MB, I’d prefer it was down around 1024 to be closer to reality.

To make the change, go into the Advanced Options of HA

And set das.slotmeminmb to a value of 1024

Now when you go back and look at the current slot allocation “Advanced Runtime Info” you see that you have gone from 112 slots to 966

HA should again work as expected, but you should know that even though this will allow you to essentially overcommit your HA settings and HA working properly for your large VM in a disaster is no longer guaranteed due to the custom settings.

Here is the  text from the vSphere Availability Guide

Slot Size Calculation is comprised of two components, CPU and memory. 

VMware HA calculates the CPU component by obtaining the CPU reservation of each powered-on virtual machine and selecting the largest value. If you have not specified a CPU reservation for a virtual machine, it is assigned a default value of 256 MHz. You can change this value by using the das.vmcpuminmhz advanced attribute.)

VMware HA calculates the memory component by obtaining the memory reservation, plus memory overhead, of each powered-on virtual machine and selecting the largest value. There is no default value for the memory reservation.

If your cluster contains any virtual machines that have much larger reservations than the others, they will distort slot size calculation. To avoid this, you can specify an upper bound for the CPU or memory component of the slot size by using the das.slotcpuinmhz or das.slotmeminmb advanced attributes, respectively.

Where to find the SQL 64 bit Native Client for vCenter Installs

Microsoft® SQL Server® 2008 R2 Native Client  X64 Package (sqlncli.msi)

PowerCLI command to enumerate VM’s mac address inside of vCenter

I did not write this, I found it online but lost the link, but it works great!
ForEach ($VirtualMachine in $VM) {
    # Get the virtual machine
    $VMsView = Get-View -ViewType VirtualMachine -Property Name,Guest.Net -Filter @{"Name"="$VirtualMachine$"}
    if ($VMsView) {
      $VMsView | `
        ForEach-Object {
          $VMview = $_
          $VMView.Guest.Net | `
            Select-Object -property @{N="VM";E={$VMView.Name}},
              MacAddress,
              IpAddress,
              Connected
        }
    }
}