Lab Manager 4.0 Best Practices & Design Considerations

New Version of this article here

Lab Manager is one of my favorite technologies in the market today, but before you install, beware of the limitations!

i. 8 ESX hosts max to connect to VMFS3 Datastore (each LUN), you can use NFS to get around this, but for our use case, this is not performant enough.

ii. 2TB vmfs3 size limit, and don’t start there, we started at 1.2TB Luns so we could expand as needed.  Avoid SSMOVE if at all possible (SSMOVE is slow and painful, but works well), if you fill up your Lun, create the extend and/or disable templates and move them to a new datastore.

iii. Only available backups are SAN Snapshots (well the only realistic one for us), and for this to be useful, see #1 below

1. Recommended to put vCenter & Lab Manager Servers on VM’s inside cluster on SAN with the guests

iv. vCenter limits

1. 32 bit has max of 2000 deployed machines and 3000 registered

2. 64 bit has max of 3000 deployed machines and 4500 registered

Best Practices & What we’ve learned

i. Make Luns 10x the size of your Template(s)

ii. Shared Storage is generally the first bottleneck.

1. I used all Raid 1+0 since we observed this on our first LBM deployment and our application is database driven (disk I/O intensive)

iii. We have averaged between 80-100 VM’s per blade, so this means our LBM 4.0 environment should top out at approximately 32 hosts (2 full HP c7000’s) (one cluster, otherwise you lose the advantages of Host Spanning Transport Networks)

iv. LOTS of IP addresses, I recommend at least a /20 for LBM installs = 4096 IP’s, you do not want to have to re-IP lab manager guests, we’ve done that before.

v. Create Gold Master Libraries for your users, helps prevent the 30 disk chain limit from being hit as often.

vi. Encourage Libraries, not snapshots

vii. Do not allow users to create\import templates, Export Only

viii. Do not allow users to modify Disk, CPU or Memory on VM’s.

ix. Storage and Deployment leases are the best thing since sliced bread. Recommend between 14-28 days for both.

x. Train your users, we even went as far as to have two access levels, one for trained, one for untrained, so the untrained are less dangerous, and if they want the advanced features it forces them to get training.

Update to HP Firmware issue

I just verified that you don't need to update the iSCSI on all 8 adaptors, just the 1st one should be enough.

Setting up Virtual Center 4 ( vCenter )

If using SQL 2005/2008 then setup two databases VirtCent & VUM before installing vCenter, they can share a DB, but VMware recommends against it for performance reasons.

When Installing, your ODBC may ask for a 32bit DSN, use this to configure that “Data Source”

%systemdrive%\Windows\SysWoW64\Odbcad32.exe

Update to HP BL490c Firmware Issue

After allot of troubleshooting and a very good suggestion in an email from someone reading my blog, I have a fix and a better understanding of the issue.  When you update from Firmware CD 8.6 to 8.7, the version of the Broadcom Nic goes from 2.2.2 to 2.2.4, however, more specifically, there are two subversions of code that make up 2.2.2 and 2.2.4.   There is “bootcode” and “iSCSI”, when you do an update from 2.2.2 to 2.2.4, you only update “bootcode” from 4.8.0 to 5.0.11 and not “iSCSI” since the version theoretically didn’t change from 3.1.5 to 3.1.5.  This seems to be the issue.  If while performing the FW update, you manually rewrite-reapply the iSCSI as well, then the firmware update applies, and everything works properly.  If you are already on 2.2.4, you can just reapply just the iSCSI or both, either works well.

In order to successfully update your ISCSI and have a working firmware, do the following:

1) Boot from the HP 8.7 firmware CD:

Choose “Installation Options”, then click “Click here to enable install options”, then click “Allow Rewrites”, then of course “OK”.

It will warn you that you must manually select NICS and options when doing NIC firmware rewrites and downgrades. Click “OK”.

Now under “HP NC-Series Broadcom Online Firmware Upgrade, Choose “Select Devices”, When you see the screen with your NICs in it, choose “Device Details” for your first NIC in the list"

You will then see a screen like this below, Select the “iSCSI” box to reapply 3.1.5

Click “OK” then Select your 2nd device, and repeat 8 times.

After that, reboot and everything should be upgraded and working properly.

Error about “Bus 0 Enclosure 0” when creating LUN on EMC Clariion array, but your not creating a LUN on “Bus 0 Enclosure 0”

The Exact error is “The RAID group being selected (RAID 1/0, RAID 1) includes disks that are located both in Bus 0 Enclosure 0 as well as some other enclosure(s). This configuration could experience Rebuilds of some disk drives following a power failure event, because Bus 0 Enclosure 0 runs longer than other enclosures during a power failure event.  An alternate configuration where the disks in the RAID group are not split between enclosures in this manner, is recommended.”

The fix was to update my Flare version and everything now works normally.

Just in case you lose your password, or the Fiber Channel Cisco MDS 9124e for your c7000 blade enclosure ships with a password you don’t know like mine was

1) Plug into the HP OBA Serial port, 9600 8/N/1.

2) connect interconnect <bay number>

2b) reset switch, during boot press "CTRL+c"

3) dir bootflash:

4) boot “kickstart_image” i.e. m9100-s2ek9-kickstart-mz.3.1.2a.bin

5) config t

6) (in config) admin-password <new complex password>

7) exit

8) dir bootflash:

9) load bootflash:”system image" i.e. m9100-s2ek9-mz….(not kickstart again like the broken HP instructions tell you)#note they have fixed this document in the last week.

10) login (fyi, the default username is “admin” not “admin123” like cisco.com states)

11) config t

12) (in config) snmp user admin auth md5 <new complex password>

(you need step 12 to use Cisco Device Manager to complete setup)

13) int mgmt 0

14) ip address <ip><mask>

15) exit

16) ip default-gateway <dg-ip>

17) exit

18) copy run start

19) (bonus info) if you want to leave the current bay to setup the redundant switch, hit CTRL+SHIFT+_ to exit back to the OBA so you can start over by connecting to the redundant bay.

Helped HP trace down a firmware issue today

My BL490c G6 blades were unable to get network connectivity to our chassis.  I had setup all the networking in HP Virtual Connect, but 6 blades worked, and 9 did not (we only purchased 15, not 16, don’t ask)

The error I received was different depending on the OS I was loading.

ESX 4.0 Update 1 gave this error: “The script 32.networking-drivers failed to execute and the installation can not continue.”

ESX 4.0 (no update 1) gave this error: Network ports Disconnected.

I was able to isolate this to a blade specific issue by swapping blades 4 and 6 and the issue followed the blades, not the bay.  There are 4 types of firmware on these HP blades that I can see.  BIOS, iLo, Qlogic(HBA), BC (nic).  The UI shows everything but the NIC firmware version.  All other blades with the latest Nic BIOS do not have any active network ports.  If I downgrade them from 2.2.4 to 2.2.2(more specifically the Bootcode from 5.0.11 to 4.8.0), then they work(problem solved).  It’s a real pain to see the HP Nic Bios since it only shows up when booting from the HP firmware CD.  I do have the latest firmware on my c7000 chassis and flex10 switches.  Just to verify this was a blade specific issue, I also swapped hard drives from a non working blade to a working blade, the issue followed the blade again.

I think I have this solved, I’ve “fixed” 5 blades so far by downgrading them, the only way I can find to downgrade the NIC bios is from the HP firmware boot CD, 8.7 has 2.2.4 and 8.6 has 2.2.2.  There is a version 2.2.3, but best I can tell it’s not easy to install it, it must be done from inside a guest OS(no bootable CD) and ESX 4 Update1 does not install because of this error, I’d have to install another OS, then change FW version, then reload the blade.

I have an open HP case to get this resolved, hopefully we can get an updated firmware soon.