Brian's Tech Blog

Where to find the SQL 64 bit Native Client for vCenter Installs

2012-01-26T11:27:00.002-07:00

Microsoft® SQL Server® 2008 R2 Native Client X64 Package (sqlncli.msi)

VMware Host Profiles giving errors about Path Selection Policy for naa and mpx devices

2012-01-26T10:20:00.002-07:00

I am trying to use host profiles in a full NFS vCenter, no block level storage other than the local SSD disks for the ESXi 5 servers to boot from and some various emulated CDROM’s attached to some of the hosts. Because I don’t really care about the uniformity of my local storage I just want to ignore these errors and get a Green “Compliant” result for my cluster so I can fix the important issues. I tried going into Edit Profile and removing a lot of data from under Storage Configuration/PSA & NMP & iSCSI, but this led to more errors, such as below:

Failures Against Host Profile

Host state doesn’t match specification: device mpx.vmhba32:xx:xx:xx Path Selection Policy needs to be set to default for claiming SATP

Host state doesn’t match specification: device mpx.vmhba32:xx:xx:xx needs to be reset

To resolve the issue, you need to right click on the Actual storage profile under “Host Profiles” on the left hand side of the screen and choose “Enable/Disable Profile Configuration…”

Then unselect items from here such as PSA device configuration and whatever else causing you unnecessary grief.

After hitting OK, rescan your hosts and you should finally get your Green Light!

Credits go to the knowledge base and to Kevin Space.

Setup VLAN tagging for ESXi on a Dell Blade Server using Dell PowerConnect M8024-k chassis switches

2012-01-23T12:27:00.002-07:00

When using ESX on a server it is good to have a lot of network cards, especially if using vCD (vCloud) . This is a follow up to my previous post about dividing a single Dell NIC into multiple nics (partitioning) http://bsmith9999.blogspot.com/2012/01/divide-single-network-into-multiple.html.

Before you begin, you need to design your setup, for my example I want 9 VLAN’s.

1010 – mgmt (the vCenter & vCM & vCell, etc…)

1020 – guests (external facing)

1030 – vMotion

1040 – NFS (not using FC)

4010 – vcdni1

4020 – vcdni2

4030 – vcdni3

4040 – vcdni4

4050 – vcdni5

Step 1 Login and look around.

Login to Dell OpenManage, Open Switching/VLAN/ choose “VLAN Membership”. Out of the box you will only see the default VLAN 1. Assuming you are using multiple chassis with clusters that will span these chassis like I am, you will need Tagging to flow through your core switches into these. On this default VLAN, you will see the “Lags” at the bottom left hand side, the Current is set to “F” which is forbidden, meaning this default VLAN will not pass through the trunk into the core switch and cross chassis, for the default VLAN 1, this is good, for the other VLANs, we will change it.

Step 2, Add VLAN’s.

Under “VLAN Membership”, click “Add” near the top, type in your VLAN ID and name, then click Apply. Repeat step to create all of your VLAN’s.

Step 3, Change VLAN Lag type to Tagged.

Click on Detail after you have created your VLANs. Choose your VLAN under “Show VLAN”, Under Lags, change the “Static” box from “F” to “T” by clicking it a couple times. Then click apply, repeat for all new VLAN’s you’ve created that you want to have flow through your core network.

Step 4, Change VLAN Port Settings to “Trunk”

**Warning** before you do this, you must have iDRAC console access to your blade or you may lose connectivity to it****

Click on “Port Settings” which is just below “VLAN Membership”. You will now see the port Detail page. For each Port Te1/0/1 through Te1/0/16 change the Port VLAN Mode from Access to Trunk, then Click Apply. This will allow the blades to pass multiple VLAN networks to and from themselves. Repeat this step 16 times.

Step 5, Modify ESXi to accommodate the new VLANs

Open a iDRAC session to your blade(s). F2 to Login, Choose “Configure Management Network”, Choose “VLAN (optional)”, then type in your VLAN ID

Hit Enter, then ESC. It will ask if you want to Apply changes and restart mangement network? Say (Y)es.

NOTE ***In the dell Switch UI, make SURE to click the little floppy disk picture in the upper right to Save your work when your done or you’ll get to repeat it after your next power outage like I did***

You should be done, repeat these steps to get all your blades online and using VLAN’s.

Divide single Dell NIC into multiple NICs, going from 2 to 8 nics per blade Dell Blade Server

2012-01-18T14:32:00.002-07:00

I was looking for the Dell equivolent to HP Flex-10’s FlexFabric Adapter. In dell speak this is called “independent NIC partitioning“ or just “NIC partitioning” or NPAR.

First let me give you some background into my new system. It consists of M1000E blade cabinets, M710HD Blade Servers with Broadcom 57712-k 10GbE 2P nics, and PowerConnect M8024-k Cabinet Switches. My goal is to turn two 10GB NIC’s into four NIC’s made up of two 1GB NIC’s and two 9GB NIC’s

Before you connect to iDRAC, If you want to use your mouse, you must set the Mouse Mode to USC\Diags (also don’t do this through remote desktop) Make this change from the iDRAC GUI. I always change the media to attach or I won’t be able to install ESX later.

Apply the setting, make sure to wait for the confirmation, otherwise it didn’t happen.

To make the actual NIC Partitioning changes, you must use the ""Dell Unified Server Configurator”, which being the noob I am, I tried to find for download, but apparently you access it during the blade server boot by pressing F10 to access the UEFI (System Services")

Then you will see something like this when it boots to UEFI

Select “Hardware Configuration”, you can do that with your mouse or the arrows on the keyboard.

Then choose tab over to HII Advanced Configuration

You will need to do the following twice, once per nic.

Then Device Configuration Menu

Change Disabled to “Enabled”

After you hit back, you will see a new options, “NIC Partitioning Configuration Menu”, select it now.

Then Select “Global Bandwidth Allocation Menu”

Tab doesn’t work on the next page, some person decided to make tab only go back and forth between the top row and the back button. However, luckily you can use your UP and Down arrows to move between rows.

Unfortunately it doesn’t appear you can Partition a NIC into less than 4, I only want 2, so using that methodology I will create one with 90 and one with 10, the other two I will give 1 to because I can’t give them 0. You are allowed to over allocate the Maximum Bandwidth. 1%=100MB, so the find 2 NIC’s I will create will be 100MB adapters I won’t end up using.

Type in your number, then hit enter, then choose the next row.

HINT: if using the mouse and it gets a bit squirrly, Play with “ALT-C” and toggle Hide Local Cursor on and off to improve mouse response.

Choose Back, then Back, then Finish, It will prompt you to Save, of course say yes, then repeat for your next adapter.

After you have done both NIC’s, Choose back, then Exit and Reboot.

Here comes the really fun part. After installing and booting into ESX, it appears that only the 2nd NIC took the partitioning command as the first one is still at 25/25/25/25 and the second one is 10/90/1/1 correctly. I made the changes again for the first NIC and rebooted again, and it works. I verified this over 3 different new blades, if anyone knows the solution, I’d love to hear it.

Credits, I found the information for this article HERE

http://www.dell.com/downloads/global/products/pwcnt/en/broadcom-npar-users-manual.pdf

http://www.dell.com/downloads/global/products/pedge/en/Dell-Broadcom-NPAR-White-Paper.pdf

http://www.dell.com/us/enterprise/p/broadcom-netxtremeii-57712-k/pd

Missing LUNS from openfiler iscsi device on ESXi 5.0

2012-01-18T13:23:00.002-07:00

My entire home vCenter cluster went down, I couldn’t get a response from anything so I rebooted the hosts and the storage, after reboots the two LUNS presented from iscsi were still there, but only one shown inside of ESX. I rebooted the openfiler iscsi with the force check disk option, after reboot I was able to see both LUNS. Thank God

See old entries in vSphere “Tasks & Events”

2012-01-17T14:53:00.002-07:00

Go into PowerCLI
Export it out so that you can search it.
[vSphere PowerCLI] C:\Program Files (x86)\VMware\Infrastructure\vSphere PowerCLI
> Get-VIEvent > c:\event.txt

How to reboot a dell Server’s iDRAC

2012-01-10T16:53:00.002-07:00

Login to iDRAC remotely, “help” is about the only command you can run besides racadm. To reboot the iDRAC management card, type “racadm racreset”, wait about a minute and it should be up again. You can run a constant ping on the iDRAC to watch it go down and back up. Other racadm commands can be found here: http://support.dell.com/support/edocs/software/smdrac3/drac5/OM53/en/ug/racugc9.htm

I love the smell of a new EMC rack in the morning

2012-01-05T09:01:00.002-07:00

PowerCLI command to enumerate VM’s mac address inside of vCenter

2011-12-15T12:52:00.002-07:00

I did not write this, I found it online but lost the link, but it works great!
ForEach ($VirtualMachine in $VM) {
    # Get the virtual machine
    $VMsView = Get-View -ViewType VirtualMachine -Property Name,Guest.Net -Filter @{"Name"="$VirtualMachine$"}
    if ($VMsView) {
      $VMsView | `
        ForEach-Object {
          $VMview = $_
          $VMView.Guest.Net | `
            Select-Object -property @{N="VM";E={$VMView.Name}},
              MacAddress,
              IpAddress,
              Connected
        }
    }
}

Trying to find a machine on a HP VirtConnect blade by Mac address

2011-12-15T10:34:00.001-07:00

->show interconnect enc0:*
Displays interconnect modules in all bays of a specific enclosure

->show interconnect-mac-table enc0:1
Displays the module MAC table for the module in bay 1 of enclosure enc0

The resulting output seems to be far too much info, with no way to sort through it. I just had putty output to a .txt file and then searched it for the mac address in question.

d15 00:50:56:3B:0D:2B Learned 4 -- --

d5 00:0C:29:73:6C:11 Learned 5 -- --

(lag) 00:50:56:3B:07:91 Learned 4 26

d7 00:50:56:8D:00:00 Learned 4 -- --

The ‘d’ number on the left tells you which blade the VM lives on.

Something I learned today about vMotion & Cisco switches

2011-11-29T16:50:00.002-07:00

When VMware says that you should put your vMotion interfaces on separate (isolated) networks from your Management adapters, they REALLY mean it. Always, always, keep your vMotion network on a dedicated nic, and on a separate isolated network. Google shows me that others have seen something similar as well.
http://www.vmadmin.info/2011/04/vmotion-unicast-flood-esxi.html

Web browser’s default auto detect proxy server feature chooses proxy server I don’t want to use.

2011-11-20T18:51:00.002-07:00

There are a couple reasons you might want to use wpad.dat that I know of.
1) you want to use a proxy server for certain websites, but not for others.
2) you don’t want to use a proxy server, you want to go directly out the internet and bypass, but machines auto detect one.
3) a mixture of the above.
The cure is wpad.dat, if you can control DHCP options, then this is a great choice. Add the following options to your scope options.
option wpad-url code 252 = text;
option wpad-url “http://webserver.com/wpad.dat”;
The wpad.dat example file below basically says, go directly to a website, unless it is in domainyouwant2useproxy4.com, then use a specified proxy
Contents of wpad.dat file below:

function FindProxyForURL(url, host)
{
   if (isPlainHostName(host))
    return "DIRECT";
   else if (shExpMatch(host, "*.domainyouwant2useproxy4.com"))
    return "PROXY proxy.domain.com:8888";
   else if (shExpMatch(host, "*"))
    return "DIRECT";
}

HP blade chassis I/O Configuration – I/O Communication - I/O Mismatch Error

2011-11-17T16:04:00.002-07:00

I encountered an issue when setting up a new hp c3000 blade chassis. I could not power on the blades due to an I/O communication issue. All bays in the back of the chassis were also reporting communication issues. Thinking that perhaps the mezzanine cards were not mapping properly to the bays, I had my team swap the Cisco 3020 Ethernet switches in the top with the Brocade FC switches on he bottom. After that, only the first bay was reporting a communication issue. If I removed the redundant bay, then I could power up and begin installing ESXi. We called hp for support, they told us that you can't have redundant FC switches, which is ridiculous. Have have hundreds of identical chassis with the same redundant FC switches. The original reason I had my team swap the bay cards is because unlike all the other BL460g7c blades they have sent us, these had the qlogic FC card in mezz port 1 and the 10Gb Ethernet mezz card in port 2. I know that the bays and mezz ports are physically mapped to one another, so to me this change made sense. After once again giving up on HP support, we put the Cisco 3020 Ethernet bay switches back into bay 1&2 and the brocade FC switches back in bay 3&4. Then the key to fixing was to swap the mezz cards in the BL460g7 servers, putting the Ethernet in mezz 1 and FC in mezz 2. After that, all I/O errors disappeared and the servers powered up as expected.

VLANs on HP BL490G7 blades using Flex-10 not working with vSphere 5 / ESXi 5

2011-11-03T11:07:00.002-06:00

Building a new vCloud 1.5 just after vSphere 5 RTM’d, we saw an issue that our guest OS’s were not working. We mapped in HP Virtual Connect Manager to multiple VLANs so that we could use vcdni with vCloud. We could ping the guest VM’s on these VLANs, but we could not do anything else to them.

The issue appears to stem from an Emulex driver & firmware version.
http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=2007397
The Solution is that HP has updated their advisory with firmware fixes to get this working
Support Flex-10 or Flex Fabric Adapters on VMware ESXi 5.0 in a Virtual Connect Environment

Determine which ESXi Host has a .vmdk file locked

2011-10-31T13:17:00.003-06:00

I was trying to remove some dead files left over from failed P2V attempts on an ESXi 5.0 host. The question I have is which host has the file locked, I don’t want to reboot more than is necessary. I am using iSCSI targets, and this command gives you a list of which Mac address has a file locked

vmkfstools –D /vmfs/volumes/<UUID>/<VMDIR>/<LOCKEDFILE.xxx>

I was able to get output that had this in it:

Hostname vmkernel: 17:00:38:46.977 cpu1:1033)Lock [type 10c00001 offset 13058048 v 20, hb offset 3499520
Hostname vmkernel: gen 532, mode 1, owner 45feb537-9c52009b-e812- 00137266e200 mtime 1174669462]

Therefore the offending Mac address is :00:13:72:66:E2:00

I opened my vSphere client and found the offending nic in my Configuration/Network Adapters by traversing each suspect host.

After rebooting that host, the file lock is gone.

Credit to this article from VMW http://kb.vmware.com/selfservice/microsites/search.do?language=en_US&cmd=displayKC&externalId=10051

Migrate VM from VMware Server 2 to vSphere ESXi 5

2011-10-31T12:38:00.002-06:00

I tried to migrate an offline VM from Server 2 to ESX 5, in the process I kept getting what seemed to be a snapshot error on a VM without snapshots. The VM would successfully migrate and then give this error on power up, or during the P2V Converter if I chose to remove snapshots.

“The parent virtual disk has been modified since the child was created. The content ID of the parent virtual disk does not match the corresponding parent content ID in the child.”

I tried about 7 times unsuccessfully to migrate the VM, then a friend suggested I modify the defaults of the P2V and choose HW version 7 instead of HW version 8, and this time it succeeded.

Storage Path Selection Policy Choices

2011-10-03T14:39:00.001-06:00

First we must talk about ALUA, it stands for “Asymmetric Logical Unit Access”, which is a feature on some mid range storage devices, (such as a Clariion) that will allow it to emulate a higher end array running Active/Active Storage Processors. Each SP still owns the LUNS, but with ALUA, an SP can process data for the other one via the backplane in the chassis.

Keep in mind:

VMware Defaults are MRU for Active/Passive & Fixed for Active/Active

MRU never falls back automatically

Always ignore my advice and follow storage vendor best practices

Great place to find recommendations of what to use on your SAN

I am assuming your not using a 3rd party PSP, SATP or MPP, such as EMC Powerpath/VE (those are generally the best option if available)

Active/Passive (think EMC CX3 without ALUA)

Fixed = Not a great choice but will work. Fixed will cause you to micro manage the ESX hosts to ensure that all hosts are on the same path. If a host has a path fail and causes it to fail over to the 'non-preferred' path this will cause trashing with the remaining hosts possibly leading to downtime.

MRU = Best practice and a good choice, this will allow the storage array to set all ESX hosts to the proper path and eliminate LUN trashing or trespassing of the LUNs; all hosts in the cluster should be set this way.

RR = Do not use, will cause trashing, data corruption and other issues.

Emulated Active/Active Mid Range Storage with ALUA Enabled, such as CX4 Clariion

Fixed = Decent Choice if you’re a control freak or have FC I/O bottlenecks.

MRU = Best Practice, vSphere 4 is aware of ALUA, this will allow the storage array to set all ESX hosts to the proper path and eliminate LUN trashing or trespassing of the LUNs; all hosts in the cluster should be set this way. Just make sure to balance the I/O among your SP’s.

RR = Works, but can cause excessive use of the backplane

Real Active/Active Higher End Storage such as Symmetrix

Fixed = Probably your best option, will require each host and LUN to be set to opposite paths; and will require micro-management of the storage infrastructure.

MRU = Works, but probably not your best option, does not load balance traffic, could force all traffic to one HBA

RR = Easiest option as long as the SAN is dedicated to the vCenter, if not, perhaps Fixed is your best option

Setting up Sysprep for vCloud

2011-08-04T14:55:00.003-06:00

Before Cloud Director can perform guest customization on virtual machines with pre-vista Windows guest operating systems, you must create a Microsoft Sysprep deployment package on each Cloud cell in your installation.

Procedure
1 Copy the Sysprep binary files for each operating system to a convenient location on a Cloud Director server host, such as /root/sysprep
Each operating system requires its own folder, MAKE SURE you use lower case for each folder, i.e. win2000 not Win2000.

Windows 2000 SysprepBinariesDirectory /win2000
Windows 2003 (32-bit) SysprepBinariesDirectory /win2k3
Windows 2003 (64-bit) SysprepBinariesDirectory /win2k3_64
Windows XP (32-bit) SysprepBinariesDirectory /winxp
Windows XP (64-bit) SysprepBinariesDirectory /winxp_64
SysprepBinariesDirectory represents a location you choose to which to copy the binaries.

Guest OS Copy Destination
1) stop the vcd services, service vmware-vcd stop

2 Run the /opt/vmware/vcloud director/deploymentPackageCreator/createSysprepPackage.sh
SysprepBinariesDirectory command.
For example, /opt/vmware/vcloud-director/deploymentPackageCreator/createSysprepPackage.sh
/root/sysprep
3 Use the service vmware-vcd restart command to restart the Cloud cell.

Installing Windows XP in the cloud (VMware vCloud Director)

2011-08-04T13:07:00.002-06:00

The first thing you’ll notice in the XP install is that it doesn’t see the default vCloud provided hard drive. Normally in vSphere, you can mount the Floppy drive with the image on ESX. In vCloud, assuming you don’t have vSphere console access, you’ll need to copy the file from one of your ESX hosts to your local hard drive, the upload it to the cloud. You’ll need to use a program such as WinSCP and grab the file from the ESX(i) host /vmimages/floppies/vmscsi.flp. Once inside of vCloud Directory, go into your Catalogs/Media tab and upload the floppy image. Assuming you’ve already uploaded your XP ISO into vCloud, mount them both and install away.

Lab Manager API deployment issue

2011-08-02T16:23:00.003-06:00

Spent the last few days troubleshooting an issue, initially, we thought it might be a fenced vs. unfenced configuration conflict, but when users try to deploy Lab Manager environments from the API tool we wrote in house, the following error was generated:

=========================================

Unable to deploy virtual machines in resource pool "LBM4".

DRS failed to find hosts to deploy the virtual machines on the resource pool "resgroup-32975".

DRS failed to find host for virtual machine "049104-DC". vCenter reported: This operation would violate a virtual machine affinity/anti-affinity rule.

Unable to find host for virtual machine "RuleViolation".

DRS failed to find host for virtual machine "049105-VCM". vCenter reported: This operation would violate a virtual machine affinity/anti-affinity rule.

Unable to find host for virtual machine "RuleViolation".

etc…..

=========================================

Since verything worked great from the Lab Manager UI, we knew it must be an API issue, it turns out the answer was that the old script from the Lab Manager 3 days was using the “Do Not Span Hosts” option.

Of course you want to span hosts in Lab Manager 4 !!

vCloud User and Resource Organization

2011-07-25T10:37:00.002-06:00

I’ve been struggling with understanding how PvDC’s(Provider vDC), Org vDC’s(Organization vDC), Organizations all relate inside of vCD. I drew up a graphic that helps me understand, hope it helps you as well.

Click the picture for a full size graphic. Thanks to our resident vCloud expert @tomralph for helping me understand

vCD (vCloud Director) Provider vDC Setup problem

2011-07-19T15:23:00.002-06:00

You click “Add Provider vDC”, then you select your vCenter server, but the clusters you want to add do not show up in the right hand panes under “resource pool” and “VC Path”. The reason is that you don’t have DRS setup on the clusters, Cloud Director will need that to create the resource pools while setting up the Provider vDC.

R.I.P. my first iPad2 4/4/11 to 7/8/11

2011-07-11T09:22:00.008-06:00

Oh how I barely knew you, only a little over 3 months old before you were dropped to you death by my 7 year old daughter on the tile floor. I always assumed we'd part because you’d get stolen out of my car. Why didn't she break my old iPad1?

Luckily, the story has a happy ending, when googling to find a repair shop, I found a number of people saying that apple had replaced their iPad for free as a one-time-only courtesy. I went down to my local apple store, they did give me a replacement iPad right there on the spot, with a warning that next time it will cost me $350 to repair it. Thank you Apple, nice job, very classy move. I was most likely going to go with a droid tablet next time, now the decision will start out leaning twards and iPad3.

Minimum vCM (VMware vCenter Configuration Manager) 5.4 install

2011-07-08T14:31:00.001-06:00

What you’ll need:

1) Hardware and Software Requirements Guide

2) Copy of vCM Software

3) License File

4) SQL 2008 R2 Software

5) SQL XML SP3

6) Windows 2008 R2 Server

7) At least one Domain Service Account

8) Add local hostname & local DC’s to the “hosts” file

9) Turn off IE ESC (at least for administrators)

10) Remote Desktop On & Firewalls Off

11) turn off UAC, enable IIS AD Auth

Can’t modify-edit-save hosts file under 2008 R2

2011-07-01T11:47:00.001-06:00

Click on the windows button, then type in “notepad” in the search box, you will see the notepad icon near the top of the window. Right click it, “run as administrator” then open the file “%windir%\System32\drivers\etc\hosts” then go ahead and edit it.

Setting google as the default search engine for IE (internet explorer)

2011-06-19T12:25:00.003-06:00

Apparently Microsoft has decided to rule the search engine world by confusion. Now when you try to select google as your search engine, you get this huge insane page called "Internet Explorer Gallery Add-ons" that suggest betty crocker and news 6 as your search engine, google is nowhere to be found, and when you do find it, it doesn't work. I say bypass that whole mess and use this link to add google as your default search engine.

http://www.microsoft.com/windows/ie/searchguide/en-en/default.mspx

NetApp can’t see our HP Blades and vice-versa

2011-06-17T15:27:00.002-06:00

After much troubleshooting, we found that the Cisco MDS did not have NPIV enabled, apparently it’s required when using a HP Flex 10 with FCoE After that we were good to go.

Moving a vCenter server into a EVC cluster.

2011-06-17T13:34:00.003-06:00

It’s a catch 22 scenario, you want to run your vCenter server in a cluster with EVC enabled, but you can’t build/manage a cluster without vCenter running so here is how you do it.

1) Build vCenter on ESX server 1

2) Build a cluster in vCenter, enable EVC and place ESX Server 2 in that cluster.

3) Power down vCenter Server(s),. open vSphere client, connect to ESX server 1, remove the vCenter VM(s) from inventory (DO NOT delete from disk). close vSphere client

4) Open vSphere client, connect directly to ESX server 2. Browse the datastore for the vCenter VM(s). Connect to them, power them up.

5) You can now connect to the vCenter and move ESX server 1 into the EVC cluster.

The problem is that I can’t find the ‘any’ key

2011-06-13T14:52:00.001-06:00

If your running HP Virtual Connect, you better upgrade to 3.17

2011-06-10T11:12:00.002-06:00

It’s hard for me to believe that DNS settings on your Flex10 will cause HP virtual Connect Manager to die, but apparently it’s true. We had this issue, one of our Flex10 adapters went offline, causing our ESX hosts to go into Isolation mode, causing our guest VM’s to all power down. I am not a happy camper with HP right now.

http://h20000.www2.hp.com/bizsupport/TechSupport/Document.jsp?objectID=c02720395&lang=en&cc=us&taskId=101&prodSeriesId=3540808&prodTypeId=329290

SUPPORT COMMUNICATION - CUSTOMER ADVISORY

Document ID: c02720395

Version: 5

Advisory: (Revision) HP Virtual Connect - Virtual Connect Manager May Be Unable to Communicate (NO_COMM) if DNS Is Enabled for Virtual Connect Ethernet Modules

NOTICE: The information in this document, including products and software versions, is current as of the Release Date. This document is subject to change without notice.

Release Date: 2011-04-14

Last Updated: 2011-04-14

DESCRIPTION

Document Version

Release Date

Details

04/14/2011

Added VC firmware v3.17 availability, VCEM clarification and an OA Customer Advisory reference to the Resolution section. Also, added clearer guidance to customers on when to perform the resolution and explanation of VC network stability in an intermittent DNS environment

04/07/2011

Updated Description to include an error message that may be seen when this issue occurs.

03/07/2011

Added clarifications to the three scenarios described in the Resolution section to ensure the full sequence of steps is followed.

03/04/2011

Added additional details regarding the circumstances in which the issue may occur. Also, added three different workaround scenarios depending on whether Enclosure Bay IP Addressing (EBIPA) or external DHCP is being used and the version of OA firmware that is in use.

02/14/2011

Original Document Release.

The HP Virtual Connect Manager (VCM) may not be able to communicate (NO_COMM) with Virtual Connect (VC) Ethernet modules in an HP BladeSystem c-Class enclosure or multiple enclosures that are part of the same Virtual Connect Domain.

IMPORTANT: Due to the possibility of a VC network outage, HP recommends that the customer follow the Resolution below as soon as possible.

The NO_COMM state may occur in a new or an existing environment when a VCM Administrator attempts to perform any of the following tasks:

Firmware Update
Add/remove/reset server blades or Onboard Administrator (OA) modules
Retrieve any VC Ethernet module status and state information (e.g. stacking links, port statistics, etc.)
Add/edit/copy/delete/assign Server Profile
Add/edit/delete VC Network
Configure Port Mirroring
Restore Domain Configuration
Change SNMP Settings
Change Advanced Ethernet Settings
Executing the "Complete VC Domain Maintenance" command in Virtual Connect Enterprise Manager (VCEM)

IMPORTANT: Attempting to execute any of the above tasks during NO_COMM adds additional risk of a network outage during the recovery steps described below.

Customers particularly susceptible to this issue have VC Modules with management IP Addresses configured in the 10.x.x.x range and configured for DNS. When this problem occurs, the VC Manager will still be accessible, but all VC Ethernet modules in the domain will be displayed with an Overall Status of "No Communication." The Virtual Connect Domain will show a "failed" status, stacking links will show "failed" and Profiles and Networks will show a status of "Unknown." In addition, the following error messages may be displayed when clicking on Domain Status from the Virtual Connect Manager Web Interface or when issuing the VC CLI command "show status":

"The domain is incapable of managing its contained VC components"

AND

"The Virtual Connect Manager is unable to communicate with the module or the Onboard Administrator. Please ensure that the module has an IP address"

This occurs if DNS is enabled for the primary VC module. The VCM may initiate a DNS reverse lookup for a very limited scope of incorrect IP addresses for the VC Ethernet modules. If this reverse lookup fails, (i.e., it is not answered by the DNS infrastructure), the primary VC module will be able to communicate correctly with the VC Ethernet modules.

If the DNS infrastructure responds to this incorrect DNS reverse lookup, then VCM attempts to communicate with the VC Ethernet modules on this incorrect IP Address and fails, triggering a NO_COMM condition. Recently, the global DNS infrastructure began responding to these limited DNS reverse lookups.

While in the NO_COMM state due to the DNS issue, the customer will not experience a VC network outage and they will still be able to pass traffic. However, if DNS environment changes cause the system to regain communication, the VC network may experience a temporary VC network outage of a few minutes. Subsequently, if the system loses communication, the customer may experience a persistent VC network outage until communication returns.

SCOPE

Any HP Virtual Connect Ethernet Modules in a c-Class BladeSystem enclosure running VC Firmware Version 1.x, 2.x or 3.x (up to and including 3.15).

RESOLUTION

This issue is resolved with Virtual Connect Firmware version 3.17 ( or later). VC 3.17 is available as follows:

http://bizsupport2.austin.hp.com/bc/docs/support/SupportManual/c02774957/c02774957.pdf

As a workaround, disable DNS for the Virtual Connect Ethernet Modules in Enclosure Bay IP Addressing (EBIPA) or external DHCP. Removing DNS from VC Modules can potentially impact the following Virtual Connect features, if configured to use DNS names:

Directory Server Settings - If a DNS name is configured for the Directory Server Address then it will no longer be resolved. The IP address will need to be configured as the Directory Server Address.
SNMP Trap Destination - If a DNS name is configured for the SNMP Trap Destination then it will no longer be resolved. The IP address will need to be configured as the SNMP Trap Destination.
From the VCM CLI - Any URL targets provided to save backup configuration or support dump will need to use an IP address and not a DNS name.

The following three workaround scenarios depend on whether Enclosure Bay IP Addressing (EBIPA) or external DHCP is being used and the version of Onboard Administrator (OA) firmware that is in use:

IMPORTANT : In all three scenarios, use the default "Administrator" account when logging into the Onboard Administrator to make EBIPA changes. Otherwise, the OA network configuration changes may not be persistent if the configuration changes were made by a non-Administrator user account, as described in OA Customer Advisory c02639172

Scenario 1 - Enclosure Bay IP Addressing is being used to provide IP Addresses to the VC Ethernet Modules and the OA firmware version is 3.00 (or higher):

Using the default Administrator account, log into the Onboard Administrator and select Enclosure Settings > Enclosure Bay IP Addressing.
Select the "Interconnect Bays" tab and remove DNS server IP address entries from the bays that include VC Ethernet Modules and click Apply.
Within 5 minutes, the DNS settings for the modules should update and normal module communication will be restored.
It is important that no VC domain changes are made until the following steps are fully completed.
If the Virtual Connect Domain is managed by Virtual Connect Enterprise Manager (VCEM):
a) If any VC Domain from the impacted VC Domain group is currently in maintenance mode go to the VCEM user interface and click "Cancel VC Domain Maintenance". Note that cancelling maintenance mode will roll back any VC Domain changes that were made while in maintenance mode. Verify that all running and pended jobs are allowed to complete before proceeding to step b).
b) Click the "VC Domains" tab, select the impacted VC domain and click "VC Domain Maintenance"
c) Click "Make Changes via VC Manager". This will release control to the VCM.
From the VCM GUI, select "Tools" => "Reset Virtual Connect Manager." This will force resynchronization of the modules if not synchronized in Step 3 above.
If the Virtual Connect Domain is managed by Virtual Connect Enterprise Manager (VCEM): Go to the VCEM user interface and click "Cancel VC Domain Maintenance". Wait for the job to complete. Cancelling maintenance mode prevents unnecessary propagation of changes to other members of the VC Domain Group.

IMPORTANT : If the NO_COMM condition was present or detected during one of the VCM administrative update tasks (listed in the DESCRIPTION section above), VCM may automatically resynchronize the modules, which would create a temporary VC domain-wide network outage during VC module initialization in either Step 3 or Step 6 above (but not both). Outage time will vary depending on the size of the VC domain.

Scenario 2 - Enclosure Bay IP Addressing is being used to provide IP Addresses to the VC Ethernet Modules and the OA firmware version is 2.60 (or earlier). If iLO DNS name registrations are statically assigned in the DNS infrastructure, move to Step 2 below:

If relying on Dynamic DNS updates for iLO, the OA firmware version must be updated to at least OA FW 3.11 before proceeding with the next step, otherwise iLO will only be reachable by IP address and there may be other ramifications to iLO LDAP Authentication.
Using the default Administrator account, log into the OA, then in Enclosure Bay IP Addressing, Select the "Interconnect Bays" tab and remove the DNS server IP address entries from the "Shared Interconnect Settings." Click Apply.
In the OA, in Enclosure Bay IP Addressing, Select the "Device Bays" tab and remove DNS server IP address entries from the "Shared Interconnect Settings" and click Apply.
Within 5 minutes, the DNS settings for the modules should update and normal module communication will be restored.
It is important that no VC domain changes are made until the following steps are fully completed.
If the Virtual Connect Domain is managed by Virtual Connect Enterprise Manager (VCEM):
a) If any VC Domain from the impacted VC Domain group is currently in maintenance mode go to the VCEM user interface and click "Cancel VC Domain Maintenance". Note that cancelling maintenance mode will roll back any VC Domain changes that were made while in maintenance mode. Verify that all running and pended jobs are allowed to complete before proceeding to step b).
b) Click the "VC Domains" tab, select the impacted VC domain and click "VC Domain Maintenance"
c) Click "Make Changes via VC Manager". This will release control to the VCM.
From the VCM GUI, select "Tools" => "Reset Virtual Connect Manager." This will force resynchronization of the modules if not synchronized in Step 4 above.
If the Virtual Connect Domain is managed by Virtual Connect Enterprise Manager (VCEM): Go to the VCEM user interface and click "Cancel VC Domain Maintenance". Wait for the job to complete. Cancelling maintenance mode prevents unnecessary propagation of changes to other members of the VC Domain Group.
IMPORTANT : If the NO_COMM condition was present or detected during one of the VCM administrative update tasks (listed in the DESCRIPTION section above), VCM may automatically resynchronize the modules, which would create a temporary VC domain-wide network outage during VC module initialization in either Step 4 or Step 7 above (but not both). Outage time will vary depending on the size of the VC domain.

Scenario 3 - External DHCP is being used to provide IP Addresses to the VC Ethernet Modules with any version of OA firmware:

On the External DHCP Scope, create an exclusion range of IP addresses (preferably only the VC Ethernet module addresses). This exclusion range needs to be configured within EBIPA on the OA.
Using the default Administrator account, log into the OA, then in Enclosure Bay IP Addressing, select the "Interconnect Bays" tab and configure the IP Addresses that were excluded in Step 1 above for bays that contain VC Ethernet modules. Do not configure DNS Server entries. Click Apply.
It is important that no VC domain changes are made until the following steps are fully completed.
Reboot the standby and primary VC modules to force them to use the new EBIPA lease. In a redundant design, the modules should be rebooted serially to mitigate downtime.

a. Reset the standby VC Module from OA.
b. Wait 15 minutes for the standby module to recover.
c. Reset the primary VC Module from OA.
d. Within 5 minutes, normal module communication will be restored.

IMPORTANT : If the NO_COMM condition was present or detected during one of the VCM administrative update tasks (listed in the DESCRIPTION section above), VCM may automatically resynchronize the modules, which would create a temporary VC domain-wide network outage during VC module initialization. Outage time will vary depending on the size of the VC domain.

This advisory will be updated if additional information becomes available.

Today is IPv6 day, what a joke.

2011-06-08T09:09:00.003-06:00

I don’t usually do editorials, but today I feel like I have to.

Being an IT person who’s spent all day, every day of the last 14 years doing IT work, I think it’s safe to say that worldwide fully routable IPv6 is not even close to a reality. I remember about 10 years ago I was told that the ‘whole internet’ was going to shut down for a few days to swap over. That simply isn’t going to happen. Newer Desktop OS’s support IPv6, on the other end, web servers like google.com support IPv6, the problem is that most ISP’s don’t support IPv6, most routers, most firewalls, IPS/IDS’s, etc… do not support it. These take months-years to fully configure, assuming you can convince the CIO to eat the cost and replace your hardware, just so other new companies who want to join the internet can have a new routable space (good luck with that). The other problem is that applications are very tied to IPv4, converting them would be a daunting task at best.

I also feel another big issue is that we really REALLY don’t need IPv6, there are FAR too many large companies that put public addresses on EVERY single desktop in the corporation, luckily most of them are not truly bidirectional routable (thank god for some security sense). In today’s economy, the smartest thing we can do is to stay on IPv4 and take back all the Class A & B networks that are not assigned to ISP’s, give those companies a class C or two, make them only use Public IP’s on their externally routed machines.

The other option is to truly move to IPv6, I have no doubt we can ‘use’ IPv6, the problem is turning off the IPv4. The only way to make this happen is to mandate it, just like the HDTV switchover, it NEVER would have happened if the government had not forced it to. If you put an end date, CIO’s will be forced to allocate the time & money to reconfigure their networks and rewrite their applications.

I made a bet with a co-worker back in 2007, the bet was a steak dinner for me if the whole internet had not moved over to IPv6 by 2011, I’m going to enjoy that steak, anybody else want to make me a bet on 2015?

So that’s what 2.7TB of ram looks like.

2011-06-06T14:28:00.001-06:00

Copy File problem in windows 7 x64

2011-05-28T10:52:00.001-06:00

I've been having problems copying from a specific desktop to my 2008R2 server for a long time now. The problem is most notable when I copy pictures to my server. To solve the problem I disabled:

IPv4 Checksum Offload
Large Send Offload (IPv4)
Large Send Offload v2 (IPv6)
TCP Checksum Offload (IPv4)
TCP Checksum Offload (IPv6)
UDP Checksum Offload (IPv4)
UDP Checksum Offload (IPv6)

Now it is working well.

My First NetApp

2011-05-25T14:36:00.003-06:00

We just purchased a NetApp FAS3240 for our new vCloud, This made me really miss my comfort zone with EMC products. But here are the initial setup challenges I faced and how I overcame them

1) Basic wiring, who puts the Shelves on the bottom really?. So the wiring diagram is giving me headaches because there is no way I’m racking top-down.

2) Basic wiring continued.. because this NetApp was purchased with HA, and there are ports labeled c0a & c0b that are described as “Controller to controller HA cable” I incorrectly assumed I needed to use them. After waiting overnight to get access to the NetApp.com website(Really? it’s 2011), I was able to download the manual and find that “they use an internal InfiniBand connector between the two controller modules, so no interconnect adapters or cabling is required.”

3) Basic console setup, This requires a serial cable (you know that thing they stopped putting on laptops 6 years ago). You’ll need to setup each controller module separately, I only configured the maintenance port, which is e0m (wouldn’t it be nice if that was on the sticker with the other e0 labels, or at least in the big blue instructions that comes with it).

4) Web Setup, https://ipaddress/api for setup then https://ipaddress/na_admin/ for management. I could only partially use the UI, it is java based, but it doesn’t work unless you modify your java setup. Start/Control Panel/Advanced Tab/Security-General/uncheck “Use TLS 1.0” then close/reopen your browser.

5) Once I had the web UI working, it was reporting that I had a Shelf Failure, this turned out to be a co-worker didn’t fully seat a power cable, but that wasn’t clearly bubbled up as a power issue, more of a vague shelf error.

6) HA was not working, instead of there being an actual problem, I went into Cluster/Manage/ “Enable Takeover” and instantly my unit went from Red to Green and everybody was happy, I could now ping both Management IP’s and HA was online.

7) I am still having issues getting email alerts and NTP working consistently, and I haven’t setup any LUNS or exports yet. That’s an adventure for next week.

Lab Manger Login Error: The username -whatever- already exists in vCenter Lab Manager. Please ask your Administrator to delete the previous user account.

2011-05-23T11:20:00.001-06:00

A user called me with this error, I had never heard of it before, but here is what happened. The user left the company, I disabled his account. He was rehired by the company, so I re-enabled his Lab Manager account. However, his AD account had been deleted (by company policy) and he had been generated a new AD account with the same login name. LBM was smart enough to know that this was not the same account and gave me a valuable error message. I deleted his LBM account, recreated him and was ready to roll.

Lab Manager HA Settings

2011-05-14T09:12:00.001-06:00

For some reason Colorado Springs has frequent late night power outages, or at least our building does. Our most common downtime event with Lab Manager (besides maintenance) is complete power outages, yes we have UPS’s, but they don’t last forever and the lab doesn’t get generators. I’ve been playing with the HA settings for a long time, this is what I’m going to try going forward. Assuming your vCenter and LBM box are hosted in the same cluster as your guests, powering on the whole environment can take hours and be a huge pain. Here is my proposed solution.

In the “Virtual Machine Options” page of the HA configuration

vCenter Server = VM Restart Priority High

LBM Server = VM Restart Priority Medium

Cluster Default settings = VM restart Priority Low

Hopefully this will allow my vCenter a chance to get up and running before all the guests boot and cripple my ESX hosts for 2+ hours.

vCD vCloud Director not allowing Media CD ISO uploads from IE

2011-05-13T10:44:00.001-06:00

It’s probably a certificate issue, I made my vCD a trusted site in IE and I installed the certificate. IE should prompt you for trusting it, but it doesn’t always. Firefox has been working well and consistently prompting me.

Best iPad Apps

2011-04-30T09:44:00.000-06:00

The apps you want depend on what you want to do. I take alot of notes and like onenote alot. I have MobilNoter because it syncs my desktop onenote to the cloud, then back/forth from my iPad. I don't use any of the office apps, I use GoodReader to allow me to open about anything. Dropbox and File Browser allow me to get files on/off the iPad. You always want the "HD" version of an app if you can find it, it will be worth it for the enhanced resolution.

Cool Apps:
Flipboard, Netflix, Directtv, Pandora, Fandango, TheOnion, Google Earth, NASA

TV:
ABC, Discovery, PBS

News:
The Weather Channel Max+, CNN, USA Today, NY Times, Drudge Report, Engadget, Macworld, Pulse News, Local News Apps, newspapers & TV stations

Search:
Google, Bing (no i'm not kidding), Yelp, Zillow

Travel:
TripAdvisor

Social:
Facebook, Twitter, Beluga, Skype

Games:
Angry Birds (all varients),Stick Golf,Risk

Sports:
Yahoo Sportacular, ESPN ScoreCenterXL

Tools:
Alarm Clock, Ping, Speed Test, RDP, Penultimate, WebEx, VMware VIEW client, VMware vSphere Client, iBooks, Kindle

Can’t sudo on a CentOS box

2011-03-30T16:00:00.001-06:00

UserXYZ is not in the sudoers file. This incident will be reported.

Just run this as root

echo 'userXYZ ALL=(ALL) ALL' >> /etc/sudoers

vSphere iPad app is pretty cool

2011-03-18T20:39:00.002-06:00

So far I really like it, gives great info at a glance, lets you do cool things like put hosts into Maint mode and gives alot of performance info.

2011-03-18T19:48:00.003-06:00

Lab Manager 4.0.3 Updated Best Practices & Design Considerations

Lab Manager is one of my favorite technologies in the market today, but before you install, beware of the limitations!

NEW IN 4.0.3 :

Windows 2008 (32-bit) support for LBM Server installation

Support for ESX & vSphere 4.1U1 & 4.0U3

End of Life for LBM

Since LBM 4 has announced and End of Life in May 1st 2013, this is the final major version of Lab Manager to be replaced to vCD (VMware Cloud Director), hence this will be my last best practices guide for it. All future guides from me will be about vCD, but as of today vCD 1.0 is primarily designed as a public cloud offering built around Multitenancy, LBM is a private cloud Software Dev/Test product, we will need the next version(s) of vCD to return many of the LBM test/dev features (which I’m assured they will). I will need those features before I make the jump in a test/dev environment to vCD from LBM.

i. 8 ESX hosts max to connect to VMFS3 Datastore (each LUN), you can use NFS to get around this, but for our use case, this is not performant enough. The VMFS limit has to do with Linked-Clones, normal ESX has a much higher limit (64).

ii. 2TB vmfs3 size limit, and don’t start there, we started at 1.2TB Luns so we could expand as needed. Avoid SSMOVE if at all possible (SSMOVE is slow and painful, but works well), if you fill up your Lun, create the extend and/or disable templates and move them to a new datastore. You can go up to 64TB with Extends, but I like smaller LUNS for performance and other reasons.

iii. Only available backups are SAN Snapshots (well the only realistic one for us), and for this to be useful, see #1 below

iv. Recommended to put vCenter & Lab Manager Servers on VM’s inside cluster on SAN with the guests (use resource pools to guarantee performance)

v. 4.0 vCenter limits

1. 32 bit has max of 2000 deployed machines and 3000 registered

2. 64 bit has max of 3000 deployed machines and 4500 registered

v. 4.1 vCenter limits (NEW!!)

64 bit has max of 10,000 deployed machines and 15,000 registered

20,000 ports per vDS (4,096 in vSphere 4.0)

(You are still limited to 3000 machines and 32 hosts per cluster, which is important for Host Spanning)

Best Practices & What we’ve learned

i. Make Luns 10x the size of your Template(s)

ii. Shared Storage is generally the first bottleneck. I used all Raid 1+0 since we observed this on our first LBM deployment and our application is database driven (disk I/O intensive). Tier your storage if possible, EMC FAST or similar technology.

iii. We have averaged between 80-120 VM’s per blade, so this means our LBM 4.0 environment should top out at approximately 80 hosts (5 full HP c7000’s) (one cluster, otherwise you lose the advantages of Host Spanning Transport Networks).

iv. LOTS of IP addresses, I recommend at least a /20 for LBM installs = 4096 IP’s, you do not want to have to re-IP lab manager guests, we’ve done that before.

v. Create a pivot Data Store, Many SAN technologies require that you present the same LUNS to a group of hosts, Think “Storage Group” from EMC. Because of this, you may want to move a VM from one storage group to another, there isn’t any way to accomplish that without having either another SAN, iSCSI, or NFS storage available that you can use for a transfer point to svmotion the VM/Template to, and then to the appropriate storage group.

vi. If using ESXi, keep one ESX per storage group for Exports, ESXi does not support SMB, so users can not export their VM’s without this.

vii. Create Gold Master Libraries for your users, helps prevent the 30 disk chain limit from being hit as often.

viii. Encourage Libraries, not snapshots

ix. Do not allow users to create\import templates, Export Only

x. Do not allow users to modify Disk, CPU or Memory on VM’s.

xi. Storage and Deployment leases are the best thing since sliced bread. Recommend between 14-28 days for both.

xii. Linked Clones are great, but Hardware Dedupe is better.

xiii. Train your users, we even went as far as to have two access levels, one for trained, one for untrained, so the untrained are less dangerous, and if they want the advanced features it forces them to get training.

NS-120 Celerra Excitement

2011-03-14T16:54:00.000-06:00

After a recent power outage, one of the 1TB Sata drives in a Raid-5 died, not a big deal. However, one of the Hot-Spare 300GB FC drives took over for it. We are nearly full allocated space-wise, so this is a scary thing. We do have a 1TB identical Hot-spare in the chassis, so my rep told me to just yank the FC Hot-spare drive that was now part of the raid (and had been for 5 days). Luckily this worked perfectly, pulling the drive forced the Celerra/Clariion to Transfer the data to the correct Hot-spare and after the new drive arrived everything went back to how it should be.

VM problems after a power outage

2011-03-14T16:48:00.002-06:00

I want to thank Colorado Springs Utilities for unexpectedly cutting our power for routine maintenance.

When trying to power our VM's back on, we got two different odd errors on separate ESX 4.0.0 hosts, both with the same resolution.

"An error occurred while communicating with the remote host."
and
"Detected an invalid snapshot configuration."

Both of which had us concerned about data corruption to say the least, however we restarted the VMW management service and they both came back to life.

service mgmt-vmware restart

Deploy from Template, can't modify unattend.xml

2011-03-08T15:03:00.002-07:00

Since you can’t modify unattend.xml or really do anything else to keep it enabled through sysprep (which auto disables it)
Do this:
Create %WINDIR%\Setup\Scripts\
Create SetupComplete.cmd (make sure no hidden extensions like SetupComplete.cmd.txt)
Edit that file, put this in there:
net user administrator /active:yes
(optional change password) net user administrator new_password
After Windows is installed, but before the logon screen appears, Windows Setup searches for the SetupComplete.cmd file in the %WINDIR%\Setup\Scripts\ directory. If a SetupComplete.cmd file is found, the file is executed. Otherwise, installation continues normally. Windows Setup logs the action in the Setupact.log file.
That should do it, now that Template should keep local admin enabled even after sysprep.

The VMware VirtualCenter Server service hung on starting

2011-02-08T15:53:00.001-07:00

Somedays I can’t see the obvious. I got this error when trying to reboot my vCenter 4.1 Lab server and the ESX 4.1 host it is on. Due to a lack of Nics I went against my own best practice and put the vCenter’s nic on a distributed switch. By doing this, in the reboot, obviously vCenter could not boot because it had no network attached to the nic (to be more specific, the distributed switch did not restart because vCenter was not running), it reported an error about not being able to connect to LDAP.

C:\ProgramData\VMware\VMware VirtualCenter\Logs\vpx*.log

[2011-02-08 13:11:32.972 02480 info 'App'] [LdapBackup] Making sure LDAP instance VMwareVCMSDS is running
[2011-02-08 13:11:32.972 02480 info 'App'] [LdapBackup] Attempting to start service ADAM_VMwareVCMSDS...
[2011-02-08 13:11:32.972 02480 info 'App'] [LdapBackup] Service stopped, starting
[2011-02-08 13:17:50.003 02480 error 'App'] [LdapBackup] Timed out waiting for service.
[2011-02-08 13:17:56.003 02480 error 'App'] [LDAP Client] Failed to connect to LDAP: 0x51 (Cannot contact the LDAP server.)
[2011-02-08 13:17:56.003 02480 error 'App'] [VpxdLdap] Failed to create LDAP client

because there was no network, the service could not start. The easiest solution was to add another nic to ESX, create a local vSwitch, tie to to that nic, then move the vCenter VM over to use it, then was able to power things back up properly.

delpart (now diskpart) in windows 7 rocks

2011-01-27T19:35:00.002-07:00

If you ever have a hard drive or USB key that has partitions you can't get rid of in Disk Manager, just use the command line tool diskpart.

Make sure to select your disk / partition / volume / vdisk (if necessary) and use the "clean command"

like Delpart, this is a VERY powerful tool that will wipe your drive clean in a second, so use with extreme caution. Make absolutely certain you have the right thing selected before you say "clean" The selected items will have "*" stars by them, see below.

DISKPART> list volume

Volume ### Ltr Label Fs Type Size Status Info
---------- --- ----------- ----- ---------- ------- --------- --------
Volume 0 H DVD-ROM 0 B No Media
Volume 1 DVD-ROM 0 B No Media
Volume 2 System Rese NTFS Partition 100 MB Healthy System
Volume 3 C NTFS Partition 297 GB Healthy Boot
* Volume 4 Removable 0 B Unusable

Happy Partitioning.

Installing VMware vCM 5.4 and trying to collect vCenter data, what version of PowerShell do I have installed?

2010-12-17T14:53:00.001-07:00

PowerShell 1.0

$Host.Version

Major  Minor  Build  Revision
-----  -----  -----  --------
1      0      0      0

PowerShell 2.0 (Windows 7 - 2008)

$Host.Version
Major  Minor  Build  Revision
-----  -----  -----  --------
2      0      -1      -1

VDR backups failing due to error -390 ( cannot quiesce virtual machine)

2010-12-08T21:27:00.000-07:00

This has been a frustrating error that started out of nowhere. Suddenly the backups of one of my VM's just stopped working. This VM does not do alot of transactions, not does it generate alot of disk activity. The error messages tend to suggest that. However, I tried powering down the VM to get the snapshot to work so the backup would complete, that worked as an interrum solution. Long term I moved the VM to other LUNS, I tried going from think to thick provisioned, that didn't help either. Finally got it fixed, I uninstalled the VMW tools, then reinstalled the tools with a 'custom' install and unchecked the option for VSS (from a suggestion I read). This seems to have fixed the issue and the online backup is running right now for the first time in a long time.

Consolidate VMDK disks

2010-12-06T15:43:00.001-07:00

I have a VM with snapshots exported from lab manager, so I am using the vmware-vdiskmanager to consolidate my 14 or so linked clones/snapshots to one flat file. the vmware-vdiskmanager can be found in server or workstation installations, it’s not in the path, you have to go to the directory to find it.

vmware-vdiskmanager -r 015495-2008R2.vmdk -t 2 2008R2.vmdk

ESXi NTP not syncing

2010-11-12T15:56:00.001-07:00

What I found was that NTP has a default maximum correction of about 1000 seconds. I had this same issue, but if I set the clocks 3-4 minutes off, they would auto correct within about 15 minutes. Using ESXi 4.1, almost none of the ntpd commands gave me any output, but I was able to see the corrections happening in the /var/log/messages.

How to build a Lab Manager Server using ESX or ESXi

2010-11-11T14:58:00.001-07:00

1) Firmware Upgrades, make sure VT & DEP are enabled.

2) If Blade Chassis

a. Make sure networking is ready, i.e. virtual connect

b. Rename blade in chassis/iLo

3) If using shared storage (and you better be)

a. Fiber Channel

i. setup VSAN’d and Zoned.

b. iSCSI

ii. Properly setup

c. Present LUN(s) to Server

4) Install ESX or ESXi

5) ESXi only

a. Change Root Password

b. Change IP to static

c. Change hostname

6) Allow SSH (if you want)

7) Install Hardwareproviders/ agents, i.e. HP ESX Agent

8) Add to vCenter

9) Apply vCenter templates (this should do the following automatically)

a. Setup Networking vNics, vMotion & Management

b. NTP

c. NFS storage

10) Patch with VUM

11) Setup Monitoring, SiteScope, vCM, etc..

12) Add to Lab Manager

The disadvantages to using ESXi instead of ESX for Lab Manager

2010-11-10T13:21:00.001-07:00

1) no timezone support, ESXi is set to UTC only. (but this doesn't cause any issues that I can find)
2) No export to SMB (window) file shares, so users can't export VMs or templates.

Where does vCenter 4 hide the sysprep location

2010-10-18T16:22:00.001-06:00

\\servername\c$\ProgramData\VMware\VMware VirtualCenter\sysprep

that’s ProgramDATA, not program files..

CentOS 5.5 iSCSI target setup with TGT

2010-10-05T11:16:00.001-06:00

I wanted a template I could add to Lab Manager so it would auto start with a new hostname and IP address every time, I used CentOS 5.5 because there is a RedHat 5 64bit tools for guest customization already available. I created a VM with a 10GB system and a 40GB data drive based off of RH5x64.

There are many good guides to get TGT installed and working, this was probably the best one.

I still had quite a few issues, but here is some good info that helped me sort it all out.

If your like me, doing something one time is nice, but ultimately not useful, you need this service to auto-start or it’s pretty much useless, therefore I think the tgtadm anc tgt-admin commands are only good for testing/setup.

Most of the guides have you setup a test file that you mount, that’s great for testing, but I wanted to map a whole physical (vm) drive, for me that was /dev/sdb. I thought for a while my problem was that I needed to partition that drive i.e. /dev/sdb1, but luckily that isn’t necessary, I tested it both ways.

All I really needed to do was this:

1) Install TGT – yum install scsi-target-utils

2) make the tgtd service auto start - chkconfig tgtd on

3) make the firewall auto-stop (this is a lab application) - chkconfig iptables off

4) modify the /etc/tgt/targets.conf file to look something like this:

<target iqn.2008-09.com.example:server.target1>
backing-store /dev/sdb
</target>

5) reboot and your done.

Here are some useful commands for testing:

Show what your current setup looks like (this is the most useful of the tgtadm commands =

tgtadm --lld iscsi --op show --mode target

The second most useful is this one, it reloads your TGT from the targets.conf file so you can see if you have it working right =

tgt-admin –e

The rest are mildly useful.

Setup a test target = tgtadm --lld iscsi --op new --mode target --tid 1 -T iqn.2001-04.com.1234:example

Setup a test LUN on above target = tgtadm --lld iscsi --op new --mode logicalunit --tid 1 --lun 1 -b /dev/sdb

Allow all IP’s to use your target = tgtadm --lld iscsi --op bind --mode target --tid 1 -I ALL (theoretically this is not needed per all documentation, but I found otherwise)

Removing or changing hosts that a Clariion sees that are Manually registered

2010-10-04T13:24:00.001-06:00

There was a mistake made on initial setup of an ESX host, and the Clariion grabbed that info and would not let go. I spent quite a while looking for “delete”, to no avail. I know the ESX hosts are correct now, but I can’t get the Clariion to refresh, the solution was to go to each SP and “Restart Management Server”, upon reboot they rescanned and found the correct info. http://SPAipaddress/setup, and http://SPBipaddress/setup

After the reboot, everything looks good and I was able to add the correct hosts to the storage groups.

Celerra Setup problems

2010-09-27T08:52:00.001-06:00

if your ever stuck trying to use /nascmd/sbin/clariion_mgmt and the command just won’t go through, try this option -skip_rules

Change Celerra IP address

2010-09-22T14:21:00.001-06:00

So I read a lot of Scott Lowe’s blog, and again he has the answer, but I needed just a bit more detail to make this work. Since i’m using his blog as a starting point i’ll reference it here. I also used this here from EMC

My Summary:

first, change the IP of the Celerra Manger, also known as the “control station”

Then change the IP of the SP’s with this command

/nasmcd/sbin/clariion_mgmt -start -spa_ip 128.221.ххх.ххх -spb_ip 128.221.ххх.ххх -use_proxy_arp

So a few things i’ll add

1) when using the clariion_mgmt tool, make sure to ssh as nasadmin, then su to root, if you ssh directly as root, the command might fail for you as it did for me with a “NAS_DB environment variable is not defined” message.

2) default passwords for nasadmin and root are “nasadmin”

3) when you setup a PPP connection to the Celerra or Clariion, do it this way:

Use the special EMC serial cable and the maintenance port on the back of the SP. Create a PPP dial up connection on your Windows laptop 115200 baud, HW flow control. Point a web browser to http://192.168.1.1/setup .

Username: clariion
Password: clariion!

I love when a question becomes a standard

2010-09-17T15:05:00.001-06:00

When VMW says that vCenter needs a static IP

2010-08-31T15:17:00.001-06:00

They mean it.

vSphere 4.1 upgrade gotchas

2010-08-27T15:16:00.000-06:00

For any of you thinking about building a new vCenter on a 64 bit OS so you can upgrade to 4.1, you should just export your database and import it on another machines. This is especially true if you use vDS's (vNetwork Distributed Switch), remember that these are stored on the vCenter and will continue to work on the ESX hosts, but will be difficult to modify/import/recreate on your new vCenter 4.1 server. To compound this difficult situation, if you are using iSCSI to reach your shared storage you will be in more deep stuff.

If you find yourself in this scenario, and you know what you named your previous vDS's, PortGroups, etc.. you may be able to recreate it without much difficulty. It gets more difficult when your using 8+ nic's per ESX Host. If you just need to start from scratch, it'll take a reboot or two of your ESX hosts (especially if you use EVC CPU masking) to get everything ripped out so you can start over again.

Remove Non-Present Devices from Device Manager

2010-08-23T14:14:00.001-06:00

After you P2V something, there are alot of old hardware devices that no longer exist, but you can’t see them by default, this article tells you how to see them so you can remove them.

http://support.microsoft.com/kb/315539

Upgrade iLo on a whole HP chassis

2010-08-20T21:20:00.001-06:00

log into HP OBA with SSH

run

update ilo all tftp://x.x.x.x/ilo2_200.bin

wait 10 minutes

How to upgrade Cisco 9124 NX-OS

2010-08-19T10:19:00.001-06:00

This is pretty simple, but for some reason hard to google4.

1) download the latest Kickstart and System image from cisco.com

2) tftp, or whatever method you like get the image onto the Cisco

3) Run this on cisco console in “config” mode

a) boot kickstart bootflash:/m9100-s2ek9-kickstart-mz.5.0.1a.bin

b) boot system bootflash:/m9100-s2ek9-mz.5.0.1a.bin

4) Save your Config, Reboot the Switch..and your done.

NOTES:

You probably don’t want to skip versions, i.e. 3-5 directly.

You can probabaly only hold 2 versions at a time, so if your on 3, upload 4, upgrade to 4, delete 3, then upload 5, update to 5, rinse, wash, repeat.

Lab Manager 4 datastore thresholds

2010-07-14T10:48:00.001-06:00

I must like to learn things the hard way, like for instance when I run out of space on a Lab Manager LUN. The default is 5GB of free space for a “yellow” alert, and 3GB for a “red” alert. For my application, we burn through 50GB in a single deployment (yes even with linked clones) because we spin up a large number of machines simultaneously. These alerts are configured on a PER LUN basis and not as a global setting (unfortunately). So you have to go into Resources/Datastores/Lun Properties, then set your limits higher. You must do that for every LUN you want to be alerted on.

Need to get a list of DC’s, but i’m not a domain admin

2010-06-29T15:04:00.001-06:00

So other than good old ADUC, lets assume you just have DNS (nslookup) access

nslookup

> set type=SRV

> _ldap._tcp.domainname.local

Install ESX 4 host certificate from windows CA

2010-06-18T09:44:00.001-06:00

I used this article and was able to get the certificate generated and installed
http://vm-help.com/esx/esx3i/change_name_and_cert.php

But after reboot I can SSH to my ESX host, but the certificate is not working the webpage is dead and I can’t add the ESX host back to vCenter

I found the problem, at the end of the article it has you put the key and cert in the wrong place

cp rui.key /host/ssl_key
cp esx.cer /host/ssl_cert

but that doesn't work.

you need to do this instead:
cp rui.key /etc/vmware/ssl/
cp esx.cer /etc/vmware/ssl/

then

mv esx.cer rui.crt (must be renamed)

How to add Media Store to Lab Manager Organization

2010-06-04T09:53:00.001-06:00

Often the media store is not available and it appears you can’t add a media store to an organizaiton. To fix this make sure you go into Organizations/OrgName Properties. Unless you add the Media Datastore to the “Datastores” it won’t let you add the Media Store to the “Media Stores” section.

Rename a Lab Manager 4 server

2010-06-04T09:50:00.001-06:00

1) Disable Deployments to that server by going to Resources/Hosts click disable.

2) Migrate live VM’s to other blades with access to same datastores by going to vCenter, choose the server, Virtual Machines tab. Select everything but the VMwareLM-ServiceVM and migrate to another host. Wait for this step to complete.

3) Unprepare the host in LabManager, wait for complete

4) In vC Put the host in Maintenance Mode, wait for complete, then Disconnect, then Remove from the cluster.

5) SSH or local access to ESX host.

Modify

/etc/hosts

/etc/sysconfig/network

Then run esxcfg-advcfg –s <HostName> /Misc/Hostname

Reboot

Wait for host to come back online

6) Add host back to vC cluster

7) Apply Host Profiles if applicable (you’ll need this for vDistributed Switches) Steps Below:

a) attach profile to host

b) check compliance

c) apply profile

d) Exit server from Maintenance Mode

8) Prepare host inside of LBM

Your done, but just to make sure everything worked as planned, migrate a VM from another host to this host, if that works, test some LBM Deployments and see if any deploy to this host.

Difficulty putting Lab Manager ESX host into maintenance mode.

2010-06-01T14:02:00.001-06:00

The job gets to 2% and hangs, you have a VM on that blade called 000000-VMwareLM-ServiceVM-x00-x00 without VMware tools installed.

The simple version is that this VM is there to help with “host spanning networks”. So inside the Lab Manager UI Resources/Hosts, you must go into properties of the host and disabled the checkbox for “Host Spanning Enabled”, then the VM will power down and you will be able to enter maintenance mode gracefully.

After the McAfee fix, your svchost.exe may be in Quarantene

2010-04-21T13:09:00.002-06:00

To fix this, copy a working svchost.exe from a good XP 32 bit SP3 machine and reboot and your good to go.

I have a copy if you want it. www.bsmith9999.com/dl/mcafee_fix.zip

Continued DAT fun with McAfee 5958

2010-04-21T12:11:00.001-06:00

I’ve had a few remote people ask how to fix this on their own.

start/run , in that box type in “cmd” (no quotes)

This should bring up a black command box window

In that window type “shutdown /a” this means abort the shutdown.

Once that is done, Go to this website.

http://www.mcafee.com/apps/downloads/security_updates/superdat.asp?region=us&segment=enterprise

Download the sdat5957.exe to a known location, such as c:\ (root of your c drive)

in the black command window you opened earlier type in “c:\sdat5957.exe /F “ this will run the patch with a “Force downgrade” to the old version of McAfee Dat.

The bad version is 5958, so once 5959 is released, that will probably be a better option that installing 5957, but again it isn’t released yet.

If you can’t go to that website on your PC due to network issues, use a second computer to download the update to a USB key or burn to a blank CD and then use that on your effected PC to repair it.

Good Luck,

Thank you McAfee for releasing Dat 5958 and destroying my windows network

2010-04-21T10:16:00.001-06:00

I hope they publish the SuperDat for 5959 soon.

windows username limit

2010-04-12T10:03:00.001-06:00

It’s really nice that windows has a 20 character username limit, but when you create one longer it doesn’t warn you, it just happily accepts the username that will never work. Yes there is a registry hack to allow windows to allow longer names, but cmon..put some intelligence into ADUC..

80003ES2LAN and Jumbo Frames

2010-04-07T11:40:00.001-06:00

No matter what Intel tells you, it doesn't work with jumbo frames.

Lab Manager Enable Local Admin Account Vista

2010-03-26T14:27:00.001-06:00

Since you can’t modify unattend.xml or really do anything else to keep it enabled through sysprep (which auto disables it)

Do this:

Create %WINDIR%\Setup\Scripts\

Create SetupComplete.cmd (make sure no hidden extensions like SetupComplete.cmd.txt)

Edit that file, put this in there:

net user administrator /active:yes

(optional change password)

net user administrator new_password

After Windows is installed, but before the logon screen appears, Windows Setup searches for the SetupComplete.cmd file in the %WINDIR%\Setup\Scripts\ directory. If a SetupComplete.cmd file is found, the file is executed. Otherwise, installation continues normally. Windows Setup logs the action in the Setupact.log file.

That should do it, now that Template should keep local admin enabled even after sysprep.

sysprep unattend.xml and Lab Manager

2010-03-26T14:07:00.001-06:00

So when the Lab Manager documentation people say don’t modify the unattend.xml, they mean it. No matter how much you modify it, the changes never make it into sysprep.

XP 64 sysprep issues in Lab Manager 4

2010-03-25T11:05:00.001-06:00

After Creating Templates for XP 64 SP1(base) and SP2 in our lab manager install, some percentage of the time after you deploy these templates you can not long into these VM’s, you type in your password, and you are re-prompted for the password. If you type the wrong password it tells you, but the right password just causes a re-prompt. After much troubleshooting I found we were not following the best practice of having machines ready to be sysprepped with the Administrator password blank. After making that change everything works again. To get the standard password back into the VM, we modified the “c:\Program Files\VMware\VMware vCenter Lab Manager\Tools\CustomizeGuest\Windows\Sysprep\WinXP_64\sysprep.inf” (that your never supposed to modify according to VMware, but i’m a rebel) from: AdminPassword="*” (* means bank) to:AdminPassword="password”

Problems with P2V of a physical server

2010-03-23T13:13:00.009-06:00

The Server is a fully patched 2003 Server. P2V should be working, everything is full 1GB, all drives are NTFS, nothing funny. I'm doing this all hot by the way, no cold booting, its a production server with no downtime allowed. Each P2V attempt died near the end, it takes several hours, so I was never watching exactly when it died. Checking the logs on the client at C:\Documents and Settings\All Users\Application Data\VMware\VMware Converter Enterprise\Logs\vmware-converter-agent-2.log . Shows me the error :

"Image processing task has failed with PlatformError fault: (1117) The request could not be performed because of an I/O device error.".

Obviously this indicates an I/O error. After some searching I found an interesting fact, if you maintain the size of the original drive the P2V or "import machine" will do a block level copy, and if there are bad blocks, then you can get this error. So if you change the destination drive size, it does a file level copy instead. There is a chance that this server's hard drive has bad blocks, but honestly I don't know. It's an old server, that's why i'm P2Ving it. After making a change to the destination partition size, this fixed my issue and the P2V was finally sucessful.

FYI, this was all done with the latest software, ESX 4.0U1, VC 4U1, etc...

XP mode for Windows 7 no longer requires processor VT

2010-03-20T16:28:00.002-06:00

Finally...

From Microsoft

Remote Desktop Error

2010-03-15T13:55:00.001-06:00

“The connection cannot be completed because the remote computer that was reached is not the one you specified. This could be caused by and outdated entry in the DNS cache. Try using the IP address of the computer instead of the name”

I saw this error connecting from Windows 7 Desktops to a specific 2008 Server in the same domain. The problem was that the clock on the server was wrong. Once I updated it, it all works again.

Lab Manager 4.0 Best Practices & Design Considerations

2010-02-17T11:24:00.002-07:00

New Version of this article here

Lab Manager is one of my favorite technologies in the market today, but before you install, beware of the limitations!

i. 8 ESX hosts max to connect to VMFS3 Datastore (each LUN), you can use NFS to get around this, but for our use case, this is not performant enough.

ii. 2TB vmfs3 size limit, and don’t start there, we started at 1.2TB Luns so we could expand as needed. Avoid SSMOVE if at all possible (SSMOVE is slow and painful, but works well), if you fill up your Lun, create the extend and/or disable templates and move them to a new datastore.

iii. Only available backups are SAN Snapshots (well the only realistic one for us), and for this to be useful, see #1 below

1. Recommended to put vCenter & Lab Manager Servers on VM’s inside cluster on SAN with the guests

iv. vCenter limits

1. 32 bit has max of 2000 deployed machines and 3000 registered

2. 64 bit has max of 3000 deployed machines and 4500 registered

Best Practices & What we’ve learned

i. Make Luns 10x the size of your Template(s)

ii. Shared Storage is generally the first bottleneck.

1. I used all Raid 1+0 since we observed this on our first LBM deployment and our application is database driven (disk I/O intensive)

iii. We have averaged between 80-100 VM’s per blade, so this means our LBM 4.0 environment should top out at approximately 32 hosts (2 full HP c7000’s) (one cluster, otherwise you lose the advantages of Host Spanning Transport Networks)

iv. LOTS of IP addresses, I recommend at least a /20 for LBM installs = 4096 IP’s, you do not want to have to re-IP lab manager guests, we’ve done that before.

v. Create Gold Master Libraries for your users, helps prevent the 30 disk chain limit from being hit as often.

vi. Encourage Libraries, not snapshots

vii. Do not allow users to create\import templates, Export Only

viii. Do not allow users to modify Disk, CPU or Memory on VM’s.

ix. Storage and Deployment leases are the best thing since sliced bread. Recommend between 14-28 days for both.

x. Train your users, we even went as far as to have two access levels, one for trained, one for untrained, so the untrained are less dangerous, and if they want the advanced features it forces them to get training.

Update to HP Firmware issue

2010-02-11T13:36:00.001-07:00

I just verified that you don't need to update the iSCSI on all 8 adaptors, just the 1st one should be enough.

Setting up Virtual Center 4 ( vCenter )

2010-02-09T15:51:00.001-07:00

If using SQL 2005/2008 then setup two databases VirtCent & VUM before installing vCenter, they can share a DB, but VMware recommends against it for performance reasons.

When Installing, your ODBC may ask for a 32bit DSN, use this to configure that “Data Source”

%systemdrive%\Windows\SysWoW64\Odbcad32.exe

Update to HP BL490c Firmware Issue

2010-02-03T11:16:00.001-07:00

After allot of troubleshooting and a very good suggestion in an email from someone reading my blog, I have a fix and a better understanding of the issue. When you update from Firmware CD 8.6 to 8.7, the version of the Broadcom Nic goes from 2.2.2 to 2.2.4, however, more specifically, there are two subversions of code that make up 2.2.2 and 2.2.4. There is “bootcode” and “iSCSI”, when you do an update from 2.2.2 to 2.2.4, you only update “bootcode” from 4.8.0 to 5.0.11 and not “iSCSI” since the version theoretically didn’t change from 3.1.5 to 3.1.5. This seems to be the issue. If while performing the FW update, you manually rewrite-reapply the iSCSI as well, then the firmware update applies, and everything works properly. If you are already on 2.2.4, you can just reapply just the iSCSI or both, either works well.

In order to successfully update your ISCSI and have a working firmware, do the following:

1) Boot from the HP 8.7 firmware CD:

Choose “Installation Options”, then click “Click here to enable install options”, then click “Allow Rewrites”, then of course “OK”.

It will warn you that you must manually select NICS and options when doing NIC firmware rewrites and downgrades. Click “OK”.

Now under “HP NC-Series Broadcom Online Firmware Upgrade, Choose “Select Devices”, When you see the screen with your NICs in it, choose “Device Details” for your first NIC in the list"

You will then see a screen like this below, Select the “iSCSI” box to reapply 3.1.5

Click “OK” then Select your 2nd device, and repeat 8 times.

After that, reboot and everything should be upgraded and working properly.

Error about “Bus 0 Enclosure 0” when creating LUN on EMC Clariion array, but your not creating a LUN on “Bus 0 Enclosure 0”

2010-01-28T16:19:00.001-07:00

The Exact error is “The RAID group being selected (RAID 1/0, RAID 1) includes disks that are located both in Bus 0 Enclosure 0 as well as some other enclosure(s). This configuration could experience Rebuilds of some disk drives following a power failure event, because Bus 0 Enclosure 0 runs longer than other enclosures during a power failure event. An alternate configuration where the disks in the RAID group are not split between enclosures in this manner, is recommended.”

The fix was to update my Flare version and everything now works normally.

Just in case you lose your password, or the Fiber Channel Cisco MDS 9124e for your c7000 blade enclosure ships with a password you don’t know like mine was

2010-01-26T15:42:00.003-07:00

1) Plug into the HP OBA Serial port, 9600 8/N/1.

2) connect interconnect <bay number>

2b) reset switch, during boot press "CTRL+c"

3) dir bootflash:

4) boot “kickstart_image” i.e. m9100-s2ek9-kickstart-mz.3.1.2a.bin

5) config t

6) (in config) admin-password <new complex password>

7) exit

8) dir bootflash:

9) load bootflash:”system image" i.e. m9100-s2ek9-mz….(not kickstart again like the broken HP instructions tell you)#note they have fixed this document in the last week.

10) login (fyi, the default username is “admin” not “admin123” like cisco.com states)

11) config t

12) (in config) snmp user admin auth md5 <new complex password>

(you need step 12 to use Cisco Device Manager to complete setup)

13) int mgmt 0

14) ip address <ip><mask>

15) exit

16) ip default-gateway <dg-ip>

17) exit

18) copy run start

19) (bonus info) if you want to leave the current bay to setup the redundant switch, hit CTRL+SHIFT+_ to exit back to the OBA so you can start over by connecting to the redundant bay.

Helped HP trace down a firmware issue today

2010-01-26T13:50:00.001-07:00

My BL490c G6 blades were unable to get network connectivity to our chassis. I had setup all the networking in HP Virtual Connect, but 6 blades worked, and 9 did not (we only purchased 15, not 16, don’t ask)

The error I received was different depending on the OS I was loading.

ESX 4.0 Update 1 gave this error: “The script 32.networking-drivers failed to execute and the installation can not continue.”

ESX 4.0 (no update 1) gave this error: Network ports Disconnected.

I was able to isolate this to a blade specific issue by swapping blades 4 and 6 and the issue followed the blades, not the bay. There are 4 types of firmware on these HP blades that I can see. BIOS, iLo, Qlogic(HBA), BC (nic). The UI shows everything but the NIC firmware version. All other blades with the latest Nic BIOS do not have any active network ports. If I downgrade them from 2.2.4 to 2.2.2(more specifically the Bootcode from 5.0.11 to 4.8.0), then they work(problem solved). It’s a real pain to see the HP Nic Bios since it only shows up when booting from the HP firmware CD. I do have the latest firmware on my c7000 chassis and flex10 switches. Just to verify this was a blade specific issue, I also swapped hard drives from a non working blade to a working blade, the issue followed the blade again.

I think I have this solved, I’ve “fixed” 5 blades so far by downgrading them, the only way I can find to downgrade the NIC bios is from the HP firmware boot CD, 8.7 has 2.2.4 and 8.6 has 2.2.2. There is a version 2.2.3, but best I can tell it’s not easy to install it, it must be done from inside a guest OS(no bootable CD) and ESX 4 Update1 does not install because of this error, I’d have to install another OS, then change FW version, then reload the blade.

I have an open HP case to get this resolved, hopefully we can get an updated firmware soon.

HP Virtual Connect Manager

2010-01-18T10:21:00.001-07:00

I was unable to definate a network set, or more accurately unable to define uplinks, the option simply wasn’t there. Turns out when they say IE8 isn’t supported, they aren’t kidding. I jumped on my ancient XP box running IE6 and suddently it works.

Ready to build a new VMWare Lab Manager 4.0 instance

2010-01-13T10:24:00.001-07:00

After months of waiting to get all the parts and pieces together, I believe I finally have them all after getting my HP GBIC’s.

Here is the basic Setup

HP c7000 with 16 BL490c Blades, Dual Quads with 72GB Ram each

Dual Flex-10 and Dual Cisco MDS 9124 blades

EMC CLARiiON CX4-960 with 15k 450GB drives

And here is the part i’ve been waiting so long for…

Diskpart in windows 7 is awesome

2009-12-29T08:53:00.001-07:00

MetaLUN Research

2009-12-15T15:33:00.002-07:00

I’ve been trying to learn about the advantages and disadvantages of EMC Clariion MetaLuns. All I really care about is performance and data protection, so this seems like a good choice

Here is some good information:

http://blog.scottlowe.org/2009/10/23/how-to-create-a-metalun/

http://www.emcstorageinfo.com/2008/03/type-and-benefits-of-meta-lun.html

http://clariionblogs.blogspot.com/2008/02/metaluns.html

http://www.emc.com/collateral/hardware/white-papers/h5773-clariion-perf-availability-release-28-firmware-wp.pdf

Does this seem bad to anyone else?

2009-12-11T13:51:00.000-07:00

My theory is that I dont need windows updates.

Copy a license to a cisco device via tftp

2009-11-24T13:38:00.001-07:00

copy tftp://ipaddress/flashimage bootflash:flashimage

red X when replying to email or sending new email via OWA on Exchange 2003

2009-11-06T13:48:00.004-07:00

This problem happens when using IE8 to connect to an exchange 2003 OWA instance that isn't fully patched (and you can't get your admin to patch it)

This fix also works for IE8 crashing while using (exchange 2003) OWA when you hit Send on an email

I grabbed the latest S/MIME Controls setupmcl.exe from the KB924334

http://www.microsoft.com/downloads/details.aspx?FamilyId=41275DEC-4C01-4C41-AA64-C9DBE5EA3F7E&displaylang=en

file version 5.3.18.6 for the executable, when installed its version 6.5.7651.60

Installed that on my Windows 7/ IE8 machine and OWA now works again.

Weird problem where a shared calendar mailbox keeps sending emails to group

2009-10-20T14:00:00.001-06:00

So I thought this going to be a case of someone adding a "outlook rule" to auto forward emails, but it turns out meeting requests were the only type of thing sent to the box where responses were forwarded to a large distro list. Turns out the fix was to go into Outlook/Tools/Options/Delegates. Someone had added a distro in there, and all meeting reuests sent to this mailbox forced a response "accepted" mail to everyone on that disto list.

Gmail's Evil account lockout tool Captcha

2009-09-30T19:05:00.002-06:00

Gmail's Evil account lockout tool, a great link for those of us who have google apps for email and have ever had gmail lock you out of your account for trying 1 bad password.

https://www.google.com/a/domainname.com/UnlockCaptcha

So you've decided to add another nic to openfiler 2.3

2009-09-22T18:52:00.002-06:00

here is the easy way (unless it auto detects, cause that's easier)

Go into /etc/modprobe.conf

Setup the eth2,3,4, whatever just like eth0,1 whatever

go into /etc/sysconfig/network-scripts

cp ifcfg-eth1 ifcfg-eth2(new interface)

open it, modify the following lines
DEVICE=eth2
and
IPADDR=192.168.10.10
so you don't have conflicts

save it, reboot, you should now see it in your UI

Openfiler 2.3 intel nics disappear after applying updates

2009-09-19T11:00:00.002-06:00

I noticed that with openfiler 2.3 my nics disappeared after applying the updates. Turns out there is a linux kernel update that changes the Nics from e1000 to e1000e .

To get these nics back, just modify your /etc/modprobe.conf and change your eth0 to e1000e from e1000.

Reboot and it should be back to a-ok

Passed the VCP4 (vcp410) exam today

2009-09-18T09:57:00.003-06:00

I got an instructor level score, but that thing isn't easy. It's probably the most bizzare test i've ever taken. Most of the questions look like this:

You did X horribly wrong and against all best practices and recommendations.

Now you want to do Y.

What happens?

I dunno..I never did X wrong, so Y always works?

I'm just glad its over.

HP ESX Management Agent Cheat Sheet

2009-09-15T11:28:00.001-06:00

To diagnose a hardware problem on an HP Server with ESX and the HP software agent installed run

hpasmcli

then you can do commands such as
show dimm
show iml
show ?
etc....