Sunday, February 23, 2014

UCS Firmware 2.2(1c) Upgrade Woes - WILL_BOOT_FAULT

This weekend while testing out the new features and compatibility of Cisco UCS firmware 2.2(1c) I ran into an issue where when a blade was upgrading using the auto upgrade feature was showing a very strange status.

I performed the normal steps:

  • Upgrade the Cisco UCS Manager software to version 2.2(1ac) or later. The required server CIMC and BIOS are also included with the download bundle.

  • Use Cisco UCS Manager to upgrade and activate the server CIMC, BIOS and boardcontroller

  • Monitor the firmware upgrade process via FSM at either the blade level (recommended) or from the service profile


Normally, this process is straight forward and very easy! One of the reasons why I love Cisco UCS, but this time after everything was said and done the blade status was showing "WILL_BOOT_FAULT."

Uh, what the heck is WILL_BOOT_FAULT and where did this come from? Ive never seen this before?! So i elected to dig into the problem and see what could be wrong.

I opened an SSH connection to UCSM and proceeded to do the following:

# scope server 2/5 (chassis 2 blade 5)
# scope boardcontroller
# show image (this will list all of the firmware versions)
# activate firmware version.0 force (select the appropriate value from the show image command)
# commit-buffer

This forced the firmware version to the blade in question and everything seemed to progress without any issues. So I thought.... Upon completion the WILL_BOOT_FAULT error came back. DARN! I then decided to move forward with an old trick that I have used before when moving from Pre-2.0.x -> 2.0.x and also when upgrading from 2.0.x -> 2.1.x which is to reset CIMC.

  • In the Navigation pane, click the Equipment tab.

  • On the Equipment tab, expand Equipment > Chassis > Chassis number > Servers, then choose your server.

  • In the Work pane, click the General tab.

  • In the Actions area, click Recover Server.

  • In the Recover Server dialog, click Reset CIMC (Server Controller), then click OK.

  • Wait for CIMC to reboot and for Cisco UCS Manager to do a shallow discovery of the server. This takes two to three minutes.


When all was said and done, the blade came happily back online and the  WILL_BOOT_FAULT error was cleared.