Cisco C9500 Stuck in ROMMON Mode

Issue Description

I have a pair of C9500 switches in StackWise Virtual mode. After I disable the Secure StackWise Virtual function, the switches are looped in the reboot and ended in the ROMMON mode.

Root Cause Analysis

The bootvar shows that the switch still boots from packages.conf file. But this file is gone.

Preparing to autoboot. [Press Ctrl-C to interrupt]  0     
boot: attempting to boot from [bootflash:packages.conf]
ERROR: failed to boot from bootflash:packages.conf (Not Found)
Please wait while the system restarts.

The packages.conf file is accidentally removed when zerotizing the secure stackwise virtual sha1 key.

C9500-SW1(config)#secure-stackwise-virtual zeroize sha1-key 

**Critical Warning** - This command is irreversible 
and will zeroize the Secure-SVL-VPK by Deleting
the IOS image and config files, please use extreme
caution and confirm with Yes on each of three
iterations to complete. The system will reboot
after the command executes successfully
Do you want to proceed ?? (yes/[no]): yes

 Proceed ?? (yes/[no]): yes

 Proceed with zeroization ?? (yes/[no]): yes
% Proceeding to zeroize image. "Reload" session to remove the loaded image.
Failed to zeoize eeprom
Failed to zeoize eeprom
Removing bootflash:packages.conf
The configuration is reset and thesystem will now rebootMar 18 07:44:24.563: %PMAN-3-RPSWITCH: R0/0: pman: RP switch initiated. Critical process linux_iosd-image has failed (rc 0)
Mar 18 07:44:25.957: %PMAN-3-RELOAD_RP: R0/0: pvp: Reloading: RP switchover initiated. This RP will be reloaded

Chassis 1 reloading, reason - Reload command

Solution

1. Find the system image

rommon 3 > dir bootflash:
Size       Attributes Name
 - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
......
1017587860   -rw-     cat9k_iosxe.17.06.04.SPA.bin
......

2. Boot from the system image

rommon 2 > set
......
BOOT=bootflash:packages.conf;
......

rommon 4 > boot bootflash:cat9k_iosxe.17.06.04.SPA.bin
boot: attempting to boot from [bootflash:cat9k_iosxe.17.06.04.SPA.bin]
boot: reading file cat9k_iosxe.17.06.04.SPA.bin
######################################################################

3. Boot system in Install Mode

The parameters are as follows after the reboot into Bundle Mode.

Switch#show ver | in System image
System image file is "bootflash:cat9k_iosxe.17.06.04.SPA.bin"
Switch#show bootvar 
BOOT variable = bootflash:packages.conf;

The bootvar is bootflash:packages.conf. I remove the boot system configuration and configure the boot from bootflash:packages.conf again. Without the “install add” command, I directly reload the system. The system is booted into ROMMON mode again.

The issue is resolved by reloading the system in Install Mode.

Switch#install add file bootflash:cat9k_iosxe.17.06.04.SPA.bin activate commit  
......
Switch#show ver

......
System returned to ROM by Image Install 
System image file is "bootflash:packages.conf"
Last reload reason: Image Install