Recently my IBM DS3400 SAN gave an alert that the controller batteries had to be changed out. So after ordering some batteries, receiving them, it was time to perform a battery exchange. The steps are quite straight forward but still require a bit of forethought. I run IBM System Storage Manager 10 from within a VM running Windows 2008 R2, it is actually my VMware vCenter Server. In order, for me to exchange the batteries the IBM System Storage Manager 10 must be able to talk to the controllers either over the network or over the fibre connection. Since this is a VM, all I can do is control the SAN over the network at this time.
The first controller went with out a hitch. I had to first identify the controller(s) which went fairly easily and then I evacuated all the LUNs from that controller to the other controller. This was simply down by changing the preferred path for the LUN. So for Controller A, I set all the LUNs using Controller A to have a preferred path through Controller B. Then I was able to Place Controller A offline.
To determine which controller was controller A was simply a case of looking for which controller was ‘dark’ I.e. no Ethernet or FC traffic. Pulling the controller and replacing the battery was pretty straightforward once this was determined. With the new battery in place the controller was once more inserted. Then Controller A was brought back online. No issues! No Downtime whatsoever!
The process follows for Controller B
- Modify all LUNs so that they had a preferred path of Controller A
- Bring Controller B offline
- Extract the Controller
- Exchange the Batteries
- Reinsert Controller B
- Bring Controller B online
- Reset the battery count down to 0 days.
Unfortunately I had a problem at the second step of this process. Controller B did not pick up the DHCP address assigned to it, in fact it referenced a completely separate subnet. I thought there was a controller issue and went through many convolutions and attempts to get the proper IP and subnet. I went so far as to set the VM hosting the IBM System Storage DS Manager to be within the subnet of Controller B and this still did not work. The cables all showed light but no activity. As a last ditch I switched out the cable and viola it worked. The DHCP address was picked up and I was finally able to proceed with the process once more upgrading the battery with zero downtime.
Even though all this worked, I discovered something interesting. I still need direct FC access to upgrade the firmware as well as sync the onboard clocks. Which bothers me somewhat as at the moment there is no way to see the FC device from within the VM.
I also discovered how to make SAN volumes with Virtual LUNs which will help me later when I redo the SAN to increase the spindles per SAN volume and therefore per LUN.
Key Take away:
- Redundant Controllers help with simple updates
- Always check your network cables when there is an issue