Solaris 11 LDOM Recovery
Today We are going to discuss Solaris 11 LDOM Recovery. There might be many scenarios when you can follow Solaris 11 LDOM Recovery steps we are going to discuss. The base is Solaris 11 Control Domain hosting Solaris 10 and Solaris 11 LDOMS. Control Domain is running Veritas for Volume Management.
Case 1: OS Disk on which LDOM is configured is totally bad and it has to be replaced by a new disk from SAN.
Case 2: Existing Storage on which LDOM OS is configured is getting replaced to meet the business needs of high performance and low in price which is the most common factor for SAN Migrations to cut costs and increase performance using the latest technology available from time to time.
Overall what I mean to say scenario maybe anything that prevails situation that OS SAN Disk needs to be replaced.
We will see how we can perform Solaris 11 LDOM Recovery after Disk is replaced.
You might have observed that Diskname is getting changed while new SAN is allocated from New Storage Array which leads to a non-booting operating system.
First of all, we always recommend running below commands before starting any disk replacement as a safety procedure to make sure recovery after disk replacement in case of the non-booting OS.
1. Save CDOM Configuration # ldm add-spconfig <$date> 2. Save LDOM Disk Configuration # ldm ls -o disk <ldom> > /var/tmp/ldm-ls-o-disk-ldom.txt 3. Save LDOM Constrains for Specific LDOM # ldm list-constraints -x ldom_name > ldom_name.xml 4. Save All LDOM constraints # ldm list-constraints -x > /var/tmp/cdom_ldoms.xml
So let’s review old output we have saved to validate what is exactly changed after new SAN disk allocation from the new storage array. We have saved LDOM config in CDOM /var/tmp before the start of the work.
Below is the saved output before change which shows OS was residing on disk mentioned below. Once storage migration is done you will get device name from storage you can easily identify it is the same device or different device.
# cat /var/tmp/ldm-ls-o-disk-ldom01.txt NAME ldom01 DISK NAME VOLUME TOUT ID DEVICE SERVER MPGROUP vspg1k0_5539-p1 vspg1k0_5539-p1@primary-vds0 0 disk@0 primary # cat /bar/tmp/ldm-ls-o-disk-ldom02.txt |grep 538e vspg1k0_538e vspg1k0_538e@primary-vds0 0 disk@0 primary vspg1k0_538e New Devices allocated are as below # vxdisk -eo alldgs list |egrep "7250|7251" tagmastore-usp1_7250 auto:none - - online invalid c6t50060E8008773322d1s2 lun fc >>>>Solaris 10 VM tagmastore-usp1_7251 auto:ZFS -- ZFS c6t50060E8008773322d2 lun fc >>>>Solaris 11 VM
From the above sample output, you can see that LDOM OS Disk was configured was on Disk ID:5539 and 538E and also they were from usp0. Storage Team informed that new devices are assigned LUNS are from the different array and picked as usp1 with Disk ID as 7250/7251.
So after storage migration, if you try to boot normally considering that those new LUNS virtualized same as old you will end up non-booting OS with below type of error.
Please enter a number [1]: Jul 4 13:15:36 solaris vdc: NOTICE: vdisk@0 disk access failed
In both cases, you need to remove the existing OS disk configuration as it got change usp0 to usp1 and LUN ID also got changed.
Table of Contents
Removing Existing Device
Remove the existing devices from the CDOM configuration using below. You need to stop VM to remove.You need to do the same for all LDOMs in which source and ID got changed during the allocation of new LUN replacement for OS Disk.
# ldm stop ldom01 # ldm remove-vdisk -f usp0_01d3-p1 ldom01 # ldm remove-vdsdev -f usp0_01d3-p1@primary-vds0 # ldm ls –o disk ldom01 >>> Validate there is no configuration exists anymore.
(A) Adding New Devices to Solaris 10 LDOMS using MPGROUP
# ldm add-vdsdev mpgroup=usp1_7250 /dev/vx/dmp/tagmastore-usp1_7250 usp1_7250-p1@primary-vds0 # ldm add-vdsdev mpgroup=usp1_7250 /dev/vx/dmp/tagmastore-usp1_7250 usp1_7250-p2@secondary-vds0 # ldm add-vdisk usp1_7250-p1 usp1_7250-p1@primary-vds0 ldom01 # ldm add-vdisk usp1_7250-p2 usp1_7250-p2@secondary-vds0 ldom01
You need to do the same for all LDOM. Primary and secondary here because the failover domain configured for availability purposes.
# ldm ls –o disk ldom01 Validate configuration done. # ldm start ldom01 # telnet localhost <port> OK > boot
It will boot.
(B) Adding New Devices to Solaris 11 LDOMS
There is small catch allocation like Solaris 10 is not going to work for Solaris 11 LDOM as mentioned above Disk Format is ZFS.It does not pick. Let us validate.
If you see partition details of new assigned disk you will find like below.
# format c6t50060E8008773322d2 >print>print
Total disk sectors available: 209698749 + 16384 (reserved sectors)
Part Tag Flag First Sector Size Last Sector
0 usr wm 256 99.99GB 209698782
1 unassigned wm 0 0 0
2 unassigned wm 0 0 0
3 unassigned wm 0 0 0 >>All these are not in use from s1-s6
4 unassigned wm 0 0 0
5 unassigned wm 0 0 0
6 unassigned wm 0 0 0
8 reserved wm 209698783 8.00MB 209715166 >>These contains vtoc/inode-index/inode tables etc.
New Disk Validation and finding boot device
# installboot -ei -F zfs /dev/vx/rdmp/tagmastore-usp1_7251s0 0.5.11,5.11-0.175.3.0.0.30.0 ae693bf2772f180ef478c46ef0d921a5 See above s0 partition of the disk contains bootblock and OS from the format and installboot enquiry.Also, you can validate the same using the veritas database using below.Remember mentioning partition is very important. # zdb -lc /dev/vx/rdmp/tagmastore-usp1_7251s0 -------------------------------------------------- LABEL 0 - VALID -------------------------------------------------- timestamp: 1590062854 UTC: Thu May 21 12:07:34 2020 version: 37 name: 'rpool' state: 0 txg: 81059 pool_guid: 12702437114737145633 hostid: 2230872562 hostname: '' top_guid: 18174691079512398175 guid: 18174691079512398175 vdev_children: 1 vdev_tree: type: 'disk' id: 0 guid: 18174691079512398175 path: '/dev/dsk/c1d0s0' devid: 'id1,vdc@n60060e8007dd6f000030dd6f00005539/a' phys_path: '/virtual-devices@100/channel-devices@200/disk@1:a' whole_disk: 1 metaslab_array: 27 metaslab_shift: 29 ashift: 9 asize: 106837311488 is_log: 0 create_txg: 4 -------------------------------------------------- LABEL 1 - VALID - CONFIG MATCHES LABEL 0 -------------------------------------------------- LABEL 2 - VALID - CONFIG MATCHES LABEL 0 -------------------------------------------------- LABEL 3 - VALID - CONFIG MATCHES LABEL 0
So for Solaris 11 LDOM you need to add devices using s0 partition like below.
# ldm stop ldom02 # ldm add-vdsdev /dev/vx/dmp/tagmastore-usp1_7251s0 usp1_7251-p1@primary-vds0 # ldm add-vdisk usp1_7251-p1 usp1_7251-p1@primary-vds0 ldom02
Please remember to add a secondary path also if you have configured the secondary domain as well. Also, you need to do the same for all your Solaris 11 LDOMS.
After that, you need to unbind VM to re-read new configuration to take effect.
# ldm unbind ldom02 # ldm bind ldom02 # ldm start ldom02
Connect to LDOM console
OK > devalias
OK> printenv boot-device
OK > setenv boot-device <bootdeviceaddress>
from devalias and zlib highlighted disk.If it says Disk0 then Disk 0 or else Disk1 if it says Disk1 in zlib output.
OK > boot
You will be able to boot VM. This is all about Solaris 11 LDOM Recovery with Solaris 10 LDOM recovery as well. There was a debate with Oracle on this especially for Solaris 11 you can use s2 partition to add disks as they recommend as there is a chance of corruption to disk but using the above method I have recovered all the LDOMS either Solaris 10 or Solaris 11 recently after SAN migration of OS Disk.
I hope you like this how we can recover Solaris 11 LDOMS after SAN Migration or Disk Replacement. Please share as much as possible if you like and in case of any query please do write to us we will get back as soon as possible.
If you have lost all LDOM configuration due to any reason like mainboard replacement and like to recover please see how to restore domain configurations.
If you need to understand how to configure ipmp in Solaris 11 please refer this.
