How to mirror hard disks on an AW51G

From Cassandra

Jump to: navigation, search

I/A version 7.x/AW51G Disk Mirroring Procedure

Corey Clingo

08/03/2004


  • Background

This document provides a procedure to mirror two disk drives on a version 7.x Foxboro I/A system, using the Solstice DiskSuite software from Sun, and standard disks and controllers. No special, expensive, bulky RAID hardware is required.

Naturally, this is completely unsupported by Invensys, so don’t call TAC if you have problems. Also, I take no responsibility for anything your I/A system does or does not do if you use all or part of this procedure on it.

Other configurations are possible (RAID 5, hot spares, logging filesystems) but are not discussed here. You can reference the Sun Solstice Disksuite 4.2.1 User’s Guide and the Sun Solstice DiskSuite 4.2.1 Reference Manual (or Google) for further information

In this document, it is assumed that the primary disk is c1t1d0, and the backup is c1t2d0. Ideally, you would use disks on different controllers, but the AW51G used here only had one Fiber Channel controller.


  • Let’s Get To It

Install the I/A software from Day 0 CD as you normally would, to the primary disk.

Install the Sun Solstice DiskSuite (SDS) software. I had to get the packages from www.sun.com, as Invensys doesn’t include all the CDs that normally come with a Sun workstation and Solaris (funny, since I seem to recall paying for a Solaris license). Make a directory on the AW/WP to hold the SDS packages, say, /opt/sds, and unzip the archive into it: 

# unzip SDS-4_2_1_108693_06-solsparc-03-aug-2001.zip –d /opt/sds

then install the packages: 

# cd /opt/sds/Packages
# pkgadd –d .

Disk space is cheaper than my time (that’s not saying much, I know), so I just installed all of them, though some may not be necessary (e.g., Japanese localization, the GUI front end, …). The Sun SDS documentation describes the packages in detail.

The install recommends a system reboot after it is done, so go ahead and do it (the Reboot_Station pick from the SftMnt menu in DM/FVwill work). When the system is booting, you will get a “no existing databases” messages from the md driver; ignore it for now (you will create them shortly).

Once the system is back up, replicate the partition table on the primary disk to the backup disk: 

# prtvtoc /dev/rdsk/c1t1d0s2 | fmthard –s - /dev/rdsk/c1t2d0s2

Enable booting when one disk of the mirror has failed: 

# echo “set md:mirrored_root_flag=1” >> /etc/system

If you don’t do this, the system will continue to run when one drive fails, but it will not reboot unless you delete the state database replicas from the failed drive.

Create state database replicas. These are used by the DiskSuite software to store information on the mirror (2 per disk): 

# metadb –a –f –c2 /dev/dsk/c1t1d0s3 /dev/dsk/c1t2d0s3

Set up the “submirror” devices, then the mirror devices, and finally add the submirrors to the mirror. Note that we must do this for each partition we want to use. For this document, we will use mirror device names of the form dn, where n is a single digit that corresponds to the physical partition number on the disks; e.g., a mirror of two disk slices c1t1d0s0 and c1t2d0s0 would be d0. For a mirror dn, the submirror devices will be dn0 and dn1. (NOTE: This is different than I/A version 6.x and previous, where mirror metadevices were numbered sequentially d1 through d5. If you prefer that scheme, you can do it that way, of course. The device files in /dev/md/dsk and /dev/md/rdsk range from d0 to d100-something, so you’ll probably want to use something in that range.) 

# metainit –f d10 1 1 c1t1d0s0
# metainit –f d20 1 1 c1t2d0s0
# metainit d0 –m d10

The first two commands create two submirror devices, and the third creates the mirror device and attaches the first (primary) submirror to it.

Now repeat for all other partitions: 

# metainit –f d11 1 1 c1t1d0s1
# metainit –f d21 1 1 c1t2d0s1
# metainit d1 –m d11
# metainit –f d14 1 1 c1t1d0s4
# metainit –f d24 1 1 c1t2d0s4
# metainit d4 –m d14
# metainit –f d15 1 1 c1t1d0s5
# metainit –f d25 1 1 c1t2d0s5
# metainit d5 –m d15
# metainit –f d16 1 1 c1t1d0s6
# metainit –f d26 1 1 c1t2d0s6
# metainit d6 –m d16
# metainit –f d17 1 1 c1t1d0s7
# metainit –f d27 1 1 c1t2d0s7
# metainit d7 –m d17

Tell the system that your root (/) filesystem is going to be on a mirror: 

# metaroot d0

This updates /etc/vfstab and /etc/system to mount / from the mirror.

Edit the /etc/vfstab to mount the other file systems from the new mirror devices (only the changed lines are listed below):

Before: 

/dev/dsk/c1t1d0s1	-	-	swap	-	no	-
/dev/dsk/c1t1d0s0	/dev/rdsk/c1t1d0s0	/	ufs	1	no	-
/dev/dsk/c1t1d0s6	/dev/rdsk/c1t1d0s6	/usr	ufs	1	no	-
/dev/dsk/c1t1d0s4	/dev/rdsk/c1t1d0s4	/var	ufs	1	no	-
/dev/dsk/c1t1d0s7	/dev/rdsk/c1t1d0s7	/opt	ufs	2	yes	-

After: 

/dev/md/dsk/d1	-	-	swap	-	no	-
/dev/md/dsk/d0	/dev/md/rdsk/d0	/	ufs	1	no	-
/dev/md/dsk/d6	/dev/md/rdsk/d6	/usr	ufs	1	no	-
/dev/md/dsk/d4	/dev/md/rdsk/d4	/var	ufs	1	no	-
/dev/md/dsk/d7	/dev/md/rdsk/d7	/opt	ufs	2	yes	-

Enable booting from the backup disk: 

# installboot /usr/platform/`uname –i`/lib/fs/ufs/bootblk /dev/rdsk/c1t2d0s0

Set up the NVRAM so you can boot from either disk. Note that these commands may have to be put into a user startup script run from userapps.dat, as /etc/rc2.d/S19FOXBORO (check) overwrites these settings on bootup. 

# eeprom “boot-device=disk0 disk1”
# eeprom use-nvramrc?=false


  • Legacy Historian

If you run the legacy historian, you have to take some extra steps to get Informix to use the mirror for its raw partition. If you are doing this right after a Day-0, you can follow this procedure; if you have existing Informix databases you need to save, I recommend a look at the documentation for QF1003991, and its companion document, “Complementary Installation Notes for QF1003991: Saving and Reloading Informix Databases with Mirrored Drives”.

First, make sure the Informix environment is set up (this assumes Bourne/Korn shell; the only C shell I want to see is on a beach): 

# INFORMIXDIR=/opt/informix
# PATH=$PATH:/opt/informix/bin
# export INFORMIXDIR PATH

Now stop the historian: 

# /opt/fox/hstorian/bin/histonoff –v <historian_name> OFF

Now for the fun part. The /usr/fox/bin/go_dbms script, which starts up Informix on boot, had its support for mirrored drives (added in QF1003991) removed, so we have to put it back. The following is an excerpt from that file showing the edits (the commented-out commands were from the original script): 

  #
  # Mods to support mirrored disks
  #
  #INFRM_DISK="$DISK"s5
  #PART_DISK=`echo $INFRM_DISK |cut -c11-18`
  #PART="rdsk\/"$PART_DISK
  INFRM_DISK=“/dev/md/rdsk/d5“
  PART=`echo $INFRM_DISK | sed 's#/dev/##' | sed ’s#/#\\\\/#g’`
  DISK_SIZE=`prtvtoc -h $INFRM_DISK | awk '{print $5}'`
  # end of mirrored disk mods
  . . . 

  # DISK_SIZE calculated in md mods above
  #DISK_SIZE=`prtvtoc -h ${DISK}s0 | sed -n '6,6p' | awk '{print $5}'`

After editing, run this script to properly modify the /opt/informix/etc/tbconfig file: 

# /usr/fox/bin/go_dbms

Now get Informix to rebuild its “dbspace” on the mirror device: 

# tbmode –ky
# tbinit –iy

Finally, initialize the historian tables: 

# dbinit50

If you backed up your Informix data, it can be restored now.


  • Final Steps

Reboot the station once more to come up on the mirror devices. Ignore the following errors during boot: 

WARNING: forceload of misc/md_trans failed
WARNING: forceload of misc/md_raid failed
WARNING: forceload of misc/md_hotspares failed

These occur due to the way drivers are loaded at boot, because / and /usr are on mirrors, and because of the fact that you have no “trans” (UFS log), software RAID, or hotspare devices present.

Once the station is back up, attach the secondary drive’s partitions to the mirrors: 

# metattach d0 d20
# metattach d1 d21
# metattach d4 d24
# metattach d5 d25
# metattach d6 d26
# metattach d7 d27

The backup in the mirror will begin syncing to the primary, so the disks will thrash, but this caused almost no CPU load on my AW51G with Fiber Channel disks (I did this on an idle system and so I had no I/O issues, either). The process may take several hours with large disks. You can check on this with the metastat command, or grab the dft_show_status script from a version 6.x system and edit as described below.


  • Replacing Failed Disks

To replace a failed disk, the disk must be first partitioned, and then the metareplace command run for each mirror partition. Using the device names in our previous examples, and assuming the secondary disk (c1t2d0) failed: 

# prtvtoc /dev/rdsk/c1t1d0s2 | fmthard –s - /dev/rdsk/c1t2d0s2
# metareplace –e d0 c1t2d0s0
# metareplace –e d1 c1t2d0s1
# metareplace –e d4 c1t2d0s4
# metareplace –e d5 c1t2d0s5
# metareplace –e d6 c1t2d0s6
# metareplace –e d7 c1t2d0s7


  • Other Comments

I don’t have any experience with other applications, like AIM*Star, FoxAMI, FoxCTS, etc. If they use the Unix filesystem exclusively for data storage, then no further action should be required, but I can’t say for certain.

The /usr/local/medusa/dft_show_status script on version 6.x, which prints a nice tabular listing of the mirror status, works fine on 7.x as well, once you change the METAFILES variable near the top to point to /usr/sbin rather than /usr/opt/SUNWdm/sbin. It lists Fiber Channel disks as ESPx/SCSIx, but I can live with that (update: I could, but I didn't want to, so I modified that script to just print the disk devices).


  • References

The following references provided valuable information used in preparing this document:

“Mirroring a Boot Drive Using Solstice DiskSuite”, John Richardson, 06/24/01 http://www.sunhelpdesk.com/static.php?file=Mirroring_Boot_Drive_Solstice_Disksuite.htm

Invensys QF1003991 documentation

Sun Solstice Disksuite 4.2.1 User’s Guide Sun Solstice DiskSuite 4.2.1 Reference Manual

Retrieved from "http://www.thecassandraproject.org/wiki/index.php/How_to_mirror_hard_disks_on_an_AW51G"
Personal tools