UNIX Essentials

Veritas Volume Manager Index

2012-12-12T12:12:00.000-08:00

Here’s a collection of my VxVM documents. Some were from my own work and some were taken from the web.

These documents were posted mainly for my personal references and for others that may find them useful.

Volume Administration

Veritas Cluster Filesystem

VCS - Implementing Cluster File System (CFS)

Veritas Disks Administration

Removing a Disk from a DiskGroup

Veritas DMP

DMP Load Balancing

Fixes, Tips and Tricks

Array Support Libraries

VxVM Manuals

Some Very Helpful Sites

Man Pages

UNIX and VxVM

VCS - Implementing Cluster File System (CFS)

2009-03-10T13:30:00.000-07:00

Veritas Cluster File System (CFS)

CFS allows the same file system to be simultaneously mounted on multiple nodes in the cluster.

The CFS is designed with master/slave architecture. Though any node can initiate an operation to create, delete, or resize data, the master node carries out the actual operation. CFS caches the metadata in memory, typically in the memory buffer cache or the vnode cache. A distributed locking mechanism, called GLM, is used for metadata and cache coherency among the multiple nodes.

The examples here are :

1. Based on VCS 5.x but should also work on 4.x
2. A new 4 node cluster with no resources defined.
3. Diskgroups and volumes will be created and shared across all nodes.

Before you configure CFS

1. Make sure you have an established Cluster and running properly.
2. Make sure these packages are installed on all nodes:

VRTScavf Veritas cfs and cvm agents by Symantec
VRTSglm Veritas LOCK MGR by Symantec

3. Make sure you have a license installed for Veritas CFS on all nodes.
4. Make sure vxfencing driver is active on all nodes (even if it is in disabled mode).

Check the status of the cluster

Here are some ways to check the status of your cluster. On these examples, CVM/CFS are not configured yet.

# cfscluster status
  NODE         CLUSTER MANAGER STATE            CVM STATE
serverA        running                        not-running                    
serverB        running                        not-running                    
serverC        running                        not-running                    
serverD        running                        not-running                    

  Error: V-35-41: Cluster not configured for data sharing application

# vxdctl -c mode
mode: enabled: cluster inactive

# /etc/vx/bin/vxclustadm nidmap
Out of cluster: No mapping information available

# /etc/vx/bin/vxclustadm -v nodestate
state: out of cluster

# hastatus -sum

-- SYSTEM STATE
-- System               State                Frozen              

A  serverA             RUNNING              0                    
A  serverB             RUNNING              0                    
A  serverC             RUNNING              0                    
A  serverD             RUNNING              0

Configure the cluster for CFS

During configuration, veritas will pick up all information that is set on your cluster configuration. And will activate CVM on all the nodes.

# cfscluster config
  
        The cluster configuration information as read from cluster
        configuration file is as follows.
                Cluster : MyCluster
                Nodes   : serverA serverB serverC serverD

  
        You will now be prompted to enter the information pertaining 
        to the cluster and the individual nodes.
  
        Specify whether you would like to use GAB messaging or TCP/UDP
        messaging. If you choose gab messaging then you will not have 
        to configure IP addresses. Otherwise you will have to provide 
        IP addresses for all the nodes in the cluster. 
   
        ------- Following is the summary of the information: ------
                Cluster         : MyCluster
                Nodes           : serverA serverB serverC serverD
                Transport       : gab
        -----------------------------------------------------------

  
        Waiting for the new configuration to be added.

        ========================================================

        Cluster File System Configuration is in progress...
        cfscluster: CFS Cluster Configured Successfully

Check the status of the cluster

Now let's check the status of the cluster. And notice that there is now a new service group cvm. CVM is required to be online before we can bring up any clustered filesystem on the nodes.

# cfscluster status

  Node             :  serverA
  Cluster Manager  :  running
  CVM state        :  running
  No mount point registered with cluster configuration


  Node             :  serverB
  Cluster Manager  :  running
  CVM state        :  running
  No mount point registered with cluster configuration


  Node             :  serverC
  Cluster Manager  :  running
  CVM state        :  running
  No mount point registered with cluster configuration


  Node             :  serverD
  Cluster Manager  :  running
  CVM state        :  running
  No mount point registered with cluster configuration

# vxdctl -c mode
mode: enabled: cluster active - MASTER
master: serverA

# /etc/vx/bin/vxclustadm nidmap
Name                             CVM Nid    CM Nid     State
serverA                         0          0          Joined: Master
serverB                         1          1          Joined: Slave
serverC                         2          2          Joined: Slave
serverD                         3          3          Joined: Slave

# /etc/vx/bin/vxclustadm -v nodestate
state: cluster member
        nodeId=0
        masterId=1
        neighborId=1
        members=0xf
        joiners=0x0
        leavers=0x0
        reconfig_seqnum=0xf0a810
        vxfen=off

# hastatus -sum

-- SYSTEM STATE
-- System               State                Frozen              

A  serverA             RUNNING              0                    
A  serverB             RUNNING              0                    
A  serverC             RUNNING              0                    
A  serverD             RUNNING              0                    

-- GROUP STATE
-- Group           System               Probed     AutoDisabled    State          

B  cvm             serverA             Y          N               ONLINE         
B  cvm             serverB             Y          N               ONLINE         
B  cvm             serverC             Y          N               ONLINE         
B  cvm             serverD             Y          N               ONLINE

Creating a Shared Disk Group and Volumes/Filesystems

This procedure creates a shared disk group for use in a cluster environment. Disks must be placed in disk groups before they can be used by the Volume Manager.

When you place a disk under Volume Manager control, the disk is initialized. Initialization destroys any existing data on the disk.

Before you begin, make sure the disks that you add to the shared-disk group must be directly attached to all the cluster nodes.

First, make sure you are on the master node:

serverA # vxdctl -c mode
mode: enabled: cluster active - MASTER
master: serverA

Initialize the disks you want to use. Make sure they are attached to all the cluster nodes. You may optionally specify the disk format.

serverA # vxdisksetup -if EMC0_1 format=cdsdisk
serverA # vxdisksetup -if EMC0_2 format=cdsdisk

Create a shared disk group with the disks you just initialized.

serverA # vxdg -s init mysharedg mysharedg01=EMC0_1 mysharedg02=EMC0_2

serverA # vxdg list
mysharedg    enabled,shared,cds   1231954112.163.serverA

Now let's add that new disk group in our cluster configuration. Giving all nodes in the cluster an option for Shared Write (sw).

serverA # cfsdgadm add mysharedg all=sw
  Disk Group is being added to cluster configuration...

Verify that the cluster configuration has been updated.

serverA # grep mysharedg /etc/VRTSvcs/conf/config/main.cf
                ActivationMode @serverA = { mysharedg = sw }
                ActivationMode @serverB = { mysharedg = sw }
                ActivationMode @serverC = { mysharedg = sw }
                ActivationMode @serverD = { mysharedg = sw }

serverA # cfsdgadm display
  Node Name : serverA
  DISK GROUP              ACTIVATION MODE
    mysharedg                    sw

  Node Name : serverB
  DISK GROUP              ACTIVATION MODE
    mysharedg                    sw

  Node Name : serverC
  DISK GROUP              ACTIVATION MODE
    mysharedg                    sw

  Node Name : serverD
  DISK GROUP              ACTIVATION MODE
    mysharedg                    sw

We can now create volumes and filesystems within the shared diskgroup.

serverA # vxassist -g mysharedg make mysharevol1 100g
serverA # vxassist -g mysharedg make mysharevol2 100g

serverA # mkfs -F vxfs /dev/vx/rdsk/mysharedg/mysharevol1
serverA # mkfs -F vxfs /dev/vx/rdsk/mysharedg/mysharevol2

Then add these volumes/filesystems to the cluster configuration so they can be mounted on any or all nodes. Mountpoints will be automatically created.

serverA # cfsmntadm add mysharedg mysharevol1 /mountpoint1
  Mount Point is being added...
  /mountpoint1 added to the cluster-configuration

serverA # cfsmntadm add mysharedg mysharevol2 /mountpoint2
  Mount Point is being added...
  /mountpoint2 added to the cluster-configuration

Display the CFS mount configurations in the cluster.

serverA # cfsmntadm display -v
  Cluster Configuration for Node: apqma519
  MOUNT POINT        TYPE      SHARED VOLUME     DISK GROUP       STATUS        MOUNT OPTIONS
  /mountpoint1    Regular      mysharevol1       mysharedg        NOT MOUNTED   crw
  /mountpoint2    Regular      mysharevol2       mysharedg        NOT MOUNTED   crw

That's it. Check you cluster configuration and try to ONLINE the filesystems on your nodes.

serverA # hastatus -sum

-- SYSTEM STATE
-- System               State                Frozen              

A  serverA             RUNNING              0                    
A  serverB             RUNNING              0                    
A  serverC             RUNNING              0                    
A  serverD             RUNNING              0                    

-- GROUP STATE
-- Group           System               Probed     AutoDisabled    State          

B  cvm             serverA             Y          N               ONLINE         
B  cvm             serverB             Y          N               ONLINE         
B  cvm             serverC             Y          N               ONLINE         
B  cvm             serverD             Y          N               ONLINE
B  vrts_vea_cfs_int_cfsmount1 serverA             Y          N               OFFLINE
B  vrts_vea_cfs_int_cfsmount1 serverB             Y          N               OFFLINE
B  vrts_vea_cfs_int_cfsmount1 serverC             Y          N               OFFLINE
B  vrts_vea_cfs_int_cfsmount1 serverD             Y          N               OFFLINE
B  vrts_vea_cfs_int_cfsmount2 serverA             Y          N               OFFLINE
B  vrts_vea_cfs_int_cfsmount2 serverB             Y          N               OFFLINE
B  vrts_vea_cfs_int_cfsmount2 serverC             Y          N               OFFLINE
B  vrts_vea_cfs_int_cfsmount2 serverD             Y          N               OFFLINE

Each volume will have its own Service group and looks really ugly, so you may want to modify your main.cf file and group them. Be creative!

Good Luck!

Determining NIC Speed

2008-05-14T11:30:00.000-07:00

To determine what NICs are configured on a system, the following command can be run:

# ifconfig -a
lo0: flags=1000849 mtu 8232 index 1
       inet 127.0.0.1 netmask ff000000
ce0: flags=1000843 mtu 1500 index 2
       inet 148.173.216.29 netmask ffffff00 broadcast 148.173.216.255
       ether 0:0:be:a9:56:5f
ce0:3: flags=1000843 mtu 1500 index 2
       inet 148.173.216.227 netmask ffffff00 broadcast 148.173.216.255
ce2: flags=1000843 mtu 1500 index 3
       inet 10.10.52.29 netmask fffffe00 broadcast 10.10.53.255
       ether 0:0:be:a9:56:5f
ce4: flags=1000843 mtu 1500 index 5
       inet 10.67.196.207 netmask fffffe00 broadcast 10.255.255.255
       ether 0:0:be:a9:56:5f

In Solaris:

To determine the link speed and/or duplex mode used by the CE driver, the following commands can be run:

    # kstat -p ce | grep link_speed
   # kstat -p ce | grep link_duplex

VxVM - Volume in NEEDSYNC State

2008-04-07T09:19:00.000-07:00

I was trying to upgrade Veritas Volume Manager from 3.5 to 4.1 when the upgrade script stopped and complained about a volume not in sync. It was a swap volume.

# vxprint -ht swapvol
Disk group: rootdg

v swapvol - ENABLED NEEDSYNC 8395200 ROUND - swap
pl swapvol-01 swapvol ENABLED ACTIVE 8395200 CONCAT - RW
sd rootdisk-01 swapvol-01 rootdisk 5159231 8395200 0 c0t8d0 ENA
pl swapvol-02 swapvol ENABLED ACTIVE 8395200 CONCAT - RW
sd disk01-04 swapvol-02 disk01 19853376 8395200 0 c1t8d0 ENA

To fix this problem, all I did was issue a resync.

# vxvol resync swapvol

# vxprint -ht swapvol
Disk group: rootdg

v swapvol - ENABLED ACTIVE 8395200 ROUND - swap
pl swapvol-01 swapvol ENABLED ACTIVE 8395200 CONCAT - RW
sd rootdisk-01 swapvol-01 rootdisk 5159231 8395200 0 c0t8d0 ENA
pl swapvol-02 swapvol ENABLED ACTIVE 8395200 CONCAT - RW
sd disk01-04 swapvol-02 disk01 19853376 8395200 0 c1t8d0 ENA

VxVM - Changing Volume Permissions

2008-04-05T23:28:00.000-07:00

I just encounter a problem with permissions on raw volumes.  The raw volumes were used by a database and permissions on these volumes were set using chown and chmod.  However, when we failover, the permissions were not carried out.

So I used the veritas function to change the ownership and permission of the volumes.

# vxedit -g  set user= group= mode=

VxVM - VxDMP Confused

2008-03-27T14:58:00.000-07:00

Today I got a call from one of my peers regarding a problem with missing LUNs.

The SAN team presented certain number of LUNs to his 3 AIX servers but all 3 had the same exact issue. All are missing 7 LUNs.

I helped out because I thought that this should be an easy one. First he told me he ran these 3 commands:

# cfgmgr
# vxdisk scandisks
# vxdctl enable

That should do it. He ran the right commands. But didn't work. So, I had to get in to the servers and look around. First, I saw some old disks from the old SHARK array which was already been disconnected. I cleared those up and re-ran those 3 commands. Still did not work.

Then, I counted all the hdisks on each HBA and compared with the hdisks listed in "vxdisk path" command. They matched. So, I thought that didn't make sense. So, I carefully reviewed the vxdisk path output and noticed that some of the devices have 4 paths and I counted 7 of those. There, I found out what the problem was. 4 paths to a LUN with only 2 HBAs active on the server. I have seen this on a Clariion but not on Symmetrix where these servers are connected to.

So, these are the steps I took to fix the problem:

1. Removed the hdisks on those 7 devices
2. Ran vxdisk scandisks ; vxdctl enable
3. Ran cfgmgr -vl fcs1 ; cfgmgr -vl fcs4
4. Ran vxdisk scandisks ; vxdctl enable

Now, how that happened, I don't know. Could be a bug in Veritas 4.0.

Here are the detailed steps:

# vxdmpadm listctlr all
CTLR-NAME       ENCLR-TYPE      STATE      ENCLR-NAME
=====================================================
scsi0           Disk            ENABLED      Disk
fscsi4          EMC             ENABLED      EMC0
fscsi1          EMC             ENABLED      EMC0
fscsi4          EMC             ENABLED      EMC1
fscsi1          EMC             ENABLED      EMC1

# lscfg | grep hdisk | grep EMC | cut -d"W" -f2 | cut -d"-" -f1 | sort -u
50060482CCB501C6
50060482CCB501C9
50060482CCAB5543
50060482CCAB554C

# lscfg | grep 50060482CCB501C6 | wc -l
      48

# lscfg | grep 50060482CCB501C9 | wc -l
      48

# lscfg | grep 50060482CCAB5543 | wc -l
      58

# lscfg | grep 50060482CCAB554C | wc -l
      58

# vxdisk list | grep EMC0 | wc -l
      48

# vxdisk list | grep EMC1 | wc -l
      51

# vxdisk list EMC1_19 | grep hdisk
hdisk97         state=enabled
hdisk107        state=enabled
hdisk211        state=enabled
hdisk182        state=enabled

# lscfg | egrep 'hdisk97|hdisk107|hdisk211|hdisk182'
* hdisk107         U5791.001.99205GR-P2-C10-T1-W50060482CCAB5543-L92000000000000  EMC Symmetrix FCP Raid5
* hdisk211         U5791.001.99205GR-P2-C10-T1-W50060482CCAB5543-LAD000000000000  EMC Symmetrix FCP Raid5
* hdisk97          U5791.001.99205LM-P2-C03-T1-W50060482CCAB554C-L92000000000000  EMC Symmetrix FCP Raid5
* hdisk182         U5791.001.99205LM-P2-C03-T1-W50060482CCAB554C-LAD000000000000  EMC Symmetrix FCP Raid5

# for i in 19 20 21 22 23 24 25
do
  vxdisk list EMC1_$i | grep hdisk | awk '{ print $1 }'
done | while read disk
do
  rmdev -dl $disk
done

# vxdisk scandisks ; vxdctl enable

# cfgmgr -vl fcs1; cfgmgr -vl fcs4

# vxdisk scandisks ; vxdctl enable

# vxdisk list | grep EMC1 | wc -l
      58

# vxdisk list EMC1_19 | grep hdisk
hdisk107        state=enabled
hdisk211        state=enabled

AIX Index

2008-03-27T09:52:00.001-07:00

Here’s a collection of my AIX documents. Some were from my own work and some were taken from the web.

These documents were posted mainly for my personal references and for others that may find them useful.

Operating System

Switching Between 32bit and 64bit mode

Tuning

Resource Limits

Redbooks

Fixes, Tips and Tricks

Some Very Helpful Sites

Man Pages

AIX - Switching Between 32bit and 64bit

2008-03-27T09:22:00.000-07:00

Before switching to 64bit mode, make sure the hardware supports it.

To verify what is running and what the hardware can support, run the following as root:

# echo "Hardware:\t`bootinfo -y` bits capable"
Hardware:       64 bits capable

# echo "Running:\t`bootinfo -K` bits mode"
Running:        32 bits mode

# ls -l /unix /usr/lib/boot/unix
lrwxrwxrwx 1 root  system  21 Aug 15 2006  /unix -> /usr/lib/boot/unix_mp
lrwxrwxrwx 1 root  system  21 Aug 15 2006  /usr/lib/boot/unix -> /usr/lib/boot/unix_mp

Switching From 32 to 64 Bit Mode

To switch from 32-bit mode to 64-bit mode run the following commands, in the given order:

1. ln -sf /usr/lib/boot/unix_64 /unix
2. ln -sf /usr/lib/boot/unix_64 /usr/lib/boot/unix
3. bosboot -ad /dev/ipldevice
4. shutdown -Fr

Switching From 64 To 32-Bit Mode

To switch from 64-bit mode to 32-bit mode run the following commands, in the given order:

1. ln -sf /usr/lib/boot/unix_mp /unix
2. ln -sf /usr/lib/boot/unix_mp /usr/lib/boot/unix
3. bosboot -ad /dev/ipldevice
4. shutdown -Fr

AIX - Resource Limits

2008-03-27T08:33:00.000-07:00

I just got a request from the development team to set certain security settings for resource limits.

Here's another good document I found from the IBM website.

Resource limits on UNIX systems (ulimit)

Resource limits is the concept where you regulate several resources consumed by a process on an UNIX operating systems. Although the resource limits are set on a per user basis, they are applied per process basis. Therefore, if a user is executing hundreds of processes, the user may consume huge amount of resources, even if the resource setting values for the user are relatively small numbers.

On UNIX systems, the ulimit command controls the limits on system resource, such as process data size, process virtual memory, and process file size. Specifically:

On Solaris systems, by default, the root user has unlimited access to these resources (for example, unlimited).
On AIX, some limits might apply to the root user.

On UNIX systems, each user can either inherit resource limits from the root user or have specific limits defined. When setting resource limits for a process, it is important to know that the limits that apply are those that are in effect for the parent process and not the limits for the user under which the process runs. For example, the IBM Directory server runs under the ldap user account that was created at install time. However, the IBM Directory server is typically started while logged in as the root user. Starting while logged in as the root user means that any limits that are in effect for the ldap user have no effect on the IBM Directory server process unless the IBM Directory server process is started while logged in as the ldap user.

To display the current user’s resource limits, use the ulimit command (see the following example):

# ulimit -Ha
time(seconds) unlimited
file(blocks) 2097151
data(kbytes) unlimited
stack(kbytes) unlimited
memory(kbytes) unlimited
coredump(blocks) unlimited
nofiles(descriptors) unlimited

# ulimit -Sa
time(seconds) unlimited
file(blocks) 2097151
data(kbytes) 131072
stack(kbytes) 32768
memory(kbytes) 32768
coredump(blocks) 2097151
nofiles(descriptors) 2000

The -H option instructs the command to display hard resource limits, while the -S option instructs the command to display soft resource limits. The hard resource limit values are set by the root user using the chuser command for each user. The soft resource limit values can be relaxed by the individual user using the ulimit command, as long as the values are smaller than the hard resource limit values.

Increasing process memory size limit

Enter the following command to check the current process data size and virtual memory size limits:

ulimit -d
ulimit -m

It is recommended that the process data size and virtual memory size be set to unlimited. Setting to unlimited can be done by modifying the following lines in the /etc/security/limits file:

default:
data = -1
rss = -1

For changes to the /etc/security/limits file to take effect, the user must log out of the current login session and log back in.

At minimum, set these size limits to 256 MB, which is the value of 256000 in the /etc/security/limits file. Increase these limits when a larger-than-default IBM Directory server cache is to be used. For more information, see the IBM Directory Server documentation.

In addition to the /etc/security/limits file, the process virtual memory size is limited by the number of segments that a process can use. By default, a process can only use one memory segment, which limits a process to 128 MB. AIX support a large memory model that is enabled through the LDR_CNTRL environment variable.

Increase file size limit

Enter the following command to check the current file size limits:

ulimit -f

It is recommended that the file size limit be set to unlimited. Setting to unlimited can be done by modifying the following lines in the /etc/security/limits file:

default:
fsize = -1

For changes to the /etc/security/limits file to take effect, the user must log out of the current login session and log back in.

Create file systems with large file support

The standard file system on AIX has a 2 GB file size limit, regardless of the ulimit setting. One way to enable files larger than the 2 GB limit is to create the file system with the Large File Enabled option. This option can be found through the Add a Journaled File System option of the smit menu. Refer to AIX documentation for additional information and file system options.

Removing a Disk from a DiskGroup

2008-03-24T21:32:00.000-07:00

Details:

Disks that are no longer required and not being used can be removed from its diskgroup.

The command to remove a disk from a diskgroup is:

vxdg -g <dg> rmdisk <disk>

The example below is from an actual server with EMC LUNs. I normally name the disks that includes the last few digits of the Storage WWN plus the actual LUN number.

# vxdisk list
DEVICE       TYPE            DISK            GROUP        STATUS
EMC0_0       auto:cdsdisk    dvgy404B23L002  dvgy404      online
EMC0_1       auto:cdsdisk    dvgy404B23L009  dvgy404      online
EMC0_2       auto:cdsdisk    dvgy404B23L057  dvgy404      online
EMC0_3       auto:cdsdisk    dvgy404B23L035  dvgy404      online
EMC0_4       auto:cdsdisk    dvgy404B23L063  dvgy404      online
EMC0_5       auto:cdsdisk    dvgy404B23L056  dvgy404      online
EMC0_6       auto:cdsdisk    dvgy404B23L027  dvgy404      online
EMC0_7       auto:cdsdisk    dvgy404B23L030  dvgy404      online
EMC0_8       auto:cdsdisk    dvgy404B23L031  dvgy404      online

# vxdg -g dvgy404 rmdisk dvgy404B23L031

# vxdisk list
DEVICE       TYPE            DISK            GROUP        STATUS
EMC0_0       auto:cdsdisk    dvgy404B23L002  dvgy404      online
EMC0_1       auto:cdsdisk    dvgy404B23L009  dvgy404      online
EMC0_2       auto:cdsdisk    dvgy404B23L057  dvgy404      online
EMC0_3       auto:cdsdisk    dvgy404B23L035  dvgy404      online
EMC0_4       auto:cdsdisk    dvgy404B23L063  dvgy404      online
EMC0_5       auto:cdsdisk    dvgy404B23L056  dvgy404      online
EMC0_6       auto:cdsdisk    dvgy404B23L027  dvgy404      online
EMC0_7       auto:cdsdisk    dvgy404B23L030  dvgy404      online
EMC0_8       auto:cdsdisk    -               -            online

The disk has been removed from the diskgroup.

VxVM - Fixing 'online dgdisabled'

2008-01-11T12:50:00.000-08:00

What to do when "vxdisk list" shows status of 'online dgdisabled'.

Details:

aixsrv01:# vxdisk -o alldgs list
DEVICE           TYPE            DISK             GROUP        STATUS
EMC_CLARiiON0_0  auto:cdsdisk    EMC_CLARiiON0_0  dygy2502     online
EMC_CLARiiON0_1  auto:cdsdisk    -               (dvgy2500)    online
EMC_CLARiiON0_2  auto:cdsdisk    EMC_CLARiiON0_4  dvgyappl     online
EMC_CLARiiON0_3  auto:cdsdisk    EMC_CLARiiON0_3  dvgy2503     online
EMC_CLARiiON0_4  auto:cdsdisk    EMC_CLARiiON0_4  dvgy2504     online
EMC_CLARiiON0_5  auto:cdsdisk    EMC_CLARiiON0_5  dvgy25       online
EMC_CLARiiON0_6  auto:cdsdisk    EMC_CLARiiON0_9  dvgy26       online dgdisabled
EMC_CLARiiON0_7  auto:cdsdisk    EMC_CLARiiON0_8  dygy2501     online
EMC_CLARiiON0_8  auto:cdsdisk    -               (dvgy2506)    online
EMC_CLARiiON0_9  auto:cdsdisk    -               (dvgy2505)    online
EMC_CLARiiON0_10 auto:cdsdisk    -               (dvgy2507)    online
EMC_CLARiiON0_11 auto:cdsdisk    EMC_CLARiiON0_11 dvgy25db2    online

This situation can happen when every disk in a disk group is lost from a bad power supply, power turned off to the disk array, cable disconnected, zoning problems, etc.

The disk group will not show in the output from vxprint -ht.

aixsrv01:# vxprint -htg dvgy26
VxVM vxprint ERROR V-5-1-582 Disk group dvgy26: No such disk group

The disk group will show as disabled in vxdg list:

aixsrv01:# vxdg list
NAME         STATE                ID
dygy2501     enabled,cds          1189621899.78.aixsrv01
dvgyappl     enabled,cds          1190904062.52.aixsrv01
dvgy25       enabled,cds          1189622068.88.aixsrv01
dvgy25db2    enabled,cds          1189622043.86.aixsrv01
dvgy26       disabled             1189538508.74.aixsrv01
dvgy2503     enabled,cds          1189621988.82.aixsrv01
dvgy2504     enabled,cds          1189622014.84.aixsrv01
dygy2502     enabled,cds          1189621955.80.aixsrv01

This is the output of vxdg list dvgy26:

aixsrv01:# vxdg list dvgy26
Group:            dvgy26
dgid:             1189538508.74.aixsrv01
import-id:        1024.22
flags:            disabled
version:          0
alignment:        0 (bytes)
local-activation: read-write
ssb:              off
detach-policy:    invalid
copies:           nconfig=default nlog=default
config:           seqno=0.1103 permlen=1280 free=1259 templen=11 loglen=192
config disk EMC_CLARiiON0_6 copy 1 len=1280 state=clean online
log disk EMC_CLARiiON0_6 copy 1 len=192

Your filesystems will of course fail and the operating system will report it as corrupted.

aixsrv01:# df -k > /dev/null
df: /db2/dwins26q: I/O error
df: /backup: I/O error
df: /db/dwdb26q/dwins25q/NODE0000: I/O error
df: /db/dwins26q/dwdb26q/syscatspace/NODE0000: I/O error
df: /db/dwins26q/dwdb26q/tempspace01/NODE0000: I/O error
df: /dba/dwins26q: I/O error
df: /db2/dwmysld: I/O error
df: /backup/wiminst: I/O error

Once you have confirmed that the disk storage is powered-up, running, and operational and if the LUNs are in a SAN, zoning is configured right, this problem can be remedied by deporting, and then importing the disk group:

# vxdg deport dvgy26

# vxdg import dvgy26
VxVM vxdg ERROR V-5-1-587 Disk group dvgy26: import failed: No valid disk found containing disk group

If volume manager can't see the disks, and your SAN or storage administrator has confirmed that the LUNs were fine and presented to your server, then rescan the disks.

aixsrv01:# vxdisk scandisks
aixsrv01:# vxdctl enable
aixsrv01:# vxdg import dvgy26

Otherwise, your diskgroup should be showing up as enabled.

aixsrv01:# vxdg list
NAME         STATE           ID
dygy2501     enabled,cds          1189621899.78.aixsrv01
dvgyappl     enabled,cds          1190904062.52.aixsrv01
dvgy25       enabled,cds          1189622068.88.aixsrv01
dvgy25db2    enabled,cds          1189622043.86.aixsrv01
dvgy26       enabled,cds          1189538508.74.aixsrv01
dvgy2503     enabled,cds          1189621988.82.aixsrv01
dvgy2504     enabled,cds          1189622014.84.aixsrv01
dygy2502     enabled,cds          1189621955.80.aixsrv01

The disk group now shows in vxprint -ht with the volumes and plexes disabled:

aixsrv01:# vxprint -htg dvgy26
DG NAME         NCONFIG      NLOG     MINORS   GROUP-ID
ST NAME         STATE        DM_CNT   SPARE_CNT         APPVOL_CNT
DM NAME         DEVICE       TYPE     PRIVLEN  PUBLEN   STATE
RV NAME         RLINK_CNT    KSTATE   STATE    PRIMARY  DATAVOLS  SRL
RL NAME         RVG          KSTATE   STATE    REM_HOST REM_DG    REM_RLNK
CO NAME         CACHEVOL     KSTATE   STATE
VT NAME         NVOLUME      KSTATE   STATE
V  NAME         RVG/VSET/CO  KSTATE   STATE    LENGTH   READPOL   PREFPLEX UTYPE
PL NAME         VOLUME       KSTATE   STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME         PLEX         DISK     DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME         PLEX         VOLNAME  NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
SC NAME         PLEX         CACHE    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
DC NAME         PARENTVOL    LOGVOL
SP NAME         SNAPVOL      DCO

dg dvgy26       default      default  9000     1189538508.74.aixsrv01

dm EMC_CLARiiON0_9 EMC_CLARiiON0_6 auto 2048   67102464 -

v  backup       -            DISABLED ACTIVE   4194304  SELECT    -        fsgen
pl backup-01    backup       DISABLED ACTIVE   4194304  CONCAT    -        RW
sd EMC_CLARiiON0_9-02 backup-01 EMC_CLARiiON0_9 8388608 4194304 0 EMC_CLARiiON0_6 ENA

v  db           -            DISABLED ACTIVE   1048576  SELECT    -        fsgen
pl db-01        db           DISABLED ACTIVE   1048576  CONCAT    -        RW
sd EMC_CLARiiON0_9-04 db-01  EMC_CLARiiON0_9 16777216 1048576 0   EMC_CLARiiON0_6 ENA

v  dba          -            DISABLED ACTIVE   4194304  SELECT    -        fsgen
pl dba-01       dba          DISABLED ACTIVE   4194304  CONCAT    -        RW
sd EMC_CLARiiON0_9-03 dba-01 EMC_CLARiiON0_9 12582912 4194304 0   EMC_CLARiiON0_6 ENA

v  db2          -            DISABLED ACTIVE   8388608  SELECT    -        fsgen
pl db2-01       db2          DISABLED ACTIVE   8388608  CONCAT    -        RW
sd EMC_CLARiiON0_9-01 db2-01 EMC_CLARiiON0_9 0 8388608  0         EMC_CLARiiON0_6 ENA

v  dwmysld      -            DISABLED ACTIVE   2097152  SELECT    -        fsgen
pl dwmysld-01   dwmysld      DISABLED ACTIVE   2097152  CONCAT    -        RW
sd EMC_CLARiiON0_9-09 dwmysld-01 EMC_CLARiiON0_9 55574528 2097152 0 EMC_CLARiiON0_6 ENA

v  lg1          -            DISABLED ACTIVE   10485760 SELECT    -        fsgen
pl lg1-01       lg1          DISABLED ACTIVE   10485760 CONCAT    -        RW
sd EMC_CLARiiON0_9-08 lg1-01 EMC_CLARiiON0_9 45088768 10485760 0  EMC_CLARiiON0_6 ENA

v  syscat       -            DISABLED ACTIVE   2097152  SELECT    -        fsgen
pl syscat-01    syscat       DISABLED ACTIVE   2097152  CONCAT    -        RW
sd EMC_CLARiiON0_9-05 syscat-01 EMC_CLARiiON0_9 17825792 2097152 0 EMC_CLARiiON0_6 ENA

v  tp01         -            DISABLED ACTIVE   4194304  SELECT    -        fsgen
pl tp01-01      tp01         DISABLED ACTIVE   4194304  CONCAT    -        RW
sd EMC_CLARiiON0_9-07 tp01-01 EMC_CLARiiON0_9 40894464 4194304 0  EMC_CLARiiON0_6 ENA

v  ts01         -            DISABLED ACTIVE   20971520 SELECT    -        fsgen
pl ts01-01      ts01         DISABLED ACTIVE   20971520 CONCAT    -        RW
sd EMC_CLARiiON0_9-06 ts01-01 EMC_CLARiiON0_9 19922944 20971520 0 EMC_CLARiiON0_6 ENA

Verify that the disks on the diskgroup are all online.

aixsrv01:# vxdisk -o alldgs list
DEVICE           TYPE            DISK             GROUP        STATUS
EMC_CLARiiON0_0  auto:cdsdisk    EMC_CLARiiON0_0  dygy2502     online
EMC_CLARiiON0_1  auto:cdsdisk    -               (dvgy2500)    online
EMC_CLARiiON0_2  auto:cdsdisk    EMC_CLARiiON0_4  dvgyappl     online
EMC_CLARiiON0_3  auto:cdsdisk    EMC_CLARiiON0_3  dvgy2503     online
EMC_CLARiiON0_4  auto:cdsdisk    EMC_CLARiiON0_4  dvgy2504     online
EMC_CLARiiON0_5  auto:cdsdisk    EMC_CLARiiON0_5  dvgy25       online
EMC_CLARiiON0_6  auto:cdsdisk    EMC_CLARiiON0_9  dvgy26       online
EMC_CLARiiON0_7  auto:cdsdisk    EMC_CLARiiON0_8  dygy2501     online
EMC_CLARiiON0_8  auto:cdsdisk    -               (dvgy2506)    online
EMC_CLARiiON0_9  auto:cdsdisk    -               (dvgy2505)    online
EMC_CLARiiON0_10 auto:cdsdisk    -               (dvgy2507)    online
EMC_CLARiiON0_11 auto:cdsdisk    EMC_CLARiiON0_11 dvgy25db2    online

Now the volumes can be started:

aixsrv01:# vxvol -g dvgy26 startall

aixsrv01:# vxprint -htg dvgy26 | egrep '^v|^pl'
v  backup       -            ENABLED  ACTIVE   4194304  SELECT    -        fsgen
pl backup-01    backup       ENABLED  ACTIVE   4194304  CONCAT    -        RW
v  db           -            ENABLED  ACTIVE   1048576  SELECT    -        fsgen
pl db-01        db           ENABLED  ACTIVE   1048576  CONCAT    -        RW
v  dba          -            ENABLED  ACTIVE   4194304  SELECT    -        fsgen
pl dba-01       dba          ENABLED  ACTIVE   4194304  CONCAT    -        RW
v  db2          -            ENABLED  ACTIVE   8388608  SELECT    -        fsgen
pl db2-01       db2          ENABLED  ACTIVE   8388608  CONCAT    -        RW
v  dwmysld      -            ENABLED  ACTIVE   2097152  SELECT    -        fsgen
pl dwmysld-01   dwmysld      ENABLED  ACTIVE   2097152  CONCAT    -        RW
v  lg1          -            ENABLED  ACTIVE   10485760 SELECT    -        fsgen
pl lg1-01       lg1          ENABLED  ACTIVE   10485760 CONCAT    -        RW
v  syscat       -            ENABLED  ACTIVE   2097152  SELECT    -        fsgen
pl syscat-01    syscat       ENABLED  ACTIVE   2097152  CONCAT    -        RW
v  tp01         -            ENABLED  ACTIVE   4194304  SELECT    -        fsgen
pl tp01-01      tp01         ENABLED  ACTIVE   4194304  CONCAT    -        RW
v  ts01         -            ENABLED  ACTIVE   20971520 SELECT    -        fsgen
pl ts01-01      ts01         ENABLED  ACTIVE   20971520 CONCAT    -        RW

The filesystems on these volumes may not be in consistent state. So, run a filesystem check before mounting them.

aixsrv01:# for i in `grep dvgy26 /etc/filesystems | awk '{ print $3 }'`
> do
>   fsck -y $i
>   mount $i
> done

note: This example was taken from an AIX server, but all veritas commands here will work on all UNIX platforms.

HBA - Using Qlogic SANsurfer CLI

2008-01-08T13:22:00.000-08:00

Introduction

The SANsurfer FC HBA CLI application provides a command line interface (CLI) that lets you easily install, configure, and deploy QLogic Fibre Channel (FC) host bus adapters (HBAs). It also provides robust diagnostic and troubleshooting capabilities and useful statistical information to optimize SAN performance. This application can only configure HBAs on the local machine upon which the application is installed.

SANsurfer FC HBA CLI is a simplified, condensed version of the SANsurfer FC HBA Manager GUI.

SANsurfer FC HBA CLI can be operated in two modes:

Interactive Mode

Do one of the following to start SANsurfer FC HBA CLI in interactive mode:

scli INT

scli

NOTE: When starting SANsurfer FC HBA CLI on a Solaris console serial port
connection, the application may be slow to launch. To resolve this issue,
specify the INT flag, as shown above.

Here is a sample:

sunsrv01# scli



        SANsurfer FC HBA CLI

        v1.7.0 Build 12

    Main Menu

    1:  Display System Information
    2:  Display HBA Settings
    3:  Display HBA Information
    4:  Display Device List
    5:  Display LUN List
    6:  Configure HBA Settings
    7:  Boot Device Settings
    8:  HBA Utilities
    9:  Flash Beacon
   10:  Diagnostics
   11:  Statistics
   12:  Help
   13:  Quit


        Enter Selection:

Non-interactive Mode

Type the following in a command window to start SANsurfer FC HBA CLI in
non-interactive mode:

scli <Parameters>

SANsurfer FC HBA CLI executes the command options, then terminates.
To list all of the available command line parameters and the SANsurfer FC HBA CLI
version, type the following:

sunsrv01# scli -h
SANsurfer FC HBA CLI
v1.7.0 Build 12
Copyright 2003-2007 QLogic Corp.
All rights reserved.
Command Line QLogic FC Host Bus Adapters
Build Type:  Release
Build Date:  Jan 31 2007 16:40:23

Usage: scli [options]


Options:

   [ int ]              - Starts interactive mode.

   -g                   - Displays the system information.


   -c  [ <all> ]         - Displays parameter settings for all HBAs.
   -c ( <hba no> | <hba wwpn> )
                        - Displays parameter settings for a specific HBA


   -i [ <all> ]         - Displays all HBAs information.
   -i ( <hba no> | <hba wwpn> )
                        - Displays a specific HBA general information.


   -t [ <all> ]         - Displays the target information on all HBAs.
   -t ( <hba no> | <hba wwpn> )
                        - Displays the target information on a specific HBA
   -t ( <hba no> | <hba wwpn> ) ( <target wwpn> | <target portid> )
                        - Displays a specific target information on a
                          specific HBA.


   -l ( <hba no> | <hba wwpn> )
                        - Displays LUN information for all HBAs.
   -l ( <hba no> | <hba wwpn> ) ( <target wwpn> | <target portid> )
                        - Displays LUN information for a specific target
   -l ( <hba no> | <hba wwpn> ) ( <target wwpn> | <target portid> ) <lun id>
                        - Displays LUN information for a specific
                          LUN on a specific target.


   -n ( <hba no> | <hba wwpn> ) { ( <param name> | <param alias> )
        <param value> }

                        - HBA Port settings (NVRAM).
   -n ( <hba no> | <hba wwpn> ) default
                        - Restore default settings (4G HBAs only).



   -p ( <hba no> | <hba wwpn> | <all> ) ( view | ? )
                        - Display target persistent binding information of
                          a specific HBA or all HBAs.
   -p ( <hba no> | <hba wwpn> ) { <target wwnn> <target wwpn> <target portid> 
        <target id> }
                        - Bind selected target(s) on a specific HBA.
   -p ( <hba no> | <hba wwpn> | <all> ) bind all
                        - Bind all target(s) on a specific HBA or all HBAs.
   -p ( <hba no> | <hba wwpn> ) remove all | unbind all
                        - Unbind all target(s) on a specific HBA.
   -p ( <hba no> | <hba wwpn> ) remove <target wwnn> | unbind <target wwnn>
                        - Unbind selected target on a specific HBA.

   -m ( <hba no> | <hba wwpn> ) ( view | ? )
                        - View the HBA 's selective LUN list
   -m ( <hba no> | <hba wwpn> ) <target wwnn> <target wwpn> <lun id> ( view
        | ? )
                        - View a LUN's selective state of specific device.
   -m ( <hba no> | <hba wwpn>  ) { <target wwnn> <target wwpn> <lun id> ( 0
        | 1 | enable | disable  | select | unselect ) }
                        - Select/Unselect a LUN of a specific target on a
                          specific HBA.
   -m ( <hba no> | <hba wwpn> ) select | enable <target wwnn> <target wwpn>
                        - Enable all LUNs of a specific target on a specific 
                          HBA.
   -m ( <hba no> | <hba wwpn> ) unselect | disable <target wwnn> <target wwpn>
                        - Disable all LUNs of a specific target on a specific
                          HBA.
   -m ( <hba no> | <hba wwpn> ) select all
                        - Select or enable all LUNs of all targets on a
                          specific HBA.
   -m ( <hba no> | <hba wwpn> ) unselect all
                        - Unselect or disable all LUNs of all targets  on a
                          specific HBA.

   -e ( view | ? )      - View the boot device of all HBAs.
   -e ( view | ? )      - View the boot device of all HBAs.
   -e ( <hba no> | <hba wwpn> ) ( view | ? ) 
                        - View the boot device of a specific HBA.
   -e <hba no> | <hba wwpn> <target wwnn> <target wwpn> <target id> <lun id>
                        - Set a specific target as boot device on a specific HBA. 
   -e ( <hba no> | <hba wwpn> ) enable 
                        - Set a default BIOS boot device on a specific HBA
                            This option is only available on x86.
   -e ( <hba no> | <hba wwpn> ) disable 
                        - Clear the boot device on a specific HBA.


   -fg ( <hba no> | <hba wwpn> ) ( view | ? )
                        - View driver settings.

   -fs ( <hba no> | <hba wwpn> ) { ( <param name> | <param alias> )
         <param value> }
                        - Configure driver settings.


   -b ( <all> | <hba no> | <hba wwpn> ) [ ( <-rg> <fw | boot| all> ) ]
         <file name>
                        - Updates the HBA's Option ROM.
         <-rg>          - Specifies Option ROM region update mode.
         <fw>           - Firmware update only.
         <boot>         - Boot code update (BIOS/Fcode/EFI) only.
         <file name>    - Specifies the image file name.
                        - Region update is only supported on QLA/QLE/QMC246x.
   -b ( <hba no> | <hba wwpn> ) save <file name>

                        - Saves the HBA's Option ROM to a file.

   -r ( <hba no> | <hba wwpn> | <all> ) <file name>
                        - Update the HBA's NVRAM.
   -r ( <hba no> | <hba wwpn> ) save <file name>
                        - Saves the HBA's NVRAM to a file.

   -do ( <hba no> | <hba wwpn> | <all> ) <rescan | rs>
                        - Tell the driver to issue a rescan command for
                          discovering newly added targets/LUNs.

   -d <file name >
                        - Update driver to HBA(s) where <file name> is
                          the full path of the driver oemsetup.inf file.


   -a ( <hba no> | <hba wwpn> ) ( view | ?)
                        - View HBA's LED flashing status.
   -a ( <hba no> | <hba wwpn> )
                        - Toggle the HBA's LED flashing state.


   -tb ( <hba no> | <hba wwpn> ){ ( <target wwpn> ) } <beacon mode> 
                        - Target Beacon: Flash the disk drive's LED to
                          locate the drive in a JBOD.


   -kl ( <hba no> | <hba wwpn> ) [ { ( <param name> | <param alias> )
         <param value> } ]
                        - Run HBA diagnostics loopback test.

   -kr ( <hba no> | <hba wwpn> ) [ { ( -ex | -exclude ) <target wwpn> } ]
       [ ( <param name>| <param alias> ) <param value> ]
                        - Run HBA diagnostics read-write buffer test.


   -gs ( <hba no> | <hba wwpn> ) { ( <param name> | <param alias> )
              <param value> }
                        - View Statistics.
   -ls ( <hba no> | <hba wwpn> ) { ( <param name> | <param alias> )
              <param value> }
                        - View Link Status.

   -dm ( <hba no> | <hba wwpn> | <all> ) general | gen | details | det 
                        - Display Digital Diagnostics Monitoring Information
                          in general or details view.
         Note: This feature is supported only with 4Gb HBAs.


   -tp | -topology ]
                        - Display host topology.


   -ha ( <hba no> | <hba wwpn> ) <alias>
                        -  Set an alias to selected HBA
   -ha ( <hba no> | <hba wwpn> ) delete
                        -  Delete the current alias of selected HBA.
   -ha ( <hba no> | <hba wwpn> ) view | ?
                        -  View the current alias of selected HBA


   -pa ( <hba no> | <hba wwpn> ) <alias>
                        -  Set a port alias to selected HBA
   -pa ( <hba no> | <hba wwpn> ) delete
                        -  Delete current port alias of selected HBA.
   -pa ( <hba no> | <hba wwpn> ) view | ?
                        -  View the current port alias of the selected HBA


   -z ( <hba no> | <hba wwpn> | <all> )
                        - Display all information for a specific
                          HBA or all HBAs.


   -v                   - Display version.

   -h | -?              - Display usage help text.

   -o <file name>       - Specifies the output to a log file.

   -f <file name>       - Specifies command line input from file.

   -x                   - Specifies the output in XML format.

   -s                   - Silent mode.
   Notes:
         1. Options -x,-s,-o can be combined with other options. However,
            they must be at the beginning or at the end of the command line.
         2. Option -f cannot be combined with any other options.
            Only one command line option per input file is valid.
         3. The '-' character can be replaced as '/' character.
            i.e.  scli -g  and scli /g are both considered as valid commands.
         4. Option -h can be combined with a command line option to display
            the usage of that individual command.
 


Legends:

   <hba no>             - HBA number (Instance number).
   <hba wwpn>           - HBA World Wide Port Name in the following format:
                          xx-xx-xx-xx-xx-xx-xx-xx or xxxxxxxxxxxxxx.
   <target wwnn>        - Target World Wide Node Name in the following format:
                          xx-xx-xx-xx-xx-xx-xx-xx or xxxxxxxxxxxxxx.
   <target wwpn>        - Target World Wide Port Name in the following format:
                          xx-xx-xx-xx-xx-xx-xx-xx or xxxxxxxxxxxxxx.
   <target portid>      - Target Port ID in the following format:
                          xx-xx-xx or xxxxxx.
   <target id>          - Target ID.
   <lun id>             - Logical Unit Number (0-255).

   HBA Port Settings (NVRAM):
   <param name>         - See column 1 - Table 1.
   <param alias>        - See column 2 - Table 1.
   <param value>        - See column 3 - Table 1.

    =======================================================================
    Parameter Name          Alias   Value                   Description 
    =======================================================================
    ConnectionOption        CO      0-3                     See note 1 below
    DataRate                DR      0-3                     See note 2 below
    FrameSize               FR      512,1024,2048
    HardLoopID              HD      0-125
    ResetDelay              RD      0-255
    EnableBIOS              EB      0,1                     See note 3,6 below
    EnableHardLoopID        HL      0,1                     See note 3 below
    EnableFCPErrRecovery    EF      0,1                     See note 3 below
    ExecutionThrottle       ET      1-65535                 See note 5 below
    EnableExtendedLogging   EL      0,1                     See note 3,4 below
    LoginReTryCount         LR      0-255
    EnableLipReset          LP      0,1                     See note 5 below
    PortDownRetryCount      PD      0-255
    EnableLIPFullLogin      FL      0,1                     See note 3 below
    LinkDownTimeOut         LT      0-240
    EnableTargetReset       TR      0,1                     See note 3,5 below
    MaximumLUNsPerTarget    ML      0,8,16,32,64,128,256    See note 5 below
    LinkDownError           LD      0,1                     See note 3,5 below
    FastErrorReporting      FE      0,1                     See note 3,5 below
    ========================================================================
    Notes:
    1. Connection Options:  0 - Loop Only.
                            1 - Point-to-Point Only.
                            2 - Loop preferred, otherwise Point-to-Point.
                            3 - Point-to-Point, otherwise Loop (QLA22xx only).
    2. Data Rate:           0 - 1Gbs, 1 - 2Gbs, 2 - Auto, 3 - 4Gbs.
    3. Others               0 - Disable, 1 - Enable
    4. Option is not available on 4Gb HBA.
    5. Option is not available with QLC driver.
    6. Option is not available on PPC64/SPARC.
    7. Option is not available on Solaris.

   Diagnostics Settings:
   -exclude | -ex:      - Specifies device by its wwpn to be excluded from
                          read/write buffer test.
   <param name>         - See column 1 - Table 2.
   <param alias>        - See column 2 - Table 2.
   <param value>        - See column 3 - Table 2.

    =======================================================================
    Parameter Name    Alias   Value                   Description 
    =======================================================================
    DataPattern       DP      00-FF                   See note 1 below
                              CRPAT                   See note 1 below
                              CSPAT                   See note 1 below
                              CJTPAT                  See note 1 below
    DataSize          DS      8,16,32,64,128,256
                              512,1024,2048           See note 2 below
    TestCount         TC      0-65535 (Loopback)      See note 3 below
                              0-10000 (R/W Buffer)    See note 3 below
    TestIncrement     TI      1-65535 (Loopback)      See note 4 below
                      TI      1-10000 (R/W Buffer)    See note 4 below
    OnError           OE      0                       Ignore
                              1                       Stop
                              2                       Loop on error
                                                      See note 5 below
    =====================================================

                     Table 2: HBA Diagnostics Configuration Settings

   Notes:

   1. DataPattern:     Test pattern in hex format.
                       Hex    Binary
                       ---    --------
                       00     00000000
                       55     01010101
                       5A     01011010
                       A5     10100101
                       AA     10101010
                       FF     11111111
                       User Defined Pattern (Hex)).
                       Random Pattern.
                       CRPAT  - Loopback test only.
                       CSPAT  - Loopback test only.
                       CJTPAT - Loopback test only.
   2. DataSize:        Specifies the data (frame payload) size in bytes. Actual
                       data that is transferred during any given pass of the test.
                       For R/W buffer test, the max data size is 128 bytes
   3. TestCount:       0 - Test Continuously
                       1 to 65535 - Total number of tests that will be 
                       executed.
   4. TestIncrement:   Must be less than the number of test count specified.
   5. OnError:         Specifies the action if an error occurs during any
                       given pass.
   Test Result:
      A) Loopback test result:
         Test Status, CRC Error, Disparity Error, Frame Length Error.
      B) Read/Write buffer test result:
         Loop ID/Status, Data Miscompare, Link Failure, Loss of Sync,
         Loss of Signal, Invalid CRC.


   Driver Settings:
   <param name>         - See column 1 - Table 3.
   <param alias>        - See column 2 - Table 3.
   <param value>        - See column 3 - Table 3.

   =======================================================================
   Parameter Name     Alias  Value                 Description 
   =======================================================================
   PersistentOnly     PO     0,1                   Present targets that are
                                                   persistently bound only.
   PersistentPlusNew  PN     0,1                   Present targets that are
                                                   persistently bound plus new
                                                   new targets.
   BindWWPN           BW     0,1                   Bind devices by WWPNs.
   BindPortID         BP     0,1                   Bind devices by Port IDs.
   =======================================================================

                     Table 3: Driver Settings


   =======================================================================
   =======================================================================
   Parameter Name     Alias  Value           Description 
   =======================================================================
   AutoPoll           AP     0               Turn on automatically update the
                                             HBA port statistics.
                             1-256           Turn on automatically update the 
                                             HBA port statistics at a specified
                                             interval.
   PollRate           SR     5-30            Set the polling interval during
                                             automatically update (seconds).
   LogToFile          LF     Log File Name   Export the statistics to a file
                                             (CSV format).
   =======================================================================
                     Table 4: HBA Port Statistics Options

   =======================================================================
   Parameter Name     Alias  Value             Description 
   =======================================================================
   AutoPoll           AP     0                 Update the link statistics
                                               automatically.
                             1-256             Update the link statistics
                                               up to a specified interval.
   PollRate           SR     5-30              Set the Statistics Sampling
                                               Rate (seconds).
   LogToFile          LF     Log File Name     Export the link statistics to a
                                               file (CSV format).
   =======================================================================
                     Table 5: Link Status Options

Sample Outputs

sunsrv01# scli -i all

-------------------------------------------------------
Host Name                  : sunsrv01
HBA Model                  : QLA2342
HBA Alias                  : 
Port                       : 1
Port Alias                 : 
Node Name                  : 20-00-00-E0-8B-86-FE-C9
Port Name                  : 21-00-00-E0-8B-86-FE-C9
Port ID                    : 63-B3-13
Serial Number              : E44926
Driver Version             : qlc-20051013-2.08
FCode Version              : 1.14.09
Firmware Version           : 3.03.116 IP
HBA Instance               : 2
OS Instance                : 2
HBA ID                     : 2-QLA2342
Actual Connection Mode     : Point to Point
Actual Data Rate           : 2 Gbps
PortType (Topology)        : NPort
Total Number of Devices    : 1
HBA Status                 : Online
-------------------------------------------------------
Host Name                  : sunsrv01
HBA Model                  : QLA2342
HBA Alias                  : 
Port                       : 2
Port Alias                 : 
Node Name                  : 20-01-00-E0-8B-A6-FE-C9
Port Name                  : 21-01-00-E0-8B-A6-FE-C9
Port ID                    : 00-00-00
Serial Number              : E44926
Driver Version             : qlc-20051013-2.08
FCode Version              : 1.14.09
Firmware Version           : 3.03.116 IP
HBA Instance               : 3
OS Instance                : 3
HBA ID                     : 3-QLA2342
Actual Connection Mode     : Unknown
Actual Data Rate           : Unknown
PortType (Topology)        : Unidentified
Total Number of Devices    : 0
HBA Status                 : Loop down
-------------------------------------------------------

Host Bus Adapters

2008-01-08T13:15:00.001-08:00

Qlogic

How to Determine what HBA is installed in a Solaris server

Manuals

HBA Drivers

VxVM - Resizing a Filesystem/Volume

2008-01-07T09:05:00.000-08:00

vxresize command syntax :

/etc/vx/bin/vxresize [-bsx] [-F fstype] [-g  diskgroup] [-t tasktag] \
                     volume new_length [medianame...]

The vxresize command will either grow or shrink both the file system and its underlying volume to match the specified new volume length. The ability to grow or shrink is file system dependent. Some file system types may require that the file system be unmounted for the operation to succeed.

Note: vxresize works with VxFS and UFS file systems only.

In some situations, when resizing large volumes, vxresize may take a long time to complete.

The new_length operand can begin with a plus (+) or minus (-) to indicate that the new length is added to or subtracted from from the current volume length.

Resizing a Filesystem/Volume

Current capacity:

# df -k /dbfiles03
Filesystem                 kbytes    used   avail capacity  Mounted on
/dev/vx/dsk/dg20/dbvol03  3079710 2709166  308950    90%    /dbfiles03

Volume information:

# vxprint dbvol03
Disk group: dg20

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
v  dbvol03      fsgen        ENABLED  6291456  -        ACTIVE   -       -
pl dbvol03-01   dbvol03      ENABLED  6298619  -        ACTIVE   -       -
sd dg2007-03    dbvol03-01   ENABLED  3149307  0        -        -       -
sd dg2006-03    dbvol03-01   ENABLED  3149307  0        -        -       -

Plex information:

# vxprint -l dbvol03-01
Disk group: dg20

Plex:     dbvol03-01
info:     len=6298619 contiglen=6298491
type:     layout=STRIPE columns=2 width=128
state:    state=ACTIVE kernel=ENABLED io=read-write
assoc:    vol=dbvol03 sd=dg2007-03,dg2006-03
flags:    busy complete

Increasing the volume to 4GB using vxresize:

# /etc/vx/bin/vxresize dbvol03 4g
/dev/vx/rdsk/dg20/dbvol03:      
        8388608 sectors in 4096 cylinders of 32 tracks, 64 sectors
        4096.0MB in 88 cyl groups (47 c/g, 47.00MB/g, 7872 i/g)
super-block backups (for fsck -F ufs -o b=#) at:
 32, 96352, 192672, 288992, 385312, 481632, 577952, 674272, 770592, 866912,
 963232, 1059552, 1155872, 1252192, 1348512, 1444832, 1541152, 1637472,
 1733792, 1830112, 1926432, 2022752, 2119072, 2215392, 2311712, 2408032,
 2504352, 2600672, 2696992, 2793312, 2889632, 2985952, 3080224, 3176544,
 3272864, 3369184, 3465504, 3561824, 3658144, 3754464, 3850784, 3947104,
 4043424, 4139744, 4236064, 4332384, 4428704, 4525024, 4621344, 4717664,
 4813984, 4910304, 5006624, 5102944, 5199264, 5295584, 5391904, 5488224,
 5584544, 5680864, 5777184, 5873504, 5969824, 6066144, 6160416, 6256736,
 6353056, 6449376, 6545696, 6642016, 6738336, 6834656, 6930976, 7027296,
 7123616, 7219936, 7316256, 7412576, 7508896, 7605216, 7701536, 7797856,
 7894176, 7990496, 8086816, 8183136, 8279456, 8375776,

New capacity:

# df -k /dbfiles03
Filesystem                 kbytes    used   avail capacity  Mounted on
/dev/vx/dsk/dg20/dbvol03  4106286 2709166 1335526    67%    /dbfiles03

New volume information (two new subdisks):

# vxprint dbvol03
Disk group: dg20

TY NAME         ASSOC        KSTATE   LENGTH   PLOFFS   STATE    TUTIL0  PUTIL0
v  dbvol03      fsgen        ENABLED  8388608  -        ACTIVE   -       -
pl dbvol03-01   dbvol03      ENABLED  8395767  -        ACTIVE   -       -
sd dg2007-03    dbvol03-01   ENABLED  3149307  0        -        -       -
sd dg2007-05    dbvol03-01   ENABLED  1048572  3149307  -        -       -
sd dg2006-03    dbvol03-01   ENABLED  3149307  0        -        -       -
sd dg2006-05    dbvol03-01   ENABLED  1048572  3149307  -        -       -

New plex information:

# vxprint -l dbvol03-01
Disk group: dg20

Plex:     dbvol03-01
info:     len=8395767 contiglen=8395639
type:     layout=STRIPE columns=2 width=128
state:    state=ACTIVE kernel=ENABLED io=read-write
assoc:    vol=dbvol03 sd=dg2007-03,dg2007-05,dg2006-03,dg2006-05
flags:    busy complete

More options:

# /etc/vx/bin/vxresize dbvol03 +1g

# /etc/vx/bin/vxresize dbvol03 +1024m

# /etc/vx/bin/vxresize dbvol03 -2g

VxVM - Extending or Shrinking a Volume

2008-01-07T08:53:00.000-08:00

vxassist command syntax :

vxassist <option> <Keyword> volume_name [attributes]

Commonly used options are given below (See man vxassist for complete list of supported options)

vxassist growto volume_name length

To extend vol3 up to 8000 sectors, type:

# vxassist growto vol3 8000

Extending by a Given Length

Command Syntax :

vxassist growby volume_name length

To extend volapp by 1000 sectors, type:

# vxassist growby volapp 1000

Shrinking a Volume

Caution - Do not shrink a volume below the size of the file system. If you have a VxFS file system, you can shrink the file system and then shrink the volume. If you do not shrink the file system first, you risk unrecoverable data loss.

Always make sure you have a good backup of the data volume to be shirnked.

Shrinking to a Given Length

Shrink a volume to a specific length as follows:

vxassist shrinkto volume_name length

Make sure you do not shrink the volume below the current size of the file system or database using the volume. This command can be safely used on empty volumes.

To shrink volcat to 1300 sectors, type:

# vxassist shrinkto volcat 1300

Shrinking by a Given Length

Shrink a volume by a specific length as follows:

vxassist shrinkby volume_name length

To extend volapp by 1000 sectors, type:

# vxassist shrinkby volapp2 8000

VxVM - Creating a Volume

2008-01-07T08:43:00.000-08:00

vxassist command syntax :

vxassist <option> <Keyword> volume_name [attributes]

Commonly used options are given below (See man vxassist for complete list of supported options)

Keywords used are make , mirror , move , growto ,growby ,shrintto ,shirnkby ,snapstart , snapshot ,snapwait

Attributes specify volumes layout disks controllar to include exclude etc

Creating a Concatenated Volume

By default, vxassist creates a concatenated volume using the space available on a disk or on the number of disks in a diskgroup if the volume size specified is more then the one available on a single disk.

Disks can be specified from a diskgroup for a volume group but if not mentioned available disks are selected by the volume manager.

Command syntax :

vxassist make volume_name volume_length

To create a new volume appvol of 100 MB in the default disk group rootdg with available disks:

# vxassist make appvol 100m

To create the volume appvol of 100MB on disk03

# vxassist make appvol 100m disk03

Creating a Striped Volume

A striped volume contains at least one plex that consists of two or more subdisks located on two or more physical disks.

Command Syntax :

vxassist make volume_name length layout=stripe

To create a striped volume appvol2 with the default stripe unit size on the default number of disks

# xassist make appvol2 100m layout=stripe

To create a striped volume appvol2 100MB striped volume on three specific disks.

# vxassist make appvol2 100m layout=stripe disk04 disk05 disk06

Creating a RAID-5 Volume

A RAID-5 volume contains a RAID-5 plex that consists of two or more subdisks located on two or more physical disks. Only one RAID-5 plex can exist per volume. A RAID-5 volume may also contain one or more RAID-5 log plexes, which are used to log information about data and parity being written to the volume.

Command Syntax :

vxassist make volume_name length layout=raid5

To create the RAID-5 volume appvol4 with the default stripe unit size on the default number of disks with RAID-5 log,

# xassist make appvol4 100m layout=raid5.

Veritas Volume Replicator Index

2008-01-06T14:46:00.000-08:00

Here’s a collection of my VVR documents. Some were from my own work and some were taken from the web.

These documents were posted mainly for my personal references and for others that may find them useful.

Contents

VVR Manuals

VVR Administration

2008-01-05T15:34:00.000-08:00

Contents

Introduction
VVR Terminolgy
Installation
Configuring VVR
Checking the Configuration
Check Replication Status
Changing The Primary Node

Introduction

This document describes how to configure and administer VVR. VVR is Veritas Volume Replicator - the binaries are automatically installed as part of VxVM (Veritas Volume Manager). To activate VVR an appropriate Veritas License Key needs installing onto all the systems using VVR.

VVR allows the contents of a set of volumes within specified disk groups to be replicated across TCP/IP to one or more remote hosts. One Primary host must exist with at least one Secondary host although more remote hosts can be configured. The data on the secondary host cannot be unless it becomes the Primary host, either by taking over the role (if the Primary host is down) or by migrating the Primary host role to the Secondary host.

VVR can be integrated into VCS with a VVR agent and Global Cluster Manager (GCM - another Veritas product) can then be used to manage the failover of clusters from site to site.

VVR Terminology

Data Change Map (DCM) - An object containing a bitmap that can be optionally associated with a data volume on the Primary RVG. The bits represent regions of data that are different between the Primary and the Secondary. DCMs are used to mark sections of volumes on the primary that have changed during extended network outages in order to minimize the amount of data that must be synchronized to the secondary site during the outage.

Data Volume - Volumes that are associated with an RVG and contain application data.

Primary Node - The node on which the primary RVG resides.

Replication Link (RLINK) - RLINKs represent the communication link to the couterpart of the RVG on another node. At the Primary node a replicated volume object has one RLINK for each of its network mirrors. On the Secondary node a replicated volume has a single RLINK object that links it to its Primary. Each RLINK on a Primary RVG represents the communication link from the Primary RVG to a corresponding Secondary RVG, via an IP connection.

Replicated Data Set (RDS) - The group of the RVG on a Primary and its corresponding Secondary hosts.

Replicated Volume Group (RVG) - A component of VVR that is made up of a set of data volumes, one or more RLINKs and an SRL. An RVG is a subset of volumes within a given VxVM disk group configured for replication to one or more secondary systems. Data is replicated from a primary RVG to a secondary RVG. The primary RVG is in use by an application, while the secondary RVG receives replication and writes to local disk. The concept of primary and secondary is per RVG, not per system. A system can simultaneously be a primary RVG for some RVGs and secondary RVG for others. The RVG also contains the storage replicator log (SRL) and replication link (RLINK).

Storage Replication Log (SRL) - Writes to the Primary RVG are saved in the SRL on the Primary side. The SRL is used to aid in recovery, as well as to buffer writes when the system operates in asynchronous mode. Each write to a data volume in the RVG generates two write requests: one to the Secondary SRL, and another to the Primary SRL.

Secondary Node - The node too which the primary RVG replicates.

Installation

VVR is installed with VxVM, follow instructions for installing VxVM.

Install the VVR licence using vxlicinst and entering the licence key when prompted.

Configuring VVR

Create Disk Groups and Volumes

Primary Node

Create a disk group for each RVG.
Create all required volumes.
Create a volume for the SRL, sized according to the amount of data to be replicated.
Optionally create DCMs for each volume.
create the filesystems as appropriate.
Mount the filesystems - this is not a requirement to configure VVR but will not prevent configuration.

Secondary Node

Create a disk group for each RVG, using the same disk group names as the Primary Node.
Create all required volumes, using the save volume names as the Primary Node. The volumes must be the same size as on the Primary Node but do not require the same underlying disk structure.
Create a volume for the SRL, as per the Primary Node.
Optionally create DCMs for each volume.
DO NOT create the filesystems, these will be replicated from the Primary Node.

Create the RVG

Secondary Node

Firstly the Secondary Node must be given permission to manage the disk group created on the Primary Node. To do this add the diskgroup ID into /etc/vx/vras/.rdg. The diskgroup ID is the value of dgid in the output of vxprint -l on the Primary Node.

Primary Node

Create the Primary RVG with the vradmin command:

vradmin -g <diskgroup> createpri <rvgname> <vol1,vol2...> <srlname>

Where:

<diskgroup> is the VxVM diskgroup name
<rvgname> is the name for the RVG, usually diskgroupnamervg
<vol1,vol2...> is a comma separated list of all volumes in the diskgroup to be replicated.
<srlname> is the name of the SRL volume

If VVR is being configured in conjunction with VCS then the Service Group name should be used as the localhost name rather than the hostname. To set this enter the following:

vradmin -g <diskgroup> set local_host=<primaryhost>

Where:

<diskgroup> is the VxVM diskgroup name
<primaryhost> is the VCS Service Group name (IP address must be resolvable from this name)

Now the Secondary Nodes are configured from the Primary Node - perform this task for each Secondary Node:

vradmin -g <diskgroup> addsec <rvgname> <primaryhost> <secondaryhost>

Where:

<diskgroup> is the VxVM diskgroup name
<rvgname> is the name of the RVG created on the Primary Node
<primaryhost> is the hostname of the Primary Node, this could be a VCS ServiceGroup name
<secondaryhost> is the hostname of the Secondary Node, this could be a VCS ServiceGroup name

Configuring Replication Mode

VVR supports both asynchronous and synchronous replication. To configure the mode of replication set the attribute synchronous to one of the following:

off - sets replication mode to asynchronous
override - sets the replication mode to soft synchronous. During normal operation replication is synchronous but if the RLINK becomes inactive then replication temporarily becomes asynchronous, storing all transactions in the SRL. When the RLINK becomes available the SRL is drained and repliction switches back to synchronous.
fail - sets replication mode to hard synchronous. During normal operation replication is synchronous and if the RLINK becomes inactive VVR failes new updates to the Primary Node.

To set the appropriate mode of replication enter the following:

vradmin -g <diskgroup> set <rvgname> <secondaryhost> synchronous=<value>

Where:

<diskgroup> is the VxVM diskgroup name
<rvgname> is the name of the RVG
<secondaryhost> is the hostname of the Secondary Node
<value> is the synchronous attribute value as listed above

Synchronising & Starting Replication

When the RVG has been configured replication can be started. When replication is started for the first time a full synchonisation takes place. The time taken to synchronise will depend on the volume of data. On the Primary Node enter the command:

vradmin -g <diskgroup> -a startrep <rvgname> <secondaryhost>

Where:

<diskgroup> is the VxVM diskgroup name
<rvgname> is the name of the RVG
<secondaryhost> is the hostname of the Secondary Node

Checking the Configuration

There are many ways of looking at how VVR has been configured. The following sections highlight the main commands and options but is not extensive. Refer to the man pages and Veritas documentation for further informtion.

VxVM Records using vxprint

The output of vxprint -Aht will show rv and rl records associated with more traditional v, pl and sd records:

# vxprint -Aht
rv intrarvg     1            ENABLED  ACTIVE   primary  2         intranet-srl
rl rlk_sv1202_intrarvg intrarvg CONNECT ACTIVE sv1202   intra1dg  rlk_sv1201_intrarvg
v  unix-intranet intrarvg    ENABLED  ACTIVE   25165824 SELECT    -        fsgen
pl unix-intranet-01 unix-intranet ENABLED ACTIVE 25176456 STRIPE  3/128    RW
sd loa0101-01   unix-intranet-01 loa0101 0     8392072  0/0       c2t0d0   ENA
sd loa0102-01   unix-intranet-01 loa0102 0     8392072  1/0       c2t1d0   ENA
sd loa0103-01   unix-intranet-01 loa0103 0     8392072  2/0       c2t2d0   ENA
pl unix-intranet-02 unix-intranet ENABLED ACTIVE 25176456 STRIPE  3/128    RW
sd loa0105-01   unix-intranet-02 loa0105 0     8392072  0/0       c3t0d0   ENA
sd loa0106-01   unix-intranet-02 loa0106 0     8392072  1/0       c3t1d0   ENA
sd loa0107-01   unix-intranet-02 loa0107 0     8392072  2/0       c3t2d0   ENA
pl unix-intranet-03 unix-intranet ENABLED ACTIVE LOGONLY CONCAT   -        RW
sd loa0102-03   unix-intranet-03 loa0102 11789424 64    LOG       c2t1d0   ENA
pl unix-intranet-04 unix-intranet ENABLED ACTIVE LOGONLY CONCAT   -        RW
sd loa0106-03   unix-intranet-04 loa0106 11789424 64    LOG       c3t1d0   ENA
v  unix-netlink intrarvg     ENABLED  ACTIVE   10187667 SELECT    -        fsgen
pl unix-netlink-01 unix-netlink ENABLED ACTIVE 10192104 STRIPE    3/128    RW
sd loa0101-02   unix-netlink-01 loa0101 8392072 3397352 0/0       c2t0d0   ENA
sd loa0102-02   unix-netlink-01 loa0102 8392072 3397352 1/0       c2t1d0   ENA
sd loa0103-02   unix-netlink-01 loa0103 8392072 3397352 2/0       c2t2d0   ENA
pl unix-netlink-02 unix-netlink ENABLED ACTIVE 10192104 STRIPE    3/128    RW
sd loa0105-02   unix-netlink-02 loa0105 8392072 3397352 0/0       c3t0d0   ENA
sd loa0106-02   unix-netlink-02 loa0106 8392072 3397352 1/0       c3t1d0   ENA
sd loa0107-02   unix-netlink-02 loa0107 8392072 3397352 2/0       c3t2d0   ENA
pl unix-netlink-03 unix-netlink ENABLED ACTIVE LOGONLY  CONCAT    -        RW
sd loa0101-03   unix-netlink-03 loa0101 11789424 64     LOG       c2t0d0   ENA
pl unix-netlink-04 unix-netlink ENABLED ACTIVE LOGONLY  CONCAT    -        RW
sd loa0105-03   unix-netlink-04 loa0105 11789424 64     LOG       c3t0d0   ENA
v  intranet-srl intrarvg     ENABLED  ACTIVE   2097152  SELECT    -        SRL
pl intranet-srl-01 intranet-srl ENABLED ACTIVE 2101552  CONCAT    -        RW
sd loa0104-01   intranet-srl-01 loa0104 0      2101552  0         c2t3d0   ENA
pl intranet-srl-02 intranet-srl ENABLED ACTIVE 2101552  CONCAT    -        RW
sd loa0108-01   intranet-srl-02 loa0108 0      2101552  0         c3t3d0   ENA

This example shows three striped and mirrored volumes, called unix-intranet, unix-netlink and intranet-srl contained within the RVG called intrarvg with an RLINK called rlk_sv1202_intrarvg

RVG Configuration

The following command will display the configuration for every RVG, or the one specified:

# vxprint -Pl [ <rvgname> ]

Disk group: intra1dg

Rlink:    rlk_sv1202_intrarvg
info:     timeout=500 packet_size=8400 rid=0.1367
          latency_high_mark=10000 latency_low_mark=9950
state:    state=ACTIVE
          synchronous=off latencyprot=off srlprot=autodcm
assoc:    rvg=intrarvg
          remote_host=sv1202 IP_addr=10.38.3.116 port=4145
          remote_dg=intra1dg
          remote_dg_dgid=1065010978.1277.sv1202
          remote_rvg_version=10
          remote_rlink=rlk_sv1201_intrarvg
          remote_rlink_rid=0.1254
          local_host=sv1201 IP_addr=10.96.103.19 port=4145
protocol: UDP/IP
flags:    write enabled attached consistent connected asynchronous

Node Configuration

To display information about the Primary and Secondary Nodes enter the command:

vradmin -g <diskgroup> printrvg <rvgname>

Where:

* <diskgroup> is the VxVM diskgroup name
* <rvgname> is the name of the RVG

# vradmin -g intra1dg printrvg intrarvg

Primary:
        HostName: sv1201       
        RvgName: intrarvg
        DgName: intra1dg
        datavol_cnt: 2
        srl: intranet-srl
        RLinks:
            name=rlk_sv1202_intrarvg, detached=off, synchronous=off
Secondary:
        HostName: sv1202
        RvgName: intrarvg
        DgName: intra1dg
        datavol_cnt: 2
        srl: intranet-srl
        RLinks:
            name=rlk_sv1201_intrarvg, detached=off, synchronous=off

Check Replication Status

Check RVG Replication Status

To check the replication status of an RVG enter the following command:

vradmin -g <diskgroup> repstatus <rvgname>

Where:

* <diskgroup> is the VxVM diskgroup name
* <rvgname> is the name of the RVG

# vradmin -g intra2dg repstatus intrarvg

Replicated Data Set: intrarvg
Primary:
  Host name:          sv1201
  RVG name:           intrarvg
  DG name:            intra1dg
  RVG state:          enabled for I/O
  Data volumes:       2
  SRL name:           intranet-srl
  SRL size:           1.00 G
  Total secondaries:  1

Secondary:
  Host name:          sv1202
  RVG name:           intrarvg
  DG name:            intra1dg
  Data status:        consistent, up-to-date
  Replication status: replicating (connected)
  Current mode:       asynchronous
  Logging to:         SRL

The key bits of information here are Data status and Replication status.

Check RLINK Replication Status

To check the replication status of an RLINK enter the command:

vxrlink -g <diskgroup> status <rlinkname>

Where:

<diskgroup> is the VxVM diskgroup name
<rlinkname> is the RLINK name

# vxrlink -g intra1dg status rlk_sv1202_intrarvg

Thu Oct 16 12:13:43 BST 2003
vxvm:vxrlink: INFO: Rlink rlk_sv1202_intrarvg has 37197 outstanding writes, occupying 537631 Kbytes (51%) on the SRL

...

Thu Oct 16 14:23:01 BST 2003
vxvm:vxrlink: INFO: Rlink rlk_sv1202_intrarvg is up to date

Changing The Primary Node

Migrate the Primary Node to a Secondary Node

If the Primary Nodes needs to be swapped to one of the Secondary Nodes and both Nodes are operational then the replication must be migrated. This performs a tidy closedown of replication and starts it up at the other end. The application must be stopped before migration can be performed, this includes unmounting the filesystems.

Stop the application.
Unmount the filesystems.
Check replication is complete, wait for it to complete if it is not.
Migrate

This process can be used to test that the Secondary Node is configured correctly to run the application.

To migrate the Primary Node enter the command:

vradmin -g <diskgroup> migrate <rvgname>

Where:

<diskgroup>: is the VxVM diskgroup name
<rvgname> is the name of the RVG to migrate
<destinationnode> is the Node to which the RVG is being migrated

Takeover the Primary Node from a Secondary Node

A takeover of the Primary Node would be used if the Primary Node is not available for any reason, including a disaster.

Firstly replication must be upto date on the Secondary Node, that is the SRL must be empty. On the Secondary Node that is to become the Primary Node enter the command:

vradmin -g <diskgroup>: takeover <rvgname>

Where:

<diskgroup>: is the VxVM diskgroup name
<rvgname> is the name of the RVG to takeover

Revert to the Original Primary Node

To revert to the Primary Node after a migration, simply migrate back.

To revert to the Primary Node after a takeover refer to the VVR Administration Guide. There are two methods to revert depending on the state of the data replication:

Fast Failback - The DCMs are used to keep note of all changes to the data volumes, these are copied to the Primary Node and replayed to apply all the changes to the original data volumes.
Difference Based Failback - This method uses checkpointing to apply the changes to the original data volumes based on the differences with the Secondary data volumes.

VxVM - DMP Load Balancing

2008-01-04T14:54:00.000-08:00

How to configure load balancing across multiple primary paths in an Active/Passive DMP environment using VERITAS Volume Manager (tm) 4.x

Details:

In an Active/Passive Dynamic Multipathing (DMP) environment, it is possible that there may be multiple primary paths to a device due to a mesh of SAN switches. Volume Manager 4.0 can be configured to allow load balancing across multiple primary paths. This TechNote explains how to do this.

Note: This feature was not available in releases prior to 4.0

In the following example, device c3t2d15 belongs to a disk group named tonydg and contains a simple concatenated volume named tony1. The device has eight primary paths:

# vxdisk list c3t2d15s2
Device:    c3t2d15s2
<..snip..>
numpaths:   8
c2t0d15s2       state=enabled   type=primary
c2t1d15s2       state=enabled   type=primary
c3t1d15s2       state=enabled   type=primary
c3t2d15s2       state=enabled   type=primary
c4t3d15s2       state=enabled   type=primary
c4t4d15s2       state=enabled   type=primary
c5t3d15s2       state=enabled   type=primary
c5t5d15s2       state=enabled   type=primary

DMP statistics are enabled and a read based workload is applied to the volume:

# vxdmpadm iostat start

# dd if=/dev/vx/rdsk/tonydg/tony1 of=/dev/null bs=512k count=6144

By displaying the DMP statistics for device c3t2d15, it can be seen that all I/Os are being directed to one subpath, c5t5d15:

# vxdmpadm iostat show dmpnodename=c3t2d15s2                         
                      cpu usage = 8023us    per cpu memory = 32768b
                        OPERATIONS             MBYTES            AVG TIME(ms)  
PATHNAME              READS     WRITES      READS     WRITES     READS   WRITES
c2t0d15s2                 0          0          0          0  0.000000 0.000000
c2t1d15s2                 0          0          0          0  0.000000 0.000000
c3t1d15s2                 0          0          0          0  0.000000 0.000000
c3t2d15s2                 0          0          0          0  0.000000 0.000000
c4t3d15s2                 0          0          0          0  0.000000 0.000000
c4t4d15s2                 0          0          0          0  0.000000 0.000000
c5t3d15s2                 0          0          0          0  0.000000 0.000000
c5t5d15s2              6144          0    3145728          0  0.011081 0.000000

Using vxdmpadm, it is clear that the default iopolicy for this enclosure is Single Active (which explains the above behavior):

# vxdmpadm getattr enclosure HDS92000 iopolicy
ENCLR_NAME     DEFAULT        CURRENT
============================================
HDS92000       Single-Active  Single-Active

To load balance across multiple primaries, the iopolicy should be set to round-robin:

# vxdmpadm setattr enclosure HDS92000 iopolicy=round-robin

# vxdmpadm getattr enclosure HDS92000 iopolicy
ENCLR_NAME     DEFAULT        CURRENT
============================================
HDS92000       Single-Active  Round-Robin

The DMP statistics are reset and the same workload is applied to the volume, load balancing across the primary paths can be seen.

# vxdmpadm iostat reset                                        

# dd if=/dev/vx/rdsk/tonydg/tony1 of=/dev/null bs=512k count=6144

# vxdmpadm iostat show dmpnodename=c3t2d15s2
                      cpu usage = 7223us    per cpu memory = 32768b
                        OPERATIONS             MBYTES            AVG TIME(ms)  
PATHNAME              READS     WRITES      READS     WRITES     READS   WRITES
c2t0d15s2               786          0     402432          0  0.011013 0.000000
c2t1d15s2               778          0     398336          0  0.011267 0.000000
c3t1d15s2               772          0     395264          0  0.011205 0.000000
c3t2d15s2               744          0     380928          0  0.011094 0.000000
c4t3d15s2               750          0     384000          0  0.011029 0.000000
c4t4d15s2               768          0     393216          0  0.011169 0.000000
c5t3d15s2               733          0     375296          0  0.010943 0.000000
c5t5d15s2               813          0     416256          0  0.011068 0.000000

The enclosure can be returned to its default policy of single active with the following:

# vxdmpadm setattr enclosure HDS92000 iopolicy=singleactive

Note: This setting is not persistent across reboots, I/O policies will revert to their default. Persistence can be configured by adding the relevant vxdmpadm command to a boot script.

VxVM - Fixing DISABLED ACTIVE Vols with DISABLED RECOVER Plexes

2008-01-03T21:38:00.000-08:00

How to recover and start a Veritas Volume Manager logical volume where the volume is DISABLED ACTIVE and has a plex that is DISABLED RECOVER

Details:

When a system encounters a problem with a volume or a plex, or if Veritas Volume Manager (VxVM) has any reason to believe that the data is not synchronized, VxVM changes the kernel state, KSTATE and state, STATE, of the volume and its plexes accordingly. The plex state can be stale, empty, nodevice, etc. A particular plex state does not necessarily mean that the data is good or bad. The plex state is representative of VxVM's perception of the data in a plex.

The output from the vxprint utility using the switches "-h" and "-t" (for more information about these switches and all applicable switches, see the man page for vxprint) displays information from records in VxVM disk group configurations, including the KSTATE and STATE of a volume and plex as indicated in columns 4 and 5 respectively in the table below. When viewing the configuration records of a VxVM disk group using the vxprint utility and the KSTATE and STATE fields display DISABLED ACTIVE for the volume and DISABLED RECOVER for the plex, recovery steps need to be followed to bring the volume back to an ENABLED ACTIVE state so it can be mounted and make the file system accessible again.

From the below output, it can be seen that the KSTATE and STATE for the volume test is DISABLED ACTIVE and its plex test-01 is DISABLED RECOVER.

# vxprint -ht -g testdg

       
DG NAME NCONFIG   NLOG    MINORS   GROUP-ID   
DM NAME DEVICE    TYPE    PRIVLEN  PUBLEN   STATE 
RV NAME RLINK_CNT KSTATE  STATE    PRIMARY  DATAVOLS  SRL
RL NAME RVG       KSTATE  STATE    REM_HOST REM_DG    REM_RLNK
V  NAME RVG       KSTATE  STATE    LENGTH   USETYPE   PREFPLEX RDPOL
PL NAME VOLUME    KSTATE  STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME PLEX      DISK    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME PLEX      VOLNAME NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
             

               
dg testdg default default 84000 970356463.1203.alu     
               
dm testdg01 c1t4d0s2 sliced 2179 8920560 -   
dm testdg02 c1t6d0s2 sliced 2179 8920560 -   
               
v test - DISABLED ACTIVE 17840128 fsgen - SELECT
pl test-01 test DISABLED RECOVER 17841120 CONCAT - RW
sd testdg01-01 test-01 testdg01 0 8920560 0 c1t4d0 ENA
sd testdg02-01 test-01 testdg02 0 8920560 8920560 c1t6d0 ENA

Follow these steps to change KSTATE and STATE of a plex that is DISABLED RECOVER to ENABLED ACTIVE so the volume can be recovered / started and the file system mounted:

1. Change the plex test-01 to the DISABLED STALE state:

vxmend -g  diskgroup fix stale <plex_name>

For example:

# vxmend -g testdg fix stale test-01

This output shows the plex test-01 as DISABLED STALE:

# vxprint -ht -g testdg

        
DG NAME NCONFIG   NLOG    MINORS   GROUP-ID   
DM NAME DEVICE    TYPE    PRIVLEN  PUBLEN   STATE 
RV NAME RLINK_CNT KSTATE  STATE    PRIMARY  DATAVOLS  SRL
RL NAME RVG       KSTATE  STATE    REM_HOST REM_DG    REM_RLNK
V  NAME RVG       KSTATE  STATE    LENGTH   USETYPE   PREFPLEX RDPOL
PL NAME VOLUME    KSTATE  STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME PLEX      DISK    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME PLEX      VOLNAME NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
               
dg testdg default default 84000 970356463.1203.alu     
               
dm testdg01 c1t4d0s2 sliced 2179 8920560 -   
dm testdg02 c1t6d0s2 sliced 2179 8920560 -   
               
v test - DISABLED ACTIVE 17840128 fsgen - SELECT
pl test-01 test DISABLED STALE 17841120 CONCAT - RW
sd testdg01-01  test-01 testdg01 0 8920560 0 c1t4d0 ENA
sd testdg02-01  test-01 testdg02 0 8920560 8920560 c1t6d0 ENA

2. Change the plex test-01 to the DISABLED CLEAN state:

vxmend -g diskgroup fix clean <plex_name>

For example:

# vxmend -g testdg fix clean test-01

This output shows the plex test-01 as DISABLED CLEAN:

# vxprint -ht -g testdg

        
DG NAME NCONFIG   NLOG    MINORS   GROUP-ID   
DM NAME DEVICE    TYPE    PRIVLEN  PUBLEN   STATE 
RV NAME RLINK_CNT KSTATE  STATE    PRIMARY  DATAVOLS  SRL
RL NAME RVG       KSTATE  STATE    REM_HOST REM_DG    REM_RLNK
V  NAME RVG       KSTATE  STATE    LENGTH   USETYPE   PREFPLEX RDPOL
PL NAME VOLUME    KSTATE  STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME PLEX      DISK    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME PLEX      VOLNAME NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
               
dg testdg default default 84000 970356463.1203.alu     
               
dm testdg01 c1t4d0s2 sliced 2179 8920560 -   
dm testdg02 c1t6d0s2 sliced 2179 8920560 -   
               
v test - DISABLED ACTIVE 17840128 fsgen - SELECT
pl test-01 test DISABLED CLEAN 17841120 CONCAT - RW
sd testdg01-01  test-01 testdg01 0 8920560 0 c1t4d0 ENA
sd testdg02-01  test-01 testdg02 0 8920560 8920560 c1t6d0 ENA

3. Start the volume test:

vxvol -g diskgroup start <volume>

For example:

# vxvol -g diskgroup start test

This output shows that the volume test and its plex test-01 are both ENABLED ACTIVE:

# vxprint -ht -g testdg
        
DG NAME NCONFIG   NLOG    MINORS   GROUP-ID   
DM NAME DEVICE    TYPE    PRIVLEN  PUBLEN   STATE 
RV NAME RLINK_CNT KSTATE  STATE    PRIMARY  DATAVOLS  SRL
RL NAME RVG       KSTATE  STATE    REM_HOST REM_DG    REM_RLNK
V  NAME RVG       KSTATE  STATE    LENGTH   USETYPE   PREFPLEX RDPOL
PL NAME VOLUME    KSTATE  STATE    LENGTH   LAYOUT    NCOL/WID MODE
SD NAME PLEX      DISK    DISKOFFS LENGTH   [COL/]OFF DEVICE   MODE
SV NAME PLEX      VOLNAME NVOLLAYR LENGTH   [COL/]OFF AM/NM    MODE
               
dg testdg default default 84000 970356463.1203.alu     
               
dm testdg01 c1t4d0s2 sliced 2179 8920560 -   
dm testdg02 c1t6d0s2 sliced 2179 8920560 -   
               
v test - ENABLED ACTIVE 17840128 fsgen - SELECT
pl test-01 test ENABLED ACTIVE 17841120 CONCAT - RW
sd testdg01-01  test-01 testdg01 0 8920560 0 c1t4d0 ENA
sd testdg02-01  test-01 testdg02 0 8920560 8920560 c1t6d0 ENA

4. Mount the volume to its associated mount point (refer to the /etc/vfstab file if the mount point location is not known) if the file system is a Veritas File System (VxFS) file system:

mount -F vxfs /dev/vx/dsk/diskgroup/volume /mount-point

For example:

# mount -F vxfs /dev/vx/dsk/testdg/test /testvol

Note: An error may be generated stating that the file system needs to be checked for consistency. If this occurs, run the VxFS specific fsck utility (/usr/lib/fs/vxfs/fsck) where the default is to replay the intent log, instead of performing a full structural file system check which is usually sufficient to set the file system to CLEAN and allow the volume to be mounted.

VxVM - Removing a Volume

2008-01-03T13:24:00.000-08:00

Removing a volume requires removing all references to the volumes to be removed like unmounting the volume if mounted and removing its reference from filesystem table.

An active volume has to be stopped first to stop all the activities to the volume only then it can be removed.

Stopping a Volume:

vxvol stop volume_name

Removing a Volume

vxedit -rf rm volume_name

How to Determine what HBA is installed in a Solaris server?

2007-12-18T15:20:00.000-08:00

The prtpicl command outputs information to accurately determine the make and model of an HBA.

The QLogic HBAs have a PCI identifier of 1077. Search through the output from the command prtpicl -v for the number 1077. A section displays similar to the following:

QLGC,qla (scsi, 44000003ac)
:DeviceID 0x4
:UnitAddress 4
:vendor-id 0x1077
:device-id 0x2300
:revision-id 0x1
:subsystem-vendor-id 0x1077
:subsystem-id 0x9
:min-grant 0x40
:max-latency 0
:cache-line-size 0x10
:latency-timer 0x40

The subsystem-ID value determines the model of HBA. Reference this chart to determine the model of HBA:

Vendor	HBA model	Vendor ID	Device ID	Subsys Vendor ID	Subsys Device ID
QLogic	QCP2340	1077	2312	1077	109
QLogic	QLA200	1077	6312	1077	119
QLogic	QLA210	1077	6322	1077	12F
QLogic	QLA2300/QLA2310	1077	2310	1077	9
QLogic	QLA2340	1077	2312	1077	100
QLogic	QLA2342	1077	2312	1077	101
QLogic	QLA2344	1077	2312	1077	102
QLogic	QLE2440	1077	2422	1077	145
QLogic	QLA2460	1077	2422	1077	133
QLogic	QLA2462	1077	2422	1077	134
QLogic	QLE2360	1077	2432	1077	117
QLogic	QLE2362	1077	2432	1077	118
QLogic	QLE2440	1077	2432	1077	147
QLogic	QLE2460	1077	2432	1077	137
QLogic	QLE2462	1077	2432	1077	138
QLogic	QSB2340	1077	2312	1077	104
QLogic	QSB2342	1077	2312	1077	105
Sun	SG-XPCI1FC-QLC	1077	6322	1077	132
Sun	6799A	1077	2200A	1077	4082
Sun	SG-XPCI1FC-QF2/x6767A	1077	2310	1077	106
Sun	SG-XPCI2FC-QF2/x6768A	1077	2312	1077	10A
Sun	X6727A	1077	2200A	1077	4083
Sun	SG-XPCI1FC-QF4	1077	2422	1077	140
Sun	SG-XPCI2FC-QF4	1077	2422	1077	141
Sun	SG-XPCIE1FC-QF4	1077	2432	1077	142
Sun	SG-XPCIE2FC-QF4	1077	2432	1077	143

VCS - Adding NIC/IP Resource

2007-12-04T13:49:00.000-08:00

Details:

Here's an actual server I worked on for adding a NIC resource.

# haconf -makerw

# hares -add vvrnic NIC db2inst_grp
VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be set before agent monitors

# hares -modify vvrnic Device ce3

# hares -modify vvrnic NetworkType ether

# hares -add vvrip IP db2inst_grp
VCS NOTICE V-16-1-10242 Resource added. Enabled attribute must be set before agent monitors

# hares -modify vvrip Device ce3
# hares -modify vvrip Address "10.67.196.191"
# hares -modify vvrip NetMask "255.255.254.0"

# hares -link vvrip vvrnic

# hagrp -enableresources db2inst_grp

# hares -online vvrip -sys server620

# haconf -dump -makero

VCS - Adding Filesystem Resource

2007-12-04T13:38:00.000-08:00

How to create a file system using VERITAS Volume Manager, controlled under VERITAS Cluster Server

Details:

Following is the algorithm to create a volume, file system and put them under VERITAS Cluster Server (VCS).

Add following resources and modify attributes:

Resources Name Attributes
1. Disk group, disk group name
2. Mount block device, FSType, MountPoint

Create dependency between following resources:

1. Mount and disk group

Enable all resources in this service group.

The following example shows how to create a raid-5 volume with a VxFS file system and put it under VCS control.

Method 1 - Using the command line

1. Create a disk group using Volume Manager with a minimum of 4 disks:

# vxdg init datadg disk01=c1t1d0s2 disk02=c1t2d0s2 disk03=c1t3d0s2 disk04=c1t4d0s2
# vxassist -g datadg make vol01 2g layout=raid5

2. Create a mount point for this volume:

# mkdir /vol01

3. Create a file system on this volume:

# mkfs -F vxfs /dev/vx/rdsk/datadg/vol01

4. Deport this disk group:

# vxdg deport datadg

5. Create a service group:

# haconf -makerw
# hagrp -add newgroup
# hagrp -modify newgroup SystemList <sysa> 0 <sysb> 1
# hagrp -modify newgroup AutoStartList <sysa>

6. Create a disk group resource and modify its attributes:

# hares -add data_dg DiskGroup newgroup
# hares -modify data_dg DiskGroup datadg

7. Create a mount resource and modify its attributes:

# hares -add vol01_mnt Mount newgroup
# hares -modify vol01_mnt BlockDevice /dev/vx/dsk/datadg/vol01
# hares -modify vol01_mnt FSType vxfs
# hares -modify vol01_mnt MountPoint /vol01
# hares -modify vol01_mnt FsckOpt %-y

8. Link the mount resource to the disk group resource:

# hares -link vol01_mnt data_dg

9. Enable the resources and close the configuration:

# hagrp -enableresources newgroup
# haconf -dump -makero

Method 2 - Editing /etc/VRTSvcs/conf/config/main.cf

# hastop -all
# cd /etc/VRTSvcs/conf/config
# haconf -makerw
# vi main.cf

Add the following line to end of this file:

group newgroup (
SystemList = { sysA =0, sysB=1}
AutoStartList = { sysA }
)

DiskGroup data_dg (
DiskGroup = datadg
)

Mount vol01_mnt (
MountPoint = "/vol01"
BlockDevice = " /dev/vx/dsk/datadg/vol01"
FSType = vxfs
)

vol01_mnt requires data_dg

# haconf -dump -makero
# hastart -local

Check status of the new service group.

------------------------------------------------------------------------------------

Here's an actual example.

# umount /backup/pdpd415

# vxdg deport bkupdg

# haconf -makerw

# hares -add bkup_dg DiskGroup pdpd415_grp
# hares -modify bkup_dg DiskGroup bkupdg

# hares -add bkupdg_bkup_mnt Mount pdpd415_grp
# hares -modify bkupdg_bkup_mnt BlockDevice /dev/vx/dsk/bkupdg/bkupvol
# hares -modify bkupdg_bkup_mnt FSType vxfs
# hares -modify bkupdg_bkup_mnt MountPoint /backup/pdpd415
# hares -modify bkupdg_bkup_mnt FsckOpt %-y

# hares -link bkupdg_bkup_mnt bkup_dg

# hagrp -enableresources pdpd415_grp

# hares -online bkup_dg -sys sppwd620
# hares -online bkupdg_bkup_mnt -sys sppwd620

# haconf -dump -makero

SAN Stuff for Solaris

2007-12-04T09:20:00.000-08:00

To verify whether an HBA is connected to a fabric or not:

# /usr/sbin/luxadm -e port

Found path to 4 HBA ports

/devices/pci@1e,600000/SUNW,qlc@3/fp@0,0:devctl                    CONNECTED
/devices/pci@1e,600000/SUNW,qlc@3,1/fp@0,0:devctl                  NOT CONNECTED
/devices/pci@1e,600000/SUNW,qlc@4/fp@0,0:devctl                    CONNECTED
/devices/pci@1e,600000/SUNW,qlc@4,1/fp@0,0:devctl                  NOT CONNECTED

Your SAN administrator will ask for the WWNs for Zoning. Here are some steps I use to get that information:

# prtconf -vp | grep wwn
            port-wwn:  210000e0.8b1d8d7d
            node-wwn:  200000e0.8b1d8d7d
            port-wwn:  210100e0.8b3d8d7d
            node-wwn:  200000e0.8b3d8d7d
            port-wwn:  210000e0.8b1eaeb0
            node-wwn:  200000e0.8b1eaeb0
            port-wwn:  210100e0.8b3eaeb0
            node-wwn:  200000e0.8b3eaeb0

Or you may use fcinfo, if installed.

# fcinfo hba-port
HBA Port WWN: 210000e08b8600c8
        OS Device Name: /dev/cfg/c11
        Manufacturer: QLogic Corp.
        Model: 375-3108-xx
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb
        Current Speed: 2Gb
        Node WWN: 200000e08b8600c8
HBA Port WWN: 210100e08ba600c8
        OS Device Name: /dev/cfg/c12
        Manufacturer: QLogic Corp.
        Model: 375-3108-xx
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb
        Current Speed: 2Gb
        Node WWN: 200100e08ba600c8
HBA Port WWN: 210000e08b86a1cc
        OS Device Name: /dev/cfg/c5
        Manufacturer: QLogic Corp.
        Model: 375-3108-xx
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb
        Current Speed: 2Gb
        Node WWN: 200000e08b86a1cc
HBA Port WWN: 210100e08ba6a1cc
        OS Device Name: /dev/cfg/c6
        Manufacturer: QLogic Corp.
        Model: 375-3108-xx
        Type: N-port
        State: online
        Supported Speeds: 1Gb 2Gb
        Current Speed: 2Gb
        Node WWN: 200100e08ba6a1cc

Here are some commands you can use for QLogic Adapters:

# modinfo | grep qlc
 76 7ba9e000  cdff8 282   1  qlc (SunFC Qlogic FCA v20060630-2.16)

# prtdiag | grep qlc
pci    66         PCI5  SUNW,qlc-pci1077,2312 (scsi-+
                  okay  /ssm@0,0/pci@18,600000/SUNW,qlc@1
pci    66         PCI5  SUNW,qlc-pci1077,2312 (scsi-+
                  okay  /ssm@0,0/pci@18,600000/SUNW,qlc@1,1
pci    33         PCI2  SUNW,qlc-pci1077,2312 (scsi-+
                  okay  /ssm@0,0/pci@19,700000/SUNW,qlc@1
pci    33         PCI2  SUNW,qlc-pci1077,2312 (scsi-+
                  okay  /ssm@0,0/pci@19,700000/SUNW,qlc@1,1

# luxadm qlgc

  Found Path to 4 FC100/P, ISP2200, ISP23xx Devices

  Opening Device: /devices/ssm@0,0/pci@19,700000/SUNW,qlc@1,1/fp@0,0:devctl
  Detected FCode Version:       ISP2312 Host Adapter Driver: 1.14.09 03/08/04

  Opening Device: /devices/ssm@0,0/pci@19,700000/SUNW,qlc@1/fp@0,0:devctl
  Detected FCode Version:       ISP2312 Host Adapter Driver: 1.14.09 03/08/04

  Opening Device: /devices/ssm@0,0/pci@18,600000/SUNW,qlc@1,1/fp@0,0:devctl
  Detected FCode Version:       ISP2312 Host Adapter Driver: 1.14.09 03/08/04

  Opening Device: /devices/ssm@0,0/pci@18,600000/SUNW,qlc@1/fp@0,0:devctl
  Detected FCode Version:       ISP2312 Host Adapter Driver: 1.14.09 03/08/04
  Complete


# luxadm -e dump_map /devices/ssm@0,0/pci@19,700000/SUNW,qlc@1,1/fp@0,0:devctl
Pos  Port_ID Hard_Addr Port WWN         Node WWN         Type
0    1f0112  0         5006048accab4f8d 5006048accab4f8d 0x0  (Disk device)
1    1f011f  0         5006048accab4e0d 5006048accab4e0d 0x0  (Disk device)
2    1f012e  0         5006048acc7034cd 5006048acc7034cd 0x0  (Disk device)
3    1f0135  0         5006048accb4fc0d 5006048accb4fc0d 0x0  (Disk device)
4    1f02ef  0         50060163306043b6 50060160b06043b6 0x0  (Disk device)
5    1f06ef  0         5006016b306043b6 50060160b06043b6 0x0  (Disk device)
6    1f0bef  0         5006016330604365 50060160b0604365 0x0  (Disk device)
7    1f19ef  0         5006016b30604365 50060160b0604365 0x0  (Disk device)
8    1f0e00  0         210100e08ba6a1cc 200100e08ba6a1cc 0x1f (Unknown Type,Host Bus Adapter)

# prtpicl -v
.
.
                 SUNW,qlc (scsi-fcp, 7f0000066b)   <---  go to qLogic website to get model number
                  :_fru_parent   (7f0000dc86H)
                  :DeviceID      0x1
                  :UnitAddress   1
                  :vendor-id     0x1077
                  :device-id     0x2312
                  :revision-id   0x2
                  :subsystem-vendor-id   0x1077
                  :subsystem-id  0x10a
                  :min-grant     0x40
                  :max-latency   0
                  :cache-line-size       0x10
                  :latency-timer         0x40

.
.

#### The subsystem-ID value determines the model of HBA.
#### For reference table Click Here

Configuring NEW LUNs:

spdma501:# format < /dev/null
Searching for disks...done


AVAILABLE DISK SELECTIONS:
       0. c1t0d0 
          /pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100000c506b2fca,0
       1. c1t1d0 
          /pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100000c506b39cf,0
Specify disk (enter its number):

spdma501:# cfgadm -o show_FCP_dev -al
Ap_Id                          Type         Receptacle   Occupant     Condition
c1                             fc-private   connected    configured   unknown
c1::2100000c506b2fca,0         disk         connected    configured   unknown
c1::2100000c506b39cf,0         disk         connected    configured   unknown
c3                             fc-fabric    connected    unconfigured unknown
c3::50060482ccaae5a3,61        disk         connected    unconfigured unknown
c3::50060482ccaae5a3,62        disk         connected    unconfigured unknown
c3::50060482ccaae5a3,63        disk         connected    unconfigured unknown
c3::50060482ccaae5a3,64        disk         connected    unconfigured unknown
c3::50060482ccaae5a3,65        disk         connected    unconfigured unknown
c3::50060482ccaae5a3,66        disk         connected    unconfigured unknown
c3::50060482ccaae5a3,67        disk         connected    unconfigured unknown
c3::50060482ccaae5a3,68        disk         connected    unconfigured unknown
c3::50060482ccaae5a3,69        disk         connected    unconfigured unknown
c3::50060482ccaae5a3,70        disk         connected    unconfigured unknown
c3::50060482ccaae5a3,71        disk         connected    unconfigured unknown
c3::50060482ccaae5a3,72        disk         connected    unconfigured unknown
c4                             fc           connected    unconfigured unknown
c5                             fc-fabric    connected    unconfigured unknown
c5::50060482ccaae5bc,61        disk         connected    unconfigured unknown
c5::50060482ccaae5bc,62        disk         connected    unconfigured unknown
c5::50060482ccaae5bc,63        disk         connected    unconfigured unknown
c5::50060482ccaae5bc,64        disk         connected    unconfigured unknown
c5::50060482ccaae5bc,65        disk         connected    unconfigured unknown
c5::50060482ccaae5bc,66        disk         connected    unconfigured unknown
c5::50060482ccaae5bc,67        disk         connected    unconfigured unknown
c5::50060482ccaae5bc,68        disk         connected    unconfigured unknown
c5::50060482ccaae5bc,69        disk         connected    unconfigured unknown
c5::50060482ccaae5bc,70        disk         connected    unconfigured unknown
c5::50060482ccaae5bc,71        disk         connected    unconfigured unknown
c5::50060482ccaae5bc,72        disk         connected    unconfigured unknown
c6                             fc           connected    unconfigured unknown

spdma501:# cfgadm -c configure c3
Nov 16 17:32:25 spdma501 last message repeated 54 times
Nov 16 17:32:26 spdma501 scsi: WARNING: /pci@8,700000/SUNW,qlc@2/fp@0,0/ssd@w50060482ccaae5a3,48 (ssd2):
Nov 16 17:32:26 spdma501        corrupt label - wrong magic number
Nov 16 17:32:26 spdma501 scsi: WARNING: /pci@8,700000/SUNW,qlc@2/fp@0,0/ssd@w50060482ccaae5a3,47 (ssd3):
Nov 16 17:32:26 spdma501        corrupt label - wrong magic number
Nov 16 17:32:26 spdma501 scsi: WARNING: /pci@8,700000/SUNW,qlc@2/fp@0,0/ssd@w50060482ccaae5a3,46 (ssd4):
Nov 16 17:32:26 spdma501        corrupt label - wrong magic number
Nov 16 17:32:26 spdma501 scsi: WARNING: /pci@8,700000/SUNW,qlc@2/fp@0,0/ssd@w50060482ccaae5a3,45 (ssd5):
Nov 16 17:32:26 spdma501        corrupt label - wrong magic number
Nov 16 17:32:26 spdma501 scsi: WARNING: /pci@8,700000/SUNW,qlc@2/fp@0,0/ssd@w50060482ccaae5a3,44 (ssd6):
Nov 16 17:32:26 spdma501        corrupt label - wrong magic number
Nov 16 17:32:26 spdma501 scsi: WARNING: /pci@8,700000/SUNW,qlc@2/fp@0,0/ssd@w50060482ccaae5a3,43 (ssd7):
Nov 16 17:32:26 spdma501        corrupt label - wrong magic number
Nov 16 17:32:26 spdma501 scsi: WARNING: /pci@8,700000/SUNW,qlc@2/fp@0,0/ssd@w50060482ccaae5a3,42 (ssd8):
Nov 16 17:32:26 spdma501        corrupt label - wrong magic number
Nov 16 17:32:26 spdma501 scsi: WARNING: /pci@8,700000/SUNW,qlc@2/fp@0,0/ssd@w50060482ccaae5a3,41 (ssd9):
Nov 16 17:32:26 spdma501        corrupt label - wrong magic number
Nov 16 17:32:26 spdma501 scsi: WARNING: /pci@8,700000/SUNW,qlc@2/fp@0,0/ssd@w50060482ccaae5a3,40 (ssd10):
Nov 16 17:32:26 spdma501        corrupt label - wrong magic number
Nov 16 17:32:26 spdma501 scsi: WARNING: /pci@8,700000/SUNW,qlc@2/fp@0,0/ssd@w50060482ccaae5a3,3f (ssd11):
Nov 16 17:32:26 spdma501        corrupt label - wrong magic number
Nov 16 17:32:26 spdma501 scsi: WARNING: /pci@8,700000/SUNW,qlc@2/fp@0,0/ssd@w50060482ccaae5a3,3e (ssd12):
Nov 16 17:32:26 spdma501        corrupt label - wrong magic number
Nov 16 17:32:26 spdma501 scsi: WARNING: /pci@8,700000/SUNW,qlc@2/fp@0,0/ssd@w50060482ccaae5a3,3d (ssd13):

spdma501:# cfgadm -c configure c5
Nov 16 17:32:55 spdma501 last message repeated 5 times
Nov 16 17:32:59 spdma501 scsi: WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060482ccaae5bc,48 (ssd14):
Nov 16 17:32:59 spdma501        corrupt label - wrong magic number
Nov 16 17:32:59 spdma501 scsi: WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060482ccaae5bc,47 (ssd15):
Nov 16 17:32:59 spdma501        corrupt label - wrong magic number
Nov 16 17:32:59 spdma501 scsi: WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060482ccaae5bc,46 (ssd16):
Nov 16 17:32:59 spdma501        corrupt label - wrong magic number
Nov 16 17:32:59 spdma501 scsi: WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060482ccaae5bc,45 (ssd17):
Nov 16 17:32:59 spdma501        corrupt label - wrong magic number
Nov 16 17:32:59 spdma501 scsi: WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060482ccaae5bc,44 (ssd18):
Nov 16 17:32:59 spdma501        corrupt label - wrong magic number
Nov 16 17:32:59 spdma501 scsi: WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060482ccaae5bc,43 (ssd19):
Nov 16 17:32:59 spdma501        corrupt label - wrong magic number
Nov 16 17:32:59 spdma501 scsi: WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060482ccaae5bc,42 (ssd20):
Nov 16 17:32:59 spdma501        corrupt label - wrong magic number
Nov 16 17:32:59 spdma501 scsi: WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060482ccaae5bc,41 (ssd21):
Nov 16 17:32:59 spdma501        corrupt label - wrong magic number
Nov 16 17:32:59 spdma501 scsi: WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060482ccaae5bc,40 (ssd22):
Nov 16 17:32:59 spdma501        corrupt label - wrong magic number
Nov 16 17:32:59 spdma501 scsi: WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060482ccaae5bc,3f (ssd23):
Nov 16 17:32:59 spdma501        corrupt label - wrong magic number
Nov 16 17:32:59 spdma501 scsi: WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060482ccaae5bc,3e (ssd24):
Nov 16 17:32:59 spdma501        corrupt label - wrong magic number
Nov 16 17:32:59 spdma501 scsi: WARNING: /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060482ccaae5bc,3d (ssd25):
Nov 16 17:32:59 spdma501        corrupt label - wrong magic number

spdma501:# format < /dev/null
Searching for disks...Nov 16 17:33:04 spdma501 last message repeated 1 time
Nov 16 17:33:07 spdma501 scsi: WARNING: /pci@8,700000/SUNW,qlc@2/fp@0,0/ssd@w50060482ccaae5a3,48 (ssd2):
Nov 16 17:33:07 spdma501        corrupt label - wrong magic numberdone

c3t50060482CCAAE5A3d61: configured with capacity of 17.04GB
c3t50060482CCAAE5A3d62: configured with capacity of 17.04GB
c3t50060482CCAAE5A3d63: configured with capacity of 17.04GB
c3t50060482CCAAE5A3d64: configured with capacity of 17.04GB
c3t50060482CCAAE5A3d65: configured with capacity of 17.04GB
c3t50060482CCAAE5A3d66: configured with capacity of 17.04GB
c3t50060482CCAAE5A3d67: configured with capacity of 17.04GB
c3t50060482CCAAE5A3d68: configured with capacity of 17.04GB
c3t50060482CCAAE5A3d69: configured with capacity of 17.04GB
c3t50060482CCAAE5A3d70: configured with capacity of 17.04GB
c3t50060482CCAAE5A3d71: configured with capacity of 17.04GB
c3t50060482CCAAE5A3d72: configured with capacity of 17.04GB
c5t50060482CCAAE5BCd67: configured with capacity of 17.04GB
c5t50060482CCAAE5BCd68: configured with capacity of 17.04GB
c5t50060482CCAAE5BCd69: configured with capacity of 17.04GB
c5t50060482CCAAE5BCd70: configured with capacity of 17.04GB
c5t50060482CCAAE5BCd71: configured with capacity of 17.04GB
c5t50060482CCAAE5BCd72: configured with capacity of 17.04GB


AVAILABLE DISK SELECTIONS:
       0. c1t0d0 
          /pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100000c506b2fca,0
       1. c1t1d0 
          /pci@8,600000/SUNW,qlc@4/fp@0,0/ssd@w2100000c506b39cf,0
       2. c3t50060482CCAAE5A3d61 
          /pci@8,700000/SUNW,qlc@2/fp@0,0/ssd@w50060482ccaae5a3,3d
       3. c3t50060482CCAAE5A3d62 
          /pci@8,700000/SUNW,qlc@2/fp@0,0/ssd@w50060482ccaae5a3,3e
       4. c3t50060482CCAAE5A3d63 
          /pci@8,700000/SUNW,qlc@2/fp@0,0/ssd@w50060482ccaae5a3,3f
       5. c3t50060482CCAAE5A3d64 
          /pci@8,700000/SUNW,qlc@2/fp@0,0/ssd@w50060482ccaae5a3,40
       6. c3t50060482CCAAE5A3d65 
          /pci@8,700000/SUNW,qlc@2/fp@0,0/ssd@w50060482ccaae5a3,41
       7. c3t50060482CCAAE5A3d66 
          /pci@8,700000/SUNW,qlc@2/fp@0,0/ssd@w50060482ccaae5a3,42
       8. c3t50060482CCAAE5A3d67 
          /pci@8,700000/SUNW,qlc@2/fp@0,0/ssd@w50060482ccaae5a3,43
       9. c3t50060482CCAAE5A3d68 
          /pci@8,700000/SUNW,qlc@2/fp@0,0/ssd@w50060482ccaae5a3,44
      10. c3t50060482CCAAE5A3d69 
          /pci@8,700000/SUNW,qlc@2/fp@0,0/ssd@w50060482ccaae5a3,45
      11. c3t50060482CCAAE5A3d70 
          /pci@8,700000/SUNW,qlc@2/fp@0,0/ssd@w50060482ccaae5a3,46
      12. c3t50060482CCAAE5A3d71 
          /pci@8,700000/SUNW,qlc@2/fp@0,0/ssd@w50060482ccaae5a3,47
      13. c3t50060482CCAAE5A3d72 
          /pci@8,700000/SUNW,qlc@2/fp@0,0/ssd@w50060482ccaae5a3,48
      14. c5t50060482CCAAE5BCd67 
          /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060482ccaae5bc,43
      15. c5t50060482CCAAE5BCd68 
          /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060482ccaae5bc,44
      16. c5t50060482CCAAE5BCd69 
          /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060482ccaae5bc,45
      17. c5t50060482CCAAE5BCd70 
          /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060482ccaae5bc,46
      18. c5t50060482CCAAE5BCd71 
          /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060482ccaae5bc,47
      19. c5t50060482CCAAE5BCd72 
          /pci@8,600000/SUNW,qlc@1/fp@0,0/ssd@w50060482ccaae5bc,48
Specify disk (enter its number):

IF YOU DON'T SEE THE NEW LUNS IN FORMAT, RUN devfsadm !!!!

# /usr/sbin/devfsadm

Label the new disks !!!!

# cd /tmp


# cat format.cmd
label
quit


# for disk in `format < /dev/null 2> /dev/null | grep "^c" | cut -d: -f1` 
do
  format -s -f /tmp/format.cmd $disk
  echo "labeled $disk ....."
done

A Guide to Understanding Veritas Volume Replicator

2007-12-03T14:05:00.000-08:00

This document is aimed at providing a solid architectural and technical overview of VERITAS Volume Manager’s IP option, VERITAS Volume Replicator.

INTRODUCTION: REPLICATION AND DISASTER RECOVERY PLANNING

In today’s business environment, reliance on information processing systems continues to grow on an almost daily basis. Information systems, which once aided a company in doing business, have now become the business itself. As companies become more reliant on critical information systems, the potential disruption to the business due to a loss of data becomes even greater.

There are many threats that organizations face today when it comes to the reliability and viability of their data. Logical corruption, or loss of data due to threats such as virus and software bugs, can be protected by ensuring there is a viable copy of data available at all times. Performing regularly scheduled backups of the organizations data typically protects against logical types of data loss. Another threat that may result in unavailable may be contributed to component failure. While most devices have begun to build in redundancy there are other technologies such as application clustering technologies that can protect against a failure of a component while continuing to enable applications to be available.

Just as the levels of protection for logical and component failures have grown, so has the reliance on the information systems being protected. Many companies now realize that logical and local protection is no longer enough to guarantee that the organization will continue to be accessible. This loss can stem from planned downtime such as site complete site maintenance to unplanned downtime such as power or cooling loss to natural disasters such as fire and flooding to acts of terrorism or war. The loss of a complete data center facility would so greatly affect an organizations capability to continue to function that protection must be established at the data center level.

Many companies have implemented significant Disaster Recovery (DR) plans to protect against the complete loss of a facility. Complete plans are in place to recover voice and data capability in a remote location. One issue that is common among many DR plans is the information processing recovery plan. For many years companies have been taking regular data backups at the primary data center, then duplicating these tapes on a regular basis for shipment offsite to the DR facility. While a tape based backup solution is the needed safety net for disaster recovery planning there is still a need in the IT environment to provide higher levels of protection for critical data. While a tape backup approach may meet the needs of much of the data within an organization there are many data types that cannot afford the levels of data loss inherent to a tape backup approach.

The first step to understanding which technologies are necessary for particular data types is to understand when and if appropriate technologies are needed. The key measure of disaster recovery technologies is based on recovery point objectives and recovery time objectives.

Recovery Point Objective (RPO) – Point in time to which applications data must be recovered to resume business transactions.
Recovery Time Objective (RTO) – Maximum elapsed time allowed before lack of business function severely impacts an organization.

A complete disaster recovery plan is not delivered by any one technology, service or vendor but rather a culmination of products that are implemented in order to provide the needed RPO and RTO of an application. When analyzing a disaster recovery solution many components must be implemented in order to guarantee application availability.

The diagram shown above outlines software technologies that map to a customer’s RPO and RTO requirements. The burst in the middle represents a disaster. To the left of the burst is the recovery point objective and has the appropriate software technology based on business needs. For example, if a particular application can afford a day or more worth of data loss then a tape backup approach is all that that is needed for that application. However, if a day or more worth of data loss will cause substantial business impact then replication technologies must be implemented into the IT environment to protect against substantial data loss. To the right of burst is the recovery time objective. If a business can afford to take a day or more in order to resume normal business activity then manual tape restore will satisfy their business needs. Organizations can improve on this RTO by using bare metal restore technologies that dramatically reduce the amount of time it takes to get a server up and running. However, if those technologies do not meet the applications RTO then clustering technologies must be implemented into the IT environment to protect against substantial downtime.

UNDERSTANDING THE NEED FOR REPLICATION

Replication is a technology designed to maintain a duplicate data set on a completely independent storage system at a difference geographical location. Replication differs from a tape backup and restore methods because replication is completely automatic and far less labor intensive. In addition, replication technologies can be used to reduce the recovery point objective of critical applications.

Whether motivated by disaster, site failure, or a planned site migration, VERITAS’ replication technologies provide the ability to distribute data for seamless availability across sites. VERITAS Volume Manager provides remote mirroring capabilities natively over Fibre Channel protocols. For organizations that wish to replicate their data natively over a standard IP network, an optional capability of VERITAS Volume Manager called VERITAS Volume Replicator can reliably, efficiently and consistently replicate data to remote locations. VERITAS’ replication technologies provide a robust storage-independent disaster recovery solution when data loss and prolonged downtime cannot be tolerated.

REPLICATION MODES
The two main types of replication are synchronous and asynchronous. Both have their advantages and disadvantages and should be available options for the IT administrator. Each uses a different process to arrive at the same goal, and each deals somewhat differently with network conditions. The performance and effectiveness of both depend ultimately on business requirements such as how soon updates must be reflected at the target location. Performance is strongly determined by the available bandwidth, network latency, the number of participating servers, the amount of data to be replicated and the geographical distance between the hosts.

Synchronous Replication
Synchronous replication ensures that a write update has been posted to the secondary location(s) and the primary location before the write operation is acknowledged to be complete at the application level. This way, in the event of a disaster at the primary location, the data recovered at the secondary location will be an exact copy of the data at the primary location. Synchronous replication produces the exact same data at both the primary and secondary location(s), which means the RPO of applications using synchronous replication would be zero. However, since the application transaction must travel to the secondary location(s) and back to the primary location before the application can continue with the next transaction there will be some application performance impact. Synchronous replication is most effective in metropolitan area networks with application environments that require zero data loss and can afford some application performance impact. For all other application, asynchronous replication should be a viable alternative.

There are many scenarios that could affect the performance of replication in synchronous mode including the amount of write activity on the system, the network pipe connecting the primary and secondary sites, and the distance between the two sites. A good rule of thumb to use is 3ms of latency for every 100 miles of distance between the primary and secondary systems. Most configurations that use synchronous replication have it set to change to asynchronous mode if the network link is lost between the primary and secondary site. This is so that the primary application is not affected by a network outage.

Asynchronous Replication
Asynchronous replication eliminates the potential performance problems of synchronous methods. The secondary site may lag behind the primary site, typically only by less then one minute, offering essentially real-time replication without the application performance impact. During asynchronous replication, application updates are written at the primary, and queued for forwarding to each secondary location as network bandwidth allows. Unlike synchronous replication, the writing application does not suffer from the application performance impact of replication and can function as if replication is not occurring. Asynchronous replication should be used in organizations that can afford minimal data loss but want to eliminate application performance impact or organizations that would like to replicate data over a wide-area network. This can also be the right choice if the network bandwidth between the two sites is large enough to handle the average amount of data, but insufficient to handle the peak write activity.

Write order fidelity
Whatever mode you select, you should ensure that the data at the secondary site is never corrupted or inconsistent. The last thing you need is a non-recoverable replicated data set at the secondary location the very moment you need it most. The only way to ensure that the data is recoverable at the secondary location(s) is to ensure that the data arrives in the same order as it was written at the primary location. This is called write order fidelity. Without write order fidelity, no guarantee exists that a secondary will have consistent recoverable data. In a database environment, updates are made to both the log and data spaces of a database management system in a fixed sequence. The log and data space are usually in different volumes, and the data itself can be spread over several additional volumes. A well-designed replication solution needs to consistently safeguard write order fidelity. This may be accomplished by a logical grouping of data volumes so the order of updates in that group is preserved within and among all secondary copies of these volumes.

Replication solutions running at the hardware level typically lacks the ability to maintain write order fidelity when running in asynchronous mode. This is due to a lack of a persistent queue of writes that have not yet been sent to the secondary. If the user using hardware replication wishes to avoid application performance impact imposed by synchronous replication, they lose recoverability on the remote site, rendering the remote copy essentially useless. Therefore, maintaining write order fidelity when using replication technologies should be an absolute requirement to ensure the recoverability of data at a remote location.

TECHNICAL REQUIREMENTS OF A REPLICATION SOLUTION

Architecturally, a complete replication solution must provide a copy of all data at the primary and secondary locations, including database files as well as any other necessary binary and control files and the replication technology must ensure that the data is accurate and recoverable.

The replication solution must be capable of being configured to support a secondary site over any distance. In today’s environment, organizations must be allowed to use current infrastructure, including current data center’s, regardless of distance. The replication solution must operate at any distance whether the data centers are a few kilometers apart or thousands of kilometers apart without adding undue cost or complexity. This means the replication technology must provide asynchronous support, over a long distance, without additional high cost items such as communication converters or additional disk space for staging data. The replication solution must be flexible enough to allow the customer to change the configuration of the data sets that are being replicated. This could include having volumes that are not replicated, because they are for temp files, or files that wouldn’t be needed in a disaster. There should also be the ability to grow and shrink the volumes with no application or customer downtime. The solution should also allow testing at the remote site in order to validate the data at the secondary is recoverable.

VERITAS REPLICATION AND REMOTE MIRRORING OVERVIEW

VERITAS replication and remote mirroring technologies can dramatically speed recovery time and eliminate data loss by making current data available immediately at an alternate location. Organizations can replicate or mirror data via a storage area network or over any IP network in order to meet their disaster recovery needs. Unlike proprietary, inflexible hardware approaches, VERITAS’ replication and remote mirroring technologies are not dependent on any specific storage hardware platform. For example, replication can occur between storage arrays from the same vendor, regardless of array model or size, or replication can occur between different storage vendor’s arrays. The only requirement is that there is matching Volume Manager volume sizes at each side.

VERITAS’ software-based replication provides a reliable, efficient and cost-effective solution for geographically mirroring data sets. It also has full database management system support, including DB2, Exchange, Oracle, SQL Server and Sybase.

VERITAS VOLUME MANAGER
VERITAS Volume Manager is the industry leader in storage virtualization. It provides an easy-to-use, online storage management tool for heterogeneous enterprise environments. Organizations can extend their storage management functionality with Volume Manager’s remote mirroring capability in order to deliver a metropolitan area disaster recovery solution. VERITAS Volume Manager can synchronously mirror data natively over storage protocols such as Fibre Channel, which makes it an ideal solution for disaster recovery within a metropolitan area network. Customer’s wishing to implement a disaster recovery solution over Fibre Channel can create “just another mirror” of their data over an extended distance, using VERITAS Volume Manager, to be made available should a complete site outage occur. This solution allows for using different storage arrays at the two sites, and is seamless to the Storage Administrators and users of the data.

VERITAS VOLUME REPLICATOR
For organizations who wish to replicate data natively over an IP network, VERITAS Volume Manager has an optional capability called Volume Replicator. VERITAS Volume Replicator reliably, efficiently and consistently replicates data to remote locations over an IP network for maximum business continuity, removing the need for expensive proprietary network hardware, and the need to have the exact same storage hardware at every site.

Since Volume Replicator (VVR) is just an optional capability to VERITAS Volume Manager (VxVM), VVR allows VxVM volumes on one system to be exactly replicated to identically sized volumes on another system. For example, an Oracle database may use several different volumes for various tablespaces, redo logs, archived redo logs, indexes and other storage. Each component is typically stored in a VxVM volume, or multiple volumes. The database may also use data stored in VERITAS File Systems, created in these volumes for better manageability. VVR can provide an exact duplicate of these volumes to another system, at another site and since the replication occurs at the volume level VVR can replicate data between any storage hardware arrays. VERITAS Volume Replicator can scale to support up to 32 secondary data storage sites and Volume Replicator elegantly handles one-to-one, one-to-many and many-to-one data replication configurations.
There are four main components that are added to the VERITAS Volume Manager code base to provide VERITAS Volume Replicator. These four components are replicated volume groups (RVG), storage replicator log (SRL), rLinks and data change maps (DCM).

The above diagram shows the architecture of VERITAS Volume Replicator. VERITAS Volume Replicator is based on the same code base as VERITAS Volume Manager and adds three components: the Rlink, SRL and RVG.

Replicated Volume Groups
VVR extends the concept of a disk group (found in VERITAS Volume Manager) to provide the concept of a replicated volume group (RVG). An RVG is a subset of volumes within a given VxVM disk group configured for replication to one or more secondary systems. An RVG can contain one or more data volumes in the disk group, up to and including all data volumes, but cannot span multiple disk groups. Multiple RVGs can be configured inside one disk group, but not the other way around. Volumes that are associated with an RVG and that contain application data are called replicated data volumes. The concept of an RVG gives the user the ability to pick and choose what data is replicated to the secondary site. Therefore, organizations need only to replicate their mission-critical data to a secondary location, which allows organization to save money on replication implementations because they need a lot less storage at their secondary site. In addition, organizations that pay for bandwidth usage can save on bandwidth costs because they are only replicating data that has stringent recovery point objectives. All other data can be protected by simply using a tape backup approach.

The data volumes in the RVG are under the control of an application, such as a database management system, that requires write-order fidelity among the updates to the volumes. Write ordering is strictly maintained within an RVG during replication to ensure that each remote volume is always consistent, both internally and with all other volumes of the group. At the simplest level, VVR exists within the VxVM code base and has the capability to intercept any write destined for a VxVM volume within an RVG and replicate the write, in the correct order, to designated secondaries before the write is passed on to the actual data volumes.

Data is replicated from a primary RVG to a secondary RVG. The primary RVG is in use by an application, while the secondary RVG receives replication and writes to local disk. The concept of primary and secondary is per

RVG, not per system. A system can simultaneously be a primary RVG for some RVGs and secondary RVG for others. The RVG also contains the storage replicator log (SRL) and replication link (RLINK) explained in the following sections.

Storage Replicator Log
All data writes destined for volumes configured for replication are first persistently queued in a log called the Storage Replicator Log. VVR implements the SRL at the primary side to store all changes for transmission to the secondary(s). The SRL is a VxVM volume configured as part of an RVG. The SRL gives VVR the ability to associate writes to specific volumes within the replicated configuration in a specific order, maintaining write order fidelity at the secondary. All writes sent to the VxVM volume layer, whether from an application such as a database writing directly to storage, or an application accessing storage via a file system are faithfully replicated in application write order to the secondary.

When implementing asynchronous replication data integrity must be guaranteed at the remote site, or it must preserve write order fidelity. This essentially means that writes applied to the secondary storage must occur in the exact same order they were applied at the primary. Asynchronous replication without this capability compromises data consistency at the disaster recovery site and may jeopardize the recoverability of the data. The SRL within Volume Replicator tracks writes in the correct order and guarantees that the data will arrive at the secondary site in that same order, whether operating in synchronous or asynchronous mode.

The SRL can also be used with synchronous or asynchronous replication to protect against network outages. Should a network outage occur when replicating in synchronous mode, Volume Replicator can automatically switch to asynchronous mode and the writes can be stored in the SRL for later transmission to secondaries when the network is restored. This functionality protects the primary location from being impacted should a network outage occur.

Rlinks
An Rlink is a VVR Replication Link to a secondary RVG. Each RLINK on a Primary RVG represents the communication link from the Primary RVG to a corresponding Secondary RVG, via an IP connection. RLinks are configured to communicate between specific host names/IP addresses and can support both TCP and UDP communication protocols between systems.

Data Change Maps
Data Change Maps (DCM) are used to mark sections of volumes on the primary that have changed during extended network outages in order to minimize the amount of data that must be synchronized to the secondary site during the outage.

VOLUME REPLICATOR TECHNICAL DETAILS

OPERATIONAL MODES AND DATA FLOW: SYNCHRONOUS
In Synchronous mode, all data writes are first posted to the SRL, and then sent to the secondary location(s). The posting of the data to the actual data volumes happens in this time period also. When the secondary location(s) receives the data write and acknowledgement is sent back to the primary location and then the application acknowledges the data write to be complete. Therefore, synchronous replication should be used in environments that cannot afford any data loss and can afford some application performance impact. Overall performance in synchronous mode is governed by the amount of time that it takes to write to the SRL plus the round trip time to send data to the secondary and receive acknowledgement.

The secondary acknowledges receipt as soon as the full write transaction (as sent by the application on the primary) is received into VVR kernel memory space. This removes actual writing to the secondary data volumes from the application latency. The primary tracks these writes in its SRL until a second acknowledgement is received from the secondary, signalling that the data has been written to physical storage. Both acknowledgements have an associated timeout so if a packet is not acknowledged, VVR will resend that packet.

In order to maximize performance, VVR does not wait for data to be written at the secondary, only received. This improves application performance. But it tracks all acknowledged, uncommitted transactions and can replay any necessary transactions if the secondary were to crash prior to actually writing its data to the physical storage.

Synchronous mode has two possible rlink settings, “fail” and “override”. These settings deal with the behavior of VVR when the network connection is lost to a secondary site. With synchronous=fail, writes will be returned as failed to the calling application if contact is lost with the secondary. This is used in rare situations where the primary and secondary storage must never differ by even one write. This is typically not used, as it will cause an application failure at the primary side when anything happens to the secondary or the interconnecting network. The “synchronous=override” is the most common setting. This will keep replication in synchronous mode unless contact with the secondary is lost, then will shift to asynchronous mode and begin tracking the writes in the SRL. This allows the primary application to continue to run and provide service to customers while the backup capability is restored.

OPERATIONAL MODES AND DATA FLOW: ASYNCHRONOUS
In asynchronous mode, an application data write is first placed in the SRL, then immediately acknowledged to the calling application at the primary site. Data is then sent as soon as possible to the secondary location, based on available bandwidth. Therefore, asynchronous replication will not have any impact to the performance of the application but the secondary location may be a few write requests behind the primary site should a disaster occur. Asynchronous replication should be used in environments where minimal data loss (typically measured in seconds for adequately sized networks) can be tolerated but where application cannot afford the performance impact of synchronous replication. In asynchronous mode, if a disaster does occur at the primary site the secondary location will be able to recover, but it may come up a few write requests behind the state of primary systems.

Using Asynchronous Replication to Decouple Application Latency
One of the most compelling features of VVR in real world environments is its ability to maintain full consistency at the secondary site while operating in asynchronous mode. Maintaining write order fidelity in asynchronous mode allows VVR to truly make use of the performance benefits available from asynchronous replication. By providing a high bandwidth connection, customers can completely remove the latency penalty from replication, and still maintain near up to the second data at the remote site. At the primary site, the application is acknowledged as soon as data is placed in the SRL. The application can continue to function as if replication is not occurring on the host. The data is then sent out almost instantaneously over the network to the secondary site. With adequate bandwidth, the SRL will not fill, so the actual data outstanding between primary and secondary is realistically whatever data is currently on the wire. This means a company can have near up to the second replication, at an arbitrary distance, with no application penalty.

Latency Protection
Latency protection allows administrators to define how far a secondary is allowed to fall behind a primary in asynchronous mode. The latency protection feature allows automatic control of excessive lag between Primary and Secondary nodes. Latency protection gives the network administrator the option to set the maximum number of updates that are allowed in the SRL, which is referred to as a latency_high_mark. When this number is reached, all update activity is delayed until the update backlog has reached a preset level, or the latency_low_mark. Latency protection ensures that the number of recent updates that could be lost in a disaster does not exceed a maximum determined amount. Latency protection is typically used to prevent the secondary from falling too far behind the primary in order to meet the recovery point objectives of the organization.

No Distance Limitations
Because VVR is truly unique in its ability to replicate data either synchronously or asynchronously over any standard IP network, there are no distance limitations. This allows organizations to utilize data centers that are already in place regardless of the distance between the locations.

INITIALIZING SECONDARY SYSTEMS FOR REPLICATION
In order to begin replication of changed blocks, a replication solution must first begin with a known duplicate data set at each site. VVR offers several ways to get a secondary site up and running in order to begin the process of replication.

Empty
The simplest method to begin replication is to start with completely empty systems at each side. This can be done if VVR is installed while initially constructing a data center, prior to production. For VVR to use an empty data set, both sides must be identically empty.

Over the Wire (Autosync)
Over the wire initialization is essentially using VVR to move all data from the primary to secondary over the network connection. Overall this is a very simple process, however, with larger data sets it can take a prohibitively long time, especially if the primary is active while attempting to initialize the secondary. In addition, in situations where organizations are leasing bandwidth lines this process can become fairly expensive. The items that must be taken into consideration to do this are:

• The network bandwidth

• The amount of write activity on the system

• The size of the SRL

This is the optimum way for doing the initial synchronization, because it can be repeated at any given time, if the solution is designed to allow for this process.

Local Mirroring
Local mirroring is an option for very large data sets. In this method, the data storage array for the secondary site is initially placed at the primary site and Volume Manager is used to mirror the data between the two storage devices. Once the mirror is complete, the Volume Manager plex is split off and the array is shipped to the secondary site. This mode will allow large amounts of data to be initialized at SAN speeds, as well as allowing subsequent data written during the shipping period to be spooled to the primary SRL. However, this can be fairly expensive if the array must be shipped over a long distance.

Tape Backup and Restore Initialization
The final option for initialization is through the use of a tape backup and restore. This is a unique feature of VVR and is not an available option for other replication technologies on the market today. It allows huge data sets to be synchronized using tape technology and immediately begin replication.

Checkpoint initialization is a hot backup of the primary side, with the SRL providing the map of what was changed during the backup. When a checkpoint initialization is started, a “check start” pointer is placed in the SRL. A full block level backup is then taken of the primary volumes. When complete, a check-end pointer is placed into the SRL. The data written between the check-start and check-end pointers represents data that was modified while the backup was taking place. This constitutes a hot backup.

The tapes are then transported to the secondary site and the data is restored to the secondary systems using the tapes. When the tape load is complete, the secondary site is connected to the primary site with a “checkpoint attach” of the rlink. The primary will then forward any data that had been written during the backup (that data between the check-start and check-end). Once this data is written to the secondary, the secondary is an exact duplicate of the primary, at the time the backup completed. At this point the secondary is consistent, and simply out of date. The SRL is then replayed to bring the secondary up to date. Therefore, tape backup and restore initialization is ideal for organizations that wish to perform an initialization of a large data set or for environments where their secondary site is located over longer distances.

RECOVERY AFTER PROBLEMS

VVR is very robust in terms of tolerating outages of the network as well as secondary systems. The SRL provides the key to recovering from outages, while maintaining a consistent secondary.

SECONDARY/NETWORK OUTAGE
VVR can be configured to handle network outages. An outage of a secondary system, or outage of the network to the secondary is identical as far as VVR is concerned. When the secondary is no longer available, as evidenced by loss of VVR heartbeat on the rlink, the primary will simply track the data write changes in the SRL. When the secondary is repaired or network problems are resolved, the SRL will then send all the changes to the secondary location(s). In addition, VVR can be configured to stop operations at the primary site should a network fail. In this case, the primary will not allow writes until the network connectivity is re-established, or the secondary is available for write activity

SECONDARY FAILURE
A failure of the secondary would be better defined as a failure of the secondary storage, resulting in a data loss on the secondary side.

There are several methods to recover from a secondary loss. The first is to rebuild the secondary storage and re-initialize using one of the methods discussed above. The second method is to take regular backups of the secondary environment using a VVR feature called “Secondary Checkpoints”. Secondary Checkpoints allow a pointer to be placed in the primary SRL to designate a location where a backup was last taken on the secondary. Assuming the primary has a large enough SRL and secondary backups are routinely taken, a failure at the secondary can be repaired by reloading the last backup and rolling the SRL forward from the last secondary checkpoint.

PRIMARY FAILURE
A failure of the primary can be broken into several possible problems. A complete failure of the primary site is handled by promotion of a secondary to a primary, affecting a disaster recovery takeover. This is exactly the scenario VVR was built for.

For primary outages, such as server failure, or server panic, the customer has the choice to wait for the primary to recover, or shift operations to a secondary server or location.

For situations involving actual data loss at the primary, the customer can shift operations to a secondary, or restore data on the primary.

VOLUME REPLICATOR ROLE CHANGES
Role changes are actions to promote system that was previously a secondary to a primary. This can be due to a complete site outage where the primary site is not available or simply a role reversal to allow a secondary site to take over operations. For simple failures of a primary or secondary server that are members of a VCS cluster, VCS can move the primary or secondary to a new server without a role change. For example, imagine a two node cluster at Site A, acting as the VVR primary, and a two-node cluster at Site B acting as the VVR secondary. If a node in Site A dies, the VVR primary will simply move to the second node under VCS control. The same is true of a single system failure at the secondary site. VCS would restart the VVR secondary on the opposite system. For situations such as a complete failure of Site A, the VVR secondary at Site B can be promoted to a primary and applications started to access the underlying data in read-write mode. This is an example of using VVR to facilitate Disaster Tolerance for a data center. To automate this entire procedure, VERITAS offers VERITAS Global Cluster Manager with the Disaster Recovery option to monitor and control applications at separate sites, connected by replication.

Primary Migration
A migration of Primary to Secondary systems is a controlled shift of Primary responsibility for an RVG. Data is flushed from the existing primary SRL if necessary, and then control is handed to the existing secondary. The original primary is demoted to a secondary, and the original secondary is promoted to a primary. This is a very simple operation carried out with one or two commands and allows rapid shift of replication primary between sites. There is zero chance for data loss, as all data outstanding at the primary site is sent to the secondary prior to allowing the migration to take place.

Secondary Takeover
A secondary takeover is a somewhat less graceful event, in that the secondary is promoted to a primary without a corresponding demotion of the original primary. These types of migrations typically happen in the event of a complete site outage without and prior notification. When a takeover is accomplished, the secondary is brought up in read-write mode in the exact state it was in at time of takeover. Any data written by the primary in ASYNC mode and not sent to the secondary is not available. After a secondary takeover, the original primary must be re-synchronized to be an exact duplicate of the new primary.

Returning to the Primary Location
After a migration has occurred to a secondary location, returning to the original primary can be accomplished using easy failback, a new feature in VVR 3.2, which allows for rapid resynchronization of an old primary after a secondary takeover. This removes the need for a complete over the wire synchronization or all the data or differential based synchronization.

When a secondary is promoted in a takeover operation, it immediately begins utilizing the Data Change Map (DCM) associated with each volume. This tracks where data has been written on the new primary. When the old primary comes back online and a failback operation is requested, the new primary communicates with the old primary and determines any data blocks that had been changed on the old primary that were not committed on the old secondary/new primary. These block locations are sent to the new primary so the corresponding blocks can be sent to the Data Change maps on the new primary. This means that any blocks written on the new primary, plus any blocks that were different on the old primary will be sent from new primary to old primary. This results in the old primary being made an exact duplicate of the new primary in a very short time. This also means that any data that was written on the old primary is permanently overwritten. VVR makes no attempt at “merging” differences between systems.

USING THE SECONDARY SYSTEM

FlashSnap is an option to VERITAS Foundation Suite and it allows a complete mirror break-off (snapshot) of a VVR volume to be taken and mounted for operation. The benefit of this option to VERITAS Volume Manager is that it allows organizations to access the data at the secondary sites for off host processing, such as reporting and backups.

USING IN BAND CONTROL MESSAGES TO CONTROL FLASHSNAP
VVR provides an advanced messaging capability to control specific events at a secondary from the primary. It does this with In Band Control (IBC) messages sent from the primary to the secondary. IBC messages are placed in the SRL like any other write traffic and are processed in SRL order. For example, consider a database running at the primary site, with VVR in asynchronous mode. The administrator places the database in hot backup mode to perform an onsite backup. An IBC can be placed in the SRL at this time to signal the secondary when to snap off a mirror, knowing that the IBC will not be received until all data ahead of it in the SRL (right to the time of shifting to hot backup) has been received. As soon as the IBC is entered into the SRL, the database can be taken out of hot backup mode at the primary site. This allows operations at the primary and secondary sites to be coordinated to occur at the exact same time in terms of data consistency.

VOLUME REPLICATOR IN THE CUSTOMER ENVIRONMENT

EFFECTS OF VOLUME REPLICATOR ON HOST RESOURCES AND APPLICATION PERFORMANCE

VVR typically has very little effect on host CPU and memory resources and have been measured in the 2-5% range. This type of CPU usage can be similar to the CPU impact one may notice doing a simple find command in UNIX. VVR also converts all of the write activity to Sequential writes. This is the fastest configuration for writes in most cases, and some customers have observed an increase in write performance using VVR when compared to not using replication within their environment.

UNDERSTANDING BANDWIDTH NEEDS USING VRADVISOR
In order to replicate data to another location, bandwidth must be available. For environments utilizing Fibre Channel connectivity VERITAS Volume Manager can be used to mirror the data between the two locations. For environments with IP connectivity, VERITAS Volume Replicator should be used. VERITAS Volume Replicator does not specifically require a network dedicated to itself, is resilient to temporary network outages and includes error-handling capabilities to alert the administrator of critical events.

A very common question is “How much bandwidth do I need?” The answer is enough bandwidth must be provided to move all write traffic to each secondary site in any given time period. For example, if 10 Gigabytes of data are written in a 24-hour period, then enough bandwidth must be provided to move 10 Gigabytes of data in 24 hours. If the configuration is set to Synchronous, then attention should be paid to the peak write activity. This activity can impact the performance of the application if the network bandwidth isn’t sufficient for the peak traffic.

The SRL can be used to spool data during time periods when write traffic exceeds replication bandwidth. In order to assist in the proper determination of the bandwidth and SRL size, the VERITAS Volume Replicator Advisor (VRAdvisor) utility is available for use. The VRAdv tool will assist the user in determining the optimal size of the SRL by taking into account the rate of data writes over a given time period, network bandwidth and different outage durations.

Collection of Data
The VRAdivisor can collect sample data write statistics based on various parameters. The data is collected over a period of time, in a file that you have specified. If VxVM is installed then the vxstat command will be used for collecting data. Otherwise, the iostat command will be used. After the data change rate has been collected the data can then be analyzed to make determinations on the optimal size of the SRL based on the different parameters that you had supplied. This result would provide you the optimum size of the SRL for immediate requirements. You can also calculate the size of the SRL based on the future requirements, changes, and other factors which you are aware of and may affect the SRL.

Analysis Results
This above screen displays the analysis results, which are generated based on the inputs that you have specified. The graphical display region displays the analysis results with the help of two graphs. The x-axis for both the graphs consists of the data write duration values based on the information collected on the system. The y-axis of the top graph highlights the SRL fill rate over the data collection period. In addition, the peak SRL fillup size is indicated against a maximum outage window. This window is displayed in yellow and would indicate a worst-case scenario. The second graph highlights the different write rates at periods throughout the collection period.

VOLUME REPLICATOR INTEGRATION WITH OTHER VERITAS PRODUCTS
Volume Replicator is a component of an overall high availability and disaster recovery solution. It fits very well into an overall high availability infrastructure provide by VERITAS Cluster Server and VERITAS Global Cluster Manager.

VERITAS CLUSTER SERVER AND VERITAS GLOBAL CLUSTER MANAGER
The full integration of VERITAS Cluster Server, VERITAS Volume Replicator and VERITAS Cluster Manager provides a powerful disaster recovery solution. VERITAS Cluster Server handles local availability issues. VERITAS Volume Replicator replicates critical data to a remote site and VERITAS Global Cluster Manager monitors and manages the clusters at each site. In the event of a site failure or complete failure of applications at the primary site, Global Cluster Manager will control the shift of replication roles to the secondary site, bring up critical applications and redirect client traffic with a single command or mouse click.

VERITAS DATABASE EDITIONS
VERITAS Database Editions offers raw device performance with the manageability of the VERITAS File System, online administration of storage, and the flexibility of storage hardware independence. Database Editions can be used within the local environment to maximize the performance of the database while Volume Replicator can be used to replicate data to the secondary site for disaster recovery protection.

VERITAS NETBACKUP
In order to have complete disaster recovery solution, every environment should be backup up on a regular basis using VERITAS NetBackup. By combining, VERITAS NetBackup and VERITAS Volume Replicator organizations can be assured that their data is protected.

VERITAS REPLICATION CAPABILITIES SUMMARY
The following section will summarize the capabilities of VVR.

STORAGE ARCHITECTURE INDEPENDENCE
VERITAS Volume Replicator replicates between any major hardware platforms to eliminate vendor-specific storage limitations. For example, using VERITAS Volume Replicator, customers can replicate between a single vendor's alike arrays, between a single vendor's dissimilar arrays, or between two different vendors' arrays. This means the only architecture restriction for VERITAS Volume Replicator is duplicate volume sizes at each end. In order to replicate a 200 Gigabyte volume, the customer must create a 200 Gigabyte volume on the primary and secondary(s). This allows the customer to create a secondary site utilizing older or less expensive hardware and only requires customers to replicate critical data, chosen based on volume, to the secondary site saving on bandwidth and storage costs. In addition, VVR provides the flexibility to change the storage configuration as data sizes grow, change, shrink or are moved without impacting replication.

MAINTAINING WRITE ORDER FIDELITY
Volume Replicator maintains write order fidelity, even in asynchronous mode to guarantee data consistency on the secondary. This is critical to providing a complete, consistent copy of data at a remote site, without requiring synchronous replication.

HIGH PERFORMING REPLICATION TECHNOLOGIES
Volume Manager and Volume Replicator are high performing replication technologies. Third party performance testing has proven that VERITAS is up to 72% faster then the leading hardware replication vendor on the market today.

NATIVE REPLICATION OVER FIBRE CHANNEL AND IP NETWORKS
VERITAS technologies can replicate over Fibre Channel and IP networks natively. Volume Manager can replicate over Fibre Channel and Volume Replicator can replicate over IP networks without the need for any expensive specialized networking devices. In addition, native replication over IP allows organizations to replicate data over any distance.

SCALABLE
Volume Replicator can scale to up to 31 separate locations for many to one and one to many replication scenarios.

INITIALIZATION OPTIONS
Volume Replicator can assist in getting your disaster recovery site up and running quickly. There are three initialization options available with Volume Replicator. The first option involves sending all of the data over the wire. The second option is by doing local mirroring between arrays and shipping the array to the disaster recovery site. The third option that is unique to Volume Replictor combines replication and backup to get the disaster recovery site up and running quickly. The organization performs a normal backup at the primary site and inserts a checkpoint. Then the tapes are sent to the disaster recovery site and a tape restore is performed. Only the data that has changed since the time the checkpoint is inserted is sent over the wire. This allows organizations to get up and running without sending large datasets over the wire or having to pay expensive shipping costs to ship storage arrays. All three operations can be performed completely online.

SUMMARY
VERITAS Volume Replicator can effectively and efficiently replicate data to another location in order to provide protection from disaster scenarios. VERITAS Volume Replicator allows organizations to replicate their data between any storage devices, over a standard IP connection, and across any distance for the ultimate in disaster recovery protection.

Setting up the look and feel of this site

2007-12-01T15:02:00.000-08:00

This entry is mainly for testing the different parameters in this template.

Sample text below....

Lesson 1: (using class:post_heading1)

A man is getting into the shower (post_text)
just as his wife is finishing up
her shower, when the doorbell rings.

The wife quickly wraps herself in
a towel and runs downstairs.

When she opens the door,
there stands Bob,
the next-door neighbour.

Before she says a word, Bob says,
"I'll give you $800 to drop that towel."

After thinking for a moment,
the woman drops her towel
and stands naked in front of Bob,
after a few seconds,
Bob hands her $800 and leaves.

The woman wraps back up in
the towel and goes back upstairs.
When she gets to the bathroom,
her husband asks, "Who was that?"
"It was Bob the next door neighbour,"
she replies.

"Great," the husband says,
"did he say anything about
the $800 he owes me?"

VVR Implementation

2007-11-02T16:36:00.000-07:00

Here’s one of the VVR projects I designed and implemented. I documented the whole process so I could use it as a reference for future projects.

The document has a detailed implementation (with screenshots). The changes were done on both primary and secondary sites. Downtime was not necessary but the customer decided to shut down the apps and database.

This implementation involves VCS,CCA and SNAPshots. I will not discuss CCA in this document.

Contents

Project Description
Server Configurations
Network Configurations
Essential Terminology
Volumes to be Replicated
Storage Replication Log (SRL)
Data Change Map (DCM)
Disk Change Object (DCO)
Technical Implementation Steps/Procedures
Detailed Implementation

Project Description

As part of the Disaster recovery project, Veritas Volume Replicator (VVR) will be implemented on EMM servers to handle the replication between the Phoenix servers and the DR servers in Minneapolis. VVR will run on top of Veritas Cluster System (VCS). VCS is already installed and running on the servers. Replicated volumes in the secondary site will be configured to have snapshot volumes.

Server Configurations

E3 (Clustered)

SUNSRV01 (Solaris 8, SunFire V440)
SUNSRV02 (Solaris 8, SunFire V440)

DR (Clustered)

SUNSRV01-DR (Solaris 8, SunFire V440)
SUNSRV02-DR (Solaris 8, SunFire V440)

Network Configurations

IP Address

SUNSRV01
- 10.10.231.130 sunsrv01 Host IP
- 10.11.196.192 sunsrv01-DRN E3/DR Link IP
SUNSRV02
- 10.10.231.131 sunsrv02 Host IP
- 10.11.196.193 sunsrv02-DRN E3/DR Link IP
SUNSRV01-DR
- 10.10.231.130 sunsrv01-dr Host IP
- 10.12.94.192 sunsrv01dr-DRN E3/DR Link IP
SUNSRV02-DR
- 10.10.231.131 sunsrv02-dr Host IP
- 10.12.94.193 sunsrv02dr-DRN E3/DR Link IP

VVR Config

RVG db2dg_rvg
SRL db2dg_srl
RLINKs rlk_db2instdr-vipvr_db2dg_rvg (E3)
rlk_db2inste3-vipvr_db2dg_rvg (DR)

Virtual IPs

SUNSRV01/SUNSRV02
Application 10.10.231.191
E3/DR Link 10.11.196.191
SUNSRV01-DR/SUNSRV02-DR
Application 10.10.231.191
E3/DR Link 10.12.94.191

VCS Heartbeats

SUNSRV01 ce0/ce4
SUNSRV02 ce0/ce4
SUNSRV01-DR ce4
SUNSRV02-DR ce4

Essential Terminology

Data Change Map (DCM) - An object containing a bitmap that can be optionally associated with a data volume on the Primary RVG. The bits represent regions of data that are different between the Primary and the Secondary. DCMs are used to mark sections of volumes on the primary that have changed during extended network outages in order to minimize the amount of data that must be synchronized to the secondary site during the outage.

Disk Change Object (DCO) - DCO volumes are used by Volume Manager FastResync (FR). FR is used to quickly resynchronize mirrored volumes that have been temporarily split and rejoined. FR works by copying only changes to the newly reattached volume using FR logging. This process reduces the time required to rejoin a split mirror, and requires less processing power than full mirror resynchronization without logging.

Data Volume - Volumes that are associated with an RVG and contain application data.

Primary Node - The node on which the primary RVG resides.

Replication Link (RLINK) - RLINKs represent the communication link to the couterpart of the RVG on another node. At the Primary node a replicated volume object has one RLINK for each of its network mirrors. On the Secondary node a replicated volume has a single RLINK object that links it to its Primary. Each RLINK on a Primary RVG represents the communication link from the Primary RVG to a corresponding Secondary RVG, via an IP connection.

Replicated Data Set (RDS) - The group of the RVG on a Primary and its corresponding Secondary hosts.

Replicated Volume Group (RVG) - A component of VVR that is made up of a set of data volumes, one or more RLINKs and an SRL. An RVG is a subset of volumes within a given VxVM disk group configured for replication to one or more secondary systems. Data is replicated from a primary RVG to a secondary RVG. The primary RVG is in use by an application, while the secondary RVG receives replication and writes to local disk. The concept of primary and secondary is per RVG, not per system. A system can simultaneously be a primary RVG for some RVGs and secondary RVG for others. The RVG also contains the storage replicator log (SRL) and replication link (RLINK).

Storage Replication Log (SRL) - Writes to the Primary RVG are saved in the SRL on the Primary side. The SRL is used to aid in recovery, as well as to buffer writes when the system operates in asynchronous mode. Each write to a data volume in the RVG generates two write requests: one to the Secondary SRL, and another to the Primary SRL.

Secondary Node - The node too which the primary RVG replicates.

Volumes to be Replicated

These are the volumes that will be configured for replication. These volumes belong to the same diskgroup db2dg.

v  bcp          -            ENABLED  ACTIVE    585105408 SELECT   -        fsgen
v  db           -            ENABLED  ACTIVE   1172343808 SELECT   -        fsgen
v  dba          -            ENABLED  ACTIVE     98566144 SELECT   -        fsgen
v  db2          -            ENABLED  ACTIVE      8388608 SELECT   -        fsgen
v  lg1          -            ENABLED  ACTIVE    396361728 SELECT   -        fsgen
v  tp01         -            ENABLED  ACTIVE    192937984 SELECT   -        fsgen

Storage Replication Log (SRL)

Storage Replication Log. All data writes destined for volumes configured for replication are first persistently queued in a log called the Storage Replicator Log. VVR implements the SRL at the primary side to store all changes for transmission to the secondary(s). The SRL is a VxVM volume configured as part of an RVG. The SRL gives VVR the ability to associate writes to specific volumes within the replicated configuration in a specific order, maintaining write order fidelity at the secondary. All writes sent to the VxVM volume layer, whether from an application such as a database writing directly to storage, or an application accessing storage via a file system are faithfully replicated in application write order to the secondary.

v  db2dg_srl  -            ENABLED  ACTIVE    856350720 SELECT   -        fsgen

Data Change Map (DCM)

Data Change Maps (DCM) are used to mark sections of volumes on the primary that have changed during extended network outages in order to minimize the amount of data that must be synchronized to the secondary site during the outage. Only one disk will be used for DCM logs at this time.

In both primary and secondary sites, the same disk device name is used.

GROUP      DISK       DEVICE       TAG          OFFSET    LENGTH    FLAGS
db2dg      db2dg95    EMC0_94      EMC0_94      0         35681280  -

Disk Change Object (DCO)

Disk Change Object (DCO) volumes are used by Volume Manager FastResync (FR). FR is used to quickly resynchronize mirrored volumes that have been temporarily split and rejoined. FR works by copying only changes to the newly reattached volume using FR logging. This process reduces the time required to rejoin a split mirror, and requires less processing power than full mirror resynchronization without logging.

GROUP      DISK       DEVICE       TAG          OFFSET    LENGTH    FLAGS
db2dg      db2dg95    EMC0_94      EMC0_94      0         35681280  -

Technical Implementation Steps/Procedures

1. Install VVR License on all nodes for both sites
2. Install VVR Packages on all nodes for both sites
3. Create a Diskgroup and volumes in the secondary site
4. Update the .rdg file on both nodes in the secondary site
5. Stop all applications – both clustered and non-clustered apps
6. Bring up the virtual IPs on all nodes on both sites
7. Create the SRL Volume in the primary site
8. Add DCM logs to the all the primary site volumes that will be replicated
9. Update the .rdg file on both nodes in the primary site
10. Ensure that VVRTypes.cf exists in /etc/VRTSvcs/conf
11. Ensure that other application types i.e. Db2udbTypes.cf exists in /etc/VRTSvcs/conf
12. Create the Primary RVG
13. Create the Secondary RVG
14. Configure VCS to integrate VVR objects – Primary Site
15. Configure VCS to integrate VVR objects – Secondary Site
16. Bring up VCS engine and bring up the vvr service group in the Secondary Site
17. Bring up VCS engine and bring up the vvr service group in the Primary Site
18. Check the Rlink and RVG status
20. Start the replication
21. Check the Replication status
22. Mount the Replicated Volumes
23. Prepare the Replicated Volumes for Snapshot in the secondary site
24. Create the SNAP Volumes in the secondary site
25. Prepare the SNAP Volumes
26. Run a point-in-time snapshot
27. Mount the SNAP Volumes

Detailed Implementation

Install VVR License on all nodes for both sites

# /opt/VRTSvlic/sbin/vxlicinst

Install VVR packages on all nodes for both sites

List of Required and Optional Packages for VVR

Required software packages for VVR:


VRTSvlic VERITAS Licensing Utilities.
VRTSvxvm VERITAS Volume Manager and Volume Replicator.
VRTSob VERITAS Enterprise Administrator Service.
VRTSvmpro VERITAS Volume Manager Management Services Provider.
VRTSvrpro VERITAS Volume Replicator Management Services Provider.
VRTSvcsvr VERITAS Cluster Server Agents for VERITAS Volume Replicator.

Optional software packages for VVR:


VRTSjre VERITAS JRE Redistribution.
VRTSweb VERITAS Java Web Server.
VRTSvrw VERITAS Volume Replicator Web Console.
VRTSobgui VERITAS Enterprise Administrator.
VRTSvmdoc VERITAS Volume Manager documentation.
VRTSvrdoc VERITAS Volume Replicator documentation.
VRTSvmman VERITAS Volume Manager manual pages.
VRTSap VERITAS Action Provider.
VRTStep VERITAS Task Execution Provider.

Installing the VVR Packages Using the pkgadd Command

1. Log in as root.

2. Mount from the software repository:

# mount software:/repos /mnt
# cd /mnt/software/os/SunOS/middleware/Veritas4.1/volume_replicator

3. Alternatively, you may choose to run the install script from the same directory. Refer to VVR Installation Guide for more information.

# ./installvvr

4. But this document follows the manual process.

# cd /mnt/software/os/SunOS/middleware/Veritas4.1/volume_replicator

Note: Install the packages in the order specified below to ensure proper installation.

5. Use the following command to install the required software packages in the specified order. Some of these may have already been installed.

# pkgadd -d . VRTSvlic VRTSvxvm VRTSob VRTSvcsvr

6. Install the following patch:

# patchadd 115209-<latest>

7. Install the following required packages in the specified order:

# pkgadd -d . VRTSvmpro VRTSvrpro

8. Use the following command to install the optional software packages:

# pkgadd -d . VRTSobgui VRTSjre VRTSweb VRTSvrw VRTSvmdoc \
              VRTSvrdoc VRTSvmman VRTSap VRTStep

The system prints out a series of status messages as the installation progresses and prompts you for any required information, such as the license key.

Create a Diskgroup and volumes in the secondary site

initialize all LUNs and assign to Diskgroup db2dg

sunsrv01-dr:# vxdiskadm
Create volumes with the same sizes as the primary site’s volumes.

sunsrv01-dr:# vxassist -g db2dg make bcp  585105408 layout=concat
sunsrv01-dr:# vxassist -g db2dg make db  1172343808 layout=concat
sunsrv01-dr:# vxassist -g db2dg make dba   98566144 layout=concat
sunsrv01-dr:# vxassist -g db2dg make db2    8388608 layout=concat
sunsrv01-dr:# vxassist -g db2dg make lg1  396361728 layout=concat
sunsrv01-dr:# vxassist -g db2dg make tp01 192937984 layout=concat

Create the SRL Volume. Allocate disks that are dedicated just for SRL volume. No other volumes should use those disks.

sunsrv01-dr:# vxassist -g db2dg make db2dg_srl 856350720 layout=concat \
> db2dg71 db2dg72 db2dg73 db2dg74 db2dg75 db2dg76 db2dg77 db2dg78 \
> db2dg79 db2dg80 db2dg81 db2dg82 db2dg83 db2dg84 db2dg85 db2dg86 \
> db2dg87 db2dg88 db2dg89 db2dg90 db2dg91 db2dg92 db2dg93 db2dg94

Add DCM logs to the volumes. Use the disk that’s been assigned only for logs.

sunsrv01-dr:# vxassist -g db2dg addlog bcp  logtype=dcm nlog=1 db2dg95
sunsrv01-dr:# vxassist -g db2dg addlog db   logtype=dcm nlog=1 db2dg95
sunsrv01-dr:# vxassist -g db2dg addlog dba  logtype=dcm nlog=1 db2dg95
sunsrv01-dr:# vxassist -g db2dg addlog db2  logtype=dcm nlog=1 db2dg95
sunsrv01-dr:# vxassist -g db2dg addlog lg1  logtype=dcm nlog=1 db2dg95
sunsrv01-dr:# vxassist -g db2dg addlog tp01 logtype=dcm nlog=1 db2dg95

Update the .rdg file on both nodes in the secondary site

The Secondary Node must be given permission to manage the disk group created on the Primary Node. To do this add the diskgroup ID into /etc/vx/vras/.rdg. The diskgroup ID is the value of dgid in the output of vxprint -l db2dg on the Primary Node.

sunsrv01:# vxprint -l db2dg | grep dgid
info:     dgid=1138140445.1393.sunsrv01 noautoimport

sunsrv01-dr:# echo “1138140445.1393.sunsrv01” > /etc/vx/vras/.rdg
sunsrv02-dr:# echo “1138140445.1393.sunsrv01” > /etc/vx/vras/.rdg

Stop all applications – both clustered and non-clustered apps

Stop all clustered and non-clustered applications that are using the volumes, which will be replicated.

Note: You may leave the applications running while replicating.

# su – db2inst –c “db2stop”

Unmount all filesystems whose underlying volumes will be replicated.

# umount ` mount | grep '/dev/vx/dsk/db2dg/' | awk '{ print $1 }'`

Forcibly stop VCS so it leaves all running resources online – especially Volumes and the diskgroup

# hastop –all -force

Bring up the virtual IPs on all nodes on both sites

These are the VVR Replication IPs (R-Link IPs). Must be unique – one at the primary site and other at the secondary site.

Primary (sunsrv01 or sunsrv02):

ce3:1 - 10.11.196.191 netmask fffffe00 broadcast 10.11.197.255

Secondary (sunsrv01-dr or sunsrv02-dr):

ce5:1 - 10.12.94.191 netmask fffffe00 broadcast 10.12.95.255

Create the SRL Volume in the primary site

Allocate disks that are dedicated just for SRL volume. No other volumes should use those disks.

sunsrv01:# vxassist -g db2dg make db2dg_srl 856350720 layout=concat \
> db2dg71 db2dg72 db2dg73 db2dg74 db2dg75 db2dg76 db2dg77 db2dg78 \
> db2dg79 db2dg80 db2dg81 db2dg82 db2dg83 db2dg84 db2dg85 db2dg86 \
> db2dg87 db2dg88 db2dg89 db2dg90 db2dg91 db2dg92 db2dg93 db2dg94



Add DCM logs to all the primary site volumes that will be replicated

Use the disk that is dedicated for logs only.

sunsrv01:# vxassist -g db2dg addlog bcp  logtype=dcm nlog=1 db2dg95
sunsrv01:# vxassist -g db2dg addlog db   logtype=dcm nlog=1 db2dg95
sunsrv01:# vxassist -g db2dg addlog dba  logtype=dcm nlog=1 db2dg95
sunsrv01:# vxassist -g db2dg addlog db2  logtype=dcm nlog=1 db2dg95
sunsrv01:# vxassist -g db2dg addlog lg1  logtype=dcm nlog=1 db2dg95
sunsrv01:# vxassist -g db2dg addlog tp01 logtype=dcm nlog=1 db2dg95


Update the .rdg file on both nodes in the primary site

The diskgroup ID is the value of dgid in the output of vxprint -l db2dg on the Secondary Node.

sunsrv02-dr:# vxprint -l db2dg | grep dgid
info:     dgid=1182230506.2373.sunsrv01-dr noautoimport 

sunsrv01:# echo “1182230506.2373.sunsrv01-dr” > /etc/vx/vras/.rdg
sunsrv02:# echo “1182230506.2373.sunsrv01-dr” > /etc/vx/vras/.rdg


Ensure that VVRTypes.cf exists in /etc/VRTSvcs/conf

This types file must exist on all nodes in the cluster.

sunsrv01:# ls -l /etc/VRTSvcs/conf/VVRTypes.cf
-rw-rw-r--   1 root     sys    1811 Jan 20  2005 /etc/VRTSvcs/conf/VVRTypes.cf


Ensure that other application types i.e. Db2udbTypes.cf exists in /etc/VRTSvcs/conf

This types file must exist on all nodes in the cluster.

sunsrv01:# ls -l /etc/VRTSvcs/conf/Db2udbTypes.cf
-rw-rw-r--   1 root     sys    1080 Jun 26  2006 /etc/VRTSvcs/conf/Db2udbTypes.cf


Create the Primary RVG

First, ensure that /usr/sbin/vradmind is running.  If not, start it with this command:

sunsrv01:# /etc/init.d/vras-vradmind.sh start


Run this from the primary site.

sunsrv01:# vradmin –g db2dg createpri db2dg_rvg bcp,db,dba,db2,lg1,tp01 \
           db2dg_srl

Usage:

vradmin -g <diskgroup> createpri <RVGname> <vol1,vol2...> <SRLname>

Where:

<diskgroup> is the VxVM diskgroup name
<RVGname> is the name for the RVG, usually <diskgroupname>_rvg
<vol1,vol2...> is a comma separated list of all volumes in the diskgroup to be replicated.
<SRLname> is the name of the SRL volume


Create the Secondary RVG

After creating the Primary RVG, go on to adding a Secondary. Use the vradmin addsec command to add a Secondary RVG.  The VIP hostnames are in the /etc/hosts file.

sunsrv01:# vradmin -g db2dg addsec db2dg_rvg db2inste3-vipvr db2instdr-vipvr


Usage:

vradmin -g <diskgroup> addsec <RVGname> <primaryhost> <secondaryhost>

Where:

<diskgroup> is the VxVM diskgroup name
<RVGname> is the name of the RVG created on the Primary Node
<primaryhost> is the hostname of the Primary Node, this could be a VCS ServiceGroup name
<secondaryhost> is the hostname of the Secondary Node, this could be a VCS ServiceGroup name


Note:  The vradmin addsec command performs the following operations:
Creates and adds a Secondary RVG of the same name as the Primary RVG to the specified RDS on the Secondary host. By default, the Secondary RVG is added to the disk group with the same name as the Primary disk group. Use the option -sdg with the vradminaddsec command to specify a different disk group on the Secondary.
Automatically adds DCMs to the Primary and Secondary data volumes if they do not have DCMs.
Associates to the Secondary RVG, existing data volumes of the same names and sizes as the Primary data volumes; it also associates an existing volume with the same name as the Primary SRL, as the Secondary SRL.
Creates and associates to the Primary and Secondary RVGs respectively, the Primary and Secondary RLINKs with default RLINK names rlk_remotehost_rvgname. 


Configure VCS to integrate VVR objects – Primary Site

There would be at least two VCS service Groups. One group represents replication group and the other group is the application (DB) group. Application group contains RVGPrimary resource. Replication group contains IP, RVG and the DiskGroup resource. 

In this particular environment, VVR Service Group and Application Service Group must be online on the same node because the diskgroup is on the VVR Service Group.

VVR Service group must be started first before the Application Service Group.  The dependencies are configured in the VCS configuration file.

VVR Service group maintains the synchronization between the primary and secondary sites.

VCS Service Groups:

db2inst_vvr VVR Service Group
Resources:

db2dgAgent (RVG)
vvrnic (NIC)
vvrip (VIP for VVR communications)
db2dg_dg (Diskgroup)

db2inst_grp Application Service Group

Resources:

_mnt (Filesystems)
adsm_db2inst (TSM Backup)

Primary Site VCS Configuration File:

include "types.cf"
include "Db2udbTypes.cf"
include "VVRTypes.cf"

cluster e3clus177 (
        UserNames = { admin = ajkCjeJgkFkkIskEjh }
        ClusterAddress = "10.10.231.191"
        Administrators = { admin }
        CredRenewFrequency = 0
        CounterInterval = 5
        )

system sunsrv01 (
        )

system sunsrv02 (
        )

group db2inst_grp (
        SystemList = { sunsrv01 = 0, sunsrv02 = 1 }
        AutoStart = 0
        )

        Application adsm_db2inst (
                Critical = 0
                User = root
                StartProgram = "/bcp/db2inst/tsm/adsmcad.db start"
                StopProgram = "/bcp/db2inst/tsm/adsmcad.db stop"
                MonitorProcesses = {
                         "/usr/bin/dsmcad -optfile=/bcp/db2inst/tsm/dsm.opt.db2inst" }
                )

        Db2udb db2inst_db (
                Critical = 0
                DB2InstOwner = db2inst
                DB2InstHome = "/db2/db2inst"
                MonScript = "/opt/VRTSvcs/bin/Db2udb/SqlTest.pl"
                )

        DiskGroup db2dgB_dg (
                DiskGroup = db2dgB
                )

        IP db2inst_ip (
                Device = ce1
                Address = "10.10.231.191"
                )

        Mount db2dgB_bkup_mnt (
                MountPoint = "/backup/db2inst"
                BlockDevice = "/dev/vx/dsk/db2dgB/bkp"
                FSType = vxfs
                FsckOpt = "-y"
                )

        Mount db2inst_bcp_mnt (
                MountPoint = "/bcp/db2inst"
                BlockDevice = "/dev/vx/dsk/db2dg/bcp"
                FSType = vxfs
                MountOpt = rw
                FsckOpt = "-y"
                )

        Mount db2inst_db2_mnt (
                MountPoint = "/db2/db2inst"
                BlockDevice = "/dev/vx/dsk/db2dg/db2"
                FSType = vxfs
                MountOpt = rw
                FsckOpt = "-y"
                )

        Mount db2inst_db_mnt (
                MountPoint = "/db/db2inst/PEMMP00P/NODE0000"
                BlockDevice = "/dev/vx/dsk/db2dg/db"
                FSType = vxfs
                MountOpt = rw
                FsckOpt = "-y"
                )

        Mount db2inst_dba_mnt (
                MountPoint = "/dba/db2inst"
                BlockDevice = "/dev/vx/dsk/db2dg/dba"
                FSType = vxfs
                MountOpt = rw
                FsckOpt = "-y"
                )

        Mount db2inst_lg1_mnt (
                MountPoint = "/db/db2inst/log1"
                BlockDevice = "/dev/vx/dsk/db2dg/lg1"
                FSType = vxfs
                MountOpt = rw
                FsckOpt = "-y"
                )

        Mount db2inst_tp01_mnt (
                MountPoint = "/db/db2inst/PEMMP00P/tempspace01/NODE0000"
                BlockDevice = "/dev/vx/dsk/db2dg/tp01"
                FSType = vxfs
                MountOpt = rw
                FsckOpt = "-y"
                )

        NIC db2inst_ce1 (
                Device = ce1
                NetworkType = ether
                )

        RVGPrimary db2dg_rvg_primary (
                Critical = 0
                RvgResourceName = db2dgAgent
                )

        requires group db2inst_vvr online local hard
        adsm_db2inst requires db2inst_bcp_mnt
        db2dgB_bkup_mnt requires db2dgB_dg
        db2inst_bcp_mnt requires db2dg_rvg_primary
        db2inst_db requires db2inst_bcp_mnt
        db2inst_db requires db2inst_db2_mnt
        db2inst_db requires db2inst_db_mnt
        db2inst_db requires db2inst_dba_mnt
        db2inst_db requires db2inst_ip
        db2inst_db requires db2inst_lg1_mnt
        db2inst_db requires db2inst_tp01_mnt
        db2inst_db2_mnt requires db2dg_rvg_primary
        db2inst_db_mnt requires db2dg_rvg_primary
        db2inst_dba_mnt requires db2dg_rvg_primary
        db2inst_ip requires db2inst_ce1
        db2inst_lg1_mnt requires db2dg_rvg_primary
        db2inst_tp01_mnt requires db2dg_rvg_primary



group db2inst_vvr (
        SystemList = { sunsrv01 = 0, sunsrv02 = 1 }
        )

        DiskGroup db2dg_dg (
                DiskGroup = db2dg
                )

        IP vvrip (
                Device = ce3
                Address = "10.11.196.191"
                NetMask = "255.255.254.0"
                )

        NIC vvrnic (
                Device = ce3
                NetworkType = ether
                )

        RVG db2dgAgent (
                Critical = 0
                RVG = db2dg_rvg
                DiskGroup = db2dg
                )

        db2dgAgent requires db2dg_dg
     vvrip requires vvrnic

Configure VCS to integrate VVR objects – Secondary Site

In this particular environment, there will be four VCS service Groups – CCA Group, Replication Group, Application (DB) Group, and SNAP Group. CCA group is for remote cluster administration. Application group contains RVGPrimary resource. Replication group contains IP, RVG and the DiskGroup resource. SNAP Group contains the resources for SNAP copy. This document will not discuss Veritas CCA.

In this particular environment, VVR Service Group, Application Service Group, and SNAP Service Group must be online on the same node because the diskgroup is on the VVR Service Group. SNAP volumes are on the same diskgroup as the application/DB volumes.

SNAP Configuration is discussed later in this document.

VVR Service group must be started first before the Application Service Group. The dependencies are configured in the VCS configuration file.

VVR Service group maintains the synchronization between the primary and secondary sites.

VCS Service Groups:

db2inst_vvr VVR Service Group
Resources:
· db2dgAgent (RVG)
· vvrnic (NIC)
· vvrip (VIP for VVR communications)
· db2dg_dg (Diskgroup)

db2inst_grp Application Service Group
Resources:
· db2dg_rvg_primary (RVG Primary)
· db2inst_db (DB2 Database)
· db2inst_ce1 (NIC)
· db2inst_ip (VIP for the application)
· db2inst__mnt (Filesystems)
· adsm_db2inst (TSM Backup)

snap_db2inst_grp Snap Copy Service Group
Resources:
· snap_db2inst_db (DB2 Database)
· snap_db2inst_ce1 (NIC)
· snap_db2inst_ip (VIP for the application)
· snap_db2inst__mnt (Filesystems)

Secondary Site VCS Configuration File:

include "types.cf"
include "ClusterMonitorConfigType.cf"
include "Db2udbTypes.cf"
include "VVRTypes.cf"

cluster e3clus177 (
        UserNames = { admin = ajkCjeJgkFkkIskEjh }
        ClusterAddress = "10.10.231.191"
        Administrators = { admin }
        CredRenewFrequency = 0
        CounterInterval = 5
        )

system sunsrv01-dr (
        )

system sunsrv02-dr (
        )

group CCAvail (
        SystemList = { sunsrv01-dr = 0, sunsrv02-dr = 1 }
        AutoStartList = { sunsrv01-dr, sunsrv02-dr }
        )

        ClusterMonitorConfig CCAvail_ClusterConfig (
                MSAddress = "10.11.198.53"
                ClusterId = 1183482234
                VCSLoggingLevel = TAG_A
                Logging = "/opt/VRTSccacm/conf/k2_logging.properties"
                ClusterMonitorVersion = "4.1.2272.1"
                )

        Process CCAvail_ClusterMonitor (
                PathName = "/opt/VRTSccacm/bin/ClusterMonitor"
                Arguments = "-config"
                )

        CCAvail_ClusterMonitor requires CCAvail_ClusterConfig


group db2inst_grp (
        SystemList = { sunsrv01-dr = 0, sunsrv02-dr = 1 }
        Enabled @sunsrv01-dr = 0
        Enabled @sunsrv02-dr = 0
        AutoStart = 0
        )

        Application adsm_db2inst (
                Enabled = 0
                Critical = 0
                User = root
                StartProgram = "/bcp/db2inst/tsm/adsmcad.db start"
                StopProgram = "/bcp/db2inst/tsm/adsmcad.db stop"
                MonitorProcesses = {
                         "/usr/bin/dsmcad -optfile=/bcp/db2inst/tsm/dsm.opt.db2inst" }
                )

        Db2udb db2inst_db (
                Enabled = 0
                Critical = 0
                DB2InstOwner = db2inst
                DB2InstHome = "/db2/db2inst"
                MonScript = "/opt/VRTSvcs/bin/Db2udb/SqlTest.pl"
                )

        DiskGroup db2dgB_dg (
                Enabled = 0
                DiskGroup = db2dgB
                )

        IP db2inst_ip (
                Enabled = 0
                Device = ce1
                Address = "10.10.231.191"
                )

        Mount db2dgB_bkup_mnt (
                Enabled = 0
                MountPoint = "/backup/db2inst"
                BlockDevice = "/dev/vx/dsk/db2dgB/bkp"
                FSType = vxfs
                FsckOpt = "-y"
                )

        Mount db2inst_bcp_mnt (
                Enabled = 0
                MountPoint = "/bcp/db2inst"
                BlockDevice = "/dev/vx/dsk/db2dg/bcp"
                FSType = vxfs
                MountOpt = rw
                FsckOpt = "-y"
                )

        Mount db2inst_db2_mnt (
                Enabled = 0
                MountPoint = "/db2/db2inst"
                BlockDevice = "/dev/vx/dsk/db2dg/db2"
                FSType = vxfs
                MountOpt = rw
                FsckOpt = "-y"
                )

        Mount db2inst_db_mnt (
                Enabled = 0
                MountPoint = "/db/db2inst/PEMMP00P/NODE0000"
                BlockDevice = "/dev/vx/dsk/db2dg/db"
                FSType = vxfs
                MountOpt = rw
                FsckOpt = "-y"
                )

        Mount db2inst_dba_mnt (
                Enabled = 0
                MountPoint = "/dba/db2inst"
                BlockDevice = "/dev/vx/dsk/db2dg/dba"
                FSType = vxfs
                MountOpt = rw
                FsckOpt = "-y"
                )

        Mount db2inst_lg1_mnt (
                Enabled = 0
                MountPoint = "/db/db2inst/log1"
                BlockDevice = "/dev/vx/dsk/db2dg/lg1"
                FSType = vxfs
                MountOpt = rw
                FsckOpt = "-y"
                )

        Mount db2inst_tp01_mnt (
                Enabled = 0
                MountPoint = "/db/db2inst/PEMMP00P/tempspace01/NODE0000"
                BlockDevice = "/dev/vx/dsk/db2dg/tp01"
                FSType = vxfs
                MountOpt = rw
                FsckOpt = "-y"
                )

        NIC db2inst_ce1 (
                Enabled = 0
                Device = ce1
                NetworkType = ether
                )

        RVGPrimary db2dg_rvg_primary (
                Enabled = 0
                Critical = 0
                RvgResourceName = db2dgAgent
                )

        requires group db2inst_vvr online local firm
        adsm_db2inst requires db2inst_bcp_mnt
        db2dgB_bkup_mnt requires db2dgB_dg
        db2inst_bcp_mnt requires db2dg_rvg_primary
        db2inst_db requires db2inst_bcp_mnt
        db2inst_db requires db2inst_db2_mnt
        db2inst_db requires db2inst_db_mnt
        db2inst_db requires db2inst_dba_mnt
        db2inst_db requires db2inst_ip
        db2inst_db requires db2inst_lg1_mnt
        db2inst_db requires db2inst_tp01_mnt
        db2inst_db2_mnt requires db2dg_rvg_primary
        db2inst_db_mnt requires db2dg_rvg_primary
        db2inst_dba_mnt requires db2dg_rvg_primary
        db2inst_ip requires db2inst_ce1
        db2inst_lg1_mnt requires db2dg_rvg_primary
        db2inst_tp01_mnt requires db2dg_rvg_primary


group db2inst_vvr (
        SystemList = { sunsrv01-dr = 0, sunsrv02-dr = 1 }
        )

        DiskGroup db2dg_dg (
                DiskGroup = db2dg
                )

        IP vvrip (
                Device = ce5
                Address = "10.12.94.191"
                NetMask = "255.255.254.0"
                )

        NIC vvrnic (
                Device = ce5
                NetworkType = ether
                )

        RVG db2dgAgent (
                Critical = 0
                RVG = db2dg_rvg
                DiskGroup = db2dg
                )

        db2dgAgent requires db2dg_dg
        vvrip requires vvrnic


group snap_db2inst_grp (
        SystemList = { sunsrv01-dr = 0, sunsrv02-dr = 1 }
        AutoStart = 0
        )

        Db2udb snap_db2inst_db (
                Critical = 0
                DB2InstOwner = db2inst
                DB2InstHome = "/db2/db2inst"
                MonScript = "/opt/VRTSvcs/bin/Db2udb/SqlTest.pl"
                )

        DiskGroup snap_db2dgB_dg (
                DiskGroup = db2dgB
                )

        IP snap_db2inst_ip (
                Device = ce1
                Address = "10.10.231.191"
                )

        Mount snap_db2dgB_bkup_mnt (
                MountPoint = "/backup/db2inst"
                BlockDevice = "/dev/vx/dsk/db2dgB/bkp"
                FSType = vxfs
                FsckOpt = "-y"
                )

        Mount snap_db2inst_bcp_mnt (
                MountPoint = "/bcp/db2inst"
                BlockDevice = "/dev/vx/dsk/db2dg/bcp_snapvol"
                FSType = vxfs
                FsckOpt = "-y"
                )

        Mount snap_db2inst_db2_mnt (
                MountPoint = "/db2/db2inst"
                BlockDevice = "/dev/vx/dsk/db2dg/db2_snapvol"
                FSType = vxfs
                FsckOpt = "-y"
                )

        Mount snap_db2inst_db_mnt (
                MountPoint = "/db/db2inst/PEMMP00P/NODE0000"
                BlockDevice = "/dev/vx/dsk/db2dg/db_snapvol"
                FSType = vxfs
                FsckOpt = "-y"
                )

        Mount snap_db2inst_dba_mnt (
                MountPoint = "/dba/db2inst"
                BlockDevice = "/dev/vx/dsk/db2dg/dba_snapvol"
                FSType = vxfs
                FsckOpt = "-y"
                )

        Mount snap_db2inst_lg1_mnt (
                MountPoint = "/db/db2inst/log1"
                BlockDevice = "/dev/vx/dsk/db2dg/lg1_snapvol"
                FSType = vxfs
                FsckOpt = "-y"
                )

        Mount snap_db2inst_tp01_mnt (
                MountPoint = "/db/db2inst/PEMMP00P/tempspace01/NODE0000"
                BlockDevice = "/dev/vx/dsk/db2dg/tp01_snapvol"
                FSType = vxfs
                FsckOpt = "-y"
                )

        NIC snap_db2inst_ce1 (
                Device = ce1
                NetworkType = ether
                )

        requires group db2inst_vvr online local firm
        snap_db2dgB_bkup_mnt requires snap_db2dgB_dg
        snap_db2inst_db requires snap_db2inst_bcp_mnt
        snap_db2inst_db requires snap_db2inst_db2_mnt
        snap_db2inst_db requires snap_db2inst_db_mnt
        snap_db2inst_db requires snap_db2inst_dba_mnt
        snap_db2inst_db requires snap_db2inst_lg1_mnt
        snap_db2inst_db requires snap_db2inst_tp01_mnt
     snap_db2inst_ip requires snap_db2inst_ce1

Bring up VCS engine and bring up the vvr service group in the Secondary Site

Start VCS on both nodes

sunsrv01-dr:# hastart
sunsrv02-dr:# hastart

Bring up the VVR Service Group on one node

sunsrv01-dr:# hagrp –online db2inst_vvr –sys sunsrv01-dr

Bring up VCS engine and bring up the vvr service group in the Primary Site

Start VCS on both nodes

sunsrv01:# hastart
sunsrv02:# hastart

Bring up the VVR Service Group on one node

sunsrv01:# hagrp –online db2inst_vvr –sys sunsrv01

Bring up the Application Service Group

sunsrv01:# hagrp –online db2inst_grp –sys sunsrv01

Check the Rlink and RVG status

Make sure that the flags are attached and connected. If for some reason links were detached or disconnected, check to make sure that the communications between the primary site and the secondary site are working fine. You should be able to ping, ssh, or telnet from each site thru the VIPs. If communications are good, you may restart VVR (see next step for restarting VVR engine).

sunsrv01:# vxprint -Pl
Disk group: db2dg

Rlink:    rlk_db2instdr-vipvr_db2dg_rvg
info:     timeout=500 packet_size=8400 rid=0.2007
          latency_high_mark=10000 latency_low_mark=9950
          bandwidth_limit=none
state:    state=ACTIVE
          synchronous=off latencyprot=off srlprot=autodcm
assoc:    rvg=db2dg_rvg
          remote_host=db2instdr-vipvr IP_addr=10.12.94.191 port=4145
          remote_dg=db2dg
          remote_dg_dgid=1182230506.2373.sunsrv01-dr
          remote_rvg_version=21
          remote_rlink=rlk_db2inste3-vipvr_db2dg_rvg
          remote_rlink_rid=0.2127
          local_host=db2inste3-vipvr IP_addr=10.11.196.191 port=4145
protocol: UDP/IP
flags:    write enabled attached consistent cant_sync connected asynchronous autosync

sunsrv01:# vradmin -l printrvg
Replicated Data Set: db2dg_rvg
Primary:
        HostName: db2inste3-vipvr       
        RvgName: db2dg_rvg
        DgName: db2dg
        datavol_cnt: 6
        srl: db2dg_srl
        RLinks:
            name=rlk_db2instdr-vipvr_db2dg_rvg, detached=off, synchronous=off
Secondary:
        HostName: db2instdr-vipvr
        RvgName: db2dg_rvg
        DgName: db2dg
        datavol_cnt: 6
        srl: db2dg_srl
        RLinks:
            name=rlk_db2inste3-vipvr_db2dg_rvg, detached=off, synchronous=off

Restart VVR engine (if the link is detached or disconnected)

This is only required if you see the links between primary and secondary as detached or disconnected. If you think that the communication between the two is working fine, then run the following commands in the primary site:

sunsrv01:# vxstart_vvr stop
sunsrv01:# vxstart_vvr start

Then, check the status again.

sunsrv01:# vxprint -Pl

Start the replication

Initiate the command from the primary site.

sunsrv01:# vradmin -g db2dg -a startrep db2dg_rvg db2instdr-vipvr
Message from Primary:
VxVM VVR vxrlink WARNING V-5-1-3359 Attaching rlink to non-empty rvg.
                                    Autosync will be performed.

VxVM VVR vxrlink INFO V-5-1-3614 Secondary data volumes detected with rvg db2dg_rvg as parent:

VxVM VVR vxrlink INFO V-5-1-6183 bcp:    len=585105408     primary_datavol=bcp
VxVM VVR vxrlink INFO V-5-1-6183 db:     len=1172343808    primary_datavol=db
VxVM VVR vxrlink INFO V-5-1-6183 db2:    len=8388608       primary_datavol=db2
VxVM VVR vxrlink INFO V-5-1-6183 dba:    len=98566144      primary_datavol=dba
VxVM VVR vxrlink INFO V-5-1-6183 lg1:    len=396361728     primary_datavol=lg1
VxVM VVR vxrlink INFO V-5-1-6183 tp01:   len=192937984     primary_datavol=tp01

VxVM VVR vxrlink INFO V-5-1-3365 Autosync operation has started

Usage:

vradmin -g <diskgroup> -a startrep <RVGname> <secondaryhost>


Where:

<diskgroup> is the VxVM diskgroup name
<RVGname> is the name for the RVG, usually <diskgroupname>_rvg
<secondaryhost> is the hostname of the Secondary Site.  Check the /etc/hosts file.



Check the Replication status

Initiate the command from the primary site.  You can tell if it’s syncing by looking at the number of bytes remaining.  If it changes, then it’s working.

sunsrv01:# vxrlink -g db2dg -i 2 status rlk_db2instdr-vipvr_db2dg_rvg
Fri Jun 22 00:01:37 MST 2007
VxVM VVR vxrlink INFO V-5-1-4464 Rlink rlk_db2instdr-vipvr_db2dg_rvg is in AUTOSYNC. 1226649984 Kbytes remaining.
VxVM VVR vxrlink INFO V-5-1-4464 Rlink rlk_db2instdr-vipvr_db2dg_rvg is in AUTOSYNC. 1226644224 Kbytes remaining.
VxVM VVR vxrlink INFO V-5-1-4464 Rlink rlk_db2instdr-vipvr_db2dg_rvg is in AUTOSYNC. 1226638464 Kbytes remaining.
VxVM VVR vxrlink INFO V-5-1-4464 Rlink rlk_db2instdr-vipvr_db2dg_rvg is in AUTOSYNC. 1226632128 Kbytes remaining.
VxVM VVR vxrlink INFO V-5-1-4464 Rlink rlk_db2instdr-vipvr_db2dg_rvg is in AUTOSYNC. 1226626368 Kbytes remaining.


Another way to check is to use the vradmin repstatus command.  The key bits of information here are Data status and Replication status.

sunsrv01:# vradmin -g db2dg repstatus db2dg_rvg
Replicated Data Set: db2dg_rvg
Primary:
  Host name:                  db2inste3-vipvr
  RVG name:                   db2dg_rvg
  DG name:                    db2dg
  RVG state:                  enabled for I/O
  Data volumes:               6
  SRL name:                   db2dg_srl
  SRL size:                   952.84 G
  Total secondaries:          1

Secondary:
  Host name:                  db2instdr-vipvr
  RVG name:                   db2dg_rvg
  DG name:                    db2dg
  Data status:                consistent, up-to-date
  Replication status:         replicating (connected)
  Current mode:               asynchronous
  Logging to:                 SRL
  Timestamp Information:      N/A


Mount the Replicated Volumes

When replication is 100% complete, you may check the volumes on the secondary site by mounting the filesystems.  But first, you need to check the status of replication.

sunsrv01:# vxrlink -g db2dg -i 2 status rlk_db2instdr-vipvr_db2dg_rvgThu Jun 28 08:54:42 MST 2007
VxVM VVR vxrlink INFO V-5-1-4467 Rlink rlk_db2instdr-vipvr_db2dg_rvg is up to date


Use VCS to mount the replicated Volumes

sunsrv01-dr:# hagrp –online db2inst_grp –sys sunsrv01-dr

Display the filesystems and compare them with the primary.

sunsrv01-dr:# df -k | grep db2dg

/dev/vx/dsk/db2dg/db2     4194304   78059   3859030 2% /db2/db2inst
/dev/vx/dsk/db2dg/bcp   292552704 1527508 272836173 1% /bcp/db2inst
/dev/vx/dsk/db2dg/dba    49283072   28555  46176114 1% /dba/db2inst
/dev/vx/dsk/db2dg/db    586171904 1733538 547911037 1% /db/db2inst/PEMMP00P/NODE0000
/dev/vx/dsk/db2dg/tp01   96468992   40176  90402086 1% /db/db2inst/PEMMP00P/tempspace01/NODE0000
/dev/vx/dsk/db2dg/lg1   198180864 1779853 184126012 1% /db/db2inst/log1

sunsrv01:# df -k | grep db2dg

/dev/vx/dsk/db2dg/db2     4194304   78059   3859030 2% /db2/db2inst
/dev/vx/dsk/db2dg/bcp   292552704 1527508 272836173 1% /bcp/db2inst
/dev/vx/dsk/db2dg/dba    49283072   28555  46176114 1% /dba/db2inst
/dev/vx/dsk/db2dg/db    586171904 1733538 547911037 1% /db/db2inst/PEMMP00P/NODE0000
/dev/vx/dsk/db2dg/tp01   96468992   40176  90402086 1% /db/db2inst/PEMMP00P/tempspace01/NODE0000
/dev/vx/dsk/db2dg/lg1   198180864 1779853 184126012 1% /db/db2inst/log1

This is the best time to test VCS for switching the service group.

sunsrv01-dr:# hagrp –switch db2inst_grp –to sunsrv02-dr



Prepare the Replicated Volumes for Snapshot in the secondary site

First, display the volume information

sunsrv02-dr:# vxprint -ht bcp
Disk group: db2dg

v  bcp           db2dg_rvg   ENABLED  ACTIVE 585105408 SELECT    -        fsgen
pl bcp-01        bcp         ENABLED  ACTIVE 585108480 CONCAT    -        RW
sd db2dg214-01   bcp-03      db2dg214 5376   285473280 0         EMC0_219 ENA
sd db2dg215-01   bcp-03      db2dg215 5376   285473280 285473280 EMC0_220 ENA
sd db2dg219-01   bcp-03      db2dg219 5376   14161920  570946560 EMC0_224 ENA
pl bcp-02        bcp         ENABLED  ACTIVE LOGONLY   CONCAT    -        RW
sd db2dg95-06    bcp-02      db2dg95  0      512       LOG       EMC0_154 ENA

Prepare the volume for snapshot by adding a DCO log.  The same disks is used for DCO and DCM logs.

sunsrv02-dr:# vxsnap -g db2dg prepare bcp ndcomirs=1 alloc=db2dg95
VxVM vxsnap INFO V-5-1-9270 Volume is under RVG, setting drl=no.

Display the volume information to see the changes.  Note that the DCO volume has been added with two logs.

sunsrv02-dr:# vxprint -ht bcp
Disk group: db2dg

v  bcp           db2dg_rvg   ENABLED  ACTIVE 585105408 SELECT    -        fsgen
pl bcp-01        bcp         ENABLED  ACTIVE 585108480 CONCAT    -        RW
sd db2dg214-01   bcp-03      db2dg214 5376   285473280 0         EMC0_219 ENA
sd db2dg215-01   bcp-03      db2dg215 5376   285473280 285473280 EMC0_220 ENA
sd db2dg219-01   bcp-03      db2dg219 5376   14161920  570946560 EMC0_224 ENA
pl bcp-02        bcp         ENABLED  ACTIVE LOGONLY   CONCAT    -        RW
sd db2dg95-06    bcp-02      db2dg95  0      512       LOG       EMC0_154 ENA
dc bcp_dco       bcp         bcp_dcl     
v  bcp_dcl       -           ENABLED  ACTIVE 40368     SELECT    -        gen
pl bcp_dcl-01    bcp_dcl     ENABLED  ACTIVE 40368     CONCAT    -        RW
sd db2dg95-07    bcp_dcl-01  db2dg95  960    40368     0         EMC0_154 ENA

Run the same steps on the rest of the volumes

sunsrv02-dr:# vxsnap -g db2dg prepare db ndcomirs=1 alloc=db2dg95
VxVM vxsnap INFO V-5-1-9270 Volume is under RVG, setting drl=no.

sunsrv02-dr:# vxsnap -g db2dg prepare dba ndcomirs=1 alloc=db2dg95
VxVM vxsnap INFO V-5-1-9270 Volume is under RVG, setting drl=no.

sunsrv02-dr:# vxsnap -g db2dg prepare db2 ndcomirs=1 alloc=db2dg95
VxVM vxsnap INFO V-5-1-9270 Volume is under RVG, setting drl=no.

sunsrv02-dr:# vxsnap -g db2dg prepare lg1 ndcomirs=1 alloc=db2dg95
VxVM vxsnap INFO V-5-1-9270 Volume is under RVG, setting drl=no.

sunsrv02-dr:# vxsnap -g db2dg prepare tp01 ndcomirs=1 alloc=db2dg95
VxVM vxsnap INFO V-5-1-9270 Volume is under RVG, setting drl=no.


Create the SNAP Volumes in the secondary site

sunsrv02-dr:# vxassist -g db2dg make bcp_snapvol  585105408 layout=concat
sunsrv02-dr:# vxassist -g db2dg make db_snapvol  1172343808 layout=concat
sunsrv02-dr:# vxassist -g db2dg make dba_snapvol   98566144 layout=concat
sunsrv02-dr:# vxassist -g db2dg make db2_snapvol    8388608 layout=concat
sunsrv02-dr:# vxassist -g db2dg make db2_snapvol    8388608 layout=concat
sunsrv02-dr:# vxassist -g db2dg make lg1_snapvol  396361728 layout=concat
sunsrv02-dr:# vxassist -g db2dg make tp01_snapvol 192937984 layout=concat


Prepare the SNAP Volumes

Identify the region size of the main volumes.  The region size must be the same for both the main volume and the snap volume.

sunsrv02-dr:# for DCONAME in `vxprint -tg db2dg | grep "^dc" | awk '{ print $2 }'`
> do
>   echo "$DCONAME\t\c"
>   vxprint -g db2dg -F%regionsz $DCONAME
> done
bcp_dco   128
db_dco    128
dba_dco   128
db2_dco   128
lg1_dco   128
tp01_dco  128


Display the volume information

sunsrv02-dr:# vxprint -ht bcp_snapvol
Disk group: db2dg

v  bcp_snapvol  fsgen        ENABLED  585105408 -       ACTIVE   -       -
pl bcp_snapvol-01 bcp_snapvol ENABLED 585105600 -       ACTIVE   -       -
sd db2dg114-01 bcp_snapvol-01 ENABLED 35726400 0      -        -       -
sd db2dg106-01 bcp_snapvol-01 ENABLED 35681280 35726400 -      -       -
sd db2dg101-01 bcp_snapvol-01 ENABLED 35681280 71407680 -      -       -
sd db2dg102-01 bcp_snapvol-01 ENABLED 35681280 107088960 -     -       -
sd db2dg103-01 bcp_snapvol-01 ENABLED 35681280 142770240 -     -       -
sd db2dg104-01 bcp_snapvol-01 ENABLED 35681280 178451520 -     -       -
sd db2dg105-01 bcp_snapvol-01 ENABLED 35681280 214132800 -     -       -
sd db2dg107-01 bcp_snapvol-01 ENABLED 35681280 249814080 -     -       -
sd db2dg108-01 bcp_snapvol-01 ENABLED 35681280 285495360 -     -       -
sd db2dg109-01 bcp_snapvol-01 ENABLED 13844160 321176640 -     -       -
sd db2dg110-01 bcp_snapvol-01 ENABLED 35726400 335020800 -     -       -
sd db2dg111-01 bcp_snapvol-01 ENABLED 35726400 370747200 -     -       -
sd db2dg112-01 bcp_snapvol-01 ENABLED 35726400 406473600 -     -       -
sd db2dg113-01 bcp_snapvol-01 ENABLED 35726400 442200000 -     -       -
sd db2dg115-01 bcp_snapvol-01 ENABLED 35726400 477926400 -     -       -
sd db2dg116-01 bcp_snapvol-01 ENABLED 35726400 513652800 -     -       -
sd db2dg117-01 bcp_snapvol-01 ENABLED 35726400 549379200 -     -       -

Prepare the snapshot volume by adding a DCO log and the same region size as the main volume.  The same disks is used for DCO and DCM logs.

sunsrv02-dr:# vxsnap -g db2dg prepare bcp_snapvol ndcomirs=1 regionsize=128 \
              alloc=db2dg95

Display the volume information to see the changes.  Note that the DCO volume has been added.

sunsrv02-dr:# vxprint -ht bcp_snapvol
Disk group: db2dg

v  bcp_snapvol  fsgen        ENABLED  585105408 -       ACTIVE   -       -
pl bcp_snapvol-01 bcp_snapvol ENABLED 585105600 -       ACTIVE   -       -
sd db2dg114-01 bcp_snapvol-01 ENABLED 35726400 0      -        -       -
sd db2dg106-01 bcp_snapvol-01 ENABLED 35681280 35726400 -      -       -
sd db2dg101-01 bcp_snapvol-01 ENABLED 35681280 71407680 -      -       -
sd db2dg102-01 bcp_snapvol-01 ENABLED 35681280 107088960 -     -       -
sd db2dg103-01 bcp_snapvol-01 ENABLED 35681280 142770240 -     -       -
sd db2dg104-01 bcp_snapvol-01 ENABLED 35681280 178451520 -     -       -
sd db2dg105-01 bcp_snapvol-01 ENABLED 35681280 214132800 -     -       -
sd db2dg107-01 bcp_snapvol-01 ENABLED 35681280 249814080 -     -       -
sd db2dg108-01 bcp_snapvol-01 ENABLED 35681280 285495360 -     -       -
sd db2dg109-01 bcp_snapvol-01 ENABLED 13844160 321176640 -     -       -
sd db2dg110-01 bcp_snapvol-01 ENABLED 35726400 335020800 -     -       -
sd db2dg111-01 bcp_snapvol-01 ENABLED 35726400 370747200 -     -       -
sd db2dg112-01 bcp_snapvol-01 ENABLED 35726400 406473600 -     -       -
sd db2dg113-01 bcp_snapvol-01 ENABLED 35726400 442200000 -     -       -
sd db2dg115-01 bcp_snapvol-01 ENABLED 35726400 477926400 -     -       -
sd db2dg116-01 bcp_snapvol-01 ENABLED 35726400 513652800 -     -       -
sd db2dg117-01 bcp_snapvol-01 ENABLED 35726400 549379200 -     -       -
dc bcp_snapvol_dco bcp_snapvol -      -        -        -        -       -
v  bcp_snapvol_dcl gen       ENABLED  40368    -        ACTIVE   -       -
pl bcp_snapvol_dcl-01 bcp_snapvol_dcl ENABLED 40368 -   ACTIVE   -       -
sd db2dg95-01 bcp_snapvol_dcl-01 ENABLED 40368 0     -        -       -


Run the same steps on the rest of the volumes

sunsrv02-dr:# vxsnap -g db2dg prepare db_snapvol ndcomirs=1 regionsize=128 \
              alloc=db2dg95
sunsrv02-dr:# vxsnap -g db2dg prepare dba_snapvol ndcomirs=1 regionsize=128 \
              alloc=db2dg95
sunsrv02-dr:# vxsnap -g db2dg prepare db2_snapvol ndcomirs=1 regionsize=128 \
              alloc=db2dg95
sunsrv02-dr:# vxsnap -g db2dg prepare lg1_snapvol ndcomirs=1 regionsize=128 \
              alloc=db2dg95
sunsrv02-dr:# vxsnap -g db2dg prepare tp01_snapvol ndcomirs=1 regionsize=128 \
              alloc=db2dg95

Verify the region sizes are the same

sunsrv02-dr:# for DCONAME in `vxprint -tg db2dg | grep "^dc" | awk '{ print $2 }'`
> do
>   echo "$DCONAME\t\c"
>   vxprint -g db2dg -F%regionsz $DCONAME
> done
bcp_dco           128
bcp_snapvol_dco   128
db_dco            128
db_snapvol_dco    128
dba_dco           128
dba_snapvol_dco   128
db2_dco           128
db2_snapvol_dco   128
lg1_dco           128
lg1_snapvol_dco   128
tp01_dco          128
tp01_snapvol_dco  128



Run a point-in-time snapshot

Verify the RLINK and RVG are active and up to date.  Run this command from the primary site.

sunsrv01:# vxrlink -g db2dg -i 2 status rlk_db2instdr-vipvr_db2dg_rvg
Thu Jun 28 08:54:42 MST 2007
VxVM VVR vxrlink INFO V-5-1-4467 Rlink rlk_db2instdr-vipvr_db2dg_rvg is up to date

sunsrv02-dr:# vxprint -Pl
Disk group: db2dg

Rlink:    rlk_db2inste3-vipvr_db2dg_rvg
info:     timeout=500 packet_size=8400 rid=0.2127
          latency_high_mark=10000 latency_low_mark=9950
          bandwidth_limit=none
state:    state=ACTIVE
          synchronous=off latencyprot=off srlprot=autodcm
assoc:    rvg=db2dg_rvg
          remote_host=db2inste3-vipvr IP_addr=10.11.196.191 port=4145
          remote_dg=db2dg
          remote_dg_dgid=1138140445.1393.sunsrv01
          remote_rvg_version=21
          remote_rlink=rlk_db2instdr-vipvr_db2dg_rvg
          remote_rlink_rid=0.2007
          local_host=db2instdr-vipvr IP_addr=10.12.94.191 port=4145
protocol: UDP/IP
flags:    write enabled attached consistent connected

Start the Snap Copy

sunsrv02-dr:# vxsnap -g db2dg make source=bcp/snapvol=bcp_snapvol \
              source=db/snapvol=db_snapvol source=dba/snapvol=dba_snapvol \
              source=db2/snapvol=db2_snapvol source=lg1/snapvol=lg1_snapvol \
              source=tp01/snapvol=tp01_snapvol

Display the sync status

sunsrv02-dr:# vxtask list
TASKID  PTID TYPE/STATE    PCT   PROGRESS
   168         SNAPSYNC/R 00.07% 0/585105408/430080  SNAPSYNC bcp_snapvol  db2dg
   170         SNAPSYNC/R 00.05% 0/1172343808/567296 SNAPSYNC db_snapvol   db2dg
   172         SNAPSYNC/R 00.46% 0/98566144/452608   SNAPSYNC dba_snapvol  db2dg
   174         SNAPSYNC/R 04.30% 0/8388608/360448    SNAPSYNC db2_snapvol  db2dg
   176         SNAPSYNC/R 00.16% 0/396361728/641024  SNAPSYNC lg1_snapvol  db2dg
   178         SNAPSYNC/R 00.18% 0/192937984/346112  SNAPSYNC tp01_snapvol db2dg

sunsrv02-dr:# vxsnap -g db2dg print

NAME          SNAPOBJECT        TYPE     PARENT   SNAPSHOT    %DIRTY    %VALID

bcp           --                volume   --       --           --       100.00
              bcp_snapvol_snp   volume   --       bcp_snapvol  0.00     --

db            --                volume   --       --           --       100.00
              db_snapvol_snp    volume   --       db_snapvol   0.00     --

dba           --                volume   --       --           --       100.00
              dba_snapvol_snp   volume   --       dba_snapvol  0.00     --

db2           --                volume   --       --           --       100.00
              db2_snapvol_snp   volume   --       db2_snapvol  0.00     --

lg1           --                volume   --       --           --       100.00
              lg1_snapvol_snp   volume   --       lg1_snapvol  0.00     --

tp01          --                volume   --       --           --       100.00
              tp01_snapvol_snp  volume   --       tp01_snapvol 0.00     --


bcp_snapvol   bcp_snp           volume   bcp      --           0.00     0.11

db_snapvol    db_snp            volume   db       --           0.00     0.02

dba_snapvol   dba_snp           volume   dba      --           0.00     0.77

db2_snapvol   db2_snp           volume   db2      --           0.00     9.13

lg1_snapvol   lg1_snp           volume   lg1      --           0.00     0.09

tp01_snapvol  tp01_snp          volume   tp01     --           0.00     0.10


Mount the SNAP Volumes

When the Sync is complete, bring up the snap service group in VCS to verify all changes.  But first, unmount the filesystems on replicated volumes.  In this particular server, the replicated volumes and the snap volumes use the same mountpoints.

sunsrv02-dr:# umount ` mount | grep '/dev/vx/dsk/db2dg/' | awk '{ print $1 }'`

sunsrv02-dr:# vxtask list
TASKID  PTID TYPE/STATE    PCT   PROGRESS


sunsrv02-dr:# hastatus -sum

-- SYSTEM STATE
-- System               State                Frozen

A  sunsrv01-dr          RUNNING              0
A  sunsrv02-dr          RUNNING              0

-- GROUP STATE
-- Group            System               Probed     AutoDisabled    State

B  CCAvail          sunsrv01-dr          Y          N               ONLINE
B  CCAvail          sunsrv02-dr          Y          N               OFFLINE
B  db2inst_grp      sunsrv01-dr          Y          N               OFFLINE
B  db2inst_grp      sunsrv02-dr          Y          N               OFFLINE
B  db2inst_vvr      sunsrv01-dr          Y          N               OFFLINE
B  db2inst_vvr      sunsrv02-dr          Y          N               ONLINE
B  snap_db2inst_grp sunsrv01-dr          Y          N               OFFLINE
B  snap_db2inst_grp sunsrv02-dr          Y          N               OFFLINE


sunsrv02-dr:# hagrp -online snap_db2inst_grp -sys sunsrv02-dr


sunsrv02-dr:# hastatus -sum

-- SYSTEM STATE
-- System               State                Frozen

A  sunsrv01-dr          RUNNING              0
A  sunsrv02-dr          RUNNING              0

-- GROUP STATE
-- Group           System               Probed     AutoDisabled    State

B  CCAvail          sunsrv01-dr          Y          N               ONLINE
B  CCAvail          sunsrv02-dr          Y          N               OFFLINE
B  db2inst_grp      sunsrv01-dr          Y          N               OFFLINE
B  db2inst_grp      sunsrv02-dr          Y          N               OFFLINE
B  db2inst_vvr      sunsrv01-dr          Y          N               OFFLINE
B  db2inst_vvr      sunsrv02-dr          Y          N               ONLINE
B  snap_db2inst_grp sunsrv01-dr          Y          N               OFFLINE
B  snap_db2inst_grp sunsrv02-dr          Y          N               ONLINE


Verify the filesystems and compare the sizes with the primary site.

sunsrv02-dr:# df -k | grep snapvol

/dev/vx/dsk/db2dg/db2_snapvol    4194304   78059 3859030   2% /db2/db2inst
/dev/vx/dsk/db2dg/bcp_snapvol  292552704 1527508 272836173 1% /bcp/db2inst
/dev/vx/dsk/db2dg/dba_snapvol   49283072   28555 46176114  1% /dba/db2inst
/dev/vx/dsk/db2dg/db_snapvol   586171904 1733538 547911037 1% /db/db2inst/PEMMP00P/NODE0000
/dev/vx/dsk/db2dg/tp01_snapvol  96468992   40176  90402086 1% /db/db2inst/PEMMP00P/tempspace01/NODE0000
/dev/vx/dsk/db2dg/lg1_snapvol  198180864 1779853 184126012 1% /db/db2inst/log1



sunsrv01:# df -k | grep db2dg

/dev/vx/dsk/db2dg/db2            4194304   78059 3859030   2% /db2/db2inst
/dev/vx/dsk/db2dg/bcp          292552704 1527508 272836173 1% /bcp/db2inst
/dev/vx/dsk/db2dg/dba           49283072   28555 46176114  1% /dba/db2inst
/dev/vx/dsk/db2dg/db           586171904 1733538 547911037 1% /db/db2inst/PEMMP00P/NODE0000
/dev/vx/dsk/db2dg/tp01          96468992   40176  90402086 1% /db/db2inst/PEMMP00P/tempspace01/NODE0000
/dev/vx/dsk/db2dg/lg1          198180864 1779853 184126012 1% /db/db2inst/log1



Migration and Resizing Replicated Volumes
2007-11-01T08:17:00.000-07:00
I just completed a project for migrating and resizing replicated volumes.  I 

documented the whole process as I implement it.

The document has a detailed implementation (with screenshots) for moving all replicated volumes to a different storage 

and then resize them afterwards.  The changes were done on both primary and secondary sites.  No downtime was 

necessary for this change.

It took me 3 nights to complete the project.


Table of Contents

Project Description
Server Configurations
Network Configurations
Essential Terminology
Existing Storage Area Network (SAN) Configuration
New SAN Allocations
Primary - NEW LUNs/Devices Distribution
DR – NEW LUNs/Devices Distribution
Technical Implementation Steps/Procedures
Detailed ImplementationNote down the existing LUN and Volume configurations
SAN Team to present/zone new LUNs to the servers
Initialize the new LUNs
Mirror all volumes including the SRL
Disassociate the SNAP volumes in the DR environment
Mirror the SNAP volumes in the DR environment
Add DCM Logs with the new 17Gb LUNs
Split and mirror and remove the ones on the original 17Gb LUNs
Remove the OLD DCM logs.
Remove the SRL mirror with the old 17Gb disks.
Remove the OLD 17Gb disks from the diskgroup and from Volume Manager control.
Resize the SRL Volume in Primary
Resize the SRL Volume in DR
Resize the Replicated Volumes
Resize the DCM Logs in DR
Prepare the DR Volumes for Snapshot
Resize the SNAP Volumes in DR
Prepare the SNAP Volumes in DR
Run a point-in-time snapshot in DR


Project Description

To migrate all volumes in Replicated Volume Group from a smaller sized LUNs (17Gb) to a new larger size LUNs (136Gb).  

This is required because the volumes are to be increased to a total SAN storage size of 7.2TB in Primary and 11.3TB in 

DR will exceed the systems’ maximum limit for number of LUNs per HBA (or devices per controller) if 17Gb LUNs are 

used.



Server Configurations

Primary (Clustered)sunsrv01        (Solaris 8, SunFire V440)
sunsrv02        (Solaris 8, SunFire V440)


Secondary (Clustered)sunsrv01-DR        (Solaris 8, SunFire V440)
sunsrv02-DR        (Solaris 8, SunFire V440)




Network Configurations

IP Address

•    sunsrv01
 o    10.10.231.130    sunsrv01        Host IP
 o    10.11.196.192    sunsrv01-DRN    Primary/DR Link IP
•    sunsrv02
 o    10.10.231.131    sunsrv02        Host IP
 o    10.11.196.193    sunsrv02-DRN    Primary/DR Link IP

•    sunsrv01-DR
 o    10.10.231.130    sunsrv01-dr     Host IP
 o    10.12.94.192     sunsrv01dr-DRN  Primary/DR Link IP
•    sunsrv02-DR
 o    10.10.231.131    sunsrv02-dr     Host IP
 o    10.12.94.193     sunsrv02dr-DRN  Primary/DR Link IP

VVR Config

•    RVG    dvgy415_rvg
•    SRL    dvgy415_srl
•    RLINKs rlk_pdpd415dr-vipvr_dvgy415_rvg  (Primary)
            rlk_pdpd415Primary-vipvr_dvgy415_rvg (DR)

Virtual IPs

•    sunsrv01/sunsrv02
 o    Application        10.10.231.191
 o    Primary/DR Link    10.11.196.191

•    sunsrv01-DR/sunsrv02-DR
 o    Application        10.10.231.191
 o    Primary/DR Link    10.12.94.191


Essential Terminology
Name 
align="center">Description
VVR Veritas 

Volume Replicator. A software that replicates data to remote locations over an IP network for maximum business 

continuity.
RVG Replication Volume Group. An RVG 

is a subset of volumes within a given VxVM disk group configured for replication to one or more secondary systems. 

Data is replicated from a primary RVG to a secondary RVG. The primary RVG is in use by an application, while the 

secondary RVG receives replication and writes to local disk. The concept of primary and secondary is per RVG, not per 

system. A system can simultaneously be a primary RVG for some RVGs and secondary RVG for others. The RVG also contains 

the storage replicator log (SRL) and replication link (RLINK).

width="60">SRL Storage Replication Log. All data writes destined for volumes 

configured for replication are first persistently queued in a log called the Storage Replicator Log. VVR implements 

the SRL at the primary side to store all changes for transmission to the secondary(s). The SRL is a VxVM volume 

configured as part of an RVG. The SRL gives VVR the ability to associate writes to specific volumes within the 

replicated configuration in a specific order, maintaining write order fidelity at the secondary. All writes sent to 

the VxVM volume layer, whether from an application such as a database writing directly to storage, or an application 

accessing storage via a file system are faithfully replicated in application write order to the 

secondary.
RLink An Rlink is a VVR Replication 

Link to a secondary RVG. Each RLINK on a Primary RVG represents the communication link from the Primary RVG to a 

corresponding Secondary RVG, via an IP connection.
DCM 
width="540">Data Change Maps (DCM) are used to mark sections of volumes on the primary that have changed during 

extended network outages in order to minimize the amount of data that must be synchronized to the secondary site 

during the outage.
DCO Disk Change Object (DCO) 

volumes are used by Volume Manager FastResync (FR). FR is used to quickly resynchronize mirrored volumes that have 

been temporarily split and rejoined. FR works by copying only changes to the newly reattached volume using FR logging. 

This process reduces the time required to rejoin a split mirror, and requires less processing power than full mirror 

resynchronization without logging.



Existing Storage Area Network (SAN) Configuration

•    Each Primary server has 2 HBAs attached to the fabric.
•    Primary servers are using Veritas DMP for Multipathing.
•    Each Primary server is presented with shared 95 x 17Gb EMC LUNs (IBM Managed)
•    Each DR server has 4 HBAs (2 dual ports) attached to the fabric.
•    DR servers are using Veritas DMP for multipathing.
•    Each DR server is presented with shared 8 x 136Gb EMC LUNs and 198 x 17Gb EMC LUNs.
•    DR SAN is managed by EMC.

New SAN Allocations



Primary - NEW LUNs/Devices Distribution

Note: Device Names may change during server reboot with reconfig option.


VOLUMES    DEVICES    DISK NAMES

DB    EMC0_139    dvgy41596
EMC0_140    dvgy41597
EMC0_142    dvgy41598
EMC0_143    dvgy41599
EMC0_144    dvgy415100
EMC0_146    dvgy415101
EMC0_150    dvgy415102
EMC0_153    dvgy415103
EMC0_154    dvgy415104
EMC0_164    dvgy415105

LG1    EMC0_148    dvgy415106
EMC0_155    dvgy415107
EMC0_156    dvgy415108

BCP    EMC0_157    dvgy415109
EMC0_158    dvgy415110

TP01    EMC0_161    dvgy415111
EMC0_138    dvgy415112
EMC0_141    dvgy415113

DB2/DBA/BCP    EMC0_145    dvgy415114



SRL AND DCM LOGS    DEVICES    DISK NAMES

SRL    EMC0_147    dvgy415115
EMC0_149    dvgy415116
EMC0_151    dvgy415117
EMC0_152    dvgy415118
EMC0_159    dvgy415119
EMC0_160    dvgy415120
EMC0_163    dvgy415121

DCM LOG    EMC0_130    dvgy415122
EMC0_162    dvgy415123



DR – NEW LUNs/Devices Distribution

Note: Device Names may change during server reboot with reconfig option.



Technical Implementation Steps/Procedures (NO DOWNTIME!)

1.    Note down the existing LUN configurations.
2.    Present new LUNs to the servers.
•    53 x 136Gb LUNs to Primary for Replicating Dataset.
•    2 x 17Gb LUNs to Primary for DCM Logs.
•    50 x 136Gb LUNs to DR for Landing area, Snap Copy, and Backup filesystems.  The existing 17Gb LUNs will be 

reformatted by EMC and change the LUN sizes to 136Gb.
•    2 x 17Gb LUNs to DR for DCM Logs.
3.    Initialize the new LUNs and assign them to the diskgroup for replicated volumes.  This step is for both Primary 

and DR.
4.    Mirror each replicated volume and the SRL volume using the new LUNs as the mirror disks.  This step is for both 

Primary and DR.
5.    Disassociate the SNAP volumes (this may not be necessary)
a.    Disassociate the snap volumes from the snapshot hierarchy.
b.    Unprepare the snap volumes.
6.    Mirror the snap volumes in DR environment using the new LUNs as the mirror disks.
7.    For each volume, add a DCM log using the 2 new  17Gb LUNs.
8.    When all volumes are completely mirrored, remove the mirror copy that were lying on the 17Gb disks.
9.    Remove the old DCM logs that were on the old LUNs.  For both Primary and DR.
10.    Remove both Primary and DR SRL mirror copy that is on the 17Gb disks.
11.    Remove all the old 17Gb disks from the diskgroup and from Veritas Volume Manager control.  For both Primary and 

DR.
12.    Resize the SRL volume in Primary.
13.    Resize the SRL volume in DR.
14.    Resize the replicated Volumes.
15.    Remove and re-add the DCM logs on the Primary and DR volumes to have bigger log sizes.  This step is necessary 

because the original logs were too small for the new volume sizes.
16.    Prepare the DR Volumes for Snapshot.
17.    Resize the SNAP Volumes in DR (or re-create if you do not want to preserve the data on the snapshot volumes.)
18.    Prepare the SNAP Volumes in DR.
19.    Run a point-in-time snapshot in DR (this will overwrite the previous data in the snap volumes.)


Detailed Implementation


1. Note down the existing LUN and Volume configurations

# format

# vxdisk list

# vxprint –ht


2. SAN Team to present/zone new LUNs to the servers

Verify the connectivity to the fabric

# luxadm -e port

Found path to 4 HBA ports

/devices/pci@1e,600000/SUNW,qlc@3/fp@0,0:devctl                    CONNECTED
/devices/pci@1e,600000/SUNW,qlc@3,1/fp@0,0:devctl                  CONNECTED
/devices/pci@1e,600000/SUNW,qlc@4/fp@0,0:devctl                    CONNECTED
/devices/pci@1e,600000/SUNW,qlc@4,1/fp@0,0:devctl                  CONNECTED

Discover the new LUNs (look for “connected unconfigured”)

# cfgadm -o show_FCP_dev –al
Ap_Id                          Type         Receptacle   Occupant     Condition
c3::50060482ccaad688,1         unavailable  connected    unconfigured unknown
c3::50060482ccaad688,2         unavailable  connected    unconfigured unknown
c3::50060482ccaad688,3         unavailable  connected    unconfigured unknown
.
.
c4                             fc-fabric    connected    unconfigured unknown
c4::50060482ccaad686           disk         connected    unconfigured unknown
c4::50060482ccaad688           disk         connected    unconfigured unknown
.
.
c5::50060482ccaad689,1         unavailable  connected    unconfigured unknown
c5::50060482ccaad689,2         unavailable  connected    unconfigured unknown
c5::50060482ccaad689,3         unavailable  connected    unconfigured unknown
.
.
c6                             fc-fabric    connected    unconfigured unknown
c6::50060482ccaad687           disk         connected    unconfigured unknown
c6::50060482ccaad689           disk         connected    unconfigured unknown

Configure the new LUNs so the operating system can see them.  Run on all Cluster Nodes.

# cfgadm -c configure c3
# cfgadm -c configure c4
# cfgadm -c configure c5
# cfgadm -c configure c6

Verify if the new LUNs have been configured

# cfgadm -o show_FCP_dev –al
Ap_Id                          Type         Receptacle   Occupant     Condition
c3::50060482ccaad688,1         disk         connected    configured   unknown
c3::50060482ccaad688,2         disk         connected    configured   unknown
c3::50060482ccaad688,3         disk         connected    configured   unknown
.
.
c4                             fc-fabric    connected    configured   unknown
c4::50060482ccaad686,0         disk         connected    configured   unknown
c4::50060482ccaad686,43        disk         connected    configured   unknown
c4::50060482ccaad686,44        disk         connected    configured   unknown
.
.
c5::50060482ccaad689,1         disk         connected    configured   unknown
c5::50060482ccaad689,2         disk         connected    configured   unknown
c5::50060482ccaad689,3         disk         connected    configured   unknown
.
.
.
c6::50060482ccaad689,21        disk         connected    configured   unknown
c6::50060482ccaad689,22        disk         connected    configured   unknown
c6::50060482ccaad689,23        disk         connected    configured   unknown

Verify if you can see the new devices in format.  If not, run devfsadm.    Run on all Cluster Nodes.

# /usr/sbin/devfsadm

Label the new disks.  Create a file /tmp/format.cmd.    Run on all Cluster Nodes.

# cat /tmp/format.cmd
label
quit

# for disk in `format < /dev/null 2> /dev/null | grep "^c" | cut -d: -f1`
> do
>  format -s -f /tmp/format.cmd $disk
>   echo "labeled $disk ....."
> done

Update volume manager.    Run on all Cluster Nodes.

# vxdisk scandisks
# vxdctl enable

Verify the number of LUNs, the LUN sizes,  and the number of paths for each LUN

# vxdmpadm listctlr all
# vxdmpadm getsubpaths ctlr=c3   [run for c3 c4 c5 c6 ]
# vxdisk list
# vxdisk path

Verify the LUN sizes.  If some of the LUNs have the wrong size, notify the SAN admin.

# for i in `vxdisk list | grep EMC | grep - | awk '{ print $1 }'`
> do
>   echo "$i \c"
>   disk=`vxdisk list $i | grep "^c3" | awk '{ print $1 }'`
>   echo "$disk\t\c"
>   prtvtoc /dev/rdsk/$disk | grep "^       2 "
> done

3. Initialize the new LUNs

# vxdiskadm

4. Mirror all volumes including the SRL

Primary:  Specify the target disks  (refer to page 6)

sunsrv01:# vxassist -g dvgy415 mirror db   dvgy41596  dvgy41597  dvgy41598 \
                                   dvgy41599  dvgy415100
sunsrv01:# vxassist -g dvgy415 mirror lg1  dvgy415106 dvgy415107
sunsrv01:# vxassist -g dvgy415 mirror bcp  dvgy415109 dvgy415110 dvgy415114
sunsrv01:# vxassist -g dvgy415 mirror tp01 dvgy415111
sunsrv01:# vxassist -g dvgy415 mirror dba  dvgy415114
sunsrv01:# vxassist -g dvgy415 mirror db2  dvgy415114

sunsrv01:# vxassist -g dvgy415 mirror dvgy415_srl dvgy415115 dvgy415116 dvgy415117

DR:  Specify the target disks  (refer to page 7)

sunsrv02-dr:# vxassist -g dvgy415 mirror db   dvgy415201 dvgy415202 dvgy415203 \
                                      dvgy415204 dvgy415205
sunsrv02-dr:# vxassist -g dvgy415 mirror lg1  dvgy415211 dvgy415212
sunsrv02-dr:# vxassist -g dvgy415 mirror bcp  dvgy415214 dvgy415215 dvgy415219
sunsrv02-dr:# vxassist -g dvgy415 mirror tp01 dvgy415216 dvgy415217 dvgy415218
sunsrv02-dr:# vxassist -g dvgy415 mirror dba  dvgy415219
sunsrv02-dr:# vxassist -g dvgy415 mirror db2  dvgy415219

sunsrv02-dr:# vxassist -g dvgy415 mirror dvgy415_srl dvgy415239 dvgy415240 \
                                             dvgy415241

5. Disassociate the SNAP volumes in the DR environment

sunsrv02-dr:# vxsnap -g dvgy415 dis db_snapvol
sunsrv02-dr:# vxsnap -g dvgy415 -f unprepare db_snapvol
sunsrv02-dr:# vxsnap -g dvgy415 unprepare db

sunsrv02-dr:# vxsnap -g dvgy415 dis lg1_snapvol
sunsrv02-dr:# vxsnap -g dvgy415 -f unprepare lg1_snapvol
sunsrv02-dr:# vxsnap -g dvgy415 unprepare lg1

sunsrv02-dr:# vxsnap -g dvgy415 dis db2_snapvol
sunsrv02-dr:# vxsnap -g dvgy415 -f unprepare db2_snapvol
sunsrv02-dr:# vxsnap -g dvgy415 unprepare db2


sunsrv02-dr:# vxsnap -g dvgy415 dis bcp_snapvol
sunsrv02-dr:# vxsnap -g dvgy415 -f unprepare bcp_snapvol
sunsrv02-dr:# vxsnap -g dvgy415 unprepare bcp

sunsrv02-dr:# vxsnap -g dvgy415 dis tp01_snapvol
sunsrv02-dr:# vxsnap -g dvgy415 -f unprepare tp01_snapvol
sunsrv02-dr:# vxsnap -g dvgy415 unprepare tp01

sunsrv02-dr:# vxsnap -g dvgy415 dis dba_snapvol
sunsrv02-dr:# vxsnap -g dvgy415 -f unprepare dba_snapvol
sunsrv02-dr:# vxsnap -g dvgy415 unprepare dba

6. Mirror the SNAP volumes in the DR environment

sunsrv02-dr:# vxassist -g dvgy415 mirror db_snapvol   dvgy415220 dvgy415221 \
                                   dvgy415222 dvgy415223 dvgy415224
sunsrv02-dr:# vxassist -g dvgy415 mirror lg1_snapvol  dvgy415230 dvgy415231
sunsrv02-dr:# vxassist -g dvgy415 mirror bcp_snapvol  dvgy415233 dvgy415234 \
                                              dvgy415238
sunsrv02-dr:# vxassist -g dvgy415 mirror tp01_snapvol dvgy415235 dvgy415236 \
                                              dvgy415237
sunsrv02-dr:# vxassist -g dvgy415 mirror dba_snapvol  dvgy415238
sunsrv02-dr:# vxassist -g dvgy415 mirror db2_snapvol  dvgy415238

7. Add DCM Logs with the new 17Gb LUNs

Primary:  Specify the target disks  (refer to page 6)

sunsrv01:# vxassist -g dvgy415 addlog bcp  logtype=dcm nlog=2 dvgy415122 dvgy415123
sunsrv01:# vxassist -g dvgy415 addlog db   logtype=dcm nlog=2 dvgy415122 dvgy415123
sunsrv01:# vxassist -g dvgy415 addlog dba  logtype=dcm nlog=2 dvgy415122 dvgy415123
sunsrv01:# vxassist -g dvgy415 addlog db2  logtype=dcm nlog=2 dvgy415122 dvgy415123
sunsrv01:# vxassist -g dvgy415 addlog lg1  logtype=dcm nlog=2 dvgy415122 dvgy415123
sunsrv01:# vxassist -g dvgy415 addlog tp01 logtype=dcm nlog=2 dvgy415122 dvgy415123

DR:  Specify the target disks  (refer to page 7)

sunsrv02-dr:# vxassist -g dvgy415 addlog bcp  logtype=dcm nlog=2 dvgy415199 \
                                                         dvgy415200
sunsrv02-dr:# vxassist -g dvgy415 addlog db   logtype=dcm nlog=2 dvgy415199 \
                                                         dvgy415200
sunsrv02-dr:# vxassist -g dvgy415 addlog dba  logtype=dcm nlog=2 dvgy415199 \
                                                         dvgy415200
sunsrv02-dr:# vxassist -g dvgy415 addlog db2  logtype=dcm nlog=2 dvgy415199 \
                                                         dvgy415200
sunsrv02-dr:# vxassist -g dvgy415 addlog lg1  logtype=dcm nlog=2 dvgy415199 \
                                                         dvgy415200
sunsrv02-dr:# vxassist -g dvgy415 addlog tp01 logtype=dcm nlog=2 dvgy415199 \
                                                         dvgy415200

8. Split and mirror and remove the ones on the original 17Gb LUNs

First, Verify that all volumes and plexes are in ENABLED ACTIVE state.

Notice the three new plexes 03, 04 and 05.  These the the mirror copy of data and the two new logs.

sunsrv01:# vxprint -htg dvgy415 | egrep "^v|^pl"

v  bcp          dvgy415_rvg  ENABLED  ACTIVE   585105408 SELECT   -        fsgen
pl bcp-01       bcp          ENABLED  ACTIVE   585105600 CONCAT   -        RW
pl bcp-02       bcp          ENABLED  ACTIVE   LOGONLY   CONCAT   -        RW
pl bcp-03       bcp          ENABLED  ACTIVE   585108480 CONCAT   -        RW
pl bcp-04       bcp          ENABLED  ACTIVE   LOGONLY   CONCAT   -        RW
pl bcp-05       bcp          ENABLED  ACTIVE   LOGONLY   CONCAT   -        RW


v  dba          dvgy415_rvg  ENABLED  ACTIVE   98566144 SELECT    -        fsgen
pl dba-01       dba          ENABLED  ACTIVE   98567040 CONCAT    -        RW
pl dba-02       dba          ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW
pl dba-03       dba          ENABLED  ACTIVE   98572800 CONCAT    -        RW
pl dba-04       dba          ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW
pl dba-05       dba          ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

v  db2          dvgy415_rvg  ENABLED  ACTIVE   8388608  SELECT    -        fsgen
pl db2-01       db2          ENABLED  ACTIVE   8389440  CONCAT    -        RW
pl db2-02       db2          ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW
pl db2-03       db2          ENABLED  ACTIVE   8394240  CONCAT    -        RW
pl db2-04       db2          ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW
pl db2-05       db2          ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

v  tp01         dvgy415_rvg  ENABLED  ACTIVE   192937984 SELECT   -        fsgen
pl tp01-01      tp01         ENABLED  ACTIVE   192938880 CONCAT   -        RW
pl tp01-02      tp01         ENABLED  ACTIVE   LOGONLY   CONCAT   -        RW
pl tp01-03      tp01         ENABLED  ACTIVE   192944640 CONCAT   -        RW
pl tp01-04      tp01         ENABLED  ACTIVE   LOGONLY   CONCAT   -        RW
pl tp01-05      tp01         ENABLED  ACTIVE   LOGONLY   CONCAT   -        RW

v  lg1          dvgy415_rvg  ENABLED  ACTIVE   396361728 SELECT   -        fsgen
pl lg1-01       lg1          ENABLED  ACTIVE   396361920 CONCAT   -        RW
pl lg1-02       lg1          ENABLED  ACTIVE   LOGONLY   CONCAT   -        RW
pl lg1-03       lg1          ENABLED  ACTIVE   396364800 CONCAT   -        RW
pl lg1-04       lg1          ENABLED  ACTIVE   LOGONLY   CONCAT   -        RW
pl lg1-05       lg1          ENABLED  ACTIVE   LOGONLY   CONCAT   -        RW

v  db           dvgy415_rvg  ENABLED  ACTIVE   1172343808 SELECT  -        fsgen
pl db-01        db           ENABLED  ACTIVE   1172344320 CONCAT  -        RW
pl db-02        db           ENABLED  ACTIVE   LOGONLY    CONCAT  -        RW
pl db-03        db           ENABLED  ACTIVE   1172344320 CONCAT  -        RW
pl db-04        db           ENABLED  ACTIVE   LOGONLY    CONCAT  -        RW
pl db-05        db           ENABLED  ACTIVE   LOGONLY    CONCAT  -        RW

v  dvgy415_srl  dvgy415_rvg  ENABLED  ACTIVE   856350720 SELECT   -        SRL
pl dvgy415_srl-01 dvgy415_srl ENABLED ACTIVE   856350720 CONCAT   -        RW
pl dvgy415_srl-02 dvgy415_srl ENABLED ACTIVE   856350720 CONCAT   -        RW


In the DR environment, take note that the SRL has not completed the mirroring at this time.

sunsrv02-dr:# vxprint -htg dvgy415 | egrep "^v|^pl"
v  bcp_snapvol  -            ENABLED  ACTIVE   585105408 SELECT   -        fsgen
pl bcp_snapvol-01 bcp_snapvol ENABLED ACTIVE   585105600 CONCAT   -        RW
pl bcp_snapvol-02 bcp_snapvol ENABLED ACTIVE   585108480 CONCAT   -        RW

v  db_snapvol   -            ENABLED  ACTIVE   1172343808 SELECT  -        fsgen
pl db_snapvol-01 db_snapvol  ENABLED  ACTIVE   1172344320 CONCAT  -        RW
pl db_snapvol-02 db_snapvol  ENABLED  ACTIVE   1172344320 CONCAT  -        RW


v  dba_snapvol  -            ENABLED  ACTIVE   98566144 SELECT    -        fsgen
pl dba_snapvol-01 dba_snapvol ENABLED ACTIVE   98567040 CONCAT    -        RW
pl dba_snapvol-02 dba_snapvol ENABLED ACTIVE   98572800 CONCAT    -        RW

v  db2_snapvol  -            ENABLED  ACTIVE   8388608  SELECT    -        fsgen
pl db2_snapvol-01 db2_snapvol ENABLED ACTIVE   8389440  CONCAT    -        RW
pl db2_snapvol-02 db2_snapvol ENABLED ACTIVE   8394240  CONCAT    -        RW

v  lg1_snapvol  -            ENABLED  ACTIVE   396361728 SELECT   -        fsgen
pl lg1_snapvol-01 lg1_snapvol ENABLED ACTIVE   396361920 CONCAT   -        RW
pl lg1_snapvol-02 lg1_snapvol ENABLED ACTIVE   396364800 CONCAT   -        RW

v  tp01_snapvol -            ENABLED  ACTIVE   192937984 SELECT   -        fsgen
pl tp01_snapvol-01 tp01_snapvol ENABLED ACTIVE 192938880 CONCAT   -        RW
pl tp01_snapvol-02 tp01_snapvol ENABLED ACTIVE 192944640 CONCAT   -        RW

v  bcp          dvgy415_rvg  ENABLED  ACTIVE   585105408 SELECT   -        fsgen
pl bcp-01       bcp          ENABLED  ACTIVE   585105600 CONCAT   -        RW
pl bcp-02       bcp          ENABLED  ACTIVE   LOGONLY   CONCAT   -        RW
pl bcp-03       bcp          ENABLED  ACTIVE   585108480 CONCAT   -        RW
pl bcp-04       bcp          ENABLED  ACTIVE   LOGONLY   CONCAT   -        RW
pl bcp-05       bcp          ENABLED  ACTIVE   LOGONLY   CONCAT   -        RW

v  db           dvgy415_rvg  ENABLED  ACTIVE   1172343808 SELECT  -        fsgen
pl db-01        db           ENABLED  ACTIVE   1172344320 CONCAT  -        RW
pl db-02        db           ENABLED  ACTIVE   LOGONLY    CONCAT  -        RW
pl db-03        db           ENABLED  ACTIVE   1172344320 CONCAT  -        RW
pl db-04        db           ENABLED  ACTIVE   LOGONLY    CONCAT  -        RW
pl db-05        db           ENABLED  ACTIVE   LOGONLY    CONCAT  -        RW

v  dba          dvgy415_rvg  ENABLED  ACTIVE   98566144 SELECT    -        fsgen
pl dba-01       dba          ENABLED  ACTIVE   98567040 CONCAT    -        RW
pl dba-02       dba          ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW
pl dba-03       dba          ENABLED  ACTIVE   98572800 CONCAT    -        RW
pl dba-04       dba          ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW
pl dba-05       dba          ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

v  db2          dvgy415_rvg  ENABLED  ACTIVE   8388608  SELECT    -        fsgen
pl db2-01       db2          ENABLED  ACTIVE   8389440  CONCAT    -        RW
pl db2-02       db2          ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW
pl db2-03       db2          ENABLED  ACTIVE   8394240  CONCAT    -        RW
pl db2-04       db2          ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW
pl db2-05       db2          ENABLED  ACTIVE   LOGONLY  CONCAT    -        RW

v  lg1          dvgy415_rvg  ENABLED  ACTIVE   396361728 SELECT   -        fsgen
pl lg1-01       lg1          ENABLED  ACTIVE   396361920 CONCAT   -        RW
pl lg1-02       lg1          ENABLED  ACTIVE   LOGONLY   CONCAT   -        RW
pl lg1-03       lg1          ENABLED  ACTIVE   396364800 CONCAT   -        RW
pl lg1-04       lg1          ENABLED  ACTIVE   LOGONLY   CONCAT   -        RW
pl lg1-05       lg1          ENABLED  ACTIVE   LOGONLY   CONCAT   -        RW

v  tp01         dvgy415_rvg  ENABLED  ACTIVE   192937984 SELECT   -        fsgen
pl tp01-01      tp01         ENABLED  ACTIVE   192938880 CONCAT   -        RW
pl tp01-02      tp01         ENABLED  ACTIVE   LOGONLY   CONCAT   -        RW
pl tp01-03      tp01         ENABLED  ACTIVE   192944640 CONCAT   -        RW
pl tp01-04      tp01         ENABLED  ACTIVE   LOGONLY   CONCAT   -        RW
pl tp01-05      tp01         ENABLED  ACTIVE   LOGONLY   CONCAT   -        RW

v  dvgy415_srl  dvgy415_rvg  ENABLED  ACTIVE   856350720 SELECT   -        SRL
pl dvgy415_srl-01 dvgy415_srl ENABLED ACTIVE   856350720 CONCAT   -        RW
pl dvgy415_srl-02 dvgy415_srl ENABLED TEMPRMSD 856350720 CONCAT   -        WO


When all volumes and plexes are in ENABLED ACTIVE state, remove the old mirrors.

sunsrv01:# vxplex -g dvgy415 -o rm dis dba-01
sunsrv01:# vxplex -g dvgy415 -o rm dis db2-01
sunsrv01:# vxplex -g dvgy415 -o rm dis bcp-01
sunsrv01:# vxplex -g dvgy415 -o rm dis tp01-01
sunsrv01:# vxplex -g dvgy415 -o rm dis lg1-01
sunsrv01:# vxplex -g dvgy415 -o rm dis db-01

sunsrv02-dr:# vxplex -g dvgy415 -o rm dis dba-01
sunsrv02-dr:# vxplex -g dvgy415 -o rm dis db2-01
sunsrv02-dr:# vxplex -g dvgy415 -o rm dis bcp-01
sunsrv02-dr:# vxplex -g dvgy415 -o rm dis tp01-01
sunsrv02-dr:# vxplex -g dvgy415 -o rm dis lg1-01
sunsrv02-dr:# vxplex -g dvgy415 -o rm dis db-01

9. Remove the OLD DCM logs.

sunsrv01:# vxplex -g dvgy415 -o rm dis dba-02
sunsrv01:# vxplex -g dvgy415 -o rm dis db2-02
sunsrv01:# vxplex -g dvgy415 -o rm dis bcp-02
sunsrv01:# vxplex -g dvgy415 -o rm dis tp01-02
sunsrv01:# vxplex -g dvgy415 -o rm dis lg1-02
sunsrv01:# vxplex -g dvgy415 -o rm dis db-02

sunsrv02-dr:# vxplex -g dvgy415 -o rm dis dba-02
sunsrv02-dr:# vxplex -g dvgy415 -o rm dis db2-02
sunsrv02-dr:# vxplex -g dvgy415 -o rm dis bcp-02
sunsrv02-dr:# vxplex -g dvgy415 -o rm dis tp01-02
sunsrv02-dr:# vxplex -g dvgy415 -o rm dis lg1-02
sunsrv02-dr:# vxplex -g dvgy415 -o rm dis db-02

10. Remove the SRL mirror with the old 17Gb disks.

First, Verify the state of RVG.  Make sure it’s still in active state and up to date.

sunsrv01:# vxrlink -g dvgy415 status rlk_pdpd415dr-vipvr_dvgy415_rvg
Thu Dec 13 10:31:40 MST 2007
VxVM VVR vxrlink INFO V-5-1-4467 Rlink rlk_pdpd415dr-vipvr_dvgy415_rvg is up to date

sunsrv01:# vxprint -Pl
Disk group: dvgy415

Rlink:    rlk_pdpd415dr-vipvr_dvgy415_rvg
info:     timeout=500 packet_size=8400 rid=0.2007
  latency_high_mark=10000 latency_low_mark=9950
  bandwidth_limit=none
state:    state=ACTIVE
  synchronous=off latencyprot=off srlprot=autodcm
assoc:    rvg=dvgy415_rvg
  remote_host=pdpd415dr-vipvr IP_addr=10.12.94.191 port=4145
  remote_dg=dvgy415
  remote_dg_dgid=1182230506.2373.sunsrv01-dr
  remote_rvg_version=21
  remote_rlink=rlk_pdpd415Primary-vipvr_dvgy415_rvg
  remote_rlink_rid=0.2127
  local_host=pdpd415Primary-vipvr IP_addr=10.11.196.191 port=4145
protocol: UDP/IP
flags:    write enabled attached consistent connected asynchronous

Remove the SRL mirror.  Remember that the first copy is the older copy.

sunsrv01:# vxplex -g dvgy415 -o rm dis dvgy415_srl-01

sunsrv02-dr:# vxplex -g dvgy415 -o rm dis dvgy415_srl-01

Again, verify the state of RVG.  Make sure it’s still in active state and up to date.

sunsrv01:# vxrlink -g dvgy415 status rlk_pdpd415dr-vipvr_dvgy415_rvg
Thu Dec 13 10:35:06 MST 2007
VxVM VVR vxrlink INFO V-5-1-4467 Rlink rlk_pdpd415dr-vipvr_dvgy415_rvg is up to date

sunsrv01:# vxprint -Pl
Disk group: dvgy415

Rlink:    rlk_pdpd415dr-vipvr_dvgy415_rvg
info:     timeout=500 packet_size=8400 rid=0.2007
  latency_high_mark=10000 latency_low_mark=9950
  bandwidth_limit=none
state:    state=ACTIVE
  synchronous=off latencyprot=off srlprot=autodcm
assoc:    rvg=dvgy415_rvg
  remote_host=pdpd415dr-vipvr IP_addr=10.12.94.191 port=4145
  remote_dg=dvgy415
  remote_dg_dgid=1182230506.2373.sunsrv01-dr
  remote_rvg_version=21
  remote_rlink=rlk_pdpd415Primary-vipvr_dvgy415_rvg
  remote_rlink_rid=0.2127
  local_host=pdpd415Primary-vipvr IP_addr=10.11.196.191 port=4145
protocol: UDP/IP
flags:    write enabled attached consistent connected asynchronous

11. Remove the OLD 17Gb disks from the diskgroup and from Volume Manager control.

We know that there were originally 95 17Gb LUNs (see third page of this document under “Existing SAN Configuration”) 

on the Primary server and by now all those 95 LUNs should be unused.  Use vxdg free to verify it.  17Gb should be 

35681280 blocks.

sunsrv01:# vxdg -g dvgy415 free | grep 35681280 | wc -l
95

Now a small script to remove them:

sunsrv01:# vxdg -g dvgy415 free | grep 35681280 | awk '{ print $1" "$2 }' | while read disk device
> do
>   vxdg -g dvgy415 rmdisk $disk
>   vxdisk rm $device
>   echo "disk:$disk  device:$device  REMOVED!"
> done

disk:dvgy41501  device:EMC0_36  REMOVED!
disk:dvgy41502  device:EMC0_29  REMOVED!
disk:dvgy41503  device:EMC0_25  REMOVED!
.
.
.
disk:dvgy41590  device:EMC0_89  REMOVED!
disk:dvgy41591  device:EMC0_90  REMOVED!
disk:dvgy41592  device:EMC0_91  REMOVED!
disk:dvgy41593  device:EMC0_92  REMOVED!
disk:dvgy41594  device:EMC0_93  REMOVED!
disk:dvgy41595  device:EMC0_94  REMOVED!

Since the server is part of a cluster, the other node must be cleaned up as well.   Different logic this time.

sunsrv02:# for i in `vxdisk -o alldgs list | grep EMC | grep -v dvgy | awk '{ print $1 }'`
> do
>   vxdisk rm $i
> done

Now in the DR server, there were originally 198 17Gb disks.  We should remove all of them.  There are two different 

17Gb sizes on the servers.

sunsrv02-dr:# vxdg -g dvgy415 free | egrep "35726400|35681280" | wc -l
198

In removing the disks, I displayed the WWN/LUN numbers so I can document and give to the SAN folks which LUNs should 

be removed and reconfigured for the new 136Gb sizes (see project description on the first page).

sunsrv02-dr:# vxdg -g dvgy415 free | egrep "35726400|35681280" | awk '{ print $1" "$2 }' | while read disk device
> do
>   physical=`vxdisk list $device | egrep "^c3|^c5" | awk '{ print $1 }'`
>   vxdg -g dvgy415 rmdisk $disk
>   vxdisk rm $device
>   echo "$physical  disk:$disk  device:$device  REMOVED!"
> done
c3t50060482CC3722B9d204s2
c5t50060482CC3722B6d204s2  disk:dvgy41501  device:EMC0_21  REMOVED!
c3t50060482CC3722B9d203s2
c5t50060482CC3722B6d203s2  disk:dvgy41502  device:EMC0_29  REMOVED!
.
.
.
.
c5t50060482CC3722B6d101s2  disk:dvgy415194  device:EMC0_10  REMOVED!
c3t50060482CC3722B9d102s2
c5t50060482CC3722B6d102s2  disk:dvgy415195  device:EMC0_147  REMOVED!
c3t50060482CC3722B9d103s2
c5t50060482CC3722B6d103s2  disk:dvgy415196  device:EMC0_148  REMOVED!
c3t50060482CC3722B9d104s2
c5t50060482CC3722B6d104s2  disk:dvgy415197  device:EMC0_149  REMOVED!
c3t50060482CC3722B9d105s2
c5t50060482CC3722B6d105s2  disk:dvgy415198  device:EMC0_150  REMOVED!


Since the server is part of a cluster, the other node must be cleaned up as well.

sunsrv01-dr:# for i in `vxdisk -o alldgs list | grep EMC | grep -v dvgy | awk '{ print $1 }'`
> do
>   vxdisk rm $i
> done

12. Resize the SRL Volume in Primary

First, display the SRL volume information

sunsrv01:# vxprint -ht dvgy415_srl
v  dvgy415_srl    dvgy415_rvg ENABLED ACTIVE   856350720 SELECT     -        SRL
pl dvgy415_srl-02 dvgy415_srl ENABLED ACTIVE   856350720 CONCAT     -        RW
sd dvgy415115-01 dvgy415_srl-02 dvgy415115 5376 285473280 0         EMC0_147 ENA
sd dvgy415116-01 dvgy415_srl-02 dvgy415116 5376 285473280 285473280 EMC0_149 ENA
sd dvgy415117-01 dvgy415_srl-02 dvgy415117 5376 285404160 570946560 EMC0_151 ENA

Check the rlink status and make sure it’s up to date

sunsrv01:# vxrlink -g dvgy415 status rlk_pdpd415dr-vipvr_dvgy415_rvg
Fri Dec 14 21:24:49 MST 2007
VxVM VVR vxrlink INFO V-5-1-4467 Rlink rlk_pdpd415dr-vipvr_dvgy415_rvg is up to date

Display the maximum space we can add using the remaining disks we have alloted for SRL.

sunsrv01:# vxassist -g dvgy415 -p maxsize dvgy415118 dvgy415119 dvgy415120 dvgy415121

1141893120

Increase the SRL size by 1141893120 blocks.

sunsrv01:# vradmin -g dvgy415 resizesrl dvgy415_rvg +1141893120

Check the volume information again and verify if the size has been increased.

sunsrv01:# vxprint -ht dvgy415_srl
Disk group: dvgy415

v  dvgy415_srl    dvgy415_rvg ENABLED ACTIVE   1998243840 SELECT     -        SRL
pl dvgy415_srl-02 dvgy415_srl ENABLED ACTIVE   1998243840 CONCAT     -        RW
sd dvgy415115-01 dvgy415_srl-02 dvgy415115 5376 285473280 0          EMC0_147 ENA
sd dvgy415116-01 dvgy415_srl-02 dvgy415116 5376 285473280 285473280  EMC0_149 ENA
sd dvgy415117-01 dvgy415_srl-02 dvgy415117 5376 285473280 570946560  EMC0_151 ENA
sd dvgy415118-01 dvgy415_srl-02 dvgy415118 5376 285473280 856419840  EMC0_152 ENA
sd dvgy415119-01 dvgy415_srl-02 dvgy415119 5376 285473280 1141893120 EMC0_159 ENA
sd dvgy415120-01 dvgy415_srl-02 dvgy415120 5376 285473280 1427366400 EMC0_160 ENA
sd dvgy415121-01 dvgy415_srl-02 dvgy415121 5376 285404160 1712839680 EMC0_163 ENA

13. Resize the SRL Volume in DR

First, display the SRL volume information

sunsrv02-dr:# vxprint -ht dvgy415_srl
Disk group: dvgy415

v  dvgy415_srl    dvgy415_rvg ENABLED ACTIVE   856350720 SELECT     -        SRL
pl dvgy415_srl-02 dvgy415_srl ENABLED ACTIVE   856350720 CONCAT     -        RW
sd dvgy415239-01 dvgy415_srl-02 dvgy415239 5376 285473280 0         EMC0_244 ENA
sd dvgy415240-01 dvgy415_srl-02 dvgy415240 5376 285473280 285473280 EMC0_245 ENA
sd dvgy415241-01 dvgy415_srl-02 dvgy415241 5376 285404160 570946560 EMC0_246 ENA

Check the rlink status and make sure it’s up to date

sunsrv02-dr:# vxrlink -g dvgy415 det rlk_pdpd415Primary-vipvr_dvgy415_rvg

Disassociate the SRL volume from the RVG

sunsrv02-dr:# vxvol -g dvgy415 dis dvgy415_srl

Verify if the volume has been disassociated.  The 3rd column should not show the RVG.

sunsrv02-dr:# vxprint -ht dvgy415_srl
Disk group: dvgy415

v  dvgy415_srl    -             ENABLED ACTIVE   856350720 SELECT    -        fsgen
pl dvgy415_srl-02 dvgy415_srl   ENABLED ACTIVE   856350720 CONCAT    -        RW
sd dvgy415239-01  dvgy415_srl-02 dvgy415239 5376 285473280 0         EMC0_244 ENA
sd dvgy415240-01  dvgy415_srl-02 dvgy415240 5376 285473280 285473280 EMC0_245 ENA
sd dvgy415241-01  dvgy415_srl-02 dvgy415241 5376 285404160 570946560 EMC0_246 ENA

Display the RVG status.  Note down the state is no longer active and the flag says detached.

sunsrv02-dr:# vxprint -Pl
Disk group: dvgy415

Rlink:    rlk_pdpd415Primary-vipvr_dvgy415_rvg
info:     timeout=500 packet_size=8400 rid=0.2127
  latency_high_mark=10000 latency_low_mark=9950
  bandwidth_limit=none
state:    state=STALE
  synchronous=off latencyprot=off srlprot=autodcm
assoc:    rvg=dvgy415_rvg
  remote_host=pdpd415Primary-vipvr IP_addr=10.11.196.191 port=4145
  remote_dg=dvgy415
  remote_dg_dgid=1138140445.1393.sunsrv01
  remote_rvg_version=21
  remote_rlink=rlk_pdpd415dr-vipvr_dvgy415_rvg
  remote_rlink_rid=0.2007
  local_host=pdpd415dr-vipvr IP_addr=10.12.94.191 port=4145
protocol: UDP/IP
flags:    write enabled detached consistent disconnected

Increase the size of the SRL by the same size we used in the Primary SRL.

sunsrv02-dr:# vxassist -g dvgy415 growby dvgy415_srl 1141893120 dvgy415241 dvgy415242 dvgy415243 dvgy415244 dvgy415245

Check the volume information again and verify if the size has been increased and has the same size as the Primary SRL 

volume.

sunsrv02-dr:# vxprint -ht dvgy415_srl
Disk group: dvgy415


v  dvgy415_srl   -              ENABLED ACTIVE   1998243840 SELECT   -        fsgen
pl dvgy415_srl-02 dvgy415_srl   ENABLED ACTIVE   1998243840 CONCAT   -        RW
sd dvgy415239-01 dvgy415_srl-02 dvgy415239 5376 285473280 0          EMC0_244 ENA
sd dvgy415240-01 dvgy415_srl-02 dvgy415240 5376 285473280 285473280  EMC0_245 ENA
sd dvgy415241-01 dvgy415_srl-02 dvgy415241 5376 285473280 570946560  EMC0_246 ENA
sd dvgy415242-01 dvgy415_srl-02 dvgy415242 5376 285473280 856419840  EMC0_247 ENA
sd dvgy415243-01 dvgy415_srl-02 dvgy415243 5376 285826560 1141893120 EMC0_248 ENA
sd dvgy415244-01 dvgy415_srl-02 dvgy415244 5376 285473280 1427719680 EMC0_249 ENA
sd dvgy415245-01 dvgy415_srl-02 dvgy415245 5376 285050880 1713192960 EMC0_250 ENA

Re-associate the SRL volume from the RVG

sunsrv02-dr:# vxvol -g dvgy415 aslog dvgy415_rvg dvgy415_srl

Re-attach the Rlink

sunsrv02-dr:# vxrlink -g dvgy415 att rlk_pdpd415Primary-vipvr_dvgy415_rvg

Check the state of the RVG.  It should be active and attached.

sunsrv02-dr:# vxprint -Pl
Disk group: dvgy415

Rlink:    rlk_pdpd415Primary-vipvr_dvgy415_rvg
info:     timeout=500 packet_size=8400 rid=0.2127
  latency_high_mark=10000 latency_low_mark=9950
  bandwidth_limit=none
state:    state=ACTIVE
  synchronous=off latencyprot=off srlprot=autodcm
assoc:    rvg=dvgy415_rvg
  remote_host=pdpd415Primary-vipvr IP_addr=10.11.196.191 port=4145
  remote_dg=dvgy415
  remote_dg_dgid=1138140445.1393.sunsrv01
  remote_rvg_version=21
  remote_rlink=rlk_pdpd415dr-vipvr_dvgy415_rvg
  remote_rlink_rid=0.2007
  local_host=pdpd415dr-vipvr IP_addr=10.12.94.191 port=4145
protocol: UDP/IP
flags:    write enabled attached consistent connected

14. Resize the Replicated Volumes

Resize the volume using vradmin and run it from the primary site – Primary.  The change will be applied on both the 

primary and the secondary site – Primary and DR.

sunsrv01:# vradmin -g dvgy415 resizevol dvgy415_rvg db 1361g
Message from Host pdpd415dr-vipvr:
VxVM vxassist WARNING V-5-1-9592 DCM log size is smaller than recommended due to increased volume size.

The DCM log will be fixed later.

Verify that both sites have the same size and both in ENABLED ACTIVE state.

sunsrv01:# vxprint -ht db
Disk group: dvgy415

v  db            dvgy415_rvg ENABLED    ACTIVE 2854223872 SELECT     -        fsgen
pl db-03         db          ENABLED    ACTIVE 2854225920 CONCAT     -        RW
sd dvgy41596-01  db-03       dvgy41596  5376   285473280  0          EMC0_139 ENA
sd dvgy41597-01  db-03       dvgy41597  5376   285473280  285473280  EMC0_140 ENA
sd dvgy41598-01  db-03       dvgy41598  5376   285473280  570946560  EMC0_142 ENA
sd dvgy41599-01  db-03       dvgy41599  5376   285473280  856419840  EMC0_143 ENA
sd dvgy415100-01 db-03       dvgy415100 5376   285473280  1141893120 EMC0_144 ENA
sd dvgy415101-01 db-03       dvgy415101 5376   285473280  1427366400 EMC0_146 ENA
sd dvgy415102-01 db-03       dvgy415102 5376   285473280  1712839680 EMC0_150 ENA
sd dvgy415103-01 db-03       dvgy415103 5376   285473280  1998312960 EMC0_153 ENA
sd dvgy415104-01 db-03       dvgy415104 5376   285473280  2283786240 EMC0_154 ENA
sd dvgy415105-01 db-03       dvgy415105 5376   284966400  2569259520 EMC0_164 ENA
pl db-04         db          ENABLED    ACTIVE LOGONLY    CONCAT     -        RW
sd dvgy415122-02 db-04       dvgy415122 576    512        LOG        EMC0_130 ENA
pl db-05         db          ENABLED    ACTIVE LOGONLY    CONCAT     -        RW
sd dvgy415123-02 db-05       dvgy415123 576    512        LOG        EMC0_162 ENA

sunsrv02-dr:# vxprint -ht db
Disk group: dvgy415

v  db            dvgy415_rvg ENABLED    ACTIVE 2854223872 SELECT     -        fsgen
pl db-03         db          ENABLED    ACTIVE 2854225920 CONCAT     -        RW
sd dvgy415201-01 db-03       dvgy415201 5376   285473280  0          EMC0_206 ENA
sd dvgy415202-01 db-03       dvgy415202 5376   285473280  285473280  EMC0_207 ENA
sd dvgy415203-01 db-03       dvgy415203 5376   285473280  570946560  EMC0_208 ENA
sd dvgy415204-01 db-03       dvgy415204 5376   285473280  856419840  EMC0_209 ENA
sd dvgy415205-01 db-03       dvgy415205 5376   285473280  1141893120 EMC0_210 ENA
sd dvgy415206-01 db-03       dvgy415206 5376   285473280  1427366400 EMC0_211 ENA
sd dvgy415207-01 db-03       dvgy415207 5376   285473280  1712839680 EMC0_212 ENA
sd dvgy415208-01 db-03       dvgy415208 5376   285473280  1998312960 EMC0_213 ENA
sd dvgy415209-01 db-03       dvgy415209 5376   285473280  2283786240 EMC0_214 ENA
sd dvgy415210-01 db-03       dvgy415210 5376   284966400  2569259520 EMC0_215 ENA
pl db-04         db          ENABLED    ACTIVE LOGONLY    CONCAT     -        RW
sd dvgy415199-02 db-04       dvgy415199 64     64         LOG        EMC0_256 ENA
pl db-05         db          ENABLED    ACTIVE LOGONLY    CONCAT     -        RW
sd dvgy415200-02 db-05       dvgy415200 64     64         LOG        EMC0_257 ENA

Check the status of the Rlink

sunsrv01:# vxrlink -g dvgy415 status rlk_pdpd415dr-vipvr_dvgy415_rvg
Fri Dec 14 22:16:21 MST 2007
VxVM VVR vxrlink INFO V-5-1-4467 Rlink rlk_pdpd415dr-vipvr_dvgy415_rvg is up to date

Verify that the filesystem has the new size

sunsrv01:# df -k /dev/vx/dsk/dvgy415/db
Filesystem             kbytes     used     avail    capacity  Mounted on
/dev/vx/dsk/dvgy415/db 1427111936 30333310 1309480015     3%    /db/pdpd415/PEMMP00P/NODE0000

Follow the same steps for the rest of the volumes in the RVG

sunsrv01:# df -k /dev/vx/dsk/dvgy415/lg1

Filesystem            kbytes    used   avail capacity  Mounted on
/dev/vx/dsk/dvgy415/lg1
             198180864 3231502 182765091     2%    /db/pdpd415/log1

sunsrv01:# vxprint -ht lg1
Disk group: dvgy415

v  lg1           dvgy415_rvg ENABLED    ACTIVE 396361728 SELECT    -        fsgen
pl lg1-03        lg1         ENABLED    ACTIVE 396364800 CONCAT    -        RW
sd dvgy415106-01 lg1-03      dvgy415106 5376   285473280 0         EMC0_148 ENA
sd dvgy415107-01 lg1-03      dvgy415107 5376   110891520 285473280 EMC0_155 ENA
pl lg1-04        lg1         ENABLED    ACTIVE LOGONLY   CONCAT    -        RW
sd dvgy415122-05 lg1-04      dvgy415122 2496   512       LOG       EMC0_130 ENA
pl lg1-05        lg1         ENABLED    ACTIVE LOGONLY   CONCAT    -        RW
sd dvgy415123-05 lg1-05      dvgy415123 2496   512       LOG       EMC0_162 ENA

sunsrv01:# vradmin -g dvgy415 resizevol dvgy415_rvg lg1 +460054528
Message from Host pdpd415dr-vipvr:
VxVM vxassist WARNING V-5-1-9592 DCM log size is smaller than recommended due to increased volume size.

sunsrv01:# df -k /dev/vx/dsk/dvgy415/lg1
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/vx/dsk/dvgy415/lg1
             428208128 3287884 398362793     1%    /db/pdpd415/log1

sunsrv01:# vxprint -ht lg1
Disk group: dvgy415

v  lg1           dvgy415_rvg ENABLED    ACTIVE   856416256 SELECT    -        fsgen
pl lg1-03        lg1         ENABLED    ACTIVE   856419840 CONCAT    -        RW
sd dvgy415106-01 lg1-03      dvgy415106 5376     285473280 0         EMC0_148 ENA
sd dvgy415107-01 lg1-03      dvgy415107 5376     285473280 285473280 EMC0_155 ENA
sd dvgy415108-01 lg1-03      dvgy415108 5376     285473280 570946560 EMC0_156 ENA
pl lg1-04        lg1         ENABLED    ACTIVE   LOGONLY   CONCAT    -        RW
sd dvgy415122-05 lg1-04      dvgy415122 2496     512       LOG       EMC0_130 ENA
pl lg1-05        lg1         ENABLED    ACTIVE   LOGONLY   CONCAT    -        RW
sd dvgy415123-05 lg1-05      dvgy415123 2496     512       LOG       EMC0_162 ENA

sunsrv02-dr:# vxprint -ht lg1
Disk group: dvgy415

v  lg1           dvgy415_rvg ENABLED    ACTIVE   856416256 SELECT    -        fsgen
pl lg1-03        lg1         ENABLED    ACTIVE   856419840 CONCAT    -        RW
sd dvgy415211-01 lg1-03      dvgy415211 5376     285473280 0         EMC0_216 ENA
sd dvgy415212-01 lg1-03      dvgy415212 5376     285473280 285473280 EMC0_217 ENA
sd dvgy415213-01 lg1-03      dvgy415213 5376     285473280 570946560 EMC0_218 ENA
pl lg1-04        lg1         ENABLED    ACTIVE   LOGONLY   CONCAT    -        RW
sd dvgy415199-05 lg1-04      dvgy415199 256      64        LOG       EMC0_256 ENA
pl lg1-05        lg1         ENABLED    ACTIVE   LOGONLY   CONCAT    -        RW
sd dvgy415200-05 lg1-05      dvgy415200 256      64        LOG       EMC0_257 ENA

sunsrv01:# df -k /dev/vx/dsk/dvgy415/tp01
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/vx/dsk/dvgy415/tp01
             96468992   40177 90402085     1%    /db/pdpd415/PEMMP00P/tempspace01/NODE0000
sunsrv01:# vxprint -ht tp01
Disk group: dvgy415

v  tp01          dvgy415_rvg  ENABLED    ACTIVE   192937984 SELECT   -        fsgen
pl tp01-03       tp01         ENABLED    ACTIVE   192944640 CONCAT   -        RW
sd dvgy415111-01 tp01-03      dvgy415111 5376     192944640 0        EMC0_161 ENA
pl tp01-04       tp01         ENABLED    ACTIVE   LOGONLY   CONCAT   -        RW
sd dvgy415122-06 tp01-04      dvgy415122 3456     512       LOG      EMC0_130 ENA
pl tp01-05       tp01         ENABLED    ACTIVE   LOGONLY   CONCAT   -        RW
sd dvgy415123-06 tp01-05      dvgy415123 3456     512       LOG      EMC0_162 ENA

sunsrv01:# vxrlink -g dvgy415 status rlk_pdpd415dr-vipvr_dvgy415_rvg
Fri Dec 14 22:44:56 MST 2007
VxVM VVR vxrlink INFO V-5-1-4467 Rlink rlk_pdpd415dr-vipvr_dvgy415_rvg is up to date

sunsrv01:# vradmin -g dvgy415 resizevol dvgy415_rvg tp01 +663474176
Message from Host pdpd415dr-vipvr:
VxVM vxassist WARNING V-5-1-9592 DCM log size is smaller than recommended due to increased volume size.


sunsrv01:# df -k /dev/vx/dsk/dvgy415/tp01
Filesystem            kbytes    used   avail capacity  Mounted on
/dev/vx/dsk/dvgy415/tp01
             428206080  121489 401329375     1%    /db/pdpd415/PEMMP00P/tempspace01/NODE0000

sunsrv01:# vxprint -ht tp01
Disk group: dvgy415

v  tp01          dvgy415_rvg ENABLED    ACTIVE 856412160 SELECT    -        fsgen
pl tp01-03       tp01        ENABLED    ACTIVE 856412160 CONCAT    -        RW
sd dvgy415111-01 tp01-03     dvgy415111 5376   285473280 0         EMC0_161 ENA
sd dvgy415112-01 tp01-03     dvgy415112 5376   285473280 285473280 EMC0_138 ENA
sd dvgy415113-01 tp01-03     dvgy415113 5376   285465600 570946560 EMC0_141 ENA
pl tp01-04       tp01        ENABLED    ACTIVE LOGONLY   CONCAT    -        RW
sd dvgy415122-06 tp01-04     dvgy415122 3456   512       LOG       EMC0_130 ENA
pl tp01-05       tp01        ENABLED    ACTIVE LOGONLY   CONCAT    -        RW
sd dvgy415123-06 tp01-05     dvgy415123 3456   512       LOG       EMC0_162 ENA


sunsrv02-dr:# vxprint -ht tp01
Disk group: dvgy415

v  tp01          dvgy415_rvg ENABLED    ACTIVE 856412160 SELECT    -        fsgen
pl tp01-03       tp01        ENABLED    ACTIVE 856412160 CONCAT    -        RW
sd dvgy415216-01 tp01-03     dvgy415216 5376   285473280 0         EMC0_221 ENA
sd dvgy415217-01 tp01-03     dvgy415217 5376   285473280 285473280 EMC0_222 ENA
sd dvgy415218-01 tp01-03     dvgy415218 5376   285465600 570946560 EMC0_223 ENA
pl tp01-04       tp01        ENABLED    ACTIVE LOGONLY   CONCAT    -        RW
sd dvgy415199-06 tp01-04     dvgy415199 320    64        LOG       EMC0_256 ENA
pl tp01-05       tp01        ENABLED    ACTIVE LOGONLY   CONCAT    -        RW
sd dvgy415200-06 tp01-05     dvgy415200 320    64        LOG       EMC0_257 ENA

15. Resize the DCM Logs in DR.

This is necessary because the original logs were too small for the new volume sizes.

Display the current LOG sizes.

sunsrv02-dr:# vxprint -tg dvgy415 | grep " LOG "

sd dvgy415199-01 bcp-04      dvgy415199 0      64       LOG       EMC0_256 ENA
sd dvgy415199-02 db-04       dvgy415199 64     64       LOG       EMC0_256 ENA
sd dvgy415199-03 dba-04      dvgy415199 128    64       LOG       EMC0_256 ENA
sd dvgy415199-04 db2-04      dvgy415199 192    64       LOG       EMC0_256 ENA
sd dvgy415199-05 lg1-04      dvgy415199 256    64       LOG       EMC0_256 ENA
sd dvgy415199-06 tp01-04     dvgy415199 320    64       LOG       EMC0_256 ENA

sd dvgy415200-01 bcp-05      dvgy415200 0      64       LOG       EMC0_257 ENA
sd dvgy415200-02 db-05       dvgy415200 64     64       LOG       EMC0_257 ENA
sd dvgy415200-03 dba-05      dvgy415200 128    64       LOG       EMC0_257 ENA
sd dvgy415200-04 db2-05      dvgy415200 192    64       LOG       EMC0_257 ENA
sd dvgy415200-05 lg1-05      dvgy415200 256    64       LOG       EMC0_257 ENA
sd dvgy415200-06 tp01-05     dvgy415200 320    64       LOG       EMC0_257 ENA

Turn off the SRL protection

sunsrv02-dr:# vxedit -g dvgy415 set srlprot=off rlk_pdpd415Primary-vipvr_dvgy415_rvg

Verify the state of srlprot

sunsrv02-dr:# vxprint -Pl
Disk group: dvgy415

Rlink:    rlk_pdpd415Primary-vipvr_dvgy415_rvg
info:     timeout=500 packet_size=8400 rid=0.2127
  latency_high_mark=10000 latency_low_mark=9950
  bandwidth_limit=none
state:    state=ACTIVE
  synchronous=off latencyprot=off srlprot=off
assoc:    rvg=dvgy415_rvg
  remote_host=pdpd415Primary-vipvr IP_addr=10.11.196.191 port=4145
  remote_dg=dvgy415
  remote_dg_dgid=1138140445.1393.sunsrv01
  remote_rvg_version=21
  remote_rlink=rlk_pdpd415dr-vipvr_dvgy415_rvg
  remote_rlink_rid=0.2007
  local_host=pdpd415dr-vipvr IP_addr=10.12.94.191 port=4145
protocol: UDP/IP
flags:    write enabled attached consistent connected

Remove the DCM Logs on resized volumes.  Run it twice because each volume has 2 logs.  This is a different method than 

step 9.

sunsrv02-dr:# vxassist -g dvgy415 remove log tp01
sunsrv02-dr:# vxassist -g dvgy415 remove log tp01
sunsrv02-dr:# vxassist -g dvgy415 remove log lg1
sunsrv02-dr:# vxassist -g dvgy415 remove log lg1
sunsrv02-dr:# vxassist -g dvgy415 remove log db
sunsrv02-dr:# vxassist -g dvgy415 remove log db

Recreate the DCM Logs on resized volumes.

sunsrv02-dr:# vxassist -g dvgy415 addlog bcp logtype=dcm nlog=2 dvgy415199 dvgy415200
sunsrv02-dr:# vxassist -g dvgy415 addlog db  logtype=dcm nlog=2 dvgy415199 dvgy415200
sunsrv02-dr:# vxassist -g dvgy415 addlog dba logtype=dcm nlog=2 dvgy415199 dvgy415200
sunsrv02-dr:# vxassist -g dvgy415 addlog db2 logtype=dcm nlog=2 dvgy415199 dvgy415200
sunsrv02-dr:# vxassist -g dvgy415 addlog lg1 logtype=dcm nlog=2 dvgy415199 dvgy415200
sunsrv02-dr:# vxassist -g dvgy415 addlog tp01 logtype=dcm nlog=2 dvgy415199 dvgy415200

Display the new LOG sizes.

sunsrv02-dr:# vxprint -tg dvgy415 | grep " LOG "

sd dvgy415199-01 bcp-01      dvgy415199 0      512      LOG       EMC0_256 ENA
sd dvgy415199-02 db-01       dvgy415199 576    512      LOG       EMC0_256 ENA
sd dvgy415199-03 dba-01      dvgy415199 1536   512      LOG       EMC0_256 ENA
sd dvgy415199-04 db2-01      dvgy415199 1088   132      LOG       EMC0_256 ENA
sd dvgy415199-05 lg1-01      dvgy415199 2496   512      LOG       EMC0_256 ENA
sd dvgy415199-06 tp01-01     dvgy415199 3456   512      LOG       EMC0_256 ENA

sd dvgy415200-01 bcp-02      dvgy415200 0      512      LOG       EMC0_257 ENA
sd dvgy415200-02 db-02       dvgy415200 576    512      LOG       EMC0_257 ENA
sd dvgy415200-03 dba-02      dvgy415200 1536   512      LOG       EMC0_257 ENA
sd dvgy415200-04 db2-02      dvgy415200 1088   132      LOG       EMC0_257 ENA
sd dvgy415200-05 lg1-02      dvgy415200 2496   512      LOG       EMC0_257 ENA
sd dvgy415200-06 tp01-02     dvgy415200 3456   512      LOG       EMC0_257 ENA

Set the SRL protection back to autodcm

sunsrv02-dr:# vxedit -g dvgy415 set srlprot=autodcm rlk_pdpd415Primary-vipvr_dvgy415_rvg

Display the RVG status and the Rlink status

sunsrv02-dr:# vxprint -Pl
Disk group: dvgy415

Rlink:    rlk_pdpd415Primary-vipvr_dvgy415_rvg
info:     timeout=500 packet_size=8400 rid=0.2127
  latency_high_mark=10000 latency_low_mark=9950
  bandwidth_limit=none
state:    state=ACTIVE
  synchronous=off latencyprot=off srlprot=autodcm
assoc:    rvg=dvgy415_rvg
  remote_host=pdpd415Primary-vipvr IP_addr=10.11.196.191 port=4145
  remote_dg=dvgy415
  remote_dg_dgid=1138140445.1393.sunsrv01
  remote_rvg_version=21
  remote_rlink=rlk_pdpd415dr-vipvr_dvgy415_rvg
  remote_rlink_rid=0.2007
  local_host=pdpd415dr-vipvr IP_addr=10.12.94.191 port=4145
protocol: UDP/IP
flags:    write enabled attached consistent connected


16. Prepare the DR Volumes for Snapshot

First, display the volume information

sunsrv02-dr:# vxprint -ht bcp
Disk group: dvgy415

v  bcp           dvgy415_rvg ENABLED    ACTIVE 585105408 SELECT    -        fsgen
pl bcp-01        bcp         ENABLED    ACTIVE LOGONLY   CONCAT    -        RW
sd dvgy415199-01 bcp-01      dvgy415199 0      512       LOG       EMC0_256 ENA
pl bcp-02        bcp         ENABLED    ACTIVE LOGONLY   CONCAT    -        RW
sd dvgy415200-01 bcp-02      dvgy415200 0      512       LOG       EMC0_257 ENA
pl bcp-03        bcp         ENABLED    ACTIVE 585108480 CONCAT    -        RW
sd dvgy415214-01 bcp-03      dvgy415214 5376   285473280 0         EMC0_219 ENA
sd dvgy415215-01 bcp-03      dvgy415215 5376   285473280 285473280 EMC0_220 ENA
sd dvgy415219-01 bcp-03      dvgy415219 5376   14161920  570946560 EMC0_224 ENA

Prepare the volume for snapshot by adding a DCO log.  The same disks is used for DCO and DCM logs.

sunsrv02-dr:# vxsnap -g dvgy415 prepare bcp ndcomirs=2 alloc=dvgy415199,dvgy415200
VxVM vxsnap INFO V-5-1-9270 Volume is under RVG, setting drl=no.

Display the volume information to see the changes.  Note that the DCO volume has been added with two logs.

sunsrv02-dr:# vxprint -ht bcp
Disk group: dvgy415

v  bcp           dvgy415_rvg ENABLED    ACTIVE 585105408 SELECT    -        fsgen
pl bcp-01        bcp         ENABLED    ACTIVE LOGONLY   CONCAT    -        RW
sd dvgy415199-01 bcp-01      dvgy415199 0      512       LOG       EMC0_256 ENA
pl bcp-02        bcp         ENABLED    ACTIVE LOGONLY   CONCAT    -        RW
sd dvgy415200-01 bcp-02      dvgy415200 0      512       LOG       EMC0_257 ENA
pl bcp-03        bcp         ENABLED    ACTIVE 585108480 CONCAT    -        RW
sd dvgy415214-01 bcp-03      dvgy415214 5376   285473280 0         EMC0_219 ENA
sd dvgy415215-01 bcp-03      dvgy415215 5376   285473280 285473280 EMC0_220 ENA
sd dvgy415219-01 bcp-03      dvgy415219 5376   14161920  570946560 EMC0_224 ENA
dc bcp_dco       bcp         bcp_dcl
v  bcp_dcl       -           ENABLED    ACTIVE 40368     SELECT    -        gen
pl bcp_dcl-01    bcp_dcl     ENABLED    ACTIVE 40368     CONCAT    -        RW
sd dvgy415199-07 bcp_dcl-01  dvgy415199 4416   40368     0         EMC0_256 ENA
pl bcp_dcl-02    bcp_dcl     ENABLED    ACTIVE 40368     CONCAT    -        RW
sd dvgy415200-07 bcp_dcl-02  dvgy415200 4416   40368     0         EMC0_257 ENA

Run the same steps on the rest of the volumes

sunsrv02-dr:# vxsnap -g dvgy415 prepare db  ndcomirs=2  alloc=dvgy415199,dvgy415200
VxVM vxsnap INFO V-5-1-9270 Volume is under RVG, setting drl=no.

sunsrv02-dr:# vxsnap -g dvgy415 prepare dba ndcomirs=2  alloc=dvgy415199,dvgy415200
VxVM vxsnap INFO V-5-1-9270 Volume is under RVG, setting drl=no.

sunsrv02-dr:# vxsnap -g dvgy415 prepare db2 ndcomirs=2  alloc=dvgy415199,dvgy415200
VxVM vxsnap INFO V-5-1-9270 Volume is under RVG, setting drl=no.

sunsrv02-dr:# vxsnap -g dvgy415 prepare lg1 ndcomirs=2  alloc=dvgy415199,dvgy415200
VxVM vxsnap INFO V-5-1-9270 Volume is under RVG, setting drl=no.

sunsrv02-dr:# vxsnap -g dvgy415 prepare tp01 ndcomirs=2 alloc=dvgy415199,dvgy415200
VxVM vxsnap INFO V-5-1-9270 Volume is under RVG, setting drl=no.


17. Resize the SNAP Volumes in DR

If you do not want to preserve the data on the snapshot volumes, you may recreate all the snapshot volumes instead of 

resize.

Resize the SNAP Volumes using the exact size as the replicated volumes

sunsrv02-dr:# /etc/vx/bin/vxresize -g dvgy415 db_snapvol   1361g      dvgy415224 \
                             dvgy415225 dvgy415226 dvgy415227 dvgy415228
sunsrv02-dr:# /etc/vx/bin/vxresize -g dvgy415 lg1_snapvol  +460054528 dvgy415231 \
                                                              dvgy415232

sunsrv02-dr:# /etc/vx/bin/vxresize -g dvgy415 tp01_snapvol +663474176 dvgy415235 \
                                                   dvgy415236 dvgy415237

sunsrv02-dr:# df -k | grep snapvol

/dev/vx/dsk/dvgy415/db_snapvol   1427111936 3555377 1334584344     1%    /db/pdpd415/PEMMP00P/NODE0000
/dev/vx/dsk/dvgy415/lg1_snapvol   428208128 3078084  398559480     1%    /db/pdpd415/log1
/dev/vx/dsk/dvgy415/tp01_snapvol  428206080  121489  401329375     1%    /db/pdpd415/PEMMP00P/tempspace01/NODE0000


18. Prepare the SNAP Volumes in DR

Identify the region size of the main volumes.  The region size must be the same for both the main volume and the snap 

volume.

sunsrv02-dr:# for DCONAME in `vxprint -tg dvgy415 | grep "^dc" | awk '{ print $2 }'`
> do
>   echo "$DCONAME\t\c"
>   vxprint -g dvgy415 -F%regionsz $DCONAME
> done
bcp_dco   128
db_dco    128
dba_dco   128
db2_dco   128
lg1_dco   128
tp01_dco  128

Display the volume information

sunsrv02-dr:# vxprint -ht bcp_snapvol
Disk group: dvgy415

v  bcp_snapvol    -              ENABLED  ACTIVE 585105408 SELECT    -        fsgen
pl bcp_snapvol-02 bcp_snapvol    ENABLED  ACTIVE 585108480 CONCAT    -        RW
sd dvgy415233-01  bcp_snapvol-02 dvgy415233 5376 285473280 0         EMC0_238 ENA
sd dvgy415234-01  bcp_snapvol-02 dvgy415234 5376 285473280 285473280 EMC0_239 ENA
sd dvgy415238-01  bcp_snapvol-02 dvgy415238 5376 14161920 570946560  EMC0_243 ENA

Prepare the snapshot volume by adding a DCO log and the same region size as the main volume.  The same disks is used 

for DCO and DCM logs.

sunsrv02-dr:# vxsnap -g dvgy415 prepare bcp_snapvol ndcomirs=2 regionsize=128 \
      alloc=dvgy415199,dvgy415200

Display the volume information to see the changes.  Note that the DCO volume has been added with two logs.

sunsrv02-dr:# vxprint -ht bcp_snapvol
Disk group: dvgy415

v  bcp_snapvol  -            ENABLED  ACTIVE   585105408 SELECT   -        fsgen
pl bcp_snapvol-02 bcp_snapvol ENABLED ACTIVE   585108480 CONCAT   -        RW
sd dvgy415233-01 bcp_snapvol-02 dvgy415233 5376 285473280 0       EMC0_238 ENA
sd dvgy415234-01 bcp_snapvol-02 dvgy415234 5376 285473280 285473280 EMC0_239 ENA
sd dvgy415238-01 bcp_snapvol-02 dvgy415238 5376 14161920 570946560 EMC0_243 ENA
dc bcp_snapvol_dco bcp_snapvol bcp_snapvol_dcl
v  bcp_snapvol_dcl -         ENABLED  ACTIVE   40368    SELECT    -        gen
pl bcp_snapvol_dcl-01 bcp_snapvol_dcl ENABLED ACTIVE 40368 CONCAT -        RW
sd dvgy415199-13 bcp_snapvol_dcl-01 dvgy415199 370176 40368 0     EMC0_256 ENA
pl bcp_snapvol_dcl-02 bcp_snapvol_dcl ENABLED ACTIVE 40368 CONCAT -        RW
sd dvgy415200-13 bcp_snapvol_dcl-02 dvgy415200 370176 40368 0     EMC0_257 ENA

Run the same steps on the rest of the volumes

sunsrv02-dr:# vxsnap -g dvgy415 prepare db_snapvol ndcomirs=2 regionsize=128 \
      alloc=dvgy415199,dvgy415200
sunsrv02-dr:# vxsnap -g dvgy415 prepare dba_snapvol ndcomirs=2 regionsize=128 \
      alloc=dvgy415199,dvgy415200
sunsrv02-dr:# vxsnap -g dvgy415 prepare db2_snapvol ndcomirs=2 regionsize=128 \
      alloc=dvgy415199,dvgy415200
sunsrv02-dr:# vxsnap -g dvgy415 prepare lg1_snapvol ndcomirs=2 regionsize=128 \
      alloc=dvgy415199,dvgy415200
sunsrv02-dr:# vxsnap -g dvgy415 prepare tp01_snapvol ndcomirs=2 regionsize=128 \
      alloc=dvgy415199,dvgy415200

Verify the region sizes are the same

sunsrv02-dr:# for DCONAME in `vxprint -tg dvgy415 | grep "^dc" | awk '{ print $2 }'`
> do
>   echo "$DCONAME\t\c"
>   vxprint -g dvgy415 -F%regionsz $DCONAME
> done
bcp_dco           128
bcp_snapvol_dco   128
db_dco            128
db_snapvol_dco    128
dba_dco           128
dba_snapvol_dco   128
db2_dco           128
db2_snapvol_dco   128
lg1_dco           128
lg1_snapvol_dco   128
tp01_dco          128
tp01_snapvol_dco  128

19. Run a point-in-time snapshot in DR (This will overwrite your previous data on the snap 

volumes)

Verify the RLINK and RVG are active and up to date

sunsrv01:# vxrlink -g dvgy415 status rlk_pdpd415dr-vipvr_dvgy415_rvg
Mon Dec 17 17:40:19 MST 2007
VxVM VVR vxrlink INFO V-5-1-4467 Rlink rlk_pdpd415dr-vipvr_dvgy415_rvg is up to date

sunsrv02-dr:# vxprint -Pl
Disk group: dvgy415

Rlink:    rlk_pdpd415Primary-vipvr_dvgy415_rvg
info:     timeout=500 packet_size=8400 rid=0.2127
  latency_high_mark=10000 latency_low_mark=9950
  bandwidth_limit=none
state:    state=ACTIVE
  synchronous=off latencyprot=off srlprot=autodcm
assoc:    rvg=dvgy415_rvg
  remote_host=pdpd415Primary-vipvr IP_addr=10.11.196.191 port=4145
  remote_dg=dvgy415
  remote_dg_dgid=1138140445.1393.sunsrv01
  remote_rvg_version=21
  remote_rlink=rlk_pdpd415dr-vipvr_dvgy415_rvg
  remote_rlink_rid=0.2007
  local_host=pdpd415dr-vipvr IP_addr=10.12.94.191 port=4145
protocol: UDP/IP
flags:    write enabled attached consistent connected

Start the Snap Copy

sunsrv02-dr:# vxsnap -g dvgy415 make source=bcp/snapvol=bcp_snapvol \
      source=db/snapvol=db_snapvol source=dba/snapvol=dba_snapvol \
      source=db2/snapvol=db2_snapvol source=lg1/snapvol=lg1_snapvol \
      source=tp01/snapvol=tp01_snapvol

Display the sync status

sunsrv02-dr:# vxtask list
TASKID  PTID TYPE/STATE    PCT   PROGRESS
286         SNAPSYNC/R 00.02% 0/585105408/118784 SNAPSYNC bcp_snapvol  dvgy415
288         SNAPSYNC/R 00.00% 0/2854223872/76160 SNAPSYNC db_snapvol   dvgy415
290         SNAPSYNC/R 00.13% 0/98566144/126976  SNAPSYNC dba_snapvol  dvgy415
292         SNAPSYNC/R 01.46% 0/8388608/122624   SNAPSYNC db2_snapvol  dvgy415
294         SNAPSYNC/R 00.01% 0/856416256/108544 SNAPSYNC lg1_snapvol  dvgy415
296         SNAPSYNC/R 00.01% 0/856412160/122880 SNAPSYNC tp01_snapvol dvgy415

sunsrv02-dr:# vxsnap -g dvgy415 print

NAME          SNAPOBJECT        TYPE     PARENT   SNAPSHOT    %DIRTY    %VALID

bcp           --                volume   --       --           --       100.00
      bcp_snapvol_snp   volume   --       bcp_snapvol  0.00     --

db            --                volume   --       --           --       100.00
      db_snapvol_snp    volume   --       db_snapvol   0.00     --

dba           --                volume   --       --           --       100.00
      dba_snapvol_snp   volume   --       dba_snapvol  0.00     --

db2           --                volume   --       --           --       100.00
      db2_snapvol_snp   volume   --       db2_snapvol  0.00     --

lg1           --                volume   --       --           --       100.00
      lg1_snapvol_snp   volume   --       lg1_snapvol  0.00     --

tp01          --                volume   --       --           --       100.00
      tp01_snapvol_snp  volume   --       tp01_snapvol 0.00     --


bcp_snapvol   bcp_snp           volume   bcp      --           0.00     0.11

db_snapvol    db_snp            volume   db       --           0.00     0.02

dba_snapvol   dba_snp           volume   dba      --           0.00     0.77

db2_snapvol   db2_snp           volume   db2      --           0.00     9.13

lg1_snapvol   lg1_snp           volume   lg1      --           0.00     0.09

tp01_snapvol  tp01_snp          volume   tp01     --           0.00     0.10



When the Sync is complete, bring up the snapvol service group in VCS to verify all changes

sunsrv02-dr:# vxtask list
TASKID  PTID TYPE/STATE    PCT   PROGRESS


sunsrv02-dr:# hastatus -sum

-- SYSTEM STATE
-- System               State                Frozen

A  sunsrv01-dr          RUNNING              0
A  sunsrv02-dr          RUNNING              0

-- GROUP STATE
-- Group            System               Probed     AutoDisabled    State

B  CCAvail          sunsrv01-dr          Y          N               ONLINE
B  CCAvail          sunsrv02-dr          Y          N               OFFLINE
B  pdpd415_grp      sunsrv01-dr          Y          N               OFFLINE
B  pdpd415_grp      sunsrv02-dr          Y          N               OFFLINE
B  pdpd415_vvr      sunsrv01-dr          Y          N               OFFLINE
B  pdpd415_vvr      sunsrv02-dr          Y          N               ONLINE
B  snap_pdpd415_grp sunsrv01-dr          Y          N               OFFLINE
B  snap_pdpd415_grp sunsrv02-dr          Y          N               OFFLINE


sunsrv02-dr:# hagrp -online snap_pdpd415_grp -sys sunsrv02-dr


sunsrv02-dr:# hastatus -sum

-- SYSTEM STATE
-- System               State                Frozen

A  sunsrv01-dr          RUNNING              0
A  sunsrv02-dr          RUNNING              0

-- GROUP STATE
-- Group           System               Probed     AutoDisabled    State

B  CCAvail          sunsrv01-dr          Y          N               ONLINE
B  CCAvail          sunsrv02-dr          Y          N               OFFLINE
B  pdpd415_grp      sunsrv01-dr          Y          N               OFFLINE
B  pdpd415_grp      sunsrv02-dr          Y          N               OFFLINE
B  pdpd415_vvr      sunsrv01-dr          Y          N               OFFLINE
B  pdpd415_vvr      sunsrv02-dr          Y          N               ONLINE
B  snap_pdpd415_grp sunsrv01-dr          Y          N               OFFLINE
B  snap_pdpd415_grp sunsrv02-dr          Y          N               ONLINE


Verify the filesystems and compare the sizes with the primary site

sunsrv02-dr:# df -k | grep snapvol

/dev/vx/dsk/dvgy415/bcp_snapvol  292552704  2059883  272337156   1%    /bcp/pdpd415

/dev/vx/dsk/dvgy415/db_snapvol  1427111936 30338312 1309475336   3%    /db/pdpd415/PEMMP00P/NODE0000

/dev/vx/dsk/dvgy415/db2_snapvol    4194304    79861    3857337   3%    /db2/pdpd415

/dev/vx/dsk/dvgy415/dba_snapvol   49283072  6830907   39799049  15%    /dba/pdpd415

/dev/vx/dsk/dvgy415/lg1_snapvol  428208128  3171480  398471922   1%    /db/pdpd415/log1

/dev/vx/dsk/dvgy415/tp01_snapvol 428206080   121489  401329375   1%    /db/pdpd415/PEMMP00P/tempspace01/NODE0000



sunsrv01:# df -k | grep dvgy415 | sort

/dev/vx/dsk/dvgy415/bcp          292552704  2059883  272337156   1%    /bcp/pdpd415

/dev/vx/dsk/dvgy415/db          1427111936 30337038 1309476520   3%    /db/pdpd415/PEMMP00P/NODE0000

/dev/vx/dsk/dvgy415/db2            4194304    79401    3857768   3%    /db2/pdpd415

/dev/vx/dsk/dvgy415/dba           49283072  6830907   39799049  15%    /dba/pdpd415

/dev/vx/dsk/dvgy415/lg1          428208128  3183012  398461110   1%    /db/pdpd415/log1

/dev/vx/dsk/dvgy415/tp01         428206080   121489  401329375   1%    /db/pdpd415/PEMMP00P/tempspace01/NODE0000

Name	align="center">Description
VVR	Veritas Volume Replicator. A software that replicates data to remote locations over an IP network for maximum business continuity.
RVG	Replication Volume Group. An RVG is a subset of volumes within a given VxVM disk group configured for replication to one or more secondary systems. Data is replicated from a primary RVG to a secondary RVG. The primary RVG is in use by an application, while the secondary RVG receives replication and writes to local disk. The concept of primary and secondary is per RVG, not per system. A system can simultaneously be a primary RVG for some RVGs and secondary RVG for others. The RVG also contains the storage replicator log (SRL) and replication link (RLINK).
width="60"> SRL	Storage Replication Log. All data writes destined for volumes configured for replication are first persistently queued in a log called the Storage Replicator Log. VVR implements the SRL at the primary side to store all changes for transmission to the secondary(s). The SRL is a VxVM volume configured as part of an RVG. The SRL gives VVR the ability to associate writes to specific volumes within the replicated configuration in a specific order, maintaining write order fidelity at the secondary. All writes sent to the VxVM volume layer, whether from an application such as a database writing directly to storage, or an application accessing storage via a file system are faithfully replicated in application write order to the secondary.
RLink	An Rlink is a VVR Replication Link to a secondary RVG. Each RLINK on a Primary RVG represents the communication link from the Primary RVG to a corresponding Secondary RVG, via an IP connection.
DCM	width="540"> Data Change Maps (DCM) are used to mark sections of volumes on the primary that have changed during extended network outages in order to minimize the amount of data that must be synchronized to the secondary site during the outage.
DCO	Disk Change Object (DCO) volumes are used by Volume Manager FastResync (FR). FR is used to quickly resynchronize mirrored volumes that have been temporarily split and rejoined. FR works by copying only changes to the newly reattached volume using FR logging. This process reduces the time required to rejoin a split mirror, and requires less processing power than full mirror resynchronization without logging.

`<diskgroup>`	is the VxVM diskgroup name
`<RVGname>`	is the name for the RVG, usually <diskgroupname>_rvg
`<vol1,vol2...>`	is a comma separated list of all volumes in the diskgroup to be replicated.
`<SRLname>`	is the name of the SRL volume