Linux LVM with Veritas DMP as multipathing

"dmp_native_support" parameter is required to turn on in order for the DMP to provide Linux LVM VG multipathing.

# vxdmpadm settune dmp_native_support=on
# vxdmpadm gettune  |grep support
dmp_native_support                       on              off
# lvmdiskscan |grep emc |grep LVM
  /dev/vx/dmp/emc_clariion0_1000   [       15.00 GB] LVM physical volume
  /dev/vx/dmp/emc0_05ff            [        1.01 GB] LVM physical volume
  /dev/vx/dmp/emc0_05fe            [        1.01 GB] LVM physical volume
  /dev/vx/dmp/emc0_05fd            [        1.01 GB] LVM physical volu

Linux Storage Foundatione Oracle RAC diskgroup and volume failure.

When all paths are disconnected/reconnected to SF Oracle RAC cluster node, some strange behaviour might occur. The vmax_sfracdg diskgroup appeared to be stale,  some disks like emc_0604 was showing 2 entries in the diskgroup. The diskgroup failed to deport and the volume denied to be stopped. The volume was still active in the kernel as explained in Symantec site. Reboot system is the only solution.


# vxdisk list -o alldgs |grep vmax_sfrac
emc0_0604    auto:cdsdisk    emcpoweral   vmax_sfracdg online shared
emc0_0604    auto:cdsdisk    -            (vmax_sfracdg) online shared
emc0_0605    auto:cdsdisk    emcpowerak   vmax_sfracdg online shared
emc0_0606    auto:cdsdisk    emcpoweraj   vmax_sfracdg online shared
emc0_0606    auto:cdsdisk    -            (vmax_sfracdg) online shared
emc0_0608    auto:cdsdisk    emcpowerah   vmax_sfracdg online shared

# vxdg deport vmax_sfracdg
VxVM vxdg ERROR V-5-1-584 Disk group vmax_sfracdg: Some volumes in the disk group are in use

# vxvol -g vmax_sfracdg stopall
VxVM vxvol ERROR V-5-1-1220 Volume sfracvol1 is currently open or mounted
VxVM vxvol ERROR V-5-1-1220 Volume sfracvol2 is currently open or mounted
VxVM vxvol ERROR V-5-1-1220 Volume sfracvol3 is currently open or mounted
VxVM vxvol ERROR V-5-1-1220 Volume sfracvol4 is currently open or mounted
VxVM vxvol ERROR V-5-1-1220 Volume sfracvol5 is currently open or mounted

 #vxprint -g vmax_sfracdg -m sfracvol1 |grep devopen
        devopen=on

VCS NOTICE V-16-20011-1021 CFSMount error in Storage Foundation Oracle RAC

 If CFSMount is unable to mount the resources in the service group, it might be the previous mounting is still in stale state during the previous failure. Solution is to remove the stale mounting and online the resouce again.


# df -k |grep error
df: `/vmax_sfracfs1': Input/output error
df: `/vmax_sfracfs3': Input/output error
df: `/vmax_sfracfs5': Input/output error


/var/VRTSvcs/log/CFSMount_A.log
sgelxha196:/var/VRTSvcs/log # less CFSMount_A.log
2011/09/28 16:40:30 VCS NOTICE V-16-20011-1021 CFSMount:vmax_sfrac3:monitor:File System Disabled: MountPoint: /vmax_sfracfs3
2011/09/28 16:40:30 VCS NOTICE V-16-20011-1021 CFSMount:vmax_sfrac5:monitor:File System Disabled: MountPoint: /vmax_sfracfs5
2011/09/28 16:40:30 VCS NOTICE V-16-20011-1021 CFSMount:vmax_sfrac1:monitor:File System Disabled: MountPoint: /vmax_sfracfs1

Solution:
# umount -f /vmax_sfracfs1
# umount -f /vmax_sfracfs3
# umount -f /vmax_sfracfs5

# hagrp -online vmax_sfrac -sys host1

Verify key registration for Veritas disks

Sometime there are disks SCSI3-PR compliance or vxfentsthdw script failure for vxfencing. Excute below to verify the keys on all disks.

Command to check all the keys and SCSI-3 PR compliance disks for VCS
vxdisk list|grep emc0 |awk '{print "/dev/vx/rdmp/"$1}' > /tmp/disks
vxdisk list|grep emc0 |awk '{print "/dev/vx/dmp/"$1}' > /tmp/disks - (Try with dmp path only if rdmp fails)
vxfenadm -a -k 1 -f /tmp/disks - register key 1 to the all the disks
vxfenadm -g all -f /tmp/disks - list the key register
vxfenadm -s all -f /tmp/disks - clear the key

Eg. Command to query registeration keys in coordinator diskgroup
host85-root>vxdisk -o alldgs list |grep vxfencoorddg |awk '{print "/dev/vx/rdmp/"$1}' > /tmp/disks
host85-root>vxfenadm -s all -f /tmp/disks
Device Name: /dev/vx/rdmp/pp_emc0_4
Total Number Of Keys: 3
key[0]:
[Numeric Format]: 86,70,68,67,55,65,48,48
[Character Format]: VFDC7A00
* [Node Format]: Cluster ID: 56442 Node ID: 0 Node Name: host85
key[1]:
[Numeric Format]: 86,70,68,67,55,65,48,49
[Character Format]: VFDC7A01
* [Node Format]: Cluster ID: 56442 Node ID: 1 Node Name: host83
key[2]:
[Numeric Format]: 86,70,68,67,55,65,48,50
[Character Format]: VFDC7A02
* [Node Format]: Cluster ID: 56442 Node ID: 2 Node Name: host81
Device Name: /dev/vx/rdmp/pp_emc0_6
Total Number Of Keys: 3
key[0]:
[Numeric Format]: 86,70,68,67,55,65,48,48
[Character Format]: VFDC7A00
* [Node Format]: Cluster ID: 56442 Node ID: 0 Node Name: host85
key[1]:
[Numeric Format]: 86,70,68,67,55,65,48,49
[Character Format]: VFDC7A01
* [Node Format]: Cluster ID: 56442 Node ID: 1 Node Name: host83
key[2]:
[Numeric Format]: 86,70,68,67,55,65,48,50
[Character Format]: VFDC7A02
* [Node Format]: Cluster ID: 56442 Node ID: 2 Node Name: host81
Device Name: /dev/vx/rdmp/pp_emc0_5
Total Number Of Keys: 3
key[0]:
[Numeric Format]: 86,70,68,67,55,65,48,48
[Character Format]: VFDC7A00
* [Node Format]: Cluster ID: 56442 Node ID: 0 Node Name: host85
key[1]:
[Numeric Format]: 86,70,68,67,55,65,48,49
[Character Format]: VFDC7A01
* [Node Format]: Cluster ID: 56442 Node ID: 1 Node Name: host83
key[2]:
[Numeric Format]: 86,70,68,67,55,65,48,50
[Character Format]: VFDC7A02
* [Node Format]: Cluster ID: 56442 Node ID: 2 Node Name: host81

VCS ASM/Oracle agent for single oracle database instance with ASM

Steps to allow VCS Oracle Agent to manage Single Oracle Database Instance with ASM:

2. Oracle ASM and Database single instance setup

 This configuration consists of 3 VCS cluster nodes. VCS ASM and Oracle Agent will manage the ASM and Oracle resources across the 3 nodes with only 1 single node posses the single database instance at any point of time. 

2.1 Select and Install Grid Infrastructure Software only and execute below for standalone server. (1^st and subsequent nodes)

/u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/roothas.pl

2.2  Configure Oracle ASM with “asmca”. (1^st node only)

2.3 Select and Install Oracle Database Software only  (1^st and subsequent nodes)

2.4 Configure listener with “netca” (1^st and subsequent nodes)

2.5 Configure Oracle DB with “dbca” (1^st node only)

2.6 Create ASM and add ASM on remote host (subsequent nodes)

1st node (host85)
host85-oracle>export ORACLE_SID=+ASM
host85-oracle>export ORACLE_HOME=/u01/app/11.2.0/grid        
host85-oracle>asmcmd
ASMCMD> ls
REGISTRY.253.762436131
ASMCMD> spget +DATA/asm/ASMPARAMETERFILE/REGISTRY.253.762436131
+DATA/asm/asmparameterfile/registry.253.762436131
ASMCMD> spcopy +DATA/asm/ASMPARAMETERFILE/REGISTRY.253.762436131 /u01/app/11.2.0/grid/dbs/spfileASM.ora

2nd node (host83) and subsequent nodes (grid owner)
host85-oracle>crsctl stop resource ora.asm –f
host85-oracle>scp -p spfileASM.ora 10.10.10.83:/u01/app/11.2.0/grid/dbs
host83-oracle>srvctl add asm -l LISTENER –p\ /u01/app/11.2.0/grid/dbs/spfileASM.ora -d '/dev/rdisk/*'
host83-oracle>crsctl start resource ora.cssd
host83-oracle>crsctl start resource ora.asm
host83-oracle>export ORACLE_SID=+ASM
host83-oracle>export ORACLE_HOME=/u01/app/11.2.0/grid        
host83-oracle>sqlplus / as sysasm
SQL> alter diskgroup DATA mount;

host83-oracle>crsctl status resource –t  (Output display register ASM DG show in RED)
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS      
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       host83                                    
ora.LISTENER.lsnr
               ONLINE  ONLINE       host83                                    
ora.asm
               ONLINE  ONLINE       host83                 Started             
ora.ons
               OFFLINE OFFLINE      host83                                    
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       host83                                    
ora.diskmon
      1        ONLINE  ONLINE       host83                                    
ora.evmd
      1        ONLINE  ONLINE       host83        

2.7 Enable Oracle Cluster Synchronization Service daemon to start automatically (1^st and subsequent nodes)

host85-oracle>srvctl disable asm
host85-oracle>crsctl modify resource ora.asm -attr ENABLED=0
host85-oracle>crsctl modify resource ora.cssd -attr AUTO_START=always

2.8 Setting MLOCK priviledge for DBA user (1^st and subsequent nodes)

host85-root>setprivgrp dba MLOCK
host85-root>echo "dba MLOCK" > /etc/privgroup
host85-root>/usr/bin/getprivgrp dba

2.9 Copy the $ORACLE_BASE/admin/SID (from 1^st node to subsequent nodes)

host85-root>pwd
/u01/app/oracle/admin

host85-root>tar xvf - HPUX | ssh 10.10.10.83 "cd /u01/app/oracle/admin; tar \ xvf - "

2.10 Copy the $ORACLE/dbs (from 1^st node to subsequent nodes)

host85-root>pwd
/u01/app/oracle/product/11.2.0/dbhome_1
host85-root>tar cvf - dbs | ssh 10.10.10.81 "cd /u01/app/oracle/product/11.2.0/dbhome_1; tar xvf -"

3 VCS ASM and VCS Service Group configuration

This diagram shows all VCS service groups and their resource types as well as their interlink relationship.

3.1 Setup ASM Instance Service group and add ASM Instance Resource type in the group.

hagrp -add  asminstgrp
hagrp -modify asminstgrp SystemList  host85 0 host83 1 host81 2
hagrp -modify asminstgrp AutoStartList host85 host83 host81
hagrp -modify asminstgrp Parallel 1

hares -add  ASM_asminst  ASMInst  asminstgrp
hares -modify ASM_asminst Critical 1
hares -modify ASM_asminst MonitorOption  0
hares -modify ASM_asminst Sid  +ASM
hares -modify ASM_asminst Owner  oracle
hares -modify ASM_asminst Home  /u01/app/11.2.0/grid
hares -modify ASM_asminst DBAUser  
hares -modify ASM_asminst DBAPword  
hares -modify ASM_asminst Pfile  
hares -modify ASM_asminst StartUpOpt  
hares -modify ASM_asminst ShutDownOpt  
hares -modify ASM_asminst EnvFile  
hares -modify ASM_asminst Encoding  
hares -modify ASM_asminst Enabled 1

3.2  Setup oraasm_group service group for single Oracle DB and ASM instance. ASMDG, IP, NIC and Netlsnr resource type will be add into this service group. Oracle resource type also require if Oracle DB is configure.

hagrp -add  oraasm_grp
hagrp -modify oraasm_grp SystemList  host85 0 host83 1 host81 2
hagrp -modify oraasm_grp AutoStartList host85
hagrp -modify oraasm_grp Parallel 0

hares -add  ASM_asmdg  ASMDG  oraasm_grp
hares -modify ASM_asmdg Critical 1
hares -modify ASM_asmdg Sid  +ASM
hares -modify ASM_asmdg Owner  oracle
hares -modify ASM_asmdg Home  /u01/app/11.2.0/grid
hares -modify ASM_asmdg DBAUser  
hares -modify ASM_asmdg DBAPword  
hares -modify ASM_asmdg DiskGroups  DATA
hares -modify ASM_asmdg EnvFile  
hares -modify ASM_asmdg Encoding  
hares -modify ASM_asmdg Enabled 1

hares -add  IP_oraprod  IP  oraasm_grp
hares -modify IP_oraprod Critical 1
hares -modify IP_oraprod ArpDelay  1
hares -modify IP_oraprod IfconfigTwice  0
hares -modify IP_oraprod PrefixLen  0
hares -modify IP_oraprod Device  lan2
hares -modify IP_oraprod Address  10.10.10.146
hares -modify IP_oraprod NetMask  255.255.252.0
hares -modify IP_oraprod Options  
hares -modify IP_oraprod RouteOptions  
hares -modify IP_oraprod Enabled 1

hares -add  NIC_oraprod  NIC  oraasm_grp
hares -modify NIC_oraprod Critical 1
hares -modify NIC_oraprod NetworkType  ether
hares -modify NIC_oraprod PingOptimize  1
hares -modify NIC_oraprod Protocol  IPv4
hares -modify NIC_oraprod Device  lan2
hares -modify NIC_oraprod NetworkHosts host85  host83  host81
hares -modify NIC_oraprod Enabled 1

hares -add  LSNR_oraprod_lsnr  Netlsnr  oraasm_grp
hares -modify LSNR_oraprod_lsnr Critical 1
hares -modify LSNR_oraprod_lsnr Listener  LISTENER
hares -modify LSNR_oraprod_lsnr MonScript  ./bin/Netlsnr/LsnrTest.pl
hares -modify LSNR_oraprod_lsnr Owner  oracle
hares -modify LSNR_oraprod_lsnr Home  /u01/app/11.2.0/grid
hares -modify LSNR_oraprod_lsnr TnsAdmin  /u01/app/11.2.0/grid/admin
hares -modify LSNR_oraprod_lsnr EnvFile  
hares -modify LSNR_oraprod_lsnr LsnrPwd  
hares -modify LSNR_oraprod_lsnr Encoding  
hares -modify LSNR_oraprod_lsnr Enabled 1

ares -add  ORA_oraprod  Oracle  oraasm_grp
hares -modify ORA_oraprod Critical 1
hares -modify ORA_oraprod StartUpOpt  STARTUP_FORCE
hares -modify ORA_oraprod ShutDownOpt  IMMEDIATE
hares -modify ORA_oraprod AutoEndBkup  1
hares -modify ORA_oraprod MonScript  ./bin/Oracle/SqlTest.pl
hares -modify ORA_oraprod MonitorOption  0
hares -modify ORA_oraprod ManagedBy  ADMIN
hares -modify ORA_oraprod Sid  HPUX
hares -modify ORA_oraprod Owner  oracle
hares -modify ORA_oraprod Home  /u01/app/oracle/product/11.2.0/dbhome_1
hares -modify ORA_oraprod Pfile  
hares -modify ORA_oraprod DBAUser  system
hares -modify ORA_oraprod DBAPword  ameMbmOmhMfoCmpOfoD
hares -modify ORA_oraprod EnvFile  
hares -modify ORA_oraprod User  
hares -modify ORA_oraprod Pword  
hares -modify ORA_oraprod Table   
hares -modify ORA_oraprod Encoding  
hares -modify ORA_oraprod DBName  
hares -modify ORA_oraprod Enabled 1

Add below in main.cf to create the link relationship:
requires group asminstgrp online local firm
IP_oraprod requires NIC_oraprod
LSNR_oraprod_lsnr requires IP_oraprod
LSNR_oraprod_lsnr requires ORA_oraprod
ORA_oraprod requires ASM_asmdg


Sample main.cf
host81-root>cat /etc/VRTSvcs/conf/config/main.cf
include "OracleASMTypes.cf"
include "types.cf"
include "CFSTypes.cf"
include "CRSResource.cf"
include "CVMTypes.cf"
include "Db2udbTypes.cf"
include "MultiPrivNIC.cf"
include "OracleTypes.cf"
include "PrivNIC.cf"
include "SybaseTypes.cf"

cluster hp818385 (
        UserNames = { admin = hqrJqlQnrMrrPzrLqo }
        Administrators = { admin }
        UseFence = SCSI3
        HacliUserLevel = COMMANDROOT
        )

system host81 (
        )
system host83 (
        )
system host85 (
        )

group asminstgrp (
        SystemList = { host85 = 0, host83 = 1, host81 = 2 }
        Parallel = 1
        AutoStartList = { host85, host83, host81 }
        )

        ASMInst ASM_asminst (
                Sid = "+ASM"
                Owner = oracle
                Home = "/u01/app/11.2.0/grid"
                )
        // resource dependency tree
        //
        //      group asminstgrp
        //      {
        //      ASMInst ASM_asminst
        //      }

group oraasm_grp (
        SystemList = { host85 = 0, host83 = 1, host81 = 2 }
        AutoStartList = { host85 }
        )

        ASMDG ASM_asmdg (
                Sid = "+ASM"
                Owner = oracle
                Home = "/u01/app/11.2.0/grid"
                DiskGroups = { DATA }
                )

        IP IP_oraprod (
                Device = lan2
                Address = "10.10.10.146"
                NetMask = "255.255.252.0"
                )

        NIC NIC_oraprod (
                Device = lan2
              NetworkHosts = { "10.10.10.85", "10.10.10.83", "10.10.10.81" }
                )

        Netlsnr LSNR_oraprod_lsnr (
                Owner = oracle
                Home = "/u01/app/11.2.0/grid"
                TnsAdmin = "/u01/app/11.2.0/grid/admin"
                )

        Oracle ORA_oraprod (
                Sid = HPUX
                Owner = oracle
                Home = "/u01/app/oracle/product/11.2.0/dbhome_1"
                DBAUser = "system"
                DBAPword = ameMbmOmhMfoCmpOfoD
                )

        requires group asminstgrp online local firm
        IP_oraprod requires NIC_oraprod
        LSNR_oraprod_lsnr requires IP_oraprod
        LSNR_oraprod_lsnr requires ORA_oraprod
        ORA_oraprod requires ASM_asmdg


        // resource dependency tree
        //
        //      group oraasm_grp
        //      {
        //      Netlsnr LSNR_oraprod_lsnr
        //          {
        //          IP IP_oraprod
        //              {
        //              NIC NIC_oraprod
        //              }
        //          Oracle ORA_oraprod
        //              {
        //              ASMDG ASM_asmdg
        //              }
        //          }
        //      }