VCS NOTICE V-16-20011-1021 CFSMount error in Storage Foundation Oracle RAC

 If CFSMount is unable to mount the resources in the service group, it might be the previous mounting is still in stale state during the previous failure. Solution is to remove the stale mounting and online the resouce again.


# df -k |grep error
df: `/vmax_sfracfs1': Input/output error
df: `/vmax_sfracfs3': Input/output error
df: `/vmax_sfracfs5': Input/output error


/var/VRTSvcs/log/CFSMount_A.log
sgelxha196:/var/VRTSvcs/log # less CFSMount_A.log
2011/09/28 16:40:30 VCS NOTICE V-16-20011-1021 CFSMount:vmax_sfrac3:monitor:File System Disabled: MountPoint: /vmax_sfracfs3
2011/09/28 16:40:30 VCS NOTICE V-16-20011-1021 CFSMount:vmax_sfrac5:monitor:File System Disabled: MountPoint: /vmax_sfracfs5
2011/09/28 16:40:30 VCS NOTICE V-16-20011-1021 CFSMount:vmax_sfrac1:monitor:File System Disabled: MountPoint: /vmax_sfracfs1

Solution:
# umount -f /vmax_sfracfs1
# umount -f /vmax_sfracfs3
# umount -f /vmax_sfracfs5

# hagrp -online vmax_sfrac -sys host1

VCS ASM/Oracle agent for single oracle database instance with ASM

Steps to allow VCS Oracle Agent to manage Single Oracle Database Instance with ASM:

2. Oracle ASM and Database single instance setup

 This configuration consists of 3 VCS cluster nodes. VCS ASM and Oracle Agent will manage the ASM and Oracle resources across the 3 nodes with only 1 single node posses the single database instance at any point of time. 

2.1 Select and Install Grid Infrastructure Software only and execute below for standalone server. (1^st and subsequent nodes)

/u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/roothas.pl

2.2  Configure Oracle ASM with “asmca”. (1^st node only)

2.3 Select and Install Oracle Database Software only  (1^st and subsequent nodes)

2.4 Configure listener with “netca” (1^st and subsequent nodes)

2.5 Configure Oracle DB with “dbca” (1^st node only)

2.6 Create ASM and add ASM on remote host (subsequent nodes)

1st node (host85)
host85-oracle>export ORACLE_SID=+ASM
host85-oracle>export ORACLE_HOME=/u01/app/11.2.0/grid        
host85-oracle>asmcmd
ASMCMD> ls
REGISTRY.253.762436131
ASMCMD> spget +DATA/asm/ASMPARAMETERFILE/REGISTRY.253.762436131
+DATA/asm/asmparameterfile/registry.253.762436131
ASMCMD> spcopy +DATA/asm/ASMPARAMETERFILE/REGISTRY.253.762436131 /u01/app/11.2.0/grid/dbs/spfileASM.ora

2nd node (host83) and subsequent nodes (grid owner)
host85-oracle>crsctl stop resource ora.asm –f
host85-oracle>scp -p spfileASM.ora 10.10.10.83:/u01/app/11.2.0/grid/dbs
host83-oracle>srvctl add asm -l LISTENER –p\ /u01/app/11.2.0/grid/dbs/spfileASM.ora -d '/dev/rdisk/*'
host83-oracle>crsctl start resource ora.cssd
host83-oracle>crsctl start resource ora.asm
host83-oracle>export ORACLE_SID=+ASM
host83-oracle>export ORACLE_HOME=/u01/app/11.2.0/grid        
host83-oracle>sqlplus / as sysasm
SQL> alter diskgroup DATA mount;

host83-oracle>crsctl status resource –t  (Output display register ASM DG show in RED)
--------------------------------------------------------------------------------
NAME           TARGET  STATE        SERVER                   STATE_DETAILS      
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.DATA.dg
               ONLINE  ONLINE       host83                                    
ora.LISTENER.lsnr
               ONLINE  ONLINE       host83                                    
ora.asm
               ONLINE  ONLINE       host83                 Started             
ora.ons
               OFFLINE OFFLINE      host83                                    
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.cssd
      1        ONLINE  ONLINE       host83                                    
ora.diskmon
      1        ONLINE  ONLINE       host83                                    
ora.evmd
      1        ONLINE  ONLINE       host83        

2.7 Enable Oracle Cluster Synchronization Service daemon to start automatically (1^st and subsequent nodes)

host85-oracle>srvctl disable asm
host85-oracle>crsctl modify resource ora.asm -attr ENABLED=0
host85-oracle>crsctl modify resource ora.cssd -attr AUTO_START=always

2.8 Setting MLOCK priviledge for DBA user (1^st and subsequent nodes)

host85-root>setprivgrp dba MLOCK
host85-root>echo "dba MLOCK" > /etc/privgroup
host85-root>/usr/bin/getprivgrp dba

2.9 Copy the $ORACLE_BASE/admin/SID (from 1^st node to subsequent nodes)

host85-root>pwd
/u01/app/oracle/admin

host85-root>tar xvf - HPUX | ssh 10.10.10.83 "cd /u01/app/oracle/admin; tar \ xvf - "

2.10 Copy the $ORACLE/dbs (from 1^st node to subsequent nodes)

host85-root>pwd
/u01/app/oracle/product/11.2.0/dbhome_1
host85-root>tar cvf - dbs | ssh 10.10.10.81 "cd /u01/app/oracle/product/11.2.0/dbhome_1; tar xvf -"

3 VCS ASM and VCS Service Group configuration

This diagram shows all VCS service groups and their resource types as well as their interlink relationship.

3.1 Setup ASM Instance Service group and add ASM Instance Resource type in the group.

hagrp -add  asminstgrp
hagrp -modify asminstgrp SystemList  host85 0 host83 1 host81 2
hagrp -modify asminstgrp AutoStartList host85 host83 host81
hagrp -modify asminstgrp Parallel 1

hares -add  ASM_asminst  ASMInst  asminstgrp
hares -modify ASM_asminst Critical 1
hares -modify ASM_asminst MonitorOption  0
hares -modify ASM_asminst Sid  +ASM
hares -modify ASM_asminst Owner  oracle
hares -modify ASM_asminst Home  /u01/app/11.2.0/grid
hares -modify ASM_asminst DBAUser  
hares -modify ASM_asminst DBAPword  
hares -modify ASM_asminst Pfile  
hares -modify ASM_asminst StartUpOpt  
hares -modify ASM_asminst ShutDownOpt  
hares -modify ASM_asminst EnvFile  
hares -modify ASM_asminst Encoding  
hares -modify ASM_asminst Enabled 1

3.2  Setup oraasm_group service group for single Oracle DB and ASM instance. ASMDG, IP, NIC and Netlsnr resource type will be add into this service group. Oracle resource type also require if Oracle DB is configure.

hagrp -add  oraasm_grp
hagrp -modify oraasm_grp SystemList  host85 0 host83 1 host81 2
hagrp -modify oraasm_grp AutoStartList host85
hagrp -modify oraasm_grp Parallel 0

hares -add  ASM_asmdg  ASMDG  oraasm_grp
hares -modify ASM_asmdg Critical 1
hares -modify ASM_asmdg Sid  +ASM
hares -modify ASM_asmdg Owner  oracle
hares -modify ASM_asmdg Home  /u01/app/11.2.0/grid
hares -modify ASM_asmdg DBAUser  
hares -modify ASM_asmdg DBAPword  
hares -modify ASM_asmdg DiskGroups  DATA
hares -modify ASM_asmdg EnvFile  
hares -modify ASM_asmdg Encoding  
hares -modify ASM_asmdg Enabled 1

hares -add  IP_oraprod  IP  oraasm_grp
hares -modify IP_oraprod Critical 1
hares -modify IP_oraprod ArpDelay  1
hares -modify IP_oraprod IfconfigTwice  0
hares -modify IP_oraprod PrefixLen  0
hares -modify IP_oraprod Device  lan2
hares -modify IP_oraprod Address  10.10.10.146
hares -modify IP_oraprod NetMask  255.255.252.0
hares -modify IP_oraprod Options  
hares -modify IP_oraprod RouteOptions  
hares -modify IP_oraprod Enabled 1

hares -add  NIC_oraprod  NIC  oraasm_grp
hares -modify NIC_oraprod Critical 1
hares -modify NIC_oraprod NetworkType  ether
hares -modify NIC_oraprod PingOptimize  1
hares -modify NIC_oraprod Protocol  IPv4
hares -modify NIC_oraprod Device  lan2
hares -modify NIC_oraprod NetworkHosts host85  host83  host81
hares -modify NIC_oraprod Enabled 1

hares -add  LSNR_oraprod_lsnr  Netlsnr  oraasm_grp
hares -modify LSNR_oraprod_lsnr Critical 1
hares -modify LSNR_oraprod_lsnr Listener  LISTENER
hares -modify LSNR_oraprod_lsnr MonScript  ./bin/Netlsnr/LsnrTest.pl
hares -modify LSNR_oraprod_lsnr Owner  oracle
hares -modify LSNR_oraprod_lsnr Home  /u01/app/11.2.0/grid
hares -modify LSNR_oraprod_lsnr TnsAdmin  /u01/app/11.2.0/grid/admin
hares -modify LSNR_oraprod_lsnr EnvFile  
hares -modify LSNR_oraprod_lsnr LsnrPwd  
hares -modify LSNR_oraprod_lsnr Encoding  
hares -modify LSNR_oraprod_lsnr Enabled 1

ares -add  ORA_oraprod  Oracle  oraasm_grp
hares -modify ORA_oraprod Critical 1
hares -modify ORA_oraprod StartUpOpt  STARTUP_FORCE
hares -modify ORA_oraprod ShutDownOpt  IMMEDIATE
hares -modify ORA_oraprod AutoEndBkup  1
hares -modify ORA_oraprod MonScript  ./bin/Oracle/SqlTest.pl
hares -modify ORA_oraprod MonitorOption  0
hares -modify ORA_oraprod ManagedBy  ADMIN
hares -modify ORA_oraprod Sid  HPUX
hares -modify ORA_oraprod Owner  oracle
hares -modify ORA_oraprod Home  /u01/app/oracle/product/11.2.0/dbhome_1
hares -modify ORA_oraprod Pfile  
hares -modify ORA_oraprod DBAUser  system
hares -modify ORA_oraprod DBAPword  ameMbmOmhMfoCmpOfoD
hares -modify ORA_oraprod EnvFile  
hares -modify ORA_oraprod User  
hares -modify ORA_oraprod Pword  
hares -modify ORA_oraprod Table   
hares -modify ORA_oraprod Encoding  
hares -modify ORA_oraprod DBName  
hares -modify ORA_oraprod Enabled 1

Add below in main.cf to create the link relationship:
requires group asminstgrp online local firm
IP_oraprod requires NIC_oraprod
LSNR_oraprod_lsnr requires IP_oraprod
LSNR_oraprod_lsnr requires ORA_oraprod
ORA_oraprod requires ASM_asmdg


Sample main.cf
host81-root>cat /etc/VRTSvcs/conf/config/main.cf
include "OracleASMTypes.cf"
include "types.cf"
include "CFSTypes.cf"
include "CRSResource.cf"
include "CVMTypes.cf"
include "Db2udbTypes.cf"
include "MultiPrivNIC.cf"
include "OracleTypes.cf"
include "PrivNIC.cf"
include "SybaseTypes.cf"

cluster hp818385 (
        UserNames = { admin = hqrJqlQnrMrrPzrLqo }
        Administrators = { admin }
        UseFence = SCSI3
        HacliUserLevel = COMMANDROOT
        )

system host81 (
        )
system host83 (
        )
system host85 (
        )

group asminstgrp (
        SystemList = { host85 = 0, host83 = 1, host81 = 2 }
        Parallel = 1
        AutoStartList = { host85, host83, host81 }
        )

        ASMInst ASM_asminst (
                Sid = "+ASM"
                Owner = oracle
                Home = "/u01/app/11.2.0/grid"
                )
        // resource dependency tree
        //
        //      group asminstgrp
        //      {
        //      ASMInst ASM_asminst
        //      }

group oraasm_grp (
        SystemList = { host85 = 0, host83 = 1, host81 = 2 }
        AutoStartList = { host85 }
        )

        ASMDG ASM_asmdg (
                Sid = "+ASM"
                Owner = oracle
                Home = "/u01/app/11.2.0/grid"
                DiskGroups = { DATA }
                )

        IP IP_oraprod (
                Device = lan2
                Address = "10.10.10.146"
                NetMask = "255.255.252.0"
                )

        NIC NIC_oraprod (
                Device = lan2
              NetworkHosts = { "10.10.10.85", "10.10.10.83", "10.10.10.81" }
                )

        Netlsnr LSNR_oraprod_lsnr (
                Owner = oracle
                Home = "/u01/app/11.2.0/grid"
                TnsAdmin = "/u01/app/11.2.0/grid/admin"
                )

        Oracle ORA_oraprod (
                Sid = HPUX
                Owner = oracle
                Home = "/u01/app/oracle/product/11.2.0/dbhome_1"
                DBAUser = "system"
                DBAPword = ameMbmOmhMfoCmpOfoD
                )

        requires group asminstgrp online local firm
        IP_oraprod requires NIC_oraprod
        LSNR_oraprod_lsnr requires IP_oraprod
        LSNR_oraprod_lsnr requires ORA_oraprod
        ORA_oraprod requires ASM_asmdg


        // resource dependency tree
        //
        //      group oraasm_grp
        //      {
        //      Netlsnr LSNR_oraprod_lsnr
        //          {
        //          IP IP_oraprod
        //              {
        //              NIC NIC_oraprod
        //              }
        //          Oracle ORA_oraprod
        //              {
        //              ASMDG ASM_asmdg
        //              }
        //          }
        //      }

Oracle SGA and PGA fine tuning

If the system memory is insufficient due to Oracle memory allocation, 1st link will show how the PGA and SGA memory allocation is calculated. 2nd link will show how to set the memory.

http://www.dba-oracle.com/art_dbazine_ram.htm
http://orafaq.com/wiki/SGA_target

Click here to view the full explanation on Memory Architecture from Oracle for 11gR2.

Remove clone_disk flag on VxVM

When VxVM devices are shown as clone disk, below command allows the flag to be cleared.
Refer to http://www.symantec.com/business/support/index?page=content&id=TECH70049 for more detail.

 # vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
emcpoweraa   auto:cdsdisk    emcpowerq    vmax_vcsdg   online clone_disk
emcpowerab   auto:cdsdisk    emcpowerp    vmax_vcsdg   online clone_disk

# for i in `vxdisk list |grep clone_disk | awk '{print $1}'`; do echo $i; vxdisk set $i clone=off; done

# vxdisk list
DEVICE       TYPE            DISK         GROUP        STATUS
emcpoweraa   auto:cdsdisk    emcpowerq    vmax_vcsdg   online
emcpowerab   auto:cdsdisk    emcpowerp    vmax_vcsdg   online

Oracle RAC 11gR2 Issues

This post is some of the oracle articles I found in the Oracle support knowledge base website during my Oracle RAC 11gR2 installation. You might find them useful if you encounter similar issue. Take note a login account is required to access the website.

Article ID
CLUVFY Fails with TCP Check PRVF-7617 Due to Case of Node Names [ID 1286394.1]
Troubleshooting 11.2 Grid Infastructure Installation Root.sh Issues [ID 1053970.1]
CRS Does not Start after Node Reboot in 11gR2 Grid Infrastructure [ID 1215893.1]
Grid Infrastructure Installation: root.sh Stalls Due to Improper localhost Setting [ID 1155903.1]
Recommendation for the Real Application Cluster Interconnect and Jumbo Frames [ID 341788.1]

Oracle Cluster Verification Utility

Oracle provides Cluster node verification prior to installation. It is very helpful for Oracle RAC pre-installation to check the nodes meeting the system minimum requirement, the network interfaces availability, the patches update and etc.

Oracle Grid pre-installation
To verify whether your system meets all of the criteria for an Oracle Clusterware installation:
Window:

cluvfy stage -pre crsinst -n node1,node2,node3

Unix:

runcluvfy.sh stage -pre crsinst -n node1,node2,node3

After you have completed phase one, verify that Oracle Clusterware is functioning properly before proceeding with phase two of your Oracle RAC installation:
Window:

cluvfy stage -post crsinst -n node1,node2,node3 -v

Unix:

runcluvfy.sh stage -post crsinst -n node1,node2,node3 -v

To verify whether your system meets all of the criteria for an Oracle RAC installation:
Window:

cluvfy stage -pre dbinst -n node1,node2,node3 -v 

Unix:

Oracle Database pre-installation

To verify whether your system meets all of the criteria for creating a database or for making a database configuration change:

Window:

cluvfy stage -pre dbcfg -n -n node1,node2,node3 -d oracle_home -v

Unix:

runcluvfy.sh stage -pre dbcfg -n -n node1,node2,node3 -d oracle_home -v

Refer to Cluster Verification Utility Reference for more detail explanation.

runcluvfy.sh stage -pre dbinst -n node1,node2,node3 -v 

VCS file system mount failure due to timeout

If there're many mount points under VCS control, increasing OnlineTimeout value for mount resource will prevent VCS mounting resource failure due to the time constraint during switch over or failover. Refer to Modifying mount resource attributes for more detail.

The commands are:

/opt/VRTS/bin/haconf -makerw

/opt/VRTS/bin/hatype -modify Mount OnlineTimeout 600

/opt/VRTS/bin/haconf –dump

Error grabbed from engine_A.log
2010/08/10 15:21:24 VCS INFO V-16-2-13078 (node2) Resource(vcsfs1) - clean completed successfully after 1 failed attempts.
2010/08/10 15:21:24 VCS INFO V-16-2-13071 (node2) Resource(vcsfs1): reached OnlineRetryLimit(0).
2010/08/10 15:21:24 VCS ERROR V-16-1-10303 Resource vcsfs1 (Owner: Unspecified, Group: vcs) is FAULTED (timed out) on sys node2

SF Oracle RAC (SFRAC) Service Group go into partial state after paths failure

When there are paths failure on SFRAC cluster node1 causing the node2 and/or node3 Service Group goes into partial state. This is due to the default disk detach policy set as global in the share diskgroup. "vxdg" can be used to view the setting and "vxedit" can be used to change the setting. Refer to What is the disk detach policy for shared disk groups and how can it be changed? for more detail.

#vxdg list sfracdg

#vxedit -g sfracdg set diskdetpolicy=local sfracdg

# hastatus -sum
-- SYSTEM STATE
-- System               State                Frozen
A  node1           RUNNING              0
A  node2           RUNNING              0

-- GROUP STATE
-- Group           System               Probed     AutoDisabled    State
B  cvm             node1           Y          N   ONLINE
B  cvm             node2           Y          N   ONLINE
B  sfrac           node1           Y          N   PARTIAL
B  sfrac           node2           Y          N   PARTIAL