Tuesday, March 15, 2016

Monitoring FUSION-WRITE RAC 12c

In RAC, blocks are written from cache to disk when checkpoint occurs, aging and when user request new data i.e cache replacement. Oracle send's message to notify other instances that oracle will perform FUSION-WRITE to move datablock from disk :)

Fusion - Writes can be monitored with following sql

select
a.inst_id "Instance",
A.value/B.value "cache fusion writes Ratio"
FROM
GV$SYSSTAT A,
GV$SYSSTAT B
WHERE
a.name='DBWR fusion writes'
AND b.name='physical writes'
AND b.inst_id=a.inst_id
ORDER BY
A.INST_ID;

If the ratio is larger we can notify more blocks are copied with previous changes between instances and we have to work on Tuning cache sizes and Checkpoints

--Nikhil Tatineni--
--12C RAC ---

Knowledge on RAC
12c: RAC Environment
http://oracle-in-memory.blogspot.com/2016/02/rac-environment.html
12c: RAC Blocked Listener

http://oracle-in-memory.blogspot.com/2016/02/12c-rac-blocked-listener.html
12c: RAC Health Checks
http://oracle-in-memory.blogspot.com/2016/02/rac-health-checks-12c.html
RAC Diagwait
http://oracle-in-memory.blogspot.com/2016/02/rac-diagwait.html
12c: Oracle Local Registry(OLR)
http://oracle-in-memory.blogspot.com/2016/02/12c-olr-oracle-local-registry.html
12c: Applying Database Rolling Patch on RAC Cluster
http://oracle-in-memory.blogspot.com/2016/02/applying-database-rolling-patch.html
12c: Resize Online redolog files 
http://oracle-in-memory.blogspot.com/2016/02/resize-online-redologs-files-12c-rac.html
RAC: Creating service using srvctl on pdb 
http://oracle-in-memory.blogspot.com/2016/02/rac-creating-service-on-pdb.html
12c: Mirror copy of OCR onto different asm diskgroup
http://oracle-in-memory.blogspot.com/2016/02/oracle-rac-mirror-copy-of-ocr-onto.html 
12c: Pin Nodes On Cluster

http://oracle-in-memory.blogspot.com/2016/02/12c-pin-nodes-on-cluster.html
12c: RAC Modify Default scan port Number

http://oracle-in-memory.blogspot.com/2016/02/12c-rac-modify-default-scan-port-number.html
12c: Mutiplex Voting Files on Oracle RAC 12C
http://oracle-in-memory.blogspot.com/2016/02/multiplex-voting-files-on-oracle-rac-12c.html
12c:converting database from No Archivelog mode to Archivelog Mode
http://oracle-in-memory.blogspot.com/2016/03/converting-12c-rac-database-to.html
12c: Creating New Virtual IP to manager HA 
http://oracle-in-memory.blogspot.com/2016/03/oracle-goldengate-creating-virtual-ip.html
CRS-4535 12C RAC Cluster
http://oracle-in-memory.blogspot.com/2016/02/12c-rac-crs-4535.html
CRS-8503 12C RAC Cluster
http://oracle-in-memory.blogspot.com/2016/03/crs-8503.html

Monday, March 7, 2016

ACFS: Automatic Cluster FileSystem on RAC 12c

Steps to create ACFS using ASMCA 
Create Diskgroup
Create ASM volume
Create ACFS 

Creating New Diskgroup "DTRAILS" for creating ACFS 
Create New Volume on newly created diskgroup 
After Creating New Volume, Created ACFS as follows

After ---

Created new Directories required and change permissions, where you want to mount ACFS on Nodes of Cluster 
[root@tnc1 ~]# 
[root@tnc1 ~]# mkdir -p /u03/trails
[root@tnc1 ~]# chown -R oracle:oinstall /u03/trails
[root@tnc1 ~]# 

Fill details as required and click ok to create ACFS. At the end it will ask you to run .sh script, which mounts the ACFS on both file system 

[root@tnc1 ~]# /u01/app/oracle/cfgtoollogs/asmca/scripts/acfs_script.sh
ACFS file system /u03/trails is mounted on nodes tnc1,tnc2

[root@tnc2 ~]# df -h
Filesystem            Size  Used Avail Use% Mounted on
/dev/mapper/VolGroup00-LogVol00
                       26G   20G  4.2G  83% /
/dev/sda1              99M   23M   72M  24% /boot
tmpfs                  12G  169M   12G   2% /dev/shm
none                  1.1G  104K  1.1G   1% /var/lib/xenstored
/dev/asm/sharkoki-6   3.0G  1.1G  2.0G  35% /u02/goldengate
/dev/asm/strail-174   1.9G   81M  1.8G   5% /u03/trails

---Nikhil Tatineni---

Sunday, March 6, 2016

ERROR OGG-00446 Unable to lock file "ECBN.cpe


Extract and Replicats running On RAC ACFS got abended report file says process abended with "OGG-00446  Unable to lock file"

GGSCI (tnc1.localdomain) 4> info all
Program     Status      Group       Lag at Chkpt  Time Since Chkpt                                        
EXTRACT     ABENDED     ECBN        0 0:00:09      00:32:52    
REPLICAT    ABENDED     RCBN        00:00:00      00:50:36    

GGSCI (tnc1.localdomain) 5> view report ECBN
ECBN0.rpt:2016-03-04 10:55:18  ERROR   OGG-00446  Unable to lock file "/u02/goldengate/12.1/dirchk/ECBN.cpe" (error 11, Resource temporarily unavailable).
ECBN0.rpt:2016-03-04 10:55:18  ERROR   OGG-01668  PROCESS ABENDING.
ECBN1.rpt:2016-03-04 10:51:38  ERROR   OGG-00446  Unable to lock file "/u02/goldengate/12.1/dirchk/ECBN.cpe" (error 11, Resource temporarily unavailable).  Lock currently held by process id (PID) 14053.

Root cause as follows 
"FROM Doc ID 1330326.1" 
As we Installed GoldenGate on Shared Filesystem (ACFS), OS dosn't shutdown cleanly 
and we have Exclusive locks held on files ""/u02/goldengate/12.1/dirchk/ECBN.cpe"" by Filesystem and makes GG process abended on GoldenGate Home

Resolution: 
Rename the "ExtractName.CPE" 
copy the information in New.cpe to Old.cpe file and Restart the extract  
After restarting extract delete the Rename.cpe File 

[oracle@tnc1 dirchk]$ ls -ltr
total 92
-rw-r----- 1 oracle oinstall  2048 Mar  6 00:32 ECBN.cpb
-rw-r----- 1 oracle oinstall 30720 Mar  6 00:32 RCBN.cpr
-rw-r----- 1 oracle oinstall 22528 Mar  6 00:32 ECBN.cpe

[oracle@tnc1 dirchk]$ mv ECBN.cpe ECBN_BACKUP.cpe
[oracle@tnc1 dirchk]$ ls -ltr
-rw-r----- 1 oracle oinstall 22528 Mar  6 00:32 ECBN_BACKUP.cpe
[oracle@tnc1 dirchk]$ cp ECBN_BACKUP.cpe ECBN.cpe
[oracle@tnc1 dirchk]$ 
[oracle@tnc1 dirchk]$ ggn
[oracle@tnc1 12.1]$ pwd
/u02/goldengate/12.1
[oracle@tnc1 12.1]$ ./ggsci
Oracle GoldenGate Command Interpreter for Oracle
Version 12.2.0.1.1 OGGCORE_12.2.0.1.0_PLATFORMS_151211.1401_FBO
Linux, x64, 64bit (optimized), Oracle 12c on Dec 12 2015 02:56:48
Operating system character set identified as UTF-8.

Copyright (C) 1995, 2015, Oracle and/or its affiliates. All rights reserved.
GGSCI (tnc1.localdomain) 1> info all
Program     Status      Group       Lag at Chkpt  Time Since Chkpt
MANAGER     RUNNING                                           
EXTRACT     ABENDED     ECBN        00:00:09          47:41:59    
EXTRACT     ABENDED     ECBN_BACKUP  00:00:09   47:41:59    
REPLICAT    ABENDED     RCBN        00:00:00           47:55:21    

GGSCI (tnc1.localdomain) 8> info all
Program     Status      Group       Lag at Chkpt  Time Since Chkpt
MANAGER     RUNNING                                           
EXTRACT     RUNNING     ECBN        47:43:16      00:00:09   

After Remove the "Rename.cpe" File 
[oracle@tnc1 dirchk]$ rm ECBN_BACKUP.cpe 
[oracle@tnc1 dirchk]$ ls -ltr

GGSCI (tnc1.localdomain) 2> info all
Program     Status      Group       Lag at Chkpt  Time Since Chkpt
MANAGER     RUNNING                                           
EXTRACT     RUNNING     ECBN        00:00:04      00:00:09    

---Nikhil Tatineni--



Converting 12c RAC database to Archivelog mode

In-order to convert RAC database from No Archivelog to Archivelog Log Mode 

Step1: set "cluster_database" to False 
Step2: Bring down Database using srvctl 
Step3: open database in mount stage from any one of the instance 
step4: convert database to archivelog mode in mount stage 
step5: set cluster_database to true again
Step6: start up database using srvctl 

[oracle@tnc1 admin]$ db
[oracle@tnc1 admin]$ sql
SQL*Plus: Release 12.1.0.2.0 Production on Thu Mar 3 01:16:17 2016
Copyright (c) 1982, 2014, Oracle.  All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options
SQL> archive log list;  
Database log mode       No Archive Mode
Automatic archival       Disabled
Archive destination       USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     38
Current log sequence       39
SQL> show parameter db_recovery
NAME     TYPE VALUE
------------------------------------ ----------- ------------------------------
db_recovery_file_dest     string +FRA
db_recovery_file_dest_size     big integer 4785M

SQL> alter system set cluster_database=FALSE scope=spfile;
#---
[oracle@tnc1 ~]$ srvctl stop database -d p1tncd

[oracle@tnc1 ~]$ db
[oracle@tnc1 ~]$ sql
SQL*Plus: Release 12.1.0.2.0 Production on Thu Mar 3 01:43:52 2016
Copyright (c) 1982, 2014, Oracle.  All rights reserved.
Connected to an idle instance.
SQL> startup mount;
ORACLE instance started.
Total System Global Area  549453824 bytes
Fixed Size     2926616 bytes
Variable Size   293603304 bytes
Database Buffers   247463936 bytes
Redo Buffers     5459968 bytes
Database mounted.

SQL> alter database archivelog;
Database altered.

SQL> alter system set cluster_database=TRUE scope=spfile;
System altered.

SQL> show parameter cluster
NAME     TYPE VALUE
------------------------------------ ----------- ------------------------------
cluster_database     boolean FALSE
cluster_database_instances     integer 1
cluster_interconnects     string
SQL> 
SQL> shut immediate;
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.

[oracle@tnc1 ~]$ srvctl start database -d p1tncd
[oracle@tnc2 ~]$ db
[oracle@tnc2 ~]$ sql

SQL*Plus: Release 12.1.0.2.0 Production on Thu Mar 3 01:49:06 2016
Copyright (c) 1982, 2014, Oracle.  All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Real Application Clusters, Automatic Storage Management, OLAP,
Advanced Analytics and Real Application Testing options
SQL> show parameter db_name;
NAME     TYPE VALUE
------------------------------------ ----------- ------------------------------
db_name      string p1tncd 

SQL> archive log list;
Database log mode       Archive Mode
Automatic archival       Enabled
Archive destination       USE_DB_RECOVERY_FILE_DEST
Oldest online log sequence     33
Next log sequence to archive   34
Current log sequence           34

---Nikhil Tatineni--
---12c RAC Database--

Saturday, March 5, 2016

CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []

2016-03-05 20:02:45.720 [ORAROOTAGENT(26842)]CRS-8500: Oracle Clusterware ORAROOTAGENT process is starting with operating system process ID 26842
2016-03-05 20:02:53.572 [CLSECHO(27164)]CRS-10001: 05-Mar-16 20:02 AFD-9204: false
2016-03-05 20:03:45.967 [ORAROOTAGENT(26842)]CRS-5818: Aborted command 'check' for resource 'ora.storage'. Details at (:CRSAGF00113:) {0:21:2} in /u01/app/oracle/diag/crs/tnc2/crs/trace/ohasd_orarootagent_root.trc.
2016-03-05 20:04:46.133 [ORAROOTAGENT(28430)]CRS-8500: Oracle Clusterware ORAROOTAGENT process is starting with operating system process ID 28430
2016-03-05 20:05:01.921 [MDNSD(4300)]CRS-5602: mDNS service stopping by request.
2016-03-05 20:05:05.559 [MDNSD(4300)]CRS-8504: Oracle Clusterware MDNSD process with operating system process ID 4300 is exiting
2016-03-05 20:05:10.100 [OHASD(3131)]CRS-10301: Unable to contact Oracle high availability service on peer nodes for cluster wide commands. Details at (:CCL00001:) in /u01/app/oracle/diag/crs/tnc2/crs/trace/ohasd.trc.
2016-03-05 20:05:46.196 [ORAROOTAGENT(28430)]CRS-5818: Aborted command 'check' for resource 'ora.storage'. Details at (:CRSAGF00113:) {0:23:2} in /u01/app/oracle/diag/crs/tnc2/crs/trace/ohasd_orarootagent_root.trc.
2016-03-05 20:06:46.354 [ORAROOTAGENT(28952)]CRS-8500: Oracle Clusterware ORAROOTAGENT process is starting with operating system process ID 28952
2016-03-05 20:07:46.412 [ORAROOTAGENT(28952)]CRS-5818: Aborted command 'check' for resource 'ora.storage'. Details at (:CRSAGF00113:) {0:25:2} in /u01/app/oracle/diag/crs/tnc2/crs/trace/ohasd_orarootagent_root.trc.
2016-03-05 20:08:46.588 [ORAROOTAGENT(29487)]CRS-8500: Oracle Clusterware ORAROOTAGENT process is starting with operating system process ID 29487
2016-03-05 20:09:46.644 [ORAROOTAGENT(29487)]CRS-5818: Aborted command 'check' for resource 'ora.storage'. Details at (:CRSAGF00113:) {0:27:2} in /u01/app/oracle/diag/crs/tnc2/crs/trace/ohasd_orarootagent_root.trc.
2016-03-05 20:10:46.819 [ORAROOTAGENT(30584)]CRS-8500: Oracle Clusterware ORAROOTAGENT process is starting with operating system process ID 30584
2016-03-05 20:10:46.857 [OCSSD(21147)]CRS-1656: The CSS daemon is terminating due to a fatal error; Details at (:CSSSC00012:) in /u01/app/oracle/diag/crs/tnc2/crs/trace/ocssd.trc
2016-03-05 20:10:46.857 [OCSSD(21147)]CRS-1603: CSSD on node tnc2 shutdown by user.
2016-03-05 20:10:46.856 [CSSDAGENT(21135)]CRS-5818: Aborted command 'start' for resource 'ora.cssd'. Details at (:CRSAGF00113:) {0:15:27} in /u01/app/oracle/diag/crs/tnc2/crs/trace/ohasd_cssdagent_root.trc.
2016-03-05 20:10:47.001 [ORAAGENT(4287)]CRS-5818: Aborted command 'clean' for resource 'ora.asm'. Details at (:CRSAGF00113:) {0:9:6} in /u01/app/oracle/diag/crs/tnc2/crs/trace/ohasd_oraagent_oracle.trc.
2016-03-05 20:10:51.009 [ORAAGENT(4287)]CRS-5011: Check of resource "ora.asm" failed: details at "(:CLSN00006:)" in "/u01/app/oracle/diag/crs/tnc2/crs/trace/ohasd_oraagent_oracle.trc"
2016-03-05 20:10:51.013 [OHASD(3131)]CRS-2757: Command 'Clean' timed out waiting for response from the resource 'ora.asm'. Details at (:CRSPE00163:) {0:9:6} in /u01/app/oracle/diag/crs/tnc2/crs/trace/ohasd.trc.
Sat Mar 05 20:10:51 2016
Errors in file /u01/app/oracle/diag/crs/tnc2/crs/trace/ocssd.trc  (incident=193):
CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []
Incident details in: /u01/app/oracle/diag/crs/tnc2/crs/incident/incdir_193/ocssd_i193.trc

2016-03-05 20:10:52.062 [OCSSD(21147)]CRS-8503: Oracle Clusterware OCSSD process with operating system process ID 21147 experienced fatal signal or exception code 6
Sweep [inc][193]: completed
2016-03-05 20:10:57.452 [OHASD(3131)]CRS-2757: Command 'Start' timed out waiting for response from the resource 'ora.cssd'. Details at (:CRSPE00163:) {0:15:27} in /u01/app/oracle/diag/crs/tnc2/crs/trace/ohasd.trc.
2016-03-05 20:10:57.541 [CSSDAGENT(30935)]CRS-8500: Oracle Clusterware CSSDAGENT process is starting with operating system process ID 30935
2016-03-05 20:10:57.696 [OHASD(3131)]CRS-2878: Failed to restart resource 'ora.asm'
2016-03-05 20:10:57.711 [OHASD(3131)]CRS-2878: Failed to restart resource 'ora.cssd'
2016-03-05 20:10:57.721 [GPNPD(4339)]CRS-2329: GPNPD on node tnc2 shut down. 
2016-03-05 20:10:57.735 [GPNPD(4339)]CRS-8504: Oracle Clusterware GPNPD process with operating system process ID 4339 is exiting
2016-03-05 20:10:58.862 [GIPCD(4408)]CRS-8504: Oracle Clusterware GIPCD process with operating system process ID 4408 is exiting
2016-03-05 20:23:00.782 [CLSECHO(9793)]CRS-10001: 05-Mar-16 20:23 AFD-9204: false
2016-03-05 21:32:19.942 [CLSECHO(11011)]CRS-10001: 05-Mar-16 21:32 AFD-9204: false

$ tail -100f  /u01/app/oracle/diag/crs/tnc2/crs/trace/ocssd.trc
016-03-05 20:10:50.299585 :    CSSD:1096018240: clssscWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 1010 with cvtimewait status 4294967186
2016-03-05 20:10:50.401486 :    CSSD:1121986880: clssnmvDHBValidateNCopy: node 1, tnc1, has a disk HB, but no network HB, DHB has rcfg 352673332, wrtcnt, 1011537, LATS 2188456, lastSeqNo 1011534, uniqueness 1457225306, timestamp 1457226649/2189006
2016-03-05 20:10:50.401595 :    CSSD:1118832960: clssnmvDHBValidateNCopy: node 1, tnc1, has a disk HB, but no network HB, DHB has rcfg 352673332, wrtcnt, 1011538, LATS 2188456, lastSeqNo 1011535, uniqueness 1457225306, timestamp 1457226649/2189396
2016-03-05 20:10:50.416981 :    CSSD:1085483328: clssnmvDHBValidateNCopy: node 1, tnc1, has a disk HB, but no network HB, DHB has rcfg 352673332, wrtcnt, 1011539, LATS 2188466, lastSeqNo 1011536, uniqueness 1457225306, timestamp 1457226649/2189446
2016-03-05 20:10:51.303616 :    CSSD:1096018240: clssscWaitOnEventValue: after CmInfo State  val 3, eval 1 waited 1000 with cvtimewait status 4294967186
2016-03-05 20:10:51.351707 :GIPCHTHR:1114102080: gipchaWorkerWork: workerThread heart beat, time interval since last heartBeat 30130loopCount 32
2016-03-05 20:10:51.404980 :    CSSD:1121986880: clssnmvDHBValidateNCopy: node 1, tnc1, has a disk HB, but no network HB, DHB has rcfg 352673332, wrtcnt, 1011540, LATS 2189456, lastSeqNo 1011537, uniqueness 1457225306, timestamp 1457226650/2190006
2016-03-05 20:10:51.405087 :    CSSD:1118832960: clssnmvDHBValidateNCopy: node 1, tnc1, has a disk HB, but no network HB, DHB has rcfg 352673332, wrtcnt, 1011541, LATS 2189456, lastSeqNo 1011538, uniqueness 1457225306, timestamp 1457226650/2190406
2016-03-05 20:10:51.420767 :    CSSD:1085483328: clssnmvDHBValidateNCopy: node 1, tnc1, has a disk HB, but no network HB, DHB has rcfg 352673332, wrtcnt, 1011542, LATS 2189476, lastSeqNo 1011539, uniqueness 1457225306, timestamp 1457226650/2190446
Trace file /u01/app/oracle/diag/crs/tnc2/crs/trace/ocssd.trc
Oracle Database 12c Clusterware Release 12.1.0.2.0 - Production Copyright 1996, 2014 Oracle. All rights reserved.
DDE: Flood control is not active
Incident 193 created, dump file: /u01/app/oracle/diag/crs/tnc2/crs/incident/incdir_193/ocssd_i193.trc
CRS-8503 [] [] [] [] [] [] [] [] [] [] [] []
    CLSB:1106217280: Oracle Clusterware infrastructure error in OCSSD (OS PID 21147): Fatal signal 6 has occurred in program ocssd thread 1106217280; nested signal count is 1

Action:

Initially Analyze the alert log files and trace files

Start investigating on Failure Node and Living node in a cluster, Possible reasons for node eviction as follows 

1) Check alert log file on Node 1,  Some times If there is problem with Interconnect, Node 1 kicks out Node 2 from the cluster.
2) Blocking port can Make Node 2 Evicts from Cluster
Stop and Start CRS on the Failure Node will resolve the issue

--Nikhil Tatineni--
--12c RAC Cluster--

Friday, March 4, 2016

Oracle GoldenGate: Creating Virtual IP address (VIP) to manage high availability


Step1:
Update /etc/hosts file with new Goldengate VIP 
vi /etc/hosts
#Goldengate
192.168.1.23 goldengate-vip gg-vip
##Update /etc/hosts Completed##

Step2:
Create new vip with root user  using "appvipcfg"
/u01/app/12.1.0.2/grid/bin/appvipcfg create -network=1 -ip=192.168.1.23 -vipname=goldengate-vip -user=root
#-----
[root@tnc1 ~]# crsctl stat res -p |grep -ie .network -ie subnet |grep -ie name -ie subnet
REGISTRATION_INVITED_SUBNETS=
NAME=ora.net1.network
USR_ORA_SUBNET=192.168.1.0

Step3:
Set permissions for oracle on new goldengate VIP 
[root@tnc1 ~]# /u01/app/12.1.0.2/grid/bin/crsctl setperm resource goldengate-vip -u user:oracle:r-x
#------
checking status of resources 
[root@tnc1 ~]# crsctl status resource -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.FRA.dg
               ONLINE  ONLINE       tnc1                     STABLE
               ONLINE  ONLINE       tnc2                     STABLE
ora.GRID.dg
               ONLINE  ONLINE       tnc1                     STABLE
               ONLINE  ONLINE       tnc2                     STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       tnc1                     STABLE
               ONLINE  OFFLINE      tnc2                     STABLE
ora.NEVOT.dg
               ONLINE  ONLINE       tnc1                     STABLE
               ONLINE  ONLINE       tnc2                     STABLE
ora.OCNEW.dg
               ONLINE  ONLINE       tnc1                     STABLE
               ONLINE  ONLINE       tnc2                     STABLE
ora.SDXH.dg
               ONLINE  ONLINE       tnc1                     STABLE
               ONLINE  ONLINE       tnc2                     STABLE
ora.asm
               ONLINE  ONLINE       tnc1                     Started,STABLE
               ONLINE  ONLINE       tnc2                     Started,STABLE
ora.net1.network
               ONLINE  ONLINE       tnc1                     STABLE
               ONLINE  OFFLINE      tnc2                     STABLE
ora.ons
               ONLINE  ONLINE       tnc1                     STABLE
               ONLINE  OFFLINE      tnc2                     STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
goldengate-vip
      1        OFFLINE OFFLINE                               STABLE
ora.LISTENER_SCAN1.lsnr

Step4:
Start new Goldengate VIP with Oracle User: 
[oracle@tnc1 ~]$ crsctl start resource goldengate-vip
CRS-2672: Attempting to start 'goldengate-vip' on 'tnc2'
CRS-2676: Start of 'goldengate-vip' on 'tnc2' succeeded

[oracle@tnc1 ~]$ crsctl status resource goldengate-vip
NAME=goldengate-vip
TYPE=app.appvipx.type
TARGET=ONLINE
STATE=ONLINE on tnc2

Step5:
ping new Goldengate VIP 
[oracle@tnc1 ~]$  ping -c4  goldengate-vip
PING goldengate-vip (10.10.10.191) 56(84) bytes of data.
64 bytes from goldengate-vip (10.10.10.191): icmp_seq=1 ttl=64 time=2.43 ms
64 bytes from goldengate-vip (10.10.10.191): icmp_seq=2 ttl=64 time=0.328 ms
64 bytes from goldengate-vip (10.10.10.191): icmp_seq=3 ttl=64 time=0.458 ms
64 bytes from goldengate-vip (10.10.10.191): icmp_seq=4 ttl=64 time=0.476 ms

--- goldengate-vip ping statistics ---
4 packets transmitted, 4 received, 0% packet loss, time 3010ms
rtt min/avg/max/mdev = 0.328/0.923/2.431/0.872 ms

New VIP is configured for Goldengate for Automatic Failover 

--Nikhil Tatineni--
--12c GoldenGate--

Querys to monitor RAC

following few  Query's will help to find out culprits-  Query to check long running transaction from last 8 hours  Col Sid Fo...