Database In Memory

Monday, May 9, 2016

Applying APRIL PSU "Patch Number : Patch 22738793

If we are applying APRIL PSU "Patch Number : 22646198: Component 11.2.0.4.160419 DB PSU + GI PSU 11.2.0.4.160419 (Apr2016) &
Patch 23274134 - Oracle Grid Infrastructure Patch Set Update 11.2.0.4.160719 (Jul2016) (Includes Database PSU 11.2.0.4.160719)

Procedure is same as follows #

Note: when you unzip the Combo patch, if you don't see patches listed in readme.html download the patch separately
Patch Can be applied in Rolling Fashion #

STEP1: Before Applying Any Patch #
Take backup of all Grid_home & Database_homes in all nodes of cluster #

STEP2:
Make sure ocm.rsp " response file is created to apply patch in all nodes of cluster #

For patch 22646198/
11.2.0.4.160419 DB PSU + GI PSU 11.2.0.4.160419 (Apr2016)
This patch can be applied in rolling Fashion # There is No downTime Required for this Patch ##

STEP3: Stop EM Agent Processes Prior to Patching and Prior to Rolling Back the Patch as ORACLE HOME owner #
$ <ORACLE_HOME>/bin/emctl stop dbconsole

b)Validation of Oracle Inventory - check the consistency of inventory information for GI home and each database home

$ opatch lsinventory -detail -oh <GRID_HOME>

$ opatch lsinventory -detail -oh <ORACLE_HOME>

STEP4:
Check for one-off patch conflict detection and resolution or Run it with Root User
Run it with Oracle / GRID User:

FOR ORACLE_HOME
/u01/app/11.2.0/grid/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -oh /u01/app/oracle/product/11.2.0/db_1 -phBaseDir /stage/22646198

For GRID HOME
/u01/app/11.2.0/grid/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -oh /u01/app/11.2.0/grid -phBaseDir /stage/22646198

STEP5: Once we confirm, there is no conflicts, go ahead and apply patch to database home & grid home USING "ocm.rsp" file.

If GI Home and the Database Homes that are not shared and ACFS file system is not configured apply patch as follows:
Syntax: opatch auto <Patch Location> -ocmrf <ocm response file>

# opatch auto /stage/22646198 -ocmrf /stage/apply_april_psu.rsp
OutPut as follows # Executing /u01/app/11.2.0/grid/perl/bin/perl /u01/app/oracle/product/11.2.0/db_1/OPatch/crs/patch11203.pl -patchdir /stage -patchn 22646198 -ocmrf /stage/apply_april_psu.rsp -paramfile /u01/app/11.2.0/grid/crs/install/crsconfig_params

This is the main log file: /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatchauto2016-05-09_10-41-31.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatchauto2016-05-09_10-41-31.report.log

2016-05-09 10:41:31: Starting Clusterware Patch Setup
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params

Stopping RAC /u01/app/oracle/product/11.2.0/db_1 ...
Stopped RAC /u01/app/oracle/product/11.2.0/db_1 successfully

patch /stage/22646198/22502456 apply successful for home /u01/app/oracle/product/11.2.0/db_1
patch /stage/22646198/22502549/custom/server/22502549 apply successful for home /u01/app/oracle/product/11.2.0/db_1

Stopping CRS...
Stopped CRS successfully

patch /stage/22646198/22502456 apply successful for home /u01/app/11.2.0/grid
patch /stage/22646198/22502549 apply successful for home /u01/app/11.2.0/grid
patch /stage/22646198/22502505 apply successful for home /u01/app/11.2.0/grid

Starting CRS...
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.

Starting RAC /u01/app/oracle/product/11.2.0/db_1 ...
Started RAC /u01/app/oracle/product/11.2.0/db_1 successfully
opatch auto succeeded.

Things You can observer on Node 2 # and in alert log file #

[root@tnc2 tnc2]# crsctl status resource -t
--------------------------------------------------------------------------------
NAME           TARGET STATE        SERVER                   STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.GRID.dg
               ONLINE ONLINE       tnc2
ora.LISTENER.lsnr
               ONLINE ONLINE       tnc2
ora.OVAL009P.dg
               ONLINE ONLINE       tnc2
ora.asm
               ONLINE ONLINE       tnc2                     Started
ora.gsd
               OFFLINE OFFLINE      tnc2
ora.net1.network
               ONLINE ONLINE       tnc2
ora.ons
               ONLINE ONLINE       tnc2
ora.registry.acfs
               ONLINE ONLINE       tnc2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE ONLINE       tnc2
ora.cvu
      1        ONLINE ONLINE       tnc2
ora.oc4j
      1        ONLINE ONLINE       tnc2
ora.oval009.db
      1        OFFLINE OFFLINE                               Instance Shutdown
      2        ONLINE ONLINE       tnc2                     Open
ora.scan1.vip
      1        ONLINE ONLINE       tnc2
ora.tnc1.vip
      1        ONLINE INTERMEDIATE tnc2                     FAILED OVER
ora.tnc2.vip
      1        ONLINE ONLINE       tnc2

[oracle@tnc2 tnc2]$ pwd
/u01/app/11.2.0/grid/log/tnc2
[oracle@tnc2 tnc2]$ tail -100f alerttnc2.log
[crsd(15372)]CRS-2772:Server 'tnc1' has been assigned to pool 'ora.oval009'.
2016-05-09 09:32:35.749:
[crsd(15372)]CRS-2772:Server 'tnc2' has been assigned to pool 'Generic'.
2016-05-09 09:32:35.750:
[crsd(15372)]CRS-2772:Server 'tnc2' has been assigned to pool 'ora.oval009'.
2016-05-09 10:49:02.615:
[cssd(14888)]CRS-1625:Node tnc1, number 1, was manually shut down
2016-05-09 10:49:02.627:
[cssd(14888)]CRS-1601:CSSD Reconfiguration complete. Active nodes are tnc2 .
2016-05-09 10:49:02.792:
[crsd(15372)]CRS-5504:Node down event reported for node 'tnc1'.
2016-05-09 10:49:02.805:
[crsd(15372)]CRS-2773:Server 'tnc1' has been removed from pool 'Generic'.
2016-05-09 10:49:02.806:
[crsd(15372)]CRS-2773:Server 'tnc1' has been removed from pool 'ora.oval009'.

[root@tnc1 ~]# ps -ef | grep pmon
oracle   15691     1 0 10:59 ?        00:00:00 asm_pmon_+ASM1
oracle   17336     1 0 11:00 ?        00:00:00 ora_pmon_oval0091
root     18623 14908 0 11:04 pts/4    00:00:00 grep pmon

##### Once the patch is successfully applied to both ORACLE & GRID_HOME, make sure Services are up & running on 1st node of the cluster #
After Apply the same step on all nodes of the cluster #

STEP6:
Run catbundle.sql scripts #
For each database instance running on the Oracle home being patched, connect to the database using SQL*Plus. Connect as SYSDBA and run the catbundle.sql script as follows
cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> STARTUP
SQL> @catbundle.sql psu apply
SQL> QUIT

STEP7:
Check for the list of patches applied to the database.
SQL> select action_time, patch_id, patch_uid, version, status, bundle_series, description from dba_registry_sqlpatch;

STEP8:
After running " catbundle.sql " step in all nodes of the cluster
run following command for patch validation # #
[oracle@tnc1 ~]$ opatch lsinventory -all_nodes

---Nikhil Tatineni--

Sunday, May 8, 2016

Creating OCM response File to Apply patch : RAC

How to Create an OCM Response file to Apply a Patch in Silent Mode - opatch silent (Doc ID 966023.1)

[oracle@tnc1 ocm]$ pwd

/u01/app/11.2.0/grid/OPatch/ocm

[oracle@tnc1 ocm]$ cd bin

[oracle@tnc1 bin]$ ls -ltr emocmrsp

ls: emocmrsp: No such file or directory

If You find There is no "emocmrsp" in GRID_HOME/OPatch/ocm/bin

DownLoad Recent OPatch to upgrade OPATCH version #

-rwxrwxr-x 1 oracle oinstall 52733900 May 8 01:02 p6880880_112000_Linux-x86-64.zip

drwxrwxr-x 2 oracle oinstall 4096 May 8 01:10 OPatch

[root@tnc2 grid]# pwd

/u01/app/11.2.0/grid

[oracle@tnc2 grid]$ unzip p6880880_112000_Linux-x86-64.zip

Archive: p6880880_112000_Linux-x86-64.zip

inflating: OPatch/opatchdiag

inflating: OPatch/opatch.pl

creating: OPatch/jlib/

inflating: OPatch/jlib/oracle.opatchcore.classpath.jar

inflating: OPatch/jlib/opatch.jar

inflating: OPatch/jlib/oracle.opatch.classpath.jar

inflating: OPatch/jlib/oracle.opatch.classpath.unix.jar

inflating: OPatch/jlib/oracle.opatch.classpath.windows.jar

inflating: OPatch/jlib/opatchsdk.jar

inflating: OPatch/jlib/oracle.opatchcore.classpath.unix.jar

inflating: OPatch/jlib/oracle.opatchcore.classpath.windows.jar

creating: OPatch/crs/

inflating: OPatch/crs/patch11203.pl

inflating: OPatch/crs/installPatch.excl

inflating: OPatch/crs/patch11202.pl

inflating: OPatch/crs/auto_patch.pl

creating: OPatch/crs/log/

inflating: OPatch/crs/patch112.pl

inflating: OPatch/crs/patchDB.pl

inflating: OPatch/crs/opatchauto

inflating: OPatch/crs/CRSProductDriver.jar

inflating: OPatch/crs/driver.jar

creating: OPatch/ocm/

creating: OPatch/ocm/doc/

extracting: OPatch/ocm/ocm.zip

inflating: OPatch/ocm/ocm_platforms.txt

creating: OPatch/ocm/lib/

inflating: OPatch/ocm/lib/emocmclnt.jar

inflating: OPatch/ocm/lib/xmlparserv2.jar

inflating: OPatch/ocm/lib/http_client.jar

inflating: OPatch/ocm/lib/emocmclnt-14.jar

inflating: OPatch/ocm/lib/regexp.jar

inflating: OPatch/ocm/lib/jnet.jar

inflating: OPatch/ocm/lib/jsse.jar

inflating: OPatch/ocm/lib/osdt_core3.jar

inflating: OPatch/ocm/lib/osdt_jce.jar

inflating: OPatch/ocm/lib/emocmcommon.jar

inflating: OPatch/ocm/lib/log4j-core.jar

inflating: OPatch/ocm/lib/jcert.jar

creating: OPatch/ocm/bin/

inflating: OPatch/ocm/bin/emocmrsp

inflating: OPatch/opatch

creating: OPatch/opatchauto-dir/

creating: OPatch/opatchauto-dir/opatchautocore/

inflating: OPatch/opatchauto-dir/opatchautocore/oplan

inflating: OPatch/opatchauto-dir/opatchautocore/opatchautobinary

creating: OPatch/opatchauto-dir/opatchautocore/jlib/

inflating: OPatch/opatchauto-dir/opatchautocore/jlib/ValidationRules.jar

inflating: OPatch/opatchauto-dir/opatchautocore/jlib/patchsdk.jar

inflating: OPatch/opatchauto-dir/opatchautocore/jlib/ProductDriver.jar

inflating: OPatch/opatchauto-dir/opatchautocore/jlib/osysmodel-utils.jar

inflating: OPatch/opatchauto-dir/opatchautocore/jlib/OsysModel.jar

creating: OPatch/opatchauto-dir/opatchautocore/jlib/apache-commons/

inflating: OPatch/opatchauto-dir/opatchautocore/jlib/apache-commons/commons-compress-1.4.jar

inflating: OPatch/opatchauto-dir/opatchautocore/jlib/apache-commons/commons-cli-1.0.jar

inflating: OPatch/opatchauto-dir/opatchautocore/jlib/oplan_core.jar

inflating: OPatch/opatchauto-dir/opatchautocore/jlib/oracle.oplan.classpath.jar

inflating: OPatch/opatchauto-dir/opatchautocore/jlib/bundle.jar

inflating: OPatch/opatchauto-dir/opatchautocore/jlib/oplan_sample.jar

inflating: OPatch/opatchauto-dir/opatchautocore/jlib/Validation.jar

creating: OPatch/opatchauto-dir/opatchautocore/jlib/jaxb/

inflating: OPatch/opatchauto-dir/opatchautocore/jlib/jaxb/activation.jar

inflating: OPatch/opatchauto-dir/opatchautocore/jlib/jaxb/jsr173_1.0_api.jar

inflating: OPatch/opatchauto-dir/opatchautocore/jlib/jaxb/jaxb-api.jar

inflating: OPatch/opatchauto-dir/opatchautocore/jlib/jaxb/jaxb-impl.jar

inflating: OPatch/opatchauto-dir/opatchautocore/jlib/automation.jar

inflating: OPatch/opatchauto-dir/opatchautocore/oplan.bat

inflating: OPatch/opatchauto-dir/opatchautocore/README.txt

inflating: OPatch/opatchauto-dir/opatchautocore/README.html

creating: OPatch/opatchauto-dir/opatchautodb/

creating: OPatch/opatchauto-dir/opatchautodb/jlib/

inflating: OPatch/opatchauto-dir/opatchautodb/jlib/oracle.oplan.db.classpath.jar

inflating: OPatch/opatchauto-dir/opatchautodb/jlib/oplan_db.jar

inflating: OPatch/emdpatch.pl

creating: OPatch/scripts/

inflating: OPatch/scripts/opatch_wls.bat

inflating: OPatch/scripts/opatch_jvm_discovery.bat

inflating: OPatch/scripts/opatch_wls

inflating: OPatch/scripts/opatch_jvm_discovery

inflating: OPatch/operr

inflating: OPatch/operr_readme.txt

inflating: OPatch/operr.bat

creating: OPatch/opatchprereqs/

inflating: OPatch/opatchprereqs/prerequisite.properties

creating: OPatch/opatchprereqs/oui/

inflating: OPatch/opatchprereqs/oui/knowledgesrc.xml

creating: OPatch/opatchprereqs/opatch/

inflating: OPatch/opatchprereqs/opatch/runtime_prereq.xml

inflating: OPatch/opatchprereqs/opatch/rulemap.xml

inflating: OPatch/opatchprereqs/opatch/opatch_prereq.xml

inflating: OPatch/opatch.bat

creating: OPatch/oplan/

creating: OPatch/oplan/jlib/

creating: OPatch/oplan/jlib/apache-commons/

inflating: OPatch/oplan/jlib/apache-commons/commons-cli-1.0.jar

creating: OPatch/oplan/jlib/jaxb/

inflating: OPatch/oplan/jlib/jaxb/jsr173_1.0_api.jar

inflating: OPatch/oplan/jlib/jaxb/jaxb-impl.jar

inflating: OPatch/oplan/jlib/jaxb/activation.jar

inflating: OPatch/oplan/jlib/jaxb/jaxb-api.jar

inflating: OPatch/oplan/jlib/Validation.jar

inflating: OPatch/oplan/jlib/oracle.oplan.classpath.jar

inflating: OPatch/oplan/jlib/OuiDriver.jar

inflating: OPatch/oplan/jlib/EMrepoDrivers.jar

inflating: OPatch/oplan/jlib/JMXDrivers.jar

inflating: OPatch/oplan/jlib/bundle.jar

inflating: OPatch/oplan/jlib/CRSProductDriver.jar

inflating: OPatch/oplan/jlib/oplan.jar

inflating: OPatch/oplan/jlib/ValidationRules.jar

inflating: OPatch/oplan/jlib/osysmodel-utils.jar

inflating: OPatch/oplan/jlib/automation.jar

inflating: OPatch/oplan/README.html

inflating: OPatch/oplan/README.txt

inflating: OPatch/oplan/oplan.bat

inflating: OPatch/oplan/oplan

inflating: OPatch/opatchdiag.bat

inflating: OPatch/README.txt

extracting: OPatch/version.txt

creating: OPatch/docs/

inflating: OPatch/docs/cversion.txt

inflating: OPatch/docs/Prereq_Users_Guide.txt

inflating: OPatch/docs/FAQ

inflating: OPatch/docs/Users_Guide.txt

[oracle@tnc2 grid]$

[oracle@tnc2 grid]$ pwd

/u01/app/11.2.0/grid

[oracle@tnc2 grid]$ cd OPatch

[oracle@tnc2 OPatch]$ ls -ltr

total 156

-rwxr-x--- 1 oracle oinstall 31056 Aug 18 2015 opatch

-rw-r----- 1 oracle oinstall 27 Aug 18 2015 version.txt

drwxr-x--- 2 oracle oinstall 4096 Aug 18 2015 scripts

-rw-r----- 1 oracle oinstall 2915 Aug 18 2015 README.txt

drwxr-x--- 3 oracle oinstall 4096 Aug 18 2015 oplan

-rw-r----- 1 oracle oinstall 3177 Aug 18 2015 operr_readme.txt

-rwxr-x--- 1 oracle oinstall 4220 Aug 18 2015 operr.bat

-rwxr-x--- 1 oracle oinstall 3161 Aug 18 2015 operr

-rwxr-x--- 1 oracle oinstall 2576 Aug 18 2015 opatch.pl

-rwxr-x--- 1 oracle oinstall 9352 Aug 18 2015 opatchdiag.bat

-rwxr-x--- 1 oracle oinstall 10056 Aug 18 2015 opatchdiag

-rwxr-x--- 1 oracle oinstall 15184 Aug 18 2015 opatch.bat

drwxr-x--- 2 oracle oinstall 4096 Aug 18 2015 jlib

-rwxr-x--- 1 oracle oinstall 23695 Aug 18 2015 emdpatch.pl

drwxr-x--- 4 oracle oinstall 4096 Aug 18 2015 opatchprereqs

drwxr-x--- 3 oracle oinstall 4096 Aug 18 2015 crs

drwxr-x--- 5 oracle oinstall 4096 Aug 18 2015 ocm

drwxr-x--- 4 oracle oinstall 4096 Aug 18 2015 opatchauto-dir

drwxr-x--- 2 oracle oinstall 4096 Aug 18 2015 docs

[oracle@tnc2 OPatch]$ cd ocm

[oracle@tnc2 ocm]$ ls -ltr

total 41628

-rw-r----- 1 oracle oinstall 638 Aug 18 2015 ocm_platforms.txt

drwxr-x--- 2 oracle oinstall 4096 Aug 18 2015 doc

-rwxr-x--- 1 oracle oinstall 42560960 Aug 18 2015 ocm.zip

drwxr-x--- 2 oracle oinstall 4096 Aug 18 2015 lib

drwxr-x--- 2 oracle oinstall 4096 Aug 18 2015 bin

[oracle@tnc2 ocm]$ cd bin

[oracle@tnc2 bin]$ ls -ltr

total 12

-rwxr-x--- 1 oracle oinstall 9063 Nov 27 2009 emocmrsp

[oracle@tnc2 bin]$

[oracle@tnc2 bin]$ pwd
/u01/app/11.2.0/grid/OPatch/ocm/bin

Creating ocm.rsp file as follows #

[oracle@tnc1 bin]$ ./emocmrsp -no_banner -output /u01/stage/file.rsp

Provide your email address to be informed of security issues, install and

initiate Oracle Configuration Manager. Easier for you if you use your My

Oracle Support Email address/User Name.

Visit http://www.oracle.com/support/policies.html for details.

Email address/User Name:

You have not provided an email address for notification of security issues.

Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y

The OCM configuration response file (/u01/app/oracle/file.rsp) was successfully created.

--Nikhil Tatineni--
--12C RAC --

Tuesday, May 3, 2016

read position and write position of GoldenGate pump process

How do we check current read position and write position of GoldenGate pump process

GGSCI (hidb401) 60> send PTNMLDB, status

Sending STATUS request to EXTRACT PTNMLDB ...
EXTRACT PTNMLDB (PID 39845948)
Current status: Recovery complete: At EOF

Current read position:
Sequence #: 6
RBA: 1113
Timestamp: 2016-04-04 14:14:55.000000
Extract Trail: /oraarch/oval013t/fs01/ggs01/trails/pg

Current write position:
Sequence #: 0
RBA: 0
Timestamp: 2016-04-04 14:18:43.860423
Extract Trail: /opt/app/t1ova1d/ggs/product/11.1.1/dirdat/pg

---Nikhil Tatineni--
---GoldenGate --

Virtual IP address is not configured for cluster node

scenario:# Creating database using dbca in RAC cluster & failed with following error.
Virtual IP address is not configured for cluster node " run /u01/app/11.2.0/grid/bin/srvctl as root to configure before performing the "create a database" operation

-------#
We found, there is NO VIP on second node of RAC cluster #
Configured VIP on second node as follows

On Node 1#

[oracle@tnc1 ~]$ srvctl config nodeapps -a

Network exists: 1/192.168.1.0/255.255.255.0/eth0, type static

VIP exists: /tnc1-vip/192.168.1.11/192.168.1.0/255.255.255.0/eth0, hosting node tnc1

On Node 2 # As a root User #

[root@tnc2 grid]# srvctl add vip -n tnc2 -k 1 -A 192.168.1.21/255.255.255.0/eth0

[root@tnc2 grid]# srvctl start nodeapps -n tnc2

[root@tnc2 grid]# srvctl config nodeapps -a

Network exists: 1/192.168.1.0/255.255.255.0/eth0, type static

VIP exists: /tnc1-vip/192.168.1.11/192.168.1.0/255.255.255.0/eth0, hosting node tnc1

VIP exists: /192.168.1.21/192.168.1.21/192.168.1.0/255.255.255.0/eth0, hosting node tnc2

After creating VIP, Installed database successfully

---Nikhil Tatineni--
---12c RAC---

Sunday, May 1, 2016

CRS-4402, CRS-2800, CRS-4000 : root.sh failed on second Node of RAC cluster

### Scenario:

/u01/app/11.2.0/grid/root.sh is failed on the second Node of the RAC cluster ##

[root@tnc2 ~]# /u01/app/11.2.0/grid/root.sh

Performing root user operation for Oracle 11g

The following environment variables are set as:

ORACLE_OWNER= oracle

ORACLE_HOME= /u01/app/11.2.0/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:

Copying dbhome to /usr/local/bin ...

Copying oraenv to /usr/local/bin ...

Copying coraenv to /usr/local/bin ...

Creating /etc/oratab file...

Entries will be added to the /etc/oratab file as needed by

Database Configuration Assistant when a database is created

Finished running generic part of root script.

Now product-specific root actions will be performed.

Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params

Creating trace directory

User ignored Prerequisites during installation

Installing Trace File Analyzer

OLR initialization - successful

Adding Clusterware entries to inittab

CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node tnc1, number 1, and is terminating

An active cluster was found during exclusive startup, restarting to join the cluster

Start of resource "ora.crsd" failed

CRS-2800: Cannot start resource 'ora.asm' as it is already in the INTERMEDIATE state on server 'tnc2'

CRS-4000: Command Start failed, or completed with errors.

Failed to start Oracle Grid Infrastructure stack

Failed to start Cluster Ready Services at /u01/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 1353.

/u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed

Node Alert log file shows as follows

[/u01/app/11.2.0/grid/bin/oraagent.bin(28566)]CRS-5019:All OCR locations are on ASM disk groups [GRID], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/tnc2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".

2016-04-30 16:28:08.453:

2016-04-30 16:28:38.628:

2016-04-30 16:29:08.727:

2016-04-30 16:29:38.867:

error in ohasd logs as follows "/u01/app/11.2.0/grid/log/tnc2/agent/ohasd/oraagent_oracle/oraagent_oracle.log"

2016-04-30 16:53:26.893: [ora.asm][1075857728]{0:0:2} [check] checkCrsStat 2 CLSCRS_STAT ret: 184

2016-04-30 16:53:26.893: [ora.asm][1075857728]{0:0:2} [check] clsnUtils::error Exception type=2 string=

2016-04-30 16:53:26.893: [ora.asm][1075857728]{0:0:2} [check] AsmAgent::checkCbk: Exception UserErrorException

2016-04-30 16:53:26.893: [ora.asm][1075857728]{0:0:2} [check]

2016-04-30 16:53:26.893: [ora.asm][1075857728]{0:0:2} [check] InstAgent::check 1 prev clsagfw_res_status 4 current clsagfw_res_status 4

2016-04-30 16:53:27.896: [ora.asm][1077958976]{0:0:2} [check] AsmAgent::check ocrCheck 1 m_OcrOnline 0 m_OcrTimer 161

2016-04-30 16:53:27.896: [ora.asm][1077958976]{0:0:2} [check] CrsCmd::ClscrsCmdData::stat entity 5 statflag 32 useFilter 1

2016-04-30 16:53:27.897: [ COMMCRS][1133730112]clsc_connect: (0x1f159cb0) no listener at (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_UI_SOCKET))

2016-04-30 16:53:27.897: [ora.asm][1077958976]{0:0:2} [check] checkCrsStat 2 CLSCRS_STAT ret: 184

2016-04-30 16:53:27.897: [ora.asm][1077958976]{0:0:2} [check] clsnUtils::error Exception type=2 string=

2016-04-30 16:53:27.897: [ora.asm][1077958976]{0:0:2} [check] AsmAgent::checkCbk: Exception UserErrorException

2016-04-30 16:53:27.897: [ora.asm][1077958976]{0:0:2} [check]

2016-04-30 16:53:27.897: [ora.asm][1077958976]{0:0:2} [check] InstAgent::check 1 prev clsagfw_res_status 4 current clsagfw_res_status 4

2016-04-30 16:53:28.900: [ora.asm][1077958976]{0:0:2} [check] AsmAgent::check ocrCheck 1 m_OcrOnline 0 m_OcrTimer 162

2016-04-30 16:53:28.900: [ora.asm][1077958976]{0:0:2} [check] CrsCmd::ClscrsCmdData::stat entity 5 statflag 32 useFilter 1

2016-04-30 16:53:28.901: [ COMMCRS][1133730112]clsc_connect: (0x1f159cb0) no listener at (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_UI_SOCKET))

2016-04-30 16:53:28.901: [ora.asm][1077958976]{0:0:2} [check] checkCrsStat 2 CLSCRS_STAT ret: 184

2016-04-30 16:53:28.901: [ora.asm][1077958976]{0:0:2} [check] clsnUtils::error Exception type=2 string=2016-04-30 16:53:2

logged into ASM instance on second Node and query the disk status, it showed diskgroups is Dismounted. It clearly shows that DISKGROUP is not mounted on second node of the Cluster

######

SQL> select GROUP_NUMBER,NAME,OFFLINE_DISKS, state from v$asm_diskgroup;

GROUP_NUMBER NAME OFFLINE_DISKS STATE

------------ ------------------------------ ------------- -------------

0 GRID 0 DISMOUNTED

[root@tnc2 bin]# ./crsctl query css votedisk

## STATE File Universal Id File Name Disk group

-- ----- ----------------- --------- ---------

1. ONLINE 357fa62799a74fcdbf518b0f0d6d9a29 (/dev/raw/raw1) [GRID]

2. OFFLINE 216d1cb038d14fc1bf8ccc05b0af6793 () []

3. ONLINE 061664871d014ff7bfac218182629a7d (/dev/raw/raw3) [GRID]

Located 3 voting disk(s).

Continued to install & Successfully Installed the GRID cluster : CVU is failed at end of grid installation, I found that there is no permission on disks on second node of the cluster

Later worked with SA by setting Udev Rules on the disks attached to Server

stopped crs force on node of the cluster and After restarted the crs on both nodes of the cluster ##

[root@tnc2 ~]# crsctl stop crs -f

CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'tnc2'

CRS-2673: Attempting to stop 'ora.mdnsd' on 'tnc2'

CRS-2673: Attempting to stop 'ora.crf' on 'tnc2'

CRS-2673: Attempting to stop 'ora.ctssd' on 'tnc2'

CRS-2673: Attempting to stop 'ora.evmd' on 'tnc2'

CRS-2673: Attempting to stop 'ora.asm' on 'tnc2'

CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'tnc2'

CRS-2677: Stop of 'ora.evmd' on 'tnc2' succeeded

CRS-2677: Stop of 'ora.mdnsd' on 'tnc2' succeeded

CRS-2677: Stop of 'ora.crf' on 'tnc2' succeeded

CRS-2677: Stop of 'ora.asm' on 'tnc2' succeeded

CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'tnc2'

CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'tnc2' succeeded

CRS-2677: Stop of 'ora.drivers.acfs' on 'tnc2' succeeded

CRS-2677: Stop of 'ora.ctssd' on 'tnc2' succeeded

CRS-2673: Attempting to stop 'ora.cssd' on 'tnc2'

CRS-2677: Stop of 'ora.cssd' on 'tnc2' succeeded

CRS-2673: Attempting to stop 'ora.gipcd' on 'tnc2'

CRS-2677: Stop of 'ora.gipcd' on 'tnc2' succeeded

CRS-2673: Attempting to stop 'ora.gpnpd' on 'tnc2'

CRS-2677: Stop of 'ora.gpnpd' on 'tnc2' succeeded

CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'tnc2' has completed

CRS-4133: Oracle High Availability Services has been stopped.

# After #
# crsctl start crs

## Issue is resolved ##

--Nikhil Tatineni--

--RAC --

Tuesday, April 12, 2016

ERROR OGG-01705 Input checkpoint position 119195676...

Replicat Abended with following Error
ERROR OGG-01705 Input checkpoint position 119195676 for input trail file '/opt/app/ggs/trails/af000047' is greater than the size of the file (119194872). Please consult Oracle Knowledge Management Doc ID 1138409.1. for instructions.

Rootcause: Checkpoint is struck inside the trail and RBA keep on increasing and make Replicat Abended

Potential Fix: In-order to run fix replicat and avoid duplicate processing of records follow the steps as follows
From report File
A) 119195676 ( Check point Position on Trail )
B) 119194872 ( Size of the Trail )

In order to start Replicat We have to follow Formulae
Replicat RBA = (checkpoint position on Trail)-(Size of the Trail) + (RBA of the First record on next trail after the restart abended)

Procedure to get info about " (RBA of the First record on next trail after the restart abended)"
GGSCI (tnc63) 39> sh ls -ltr /opt/app/oval007p/ggs/trails/af*
-rw-r----- 1 ggs01 ggsoper 119194872 Apr 09 14:51 /opt/app/ggs/trails/af000047
-rw-r----- 1 ggs01 ggsoper 1980582 Apr 09 16:46 /opt/app/ggs/trails/af000048

./logdump
logdump> open /opt/app/ggs/trails/af000048
logdump> ghdr on
logdump> detail data
logdump> pos 0
logdump> n
2016/04/09 14:51:34.054.664 RestartAbend Len 0 RBA 1128
Name:
After Image: Partition 0 G s

Logdump>n
Hdr-Ind : E (x45) Partition : . (x04)
UndoFlag : . (x00) BeforeAfter: A (x41)
RecLength : 345 (x0159) IO Time : 2016/04/09 13:50:29.987.473
IOType : 5 (x05) OrigNode : 255 (xff)
TransInd : . (x03) FormatType : R (x52)
SyskeyLen : 0 (x00) Incomplete : . (x00)
AuditRBA : 173237 AuditPos : 165583888
Continued : N (x00) RecCount : 1 (x01)
2016/04/09 13:50:29.987.473 Insert Len 345 RBA 1190
Name:

####
RBA= 119195676 - 119194872 + 1190 = 1994

Now Alter replicat with n+1 seq number and RBA
ggsci> alter repone, extseqno 48 extrba 1994
ggsci> start repone

Regards,
Nikhil Tatineni,
GoldenGate.

Tuesday, April 5, 2016

ERROR OGG-01755 Cannot register or unregister EXTRACT

Encountered following Error while registering integrated extract with database

ERROR OGG-01755 Cannot register or unregister EXTRACT extcbn because of the following SQL error: OCI Error retrieving bind info for query (status = 100). See Extract user privileges in the Oracle GoldenGate for Oracle Installation and Setup Guide

Root Cause as follows

Server crash, rebooted, OGG or database is not shutdown cleanly, the Integrated Extract is left in an indeterminate state

Unregister Extract Manually from database as follows

SQL> exec DBMS_XSTREAM_GG_ADM.STOP_OUTBOUND('OGG$MYEXTRACT',true);
SQL> exec DBMS_XSTREAM_ADM.DROP_OUTBOUND('OGG$MYEXTRACT');
SQL> exec DBMS_STREAMS_ADM.REMOVE_QUEUE(queue_name => 'GGMINING.OGG$Q_MYEXTRACT', cascade => true, drop_unused_queue_table => true);

If the above works correctly, execute these selects to check.

SQL> select * from DBA_XSTREAM_OUTBOUND;
SQL> select * from SYS.XSTREAM$_SERVER;
Make sure you don't see anything to do with this extract

I got this whole information from METALINK, we have to replace "MYEXTRACT" with your Extract Name

---Nikhil Tatineni---

---Oracle In Memory---