using opatch to check patches installed on All Nodes of the RAC Cluster #
[oracle@tnc1 ~]$ opatch lsinventory -all_nodes
Oracle Interim Patch Installer version 11.2.0.3.12
Copyright (c) 2016, Oracle Corporation. All rights reserved.
Oracle Home : /u01/app/oracle/product/11.2.0/db_1
Central Inventory : /u01/app/oraInventory
from : /u01/app/oracle/product/11.2.0/db_1/oraInst.loc
OPatch version : 11.2.0.3.12
OUI version : 11.2.0.4.0
Log file location : /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/opatch2016-05-09_11-13-45AM_1.log
Lsinventory Output file location : /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatch/lsinv/lsinventory2016-05-09_11-13-45AM.txt
--------------------------------------------------------------------------------
Local Machine Information::
Hostname: tnc1.localdomain
ARU platform id: 226
ARU platform description:: Linux x86-64
Rac system comprising of multiple nodes
Local node = tnc1
Remote node = tnc2
Installed Patch List(s)
======================
Node Name : tnc1
----------------
Interim patches (2) :
Patch 22502549 : applied on Mon May 09 10:48:30 EDT 2016
Unique Patch ID: 19880366
Patch description: "OCW Patch Set Update : 11.2.0.4.160419 (22502549)"
Created on 11 Mar 2016, 01:34:41 hrs PST8PDT
Bugs fixed:
19270660, 18328800, 18508710, 18691572, 20038451, 21251192, 21232394
20365005, 17750548, 17387214, 17617807, 14497275, 20219458, 17733927
18180541, 18962892, 17292250, 17378618, 16759171, 20110156, 17843489
17065496, 13991403, 17273020, 17155238, 20012766, 21245437, 18261183
18053580, 20218012, 17013634, 17886392, 20995001, 17039197, 17947785
16317771, 10052729, 22353346, 20340620, 16237657, 20317221, 15917869
18199185, 18399991, 20186278, 17374271, 18024089, 16849642, 20746251
20246071, 14270845, 20552947, 18882642, 18414137, 17001914, 17927970
14378120, 16346413, 15986647, 18068871, 21222147, 18143836, 16206997
21982225, 19168690, 20235511, 18343490, 21875360, 16613232, 19276791
17722664, 20440643, 12928658, 18952577, 18520351, 16249829, 18226143
16076412, 18265482, 18229842, 17172091, 20676340, 17818075, 20091753
18231837, 14373486, 20136892, 17483479, 20551654, 18120545, 18729166
13843841, 21225209, 17405302, 18709496, 18330979, 18744838, 17087371
20531190, 14525998, 18187697, 20598625, 14385860, 18348155, 19479503
12928592, 17516024, 18370031, 17764053, 19272663, 17551223, 14671408
18272135, 14207615, 21255373, 17500165, 18875012, 14769643, 18464784
19558324, 18848125, 19241857, 14851828, 17955615, 20315294, 14693336
16284825, 17352230, 20014326, 17238586, 17089344, 17405605, 21327402
17531342, 19398098, 17159489, 17640316, 13823394, 16543190, 22024217
17983675, 20795241, 17598201, 17481314, 16281493, 18346135, 15986311
19601468, 17208793, 18700935, 18999857, 14076173, 18428146, 18352845
17435488, 20408163, 17592037, 18352846, 19616601, 17391726, 17387779
14777968, 15851860, 16206882, 20141091, 21113068, 20175174, 17305100
15832129, 19885321, 16901346, 17985714, 18536826, 17780903, 18752378
18946768, 16876500, 16875342, 17769597, 19955755, 16429265, 18336452
17273003, 17209968, 16988311, 20094984, 19319357, 17059927, 17046460
18053631, 16867761, 18774591, 21442094, 20235486, 19359787, 15869775
19642566, 17447588, 16798862, 15920201
Patch 22502456 : applied on Mon May 09 10:46:10 EDT 2016
Unique Patch ID: 19856194
Patch description: "Database Patch Set Update : 11.2.0.4.160419 (22502456)"
Created on 21 Mar 2016, 11:49:22 hrs
Sub-patch 21948347; "Database Patch Set Update : 11.2.0.4.160119 (21948347)"
Sub-patch 21352635; "Database Patch Set Update : 11.2.0.4.8 (21352635)"
Sub-patch 20760982; "Database Patch Set Update : 11.2.0.4.7 (20760982)"
Sub-patch 20299013; "Database Patch Set Update : 11.2.0.4.6 (20299013)"
Sub-patch 19769489; "Database Patch Set Update : 11.2.0.4.5 (19769489)"
Sub-patch 19121551; "Database Patch Set Update : 11.2.0.4.4 (19121551)"
Sub-patch 18522509; "Database Patch Set Update : 11.2.0.4.3 (18522509)"
Sub-patch 18031668; "Database Patch Set Update : 11.2.0.4.2 (18031668)"
Sub-patch 17478514; "Database Patch Set Update : 11.2.0.4.1 (17478514)"
Bugs fixed:
17288409, 21051852, 17811429, 18607546, 17205719, 20506699, 17816865
17922254, 17754782, 16934803, 13364795, 17311728, 17441661, 17284817
16992075, 17446237, 14015842, 19972569, 21756677, 21538558, 20925795
17449815, 17375354, 19463897, 13866822, 17982555, 17235750, 17478514
18317531, 14338435, 18235390, 20803583, 13944971, 20142975, 17811789
16929165, 18704244, 20506706, 17546973, 20334344, 14054676, 17088068
17346091, 18264060, 17343514, 21538567, 19680952, 18471685, 19211724
13951456, 21847223, 16315398, 18744139, 16850630, 19049453, 18673304
17883081, 19915271, 18641419, 18262334, 17006183, 16065166, 18277454
16833527, 10136473, 18051556, 17865671, 17852463, 18554871, 17853498
18334586, 17551709, 17588480, 19827973, 17344412, 17842825, 18828868
17025461, 11883252, 13609098, 17239687, 17602269, 19197175, 22195457
18316692, 17313525, 12611721, 19544839, 18964939, 17600719, 18191164
19393542, 17571306, 18482502, 20777150, 19466309, 17040527, 17165204
18098207, 16785708, 17465741, 17174582, 16180763, 16777840, 12982566
19463893, 22195465, 16875449, 12816846, 17237521, 19358317, 17811438
17811447, 21983325, 17945983, 18762750, 16912439, 17184721, 18061914
17282229, 18331850, 18202441, 17082359, 18723434, 21972320, 19554106
14034426, 18339044, 19458377, 17752995, 20448824, 17891943, 17258090
17767676, 16668584, 18384391, 17040764, 17381384, 15913355, 18356166
14084247, 20596234, 20506715, 21756661, 13853126, 18203837, 14245531
21756699, 16043574, 22195441, 17848897, 17877323, 21453153, 17468141
20861693, 17786518, 17912217, 17037130, 18155762, 16956380, 17478145
17394950, 18641461, 18189036, 18619917, 17027426, 21352646, 16268425
22195492, 19584068, 18436307, 17265217, 17634921, 13498382, 21526048
19258504, 20004087, 17443671, 22195485, 18000422, 20004021, 22321756
17571039, 21067387, 16344544, 18009564, 14354737, 21286665, 18135678
18614015, 20441797, 18362222, 17835048, 16472716, 17936109, 17050888
17325413, 14010183, 18747196, 17761775, 16721594, 17082983, 20067212
21179898, 17302277, 18084625, 15990359, 18203835, 17297939, 17811456
16731148, 21168487, 13829543, 17215560, 14133975, 17694209, 17385178
18091059, 8322815, 17586955, 17201159, 17655634, 18331812, 19730508
18868646, 17648596, 16220077, 16069901, 17348614, 17393915, 17274537
17957017, 18096714, 17308789, 18436647, 14285317, 19289642, 14764829
18328509, 17622427, 22195477, 16943711, 22502493, 14368995, 17346671
18996843, 17783588, 21343838, 16618694, 17672719, 18856999, 18783224
17851160, 17546761, 17798953, 18273830, 22092979, 16596890, 19972566
16384983, 17726838, 17360606, 22321741, 13645875, 18199537, 16542886
21787056, 17889549, 14565184, 17071721, 17610798, 20299015, 21343897
22893153, 20657441, 17397545, 18230522, 16360112, 19769489, 12905058
18641451, 12747740, 18430495, 17016369, 17042658, 14602788, 17551063
19972568, 21517440, 18508861, 19788842, 14657740, 17332800, 13837378
19972564, 17186905, 18315328, 19699191, 17437634, 22353199, 18093615
19006849, 19013183, 17296856, 18674024, 17232014, 16855292, 17762296
14692762, 21051840, 17705023, 19121551, 21330264, 19854503, 21868720
19309466, 18681862, 18554763, 20558005, 17390160, 18456514, 16306373
13955826, 18139690, 17501491, 17752121, 21668627, 17299889, 17889583
18673325, 19721304, 18293054, 17242746, 17951233, 17649265, 18094246
19615136, 17011832, 16870214, 17477958, 18522509, 20631274, 16091637
17323222, 16595641, 16524926, 18228645, 18282562, 17596908, 17156148
18031668, 16494615, 22683225, 17545847, 17655240, 17614134, 13558557
17341326, 17891946, 17716305, 16392068, 19271443, 21351877, 18092127
18440047, 17614227, 14106803, 16903536, 18973907, 18673342, 19032867
17389192, 17612828, 16194160, 17006570, 17721717, 17390431, 17570240
16863422, 18325460, 19727057, 16422541, 19972570, 17267114, 18244962
21538485, 18765602, 18203838, 16198143, 17246576, 14829250, 17835627
18247991, 14458214, 21051862, 16692232, 17786278, 17227277, 16042673
16314254, 16228604, 16837842, 17393683, 17787259, 20331945, 20074391
15861775, 16399083, 18018515, 22683212, 18260550, 21051858, 17036973
16613964, 17080436, 16579084, 18384537, 18280813, 20296213, 16901385
15979965, 18441944, 16450169, 9756271, 17892268, 11733603, 16285691
17587063, 21343775, 16538760, 18180390, 18193833, 21387964, 21051833
17238511, 17824637, 16571443, 18306996, 14852021, 18674047, 17853456
12364061, 22195448
Node Name : tnc2
----------------
There are no Interim patches installed in this Oracle Home.
Binary & Checksum Information
==============================
Binary Location : /u01/app/oracle/product/11.2.0/db_1/bin/oracle
Node Size Checksum
---- ---- --------
tnc1 239840518 CA25EF1480C4A31241F7E1D95F9BE47D10B16318882A383FE5A55D584F0620FC
tnc2 239627031 C2A63A5807C20F7415001E62A0FCFBE93ECED91352A330C24A15C3A544553A62
The binary and checksum values of the following nodes does not match with that of tnc1 (local node)
tnc2
--------------------------------------------------------------------------------
OPatch Session completed with warnings.
OPatch completed with warnings.
-- Nikhil Tatineni--
-- RAC --
Monday, May 9, 2016
Applying APRIL PSU "Patch Number : Patch 22738793
If we are applying APRIL PSU "Patch Number : 22646198: Component 11.2.0.4.160419 DB PSU + GI PSU 11.2.0.4.160419 (Apr2016) &
Patch 23274134 - Oracle Grid Infrastructure Patch Set Update 11.2.0.4.160719 (Jul2016) (Includes Database PSU 11.2.0.4.160719)
Procedure is same as follows #
Note: when you unzip the Combo patch, if you don't see patches listed in readme.html download the patch separately
Patch Can be applied in Rolling Fashion #
STEP1: Before Applying Any Patch #
Take backup of all Grid_home & Database_homes in all nodes of cluster #
STEP2:
Make sure ocm.rsp " response file is created to apply patch in all nodes of cluster #
For patch 22646198/
11.2.0.4.160419 DB PSU + GI PSU 11.2.0.4.160419 (Apr2016)
This patch can be applied in rolling Fashion # There is No downTime Required for this Patch ##
STEP3: Stop EM Agent Processes Prior to Patching and Prior to Rolling Back the Patch as ORACLE HOME owner #
$ <ORACLE_HOME>/bin/emctl stop dbconsole
b)Validation of Oracle Inventory - check the consistency of inventory information for GI home and each database home
$ opatch lsinventory -detail -oh <GRID_HOME>
$ opatch lsinventory -detail -oh <ORACLE_HOME>STEP4:
Check for one-off patch conflict detection and resolution or Run it with Root User
Run it with Oracle / GRID User:
FOR ORACLE_HOME
/u01/app/11.2.0/grid/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -oh /u01/app/oracle/product/11.2.0/db_1 -phBaseDir /stage/22646198
For GRID HOME
/u01/app/11.2.0/grid/OPatch/opatch prereq CheckConflictAgainstOHWithDetail -oh /u01/app/11.2.0/grid -phBaseDir /stage/22646198
STEP5: Once we confirm, there is no conflicts, go ahead and apply patch to database home & grid home USING "ocm.rsp" file.
If GI Home and the Database Homes that are not shared and ACFS file system is not configured apply patch as follows:
Syntax: opatch auto <Patch Location> -ocmrf <ocm response file>
# opatch auto /stage/22646198 -ocmrf /stage/apply_april_psu.rsp
OutPut as follows # Executing /u01/app/11.2.0/grid/perl/bin/perl /u01/app/oracle/product/11.2.0/db_1/OPatch/crs/patch11203.pl -patchdir /stage -patchn 22646198 -ocmrf /stage/apply_april_psu.rsp -paramfile /u01/app/11.2.0/grid/crs/install/crsconfig_params
This is the main log file: /u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatchauto2016-05-09_10-41-31.log
This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:
/u01/app/oracle/product/11.2.0/db_1/cfgtoollogs/opatchauto2016-05-09_10-41-31.report.log
2016-05-09 10:41:31: Starting Clusterware Patch Setup
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Stopping RAC /u01/app/oracle/product/11.2.0/db_1 ...
Stopped RAC /u01/app/oracle/product/11.2.0/db_1 successfully
patch /stage/22646198/22502456 apply successful for home /u01/app/oracle/product/11.2.0/db_1
patch /stage/22646198/22502549/custom/server/22502549 apply successful for home /u01/app/oracle/product/11.2.0/db_1
Stopping CRS...
Stopped CRS successfully
patch /stage/22646198/22502456 apply successful for home /u01/app/11.2.0/grid
patch /stage/22646198/22502549 apply successful for home /u01/app/11.2.0/grid
patch /stage/22646198/22502505 apply successful for home /u01/app/11.2.0/grid
Starting CRS...
Installing Trace File Analyzer
CRS-4123: Oracle High Availability Services has been started.
Starting RAC /u01/app/oracle/product/11.2.0/db_1 ...
Started RAC /u01/app/oracle/product/11.2.0/db_1 successfully
opatch auto succeeded.
Things You can observer on Node 2 # and in alert log file #
[root@tnc2 tnc2]# crsctl status resource -t
--------------------------------------------------------------------------------
NAME TARGET STATE SERVER STATE_DETAILS
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.GRID.dg
ONLINE ONLINE tnc2
ora.LISTENER.lsnr
ONLINE ONLINE tnc2
ora.OVAL009P.dg
ONLINE ONLINE tnc2
ora.asm
ONLINE ONLINE tnc2 Started
ora.gsd
OFFLINE OFFLINE tnc2
ora.net1.network
ONLINE ONLINE tnc2
ora.ons
ONLINE ONLINE tnc2
ora.registry.acfs
ONLINE ONLINE tnc2
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
1 ONLINE ONLINE tnc2
ora.cvu
1 ONLINE ONLINE tnc2
ora.oc4j
1 ONLINE ONLINE tnc2
ora.oval009.db
1 OFFLINE OFFLINE Instance Shutdown
2 ONLINE ONLINE tnc2 Open
ora.scan1.vip
1 ONLINE ONLINE tnc2
ora.tnc1.vip
1 ONLINE INTERMEDIATE tnc2 FAILED OVER
ora.tnc2.vip
1 ONLINE ONLINE tnc2
[oracle@tnc2 tnc2]$ pwd
/u01/app/11.2.0/grid/log/tnc2
[oracle@tnc2 tnc2]$ tail -100f alerttnc2.log
[crsd(15372)]CRS-2772:Server 'tnc1' has been assigned to pool 'ora.oval009'.
2016-05-09 09:32:35.749:
[crsd(15372)]CRS-2772:Server 'tnc2' has been assigned to pool 'Generic'.
2016-05-09 09:32:35.750:
[crsd(15372)]CRS-2772:Server 'tnc2' has been assigned to pool 'ora.oval009'.
2016-05-09 10:49:02.615:
[cssd(14888)]CRS-1625:Node tnc1, number 1, was manually shut down
2016-05-09 10:49:02.627:
[cssd(14888)]CRS-1601:CSSD Reconfiguration complete. Active nodes are tnc2 .
2016-05-09 10:49:02.792:
[crsd(15372)]CRS-5504:Node down event reported for node 'tnc1'.
2016-05-09 10:49:02.805:
[crsd(15372)]CRS-2773:Server 'tnc1' has been removed from pool 'Generic'.
2016-05-09 10:49:02.806:
[crsd(15372)]CRS-2773:Server 'tnc1' has been removed from pool 'ora.oval009'.
[root@tnc1 ~]# ps -ef | grep pmon
oracle 15691 1 0 10:59 ? 00:00:00 asm_pmon_+ASM1
oracle 17336 1 0 11:00 ? 00:00:00 ora_pmon_oval0091
root 18623 14908 0 11:04 pts/4 00:00:00 grep pmon
##### Once the patch is successfully applied to both ORACLE & GRID_HOME, make sure Services are up & running on 1st node of the cluster #
After Apply the same step on all nodes of the cluster #
STEP6:
Run catbundle.sql scripts #
For each database instance running on the Oracle home being patched, connect to the database using SQL*Plus. Connect as SYSDBA and run the catbundle.sql script as follows
cd $ORACLE_HOME/rdbms/admin
sqlplus /nolog
SQL> CONNECT / AS SYSDBA
SQL> STARTUP
SQL> @catbundle.sql psu apply
SQL> QUIT
STEP7:
Check for the list of patches applied to the database.
SQL> select action_time, patch_id, patch_uid, version, status, bundle_series, description from dba_registry_sqlpatch;
STEP8:
After running " catbundle.sql " step in all nodes of the cluster
run following command for patch validation # #
[oracle@tnc1 ~]$ opatch lsinventory -all_nodes
---Nikhil Tatineni--
Sunday, May 8, 2016
Creating OCM response File to Apply patch : RAC
How to Create an OCM Response file to Apply a Patch in Silent Mode - opatch silent (Doc ID 966023.1)
|
[oracle@tnc1 ocm]$ pwd
/u01/app/11.2.0/grid/OPatch/ocm
[oracle@tnc1 ocm]$ cd bin
[oracle@tnc1 bin]$ ls -ltr emocmrsp
ls: emocmrsp: No such file or directory
If You find There is no "emocmrsp" in GRID_HOME/OPatch/ocm/bin
DownLoad Recent OPatch to upgrade OPATCH version #
Creating ocm.rsp file as follows #
-rwxrwxr-x 1 oracle oinstall 52733900 May 8 01:02 p6880880_112000_Linux-x86-64.zip
drwxrwxr-x 2 oracle oinstall 4096 May 8 01:10 OPatch
[root@tnc2 grid]# pwd
/u01/app/11.2.0/grid
[oracle@tnc2 grid]$ unzip p6880880_112000_Linux-x86-64.zip
Archive: p6880880_112000_Linux-x86-64.zip
inflating: OPatch/opatchdiag
inflating: OPatch/opatch.pl
creating: OPatch/jlib/
inflating: OPatch/jlib/oracle.opatchcore.classpath.jar
inflating: OPatch/jlib/opatch.jar
inflating: OPatch/jlib/oracle.opatch.classpath.jar
inflating: OPatch/jlib/oracle.opatch.classpath.unix.jar
inflating: OPatch/jlib/oracle.opatch.classpath.windows.jar
inflating: OPatch/jlib/opatchsdk.jar
inflating: OPatch/jlib/oracle.opatchcore.classpath.unix.jar
inflating: OPatch/jlib/oracle.opatchcore.classpath.windows.jar
creating: OPatch/crs/
inflating: OPatch/crs/patch11203.pl
inflating: OPatch/crs/installPatch.excl
inflating: OPatch/crs/patch11202.pl
inflating: OPatch/crs/auto_patch.pl
creating: OPatch/crs/log/
inflating: OPatch/crs/patch112.pl
inflating: OPatch/crs/patchDB.pl
inflating: OPatch/crs/opatchauto
inflating: OPatch/crs/CRSProductDriver.jar
inflating: OPatch/crs/driver.jar
creating: OPatch/ocm/
creating: OPatch/ocm/doc/
extracting: OPatch/ocm/ocm.zip
inflating: OPatch/ocm/ocm_platforms.txt
creating: OPatch/ocm/lib/
inflating: OPatch/ocm/lib/emocmclnt.jar
inflating: OPatch/ocm/lib/xmlparserv2.jar
inflating: OPatch/ocm/lib/http_client.jar
inflating: OPatch/ocm/lib/emocmclnt-14.jar
inflating: OPatch/ocm/lib/regexp.jar
inflating: OPatch/ocm/lib/jnet.jar
inflating: OPatch/ocm/lib/jsse.jar
inflating: OPatch/ocm/lib/osdt_core3.jar
inflating: OPatch/ocm/lib/osdt_jce.jar
inflating: OPatch/ocm/lib/emocmcommon.jar
inflating: OPatch/ocm/lib/log4j-core.jar
inflating: OPatch/ocm/lib/jcert.jar
creating: OPatch/ocm/bin/
inflating: OPatch/ocm/bin/emocmrsp
inflating: OPatch/opatch
creating: OPatch/opatchauto-dir/
creating: OPatch/opatchauto-dir/opatchautocore/
inflating: OPatch/opatchauto-dir/opatchautocore/oplan
inflating: OPatch/opatchauto-dir/opatchautocore/opatchautobinary
creating: OPatch/opatchauto-dir/opatchautocore/jlib/
inflating: OPatch/opatchauto-dir/opatchautocore/jlib/ValidationRules.jar
inflating: OPatch/opatchauto-dir/opatchautocore/jlib/patchsdk.jar
inflating: OPatch/opatchauto-dir/opatchautocore/jlib/ProductDriver.jar
inflating: OPatch/opatchauto-dir/opatchautocore/jlib/osysmodel-utils.jar
inflating: OPatch/opatchauto-dir/opatchautocore/jlib/OsysModel.jar
creating: OPatch/opatchauto-dir/opatchautocore/jlib/apache-commons/
inflating: OPatch/opatchauto-dir/opatchautocore/jlib/apache-commons/commons-compress-1.4.jar
inflating: OPatch/opatchauto-dir/opatchautocore/jlib/apache-commons/commons-cli-1.0.jar
inflating: OPatch/opatchauto-dir/opatchautocore/jlib/oplan_core.jar
inflating: OPatch/opatchauto-dir/opatchautocore/jlib/oracle.oplan.classpath.jar
inflating: OPatch/opatchauto-dir/opatchautocore/jlib/bundle.jar
inflating: OPatch/opatchauto-dir/opatchautocore/jlib/oplan_sample.jar
inflating: OPatch/opatchauto-dir/opatchautocore/jlib/Validation.jar
creating: OPatch/opatchauto-dir/opatchautocore/jlib/jaxb/
inflating: OPatch/opatchauto-dir/opatchautocore/jlib/jaxb/activation.jar
inflating: OPatch/opatchauto-dir/opatchautocore/jlib/jaxb/jsr173_1.0_api.jar
inflating: OPatch/opatchauto-dir/opatchautocore/jlib/jaxb/jaxb-api.jar
inflating: OPatch/opatchauto-dir/opatchautocore/jlib/jaxb/jaxb-impl.jar
inflating: OPatch/opatchauto-dir/opatchautocore/jlib/automation.jar
inflating: OPatch/opatchauto-dir/opatchautocore/oplan.bat
inflating: OPatch/opatchauto-dir/opatchautocore/README.txt
inflating: OPatch/opatchauto-dir/opatchautocore/README.html
creating: OPatch/opatchauto-dir/opatchautodb/
creating: OPatch/opatchauto-dir/opatchautodb/jlib/
inflating: OPatch/opatchauto-dir/opatchautodb/jlib/oracle.oplan.db.classpath.jar
inflating: OPatch/opatchauto-dir/opatchautodb/jlib/oplan_db.jar
inflating: OPatch/emdpatch.pl
creating: OPatch/scripts/
inflating: OPatch/scripts/opatch_wls.bat
inflating: OPatch/scripts/opatch_jvm_discovery.bat
inflating: OPatch/scripts/opatch_wls
inflating: OPatch/scripts/opatch_jvm_discovery
inflating: OPatch/operr
inflating: OPatch/operr_readme.txt
inflating: OPatch/operr.bat
creating: OPatch/opatchprereqs/
inflating: OPatch/opatchprereqs/prerequisite.properties
creating: OPatch/opatchprereqs/oui/
inflating: OPatch/opatchprereqs/oui/knowledgesrc.xml
creating: OPatch/opatchprereqs/opatch/
inflating: OPatch/opatchprereqs/opatch/runtime_prereq.xml
inflating: OPatch/opatchprereqs/opatch/rulemap.xml
inflating: OPatch/opatchprereqs/opatch/opatch_prereq.xml
inflating: OPatch/opatch.bat
creating: OPatch/oplan/
creating: OPatch/oplan/jlib/
creating: OPatch/oplan/jlib/apache-commons/
inflating: OPatch/oplan/jlib/apache-commons/commons-cli-1.0.jar
creating: OPatch/oplan/jlib/jaxb/
inflating: OPatch/oplan/jlib/jaxb/jsr173_1.0_api.jar
inflating: OPatch/oplan/jlib/jaxb/jaxb-impl.jar
inflating: OPatch/oplan/jlib/jaxb/activation.jar
inflating: OPatch/oplan/jlib/jaxb/jaxb-api.jar
inflating: OPatch/oplan/jlib/Validation.jar
inflating: OPatch/oplan/jlib/oracle.oplan.classpath.jar
inflating: OPatch/oplan/jlib/OuiDriver.jar
inflating: OPatch/oplan/jlib/EMrepoDrivers.jar
inflating: OPatch/oplan/jlib/JMXDrivers.jar
inflating: OPatch/oplan/jlib/bundle.jar
inflating: OPatch/oplan/jlib/CRSProductDriver.jar
inflating: OPatch/oplan/jlib/oplan.jar
inflating: OPatch/oplan/jlib/ValidationRules.jar
inflating: OPatch/oplan/jlib/osysmodel-utils.jar
inflating: OPatch/oplan/jlib/automation.jar
inflating: OPatch/oplan/README.html
inflating: OPatch/oplan/README.txt
inflating: OPatch/oplan/oplan.bat
inflating: OPatch/oplan/oplan
inflating: OPatch/opatchdiag.bat
inflating: OPatch/README.txt
extracting: OPatch/version.txt
creating: OPatch/docs/
inflating: OPatch/docs/cversion.txt
inflating: OPatch/docs/Prereq_Users_Guide.txt
inflating: OPatch/docs/FAQ
inflating: OPatch/docs/Users_Guide.txt
[oracle@tnc2 grid]$
[oracle@tnc2 grid]$
[oracle@tnc2 grid]$ pwd
/u01/app/11.2.0/grid
[oracle@tnc2 grid]$ cd OPatch
[oracle@tnc2 OPatch]$ ls -ltr
total 156
-rwxr-x--- 1 oracle oinstall 31056 Aug 18 2015 opatch
-rw-r----- 1 oracle oinstall 27 Aug 18 2015 version.txt
drwxr-x--- 2 oracle oinstall 4096 Aug 18 2015 scripts
-rw-r----- 1 oracle oinstall 2915 Aug 18 2015 README.txt
drwxr-x--- 3 oracle oinstall 4096 Aug 18 2015 oplan
-rw-r----- 1 oracle oinstall 3177 Aug 18 2015 operr_readme.txt
-rwxr-x--- 1 oracle oinstall 4220 Aug 18 2015 operr.bat
-rwxr-x--- 1 oracle oinstall 3161 Aug 18 2015 operr
-rwxr-x--- 1 oracle oinstall 2576 Aug 18 2015 opatch.pl
-rwxr-x--- 1 oracle oinstall 9352 Aug 18 2015 opatchdiag.bat
-rwxr-x--- 1 oracle oinstall 10056 Aug 18 2015 opatchdiag
-rwxr-x--- 1 oracle oinstall 15184 Aug 18 2015 opatch.bat
drwxr-x--- 2 oracle oinstall 4096 Aug 18 2015 jlib
-rwxr-x--- 1 oracle oinstall 23695 Aug 18 2015 emdpatch.pl
drwxr-x--- 4 oracle oinstall 4096 Aug 18 2015 opatchprereqs
drwxr-x--- 3 oracle oinstall 4096 Aug 18 2015 crs
drwxr-x--- 5 oracle oinstall 4096 Aug 18 2015 ocm
drwxr-x--- 4 oracle oinstall 4096 Aug 18 2015 opatchauto-dir
drwxr-x--- 2 oracle oinstall 4096 Aug 18 2015 docs
[oracle@tnc2 OPatch]$ cd ocm
[oracle@tnc2 ocm]$ ls -ltr
total 41628
-rw-r----- 1 oracle oinstall 638 Aug 18 2015 ocm_platforms.txt
drwxr-x--- 2 oracle oinstall 4096 Aug 18 2015 doc
-rwxr-x--- 1 oracle oinstall 42560960 Aug 18 2015 ocm.zip
drwxr-x--- 2 oracle oinstall 4096 Aug 18 2015 lib
drwxr-x--- 2 oracle oinstall 4096 Aug 18 2015 bin
[oracle@tnc2 ocm]$ cd bin
[oracle@tnc2 bin]$ ls -ltr
total 12
-rwxr-x--- 1 oracle oinstall 9063 Nov 27 2009 emocmrsp
[oracle@tnc2 bin]$
[oracle@tnc2 bin]$ pwd
/u01/app/11.2.0/grid/OPatch/ocm/bin
/u01/app/11.2.0/grid/OPatch/ocm/bin
Creating ocm.rsp file as follows #
[oracle@tnc1 bin]$ ./emocmrsp -no_banner -output /u01/stage/file.rsp
Provide your email address to be informed of security issues, install and
initiate Oracle Configuration Manager. Easier for you if you use your My
Oracle Support Email address/User Name.
Visit http://www.oracle.com/support/policies.html for details.
Email address/User Name:
You have not provided an email address for notification of security issues.
Do you wish to remain uninformed of security issues ([Y]es, [N]o) [N]: Y
The OCM configuration response file (/u01/app/oracle/file.rsp) was successfully created.
--Nikhil Tatineni--
--12C RAC --
Tuesday, May 3, 2016
read position and write position of GoldenGate pump process
How do we check current read position and write position of GoldenGate pump process
GGSCI (hidb401) 60> send PTNMLDB, status
Sending STATUS request to EXTRACT PTNMLDB ...
EXTRACT PTNMLDB (PID 39845948)
Current status: Recovery complete: At EOF
Current read position:
Sequence #: 6
RBA: 1113
Timestamp: 2016-04-04 14:14:55.000000
Extract Trail: /oraarch/oval013t/fs01/ggs01/trails/pg
Current write position:
Sequence #: 0
RBA: 0
Timestamp: 2016-04-04 14:18:43.860423
Extract Trail: /opt/app/t1ova1d/ggs/product/11.1.1/dirdat/pg
---Nikhil Tatineni--
---GoldenGate --
GGSCI (hidb401) 60> send PTNMLDB, status
Sending STATUS request to EXTRACT PTNMLDB ...
EXTRACT PTNMLDB (PID 39845948)
Current status: Recovery complete: At EOF
Current read position:
Sequence #: 6
RBA: 1113
Timestamp: 2016-04-04 14:14:55.000000
Extract Trail: /oraarch/oval013t/fs01/ggs01/trails/pg
Current write position:
Sequence #: 0
RBA: 0
Timestamp: 2016-04-04 14:18:43.860423
Extract Trail: /opt/app/t1ova1d/ggs/product/11.1.1/dirdat/pg
---Nikhil Tatineni--
---GoldenGate --
Virtual IP address is not configured for cluster node
scenario:# Creating database using dbca in RAC cluster & failed with following error.
Virtual IP address is not configured for cluster node " run /u01/app/11.2.0/grid/bin/srvctl as root to configure before performing the "create a database" operation
-------#
We found, there is NO VIP on second node of RAC cluster #
Configured VIP on second node as follows
On Node 1#
After creating VIP, Installed database successfully
---Nikhil Tatineni--
---12c RAC---
Virtual IP address is not configured for cluster node " run /u01/app/11.2.0/grid/bin/srvctl as root to configure before performing the "create a database" operation
-------#
We found, there is NO VIP on second node of RAC cluster #
Configured VIP on second node as follows
On Node 1#
[oracle@tnc1 ~]$ srvctl config nodeapps -a
Network exists: 1/192.168.1.0/255.255.255.0/eth0, type static
VIP exists: /tnc1-vip/192.168.1.11/192.168.1.0/255.255.255.0/eth0, hosting node tnc1
On Node 2 # As a root User #
[root@tnc2 grid]# srvctl add vip -n tnc2 -k 1 -A 192.168.1.21/255.255.255.0/eth0
[root@tnc2 grid]# srvctl start nodeapps -n tnc2
[root@tnc2 grid]# srvctl config nodeapps -a
Network exists: 1/192.168.1.0/255.255.255.0/eth0, type static
VIP exists: /tnc1-vip/192.168.1.11/192.168.1.0/255.255.255.0/eth0, hosting node tnc1
VIP exists: /192.168.1.21/192.168.1.21/192.168.1.0/255.255.255.0/eth0, hosting node tnc2After creating VIP, Installed database successfully
---Nikhil Tatineni--
---12c RAC---
Sunday, May 1, 2016
CRS-4402, CRS-2800, CRS-4000 : root.sh failed on second Node of RAC cluster
### Scenario:
/u01/app/11.2.0/grid/root.sh is failed on the second Node of the RAC cluster ##
[root@tnc2 ~]# /u01/app/11.2.0/grid/root.sh
Performing root user operation for Oracle 11g
The following environment variables are set as:
ORACLE_OWNER= oracle
ORACLE_HOME= /u01/app/11.2.0/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
Copying dbhome to /usr/local/bin ...
Copying oraenv to /usr/local/bin ...
Copying coraenv to /usr/local/bin ...
Creating /etc/oratab file...
Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/11.2.0/grid/crs/install/crsconfig_params
Creating trace directory
User ignored Prerequisites during installation
Installing Trace File Analyzer
OLR initialization - successful
Adding Clusterware entries to inittab
CRS-4402: The CSS daemon was started in exclusive mode but found an active CSS daemon on node tnc1, number 1, and is terminating
An active cluster was found during exclusive startup, restarting to join the cluster
Start of resource "ora.crsd" failed
CRS-2800: Cannot start resource 'ora.asm' as it is already in the INTERMEDIATE state on server 'tnc2'
CRS-4000: Command Start failed, or completed with errors.
Failed to start Oracle Grid Infrastructure stack
Failed to start Cluster Ready Services at /u01/app/11.2.0/grid/crs/install/crsconfig_lib.pm line 1353.
/u01/app/11.2.0/grid/perl/bin/perl -I/u01/app/11.2.0/grid/perl/lib -I/u01/app/11.2.0/grid/crs/install /u01/app/11.2.0/grid/crs/install/rootcrs.pl execution failed
Node Alert log file shows as follows
[/u01/app/11.2.0/grid/bin/oraagent.bin(28566)]CRS-5019:All OCR locations are on ASM disk groups [GRID], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/tnc2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
2016-04-30 16:28:08.453:
[/u01/app/11.2.0/grid/bin/oraagent.bin(28566)]CRS-5019:All OCR locations are on ASM disk groups [GRID], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/tnc2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
2016-04-30 16:28:38.628:
[/u01/app/11.2.0/grid/bin/oraagent.bin(28566)]CRS-5019:All OCR locations are on ASM disk groups [GRID], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/tnc2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
2016-04-30 16:29:08.727:
[/u01/app/11.2.0/grid/bin/oraagent.bin(28566)]CRS-5019:All OCR locations are on ASM disk groups [GRID], and none of these disk groups are mounted. Details are at "(:CLSN00100:)" in "/u01/app/11.2.0/grid/log/tnc2/agent/ohasd/oraagent_oracle/oraagent_oracle.log".
2016-04-30 16:29:38.867:
error in ohasd logs as follows "/u01/app/11.2.0/grid/log/tnc2/agent/ohasd/oraagent_oracle/oraagent_oracle.log"
2016-04-30 16:53:26.893: [ora.asm][1075857728]{0:0:2} [check] checkCrsStat 2 CLSCRS_STAT ret: 184
2016-04-30 16:53:26.893: [ora.asm][1075857728]{0:0:2} [check] clsnUtils::error Exception type=2 string=
2016-04-30 16:53:26.893: [ora.asm][1075857728]{0:0:2} [check] AsmAgent::checkCbk: Exception UserErrorException
2016-04-30 16:53:26.893: [ora.asm][1075857728]{0:0:2} [check]
2016-04-30 16:53:26.893: [ora.asm][1075857728]{0:0:2} [check] InstAgent::check 1 prev clsagfw_res_status 4 current clsagfw_res_status 4
2016-04-30 16:53:27.896: [ora.asm][1077958976]{0:0:2} [check] AsmAgent::check ocrCheck 1 m_OcrOnline 0 m_OcrTimer 161
2016-04-30 16:53:27.896: [ora.asm][1077958976]{0:0:2} [check] CrsCmd::ClscrsCmdData::stat entity 5 statflag 32 useFilter 1
2016-04-30 16:53:27.897: [ COMMCRS][1133730112]clsc_connect: (0x1f159cb0) no listener at (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_UI_SOCKET))
2016-04-30 16:53:27.897: [ora.asm][1077958976]{0:0:2} [check] checkCrsStat 2 CLSCRS_STAT ret: 184
2016-04-30 16:53:27.897: [ora.asm][1077958976]{0:0:2} [check] clsnUtils::error Exception type=2 string=
2016-04-30 16:53:27.897: [ora.asm][1077958976]{0:0:2} [check] AsmAgent::checkCbk: Exception UserErrorException
2016-04-30 16:53:27.897: [ora.asm][1077958976]{0:0:2} [check]
2016-04-30 16:53:27.897: [ora.asm][1077958976]{0:0:2} [check] InstAgent::check 1 prev clsagfw_res_status 4 current clsagfw_res_status 4
2016-04-30 16:53:28.900: [ora.asm][1077958976]{0:0:2} [check] AsmAgent::check ocrCheck 1 m_OcrOnline 0 m_OcrTimer 162
2016-04-30 16:53:28.900: [ora.asm][1077958976]{0:0:2} [check] CrsCmd::ClscrsCmdData::stat entity 5 statflag 32 useFilter 1
2016-04-30 16:53:28.901: [ COMMCRS][1133730112]clsc_connect: (0x1f159cb0) no listener at (ADDRESS=(PROTOCOL=IPC)(KEY=CRSD_UI_SOCKET))
2016-04-30 16:53:28.901: [ora.asm][1077958976]{0:0:2} [check] checkCrsStat 2 CLSCRS_STAT ret: 184
2016-04-30 16:53:28.901: [ora.asm][1077958976]{0:0:2} [check] clsnUtils::error Exception type=2 string=2016-04-30 16:53:2
logged into ASM instance on second Node and query the disk status, it showed diskgroups is Dismounted. It clearly shows that DISKGROUP is not mounted on second node of the Cluster
######
SQL> select GROUP_NUMBER,NAME,OFFLINE_DISKS, state from v$asm_diskgroup;
GROUP_NUMBER NAME OFFLINE_DISKS STATE
------------ ------------------------------ ------------- -------------
0 GRID 0 DISMOUNTED
[root@tnc2 bin]# ./crsctl query css votedisk
## STATE File Universal Id File Name Disk group
-- ----- ----------------- --------- ---------
1. ONLINE 357fa62799a74fcdbf518b0f0d6d9a29 (/dev/raw/raw1) [GRID]
2. OFFLINE 216d1cb038d14fc1bf8ccc05b0af6793 () []
3. ONLINE 061664871d014ff7bfac218182629a7d (/dev/raw/raw3) [GRID]
Located 3 voting disk(s).
Continued to install & Successfully Installed the GRID cluster : CVU is failed at end of grid installation, I found that there is no permission on disks on second node of the cluster
Later worked with SA by setting Udev Rules on the disks attached to Server
stopped crs force on node of the cluster and After restarted the crs on both nodes of the cluster ##
[root@tnc2 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'tnc2'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'tnc2'
CRS-2673: Attempting to stop 'ora.crf' on 'tnc2'
CRS-2673: Attempting to stop 'ora.ctssd' on 'tnc2'
CRS-2673: Attempting to stop 'ora.evmd' on 'tnc2'
CRS-2673: Attempting to stop 'ora.asm' on 'tnc2'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'tnc2'
CRS-2677: Stop of 'ora.evmd' on 'tnc2' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'tnc2' succeeded
CRS-2677: Stop of 'ora.crf' on 'tnc2' succeeded
CRS-2677: Stop of 'ora.asm' on 'tnc2' succeeded
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'tnc2'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'tnc2' succeeded
CRS-2677: Stop of 'ora.drivers.acfs' on 'tnc2' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'tnc2' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'tnc2'
CRS-2677: Stop of 'ora.cssd' on 'tnc2' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'tnc2'
CRS-2677: Stop of 'ora.gipcd' on 'tnc2' succeeded
CRS-2673: Attempting to stop 'ora.gpnpd' on 'tnc2'
CRS-2677: Stop of 'ora.gpnpd' on 'tnc2' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'tnc2' has completed
CRS-4133: Oracle High Availability Services has been stopped.
# After #
# crsctl start crs
# After #
# crsctl start crs
## Issue is resolved ##
--Nikhil Tatineni--
--RAC --
Subscribe to:
Posts (Atom)
-
How to kill blocking session on object ? STEP1: Initially we need to find out blocking session on object SELECT o.owner, o...
-
When goldengate process is unable to process transaction it generates a discard and makes process abended. When ever a discard is gene...
-
Hybrid Columnar Compression enables us to implement different levels of data compression and provides cost-savings at storage le...
