Saturday, February 20, 2016

Applying Database Rolling Patch on 12c RAC Cluster

Applying Rolling Patch "21948354" on RAC ORACLE_HOME's

a) On Node1: Stop CRS on Node 1, which failover all services running from Node 1  to Node 2. Make sure management database is Failed Over to Node 2 
#crsctl stop crs 

b) Update OPatch Version if it is less than 12.1.0.1.7 by using patch 6880880 by selecting the 12.1.0.1.0 release


c) Set appropriate environment variables 
export ORACLE_BASE=/u01/app/oracle
export DB_HOME=$ORACLE_BASE/product/12.1.0.2/db_1
export PATH=$ORACLE_HOME/OPatch:$PATH:.

d) Take Backup Of ORACLE_HOME 

$ cp -r db_1 db_1_backup

e) Check for previous patches conflicts with New Rolling  PSU Patch "21948354"

$ cd 21948354
$ opatch prereq CheckConflictAgainstOHWithDetail -ph ./

f) When ever above script completed without any conflicts go ahead and apply rolling patch 
$ opatch apply -local 

g) When Patching completed on ORACLE_HOME, go ahead and bringup CRS on NODE1 and make sure all services are running on NODE 1

Follow the same steps and Apply patch on Node 2
After applying Patch on both ORACLE_HOMES / Nodes of a cluster, make sure all resources are up and running 

[oracle@tnc1 ~]$ crsctl status resource -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.FRA.dg
               ONLINE  ONLINE       tnc1                     STABLE
               ONLINE  ONLINE       tnc2                     STABLE
ora.GRID.dg
               ONLINE  ONLINE       tnc1                     STABLE
               ONLINE  ONLINE       tnc2                     STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       tnc1                     STABLE
               ONLINE  ONLINE       tnc2                     STABLE
ora.SDXH.dg
               ONLINE  ONLINE       tnc1                     STABLE
               ONLINE  ONLINE       tnc2                     STABLE
ora.asm
               ONLINE  ONLINE       tnc1                     Started,STABLE
               ONLINE  ONLINE       tnc2                     Started,STABLE
ora.net1.network
               ONLINE  ONLINE       tnc1                     STABLE
               ONLINE  ONLINE       tnc2                     STABLE
ora.ons
               ONLINE  ONLINE       tnc1                     STABLE
               ONLINE  ONLINE       tnc2                     STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       tnc2                     STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       tnc1                     169.254.248.60,STABLE                                
ora.cvu
      1        ONLINE  ONLINE       tnc2                     STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       tnc1                     Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       tnc1                     STABLE
ora.p1tncd.db
      1        ONLINE  ONLINE       tnc2                     Open,STABLE
      2        ONLINE  ONLINE       tnc1                     Open,STABLE
ora.p1tncd.srvnsp.svc
      1        ONLINE  ONLINE       tnc2                     STABLE
      2        ONLINE  ONLINE       tnc1                     STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       tnc2                     STABLE
ora.tnc1.vip
      1        ONLINE  ONLINE       tnc1                     STABLE
ora.tnc2.vip
      1        ONLINE  ONLINE       tnc2                     STABLE
--------------------------------------------------------------------------------

When all resources are online upgrade the database using 12c New OPTION datapatch as follows 

[oracle@tnc2 OPatch]$ ./datapatch -verbose
SQL Patching tool version 12.1.0.2.0 on Sat Feb 20 07:15:30 2016
Copyright (c) 2015, Oracle.  All rights reserved.
Log file for this invocation: /u01/app/oracle/cfgtoollogs/sqlpatch/sqlpatch_21603_2016_02_20_07_15_30/sqlpatch_invocation.log
Connecting to database...OK
Note:  Datapatch will only apply or rollback SQL fixes for PDBs
       that are in an open state, no patches will be applied to closed PDBs.
       Please refer to Note: Datapatch: Database 12c Post Patch SQL Automation
       (Doc ID 1585822.1)
Bootstrapping registry and package to current versions...done
Determining current state...done
Current state of SQL patches:
Bundle series PSU:
  ID 160119 in the binary registry and not installed in any PDB
Adding patches to installation queue and performing prereq checks...
Installation queue:
  For the following PDBs: CDB$ROOT PDB$SEED CBN
    Nothing to roll back
    The following patches will be applied:
      21948354 (Database Patch Set Update : 12.1.0.2.160119 (21948354))
Installing patches...
Patch installation complete.  Total patches installed: 3
Validating logfiles...
Patch 21948354 apply (pdb CDB$ROOT): SUCCESS
  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/21948354/19553095/21948354_apply_P1TNCD_CDBROOT_2016Feb20_07_18_04.log (no errors)
Patch 21948354 apply (pdb PDB$SEED): SUCCESS
  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/21948354/19553095/21948354_apply_P1TNCD_PDBSEED_2016Feb20_07_18_24.log (no errors)
Patch 21948354 apply (pdb CBN): SUCCESS
  logfile: /u01/app/oracle/cfgtoollogs/sqlpatch/21948354/19553095/21948354_apply_P1TNCD_CBN_2016Feb20_07_18_25.log (no errors)

Database is Upgraded Successfully :) 
Check what are the patches Installed on DATABASE_HOME 


[oracle@tnc1 OPatch]$ ./opatch lsinventory
Oracle Interim Patch Installer version 12.1.0.1.10
Copyright (c) 2016, Oracle Corporation.  All rights reserved.
Oracle Home       : /u01/app/oracle/product/12.1.0.2/db_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/12.1.0.2/db_1/oraInst.loc
OPatch version    : 12.1.0.1.10
OUI version       : 12.1.0.2.0
Log file location : /u01/app/oracle/product/12.1.0.2/db_1/cfgtoollogs/opatch/opatch2016-02-20_19-21-31PM_1.log
Lsinventory Output file location : /u01/app/oracle/product/12.1.0.2/db_1/cfgtoollogs/opatch/lsinv/lsinventory2016-02-20_19-21-31PM.txt
--------------------------------------------------------------------------------
Local Machine Information::
Hostname: tnc1.localdomain
ARU platform id: 226
ARU platform description:: Linux x86-64
Installed Top-level Products (1): 
Oracle Database 12c                                                  12.1.0.2.0
There are 1 products installed in this Oracle Home.
Interim patches (1) :
Patch  21948354     : applied on Sat Feb 20 05:12:02 EST 2016
Unique Patch ID:  19553095
Patch description:  "Database Patch Set Update : 12.1.0.2.160119 (21948354)"
   Created on 20 Dec 2015, 23:39:33 hrs PST8PDT
Sub-patch  21359755; "Database Patch Set Update : 12.1.0.2.5 (21359755)"
Sub-patch  20831110; "Database Patch Set Update : 12.1.0.2.4 (20831110)"
Sub-patch  20299023; "Database Patch Set Update : 12.1.0.2.3 (20299023)"
Sub-patch  19769480; "Database Patch Set Update : 12.1.0.2.2 (19769480)"
   Bugs fixed:
     19189525, 19075256, 19141838, 19865345, 19791273, 19280225, 18845653
     20951038, 19243521, 19248799, 21756699, 18988834, 19238590, 21281532
     20245930, 18921743, 18799063, 19134173, 19571367, 20476175, 20925795
     19018206, 20509482, 20387265, 20588502, 19149990, 18849537, 18886413
     17551063, 19183343, 19703301, 19001390, 18202441, 19189317, 19644859
     19358317, 19390567, 19279273, 19706965, 19068970, 19619732, 20348653
     18607546, 18940497, 19670108, 19649152, 18948177, 19315691, 19676905
     18964978, 19035573, 20165574, 19176326, 20413820, 20558005, 19176223
     19532017, 20134339, 19074147, 18411216, 20361671, 20425790, 18966843
     20294666, 19307662, 19371175, 19195895, 19154375, 19468991, 19174521
     19520602, 19382851, 21875360, 19326908, 19658708, 20093776, 20618595
     21787056, 17835294, 19791377, 19068610, 20048359, 20746251, 19143550
     19185876, 19627012, 20281121, 19577410, 22092979, 19001359, 19518079
     18610915, 19490948, 18674024, 18306996, 19309466, 19081128, 19915271
     20122715, 21188532, 20284155, 18791688, 20890311, 21442094, 18973548
     19303936, 19597439, 20235511, 18964939, 19430401, 19044962, 19409212
     19879746, 20657441, 19684504, 19024808, 18799993, 20877664, 19028800
     19065556, 19723336, 19077215, 19604659, 21421886, 19524384, 19048007
     18288842, 19689979, 20446883, 18952989, 16870214, 19928926, 21526048
     19180770, 19197175, 19902195, 20318889, 19730508, 19012119, 19067244
     20074391, 19512341, 19841800, 14643995, 20331945, 19587324, 19547370
     19065677, 19637186, 21225209, 20397490, 18967382, 19174430, 18674047
     19054077, 19536415, 19708632, 19289642, 20869721, 19335438, 17365043
     18856999, 19869255, 20471920, 19468347, 21620471, 16359751, 18990693
     17890099, 19439759, 19769480, 19272708, 19978542, 20101006, 21300341
     20402832, 19329654, 19873610, 21668627, 21517440, 19304354, 19052488
     20794034, 19291380, 18681056, 19896336, 19076343, 19561643, 18618122
     20440930, 18456643, 19699191, 18909599, 19487147, 18250893, 19016730
     18743542, 20347562, 16619249, 18354830, 19687159, 19174942, 20424899
     19989009, 20688221, 20441797, 19157754, 19032777, 19058490, 19399918
     18885870, 19434529, 19018447, 18417036, 20919320, 19022470, 19284031
     20474192, 20173897, 22062026, 19385656, 19501299, 17274537, 20899461
     19440586, 16887946, 19606174, 18436647, 17655240, 19023822, 19178851
     19124589, 19805359, 19597583, 19155797, 19393542, 19050649
Rac system comprising of multiple nodes
  Local node = tnc1
  Remote node = tnc2
--------------------------------------------------------------------------------
OPatch succeeded.

--Nikhil Tatineni--
--12c RAC Cluster---

Querys to monitor RAC

following few  Query's will help to find out culprits-  Query to check long running transaction from last 8 hours  Col Sid Fo...