Sunday, February 14, 2016

12c-RAC-CRS-4535

[root@tnc1 ~]# crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online

Checked CRS log on the Node 1 and found following info --- 


[oracle@tnc1 trace]$ pwd
/u01/app/oracle/diag/crs/tnc1/crs/trace

'''''''''''''''''''''''''''''''''''''''''''''''''''''''''''''
[oracle@tnc1 trace]$ ls -ltr alert.log 
-rw-rw---- 1 root oinstall 90611 Feb 13 23:18 alert.log

'''''''''''''''''''''''''''''''''''''''''''''''''''''


2016-02-13 19:45:50.055 [CLSECHO(10483)]CRS-10001: 13-Feb-16 19:45 AFD-9204: false
2016-02-13 19:59:42.749 [ORAAGENT(5595)]CRS-5011: Check of resource "p1tncd" failed: details at "(:CLSN00007:)" in "/u01/app/oracle/diag/crs/tnc1/crs/trace/crsd_oraagent_oracle.trc"
2016-02-13 20:01:36.304 [ORAROOTAGENT(5604)]CRS-5017: The resource action "ora.net1.network start" encountered the following error: 
2016-02-13 20:01:36.304+CRS-5008: Invalid attribute value: eth0 for the network interface
. For details refer to "(:CLSN00107:)" in "/u01/app/oracle/diag/crs/tnc1/crs/trace/crsd_orarootagent_root.trc".
2016-02-13 20:01:36.340 [CRSD(5467)]CRS-2878: Failed to restart resource 'ora.net1.network'
2016-02-13 20:01:36.358 [CRSD(5467)]CRS-2769: Unable to failover resource 'ora.net1.network'.
2016-02-13 20:01:37.056 [CRSD(5467)]CRS-2771: Maximum restart attempts reached for resource 'ora.tnc1.vip'; will not restart.
2016-02-13 20:01:37.218 [CRSD(5467)]CRS-2771: Maximum restart attempts reached for resource 'ora.scan1.vip'; will not restart.
2016-02-13 20:01:42.686 [ORAAGENT(5595)]CRS-5016: Process "/u01/app/12.1.0.2/grid/bin/lsnrctl" spawned by agent "ORAAGENT" for action "check" failed: details at "(:CLSN00010:)" in "/u01/app/oracle/diag/crs/tnc1/crs/trace/crsd_oraagent_oracle.trc"
2016-02-13 20:25:49.009 [OCSSD(4377)]CRS-1612: Network communication with node tnc2 (2) missing for 50% of timeout interval.  Removal of this node from cluster in 14.230 seconds
2016-02-13 20:25:51.325 [ORAAGENT(5595)]CRS-5011: Check of resource "p1tncd" failed: details at "(:CLSN00007:)" in "/u01/app/oracle/diag/crs/tnc1/crs/trace/crsd_oraagent_oracle.trc"
2016-02-13 20:25:56.070 [OCSSD(4377)]CRS-1611: Network communication with node tnc2 (2) missing for 75% of timeout interval.  Removal of this node from cluster in 7.170 seconds
2016-02-13 20:25:54.158 [ORAAGENT(3949)]CRS-5011: Check of resource "ora.asm" failed: details at "(:CLSN00006:)" in "/u01/app/oracle/diag/crs/tnc1/crs/trace/ohasd_oraagent_oracle.trc"
2016-02-13 20:26:00.852 [ORAAGENT(5595)]CRS-5011: Check of resource "_mgmtdb" failed: details at "(:CLSN00007:)" in "/u01/app/oracle/diag/crs/tnc1/crs/trace/crsd_oraagent_oracle.trc"
2016-02-13 20:26:01.112 [OCSSD(4377)]CRS-1610: Network communication with node tnc2 (2) missing for 90% of timeout interval.  Removal of this node from cluster in 2.120 seconds
2016-02-13 20:26:03.259 [OCSSD(4377)]CRS-1607: Node tnc2 is being evicted in cluster incarnation 350858546; details at (:CSSNM00007:) in /u01/app/oracle/diag/crs/tnc1/crs/trace/ocssd.trc.
Sat Feb 13 20:26:09 2016

Fatal NI connect error 12514, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=169.254.248.60)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=tnc_cluster)(CID=(PROGRAM=ologgerd)(HOST=tnc1.localdomain)(USER=root))))

  VERSION INFORMATION:
TNS for Linux: Version 12.1.0.2.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.1.0.2.0 - Production
  Time: 13-FEB-2016 20:26:09
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12564
    
TNS-12564: TNS:connection refused
    ns secondary err code: 0
    nt main err code: 0
    nt secondary err code: 0
    nt OS err code: 0

Fatal NI connect error 12514, connecting to:
 (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=10.10.10.180)(PORT=1521))(CONNECT_DATA=(SERVICE_NAME=tnc_cluster)(CID=(PROGRAM=ologgerd)(HOST=tnc1.localdomain)(USER=root))))

  VERSION INFORMATION:
TNS for Linux: Version 12.1.0.2.0 - Production
TCP/IP NT Protocol Adapter for Linux: Version 12.1.0.2.0 - Production
  Time: 13-FEB-2016 20:26:09
  Tracing not turned on.
  Tns error struct:
    ns main err code: 12564
    
TNS-12564: TNS:connection refused
    ns secondary err code: 0
    nt main err code: 0
    nt secondary err code: 0
    nt OS err code: 0
2016-02-13 20:26:10.976 [ORAAGENT(5595)]CRS-5011: Check of resource "ora.asm" failed: details at "(:CLSN00006:)" in "/u01/app/oracle/diag/crs/tnc1/crs/trace/crsd_oraagent_oracle.trc"
2016-02-13 20:26:11.262 [ORAROOTAGENT(4906)]CRS-5818: Aborted command 'check' for resource 'ora.drivers.acfs'. Details at (:CRSAGF00113:) {0:9:3} in /u01/app/oracle/diag/crs/tnc1/crs/trace/ohasd_orarootagent_root.trc.
2016-02-13 20:26:11.902 [ORAROOTAGENT(4906)]CRS-5014: Agent "ORAROOTAGENT" timed out starting process "/u01/app/12.1.0.2/grid/bin/acfsload" for action "check": details at "(:CLSN00009:)" in "/u01/app/oracle/diag/crs/tnc1/crs/trace/ohasd_orarootagent_root.trc"
2016-02-13 20:26:14.393 [ORAAGENT(3949)]CRS-5011: Check of resource "ora.asm" failed: details at "(:CLSN00006:)" in "/u01/app/oracle/diag/crs/tnc1/crs/trace/ohasd_oraagent_oracle.trc"
2016-02-13 20:26:14.690 [CRSD(5467)]CRS-1024: The Cluster Ready Service on this node terminated because the ASM instance was not active on this node. Details at (:PROCR00009:) in /u01/app/oracle/diag/crs/tnc1/crs/trace/crsd.trc.


Rootcause: I found, when it is "ORAROOTAGENT" is trying to bringup network sources and it is failed on server & eventually Stopped CRS on the Node forcefully 

&  also I found one more error in the crs alert logfile 
TNS-12564: TNS:connection refused
    ns secondary err code: 0
    nt main err code: 0
    nt secondary err code: 0
    nt OS err code: 0

Resolution: As a root, stopped CRS Stack on Node 1 forcefully  
checked bash_profile and network configuration files to avoid "TNS-12564" errors 


[root@tnc1 ~]# crsctl stop crs -f
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'tnc1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'tnc1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'tnc1'
CRS-2677: Stop of 'ora.gpnpd' on 'tnc1' succeeded
CRS-2677: Stop of 'ora.mdnsd' on 'tnc1' succeeded

[root@tnc1 ~]# crsctl start crs

[oracle@tnc1 ~]$ crsctl status resource -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.FRA.dg
               ONLINE  ONLINE       tnc1                     STABLE
               ONLINE  ONLINE       tnc2                     STABLE
ora.GRID.dg
               ONLINE  ONLINE       tnc1                     STABLE
               ONLINE  ONLINE       tnc2                     STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       tnc1                     STABLE
               ONLINE  ONLINE       tnc2                     STABLE
ora.SDXH.dg
               ONLINE  ONLINE       tnc1                     STABLE
               ONLINE  ONLINE       tnc2                     STABLE
ora.asm
               ONLINE  ONLINE       tnc1                     Started,STABLE
               ONLINE  ONLINE       tnc2                     Started,STABLE
ora.net1.network
               ONLINE  ONLINE       tnc1                     STABLE
               ONLINE  ONLINE       tnc2                     STABLE
ora.ons
               ONLINE  ONLINE       tnc1                     STABLE
               ONLINE  ONLINE       tnc2                     STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       tnc1                     STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       tnc2                     169.254.89.38 10.10.
                                                                          10.190,STABLE
ora.cvu
      1        ONLINE  ONLINE       tnc2                     STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       tnc2                     Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       tnc2                     STABLE
ora.p1tncd.db
      1        ONLINE  ONLINE       tnc1                     Open,STABLE
      2        ONLINE  ONLINE       tnc2                     Open,STABLE
ora.p1tncd.srvnsp.svc
      1        ONLINE  ONLINE       tnc1                     STABLE
      2        OFFLINE OFFLINE                                 STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       tnc1                     STABLE
ora.tnc1.vip
      1        ONLINE  ONLINE       tnc1                     STABLE
ora.tnc2.vip
      1        ONLINE  ONLINE       tnc2                     STABLE
--------------------------------------------------------------------------------

--NIKHIL TATINENI--
--12C Oracle RAC-- 

Saturday, February 13, 2016

RAC :Environment

Environment Description: 
##  Multi-tenant database "p1tncd" version 12.1.0.2 running on 12.1.0.2 GRID HOME

Number of Nodes in a Cluster
[oracle@tnc1 ~]$ olsnodes -n
tnc1 1
tnc2 2

To check Oracle Clusterware Version 
[oracle@tnc2 dbs]$ crsctl query crs softwareversion
Oracle Clusterware version on node [tnc2] is [12.1.0.2.0]

To check name of the cluster
[oracle@tnc1 bin]$ ./cemutlo -n
tnc-cluster

To check network Interface on the node
[oracle@tnc2 bin]$ ./oifcfg iflist
eth0  192.168.1.0
eth1  10.10.10.0
eth1  169.254.0.0
virbr0  192.168.122.0
eth2  10.0.0.0

To check Status of scan_listener on node of the cluster
[oracle@tnc2 ~]$ srvctl status scan_listener
SCAN Listener LISTENER_SCAN1 is enabled
SCAN listener LISTENER_SCAN1 is running on node tnc1 

Databases Running on RAC cluster 
[oracle@tnc1 ~]$ ps -ef | grep pmon
oracle    5066     1  0 09:01 ?        00:00:00 asm_pmon_+ASM1
oracle    5755     1  0 09:01 ?        00:00:00 mdb_pmon_-MGMTDB
oracle   13278     1  0 10:38 ?        00:00:00 ora_pmon_p1tncd1
oracle   23533 25200  0 15:32 pts/1    00:00:00 grep pmon 

"p1tncd" is CDB ROOT database running on 2 nodes on the cluster 

End point listener running on Node of the cluster
[oracle@tnc1 bin]$ lsnrctl status
LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 13-FEB-2016 15:47:42
Copyright (c) 1991, 2014, Oracle.  All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER)))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 12.1.0.2.0 - Production
Start Date                13-FEB-2016 13:24:13
Uptime                    0 days 2 hr. 23 min. 29 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/12.1.0.2/grid/network/admin/listener.ora
Listener Log File         /u01/app/oracle/diag/tnslsnr/tnc1/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=ipc)(KEY=LISTENER)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=127.0.0.1)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=192.168.1.11)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=tnc1.localdomain)(PORT=5500))(Security=(my_wallet_directory=/u01/app/oracle/product/12.1.0.2/db_1/admin/p1tncd/xdb_wallet))(Presentation=HTTP)(Session=RAW))
Services Summary...
Service "+ASM" has 1 instance(s).
  Instance "+ASM1", status READY, has 1 handler(s) for this service...
Service "-MGMTDBXDB" has 1 instance(s).
  Instance "-MGMTDB", status READY, has 1 handler(s) for this service...
Service "_mgmtdb" has 1 instance(s).
  Instance "-MGMTDB", status READY, has 1 handler(s) for this service...
Service "cbn" has 1 instance(s).
  Instance "p1tncd1", status READY, has 1 handler(s) for this service...
Service "p1tncd" has 1 instance(s).
  Instance "p1tncd1", status READY, has 1 handler(s) for this service...
Service "p1tncdXDB" has 1 instance(s).
  Instance "p1tncd1", status READY, has 1 handler(s) for this service...
Service "srvnsp" has 1 instance(s).
  Instance "p1tncd1", status READY, has 1 handler(s) for this service...
Service "tnc_cluster" has 1 instance(s).
  Instance "-MGMTDB", status READY, has 1 handler(s) for this service...
The command completed successfully

Performing Health Checks on the cluster

[oracle@tnc1 bin]$ crsctl check cluster -all
**************************************************************
tnc1:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************
tnc2:
CRS-4537: Cluster Ready Services is online
CRS-4529: Cluster Synchronization Services is online
CRS-4533: Event Manager is online
**************************************************************

To check status of resources running on cluster

[oracle@tnc2 ~]$ crsctl status resource -t
--------------------------------------------------------------------------------
Name           Target  State        Server                   State details       
--------------------------------------------------------------------------------
Local Resources
--------------------------------------------------------------------------------
ora.FRA.dg
               ONLINE  ONLINE       tnc1                     STABLE
               ONLINE  ONLINE       tnc2                     STABLE
ora.GRID.dg
               ONLINE  ONLINE       tnc1                     STABLE
               ONLINE  ONLINE       tnc2                     STABLE
ora.LISTENER.lsnr
               ONLINE  ONLINE       tnc1                     STABLE
               ONLINE  ONLINE       tnc2                     STABLE
ora.SDXH.dg
               ONLINE  ONLINE       tnc1                     STABLE
               ONLINE  ONLINE       tnc2                     STABLE
ora.asm
               ONLINE  ONLINE       tnc1                     Started,STABLE
               ONLINE  ONLINE       tnc2                     Started,STABLE
ora.net1.network
               ONLINE  ONLINE       tnc1                     STABLE
               ONLINE  ONLINE       tnc2                     STABLE
ora.ons
               ONLINE  ONLINE       tnc1                     STABLE
               ONLINE  ONLINE       tnc2                     STABLE
--------------------------------------------------------------------------------
Cluster Resources
--------------------------------------------------------------------------------
ora.LISTENER_SCAN1.lsnr
      1        ONLINE  ONLINE       tnc1                     STABLE
ora.MGMTLSNR
      1        ONLINE  ONLINE       tnc1                     169.254.248.60 10.10
                                                                                   .10.180,STABLE
ora.cvu
      1        ONLINE  ONLINE       tnc2                     STABLE
ora.mgmtdb
      1        ONLINE  ONLINE       tnc1                     Open,STABLE
ora.oc4j
      1        ONLINE  ONLINE       tnc1                     STABLE
ora.p1tncd.db
      1        ONLINE  ONLINE       tnc1                     Open,STABLE
      2        ONLINE  ONLINE       tnc2                     Open,STABLE
ora.p1tncd.srvnsp.svc
      1        ONLINE  ONLINE       tnc1                     STABLE
      2        ONLINE  ONLINE       tnc2                     STABLE
ora.scan1.vip
      1        ONLINE  ONLINE       tnc1                     STABLE
ora.tnc1.vip
      1        ONLINE  ONLINE       tnc1                     STABLE
ora.tnc2.vip
      1        ONLINE  ONLINE       tnc2                     STABLE
--------------------------------------------------------------------------------

Services created on database 

"srvnsp" is the service created on Pluggable database "CBN" 

[oracle@tnc2 ~]$ srvctl config database -f
PRKO-2002 : Invalid command line option: -f
[oracle@tnc2 ~]$ srvctl config database -d p1tncd
Database unique name: p1tncd
Database name: p1tncd
Oracle home: /u01/app/oracle/product/12.1.0.2/db_1
Oracle user: oracle
Spfile: +SDXH/P1TNCD/PARAMETERFILE/spfile.271.903695731
Password file: +SDXH/P1TNCD/PASSWORD/pwdp1tncd.256.903695139
Domain: 
Start options: open
Stop options: immediate
Database role: PRIMARY
Management policy: AUTOMATIC
Server pools: 
Disk Groups: FRA,SDXH
Mount point paths: 
Services: srvnsp
Type: RAC
Start concurrency: 
Stop concurrency: 
OSDBA group: dba
OSOPER group: oper
Database instances: p1tncd1,p1tncd2
Configured nodes: tnc1,tnc2
Database is administrator managed

---Nikhil Tatineni--
--12c RAC--

Friday, February 12, 2016

12c: Integrated GoldenGate Extract

In order to increase the performance of Goldengate capture process, we can enable the log mining server so that extract will capture the logical records from log mining server and writes to the local trail 

In order to configure and invoke log mining server and  addition to goldengate privileges to assign, we need additional privilege's to enable the integrated capture on database 

SQL> exec dbms_goldengate_auth.grant_admin_privilege('ggsuser');
PL/SQL procedure successfully completed.

perform dblogin using ggsuser and password & Initially register the extract from the goldengate command utility as shown below

GGSCI (tnc61.ffdc.com) 2> dblogin userid ggsuser, password oracle
Successfully logged into database.

GGSCI (tnc61.ffdc.com as ggsuser@oval007p) 3> register extract extnew, database 
Extract EXTNEW successfully registered with database at SCN 3305172.

GGSCI (tnc61.ffdc.com as ggsuser@oval007p) 4> add extract extnew, integrated tranlog, begin now
EXTRACT added.

ggsci>edit param <extnew>


-- add extract extnew, EXTTRAILSOURCE /u01/app/trails/ca, integrated tranlog, begin now
extract extnew
SETENV (ORACLE_SID=oval007p)
SETENV (ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1)
userid ggsuser, password oracle
exttrail /u01/app/trails/ca
nocompressupdates 
nocompressdeletes 
REPORTCOUNT EVERY 10 MINUTES, RATE
tranlogoptions integrated params (max_sga_size 200, parallelism 1) 
---Include the full record
table nspdba.symbol, getbeforecols(on update all, on delete all); 


--Nikhil Tatineni--
--12c: GoldenGate---



12c: creating service on PDB

Update local_listener parameter for container database 

SQL> alter system set local_listener='tnc63.sfdc.sbc.com:1521';
System altered.

SQL> show parameter local_listener
NAME     TYPE VALUE
------------------------------------ ----------- ------------------------------
local_listener     string tnc63.sfdc.sbc.com:1521

 To create new service, alter session to Pluggable database "NSPDBA"

SQL> alter session set container=nspdba;
Session altered.

SQL> show con_name;
CON_NAME
------------------------------
NSPDBA

SQL> show parameter local_listener

NAME     TYPE VALUE
------------------------------------ ----------- ------------------------------
local_listener     string tnc63.sfdc.sbc.com:1521


Create and start Service using DBMS_SERVICE as follows 

SQL> exec dbms_service.create_service('ovals','ovals');
PL/SQL procedure successfully completed.

SQL> exec dbms_service.start_service('ovals');
PL/SQL procedure successfully completed.

when ever new service is created on pluggable database, Background process PMON register's with database  listener 

[admind@tnc63 ~]$ lsnrctl status
LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 12-FEB-2016 11:02:40
Copyright (c) 1991, 2014, Oracle.  All rights reserved.

Connecting to (ADDRESS=(PROTOCOL=tcp)(HOST=)(PORT=1521))
STATUS of the LISTENER
------------------------
Alias                     LISTENER
Version                   TNSLSNR for Linux: Version 12.1.0.2.0 - Production
Start Date                12-FEB-2016 10:50:04
Uptime                    0 days 0 hr. 12 min. 36 sec
Trace Level               off
Security                  ON: Local OS Authentication
SNMP                      OFF
Listener Parameter File   /u01/app/admind/product/12.1.0/dbhome_1/network/admin/listener.ora
Listener Log File         /u01/app/admind/diag/tnslsnr/tnc63/listener/alert/log.xml
Listening Endpoints Summary...
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcp)(HOST=tnc63.sfdc.sbc.com)(PORT=1521)))
  (DESCRIPTION=(ADDRESS=(PROTOCOL=tcps)(HOST=tnc63.sfdc.sbc.com)(PORT=5500))(Security=(my_wallet_directory=/u01/app/admind/product/12.1.0/dbhome_1/admin/admin/xdb_wallet))(Presentation=HTTP)(Session=RAW))
Services Summary...
Service "admin" has 1 instance(s).
  Instance "admin", status READY, has 1 handler(s) for this service...
Service "adminXDB" has 1 instance(s).
  Instance "admin", status READY, has 1 handler(s) for this service...
Service "nspdba" has 1 instance(s).
  Instance "admin", status READY, has 1 handler(s) for this service...
Service "ovals" has 1 instance(s).
  Instance "admin", status READY, has 1 handler(s) for this service...
The command completed successfully

Add following Entry in tnsnames.ora

OVALS =
  (DESCRIPTION =
    (ADDRESS_LIST =
      (ADDRESS = (PROTOCOL = TCP)(HOST = 10.0.0.140)(PORT = 1521))
    )
    (CONNECT_DATA =
      (SERVICE_NAME = ovals)
    )
  )

Try to connect to pluggable database using new service "ovals" created on pluggable database 

[admind@tnc63 ~]$ sqlplus nspdba/nspdba@ovals

SQL*Plus: Release 12.1.0.2.0 Production on Fri Feb 12 10:38:15 2016
Copyright (c) 1982, 2014, Oracle.  All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, OLAP, Advanced Analytics and Real Application Testing options

SQL> 
SQL> show con_name;
CON_NAME
------------------------------
NSPDBA

Note: when ever you bounce the CDB ROOT database, you need to restart the service created on pluggable database 

---Nikhil Tatineni--
---12c: Pluggable database--

Thursday, February 11, 2016

12c: PSU January 2016 New Notation (12.1.0.2. 160119)

Hi Guys,

Recently Upgraded my oracle 12c Database with 12c PSU January 2016 
Found New Notation for patching Number 12.1.0.2.160119 

[admind@tnc63 dbhome_1]$ opatch lsinventory
Oracle Interim Patch Installer version 12.1.0.1.10
Copyright (c) 2016, Oracle Corporation.  All rights reserved.
Oracle Home       : /u01/app/admind/product/12.1.0/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/admind/product/12.1.0/dbhome_1/oraInst.loc
OPatch version    : 12.1.0.1.10
OUI version       : 12.1.0.2.0
Log file location : /u01/app/admind/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2016-02-11_15-37-19PM_1.log
Lsinventory Output file location : /u01/app/admind/product/12.1.0/dbhome_1/cfgtoollogs/opatch/lsinv/lsinventory2016-02-11_15-37-19PM.txt
--------------------------------------------------------------------------------
Local Machine Information::
Hostname: tnc63.sfdc.sbc.com
ARU platform id: 226
ARU platform description:: Linux x86-64
Installed Top-level Products (1): 
Oracle Database 12c                                                  12.1.0.2.0
There are 1 products installed in this Oracle Home.
Interim patches (1) :
Patch  21948354     : applied on Thu Feb 11 14:21:05 EST 2016
Unique Patch ID:  19553095
Patch description:  "Database Patch Set Update : 12.1.0.2.160119 (21948354)"
   Created on 20 Dec 2015, 23:39:33 hrs PST8PDT
Sub-patch  21359755; "Database Patch Set Update : 12.1.0.2.5 (21359755)"
Sub-patch  20831110; "Database Patch Set Update : 12.1.0.2.4 (20831110)"
Sub-patch  20299023; "Database Patch Set Update : 12.1.0.2.3 (20299023)"
Sub-patch  19769480; "Database Patch Set Update : 12.1.0.2.2 (19769480)"
   Bugs fixed:
     19189525, 19075256, 19141838, 19865345, 19791273, 19280225, 18845653
     20951038, 19243521, 19248799, 21756699, 18988834, 19238590, 21281532
     20245930, 18921743, 18799063, 19134173, 19571367, 20476175, 20925795
     19018206, 20509482, 20387265, 20588502, 19149990, 18849537, 18886413
     17551063, 19183343, 19703301, 19001390, 18202441, 19189317, 19644859
     19358317, 19390567, 19279273, 19706965, 19068970, 19619732, 20348653
     18607546, 18940497, 19670108, 19649152, 18948177, 19315691, 19676905
     18964978, 19035573, 20165574, 19176326, 20413820, 20558005, 19176223
     19532017, 20134339, 19074147, 18411216, 20361671, 20425790, 18966843
     20294666, 19307662, 19371175, 19195895, 19154375, 19468991, 19174521
     19520602, 19382851, 21875360, 19326908, 19658708, 20093776, 20618595
     21787056, 17835294, 19791377, 19068610, 20048359, 20746251, 19143550
     19185876, 19627012, 20281121, 19577410, 22092979, 19001359, 19518079
     18610915, 19490948, 18674024, 18306996, 19309466, 19081128, 19915271
     20122715, 21188532, 20284155, 18791688, 20890311, 21442094, 18973548
     19303936, 19597439, 20235511, 18964939, 19430401, 19044962, 19409212
     19879746, 20657441, 19684504, 19024808, 18799993, 20877664, 19028800
     19065556, 19723336, 19077215, 19604659, 21421886, 19524384, 19048007
     18288842, 19689979, 20446883, 18952989, 16870214, 19928926, 21526048
     19180770, 19197175, 19902195, 20318889, 19730508, 19012119, 19067244
     20074391, 19512341, 19841800, 14643995, 20331945, 19587324, 19547370
     19065677, 19637186, 21225209, 20397490, 18967382, 19174430, 18674047
     19054077, 19536415, 19708632, 19289642, 20869721, 19335438, 17365043
     18856999, 19869255, 20471920, 19468347, 21620471, 16359751, 18990693
     17890099, 19439759, 19769480, 19272708, 19978542, 20101006, 21300341
     20402832, 19329654, 19873610, 21668627, 21517440, 19304354, 19052488
     20794034, 19291380, 18681056, 19896336, 19076343, 19561643, 18618122
     20440930, 18456643, 19699191, 18909599, 19487147, 18250893, 19016730
     18743542, 20347562, 16619249, 18354830, 19687159, 19174942, 20424899
     19989009, 20688221, 20441797, 19157754, 19032777, 19058490, 19399918
     18885870, 19434529, 19018447, 18417036, 20919320, 19022470, 19284031
     20474192, 20173897, 22062026, 19385656, 19501299, 17274537, 20899461
     19440586, 16887946, 19606174, 18436647, 17655240, 19023822, 19178851
     19124589, 19805359, 19597583, 19155797, 19393542, 19050649
--------------------------------------------------------------------------------
OPatch succeeded.

currently my database is upgraded to version 
Patch description:  "Database Patch Set Update : 12.1.0.2.160119 (21948354)"

--Nikhil Tatineni--
--12c: Patching PDBS & CDB'S---

Querys to monitor RAC

following few  Query's will help to find out culprits-  Query to check long running transaction from last 8 hours  Col Sid Fo...