Monday, February 15, 2016

Oracle RAC Blocked Listener 12c

Checked services supported by scan_listener and I found blocked state of the listener / To find out Root cause, we have to understand functionality of the "scan_listener" on RAC Cluster. When ever SCAN receives client  request, one of the SCAN IP receives request and perform load balancing using  VIP. Finally VIP allows or makes connection with database. As we know that local_listener are registered with VIP. when ever we see blocked state of the listener, we have to validate the local_listener and remote listener of the database including port Number. Based on error message on the blocked state of the listener we can investigate further  

[oracle@tnc1 admin]$ lsnrctl services LISTENER_SCAN1


LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 14-FEB-2016 19:15:02Copyright (c) 1991, 2014, Oracle.  All rights reserved.Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))Services Summary...Service "-MGMTDBXDB" has 1 instance(s).  Instance "-MGMTDB", status READY, has 1 handler(s) for this service...    Handler(s):      "D000" established:0 refused:0 current:0 max:1022 state:ready         DISPATCHER <machine: tnc1.localdomain, pid: 6449>         (ADDRESS=(PROTOCOL=tcp)(HOST=tnc1.localdomain)(PORT=15726))Service "_mgmtdb" has 1 instance(s).  Instance "-MGMTDB", status READY, has 1 handler(s) for this service...    Handler(s):      "DEDICATED" established:0 refused:0 state:ready         REMOTE SERVER         (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.1.11)(PORT=1521))Service "cbn" has 2 instance(s).  Instance "p1tncd1", status READY, has 1 handler(s) for this service...    Handler(s):      "DEDICATED" established:0 refused:0 state:blocked         REMOTE SERVER (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.1.22)(PORT=1525))  Instance "p1tncd2", status READY, has 1 handler(s) for this service...    

Handler(s):      "DEDICATED" established:0 refused:0 state:blocked         
REMOTE SERVER         (ADDRESS=(PROTOCOL=TCP (HOST=192.168.1.22)(PORT=1525))Service "p1tncd" has 2 instance(s).  Instance "p1tncd1", status READY, has 1 handler(s) for this service...    Handler(s):      "DEDICATED" established:0 refused:0 state:blocked    REMOTE SERVER (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.1.22)(PORT=1525))  Instance "p1tncd2", status READY, has 1 handler(s) for this service...    Handler(s):      "DEDICATED" established:0 refused:0 state:blocked  REMOTE SERVER  (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.1.22)(PORT=1521))

Here when you check, Number of connections established is zero, this Indicates clearly there is problem with listener configuration. If column "established" shows some value other than "0"  and then we need to check resources available on the server / on database. In our scenario we need to  check local_listener parameter & port number associated with VIP  Update “local_listener” parameter with new port number “1521”


On Node 1

SQL> alter system set local_listener='(ADDRESS = (PROTOCOL=TCP)(HOST=192.168.1.11)(PORT=1521))’ scope=both sid='p1tncd1';

On Node 2

SQL> alter system set local_listener='(ADDRESS = (PROTOCOL=TCP)(HOST=192.168.1.21)(PORT=1521))’ scope=both sid='p1tncd2’;

After stop and start listener using srvctl stop and start listener 

[oracle@tnc1 ~]$ srvctl stop listener
[oracle@tnc1 ~]$ srvctl start listener



[oracle@tnc1 ~]$ lsnrctl services LISTENER_SCAN1

LSNRCTL for Linux: Version 12.1.0.2.0 - Production on 15-FEB-2016 11:29:13
Copyright (c) 1991, 2014, Oracle.  All rights reserved.
Connecting to (DESCRIPTION=(ADDRESS=(PROTOCOL=IPC)(KEY=LISTENER_SCAN1)))
Services Summary...
Service "-MGMTDBXDB" has 1 instance(s).
  Instance "-MGMTDB", status READY, has 1 handler(s) for this service...
    Handler(s):
      "D000" established:0 refused:0 current:0 max:1022 state:ready
         DISPATCHER <machine: tnc1.localdomain, pid: 6912>
         (ADDRESS=(PROTOCOL=tcp)(HOST=tnc1.localdomain)(PORT=10546))
Service "_mgmtdb" has 1 instance(s).
  Instance "-MGMTDB", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:ready
         REMOTE SERVER
         (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.1.11)(PORT=1521))
Service "cbn" has 2 instance(s).
  Instance "p1tncd1", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:ready
         REMOTE SERVER
         (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.1.11)(PORT=1521))
  Instance "p1tncd2", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:ready
         REMOTE SERVER
         (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.1.21)(PORT=1521))
Service "p1tncd" has 2 instance(s).
  Instance "p1tncd1", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:ready
         REMOTE SERVER
         (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.1.11)(PORT=1521))
  Instance "p1tncd2", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:ready
         REMOTE SERVER
         (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.1.21)(PORT=1521))
Service "p1tncdXDB" has 2 instance(s).
  Instance "p1tncd1", status READY, has 1 handler(s) for this service...
    Handler(s):
      "D000" established:0 refused:0 current:0 max:1022 state:ready
         DISPATCHER <machine: tnc1.localdomain, pid: 7257>
         (ADDRESS=(PROTOCOL=tcp)(HOST=tnc1.localdomain)(PORT=37946))
  Instance "p1tncd2", status READY, has 1 handler(s) for this service...
    Handler(s):
      "D000" established:0 refused:0 current:0 max:1022 state:ready
         DISPATCHER <machine: tnc2.localdomain, pid: 6582>
         (ADDRESS=(PROTOCOL=tcp)(HOST=tnc2.localdomain)(PORT=33674))
Service "srvnsp" has 2 instance(s).
  Instance "p1tncd1", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:ready
         REMOTE SERVER
         (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.1.11)(PORT=1521))
  Instance "p1tncd2", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:ready
         REMOTE SERVER
         (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.1.21)(PORT=1521))
Service "tnc_cluster" has 1 instance(s).
  Instance "-MGMTDB", status READY, has 1 handler(s) for this service...
    Handler(s):
      "DEDICATED" established:0 refused:0 state:ready
         REMOTE SERVER
         (ADDRESS=(PROTOCOL=TCP)(HOST=192.168.1.11)(PORT=1521))
The command completed successfully

---Nikhil Tatineni----
---12c Oracle RAC---




Querys to monitor RAC

following few  Query's will help to find out culprits-  Query to check long running transaction from last 8 hours  Col Sid Fo...