Saturday, September 17, 2016

oratop # DBA friendly

The “oratop” displays relevant database activity information presented in four sections. 

Section 1 - HEADER: Global database information  
Section 2 - DATABASS: Database instance Activity  
Section 3 - DB EVENTS: AWR like “Top 5 Timed Events“  

Section 4 – PROCESS/SQL: Processes/SQL information Specific detail of the various sections data can be obtained using the interactive help (see Runtime usage Sections Detail below) 


[oracle@tnc2 ~]$ tfactl print status
.-------------------------------------------------------------------------------------------.
| Host | Status of TFA | PID  | Port | Version    | Build ID             | Inventory Status |
+------+---------------+------+------+------------+----------------------+------------------+
| tnc2 | RUNNING       | 4280 | 5000 | 12.1.2.8.0 | 12128020160623222219 | COMPLETE         |
'------+---------------+------+------+------------+----------------------+------------------'
[oracle@tnc2 ~]$ tfactl
tfactl> db12c                                                                                        

Usage : /u01/app/12.1.0.2/grid/bin/tfactl <command> [options]
<command> =
          print          Print requested details
          purge          Delete collections from TFA repository
          analyze        List events summary and search strings in alert logs.
          diagcollect    Collect logs from across nodes in cluster
          collection     Manage TFA collections
          directory      Add or Remove or Modify directory in TFA
          toolstatus     Prints the status of TFA Support Tools
          run <tool>     Run the desired support tool
          start <tool>   Starts the desired support tool
          stop <tool>    Stops the desired support tool

For help with a command: /u01/app/12.1.0.2/grid/bin/tfactl <command> -help

tfactl> db db12c                                                                                     
Set db to DB12C
DB12C tfactl> oratop -i 10 -bn10                                                                     
Cycle 1 - oratop: Release 14.1.2 Production on Wed Sep 14 11:00:16 2016

Oracle 12c - db1 11:00:08 up: 0.4h,   2 ins,    0 sn,   0 us, 1.8G mt,  1.9% db
ID %CPU LOAD %DCU   AAS  ASC  ASI  ASW  AST IOPS %FR   PGA UTPS UCPS SSRT  %DBT
-------------------------------------------------------------------------------
 1   21    1    0     1    0    0    0    0    4   6  368M    0    2 975u  84.3
 2   11    1    0     1    0    0    0    0    3   4  323M    0    2 462u  15.7

EVENT (C)                        TOT WAITS   TIME(s)  AVG_MS  PCT    WAIT_CLASS
----------------------------------------------------------------------------------------------
PX Deq: Slave Session Stats           2760         453   179.6   36           Other
PX Deq: reap credit                       47402       320     6.7     25         Other
DB CPU                                             188        15              
oracle thread bootstrap                     319       183   580.5   15        Other
reliable message                               890       118   168.9    9          Other
ID   SID     SPID USR PROG S  PGA SQLID/BLOCKER OPN  E/T STA STE EVENT/*LA  W/T
-------------------------------------------------------------------------------

From 
Section 4 – PROCESS/SQL: Processes/SQL information Specific detail of the various sections data 
we will get SQL_ID involved in waits 

check execution plan of SQL statement using dbms_xplan.display_cursor & SQL_TEXT as follows
sql> select * from table(dbms_xplan.display_cursor(‘&sql_id’,’&child_no’,’typical’));

SQL> col SQL_TEXT format a10000
SQL> select SQL_TEXT from v$sqltext where SQL_ID='worldoforacle';

--Nikhil Tatineni--
--RAC--





Tuesday, September 13, 2016

TFA not Moved to New GI_HOME while Upgrading from 11.2.0.4 to 12c

Scenario: Upgrading to 12.1.0.2 Grid Infrastructure but TFA wasn't installed in the new GI_HOME: 

>> TFA not Moved to New GI_HOME While Upgrading to 12.1.0.2 (Doc ID 1918428.1)
Follow to fix this issue +

Deinstall TFA from 11.2.0.4 

[root@tnc2 bin]# tfactl uninstall

TFA will be Uninstalled on Node tnc2: 
Removing TFA from tnc2 only
Please remove TFA locally on any other configured nodes

Notifying Other Nodes about TFA Uninstall...
Sleeping for 10 seconds...

Stopping TFA Support Tools...
Stopping TFA in tnc2...

Shutting down TFA
TFA is running  - Will wait 5 seconds (up to 3 times)  
TFA-00002 : Oracle Trace File Analyzer (TFA) is not running
TFA Stopped Successfully
. . . . . 
. . . 
Successfully shutdown TFA..

Deleting TFA support files on tnc2:
Removing /u01/app/oracle/tfa/tnc2/database...
Removing /u01/app/oracle/tfa/tnc2/log...
Removing /u01/app/oracle/tfa/tnc2/output...
Removing /u01/app/oracle/tfa/tnc2...
Removing /u01/app/oracle/tfa...
Removing /etc/rc.d/rc0.d/K17init.tfa
Removing /etc/rc.d/rc1.d/K17init.tfa
Removing /etc/rc.d/rc2.d/K17init.tfa
Removing /etc/rc.d/rc4.d/K17init.tfa
Removing /etc/rc.d/rc6.d/K17init.tfa
Removing /etc/init.d/init.tfa...
Removing /u01/app/11.2.0.4/grid/bin/tfactl...
Removing /u01/app/11.2.0.4/grid/tfa/bin...
Removing /u01/app/11.2.0.4/grid/tfa/tnc2...

Install TFA as root user on new 12.1.0.2 Cluster Home + 

Syntax #
<NEW_GI_HOME>/crs/install/tfa_setup -silent -crshome <NEW_GI_HOME

/u01/app/12.1.0.2/grid/crs/install/tfa_setup -silent -crshome /u01/app/12.1.0.2/grid
>>>
[root@tnc2 grid]# /u01/app/12.1.0.2/grid/crs/install/tfa_setup -silent -crshome /u01/app/12.1.0.2/grid

Starting TFA installation
Using JAVA_HOME : /u01/app/12.1.0.2/grid/jdk/jre

Running Auto Setup for TFA as user root...
Installing TFA now...

TFA Will be Installed on tnc2...
TFA will scan the following Directories
++++++++++++++++++++++++++++++++++++++++++++

.------------------------------------------------------.
|                         tnc2                         |
+-------------------------------------------+----------+
| Trace Directory                           | Resource |
+-------------------------------------------+----------+
| /u01/app/12.1.0.2/grid/OPatch/crs/log     | CRS      |
| /u01/app/12.1.0.2/grid/cfgtoollogs        | CFGTOOLS |
| /u01/app/12.1.0.2/grid/crs/log            | CRS      |
| /u01/app/12.1.0.2/grid/cv/log             | CRS      |
| /u01/app/12.1.0.2/grid/evm/admin/log      | CRS      |
| /u01/app/12.1.0.2/grid/evm/admin/logger   | CRS      |
| /u01/app/12.1.0.2/grid/evm/log            | CRS      |
| /u01/app/12.1.0.2/grid/install            | INSTALL  |
| /u01/app/12.1.0.2/grid/log                | CRS      |
| /u01/app/12.1.0.2/grid/network/log        | CRS      |
| /u01/app/12.1.0.2/grid/oc4j/j2ee/home/log | DBWLM    |
| /u01/app/12.1.0.2/grid/opmn/logs          | CRS      |
| /u01/app/12.1.0.2/grid/racg/log           | CRS      |
| /u01/app/12.1.0.2/grid/rdbms/log          | ASM      |
| /u01/app/12.1.0.2/grid/scheduler/log      | CRS      |
| /u01/app/12.1.0.2/grid/srvm/log           | CRS      |
| /u01/app/oraInventory/ContentsXML         | INSTALL  |
| /u01/app/oraInventory/logs                | INSTALL  |
| /u01/app/oracle/crsdata/tnc2/acfs         | ACFS     |
| /u01/app/oracle/crsdata/tnc2/core         | CRS      |
| /u01/app/oracle/crsdata/tnc2/crsconfig    | CRS      |
| /u01/app/oracle/crsdata/tnc2/crsdiag      | CRS      |
| /u01/app/oracle/crsdata/tnc2/cvu          | CRS      |
| /u01/app/oracle/crsdata/tnc2/evm          | CRS      |
| /u01/app/oracle/crsdata/tnc2/output       | CRS      |
| /u01/app/oracle/crsdata/tnc2/trace        | CRS      |
'-------------------------------------------+----------'

Installing TFA on tnc2:
HOST: tnc2 TFA_HOME: /u01/app/12.1.0.2/grid/tfa/tnc2/tfa_home
.------------------------------------------------------------------------.
| Host | Status of TFA | PID  | Port | Version    | Build ID             |
+------+---------------+------+------+------------+----------------------+
| tnc2 | RUNNING       | 3679 | 5000 | 12.1.2.0.0 | 12120020140619094932 |
'------+---------------+------+------+------------+----------------------'
Running Inventory in All Nodes...
Enabling Access for Non-root Users on tnc2...
Adding default users and groups to TFA Access list...

Summary of TFA Installation:
.----------------------------------------------------------------.
|                              tnc2                              |
+---------------------+------------------------------------------+
| Parameter           | Value                                    |
+---------------------+------------------------------------------+
| Install location    | /u01/app/12.1.0.2/grid/tfa/tnc2/tfa_home |
| Repository location | /u01/app/oracle/tfa/repository           |
| Repository usage    | 0 MB out of 6247 MB                      |
'---------------------+------------------------------------------'
Installing oratop extension..
TFA is successfully installed...

Usage : /u01/app/12.1.0.2/grid/bin/tfactl <command> [options]
<command> =
          start        Starts TFA
          stop         Stops TFA
          enable       Enable TFA Auto restart
          disable      Disable TFA Auto restart
          print        Print requested details
          access       Add or Remove or List TFA Users and Groups
          purge        Delete collections from TFA repository
          directory    Add or Remove or Modify directory in TFA
          host         Add or Remove host in TFA
          diagcollect  Collect logs from across nodes in cluster
          analyze      List events summary and search strings in alert logs.
          set          Turn ON/OFF or Modify various TFA features
          uninstall    Uninstall TFA from this node

For help with a command: /u01/app/12.1.0.2/grid/bin/tfactl <command> -help

[oracle@tnc2 TFA]$ tfactl print status
.--------------------------------------------------------------------------------------------.
| Host | Status of TFA | PID   | Port | Version    | Build ID             | Inventory Status |
+------+---------------+-------+------+------------+----------------------+------------------+
| tnc2 | RUNNING       |  3679 | 5000 | 12.1.2.0.0 | 12120020140619094932 | COMPLETE         |
| tnc1 | RUNNING       | 12158 | 5000 | 12.1.2.0.0 | 12120020140619094932 | COMPLETE         |
'------+---------------+-------+------+------------+----------------------+------------------'

[oracle@tnc2 TFA]$ crsctl query crs softwareversion
Oracle Clusterware version on node [tnc2] is [12.1.0.2.0]


>>>>>>

--Nikhil Tatineni--
--RAC--

Sunday, September 11, 2016

12c RAC > Clusterware logs

Recently upgraded my RAC cluster from 11.2.0.4 to 12.1.0.2
Clusterware logs are centralized in  RAC 12c 

>> In 11gR2 RAC Cluster, we have different directories to investigate or troubleshoot issues in cluster <<

/u01/app/11.2.0.4/grid/log/tnc2

[oracle@tnc2 tnc2]$ ll
total 316
drwxr-xr-x 3 root   root       4096 Jul 13 12:38 acfs
drwxr-x--- 3 oracle oinstall   4096 Aug 20 12:54 acfslog
drwxr-x--- 2 oracle oinstall   4096 Jul 13 12:37 acfsrepl
drwxr-x--- 2 root   oinstall   4096 Jul 13 12:37 acfsreplroot
drwxr-xr-x 2 root   oinstall   4096 Jul 13 12:37 acfssec
drwxr-x--- 2 oracle oinstall   4096 Jul 13 12:37 admin
drwxrwxr-t 4 root   oinstall   4096 Jul 13 12:37 agent
-rw-rw-r-- 1 oracle oinstall 221280 Sep  2 17:56 alerttnc2.log
drwxrwxrwt 2 oracle oinstall   4096 Sep  2 17:55 client
drwxr-x--- 2 root   oinstall   4096 Jul 13 12:41 crflogd
drwxr-x--- 2 root   oinstall   4096 Jul 13 12:41 crfmond
drwxr-x--- 2 root   oinstall   4096 Aug 26 13:47 crsd
drwxr-x--- 2 oracle oinstall   4096 Aug 20 12:51 cssd
drwxr-x--- 2 root   oinstall   4096 Aug 20 15:25 ctssd
drwxr-x--- 4 oracle oinstall   4096 Jul 13 12:37 cvu
drwxr-x--- 2 oracle oinstall   4096 Jul 13 12:37 diskmon
drwxr-x--- 2 oracle oinstall   4096 Jul 13 12:41 evmd
drwxr-x--- 2 oracle oinstall   4096 Sep  2 13:58 gipcd
drwxr-x--- 2 root   oinstall   4096 Jul 13 12:37 gnsd
drwxr-x--- 2 oracle oinstall   4096 Sep  2 14:32 gpnpd
drwxr-x--- 2 oracle oinstall   4096 Jul 13 12:38 mdnsd
drwxr-x--- 2 root   oinstall   4096 Aug 25 15:28 ohasd
drwxrwxr-t 5 oracle oinstall   4096 Aug 24 15:51 racg
drwxr-x--- 2 oracle oinstall   4096 Jul 13 12:37 srvm

>> But in 12c RAC cluster, Oracle centralized logs & Location as follows << 
>> It is easy to troubleshoot Issues in Clusterware Now << 

[oracle@tnc2 trace]$ pwd
/u01/app/oracle/diag/crs/tnc2/crs/trace

/u01/app/oracle/diag/crs/<Node Name>/crs/trace

> Using Alert log file, we can find Root cause in New 12c RAC Cluster 

[oracle@tnc2 trace]$ ls -ltr alert.log 
-rw-rw---- 1 oracle oinstall 124737 Sep 11 20:39 alert.log

<< << << << Use ADRCI to get ADR location :) 

[oracle@tnc2 grid]$ adrci

ADRCI: Release 12.1.0.2.0 - Production on Sun Sep 11 21:04:24 2016
Copyright (c) 1982, 2014, Oracle and/or its affiliates.  All rights reserved.

ADR base = "/u01/app/oracle"
adrci> show homes
ADR Homes: 
diag/tnslsnr/tnc2/listener_scan1
diag/tnslsnr/tnc2/mgmtlsnr
diag/tnslsnr/tnc2/listener
diag/rdbms/mipd09/mipd092
diag/rdbms/_mgmtdb/-MGMTDB
diag/rdbms/db12c/db12c_2
diag/asm/+asm/+ASM2
diag/crs/tnc2/crs

>> >> >>
--Nikhil Tatineni--
--12c RAC--


Saturday, September 10, 2016

ERROR OGG-01091 Unable to open file # Replicat Abended

Scenario # RNDELTA6 abended with Error # 
During New GoldenGate setup removed old trails manually from Remote Trail Directory # 

GGSCI (tnc61) 1> view report RNDELTA6
........
...

Source Context :
  SourceModule            : [er.api]
  SourceID                : [/scratch/aime1/adestore/views/aime1_adc4150256/oggcore/OpenSys/src/app/er/api.c]
  SourceFunction          : [XT_read]
  SourceLine              : [4145]
  ThreadBacktrace         : [9] elements                         : [/opt/app/t1c2d985/ggs/11.2.1.0.3/libgglog.so(CMessageContext::AddThreadContext()+0x1e) [0x7f8230dc270e]]  : [/opt/app/t1c2d985/ggs/11.2.1.0.3/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int,
 ...)+0x2cc) [0x7f8230dbb6ac]]    : [/opt/app/t1c2d985/ggs/11.2.1.0.3/libgglog.so(_MSG_ERR_FILE_OPEN_ERROR(CSourceContext*, char const*, CMessa
geFactory::MessageDisposition)+0x31) [0x7f8230da58dd] : [/opt/app/t1c2d985/ggs/11.2.1.0.3/replicat(XT_read(chkpt_context_t*, char*, __std_rec_hdr*, int*, int*, sho
rt, short, char*, short*, char*, short*, int, long*, char*)+0xd71) [0x4c80b1]   : [/opt/app/t1c2d985/ggs/11.2.1.0.3/replicat(READ_EXTRACT_RECORD(chkpt_context_t*, short, char*, __std_rec_hd
r*, int*, int*, long*, short, short, char*, short*, char*, short*, long*, char*)+0xe8) [0x4c8ff8]]
                          : [/opt/app/t1c2d985/ggs/11.2.1.0.3/replicat(process_extract_loop()+0x402) [0x538dc2]]
                          : [/opt/app/t1c2d985/ggs/11.2.1.0.3/replicat(main+0x732) [0x54cc02]]
                          : [/lib64/libc.so.6(__libc_start_main+0xfd) [0x3a6ca1ed5d]]
                          : [/opt/app/t1c2d985/ggs/11.2.1.0.3/replicat(__gxx_personality_v0+0x31a) [0x4c0c6a]]
2016-09-09 03:10:29  ERROR   OGG-01091  Unable to open file "/appl/tnc1/gghome/dirdat/bg000005" (error 2, No such file o
r directory).

During Initial Investigation, we found this issue from report file, trail with sequence number 5 doesn't exist physically. Validated the physical location of trails by querying from GGSCI command line utility 

GGSCI (tnc61) 3> sh ls -ltr /appl/tnc1/gghome/dirdat/bg*
-rw-rw-rw- 1 t1c2d985g t1c2d985g     0 Sep  7 14:32 /appl/tnc1/gghome/dirdat/bg000006
-rw-rw-rw- 1 t1c2d985g t1c2d985g     0 Sep  7 14:48 /appl/tnc1/gghome/dirdat/bg000007
-rw-rw-rw- 1 t1c2d985g t1c2d985g     0 Sep  7 14:49 /appl/tnc1/gghome/dirdat/bg000008
-rw-rw-rw- 1 t1c2d985g t1c2d985g 12198 Sep  9 03:09 /appl/tnc1/gghome/dirdat/bg000009

To Resolve this issue #
alter the replicat to next available sequence number on the trail directory #
GGSCI (tnc61) 4> alter RNDELTA6, extseqno 6 extrba 0
REPLICAT altered.

GGSCI (tnc61) 5> start RNDELTA6
Sending START request to MANAGER ...
REPLICAT RNDELTA6 starting

GGSCI (tnc61) 8> info RNDELTA6
REPLICAT   RNDELTA6  Last Started 2016-09-09 03:12   Status RUNNING
Checkpoint Lag       00:00:00 (updated 00:00:03 ago)
Log Read Checkpoint  File /appl/tnc1/gghome/dirdat/bg000009

                     2016-09-01 05:19:56.000685  RBA 12198

--Nikhil Tatineni--
--gg--

ERROR OGG-01031 There is a problem in network communication, a remote file problem,

Scenario # Brand New pump writing to Target server got abended with following Error # 

Database Language and Character Set:
NLS_LANG         = ".US7ASCII"
NLS_LANGUAGE     = "AMERICAN"
NLS_TERRITORY    = "AMERICA"
NLS_CHARACTERSET = "US7ASCII"
2016-09-01 10:24:30  INFO    OGG-01226  Socket buffer size set to 27985 (flush size 27985).
Source Context :
  SourceModule            : [er.extrout]
  SourceID                : [/scratch/aime1/adestore/views/aime1_adc4150256/oggcore/OpenSys/src/app/er/extrout.c]
  SourceFunction          : [complete_tcp_msg]
  SourceLine              : [1526]
  ThreadBacktrace         : [9] elements
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/libgglog.so(CMessageContext::AddThreadContext()+0x1e) [0x7f8ea9e4470e]]
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, ...)+0x2cc) [0x7f8ea9e3d6ac]]
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/libgglog.so(_MSG_ERR_ER_REMOTE_COMM_PROBLEM(CSourceContext*, char const*, CMessageFactory::MessageDisposition)+0x31) [0x7f8ea9e28789]]
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/extract(complete_tcp_msg(extract_def*)+0x421) [0x519049]]
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/extract(flush_tcp(extract_def*, int)+0x206) [0x5198f2]]
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/extract(RECOVERY_initialize()+0x371) [0x52ae51]]
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/extract(main+0x4a5) [0x573985]]
                          : [/lib64/libc.so.6(__libc_start_main+0xfd) [0x311801ed5d]]
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/extract(__gxx_personality_v0+0x38a) [0x4edaba]]
2016-09-01 10:24:30  ERROR   OGG-01031  There is a problem in network communication, a remote file problem, encryption keys for target and source do not match (if using ENCRYPT) or an unknown error. (Reply received is Output file /usr/local/opt/oracle/ggs/11.2.1.0.1/trails/gi000000 is not in any allowed output directories.).
2016-09-01 10:24:30  ERROR   OGG-01668  PROCESS ABENDING.

To find out Root Cause # 

workaround > Telnet fails, 
There is problem with network between servers (or)
Blocking port Issues on target server (or)
manager child process (collector) Problems # 

$ telnet tnc61.world.com 7809
Trying 192.168.56.212...
Connected to tnc1.world.com
Escape character is '^]'.
Connection closed by foreign host.

Here Telnet worked as expected. There is no issues with network or port Number # 
Later ...
Checked status of Manager # manager is up & running and  gathering port details 
if every thing looks good, you can confirm child process is up & running on manager end

GGSCI (tnc61) 19> SEND MANAGER GETPORTINFO DETAIL
Sending GETPORTINFO, request to MANAGER ...
Dynamic Port List
Starting Index 0
Entry Port  Error  Process     Assigned             Program
----- ----- ----- ----------   -------------------  -------
   0   7809     0
   1   7810     0
   2   7811     0
   3   7812     0
   4   7813     0
   5   7814     0
   6   7815     0
   7   7816     0
   8   7817     0
   9   7818     0
  10   7819     0
  11   7820     0
  12   7821     0
  13   7822     0
  14   7823     0
  15   7824     0
  16   7825     0
  17   7826     0
  18   7827     0
  19   7828     0
  20   7829     0
  21   7830     0
  22   7833     0
  23   7835     0

>> few messages from Targetgg_HOME/ggserr.log 


2016-08-31 09:55:02  WARNING OGG-01223  Oracle GoldenGate Collector for Oracle:  Output file /usr/local/opt/oracle/ggs/11.2.1.0.1/trails/gi000000 is not in any allowed output directories.
2016-08-31 09:55:02  INFO    OGG-01676  Oracle GoldenGate Collector for Oracle:  Terminating after client disconnect.
2016-08-31 09:58:15  INFO    OGG-00963  Oracle GoldenGate Manager for Oracle, mgr.prm:  Command received from EXTRACT on host [130.6.44.131]:39623 (START SERVER CPU -1 PRI -1  TIMEOUT 300 PARAMS ).
2016-08-31 09:58:15  INFO    OGG-00960  Oracle GoldenGate Manager for Oracle, mgr.prm:  Access granted (rule #4).
2016-08-31 09:58:15  INFO    OGG-01677  Oracle GoldenGate Collector for Oracle:  Waiting for connection (started dynamically).
2016-08-31 09:58:15  INFO    OGG-00963  Oracle GoldenGate Manager for Oracle, mgr.prm:  Command received from SERVER on host [127.0.0.1]:9264 (REPORT 1442204 7809).
2016-08-31 09:58:15  INFO    OGG-00960  Oracle GoldenGate Manager for Oracle, mgr.prm:  Access granted (rule #1).
2016-08-31 09:58:15  INFO    OGG-00974  Oracle GoldenGate Manager for Oracle, mgr.prm:  Manager started collector process (Port 7809).
2016-08-31 09:58:15  INFO    OGG-01228  Oracle GoldenGate Collector for Oracle:  Timeout in 300 seconds.
2016-08-31 09:58:20  INFO    OGG-01229  Oracle GoldenGate Collector for Oracle:  Connected to tnc1.world.com2016-08-31 09:58:20  WARNING OGG-01223  Oracle GoldenGate Collector for Oracle:  Output file /usr/local/opt/oracle/ggs/11.2.1.0.1/trails/gi000000 is not in any allowed output directories.
2016-08-31 09:58:20  INFO    OGG-01676  Oracle GoldenGate Collector for Oracle:  Terminating after client disconnect.


Later # 
Investigate file system, i.e report file says " Reply received is Output file /usr/local/opt/oracle/ggs/11.2.1.0.1/trails/gi000000 is not in any allowed output directories.)."  
check for free space available on Target server ( Target file system )

$  df -g /usr/local/opt/oracle/ggs/11.2.1.0.1/trails/
Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
/dev/ggs_lv       20.00     19.70    2%      215     1% /usr/local/opt/oracle/ggs

>> >>> 
Later I raised Oracle SR for this issue ..Followed (Doc ID 1288562.1) and resolved it

Resolution: # In order to fix this issue, 
Deleted existing pump & 
Changed Remote trail directory and granted required permissions on remote trail directory
Recreated pump and restarted the brand NEW PUMP #

Reference # 

OGG Data Pumps Error With OGG-01031 When Upgrading the Target Side (Doc ID 1288562.1)

--Nikhil Tatineni--
--GoldenGate--

Thursday, September 8, 2016

ERROR OGG-00664 OCI Error during OCIServerAttach (status = 12162-ORA-12162:

Scenario # Rebooted golden gate process, Pump Abended with following error <
Investigated  Issue and found root cause # 

ggsci> View report plct90
.....
....
............
extract plct90
passthru
userid ggsuser, password *******
Source Context :
  SourceModule            : [ggdb.ora.sess]
  SourceID                : [/scratch/aime1/adestore/views/aime1_staxj04/oggcore
/OpenSys/src/gglib/ggdbora/ocisess.c]
  SourceFunction          : [OCISESS_context_def::oci_try(int, const char *, ...)]
  SourceLine              : [650]
2016-09-04 08:48:14  ERROR   OGG-00664  OCI Error during OCIServerAttach (status

 = 12162-ORA-12162: TNS:net service name is incorrectly specified)

Pump Param file looks like as follows # 
ggsci> view param plct90
extract plct90
userid ggsuser, password oracle
PASSTHRU
RMTHOST tnc61.ffdc.com, MGRPORT 7809
RMTTRAIL /u01/app/trails/aa
REPORTCOUNT EVERY 10 MINUTES, RATE
TABLE world.orders;

Resolution: New changes in parameter file and start pump 
ggsci> view param plct90
extract plct90
userid ggsuser@tns_name, password oracle
PASSTHRU
RMTHOST tnc61.ffdc.com, MGRPORT 7809
RMTTRAIL /u01/app/trails/aa
REPORTCOUNT EVERY 10 MINUTES, RATE

ggsci> start plct90 

--pump came up automatically, Issue is resolved --

--Nikhil Tatineni--
--gg--

Links Related to GoldenGate #

>>

Adding a disk to diskgroup # ASM

>>>> >>> check OS Level ++++
[root@tnc1 dev]# ls -ltr asm*
brw-rw---- 1 oracle dba      8, 129 Sep  5 07:59 asm-disk8
brw-rw---- 1 oracle dba      8, 145 Sep  5 07:59 asm-disk9

>>>>  ## >>> Log in as "sysasm" and query header status, we can add a disk to the disk group when it's header status is CANDIDATE (or) PROVISIONED (or) FORMER +++

[oracle@tnc1 scripts]$ cat /u01/stage/scripts/check_free_space_disks.sql
set lines 255
col path for a35
col Diskgroup for a15
col DiskName for a20
col disk# for 999
col total_mb for 999,999,999
col free_mb for 999,999,999
compute sum of total_mb on DiskGroup
compute sum of free_mb on DiskGroup
break on DiskGroup skip 1 on report -

set pages 255

select a.name DiskGroup, b.disk_number Disk#, b.name DiskName, b.total_mb, b.free_mb, b.path, b.header_status
from v$asm_disk b, v$asm_diskgroup a
where a.group_number (+) =b.group_number
order by b.group_number, b.disk_number, b.name
/

set lines 122
set pages 66

>>>> ### >>>
>>>>
Log in as sysasm #  and add a disk to diskgroup ##
SQL> alter diskgroup stage add disk '/dev/asm-disk8','/dev/asm-disk9';
Diskgroup altered.

"The Number of Actual Rebalancing processes allocated Depend's upon parameter asm_power_limit" it ranges from 0 to 11 & 0 to 1024 depends on disk group compatibility 

sql> alter system set asm_power_limit=11;

>>>>  ##  >>>

Monitor Rebalancing Operation +++
SQL> select group_number,operation,state,power,est_minutes from v$asm_operation;

---Nikhil Tatineni---
---ASM --- 



Querys to monitor RAC

following few  Query's will help to find out culprits-  Query to check long running transaction from last 8 hours  Col Sid Fo...