Tuesday, December 6, 2016

Query's Used to monitor Goldengate Integrated Extract and replicat

Integrated capture GoldenGate views on Source Database >

col CAPTURE_NAME for a20;
col QUEUE_NAME for a15;
col START_SCN for 9999999999;
col STATUS for a10;
col CAPTURED_SCN for 9999999999;
col APPLIED_SCN for 9999999999;
col SOURCE_DATABASE for a10;
col LOGMINER_ID for 9999999;
col REQUIRED_CHECKPOINTSCN for a30;
col STATUS_CHANGE_TIME for a15;
col ERROR_NUMBER for a15;
col ERROR_MESSAGE for a10;
col CAPTURE_TYPE for a10;
col START_TIME for a30

SELECT CAPTURE_NAME, QUEUE_NAME, START_SCN, STATUS,
CAPTURED_SCN, APPLIED_SCN, SOURCE_DATABASE,
LOGMINER_ID, REQUIRED_CHECKPOINT_SCN,
STATUS_CHANGE_TIME, ERROR_NUMBER, ERROR_MESSAGE,
CAPTURE_TYPE, START_TIME
FROM DBA_CAPTURE;

> dynamic statistics from the GoldenGate views 
col state for a30;
SELECT sid, serial#, capture#, CAPTURE_NAME, STARTUP_TIME, CAPTURE_TIME,
state, SGA_USED, BYTES_OF_REDO_MINED,
to_char(STATE_CHANGED_TIME, 'mm-dd-yy hh24:mi') STATE_CHANGED_TIME
FROM V$GOLDENGATE_CAPTURE;

>>>>

col capture_message_create_time for a30;
col enqueue_message_create_time for a27;
col available_message_create_time for a30;

SELECT capture_name,
to_char(capture_time, 'mm-dd-yy hh24:mi') capture_time,
capture_message_number,
to_char(capture_message_create_time ,'mm-dd-yy hh24:mi') capture_message_create_time,
to_char(enqueue_time,'mm-dd-yy hh24:mi') enqueue_time,
enqueue_message_number,
to_char(enqueue_message_create_time, 'mm-dd-yy hh24:mi') enqueue_message_create_time,
available_message_number,
to_char(available_message_create_time,'mm-dd-yy hh24:mi') available_message_create_time
FROM GV$GOLDENGATE_CAPTURE;

>>>

Monitor the number of open transactions and LCRs for each capture processes >

SELECT component_name capture_name, count(*) open_transactions,
sum(cumulative_message_count) LCRs
FROM GV$GOLDENGATE_TRANSACTION
WHERE component_type='CAPTURE'
group by component_name;

>>> >>> 

 LogMiner views for LogMiner sessions and statistics > 

col db_name for a15;

select INST_ID, SESSION_ID,SESSION_NAME,SESSION_STATE, DB_NAME,
NUM_PROCESS,START_SCN,END_SCN,SPILL_SCN, PROCESSED_SCN, PREPARED_SCN,
READ_SCN MAX_MEMORY_SIZE,USED_MEMORY_SIZE PINNED_TXN, PINNED_COMMITTED_TXN
from GV$LOGMNR_SESSION;
SELECT SESSION_ID, NAME, VALUE
FROM V$LOGMNR_STATS;

>>
Integrated Capture >

SQL> select CAPTURE_NAME, QUEUE_NAME, STATUS from DBA_CAPTURE;
SQL> select OWNER, QUEUE_TABLE, QUEUE_TYPE from dba_queues where NAME='OGG$Q_TESTEXT1';


Integrated Replicat >

SQL> select REPLICAT_NAME,SERVER_NAME from DBA_GOLDENGATE_INBOUND;
SQL> select APPLY_NAME, QUEUE_NAME, status from dba_apply;
SQL> select apply_name,state from V$GG_APPLY_COORDINATOR ;


—Oracle In-Memory—

Thursday, November 17, 2016

Keygen # GoldenGate Password Encryption

Advantages # 
password protection 
Trails can be encrypted 

> PROCEDURE to implement > 
using "keygen" utility, we can generate hex keys 
after Enter the hex keys in ASCII file in goldengate home directory and encrypt the password 

Step 1# create hex keys using keygen utility # keygen utility exists in goldengate home; key length is max 256 bits 
Syntax # ./keygen <key_length> <Number of keys> 

[oracle@tnc1 home]$ ./keygen 128 4
0x05A5A84484FA4824B0BF2528ADC0E461
0xD392635258E8AC4D0C417D054988F659
0xA1801E602CD6107767C2D462E54F0852
0x6F6ED96D01C47420C3432C4081171A4A

Step 2 # Enter all 4 keys in ASCII file ENKEYS and < Name key value with key Name as follows >

[oracle@tnc1 home]$touch ENCKEYS 

[oracle@tnc1 home]$ cat ENCKEYS 
# Key name      key Value 
key128_1        0x05A5A84484FA4824B0BF2528ADC0E461
key128_2        0xD392635258E8AC4D0C417D054988F659
key128_3        0xA1801E602CD6107767C2D462E54F0852
key128_4        0x6F6ED96D01C47420C3432C4081171A4A

Usually we encrypt the ggsuser password with <encrypt_type>
encrypt_type = AES128, AES192,  AES256
Syntax # 
GGSCI> encrypt password <GOLDENGATE Password> <encrypt_type> ENCRYPTKEY <keyname> 

GGSCI (tnc1.localdomain) 2> encrypt password oracle aes128 encryptkey key128_2
Encrypted password:  AADAAAAAAAAAAAGAKHGAGDRIUAZDVENBDIJHSCIEVDFJCAFJUBXIQDEDXHLHIBHJPICCJILGSEOATIJJ
Algorithm used:  AES128

Testing database login from goldengate command line utility # ggsci >
GGSCI (tnc1.localdomain) 5> dblogin userid ggsuser password AADAAAAAAAAAAAGAKHGAGDRIUAZDVENBDIJHSCIEVDFJCAFJUBXIQDEDXHLHIBHJPICCJILGSEOATIJJ, encryptkey key128_2
Successfully logged into database.

STEP3 # copy ENKEYS file to source and Target goldengate homes  # 

--Nikhil Tatineni--
--Oracle in memory---



finding Culprit SQL in RAC cluster

Following Query’ Tells about > SQL Query's in RAC which are waiting to get CPU (or) Query is waiting on CPU > 
Depend on result of following query 
we will dive in and find set of sql’s which is waiting to get CPU 
or which is waiting on CPU :) 

SQL> select * from (
select
SQL_ID ,
sum(decode(session_state,'ON CPU',1,0)) as CPU,
sum(decode(session_state,'WAITING',1,0)) - 
sum(decode(session_state,'WAITING', decode(wait_class, 'User I/O',1,0),
0)) as WAIT,
sum(decode(session_state,'WAITING', decode(wait_class, 'User 
I/O',1,0),0)) as IO,
sum(decode(session_state,'ON CPU',1,1)) as TOTAL
from v$active_session_history
where SQL_ID is not NULL
group by sql_id
order by sum(decode(session_state,'ON CPU',1,1))
)
where rownum <11;

following .......
Run following Query to get wait events on database and match Query is waiting on CPU 
SQL> SELECT DISTINCT wait_class FROM gv$event_name ORDER BY 1;

Get wait_class from gv$event_name & replace wait_class and find out culprit SQL on database >
SQL> SELECT sql_id, COUNT(*)
FROM gv$active_session_history ash, v$event_name evt
WHERE ash.sample_time > SYSDATE - 3/24
AND ash.session_state = 'WAITING'
AND ash.event_id = evt.event_id
AND evt.wait_class = 'System I/O'
GROUP BY sql_id
ORDER BY COUNT(*) DESC;

From gv$active_session_history" we will get sql_id and helps to check execution plan of sql statement :) and Investigate further to decrease waits on cluster 

SQL> SELECT * FROM TABLE(dbms_xplan.display_cursor('xxxxxxxxxxxx'));

---Nikhil Tatineni---

---Oracle Database Admin --- 

Thursday, November 10, 2016

Remove DDL replication # GoldenGate

Removing DDL replication from GoldenGate >
[oracle@red goldengate]$ sqlplus / as sysdba
SQL*Plus: Release 12.1.0.2.0 Production on Wed Nov 2 22:16:31 2016
Copyright (c) 1982, 2014, Oracle.  All rights reserved.
Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Oracle Label Security, OLAP, Advanced Analytics
and Real Application Testing options

SQL> @ddl_disable
Trigger altered.

SQL> @ddl_remove.sql
DDL replication removal script.
WARNING: this script removes all DDL replication objects and data.
You will be prompted for the name of a schema for the Oracle GoldenGate database objects.
NOTE: The schema must be created prior to running this script.

Enter Oracle GoldenGate schema name:ggsuser
Working, please wait ...
Spooling to file ddl_remove_spool.txt
Script complete.

SQL> @marker_remove.sql
Marker removal script.
WARNING: this script removes all marker objects and data.
You will be prompted for the name of a schema for the Oracle GoldenGate database objects.
NOTE: The schema must be created prior to running this script.

Enter Oracle GoldenGate schema name:ggsuser
PL/SQL procedure successfully completed.
Sequence dropped.
Table dropped.
Script complete.

DDL replication  dropped successfully <<


---Nikhil Tatineni--
--GoldenGate--

Replicat struck on trail # goldengate

Scenario # Replicat struck on trail  and data is out of sync with source and target 
> Root cause > Replicat unable to perform " ETROLLOVER "

GGSCI (tnc0624) 5> info REP1
REPLICAT   REP1  Last Started 2016-11-06 08:07   Status RUNNING
Checkpoint Lag       00:00:00 (updated 00:00:04 ago)
Log Read Checkpoint  File /opt/app/ggs01/ggs/11.2.1.0.3/trails/m8000026
                     2016-11-06 08:21:23.805171  RBA 1534

GGSCI (tnc0624) 6> stats REP2
Sending STATS request to REPLICAT REP2 ...
No active replication maps.

GGSCI (tnc0624) 7> stats REP1
Sending STATS request to REPLICAT REP1 ...
No active replication maps.


GGSCI (tnc0624) 8> sh ls -ltr /opt/app/ggs01/ggs/11.2.1.0.3/trails/m8*
-rw-rw-rw- 1 ggs01g ggs01g 399999709 Nov  3 15:30 /opt/app/ggs01/ggs/11.2.1.0.3/trails/m8000018
-rw-rw-rw- 1 ggs01g ggs01g  19982829 Nov  4 04:46 /opt/app/ggs01/ggs/11.2.1.0.3/trails/m8000019
-rw-rw-rw- 1 ggs01g ggs01g 174809829 Nov  6 07:53 /opt/app/ggs01/ggs/11.2.1.0.3/trails/m8000020
-rw-rw-rw- 1 ggs01g ggs01g      1289 Nov  6 07:57 /opt/app/ggs01/ggs/11.2.1.0.3/trails/m8000021
-rw-rw-rw- 1 ggs01g ggs01g      1338 Nov  6 08:03 /opt/app/ggs01/ggs/11.2.1.0.3/trails/m8000022
-rw-rw-rw- 1 ggs01g ggs01g      1387 Nov  6 08:09 /opt/app/ggs01/ggs/11.2.1.0.3/trails/m8000023
-rw-rw-rw- 1 ggs01g ggs01g      1436 Nov  6 08:15 /opt/app/ggs01/ggs/11.2.1.0.3/trails/m8000024
-rw-rw-rw- 1 ggs01g ggs01g      1485 Nov  6 08:21 /opt/app/ggs01/ggs/11.2.1.0.3/trails/m8000025
-rw-rw-rw- 1 ggs01g ggs01g      1534 Nov  6 08:21 /opt/app/ggs01/ggs/11.2.1.0.3/trails/m8000026
-rw-rw-rw- 1 ggs01g ggs01g   1119647 Nov  6 11:17 /opt/app/ggs01/ggs/11.2.1.0.3/trails/m8000027

Resolution # 
GGSCI (tnc0624) 8> info REP1
REPLICAT   REP1  Last Started 2016-11-06 08:07   Status RUNNING
Checkpoint Lag       00:00:00 (updated 00:00:04 ago)
Log Read Checkpoint  File /opt/app/ggs01/ggs/11.2.1.0.3/trails/m8000026
                     2016-11-06 08:21:23.805171  RBA 1534

-rw-rw-rw- 1 ggs01g ggs01g      1534 Nov  6 08:21 /opt/app/ggs01/ggs/11.2.1.0.3/trails/m8000026
-rw-rw-rw- 1 ggs01g ggs01g   1119647 Nov  6 11:17 /opt/app/ggs01/ggs/11.2.1.0.3/trails/m8000027

Here it is unable to rollover from trail sequence number 26 to 27 :) 
perform ETROLLOVER >

GGSCI (tnc0624) 9> stop REP1
GGSCI (tnc0624) 10> stop REP2

GGSCI (tnc0624) 11> alter REP1, extseqno 27 extrba 0 
GGSCI (tnc0624) 12> alter REP2, extseqno 27 extrba 0 

GGSCI (tnc0624) 13> start REP1

--Nikhil Tatineni--
--Oracle GoldenGate--

Thursday, October 13, 2016

Media Recovery terminated with error 28365 # TDE

>>> >>>> >>>> >>>
scenario # created Oracle Wallet <<
>> MRP stopped on standby # 

SQL> select * from v$dataguard_status where severity in ('Error','Fatal') and timestamp > (sysdate -1);
FACILITY SEVERITY   DEST_ID MESSAGE_NUM ERROR_CODE CAL TIMESTAMP
------------------------ ------------- ------------------------- --------
MESSAGE
-------------------------------------------------------------------------
Log Apply Services Error 0   4         28365 YES  13-OCT-16
MRP0: Background Media Recovery terminated with error 28365

SQL> select * from v$encryption_wallet;
WRL_TYPE  WRL_PARAMETER  STATUS
————————————————————————————————
file     /u02/encrypt/    CLOSED

Resolution # >>>> 

Steps to perform to resolve Issue on standby >>>
shutdown the standby database 
scp cwallet.sso & ewallet.p12 to standby server wallet location from primary 
Mount the database 
Start MRP process 

SQL> select open_mode from v$database;
OPEN_MODE
--------------------
MOUNTED

SQL> shut immediate;
ORA-01109: database not open
Database dismounted.
ORACLE instance shut down.

From Primary > scp cwallet.sso and ewallet.p12 
[oracle@dg1 encrypt]$ scp * oracle@192.168.56.104:/u02/encrypt/
oracle@192.168.56.104's password: 
cwallet.sso                                   100% 2923     2.9KB/s   00:00    
ewallet.p12                                   100% 2845     2.8KB/s   00:00 

SQL> startup mount;
ORACLE instance started.

Total System Global Area  835104768 bytes
Fixed Size     2257840 bytes
Variable Size   541068368 bytes
Database Buffers   289406976 bytes
Redo Buffers     2371584 bytes
Database mounted.

SQL> select * from v$encryption_wallet;
WRL_TYPE  WRL_PARAMETER  STATUS
————————————————
file     /u02/encrypt/    OPEN

> Start the MRP on standby< 
SQL> alter database recover managed standby database disconnect from session;
Database altered.

SQL> select process from v$managed_standby where process like 'MRP%';
PROCESS
---------
MRP0

>>> 

--Nikhil Tatineni--
--Dataguard-- 
Links Related to Datagaurd <
Applying logs with delay # Standby database #
http://www.oracleinmemory.com/2016/07/dataguard-delay-parameter.html

Physical Standby # Active Dataguard 11g
http://www.oracleinmemory.com/2016/07/physical-standby-active-dataguard-11g.html

Snapshot standby # 11g
http://www.oracleinmemory.com/2016/07/dataguard-converting-physical-standby.html

Troubleshooting #

Dataguard # oldest online log sequence 0
http://www.oracleinmemory.com/2016/07/dataguard-oldest-online-log-sequence-0.html

ARBx # Slave Process in ASM

DBA manage databases on ASM & manages ASM disk groups.  When we notice  the disk group holding database is filled, DBA will add Brand new disk to existing disk group. Here I want to elaborate the background process of ASM i.e (ARBx), How ASM will perform rebalancing operation? ASM allocate ARBx( Actual Rebalancing process ) to perform rebalancing operation inside the disk group. ARBx is responsible for data movement between disks inside the disk group. The number of rebalancing process allocated depend on ASM_POWER_LIMIT value. Based on ASM disk group compatibility, we can alter ASM_POWER_LIMIT
if disk group asm  compatibility >= 11.2.0.2 (  alter asm_power_limit from 0 to 1024) 
if disk group asm compatibility < 11.2.0.2 ( alter asm_power_limit from 0 to 11) 

Query to check ASM disk group compatibility using v$asm_diskgroup
col COMPATIBILITY form a10
SQL> col DATABASE_COMPATIBILITY form a10
SQL> col NAME form a20
SQL> select group_number, name,
compatibility, database_compatibility from v$asm_diskgroup;

 Regularly when we will add disk to the disk group by logging into SYSASM (or) using ASMCA 
sql> conn / as sysasm
sql> alter diskgroup stage add disk '/dev/asm-disk8','/dev/asm-disk9';
sql>alter diskgroup DISKGROUP rebalance power 5;
when you say 5, 5 actual rebalancing (ARBx) process are allocated inside the diskgroup to improve performance we can alter this number to higher value, where 0 will disable the Rebalancing process

Query performance of ASM_ Rebalancing operation can be monitored using "v$asm_operation"
 SQL> select group_number,operation,state,power,est_minutes from v$asm_operation;

Note > It is always recommended to add a disk to existing disk group during off hours, rebalancing operation will impact the performance of database 

Adding a disk to diskgroup > ASM

Querys to monitor RAC

following few  Query's will help to find out culprits-  Query to check long running transaction from last 8 hours  Col Sid Fo...