Saturday, June 25, 2016

ERROR OGG-00519 Fatal error executing DDL replication:

scenario # I am trying to drop tablespace on source database, Enabled DDL replication on source table, replicat abending with following error.  Tried with disabling trigger  result is same # Ufff # 

Connected to:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
With the Partitioning, Oracle Label Security, OLAP, Advanced Analytics
and Real Application Testing options
SQL> @ddl_disable
Trigger altered.

SQL> select TABLESPACE_NAME from dba_tablespaces;
TABLESPACE_NAME
------------------------------
SYSTEM
SYSAUX
UNDOTBS1
TEMP
USERS
STAGE
GGS_STAGE
HARRY
ADMIN
FIX
FIX01
11 rows selected.

SQL> drop tablespace fix including contents;
Tablespace dropped.

GGSCI (red.gg.ora.com) 5> info all
Program     Status      Group       Lag at Chkpt  Time Since Chkpt
MANAGER     RUNNING                                           
EXTRACT     RUNNING     EDELTA      00:00:10      00:00:05    
REPLICAT    ABENDED     RMOTS       00:11:15      00:00:10   

GGSCI (red.gg.ora.com) 6> view report rmots
***********************************************************************
                 Oracle GoldenGate Delivery for Oracle
 Version 12.1.2.0.0 17185003 OGGCORE_12.1.2.0.0_PLATFORMS_130924.1316_FBO
   Linux, x64, 64bit (optimized), Oracle 12c on Sep 25 2013 02:54:11
Copyright (C) 1995, 2013, Oracle and/or its affiliates. All rights reserved.
                    Starting at 2016-06-25 00:25:20
***********************************************************************

Operating System Version:
Linux
Version #1 SMP Sat Feb 25 12:50:01 EST 2012, Release 2.6.18-308.el5xen
Node: red.gg.ora.com
Machine: x86_64
                         soft limit   hard limit
Address Space Size   :    unlimited    unlimited
Heap Size            :    unlimited    unlimited
File Size            :    unlimited    unlimited
CPU Time             :    unlimited    unlimited

Process id: 6448

Description: 

***********************************************************************
**            Running with the following parameters                  **
***********************************************************************

2016-06-25 00:25:20  INFO    OGG-03059  Operating system character set identified as UTF-8.

2016-06-25 00:25:20  INFO    OGG-02695  ANSI SQL parameter syntax is used for parameter parsing.
replicat rmots
SETENV (ORACLE_SID=mldb69)

2016-06-25 00:25:20  INFO    OGG-02095  Successfully set environment variable ORACLE_SID=mldb69.
SETENV (ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1)

2016-06-25 00:25:20  INFO    OGG-02095  Successfully set environment variable ORACLE_HOME=/u01/app/
oracle/product/12.1.0/dbhome_1.
userid ggsuser@mldb69, password ******
assumetargetdefs
reportcount every 24 hours, rate
DBOPTIONS INTEGRATEDPARAMS(parallelism 6)
discardfile /u01/app/goldengate/dirdsc/rmots.dsc, megabytes 100, append
discardrollover at 21:00
reperror (default, discard)
map delta.*, target mots.*;

2016-06-25 00:25:20  INFO    OGG-06451  Triggers will be suppressed by default.

2016-06-25 00:25:20  INFO    OGG-01971  The previous message, 'INFO OGG-06451', repeated 1 times.

2016-06-25 00:25:20  INFO    OGG-01815  Virtual Memory Facilities for: COM
    anon alloc: mmap(MAP_ANON)  anon free: munmap
    file alloc: mmap(MAP_SHARED)  file free: munmap
    target directories:
    /u01/app/goldengate/dirtmp.

CACHEMGR virtual memory values (may have been adjusted)
CACHESIZE:                                2G
CACHEPAGEOUTSIZE (default):               8M
PROCESS VM AVAIL FROM OS (min):           4G
CACHESIZEMAX (strict force to disk):   3.41G

Database Version:
Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 - 64bit Production
PL/SQL Release 12.1.0.2.0 - Production
CORE 12.1.0.2.0 Production
TNS for Linux: Version 12.1.0.2.0 - Production
NLSRTL Version 12.1.0.2.0 - Production

Database Language and Character Set:
NLS_LANGUAGE     = "AMERICAN" 
NLS_TERRITORY    = "AMERICA" 
NLS_CHARACTERSET = "WE8MSWIN1252" 
Opened trail file /u01/app/goldengate/dirdat/aa000009 at 2016-06-25 00:25:20

2016-06-25 00:25:20  INFO    OGG-03522  Setting session time zone to source database time zone 'GMT'
2016-06-25 00:25:20  INFO    OGG-01014  Positioning with begin time: Jun 25, 2016 12:18:44 AM, starting record time: Jun 25, 2016 12:24:38 AM at extseqno 9, extrba 20569.
2016-06-25 00:25:20  INFO    OGG-03506  The source database character set, as determined from the trail file, is we8mswin1252.
2016-06-25 00:25:20  INFO    OGG-02527  Integrated Replicat does not populate a trace table.
2016-06-25 00:25:20  INFO    OGG-02545  Parameter GROUPTRANSOPS is ignored by Integrated Replicat when parallelism is greater than 1.
2016-06-25 00:25:31  INFO    OGG-02530  Integrated replicat successfully attached to inbound server OGG$RMOTS.
***********************************************************************
**                     Run Time Messages                             **
***********************************************************************
Opened trail file /u01/app/goldengate/dirdat/aa000009 at 2016-06-25 00:25:32
2016-06-25 00:25:53  INFO    OGG-02529  Successfully unregistered REPLICAT RMOTS inbound server OGG$RMOTS from database.
2016-06-25 00:25:53  INFO    OGG-02525  Integrated Replicat has been unregistered.
2016-06-25 00:25:53  INFO    OGG-01407  Setting current schema for DDL operation to [SYS].
2016-06-25 00:25:53  INFO    OGG-01408  Restoring current schema for DDL operation to [ggsuser].
2016-06-25 00:25:53  INFO    OGG-01407  Setting current schema for DDL operation to [SYS].
2016-06-25 00:25:54  INFO    OGG-01408  Restoring current schema for DDL operation to [ggsuser].
2016-06-25 00:29:53  INFO    OGG-01407  Setting current schema for DDL operation to [MOTS].
2016-06-25 00:29:53  INFO    OGG-01408  Restoring current schema for DDL operation to [ggsuser].
2016-06-25 00:34:11  INFO    OGG-01407  Setting current schema for DDL operation to [MOTS].
2016-06-25 00:34:11  INFO    OGG-01408  Restoring current schema for DDL operation to [ggsuser].
2016-06-25 00:34:25  INFO    OGG-01407  Setting current schema for DDL operation to [MOTS].
2016-06-25 00:34:25  INFO    OGG-01408  Restoring current schema for DDL operation to [ggsuser].
2016-06-25 00:34:45  INFO    OGG-01407  Setting current schema for DDL operation to [MOTS].
2016-06-25 00:34:45  INFO    OGG-01408  Restoring current schema for DDL operation to [ggsuser].
2016-06-25 00:45:51  INFO    OGG-01407  Setting current schema for DDL operation to [SYS].
Source Context :
  SourceModule            : [ggapp.ddl]
  SourceID                : [/scratch/aime1/adestore/views/aime1_adc4150472/oggcore/OpenSys/src/ggl
ib/ggapp/ddlrep.c]
  SourceFunction          : [DDLREP_handleDDLError]
  SourceLine              : [857]
  ThreadBacktrace         : [15] elements
                          : [/u01/app/goldengate/libgglog.so(CMessageContext::AddThreadContext()+0x
1e) [0x2b2ef3dff7ee]]
                          : [/u01/app/goldengate/libgglog.so(CMessageFactory::CreateMessage(CSource
Context*, unsigned int, ...)+0x321) [0x2b2ef3dfab11]]
                          : [/u01/app/goldengate/libgglog.so(_MSG_ERR_DDL_ABEND_NO_HANDLER(CSourceC
ontext*, char const*, CMessageFactory::MessageDisposition)+0x31) [0x2b2ef3de533f]]
                          : [/u01/app/goldengate/replicat(DDLREP_handleDDLError(ggs::gglib::ggunico
de::UString const&, ggs::gglib::ggunicode::UString const&, ggs::gglib::ggunicode::UString const&, g
gs::gglib::ggunicode::UString&, int, RepConfig_t*, ggs::gglib::ggunicode::UString const&, ggs::ggli
b::ggunicode::UString const&, ggs::gglib::ggunicode::UString const&, ggs::gglib::ggapp::CDBObjName<
(DBObjType)8>&, ggs::gglib::ggapp::CDBObjName<(DBObjType)7>&, ggs::gglib::ggapp::CDBObjName<(DBObjT
ype)1>&, long*)+0xd4a) [0x5fb53a]]
                          : [/u01/app/goldengate/replicat [0x5fefc4]]
                          : [/u01/app/goldengate/replicat(DDLREP_process(char*, __std_rec_hdr*, uns
igned short)+0x1002) [0x603898]]
                          : [/u01/app/goldengate/replicat(replicate_io(file_def*, __std_rec_hdr*, c
har*, extr_ptr_def*, int*, int)+0x33dd) [0x59cc2d]]
                          : [/u01/app/goldengate/replicat(process_extract_loop()+0x31b7) [0x5972a7]
]
                          : [/u01/app/goldengate/replicat(replicat_main(int, char**)+0x5bc) [0x5bfd
ac]]
                          : [/u01/app/goldengate/replicat(ggs::gglib::MultiThreading::MainThread::E
xecMain()+0x6f) [0x68156f]]
                          : [/u01/app/goldengate/replicat(ggs::gglib::MultiThreading::Thread::RunTh
read(ggs::gglib::MultiThreading::Thread::ThreadArgs*)+0x11a) [0x68041a]]
                          : [/u01/app/goldengate/replicat(ggs::gglib::MultiThreading::MainThread::R
un(int, char**)+0x128) [0x681918]]
                          : [/u01/app/goldengate/replicat(main+0x3f) [0x5be84f]]
                          : [/lib64/libc.so.6(__libc_start_main+0xf4) [0x3d07e1d994]]
                          : [/u01/app/goldengate/replicat [0x508339]]
2016-06-25 00:45:51  ERROR   OGG-00519  Fatal error executing DDL replication: error [Error code [9
59], ORA-00959: tablespace 'FIX' does not exist SQL drop tablespace fix including contents /* GOLDE
NGATE_DDL_REPLICATION */], no error handler present.
***********************************************************************
*                   ** Run Time Statistics **                         *
***********************************************************************
Last record for the last committed transaction is the following: 
___________________________________________________________________

Resolution # Working On it # 

Need Suggestions.. 

--Nikhil Tatineni--
--GoldeNgate --



Querys to monitor RAC

following few  Query's will help to find out culprits-  Query to check long running transaction from last 8 hours  Col Sid Fo...