Saturday, September 10, 2016

ERROR OGG-01091 Unable to open file # Replicat Abended

Scenario # RNDELTA6 abended with Error # 
During New GoldenGate setup removed old trails manually from Remote Trail Directory # 

GGSCI (tnc61) 1> view report RNDELTA6
........
...

Source Context :
  SourceModule            : [er.api]
  SourceID                : [/scratch/aime1/adestore/views/aime1_adc4150256/oggcore/OpenSys/src/app/er/api.c]
  SourceFunction          : [XT_read]
  SourceLine              : [4145]
  ThreadBacktrace         : [9] elements                         : [/opt/app/t1c2d985/ggs/11.2.1.0.3/libgglog.so(CMessageContext::AddThreadContext()+0x1e) [0x7f8230dc270e]]  : [/opt/app/t1c2d985/ggs/11.2.1.0.3/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int,
 ...)+0x2cc) [0x7f8230dbb6ac]]    : [/opt/app/t1c2d985/ggs/11.2.1.0.3/libgglog.so(_MSG_ERR_FILE_OPEN_ERROR(CSourceContext*, char const*, CMessa
geFactory::MessageDisposition)+0x31) [0x7f8230da58dd] : [/opt/app/t1c2d985/ggs/11.2.1.0.3/replicat(XT_read(chkpt_context_t*, char*, __std_rec_hdr*, int*, int*, sho
rt, short, char*, short*, char*, short*, int, long*, char*)+0xd71) [0x4c80b1]   : [/opt/app/t1c2d985/ggs/11.2.1.0.3/replicat(READ_EXTRACT_RECORD(chkpt_context_t*, short, char*, __std_rec_hd
r*, int*, int*, long*, short, short, char*, short*, char*, short*, long*, char*)+0xe8) [0x4c8ff8]]
                          : [/opt/app/t1c2d985/ggs/11.2.1.0.3/replicat(process_extract_loop()+0x402) [0x538dc2]]
                          : [/opt/app/t1c2d985/ggs/11.2.1.0.3/replicat(main+0x732) [0x54cc02]]
                          : [/lib64/libc.so.6(__libc_start_main+0xfd) [0x3a6ca1ed5d]]
                          : [/opt/app/t1c2d985/ggs/11.2.1.0.3/replicat(__gxx_personality_v0+0x31a) [0x4c0c6a]]
2016-09-09 03:10:29  ERROR   OGG-01091  Unable to open file "/appl/tnc1/gghome/dirdat/bg000005" (error 2, No such file o
r directory).

During Initial Investigation, we found this issue from report file, trail with sequence number 5 doesn't exist physically. Validated the physical location of trails by querying from GGSCI command line utility 

GGSCI (tnc61) 3> sh ls -ltr /appl/tnc1/gghome/dirdat/bg*
-rw-rw-rw- 1 t1c2d985g t1c2d985g     0 Sep  7 14:32 /appl/tnc1/gghome/dirdat/bg000006
-rw-rw-rw- 1 t1c2d985g t1c2d985g     0 Sep  7 14:48 /appl/tnc1/gghome/dirdat/bg000007
-rw-rw-rw- 1 t1c2d985g t1c2d985g     0 Sep  7 14:49 /appl/tnc1/gghome/dirdat/bg000008
-rw-rw-rw- 1 t1c2d985g t1c2d985g 12198 Sep  9 03:09 /appl/tnc1/gghome/dirdat/bg000009

To Resolve this issue #
alter the replicat to next available sequence number on the trail directory #
GGSCI (tnc61) 4> alter RNDELTA6, extseqno 6 extrba 0
REPLICAT altered.

GGSCI (tnc61) 5> start RNDELTA6
Sending START request to MANAGER ...
REPLICAT RNDELTA6 starting

GGSCI (tnc61) 8> info RNDELTA6
REPLICAT   RNDELTA6  Last Started 2016-09-09 03:12   Status RUNNING
Checkpoint Lag       00:00:00 (updated 00:00:03 ago)
Log Read Checkpoint  File /appl/tnc1/gghome/dirdat/bg000009

                     2016-09-01 05:19:56.000685  RBA 12198

--Nikhil Tatineni--
--gg--

Querys to monitor RAC

following few  Query's will help to find out culprits-  Query to check long running transaction from last 8 hours  Col Sid Fo...