Wednesday, February 17, 2016

ERROR OGG-01454

Today GoldenGate Extract abended on Servers
During my initial investigation, goldengate error report says as follows 

2016-02-17 10:01:41  INFO    OGG-01055  Recovery initialization completed for target file /appl/local/trails/as000025, at RBA 1111.

2016-02-17 10:01:41  INFO    OGG-01478  Output file /appl/local/trails/as is using format RELEASE 11.2.
  SourceModule            : [er.extrout]
  SourceID                : [/scratch/aime1/adestore/views/aime1_staxj04/oggcore/OpenSys/src/app/er/extrout.c]
  SourceFunction          : [ext_file_open(extract_def *)]
  SourceLine              : [816]
2016-02-17 10:01:51  ERROR   OGG-01454  Unable to lock file "/appl/local/trails/as000025" (error 13, Permission denied).  Lock currently held by process id (PID) 36634658.
2016-02-17 10:01:51  ERROR   OGG-01668  PROCESS ABENDING.

Using Error report, I tried to grep for process which locked the trail on source trail directory 

ps -ef | grep 36634658


   ggs01 29491394 39256220   0 12:00:03  pts/1  0:00 grep 36634658


   ggs01 36634658 36503594   1 09:28:54      -  0:51 /usr/local/opt/oracle/ggs_admin/112106/extract PARAMFILE /appl/local/trails/gg_home/dirprm/extora.prm REPORTFILE /appl/local/trails/gg_home/dirrpt/extora.rpt PROCESSID EXTORA USESUBDIRS


some other facts where we can find similiar errors in environment
using (Doc ID 1338582.1) & (Doc ID 1576525.1) Metalink

1) Check space available on the directory
2) Check Resources available on Server / database 
3) Check Oracle Goldengate process is able to access all Process directories like dirdat, dirck and dirrpt 
4) Abnormally if any process is abended or stopped, the process id can conflict with another 
5) check trails used by goldengate process on servers
6) Goldengate files are corrupted 

In my scenario here I found, the same trail is used by another goldengate extract process running on goldengate home which created lock on the trail. This made extract abended on the server 

Resolution:
Removed the process which is associated with another extract and started the abended extract 


--Nikhil Tatineni--
--GoldenGate--




Querys to monitor RAC

following few  Query's will help to find out culprits-  Query to check long running transaction from last 8 hours  Col Sid Fo...