Sunday, February 1, 2015

ERROR OGG-01232 Receive TCP params error: TCP/IP error 73

Scenario:
we have compliant from application team, data is not getting replicated for one of the critical table to target instances,  During initial investigation,  we found  Ogg pump is abended  on source side and also we found following error in report file, ogg monitoring is disabled on servers


we found pump is abended on source side,


EXTRACT     ABENDED     PEMPRI     00:00:00      09:26:59

GGSCI> view report PEMPRI

2014-10-05 10:38:06  INFO    OGG-01226  Socket buffer size set to 27985 (flush size 27985).
Source Context :  SourceModule            : [ggnet.tpcrtn]
  SourceID                : [/scratch/aime1/adestore/views/aime1_staxj04/oggcore/OpenSys/src/gglib/ggnet/tcprtn.c]
  SourceFunction          : [opt_settle(int, char *, int32_t, char *)]
  SourceLine              : [3129]
2014-10-05 10:43:06  ERROR   OGG-01232  Receive TCP params error: TCP/IP error 73 (Connection reset by peer), endpoint:sfo.com.
2014-10-05 10:43:06  ERROR   OGG-01668  PROCESS ABENDING.

Resolution
check goldengate manager process  is  running on target  goldengate home / server

GGSCI> Info mgr
  • If the manager is not running on target goldengate home, start the manager and start abended pump on source side
  • If the manager is running, still the pump is abended state, i recommend you to bounce the associated replicat on target goldengate home. Replicat process is stale and can hold lock on the  remote trail 
  • If still pump is not coming up check RBA of the pump with size of the local trail on source server

GGSCI (sfo) 19> info PEMPRI
EXTRACT    PEMPRI   Last Started 2014-10-05 10:38   Status ABENDED
Checkpoint Lag       00:00:00 (updated 09:18:45 ago)
Log Read Checkpoint  File /u02/app/ggs/trails/ba000083
                     2014-10-05 01:18:48.000000  RBA 447538549

GGSCI (sfo) 20> exit
ggs01@sfo> cd /u02/app/ggs/trails/
ggs01@
sfo> ls -ltr ba*


-rw-r-----    1 ggs01    ggsoper   447538549 Oct 05 01:45 ba000083
-rw-r-----    1 ggs01    ggsoper    10543247 Oct 05 10:38 ba000084

when you compared the size of the trail and RBA is same, we can conclude that pump is unable to perform ETROLLOVER, Now you can forcefully ETROLLOVER the pump to next trail 

GGSCI> alter PEMPRI, ETROLLOVER

GGSCI (sfo) 19> info PEMPRI
EXTRACT    PEMPRI   Last Started 2014-10-05 10:38   Status RUNNING
Checkpoint Lag       00:00:00 (updated 09:18:45 ago)
Log Read Checkpoint  File /u02/app/ggs/trails/ba000084
                     2014-10-05 01:18:48.000000  RBA 1168

Now we can see, pump is up and running, and we can see  RBA "RBA 1168" is increasing !
! -- Oracle In-memory !!!

Querys to monitor RAC

following few  Query's will help to find out culprits-  Query to check long running transaction from last 8 hours  Col Sid Fo...