ERROR OGG-01031 There is a problem in network communication, a remote file problem,

Scenario # Brand New pump writing to Target server got abended with following Error # 

Database Language and Character Set:
NLS_LANG         = ".US7ASCII"
NLS_LANGUAGE     = "AMERICAN"
NLS_TERRITORY    = "AMERICA"
NLS_CHARACTERSET = "US7ASCII"

2016-09-01 10:24:30  INFO    OGG-01226  Socket buffer size set to 27985 (flush size 27985).

Source Context :
  SourceModule            : [er.extrout]
  SourceID                : [/scratch/aime1/adestore/views/aime1_adc4150256/oggcore/OpenSys/src/app/er/extrout.c]
  SourceFunction          : [complete_tcp_msg]
  SourceLine              : [1526]
  ThreadBacktrace         : [9] elements
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/libgglog.so(CMessageContext::AddThreadContext()+0x1e) [0x7f8ea9e4470e]]
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, ...)+0x2cc) [0x7f8ea9e3d6ac]]
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/libgglog.so(_MSG_ERR_ER_REMOTE_COMM_PROBLEM(CSourceContext*, char const*, CMessageFactory::MessageDisposition)+0x31) [0x7f8ea9e28789]]
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/extract(complete_tcp_msg(extract_def*)+0x421) [0x519049]]
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/extract(flush_tcp(extract_def*, int)+0x206) [0x5198f2]]
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/extract(RECOVERY_initialize()+0x371) [0x52ae51]]
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/extract(main+0x4a5) [0x573985]]
                          : [/lib64/libc.so.6(__libc_start_main+0xfd) [0x311801ed5d]]
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/extract(__gxx_personality_v0+0x38a) [0x4edaba]]

2016-09-01 10:24:30  ERROR   OGG-01031  There is a problem in network communication, a remote file problem, encryption keys for target and source do not match (if using ENCRYPT) or an unknown error. (Reply received is Output file /usr/local/opt/oracle/ggs/11.2.1.0.1/trails/gi000000 is not in any allowed output directories.).

2016-09-01 10:24:30  ERROR   OGG-01668  PROCESS ABENDING.

To find out Root Cause # 

workaround > Telnet fails, 
There is problem with network between servers (or)
Blocking port Issues on target server (or)
manager child process (collector) Problems # 

$ telnet tnc61.world.com 7809
Trying 192.168.56.212...
Connected to tnc1.world.com
Escape character is '^]'.
Connection closed by foreign host.

Here Telnet worked as expected. There is no issues with network or port Number # 

Later ...

Checked status of Manager # manager is up & running and  gathering port details 

if every thing looks good, you can confirm child process is up & running on manager end

GGSCI (tnc61) 19> SEND MANAGER GETPORTINFO DETAIL

Sending GETPORTINFO, request to MANAGER ...

Dynamic Port List

Starting Index 0
Entry Port  Error  Process     Assigned             Program
----- ----- ----- ----------   -------------------  -------
   0   7809     0
   1   7810     0
   2   7811     0
   3   7812     0
   4   7813     0
   5   7814     0
   6   7815     0
   7   7816     0
   8   7817     0
   9   7818     0
  10   7819     0
  11   7820     0
  12   7821     0
  13   7822     0
  14   7823     0
  15   7824     0
  16   7825     0
  17   7826     0
  18   7827     0
  19   7828     0
  20   7829     0
  21   7830     0
  22   7833     0
  23   7835     0

>> few messages from Targetgg_HOME/ggserr.log 

2016-08-31 09:55:02  WARNING OGG-01223  Oracle GoldenGate Collector for Oracle:  Output file /usr/local/opt/oracle/ggs/11.2.1.0.1/trails/gi000000 is not in any allowed output directories.
2016-08-31 09:55:02  INFO    OGG-01676  Oracle GoldenGate Collector for Oracle:  Terminating after client disconnect.
2016-08-31 09:58:15  INFO    OGG-00963  Oracle GoldenGate Manager for Oracle, mgr.prm:  Command received from EXTRACT on host [130.6.44.131]:39623 (START SERVER CPU -1 PRI -1  TIMEOUT 300 PARAMS ).
2016-08-31 09:58:15  INFO    OGG-00960  Oracle GoldenGate Manager for Oracle, mgr.prm:  Access granted (rule #4).
2016-08-31 09:58:15  INFO    OGG-01677  Oracle GoldenGate Collector for Oracle:  Waiting for connection (started dynamically).
2016-08-31 09:58:15  INFO    OGG-00963  Oracle GoldenGate Manager for Oracle, mgr.prm:  Command received from SERVER on host [127.0.0.1]:9264 (REPORT 1442204 7809).
2016-08-31 09:58:15  INFO    OGG-00960  Oracle GoldenGate Manager for Oracle, mgr.prm:  Access granted (rule #1).
2016-08-31 09:58:15  INFO    OGG-00974  Oracle GoldenGate Manager for Oracle, mgr.prm:  Manager started collector process (Port 7809).
2016-08-31 09:58:15  INFO    OGG-01228  Oracle GoldenGate Collector for Oracle:  Timeout in 300 seconds.
2016-08-31 09:58:20  INFO    OGG-01229  Oracle GoldenGate Collector for Oracle:  Connected to tnc1.world.com2016-08-31 09:58:20  WARNING OGG-01223  Oracle GoldenGate Collector for Oracle:  Output file /usr/local/opt/oracle/ggs/11.2.1.0.1/trails/gi000000 is not in any allowed output directories.
2016-08-31 09:58:20  INFO    OGG-01676  Oracle GoldenGate Collector for Oracle:  Terminating after client disconnect.

Later # 

Investigate file system, i.e report file says " Reply received is Output file /usr/local/opt/oracle/ggs/11.2.1.0.1/trails/gi000000 is not in any allowed output directories.)."  

check for free space available on Target server ( Target file system )

$  df -g /usr/local/opt/oracle/ggs/11.2.1.0.1/trails/
Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
/dev/ggs_lv       20.00     19.70    2%      215     1% /usr/local/opt/oracle/ggs

>> >>> 

Later I raised Oracle SR for this issue ..Followed (Doc ID 1288562.1) and resolved it

Resolution: # In order to fix this issue, 

Deleted existing pump & 

Changed Remote trail directory and granted required permissions on remote trail directory

Recreated pump and restarted the brand NEW PUMP #

Reference # 

OGG Data Pumps Error With OGG-01031 When Upgrading the Target Side (Doc ID 1288562.1)

--Nikhil Tatineni--

--GoldenGate--