ERROR OGG-01091 Unable to open file # Replicat Abended

Scenario # RNDELTA6 abended with Error # 

During New GoldenGate setup removed old trails manually from Remote Trail Directory # 

GGSCI (tnc61) 1> view report RNDELTA6

........

...

Source Context :

  SourceModule            : [er.api]

  SourceID                : [/scratch/aime1/adestore/views/aime1_adc4150256/oggcore/OpenSys/src/app/er/api.c]

  SourceFunction          : [XT_read]

  SourceLine              : [4145]

  ThreadBacktrace         : [9] elements                         : [/opt/app/t1c2d985/ggs/11.2.1.0.3/libgglog.so(CMessageContext::AddThreadContext()+0x1e) [0x7f8230dc270e]]  : [/opt/app/t1c2d985/ggs/11.2.1.0.3/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int,

 ...)+0x2cc) [0x7f8230dbb6ac]]    : [/opt/app/t1c2d985/ggs/11.2.1.0.3/libgglog.so(_MSG_ERR_FILE_OPEN_ERROR(CSourceContext*, char const*, CMessa

geFactory::MessageDisposition)+0x31) [0x7f8230da58dd] : [/opt/app/t1c2d985/ggs/11.2.1.0.3/replicat(XT_read(chkpt_context_t*, char*, __std_rec_hdr*, int*, int*, sho

rt, short, char*, short*, char*, short*, int, long*, char*)+0xd71) [0x4c80b1]   : [/opt/app/t1c2d985/ggs/11.2.1.0.3/replicat(READ_EXTRACT_RECORD(chkpt_context_t*, short, char*, __std_rec_hd

r*, int*, int*, long*, short, short, char*, short*, char*, short*, long*, char*)+0xe8) [0x4c8ff8]]

                          : [/opt/app/t1c2d985/ggs/11.2.1.0.3/replicat(process_extract_loop()+0x402) [0x538dc2]]

                          : [/opt/app/t1c2d985/ggs/11.2.1.0.3/replicat(main+0x732) [0x54cc02]]

                          : [/lib64/libc.so.6(__libc_start_main+0xfd) [0x3a6ca1ed5d]]

                          : [/opt/app/t1c2d985/ggs/11.2.1.0.3/replicat(__gxx_personality_v0+0x31a) [0x4c0c6a]]

2016-09-09 03:10:29  ERROR   OGG-01091  Unable to open file "/appl/tnc1/gghome/dirdat/bg000005" (error 2, No such file o

r directory).

During Initial Investigation, we found this issue from report file, trail with sequence number 5 doesn't exist physically. Validated the physical location of trails by querying from GGSCI command line utility 

GGSCI (tnc61) 3> sh ls -ltr /appl/tnc1/gghome/dirdat/bg*

-rw-rw-rw- 1 t1c2d985g t1c2d985g     0 Sep  7 14:32 /appl/tnc1/gghome/dirdat/bg000006

-rw-rw-rw- 1 t1c2d985g t1c2d985g     0 Sep  7 14:48 /appl/tnc1/gghome/dirdat/bg000007

-rw-rw-rw- 1 t1c2d985g t1c2d985g     0 Sep  7 14:49 /appl/tnc1/gghome/dirdat/bg000008

-rw-rw-rw- 1 t1c2d985g t1c2d985g 12198 Sep  9 03:09 /appl/tnc1/gghome/dirdat/bg000009

To Resolve this issue #

alter the replicat to next available sequence number on the trail directory #

GGSCI (tnc61) 4> alter RNDELTA6, extseqno 6 extrba 0

REPLICAT altered.

GGSCI (tnc61) 5> start RNDELTA6

Sending START request to MANAGER ...

REPLICAT RNDELTA6 starting

GGSCI (tnc61) 8> info RNDELTA6

REPLICAT   RNDELTA6  Last Started 2016-09-09 03:12   Status RUNNING

Checkpoint Lag       00:00:00 (updated 00:00:03 ago)

Log Read Checkpoint  File /appl/tnc1/gghome/dirdat/bg000009

                     2016-09-01 05:19:56.000685  RBA 12198

--Nikhil Tatineni--

--gg--

ERROR OGG-01031 There is a problem in network communication, a remote file problem,

Scenario # Brand New pump writing to Target server got abended with following Error # 

Database Language and Character Set:
NLS_LANG         = ".US7ASCII"
NLS_LANGUAGE     = "AMERICAN"
NLS_TERRITORY    = "AMERICA"
NLS_CHARACTERSET = "US7ASCII"

2016-09-01 10:24:30  INFO    OGG-01226  Socket buffer size set to 27985 (flush size 27985).

Source Context :
  SourceModule            : [er.extrout]
  SourceID                : [/scratch/aime1/adestore/views/aime1_adc4150256/oggcore/OpenSys/src/app/er/extrout.c]
  SourceFunction          : [complete_tcp_msg]
  SourceLine              : [1526]
  ThreadBacktrace         : [9] elements
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/libgglog.so(CMessageContext::AddThreadContext()+0x1e) [0x7f8ea9e4470e]]
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, ...)+0x2cc) [0x7f8ea9e3d6ac]]
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/libgglog.so(_MSG_ERR_ER_REMOTE_COMM_PROBLEM(CSourceContext*, char const*, CMessageFactory::MessageDisposition)+0x31) [0x7f8ea9e28789]]
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/extract(complete_tcp_msg(extract_def*)+0x421) [0x519049]]
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/extract(flush_tcp(extract_def*, int)+0x206) [0x5198f2]]
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/extract(RECOVERY_initialize()+0x371) [0x52ae51]]
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/extract(main+0x4a5) [0x573985]]
                          : [/lib64/libc.so.6(__libc_start_main+0xfd) [0x311801ed5d]]
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/extract(__gxx_personality_v0+0x38a) [0x4edaba]]

2016-09-01 10:24:30  ERROR   OGG-01031  There is a problem in network communication, a remote file problem, encryption keys for target and source do not match (if using ENCRYPT) or an unknown error. (Reply received is Output file /usr/local/opt/oracle/ggs/11.2.1.0.1/trails/gi000000 is not in any allowed output directories.).

2016-09-01 10:24:30  ERROR   OGG-01668  PROCESS ABENDING.

To find out Root Cause # 

workaround > Telnet fails, 
There is problem with network between servers (or)
Blocking port Issues on target server (or)
manager child process (collector) Problems # 

$ telnet tnc61.world.com 7809
Trying 192.168.56.212...
Connected to tnc1.world.com
Escape character is '^]'.
Connection closed by foreign host.

Here Telnet worked as expected. There is no issues with network or port Number # 

Later ...

Checked status of Manager # manager is up & running and  gathering port details 

if every thing looks good, you can confirm child process is up & running on manager end

GGSCI (tnc61) 19> SEND MANAGER GETPORTINFO DETAIL

Sending GETPORTINFO, request to MANAGER ...

Dynamic Port List

Starting Index 0
Entry Port  Error  Process     Assigned             Program
----- ----- ----- ----------   -------------------  -------
   0   7809     0
   1   7810     0
   2   7811     0
   3   7812     0
   4   7813     0
   5   7814     0
   6   7815     0
   7   7816     0
   8   7817     0
   9   7818     0
  10   7819     0
  11   7820     0
  12   7821     0
  13   7822     0
  14   7823     0
  15   7824     0
  16   7825     0
  17   7826     0
  18   7827     0
  19   7828     0
  20   7829     0
  21   7830     0
  22   7833     0
  23   7835     0

>> few messages from Targetgg_HOME/ggserr.log 

2016-08-31 09:55:02  WARNING OGG-01223  Oracle GoldenGate Collector for Oracle:  Output file /usr/local/opt/oracle/ggs/11.2.1.0.1/trails/gi000000 is not in any allowed output directories.
2016-08-31 09:55:02  INFO    OGG-01676  Oracle GoldenGate Collector for Oracle:  Terminating after client disconnect.
2016-08-31 09:58:15  INFO    OGG-00963  Oracle GoldenGate Manager for Oracle, mgr.prm:  Command received from EXTRACT on host [130.6.44.131]:39623 (START SERVER CPU -1 PRI -1  TIMEOUT 300 PARAMS ).
2016-08-31 09:58:15  INFO    OGG-00960  Oracle GoldenGate Manager for Oracle, mgr.prm:  Access granted (rule #4).
2016-08-31 09:58:15  INFO    OGG-01677  Oracle GoldenGate Collector for Oracle:  Waiting for connection (started dynamically).
2016-08-31 09:58:15  INFO    OGG-00963  Oracle GoldenGate Manager for Oracle, mgr.prm:  Command received from SERVER on host [127.0.0.1]:9264 (REPORT 1442204 7809).
2016-08-31 09:58:15  INFO    OGG-00960  Oracle GoldenGate Manager for Oracle, mgr.prm:  Access granted (rule #1).
2016-08-31 09:58:15  INFO    OGG-00974  Oracle GoldenGate Manager for Oracle, mgr.prm:  Manager started collector process (Port 7809).
2016-08-31 09:58:15  INFO    OGG-01228  Oracle GoldenGate Collector for Oracle:  Timeout in 300 seconds.
2016-08-31 09:58:20  INFO    OGG-01229  Oracle GoldenGate Collector for Oracle:  Connected to tnc1.world.com2016-08-31 09:58:20  WARNING OGG-01223  Oracle GoldenGate Collector for Oracle:  Output file /usr/local/opt/oracle/ggs/11.2.1.0.1/trails/gi000000 is not in any allowed output directories.
2016-08-31 09:58:20  INFO    OGG-01676  Oracle GoldenGate Collector for Oracle:  Terminating after client disconnect.

Later # 

Investigate file system, i.e report file says " Reply received is Output file /usr/local/opt/oracle/ggs/11.2.1.0.1/trails/gi000000 is not in any allowed output directories.)."  

check for free space available on Target server ( Target file system )

$  df -g /usr/local/opt/oracle/ggs/11.2.1.0.1/trails/
Filesystem    GB blocks      Free %Used    Iused %Iused Mounted on
/dev/ggs_lv       20.00     19.70    2%      215     1% /usr/local/opt/oracle/ggs

>> >>> 

Later I raised Oracle SR for this issue ..Followed (Doc ID 1288562.1) and resolved it

Resolution: # In order to fix this issue, 

Deleted existing pump & 

Changed Remote trail directory and granted required permissions on remote trail directory

Recreated pump and restarted the brand NEW PUMP #

Reference # 

OGG Data Pumps Error With OGG-01031 When Upgrading the Target Side (Doc ID 1288562.1)

--Nikhil Tatineni--

--GoldenGate--

ERROR OGG-00664 OCI Error during OCIServerAttach (status = 12162-ORA-12162:

Scenario # Rebooted golden gate process, Pump Abended with following error <

Investigated  Issue and found root cause # 

ggsci> View report plct90

.....

....

............

extract plct90

passthru

userid ggsuser, password *******

Source Context :

  SourceModule            : [ggdb.ora.sess]

  SourceID                : [/scratch/aime1/adestore/views/aime1_staxj04/oggcore

/OpenSys/src/gglib/ggdbora/ocisess.c]

  SourceFunction          : [OCISESS_context_def::oci_try(int, const char *, ...)]

  SourceLine              : [650]

2016-09-04 08:48:14  ERROR   OGG-00664  OCI Error during OCIServerAttach (status

 = 12162-ORA-12162: TNS:net service name is incorrectly specified)

Pump Param file looks like as follows # 

ggsci> view param plct90

extract plct90

userid ggsuser, password oracle

PASSTHRU

RMTHOST tnc61.ffdc.com, MGRPORT 7809

RMTTRAIL /u01/app/trails/aa

REPORTCOUNT EVERY 10 MINUTES, RATE

TABLE world.orders;

Resolution: New changes in parameter file and start pump 

ggsci> view param plct90

extract plct90

userid ggsuser@tns_name, password oracle

PASSTHRU

RMTHOST tnc61.ffdc.com, MGRPORT 7809

RMTTRAIL /u01/app/trails/aa

REPORTCOUNT EVERY 10 MINUTES, RATE

ggsci> start plct90 

--pump came up automatically, Issue is resolved --

--Nikhil Tatineni--

--gg--

Links Related to GoldenGate #

REPLICAT WITH HANDLECOLLISIONS

http://www.databaseinmemory.com/2015/01/replicat-with-handlecollisions.html

SPLITING OracleGoldenGate REPLICAT

http://www.databaseinmemory.com/2015/02/hey-guys-welcome-back.html

Goldengate Replicat is running very slow

http://www.databaseinmemory.com/2015/10/replicat-is-running-very-slow.html

Goldengate Replicat's blocking itself

http://www.databaseinmemory.com/2015/10/replicats-blocking-itself.html

Oracle GoldenGate: SKIP LONG RUNNING TRANSACTIONS

http://www.databaseinmemory.com/2015/03/ogg-skip-long-running-transactions.html

Goldengate Extract is not moving?

http://www.databaseinmemory.com/2015/02/extract-is-not-moving.html

ERROR OGG-00446  Missing filename opening checkpoint file

http://www.databaseinmemory.com/2015/08/error-ogg-00446.html

>>

Adding a disk to diskgroup # ASM

>>>> >>> check OS Level ++++

[root@tnc1 dev]# ls -ltr asm*

brw-rw---- 1 oracle dba      8, 129 Sep  5 07:59 asm-disk8

brw-rw---- 1 oracle dba      8, 145 Sep  5 07:59 asm-disk9

>>>>  ## >>> Log in as "sysasm" and query header status, we can add a disk to the disk group when it's header status is CANDIDATE (or) PROVISIONED (or) FORMER +++

[oracle@tnc1 scripts]$ cat /u01/stage/scripts/check_free_space_disks.sql

set lines 255

col path for a35

col Diskgroup for a15

col DiskName for a20

col disk# for 999

col total_mb for 999,999,999

col free_mb for 999,999,999

compute sum of total_mb on DiskGroup

compute sum of free_mb on DiskGroup

break on DiskGroup skip 1 on report -

set pages 255

select a.name DiskGroup, b.disk_number Disk#, b.name DiskName, b.total_mb, b.free_mb, b.path, b.header_status

from v$asm_disk b, v$asm_diskgroup a

where a.group_number (+) =b.group_number

order by b.group_number, b.disk_number, b.name

/

set lines 122

set pages 66

>>>> ### >>>

>>>>

Log in as sysasm #  and add a disk to diskgroup ##

SQL> alter diskgroup stage add disk '/dev/asm-disk8','/dev/asm-disk9';

Diskgroup altered.

"The Number of Actual Rebalancing processes allocated Depend's upon parameter asm_power_limit" it ranges from 0 to 11 & 0 to 1024 depends on disk group compatibility 

sql> alter system set asm_power_limit=11;

>>>>  ##  >>>

Monitor Rebalancing Operation +++
SQL> select group_number,operation,state,power,est_minutes from v$asm_operation;

---Nikhil Tatineni---
---ASM --- 

rootupgrade.sh failed on first node (11.2.0.4 to 12.1.0.2)

scenario # upgrading GRID infrastructure from 11.2.0.4 to 12.1.0.2 and rootupgrade.sh failed on first node 
 
[root@tnc1 ~]# /u01/app/12.1.0.2/grid/rootupgrade.sh
The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/12.1.0.2/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/12.1.0.2/grid/crs/install/crsconfig_params
2016/09/02 16:52:10 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.

2016/09/02 16:52:12 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.

2016/09/02 16:52:17 CLSRSC-464: Starting retrieval of the cluster configuration data
2016/09/02 16:52:33 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.
2016/09/02 16:52:33 CLSRSC-363: User ignored prerequisites during installation
2016/09/02 16:52:50 CLSRSC-515: Starting OCR manual backup.
2016/09/02 16:52:54 CLSRSC-516: OCR manual backup successful.
2016/09/02 16:53:03 CLSRSC-468: Setting Oracle Clusterware and ASM to rolling migration mode

2016/09/02 16:53:03 CLSRSC-482: Running command: '/u01/app/12.1.0.2/grid/bin/asmca -silent -upgradeNodeASM -nonRolling false -oldCRSHome /u01/app/11.2.0.4/grid -oldCRSVersion 11.2.0.4.0 -nodeNumber 1 -firstNode true -startRolling true'
ASM configuration upgraded in local node successfully.

2016/09/02 16:53:15 CLSRSC-469: Successfully set Oracle Clusterware and ASM to rolling migration mode
2016/09/02 16:53:15 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack
2016/09/02 16:53:49 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.
OLR initialization - successful
2016/09/02 16:57:07 CLSRSC-329: Replacing Clusterware entries in file '/etc/inittab'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2016/09/02 17:01:43 CLSRSC-115: Start of resource 'ora.asm' failed

2016/09/02 17:01:43 CLSRSC-117: Failed to start Oracle Clusterware stack
2016/09/02 17:01:43 CLSRSC-247: Failed to start ASM

Died at /u01/app/12.1.0.2/grid/crs/install/crsupgrade.pm line 967.
The command '/u01/app/12.1.0.2/grid/perl/bin/perl -I/u01/app/12.1.0.2/grid/perl/lib -I/u01/app/12.1.0.2/grid/crs/install /u01/app/12.1.0.2/grid/crs/install/rootcrs.pl  -upgrade' execution failed

 Environment looks like as follows on failed node
[root@tnc1 trace]# ps -ef | grep ohasd
root      8333     1  0 16:57 ?        00:00:00 /bin/sh /etc/init.d/init.ohasd run
root      9554     1  0 16:58 ?        00:00:05 /u01/app/12.1.0.2/grid/bin/ohasd.bin exclusive
root     20043 18187  0 17:24 pts/2    00:00:00 grep ohasd

[root@tnc1 trace]# ps -elf | egrep "PID|d.bin|ohas|oraagent|orarootagent|cssdagent|cssdmonitor" | grep -v grep

F S UID        PID  PPID  C PRI  NI ADDR SZ WCHAN  STIME TTY          TIME CMD
4 S root      8333     1  0  78   0 -  2704 pipe_w 16:57 ?        00:00:00 /bin/sh /etc/init.d/init.ohasd run
4 S root      9554     1  0  75   0 - 81343 futex_ 16:58 ?        00:00:05 /u01/app/12.1.0.2/grid/bin/ohasd.bin exclusive
4 S oracle   11044     1  0  75   0 - 68065 futex_ 16:59 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/oraagent.bin
0 S oracle   11058     1  0  75   0 - 36257 429496 16:59 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/evmd.bin
0 S oracle   11086     1  0  75   0 - 33771 -      16:59 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/mdnsd.bin
0 S oracle   11096 11058  0  75   0 - 50643 923257 16:59 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/evmlogger.bin -o /u01/app/12.1.0.2/grid/log/[HOSTNAME]/evmd/evmlogger.info -l /u01/app/12.1.0.2/grid/log/[HOSTNAME]/evmd/evmlogger.log
0 S oracle   11113     1  0  75   0 - 39287 -      16:59 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/gpnpd.bin
4 S root     11174     1  0 -40   - - 41241 futex_ 16:59 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/cssdmonitor
0 S oracle   11176     1  0  75   0 - 42690 -      16:59 ?        00:00:01 /u01/app/12.1.0.2/grid/bin/gipcd.bin
4 S root     11194     1  0 -40   - - 41385 futex_ 16:59 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/cssdagent
4 S oracle   11223     1  0 -40   - - 56853 futex_ 16:59 ?        00:00:03 /u01/app/12.1.0.2/grid/bin/ocssd.bin
4 S root     11430     1  0  75   0 - 50513 futex_ 17:00 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/orarootagent.bin
4 S root     11443     1  0  78   0 - 40670 futex_ 17:00 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/octssd.bin reboot
4 S root     11469     1  0 -40   - - 57823 -      17:00 ?        00:00:05 /u01/app/12.1.0.2/grid/bin/osysmond.bin
4 S root     11500     1  0 -40   - - 72193 923257 17:00 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/ologgerd -M -d /u01/app/12.1.0.2/grid/crf/db/tnc1

[root@tnc1 bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4534: Cannot communicate with Event Manager

[root@tnc1 bin]# cat /etc/oracle/olr.loc
olrconfig_loc=/u01/app/12.1.0.2/grid/cdata/tnc1.olr
crs_home=/u01/app/12.1.0.2/grid

 Root cause #
/dev/shm/ to store temporary storage files on server
when runs out of memory / rootupgrade.sh fails on server #

Resolution #
Increase /dev/shm/ space to resolve this issue # 

[root@tnc1 bin]# cat /etc/fstab
/dev/VolGroup00/LogVol00 /                ext3    defaults        1 1
LABEL=/boot             /boot                  ext3    defaults        1 2
tmpfs                   /dev/shm                    tmpfs   defaults        0 0
devpts                  /dev/pts                     devpts  gid=5,mode=620  0 0
sysfs                   /sys                             sysfs   defaults        0 0
proc                    /proc                           proc    defaults        0 0
/dev/VolGroup00/LogVol01 swap        swap    defaults        0 0
tmpfs  /dev/shm  tmpfs  defaults,size=3G  0 0
  
Downgrade again from 12.1.0.2 to 11.2.0.4 

[root@tnc1 install]# ./rootcrs.sh -downgrade -force
Using configuration parameter file: /u01/app/12.1.0.2/grid/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'tnc1'
CRS-2673: Attempting to stop 'ora.crf' on 'tnc1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'tnc1'
CRS-2673: Attempting to stop 'ora.evmd' on 'tnc1'
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'tnc1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'tnc1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'tnc1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'tnc1' succeeded
CRS-2677: Stop of 'ora.crf' on 'tnc1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'tnc1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'tnc1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'tnc1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'tnc1'
CRS-2677: Stop of 'ora.mdnsd' on 'tnc1' succeeded
CRS-2677: Stop of 'ora.cssd' on 'tnc1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'tnc1'
CRS-2677: Stop of 'ora.gipcd' on 'tnc1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'tnc1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2016/09/02 17:32:44 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2016/09/02 17:32:44 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
Successfully downgraded Oracle Clusterware stack on this node

[root@tnc1 install]# cat /etc/oracle/olr.loc
olrconfig_loc=/u01/app/11.2.0.4/grid/cdata/tnc1.olr
crs_home=/u01/app/11.2.0.4/grid
[root@tnc1 install]#  
 ------
Next Step # RERUN rootupgrade.sh  from 12.1.0.2 again ...
 Output as follows #

[root@tnc1 bin]# /u01/app/12.1.0.2/grid/rootupgrade.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME=  /u01/app/12.1.0.2/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "oraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin.  Overwrite it? (y/n)
[n]:

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/12.1.0.2/grid/crs/install/crsconfig_params
2016/09/02 17:43:25 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.

2016/09/02 17:43:25 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2016/09/02 17:43:31 CLSRSC-464: Starting retrieval of the cluster configuration data
2016/09/02 17:43:42 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.

2016/09/02 17:43:42 CLSRSC-363: User ignored prerequisites during installation
2016/09/02 17:44:04 CLSRSC-515: Starting OCR manual backup.
2016/09/02 17:44:08 CLSRSC-516: OCR manual backup successful.
2016/09/02 17:44:15 CLSRSC-468: Setting Oracle Clusterware and ASM to rolling migration mode

2016/09/02 17:44:15 CLSRSC-482: Running command: '/u01/app/12.1.0.2/grid/bin/asmca -silent -upgradeNodeASM -nonRolling false -oldCRSHome /u01/app/11.2.0.4/grid -oldCRSVersion 11.2.0.4.0 -nodeNumber 1 -firstNode true -startRolling true'

ASM configuration upgraded in local node successfully.
2016/09/02 17:44:21 CLSRSC-469: Successfully set Oracle Clusterware and ASM to rolling migration mode
2016/09/02 17:44:21 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack
2016/09/02 17:44:56 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.
OLR initialization - successful
2016/09/02 17:48:16 CLSRSC-329: Replacing Clusterware entries in file '/etc/inittab'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2016/09/02 17:52:03 CLSRSC-472: Attempting to export the OCR

2016/09/02 17:52:03 CLSRSC-482: Running command: 'ocrconfig -upgrade oracle oinstall'
2016/09/02 17:52:31 CLSRSC-473: Successfully exported the OCR

2016/09/02 17:52:38 CLSRSC-486:
 At this stage of upgrade, the OCR has changed.
 Any attempt to downgrade the cluster after this point will require a complete cluster outage to restore the OCR.

2016/09/02 17:52:38 CLSRSC-541:
 To downgrade the cluster:
 1. All nodes that have been upgraded must be downgraded.

2016/09/02 17:52:38 CLSRSC-542:
 2. Before downgrading the last node, the Grid Infrastructure stack on all other cluster nodes must be down.

2016/09/02 17:52:38 CLSRSC-543:
 3. The downgrade command must be run on the node tnc2 with the '-lastnode' option to restore global configuration data.
2016/09/02 17:53:06 CLSRSC-343: Successfully started Oracle Clusterware stack
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully taken the backup of node specific configuration in OCR.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2016/09/02 17:53:49 CLSRSC-474: Initiating upgrade of resource types

2016/09/02 17:54:17 CLSRSC-482: Running command: 'upgrade model  -s 11.2.0.4.0 -d 12.1.0.2.0 -p first'

2016/09/02 17:54:17 CLSRSC-475: Upgrade of resource types successfully initiated.
2016/09/02 17:54:25 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded 

--Nikhil Tatineni--
--Oracle 12c -- 

Reference #
http://www.golinuxhub.com/2012/09/how-to-fix-ora-00845-memorytarget-not.html

ERROR OGG-01161 Bad column index (17)

Scenario # Replicat got abended with "ERROR   OGG-01161  Bad column index (17) " 

Source Context :

SourceModule   : [ggstd.conv.endian]

SourceID           : [/scratch/aime1/adestore/views/aime1_staxj04/oggcore/OpenSys/src/gglib/ggstd/lecnv.c]

SourceFunction  : [convCompSQL(char *, file_def *, rowlen_t)]

SourceLine         : [533]

2016-07-13 06:48:42  ERROR   OGG-01161  Bad column index (17) specified for table world.andhra_pradesh, max columns = 17.

Last log location read:

     FILE              :     /opt/ggs/trails/yn000063

     SEQNO         :     63

     RBA               :     93183

     TIMESTAMP  :     2016-07-09 05:06:28.026375

     EOF               :      NO

     READERR     :       0

Resolution #  

create definition file using defgen utility, definition of source table is loaded into .def file and place definition file on target

Steps to create definition file
Step1: Create .prm file in dirprm directory under golden gate home

vi  /opt/app/db11g/ggs/dirprm/world.prm
defsfile ./dirdef/def-world.def
userid ggsuser password xxxxxx
table world.andhra_pradesh

Step2:  Run the defgen file from goldengate home 

./defgen paramfile /opt/app/db11g/ggs/dirprm/world.prm 

when you run defgen utility, .def file is created in dirdef directory under goldengate home. after scp or sftp .def file created under dirdef directory from source to target golden gate home /dirdef directory and add parameters in replicat paramfile as follows

Replicat param file with definition file:

-- GGSCI> add replicat rnlggsm1, exttrail /opt/ggs/trails/yn, checkpointtable ggsuser.nspckpt
replicat rnlggsm1
SETENV (ORACLE_HOME="/usr/local/opt/oracle/product/11204")
SETENV (ORACLE_SID="db11g")
userid ggsuser, password xxxxxx
sourcedefs ./dirdef/def-world.def
reperror (default, discard)
discardfile /opt/app/ggs/dsc/rnlggsm1.dsc, megabytes 200, append
discardrollover at 21:00
---REPERROR (1403, DISCARD)
---BATCHSQL BATCHERRORMODE, BATCHTRANSOPS 100000
map world.andhra_pradesh, target india.andhra_pradesh;

Issue is resolved --

--Nikhil Tatineni--

--GoldenGate--

RAC Patching # opatch auto

+++++++++++++++++++++++++++++++++++++++++++++++++++++++++

Applying opatch using response file # 

[root@tnc1 stage]# opatch auto /u01/stage/23274134 -ocmrf /u01/stage/first_node_file.rsp

Executing /u01/app/11.2.0.4/grid/perl/bin/perl /u01/app/11.2.0.4/grid/OPatch/crs/patch11203.pl -patchdir /u01/stage -patchn 23274134 -ocmrf /u01/stage/first_node_file.rsp -paramfile /u01/app/11.2.0.4/grid/crs/install/crsconfig_params

This is the main log file: /u01/app/11.2.0.4/grid/cfgtoollogs/opatchauto2016-08-26_13-35-20.log

This file will show your detected configuration and all the steps that opatchauto attempted to do on your system:

/u01/app/11.2.0.4/grid/cfgtoollogs/opatchauto2016-08-26_13-35-20.report.log

2016-08-26 13:35:20: Starting Clusterware Patch Setup

Using configuration parameter file: /u01/app/11.2.0.4/grid/crs/install/crsconfig_params

Stopping RAC /u01/app/oracle/product/11.2.0.4/db_1 ...

Stopped RAC /u01/app/oracle/product/11.2.0.4/db_1 successfully

patch /u01/stage/23274134/23054359  apply successful for home  /u01/app/oracle/product/11.2.0.4/db_1 

patch /u01/stage/23274134/23054319/custom/server/23054319  apply successful for home  /u01/app/oracle/product/11.2.0.4/db_1 

Stopping CRS...

Stopped CRS successfully

patch /u01/stage/23274134/23054359  apply successful for home  /u01/app/11.2.0.4/grid 

patch /u01/stage/23274134/23054319  apply successful for home  /u01/app/11.2.0.4/grid 

patch /u01/stage/23274134/22502505  apply successful for home  /u01/app/11.2.0.4/grid 

Starting CRS...

Installing Trace File Analyzer

CRS-4123: Oracle High Availability Services has been started.

Starting RAC /u01/app/oracle/product/11.2.0.4/db_1 ...

Started RAC /u01/app/oracle/product/11.2.0.4/db_1 successfully

opatch auto succeeded. 

#  +++++++++++      ++++++++++++++ +++++++++++++ +++++++++++++

This is the main log file: /u01/app/11.2.0.4/grid/cfgtoollogs/opatchauto2016-08-26_13-35-20.report.log

Steps executed to apply "opatch auto" in RAC cluster as follows # 

[root@tnc1 ~]# tail -100f /u01/app/11.2.0.4/grid/cfgtoollogs/opatchauto2016-08-26_13-35-20.report.log

***********  Configuration Data  ***********

* It shows only those targets that will be patched in this session *

crs_home=/u01/app/11.2.0.4/grid      owner=oracle      opatch_ver=11.2.0.3.12

rac_home=/u01/app/oracle/product/11.2.0.4/db_1      owner=oracle      opatch_ver=11.2.0.3.12

*********** Steps to be executed as owner unless specified as root ***********

1: /u01/app/oracle/product/11.2.0.4/db_1/OPatch/opatch prereq CheckComponents -ph /u01/stage/23274134/23054359 -oh /u01/app/oracle/product/11.2.0.4/db_1

2: /u01/app/oracle/product/11.2.0.4/db_1/OPatch/opatch prereq CheckComponents -ph /u01/stage/23274134/23054319/custom/server/23054319 -oh /u01/app/oracle/product/11.2.0.4/db_1

3: /u01/app/oracle/product/11.2.0.4/db_1/OPatch/opatch prereq CheckConflictAgainstOH -ph /u01/stage/23274134/23054359 -oh /u01/app/oracle/product/11.2.0.4/db_1

4: /u01/app/oracle/product/11.2.0.4/db_1/OPatch/opatch prereq CheckConflictAgainstOH -ph /u01/stage/23274134/23054319/custom/server/23054319 -oh /u01/app/oracle/product/11.2.0.4/db_1

5: /u01/app/11.2.0.4/grid/OPatch/opatch prereq CheckComponents -ph /u01/stage/23274134/23054359 -oh /u01/app/11.2.0.4/grid

6: /u01/app/11.2.0.4/grid/OPatch/opatch prereq CheckComponents -ph /u01/stage/23274134/23054319 -oh /u01/app/11.2.0.4/grid

7: /u01/app/11.2.0.4/grid/OPatch/opatch prereq CheckComponents -ph /u01/stage/23274134/22502505 -oh /u01/app/11.2.0.4/grid

8: /u01/app/11.2.0.4/grid/OPatch/opatch prereq CheckConflictAgainstOH -ph /u01/stage/23274134/23054359 -oh /u01/app/11.2.0.4/grid

9: /u01/app/11.2.0.4/grid/OPatch/opatch prereq CheckConflictAgainstOH -ph /u01/stage/23274134/23054319 -oh /u01/app/11.2.0.4/grid

10: /u01/app/11.2.0.4/grid/OPatch/opatch prereq CheckConflictAgainstOH -ph /u01/stage/23274134/22502505 -oh /u01/app/11.2.0.4/grid

11: /u01/app/oracle/product/11.2.0.4/db_1/bin/emctl stop dbconsole

12: /u01/app/oracle/product/11.2.0.4/db_1/bin/emctl stop agent

13: /u01/app/oracle/product/11.2.0.4/db_1/OPatch/opatch prereq CheckApplicable -ph /u01/stage/23274134/23054359 -oh /u01/app/oracle/product/11.2.0.4/db_1

14: /u01/app/oracle/product/11.2.0.4/db_1/OPatch/opatch prereq CheckApplicable -ph /u01/stage/23274134/23054319/custom/server/23054319 -oh /u01/app/oracle/product/11.2.0.4/db_1

15: /u01/app/oracle/product/11.2.0.4/db_1/bin/srvctl stop home -o /u01/app/oracle/product/11.2.0.4/db_1 -s /u01/app/oracle/product/11.2.0.4/db_1/srvm/admin/stophome.txt -n tnc1 -f

16: /u01/stage/23274134/23054319/custom/server/23054319/custom/scripts/prepatch.sh -dbhome /u01/app/oracle/product/11.2.0.4/db_1

17: /u01/app/oracle/product/11.2.0.4/db_1/OPatch/opatch napply /u01/stage/23274134/23054359 -local -silent -ocmrf /u01/stage/first_node_file.rsp -oh /u01/app/oracle/product/11.2.0.4/db_1 -invPtrLoc /u01/app/oracle/product/11.2.0.4/db_1/oraInst.loc

18: /u01/app/oracle/product/11.2.0.4/db_1/OPatch/opatch napply /u01/stage/23274134/23054319/custom/server/23054319 -local -silent -ocmrf /u01/stage/first_node_file.rsp -oh /u01/app/oracle/product/11.2.0.4/db_1 -invPtrLoc /u01/app/oracle/product/11.2.0.4/db_1/oraInst.loc

19: /u01/stage/23274134/23054319/custom/server/23054319/custom/scripts/postpatch.sh -dbhome /u01/app/oracle/product/11.2.0.4/db_1

20: /u01/app/11.2.0.4/grid/crs/install/rootcrs.pl -unlock : run as root

21: /sbin/fuser -k /u01/app/11.2.0.4/grid/bin/crsctl.bin : run as root

22: /u01/app/11.2.0.4/grid/OPatch/opatch prereq CheckApplicable -ph /u01/stage/23274134/23054359 -oh /u01/app/11.2.0.4/grid

23: /u01/app/11.2.0.4/grid/OPatch/opatch prereq CheckApplicable -ph /u01/stage/23274134/23054319 -oh /u01/app/11.2.0.4/grid

24: /u01/app/11.2.0.4/grid/OPatch/opatch prereq CheckApplicable -ph /u01/stage/23274134/22502505 -oh /u01/app/11.2.0.4/grid

25: /u01/app/11.2.0.4/grid/OPatch/opatch napply /u01/stage/23274134/23054359 -local -silent -ocmrf /u01/stage/first_node_file.rsp -oh /u01/app/11.2.0.4/grid -invPtrLoc /u01/app/11.2.0.4/grid/oraInst.loc

26: /u01/app/11.2.0.4/grid/OPatch/opatch napply /u01/stage/23274134/23054319 -local -silent -ocmrf /u01/stage/first_node_file.rsp -oh /u01/app/11.2.0.4/grid -invPtrLoc /u01/app/11.2.0.4/grid/oraInst.loc

27: /u01/app/11.2.0.4/grid/OPatch/opatch napply /u01/stage/23274134/22502505 -local -silent -ocmrf /u01/stage/first_node_file.rsp -oh /u01/app/11.2.0.4/grid -invPtrLoc /u01/app/11.2.0.4/grid/oraInst.loc

28: /u01/app/11.2.0.4/grid/bin/emctl start dbconsole

29: /u01/app/11.2.0.4/grid/rdbms/install/rootadd_rdbms.sh : run as root

30: /u01/app/11.2.0.4/grid/crs/install/rootcrs.pl -patch : run as root

31: /u01/app/oracle/product/11.2.0.4/db_1/bin/emctl start dbconsole

32: /u01/app/oracle/product/11.2.0.4/db_1/bin/emctl start agent

33: /u01/app/oracle/product/11.2.0.4/db_1/bin/srvctl start home -o /u01/app/oracle/product/11.2.0.4/db_1 -s /u01/app/oracle/product/11.2.0.4/db_1/srvm/admin/stophome.txt -n tnc1

### Post Installation Steps ### After ...

Patch 23274134 - Oracle Grid Infrastructure Patch Set Update 11.2.0.4.160719 (Jul2016) (Includes Database PSU 11.2.0.4.160719)

SQL> @$ORACLE_HOME/rdbms/admin/catbundle.sql psu apply

PL/SQL procedure successfully completed.

" catbundle.sql " will update "dba_registry_history" View 

SQL> column version format a10

SQL> column action_time format a30

SQL> column action format a10

SQL> select action_time,action,version,id from dba_registry_history;

ACTION_TIME                    ACTION     VERSION            ID

------------------------------ ---------- ---------- ----------

24-AUG-13 12.03.45.119862 PM   APPLY      11.2.0.4            0

19-JUL-16 03.56.57.399886 PM   APPLY      11.2.0.4            0

25-AUG-16 03.42.09.382630 PM   APPLY      11.2.0.4       160419

26-AUG-16 10.04.27.756443 PM   APPLY      11.2.0.4       160719

4 rows selected.

oracle@tnc1 ~]$ cd $ORACLE_BASE/cfgtoollogs/catbundle 

[oracle@tnc1 catbundle]$ ls -ltr

total 32

-rw-r--r-- 1 oracle oinstall 26377 Aug 26 22:04 catbundle_PSU_TEST09_GENERATE_2016Aug26_22_04_24.log

-rw-r--r-- 1 oracle oinstall  2476 Aug 26 22:04 catbundle_PSU_TEST09_APPLY_2016Aug26_22_04_27.log

[oracle@tnc1 catbundle]$

[oracle@tnc1 catbundle]$ grep ERROR *

catbundle_PSU_TEST09_APPLY_2016Aug26_22_04_27.log:SQL> Rem    SQL_IGNORABLE_ERRORS: NONE

catbundle_PSU_TEST09_GENERATE_2016Aug26_22_04_24.log: 82        RAISE_APPLICATION_ERROR(-20000, 'INVALID_PATH during openScriptFile');

catbundle_PSU_TEST09_GENERATE_2016Aug26_22_04_24.log: 84        RAISE_APPLICATION_ERROR(-20000, 'INVALID_MODE during openScriptFile');

catbundle_PSU_TEST09_GENERATE_2016Aug26_22_04_24.log: 86        RAISE_APPLICATION_ERROR(-20000,

catbundle_PSU_TEST09_GENERATE_2016Aug26_22_04_24.log: 89        RAISE_APPLICATION_ERROR(-20000,

catbundle_PSU_TEST09_GENERATE_2016Aug26_22_04_24.log:107        RAISE_APPLICATION_ERROR(-20000,

catbundle_PSU_TEST09_GENERATE_2016Aug26_22_04_24.log:110        RAISE_APPLICATION_ERROR(-20000,

catbundle_PSU_TEST09_GENERATE_2016Aug26_22_04_24.log:112      WHEN UTL_FILE.WRITE_ERROR THEN

catbundle_PSU_TEST09_GENERATE_2016Aug26_22_04_24.log:113        RAISE_APPLICATION_ERROR(-20000,

catbundle_PSU_TEST09_GENERATE_2016Aug26_22_04_24.log:114                                'WRITE_ERROR during insertScriptFile');

catbundle_PSU_TEST09_GENERATE_2016Aug26_22_04_24.log:123        RAISE_APPLICATION_ERROR(-20000,

catbundle_PSU_TEST09_GENERATE_2016Aug26_22_04_24.log:125      WHEN UTL_FILE.WRITE_ERROR THEN

catbundle_PSU_TEST09_GENERATE_2016Aug26_22_04_24.log:126        RAISE_APPLICATION_ERROR(-20000,

catbundle_PSU_TEST09_GENERATE_2016Aug26_22_04_24.log:127                                'WRITE_ERROR during closeScriptFile');

 # Compile Invalid Objects - - -

SQL> @$ORACLE_HOME/rdbms/admin/utlrp.sql 

+++Nikhil Tatineni +++