Database In Memory

Tuesday, September 13, 2016

TFA not Moved to New GI_HOME while Upgrading from 11.2.0.4 to 12c

Scenario: Upgrading to 12.1.0.2 Grid Infrastructure but TFA wasn't installed in the new GI_HOME:

>> TFA not Moved to New GI_HOME While Upgrading to 12.1.0.2 (Doc ID 1918428.1)

Follow to fix this issue +

Deinstall TFA from 11.2.0.4

[root@tnc2 bin]# tfactl uninstall

TFA will be Uninstalled on Node tnc2:

Removing TFA from tnc2 only

Please remove TFA locally on any other configured nodes

Notifying Other Nodes about TFA Uninstall...

Sleeping for 10 seconds...

Stopping TFA Support Tools...

Stopping TFA in tnc2...

Shutting down TFA

TFA is running - Will wait 5 seconds (up to 3 times)

TFA-00002 : Oracle Trace File Analyzer (TFA) is not running

TFA Stopped Successfully

. . . . .

. . .

Successfully shutdown TFA..

Deleting TFA support files on tnc2:

Removing /u01/app/oracle/tfa/tnc2/database...

Removing /u01/app/oracle/tfa/tnc2/log...

Removing /u01/app/oracle/tfa/tnc2/output...

Removing /u01/app/oracle/tfa/tnc2...

Removing /u01/app/oracle/tfa...

Removing /etc/rc.d/rc0.d/K17init.tfa

Removing /etc/rc.d/rc1.d/K17init.tfa

Removing /etc/rc.d/rc2.d/K17init.tfa

Removing /etc/rc.d/rc4.d/K17init.tfa

Removing /etc/rc.d/rc6.d/K17init.tfa

Removing /etc/init.d/init.tfa...

Removing /u01/app/11.2.0.4/grid/bin/tfactl...

Removing /u01/app/11.2.0.4/grid/tfa/bin...

Removing /u01/app/11.2.0.4/grid/tfa/tnc2...

Install TFA as root user on new 12.1.0.2 Cluster Home +

Syntax #

<NEW_GI_HOME>/crs/install/tfa_setup -silent -crshome <NEW_GI_HOME

/u01/app/12.1.0.2/grid/crs/install/tfa_setup -silent -crshome /u01/app/12.1.0.2/grid

>>>

[root@tnc2 grid]# /u01/app/12.1.0.2/grid/crs/install/tfa_setup -silent -crshome /u01/app/12.1.0.2/grid

Starting TFA installation

Using JAVA_HOME : /u01/app/12.1.0.2/grid/jdk/jre

Running Auto Setup for TFA as user root...

Installing TFA now...

TFA Will be Installed on tnc2...

TFA will scan the following Directories

++++++++++++++++++++++++++++++++++++++++++++

.------------------------------------------------------.

| tnc2 |

+-------------------------------------------+----------+

| Trace Directory | Resource |

+-------------------------------------------+----------+

| /u01/app/12.1.0.2/grid/OPatch/crs/log | CRS |

| /u01/app/12.1.0.2/grid/cfgtoollogs | CFGTOOLS |

| /u01/app/12.1.0.2/grid/crs/log | CRS |

| /u01/app/12.1.0.2/grid/cv/log | CRS |

| /u01/app/12.1.0.2/grid/evm/admin/log | CRS |

| /u01/app/12.1.0.2/grid/evm/admin/logger | CRS |

| /u01/app/12.1.0.2/grid/evm/log | CRS |

| /u01/app/12.1.0.2/grid/install | INSTALL |

| /u01/app/12.1.0.2/grid/log | CRS |

| /u01/app/12.1.0.2/grid/network/log | CRS |

| /u01/app/12.1.0.2/grid/oc4j/j2ee/home/log | DBWLM |

| /u01/app/12.1.0.2/grid/opmn/logs | CRS |

| /u01/app/12.1.0.2/grid/racg/log | CRS |

| /u01/app/12.1.0.2/grid/rdbms/log | ASM |

| /u01/app/12.1.0.2/grid/scheduler/log | CRS |

| /u01/app/12.1.0.2/grid/srvm/log | CRS |

| /u01/app/oraInventory/ContentsXML | INSTALL |

| /u01/app/oraInventory/logs | INSTALL |

| /u01/app/oracle/crsdata/tnc2/acfs | ACFS |

| /u01/app/oracle/crsdata/tnc2/core | CRS |

| /u01/app/oracle/crsdata/tnc2/crsconfig | CRS |

| /u01/app/oracle/crsdata/tnc2/crsdiag | CRS |

| /u01/app/oracle/crsdata/tnc2/cvu | CRS |

| /u01/app/oracle/crsdata/tnc2/evm | CRS |

| /u01/app/oracle/crsdata/tnc2/output | CRS |

| /u01/app/oracle/crsdata/tnc2/trace | CRS |

'-------------------------------------------+----------'

Installing TFA on tnc2:

HOST: tnc2 TFA_HOME: /u01/app/12.1.0.2/grid/tfa/tnc2/tfa_home

.------------------------------------------------------------------------.

+------+---------------+------+------+------------+----------------------+

| tnc2 | RUNNING | 3679 | 5000 | 12.1.2.0.0 | 12120020140619094932 |

'------+---------------+------+------+------------+----------------------'

Running Inventory in All Nodes...

Enabling Access for Non-root Users on tnc2...

Adding default users and groups to TFA Access list...

Summary of TFA Installation:

.----------------------------------------------------------------.

| tnc2 |

+---------------------+------------------------------------------+

| Parameter | Value |

+---------------------+------------------------------------------+

| Install location | /u01/app/12.1.0.2/grid/tfa/tnc2/tfa_home |

| Repository location | /u01/app/oracle/tfa/repository |

| Repository usage | 0 MB out of 6247 MB |

'---------------------+------------------------------------------'

Installing oratop extension..

TFA is successfully installed...

Usage : /u01/app/12.1.0.2/grid/bin/tfactl <command> [options]

<command> =

start Starts TFA

stop Stops TFA

enable Enable TFA Auto restart

disable Disable TFA Auto restart

print Print requested details

access Add or Remove or List TFA Users and Groups

purge Delete collections from TFA repository

directory Add or Remove or Modify directory in TFA

host Add or Remove host in TFA

diagcollect Collect logs from across nodes in cluster

analyze List events summary and search strings in alert logs.

set Turn ON/OFF or Modify various TFA features

uninstall Uninstall TFA from this node

For help with a command: /u01/app/12.1.0.2/grid/bin/tfactl <command> -help

[oracle@tnc2 TFA]$ tfactl print status

.--------------------------------------------------------------------------------------------.

+------+---------------+-------+------+------------+----------------------+------------------+

| tnc2 | RUNNING | 3679 | 5000 | 12.1.2.0.0 | 12120020140619094932 | COMPLETE |

| tnc1 | RUNNING | 12158 | 5000 | 12.1.2.0.0 | 12120020140619094932 | COMPLETE |

'------+---------------+-------+------+------------+----------------------+------------------'

[oracle@tnc2 TFA]$ crsctl query crs softwareversion

Oracle Clusterware version on node [tnc2] is [12.1.0.2.0]

>>>>>>

--Nikhil Tatineni--

--RAC--

Sunday, September 11, 2016

12c RAC > Clusterware logs

Recently upgraded my RAC cluster from 11.2.0.4 to 12.1.0.2

Clusterware logs are centralized in RAC 12c

>> In 11gR2 RAC Cluster, we have different directories to investigate or troubleshoot issues in cluster <<

/u01/app/11.2.0.4/grid/log/tnc2

[oracle@tnc2 tnc2]$ ll

total 316

drwxr-xr-x 3 root root 4096 Jul 13 12:38 acfs

drwxr-x--- 3 oracle oinstall 4096 Aug 20 12:54 acfslog

drwxr-x--- 2 oracle oinstall 4096 Jul 13 12:37 acfsrepl

drwxr-x--- 2 root oinstall 4096 Jul 13 12:37 acfsreplroot

drwxr-xr-x 2 root oinstall 4096 Jul 13 12:37 acfssec

drwxr-x--- 2 oracle oinstall 4096 Jul 13 12:37 admin

drwxrwxr-t 4 root oinstall 4096 Jul 13 12:37 agent

-rw-rw-r-- 1 oracle oinstall 221280 Sep 2 17:56 alerttnc2.log

drwxrwxrwt 2 oracle oinstall 4096 Sep 2 17:55 client

drwxr-x--- 2 root oinstall 4096 Jul 13 12:41 crflogd

drwxr-x--- 2 root oinstall 4096 Jul 13 12:41 crfmond

drwxr-x--- 2 root oinstall 4096 Aug 26 13:47 crsd

drwxr-x--- 2 oracle oinstall 4096 Aug 20 12:51 cssd

drwxr-x--- 2 root oinstall 4096 Aug 20 15:25 ctssd

drwxr-x--- 4 oracle oinstall 4096 Jul 13 12:37 cvu

drwxr-x--- 2 oracle oinstall 4096 Jul 13 12:37 diskmon

drwxr-x--- 2 oracle oinstall 4096 Jul 13 12:41 evmd

drwxr-x--- 2 oracle oinstall 4096 Sep 2 13:58 gipcd

drwxr-x--- 2 root oinstall 4096 Jul 13 12:37 gnsd

drwxr-x--- 2 oracle oinstall 4096 Sep 2 14:32 gpnpd

drwxr-x--- 2 oracle oinstall 4096 Jul 13 12:38 mdnsd

drwxr-x--- 2 root oinstall 4096 Aug 25 15:28 ohasd

drwxrwxr-t 5 oracle oinstall 4096 Aug 24 15:51 racg

drwxr-x--- 2 oracle oinstall 4096 Jul 13 12:37 srvm

>> But in 12c RAC cluster, Oracle centralized logs & Location as follows <<

>> It is easy to troubleshoot Issues in Clusterware Now <<

[oracle@tnc2 trace]$ pwd

/u01/app/oracle/diag/crs/tnc2/crs/trace

/u01/app/oracle/diag/crs/<Node Name>/crs/trace

> Using Alert log file, we can find Root cause in New 12c RAC Cluster

[oracle@tnc2 trace]$ ls -ltr alert.log

-rw-rw---- 1 oracle oinstall 124737 Sep 11 20:39 alert.log

<< << << << Use ADRCI to get ADR location :)

[oracle@tnc2 grid]$ adrci

ADRCI: Release 12.1.0.2.0 - Production on Sun Sep 11 21:04:24 2016

ADR base = "/u01/app/oracle"

adrci> show homes

ADR Homes:

diag/tnslsnr/tnc2/listener_scan1

diag/tnslsnr/tnc2/mgmtlsnr

diag/tnslsnr/tnc2/listener

diag/rdbms/mipd09/mipd092

diag/rdbms/_mgmtdb/-MGMTDB

diag/rdbms/db12c/db12c_2

diag/asm/+asm/+ASM2

diag/crs/tnc2/crs

>> >> >>

--Nikhil Tatineni--

--12c RAC--

Saturday, September 10, 2016

ERROR OGG-01091 Unable to open file # Replicat Abended

Scenario # RNDELTA6 abended with Error #

During New GoldenGate setup removed old trails manually from Remote Trail Directory #

GGSCI (tnc61) 1> view report RNDELTA6

........

...

Source Context :

SourceModule : [er.api]

SourceID : [/scratch/aime1/adestore/views/aime1_adc4150256/oggcore/OpenSys/src/app/er/api.c]

SourceFunction : [XT_read]

SourceLine : [4145]

ThreadBacktrace : [9] elements : [/opt/app/t1c2d985/ggs/11.2.1.0.3/libgglog.so(CMessageContext::AddThreadContext()+0x1e) [0x7f8230dc270e]] : [/opt/app/t1c2d985/ggs/11.2.1.0.3/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int,

...)+0x2cc) [0x7f8230dbb6ac]] : [/opt/app/t1c2d985/ggs/11.2.1.0.3/libgglog.so(_MSG_ERR_FILE_OPEN_ERROR(CSourceContext*, char const*, CMessa

geFactory::MessageDisposition)+0x31) [0x7f8230da58dd] : [/opt/app/t1c2d985/ggs/11.2.1.0.3/replicat(XT_read(chkpt_context_t*, char*, __std_rec_hdr*, int*, int*, sho

rt, short, char*, short*, char*, short*, int, long*, char*)+0xd71) [0x4c80b1] : [/opt/app/t1c2d985/ggs/11.2.1.0.3/replicat(READ_EXTRACT_RECORD(chkpt_context_t*, short, char*, __std_rec_hd

r*, int*, int*, long*, short, short, char*, short*, char*, short*, long*, char*)+0xe8) [0x4c8ff8]]

: [/opt/app/t1c2d985/ggs/11.2.1.0.3/replicat(process_extract_loop()+0x402) [0x538dc2]]

: [/opt/app/t1c2d985/ggs/11.2.1.0.3/replicat(main+0x732) [0x54cc02]]

: [/lib64/libc.so.6(__libc_start_main+0xfd) [0x3a6ca1ed5d]]

: [/opt/app/t1c2d985/ggs/11.2.1.0.3/replicat(__gxx_personality_v0+0x31a) [0x4c0c6a]]

2016-09-09 03:10:29 ERROR OGG-01091 Unable to open file "/appl/tnc1/gghome/dirdat/bg000005" (error 2, No such file o

r directory).

During Initial Investigation, we found this issue from report file, trail with sequence number 5 doesn't exist physically. Validated the physical location of trails by querying from GGSCI command line utility

GGSCI (tnc61) 3> sh ls -ltr /appl/tnc1/gghome/dirdat/bg*

-rw-rw-rw- 1 t1c2d985g t1c2d985g 0 Sep 7 14:32 /appl/tnc1/gghome/dirdat/bg000006

-rw-rw-rw- 1 t1c2d985g t1c2d985g 0 Sep 7 14:48 /appl/tnc1/gghome/dirdat/bg000007

-rw-rw-rw- 1 t1c2d985g t1c2d985g 0 Sep 7 14:49 /appl/tnc1/gghome/dirdat/bg000008

-rw-rw-rw- 1 t1c2d985g t1c2d985g 12198 Sep 9 03:09 /appl/tnc1/gghome/dirdat/bg000009

To Resolve this issue #

alter the replicat to next available sequence number on the trail directory #

GGSCI (tnc61) 4> alter RNDELTA6, extseqno 6 extrba 0

REPLICAT altered.

GGSCI (tnc61) 5> start RNDELTA6

Sending START request to MANAGER ...

REPLICAT RNDELTA6 starting

GGSCI (tnc61) 8> info RNDELTA6

REPLICAT RNDELTA6 Last Started 2016-09-09 03:12 Status RUNNING

Checkpoint Lag 00:00:00 (updated 00:00:03 ago)

Log Read Checkpoint File /appl/tnc1/gghome/dirdat/bg000009

2016-09-01 05:19:56.000685 RBA 12198

--Nikhil Tatineni--

--gg--

ERROR OGG-01031 There is a problem in network communication, a remote file problem,

Scenario # Brand New pump writing to Target server got abended with following Error #

Database Language and Character Set:
NLS_LANG         = ".US7ASCII"
NLS_LANGUAGE     = "AMERICAN"
NLS_TERRITORY    = "AMERICA"
NLS_CHARACTERSET = "US7ASCII"

2016-09-01 10:24:30 INFO OGG-01226 Socket buffer size set to 27985 (flush size 27985).

Source Context :
SourceModule            : [er.extrout]
SourceID                : [/scratch/aime1/adestore/views/aime1_adc4150256/oggcore/OpenSys/src/app/er/extrout.c]
SourceFunction          : [complete_tcp_msg]
SourceLine              : [1526]
ThreadBacktrace         : [9] elements
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/libgglog.so(CMessageContext::AddThreadContext()+0x1e) [0x7f8ea9e4470e]]
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/libgglog.so(CMessageFactory::CreateMessage(CSourceContext*, unsigned int, ...)+0x2cc) [0x7f8ea9e3d6ac]]
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/libgglog.so(_MSG_ERR_ER_REMOTE_COMM_PROBLEM(CSourceContext*, char const*, CMessageFactory::MessageDisposition)+0x31) [0x7f8ea9e28789]]
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/extract(complete_tcp_msg(extract_def*)+0x421) [0x519049]]
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/extract(flush_tcp(extract_def*, int)+0x206) [0x5198f2]]
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/extract(RECOVERY_initialize()+0x371) [0x52ae51]]
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/extract(main+0x4a5) [0x573985]]
                          : [/lib64/libc.so.6(__libc_start_main+0xfd) [0x311801ed5d]]
                          : [/opt/app/t1c4d103/ggs/11.2.1.0.3/extract(__gxx_personality_v0+0x38a) [0x4edaba]]

2016-09-01 10:24:30 ERROR OGG-01031 There is a problem in network communication, a remote file problem, encryption keys for target and source do not match (if using ENCRYPT) or an unknown error. (Reply received is Output file /usr/local/opt/oracle/ggs/11.2.1.0.1/trails/gi000000 is not in any allowed output directories.).

2016-09-01 10:24:30 ERROR OGG-01668 PROCESS ABENDING.

To find out Root Cause #

workaround > Telnet fails,
There is problem with network between servers (or)
Blocking port Issues on target server (or)
manager child process (collector) Problems #

$ telnet tnc61.world.com 7809
Trying 192.168.56.212...
Connected to tnc1.world.com
Escape character is '^]'.
Connection closed by foreign host.

Here Telnet worked as expected. There is no issues with network or port Number #

Later ...

Checked status of Manager # manager is up & running and gathering port details

if every thing looks good, you can confirm child process is up & running on manager end

GGSCI (tnc61) 19> SEND MANAGER GETPORTINFO DETAIL

Sending GETPORTINFO, request to MANAGER ...

Dynamic Port List

Starting Index 0
Entry Port Error Process     Assigned             Program
----- ----- ----- ----------   ------------------- -------
   0   7809     0
   1   7810     0
   2   7811     0
   3   7812     0
   4   7813     0
   5   7814     0
   6   7815     0
   7   7816     0
   8   7817     0
   9   7818     0
10   7819     0
11   7820     0
12   7821     0
13   7822     0
14   7823     0
15   7824     0
16   7825     0
17   7826     0
18   7827     0
19   7828     0
20   7829     0
21   7830     0
22   7833     0
23   7835     0

>> few messages from Targetgg_HOME/ggserr.log

2016-08-31 09:55:02 WARNING OGG-01223 Oracle GoldenGate Collector for Oracle: Output file /usr/local/opt/oracle/ggs/11.2.1.0.1/trails/gi000000 is not in any allowed output directories.
2016-08-31 09:55:02 INFO    OGG-01676 Oracle GoldenGate Collector for Oracle: Terminating after client disconnect.
2016-08-31 09:58:15 INFO    OGG-00963 Oracle GoldenGate Manager for Oracle, mgr.prm: Command received from EXTRACT on host [130.6.44.131]:39623 (START SERVER CPU -1 PRI -1 TIMEOUT 300 PARAMS ).
2016-08-31 09:58:15 INFO    OGG-00960 Oracle GoldenGate Manager for Oracle, mgr.prm: Access granted (rule #4).
2016-08-31 09:58:15 INFO    OGG-01677 Oracle GoldenGate Collector for Oracle: Waiting for connection (started dynamically).
2016-08-31 09:58:15 INFO    OGG-00963 Oracle GoldenGate Manager for Oracle, mgr.prm: Command received from SERVER on host [127.0.0.1]:9264 (REPORT 1442204 7809).
2016-08-31 09:58:15 INFO    OGG-00960 Oracle GoldenGate Manager for Oracle, mgr.prm: Access granted (rule #1).
2016-08-31 09:58:15 INFO    OGG-00974 Oracle GoldenGate Manager for Oracle, mgr.prm: Manager started collector process (Port 7809).
2016-08-31 09:58:15 INFO    OGG-01228 Oracle GoldenGate Collector for Oracle: Timeout in 300 seconds.
2016-08-31 09:58:20 INFO    OGG-01229 Oracle GoldenGate Collector for Oracle: Connected to tnc1.world.com2016-08-31 09:58:20 WARNING OGG-01223 Oracle GoldenGate Collector for Oracle: Output file /usr/local/opt/oracle/ggs/11.2.1.0.1/trails/gi000000 is not in any allowed output directories.
2016-08-31 09:58:20 INFO    OGG-01676 Oracle GoldenGate Collector for Oracle: Terminating after client disconnect.

Later #

Investigate file system, i.e report file says " Reply received is Output file /usr/local/opt/oracle/ggs/11.2.1.0.1/trails/gi000000 is not in any allowed output directories.)."

check for free space available on Target server ( Target file system )

$ df -g /usr/local/opt/oracle/ggs/11.2.1.0.1/trails/
Filesystem GB blocks Free %Used Iused %Iused Mounted on
/dev/ggs_lv 20.00 19.70 2% 215 1% /usr/local/opt/oracle/ggs

>> >>>

Later I raised Oracle SR for this issue ..Followed (Doc ID 1288562.1) and resolved it

Resolution: # In order to fix this issue,

Deleted existing pump &

Changed Remote trail directory and granted required permissions on remote trail directory

Recreated pump and restarted the brand NEW PUMP #

Reference #

OGG Data Pumps Error With OGG-01031 When Upgrading the Target Side (Doc ID 1288562.1)

--Nikhil Tatineni--

--GoldenGate--

Thursday, September 8, 2016

ERROR OGG-00664 OCI Error during OCIServerAttach (status = 12162-ORA-12162:

Scenario # Rebooted golden gate process, Pump Abended with following error <

Investigated Issue and found root cause #

ggsci> View report plct90

.....

....

............

extract plct90

passthru

userid ggsuser, password *******

Source Context :

SourceModule : [ggdb.ora.sess]

SourceID : [/scratch/aime1/adestore/views/aime1_staxj04/oggcore

/OpenSys/src/gglib/ggdbora/ocisess.c]

SourceFunction : [OCISESS_context_def::oci_try(int, const char *, ...)]

SourceLine : [650]

2016-09-04 08:48:14 ERROR OGG-00664 OCI Error during OCIServerAttach (status

= 12162-ORA-12162: TNS:net service name is incorrectly specified)

Pump Param file looks like as follows #

ggsci> view param plct90

extract plct90

userid ggsuser, password oracle

PASSTHRU

RMTHOST tnc61.ffdc.com, MGRPORT 7809

RMTTRAIL /u01/app/trails/aa

REPORTCOUNT EVERY 10 MINUTES, RATE

TABLE world.orders;

Resolution: New changes in parameter file and start pump

ggsci> view param plct90

extract plct90

userid ggsuser@tns_name, password oracle

PASSTHRU

RMTHOST tnc61.ffdc.com, MGRPORT 7809

RMTTRAIL /u01/app/trails/aa

REPORTCOUNT EVERY 10 MINUTES, RATE

ggsci> start plct90

--pump came up automatically, Issue is resolved --

--Nikhil Tatineni--

--gg--

Links Related to GoldenGate #

REPLICAT WITH HANDLECOLLISIONS

http://www.databaseinmemory.com/2015/01/replicat-with-handlecollisions.html

SPLITING OracleGoldenGate REPLICAT

http://www.databaseinmemory.com/2015/02/hey-guys-welcome-back.html

Goldengate Replicat is running very slow

http://www.databaseinmemory.com/2015/10/replicat-is-running-very-slow.html

Goldengate Replicat's blocking itself

http://www.databaseinmemory.com/2015/10/replicats-blocking-itself.html

Oracle GoldenGate: SKIP LONG RUNNING TRANSACTIONS

http://www.databaseinmemory.com/2015/03/ogg-skip-long-running-transactions.html

Goldengate Extract is not moving?

http://www.databaseinmemory.com/2015/02/extract-is-not-moving.html

ERROR OGG-00446 Missing filename opening checkpoint file

http://www.databaseinmemory.com/2015/08/error-ogg-00446.html

Adding a disk to diskgroup # ASM

>>>> >>> check OS Level ++++

[root@tnc1 dev]# ls -ltr asm*

brw-rw---- 1 oracle dba 8, 129 Sep 5 07:59 asm-disk8

brw-rw---- 1 oracle dba 8, 145 Sep 5 07:59 asm-disk9

>>>> ## >>> Log in as "sysasm" and query header status, we can add a disk to the disk group when it's header status is CANDIDATE (or) PROVISIONED (or) FORMER +++

[oracle@tnc1 scripts]$ cat /u01/stage/scripts/check_free_space_disks.sql

set lines 255

col path for a35

col Diskgroup for a15

col DiskName for a20

col disk# for 999

col total_mb for 999,999,999

col free_mb for 999,999,999

compute sum of total_mb on DiskGroup

compute sum of free_mb on DiskGroup

break on DiskGroup skip 1 on report -

set pages 255

select a.name DiskGroup, b.disk_number Disk#, b.name DiskName, b.total_mb, b.free_mb, b.path, b.header_status

from v$asm_disk b, v$asm_diskgroup a

where a.group_number (+) =b.group_number

order by b.group_number, b.disk_number, b.name

set lines 122

set pages 66

>>>> ### >>>

>>>>

Log in as sysasm # and add a disk to diskgroup ##

SQL> alter diskgroup stage add disk '/dev/asm-disk8','/dev/asm-disk9';

Diskgroup altered.

"The Number of Actual Rebalancing processes allocated Depend's upon parameter asm_power_limit" it ranges from 0 to 11 & 0 to 1024 depends on disk group compatibility

sql> alter system set asm_power_limit=11;

>>>> ## >>>

Monitor Rebalancing Operation +++
SQL> select group_number,operation,state,power,est_minutes from v$asm_operation;

---Nikhil Tatineni---
---ASM ---

Friday, September 2, 2016

rootupgrade.sh failed on first node (11.2.0.4 to 12.1.0.2)

scenario # upgrading GRID infrastructure from 11.2.0.4 to 12.1.0.2 and rootupgrade.sh failed on first node

[root@tnc1 ~]# /u01/app/12.1.0.2/grid/rootupgrade.sh
The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME= /u01/app/12.1.0.2/grid

Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/12.1.0.2/grid/crs/install/crsconfig_params
2016/09/02 16:52:10 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.

2016/09/02 16:52:12 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.

2016/09/02 16:52:17 CLSRSC-464: Starting retrieval of the cluster configuration data
2016/09/02 16:52:33 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.
2016/09/02 16:52:33 CLSRSC-363: User ignored prerequisites during installation
2016/09/02 16:52:50 CLSRSC-515: Starting OCR manual backup.
2016/09/02 16:52:54 CLSRSC-516: OCR manual backup successful.
2016/09/02 16:53:03 CLSRSC-468: Setting Oracle Clusterware and ASM to rolling migration mode

2016/09/02 16:53:03 CLSRSC-482: Running command: '/u01/app/12.1.0.2/grid/bin/asmca -silent -upgradeNodeASM -nonRolling false -oldCRSHome /u01/app/11.2.0.4/grid -oldCRSVersion 11.2.0.4.0 -nodeNumber 1 -firstNode true -startRolling true'
ASM configuration upgraded in local node successfully.

2016/09/02 16:53:15 CLSRSC-469: Successfully set Oracle Clusterware and ASM to rolling migration mode
2016/09/02 16:53:15 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack
2016/09/02 16:53:49 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.
OLR initialization - successful
2016/09/02 16:57:07 CLSRSC-329: Replacing Clusterware entries in file '/etc/inittab'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2016/09/02 17:01:43 CLSRSC-115: Start of resource 'ora.asm' failed

2016/09/02 17:01:43 CLSRSC-117: Failed to start Oracle Clusterware stack
2016/09/02 17:01:43 CLSRSC-247: Failed to start ASM

Died at /u01/app/12.1.0.2/grid/crs/install/crsupgrade.pm line 967.
The command '/u01/app/12.1.0.2/grid/perl/bin/perl -I/u01/app/12.1.0.2/grid/perl/lib -I/u01/app/12.1.0.2/grid/crs/install /u01/app/12.1.0.2/grid/crs/install/rootcrs.pl -upgrade' execution failed

Environment looks like as follows on failed node
[root@tnc1 trace]# ps -ef | grep ohasd
root      8333     1 0 16:57 ?        00:00:00 /bin/sh /etc/init.d/init.ohasd run
root      9554     1 0 16:58 ?        00:00:05 /u01/app/12.1.0.2/grid/bin/ohasd.bin exclusive
root     20043 18187 0 17:24 pts/2    00:00:00 grep ohasd

[root@tnc1 trace]# ps -elf | egrep "PID|d.bin|ohas|oraagent|orarootagent|cssdagent|cssdmonitor" | grep -v grep

F S UID        PID PPID C PRI NI ADDR SZ WCHAN STIME TTY          TIME CMD
4 S root      8333     1 0 78   0 - 2704 pipe_w 16:57 ?        00:00:00 /bin/sh /etc/init.d/init.ohasd run
4 S root      9554     1 0 75   0 - 81343 futex_ 16:58 ?        00:00:05 /u01/app/12.1.0.2/grid/bin/ohasd.bin exclusive
4 S oracle   11044     1 0 75   0 - 68065 futex_ 16:59 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/oraagent.bin
0 S oracle   11058     1 0 75   0 - 36257 429496 16:59 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/evmd.bin
0 S oracle   11086     1 0 75   0 - 33771 -      16:59 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/mdnsd.bin
0 S oracle   11096 11058 0 75   0 - 50643 923257 16:59 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/evmlogger.bin -o /u01/app/12.1.0.2/grid/log/[HOSTNAME]/evmd/evmlogger.info -l /u01/app/12.1.0.2/grid/log/[HOSTNAME]/evmd/evmlogger.log
0 S oracle   11113     1 0 75   0 - 39287 -      16:59 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/gpnpd.bin
4 S root     11174     1 0 -40   - - 41241 futex_ 16:59 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/cssdmonitor
0 S oracle   11176     1 0 75   0 - 42690 -      16:59 ?        00:00:01 /u01/app/12.1.0.2/grid/bin/gipcd.bin
4 S root     11194     1 0 -40   - - 41385 futex_ 16:59 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/cssdagent
4 S oracle   11223     1 0 -40   - - 56853 futex_ 16:59 ?        00:00:03 /u01/app/12.1.0.2/grid/bin/ocssd.bin
4 S root     11430     1 0 75   0 - 50513 futex_ 17:00 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/orarootagent.bin
4 S root     11443     1 0 78   0 - 40670 futex_ 17:00 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/octssd.bin reboot
4 S root     11469     1 0 -40   - - 57823 -      17:00 ?        00:00:05 /u01/app/12.1.0.2/grid/bin/osysmond.bin
4 S root     11500     1 0 -40   - - 72193 923257 17:00 ?        00:00:00 /u01/app/12.1.0.2/grid/bin/ologgerd -M -d /u01/app/12.1.0.2/grid/crf/db/tnc1

[root@tnc1 bin]# ./crsctl check crs
CRS-4638: Oracle High Availability Services is online
CRS-4535: Cannot communicate with Cluster Ready Services
CRS-4529: Cluster Synchronization Services is online
CRS-4534: Cannot communicate with Event Manager

[root@tnc1 bin]# cat /etc/oracle/olr.loc
olrconfig_loc=/u01/app/12.1.0.2/grid/cdata/tnc1.olr
crs_home=/u01/app/12.1.0.2/grid

Root cause #
/dev/shm/ to store temporary storage files on server
when runs out of memory / rootupgrade.sh fails on server #

Resolution #
Increase /dev/shm/ space to resolve this issue #

[root@tnc1 bin]# cat /etc/fstab
/dev/VolGroup00/LogVol00 /                ext3    defaults        1 1
LABEL=/boot             /boot                  ext3    defaults        1 2
tmpfs                   /dev/shm                    tmpfs   defaults        0 0
devpts                  /dev/pts                     devpts gid=5,mode=620 0 0
sysfs                   /sys                             sysfs   defaults        0 0
proc                    /proc                 proc    defaults        0 0
/dev/VolGroup00/LogVol01 swap        swap    defaults        0 0
tmpfs /dev/shm tmpfs defaults,size=3G 0 0

Downgrade again from 12.1.0.2 to 11.2.0.4

[root@tnc1 install]# ./rootcrs.sh -downgrade -force
Using configuration parameter file: /u01/app/12.1.0.2/grid/crs/install/crsconfig_params
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'tnc1'
CRS-2673: Attempting to stop 'ora.crf' on 'tnc1'
CRS-2673: Attempting to stop 'ora.ctssd' on 'tnc1'
CRS-2673: Attempting to stop 'ora.evmd' on 'tnc1'
CRS-2673: Attempting to stop 'ora.cluster_interconnect.haip' on 'tnc1'
CRS-2673: Attempting to stop 'ora.mdnsd' on 'tnc1'
CRS-2673: Attempting to stop 'ora.gpnpd' on 'tnc1'
CRS-2677: Stop of 'ora.cluster_interconnect.haip' on 'tnc1' succeeded
CRS-2677: Stop of 'ora.crf' on 'tnc1' succeeded
CRS-2677: Stop of 'ora.ctssd' on 'tnc1' succeeded
CRS-2677: Stop of 'ora.gpnpd' on 'tnc1' succeeded
CRS-2677: Stop of 'ora.evmd' on 'tnc1' succeeded
CRS-2673: Attempting to stop 'ora.cssd' on 'tnc1'
CRS-2677: Stop of 'ora.mdnsd' on 'tnc1' succeeded
CRS-2677: Stop of 'ora.cssd' on 'tnc1' succeeded
CRS-2673: Attempting to stop 'ora.gipcd' on 'tnc1'
CRS-2677: Stop of 'ora.gipcd' on 'tnc1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'tnc1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
2016/09/02 17:32:44 CLSRSC-4001: Installing Oracle Trace File Analyzer (TFA) Collector.
2016/09/02 17:32:44 CLSRSC-4002: Successfully installed Oracle Trace File Analyzer (TFA) Collector.
Successfully downgraded Oracle Clusterware stack on this node

[root@tnc1 install]# cat /etc/oracle/olr.loc
olrconfig_loc=/u01/app/11.2.0.4/grid/cdata/tnc1.olr
crs_home=/u01/app/11.2.0.4/grid
[root@tnc1 install]#
------
Next Step # RERUN rootupgrade.sh from 12.1.0.2 again ...
Output as follows #

[root@tnc1 bin]# /u01/app/12.1.0.2/grid/rootupgrade.sh
Performing root user operation.

The following environment variables are set as:
    ORACLE_OWNER= oracle
    ORACLE_HOME= /u01/app/12.1.0.2/grid
Enter the full pathname of the local bin directory: [/usr/local/bin]:
The file "dbhome" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
The file "oraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:
The file "coraenv" already exists in /usr/local/bin. Overwrite it? (y/n)
[n]:

Entries will be added to the /etc/oratab file as needed by
Database Configuration Assistant when a database is created
Finished running generic part of root script.
Now product-specific root actions will be performed.
Using configuration parameter file: /u01/app/12.1.0.2/grid/crs/install/crsconfig_params
2016/09/02 17:43:25 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.

2016/09/02 17:43:25 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.
2016/09/02 17:43:31 CLSRSC-464: Starting retrieval of the cluster configuration data
2016/09/02 17:43:42 CLSRSC-465: Retrieval of the cluster configuration data has successfully completed.

2016/09/02 17:43:42 CLSRSC-363: User ignored prerequisites during installation
2016/09/02 17:44:04 CLSRSC-515: Starting OCR manual backup.
2016/09/02 17:44:08 CLSRSC-516: OCR manual backup successful.
2016/09/02 17:44:15 CLSRSC-468: Setting Oracle Clusterware and ASM to rolling migration mode

2016/09/02 17:44:15 CLSRSC-482: Running command: '/u01/app/12.1.0.2/grid/bin/asmca -silent -upgradeNodeASM -nonRolling false -oldCRSHome /u01/app/11.2.0.4/grid -oldCRSVersion 11.2.0.4.0 -nodeNumber 1 -firstNode true -startRolling true'

ASM configuration upgraded in local node successfully.
2016/09/02 17:44:21 CLSRSC-469: Successfully set Oracle Clusterware and ASM to rolling migration mode
2016/09/02 17:44:21 CLSRSC-466: Starting shutdown of the current Oracle Grid Infrastructure stack
2016/09/02 17:44:56 CLSRSC-467: Shutdown of the current Oracle Grid Infrastructure stack has successfully completed.
OLR initialization - successful
2016/09/02 17:48:16 CLSRSC-329: Replacing Clusterware entries in file '/etc/inittab'

CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Oracle High Availability Services has been started.
2016/09/02 17:52:03 CLSRSC-472: Attempting to export the OCR

2016/09/02 17:52:03 CLSRSC-482: Running command: 'ocrconfig -upgrade oracle oinstall'
2016/09/02 17:52:31 CLSRSC-473: Successfully exported the OCR

2016/09/02 17:52:38 CLSRSC-486:
At this stage of upgrade, the OCR has changed.
Any attempt to downgrade the cluster after this point will require a complete cluster outage to restore the OCR.

2016/09/02 17:52:38 CLSRSC-541:
To downgrade the cluster:
1. All nodes that have been upgraded must be downgraded.

2016/09/02 17:52:38 CLSRSC-542:
2. Before downgrading the last node, the Grid Infrastructure stack on all other cluster nodes must be down.

2016/09/02 17:52:38 CLSRSC-543:
3. The downgrade command must be run on the node tnc2 with the '-lastnode' option to restore global configuration data.
2016/09/02 17:53:06 CLSRSC-343: Successfully started Oracle Clusterware stack
clscfg: EXISTING configuration version 5 detected.
clscfg: version 5 is 11g Release 2.
Successfully taken the backup of node specific configuration in OCR.
Successfully accumulated necessary OCR keys.
Creating OCR keys for user 'root', privgrp 'root'..
Operation successful.
2016/09/02 17:53:49 CLSRSC-474: Initiating upgrade of resource types

2016/09/02 17:54:17 CLSRSC-482: Running command: 'upgrade model -s 11.2.0.4.0 -d 12.1.0.2.0 -p first'

2016/09/02 17:54:17 CLSRSC-475: Upgrade of resource types successfully initiated.
2016/09/02 17:54:25 CLSRSC-325: Configure Oracle Grid Infrastructure for a Cluster ... succeeded

--Nikhil Tatineni--
--Oracle 12c --

Reference #
http://www.golinuxhub.com/2012/09/how-to-fix-ora-00845-memorytarget-not.html

Database In Memory

Tuesday, September 13, 2016

TFA not Moved to New GI_HOME while Upgrading from 11.2.0.4 to 12c

Sunday, September 11, 2016

12c RAC > Clusterware logs

Saturday, September 10, 2016

ERROR OGG-01091 Unable to open file # Replicat Abended

ERROR OGG-01031 There is a problem in network communication, a remote file problem,

Thursday, September 8, 2016

ERROR OGG-00664 OCI Error during OCIServerAttach (status = 12162-ORA-12162:

Adding a disk to diskgroup # ASM

Friday, September 2, 2016

rootupgrade.sh failed on first node (11.2.0.4 to 12.1.0.2)

Querys to monitor RAC

Followers