Wednesday, July 13, 2016

patch failed with no space left on directory

[oracle@green dbhome_1]$ ls -ltr 
dbhome_1

Before you apply a patch, we take backup of ORACLE_HOME
[oracle@green dbhome_1]$ cp -r dbhome_1 dbhome_1_backup

[oracle@green 22291127]$ opatch apply
The following actions have failed:
Archive not applied /u01/stage/22291127/22291127/files/lib/libserver12.a/kcbz.o to /u01/app/oracle/product/12.1.0/dbhome_1/lib/libserver12.a... ''

Do you want to proceed? [y|n]
y
User Responded with: Y
Make failed to invoke "/usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1"....'/usr/bin/ld: final link failed: No space left on device
collect2: ld returned 1 exit status
make: *** [/u01/app/oracle/product/12.1.0/dbhome_1/rdbms/lib/oracle] Error 1
'
Make failed to invoke "/usr/bin/make -f ins_rdbms.mk patchset_opt_all ioracle ORACLE_HOME=/u01/app/oracle/product/12.1.0/dbhome_1"....'/usr/bin/ld: final link failed: No space left on device
collect2: ld returned 1 exit status
make: *** [/u01/app/oracle/product/12.1.0/dbhome_1/rdbms/lib/oracle] Error 1"
The following make actions have failed :
Re-link fails on target "ioracle".
Re-link fails on target "patchset_opt_all ioracle".
Do you want to proceed? [y|n]
[Jul 11, 2016 8:43:14 PM]    8) OUI-67124:
 NApply was not able to restore the home.  Please invoke the following scripts:
                               - restore.[sh,bat]
                               - make.txt (Unix only)
                             to restore the ORACLE_HOME.  They are located under  "/u01/app/oracle/product/12.1.0/dbhome_1/.patch_storage/NApply/2016-07-11_20-37-20PM"
[Jul 11, 2016 8:43:14 PM]    --------------------------------------------------------------------------------
[Jul 11, 2016 8:43:14 PM]    Finishing UtilSession at Mon Jul 11 20:43:14 EDT 2016
[Jul 11, 2016 8:43:14 PM]    Log file location: /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2016-07-11_20-37-20PM_1.log
[Jul 11, 2016 8:43:14 PM]    Stack Description: java.lang.RuntimeException: Re-link fails on target "ioracle". Re-link fails on target "patchset_opt_all ioracle".
[Jul 11, 2016 8:43:14 PM]    StackTrace: oracle.opatch.OPatchSession.runMake(OPatchSession.java:772)
[Jul 11, 2016 8:43:14 PM]    StackTrace: oracle.opatch.MergedPatchObject.invokeConsolidatedMake(MergedPatchObject.java:1784)
[Jul 11, 2016 8:43:14 PM]    StackTrace: oracle.opatch.opatchutil.NApply.process(NApply.java:1268)
[Jul 11, 2016 8:43:14 PM]    StackTrace: oracle.opatch.opatchutil.OUSession.napply(OUSession.java:1089)
[Jul 11, 2016 8:43:14 PM]    StackTrace: sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[Jul 11, 2016 8:43:14 PM]    StackTrace: sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[Jul 11, 2016 8:43:14 PM]    StackTrace: sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[Jul 11, 2016 8:43:14 PM]    StackTrace: java.lang.reflect.Method.invoke(Method.java:597)
[Jul 11, 2016 8:43:14 PM]    StackTrace: oracle.opatch.UtilSession.process(UtilSession.java:317)
[Jul 11, 2016 8:43:14 PM]    StackTrace: oracle.opatch.OPatchSession.main(OPatchSession.java:2585)
[Jul 11, 2016 8:43:14 PM]    StackTrace: oracle.opatch.OPatch.main(OPatch.java:634)

when opatch failed with no space left on directory # 
before applying a patch make sure, you have enough space on oracle_home # 
If opatch fails, restore the old_home 
[oracle@green dbhome_1]$ mv dbhome_1_backup dbhome_1  

I already applied a patch on standby database home,  I need to rollback the  patch on standby # :) 
[oracle@green 22291127]$ opatch rollback -id 22291127
Oracle Interim Patch Installer version 12.1.0.1.10
Copyright (c) 2016, Oracle Corporation.  All rights reserved.
Oracle Home       : /u01/app/oracle/product/12.1.0/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
OPatch version    : 12.1.0.1.10
OUI version       : 12.1.0.2.0
Log file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/22291127_Jul_08_2016_20_48_46/rollback2016-07-08_20-48-46PM_1.log

Patches will be rolled back in the following order: 
   22291127   21948354   21359755   20831110   20299023   19769480
The following patch(es) will be rolled back: 22291127  21948354  21359755  20831110  20299023  19769480  
Sub-patches of a composite series are being rolled back. The system will be returned to a state where all subpatches are rolled back.
The following bug fixes will be removed: 22168163,18893947,19333670,20825533,20879889,21756677,16439813,20466628,22173980,21899588,19883092,21329301,13542050,20711718,18499088,20596234,22528741,18914624,21915719,19835133,20117253,20904530,18990023,21273804,20936905,20356733,22046677,21756661,22353199,21260431,20952966,21315084,19354335,19777862,19990037,21153266,21917884,21091431,20373598,22762046,21744290,19639483,19258504,21847223,19888853,22374754,19450314,16923858,20043616,20124446,19676012,22353346,19721304,20831538,19591608,20447445,20794034,19490948,19689979,19902195,19869255,21787056,17655240,21517440,21300341,20173897,21668627,21526048,16887946,17551063,19879746,20101006,20618595,22092979,20318889,19326908,18973548,18799063,20877664,21756699,21875360,18886413,21188532,20869721,20890311,20446883,20509482,17890099,20588502,19604659,20951038,18799993,19141838,19243521,20048359,19571367,20397490,20476175,20925795,21442094,20245930,20134339,21281532,21421886,20413820,18966843,20441797,20281121,21620471,19791273,20361671,18411216,19587324,19032777,19468991,17365043,22062026,20471920,18743542,20688221,20122715,20424899,19307662,20746251,20919320,20331945,20657441,20387265,20294666,19703301,20165574,20899461,20558005,19399918,20402832,21225209,19284031,20474192,19699191,20093776,19989009,20440930,18306996,19865345,19536415,18607546,19487147,19035573,19873610,19081128,19606174,19272708,17274537,19358317,19291380,19012119,19687159,19547370,20348653,20425790,19524384,19385656,19928926,18681056,19512341,18856999,16619249,19018447,19393542,20235511,19627012,19649152,19670108,19684504,18417036,20347562,19658708,19896336,19805359,19637186,18940497,19183343,19978542,19644859,18202441,19023822,19791377,18909599,19335438,19238590,19841800,19619732,19315691,19730508,19597583,19065677,19915271,20284155,19157754,18885870,19303936,19708632,19371175,18618122,19329654,19075256,19074147,19044962,19289642,19068610,18988834,19028800,19561643,19058490,19390567,18967382,19174942,19174521,19176223,19501299,19178851,18948177,18674047,19723336,19189525,19001390,19176326,19280225,19143550,18250893,19180770,19155797,19016730,19185876,18354830,19067244,18845653,18849537,18964978,19065556,19440586,19439759,19024808,18952989,18990693,19052488,19189317,19409212,19124589,19154375,19279273,19468347,19054077,19048007,19248799,19018206,18921743,14643995,18456643,16870214,19434529,19706965,17835294,20074391,18791688,19197175,19134173,19174430,19050649,19769480,19077215,19577410,18288842,18436647,19520602,19149990,19076343,19195895,18610915,19068970,19518079,19304354,19001359,19676905,19309466,19382851,18964939,16359751,19022470,19532017,19597439,18674024,19430401

Please shutdown Oracle instances running out of this ORACLE_HOME on the local system.
(Oracle Home = '/u01/app/oracle/product/12.1.0/dbhome_1')

Is the local system ready for patching? [y|n]
y
User Responded with: Y

Rolling back patch 22291127...
RollbackSession rolling back interim patch '22291127' from OH '/u01/app/oracle/product/12.1.0/dbhome_1'
Patching component oracle.ldap.rsf, 12.1.0.2.0...
Patching component oracle.ldap.client, 12.1.0.2.0...
Patching component oracle.tfa, 12.1.0.2.0...
Patching component oracle.rdbms, 12.1.0.2.0...
Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...

Patching component oracle.rdbms.rsf, 12.1.0.2.0...
Deleting "phn.o" from archive "/u01/app/oracle/product/12.1.0/dbhome_1/lib/libpls12_pic.a"
Patching component oracle.oraolap, 12.1.0.2.0...
Patching component oracle.rdbms.rman, 12.1.0.2.0...
Patching component oracle.oraolap.dbscripts, 12.1.0.2.0...
RollbackSession removing interim patch '22291127' from inventory

Rolling back patch 21948354...
RollbackSession rolling back interim patch '21948354' from OH '/u01/app/oracle/product/12.1.0/dbhome_1'

Patching component oracle.rdbms.deconfig, 12.1.0.2.0...
Patching component oracle.xdk, 12.1.0.2.0...
Patching component oracle.tfa, 12.1.0.2.0...
Patching component oracle.rdbms, 12.1.0.2.0...
Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...
Patching component oracle.xdk.parser.java, 12.1.0.2.0...
Patching component oracle.rdbms.rsf, 12.1.0.2.0...

Patching component oracle.xdk.rsf, 12.1.0.2.0...
RollbackSession removing interim patch '21948354' from inventory

Rolling back patch 21359755...
RollbackSession rolling back interim patch '21359755' from OH '/u01/app/oracle/product/12.1.0/dbhome_1'
Patching component oracle.assistants.server, 12.1.0.2.0...
Patching component oracle.rdbms.rsf, 12.1.0.2.0...
Patching component oracle.rdbms, 12.1.0.2.0...
Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...
Patching component oracle.tfa, 12.1.0.2.0...
RollbackSession removing interim patch '21359755' from inventory

Rolling back patch 20831110...
RollbackSession rolling back interim patch '20831110' from OH '/u01/app/oracle/product/12.1.0/dbhome_1'
Patching component oracle.rdbms, 12.1.0.2.0...
Patching component oracle.oraolap.dbscripts, 12.1.0.2.0...
Patching component oracle.ldap.rsf, 12.1.0.2.0...
Patching component oracle.tfa, 12.1.0.2.0...
Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...
Patching component oracle.rdbms.rsf, 12.1.0.2.0...
RollbackSession removing interim patch '20831110' from inventory
Rolling back patch 20299023...

RollbackSession rolling back interim patch '20299023' from OH '/u01/app/oracle/product/12.1.0/dbhome_1'
Patching component oracle.tfa, 12.1.0.2.0...
Patching component oracle.rdbms.deconfig, 12.1.0.2.0...
Patching component oracle.rdbms.rsf, 12.1.0.2.0...
Patching component oracle.rdbms, 12.1.0.2.0...
Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...
Patching component oracle.rdbms.rsf.ic, 12.1.0.2.0...
Patching component oracle.ldap.rsf, 12.1.0.2.0...
Patching component oracle.ldap.rsf.ic, 12.1.0.2.0...
RollbackSession removing interim patch '20299023' from inventory
Rolling back patch 19769480...
RollbackSession rolling back interim patch '19769480' from OH '/u01/app/oracle/product/12.1.0/dbhome_1'
Patching component oracle.rdbms.deconfig, 12.1.0.2.0...
Patching component oracle.xdk, 12.1.0.2.0...
Patching component oracle.tfa, 12.1.0.2.0...
Patching component oracle.rdbms.util, 12.1.0.2.0...
Patching component oracle.rdbms, 12.1.0.2.0...
Patching component oracle.rdbms.dbscripts, 12.1.0.2.0...
Patching component oracle.xdk.parser.java, 12.1.0.2.0...
Patching component oracle.oraolap, 12.1.0.2.0...
Patching component oracle.xdk.rsf, 12.1.0.2.0...
Patching component oracle.rdbms.rsf, 12.1.0.2.0...
Patching component oracle.rdbms.rman, 12.1.0.2.0...
Patching component oracle.ldap.rsf, 12.1.0.2.0...
Patching component oracle.ldap.rsf.ic, 12.1.0.2.0...
RollbackSession removing interim patch '19769480' from inventory
Log file location: /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/22291127_Jul_08_2016_20_48_46/rollback2016-07-08_20-48-46PM_1.log
OPatch succeeded.

--Nikhil Tatineni--
--Oracle in Memory--




Querys to monitor RAC

following few  Query's will help to find out culprits-  Query to check long running transaction from last 8 hours  Col Sid Fo...