Thursday, June 9, 2016

Oracle patching error executables are active


Scenario:Applying patch to the oracle_home/ patching failed with error "Following executables are active" 

Checking for conflicts before applying patch on database_home, Make sure it is passed 
[oracle@z1p12c 22291127]$ opatch prereq CheckConflictAgainstOHWithDetail -ph ./
Oracle Interim Patch Installer version 12.1.0.1.10
Copyright (c) 2016, Oracle Corporation.  All rights reserved.
PREREQ session
Oracle Home       : /u01/app/oracle/product/12.1.0/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
OPatch version    : 12.1.0.1.10
OUI version       : 12.1.0.2.0
Log file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2016-06-08_16-18-46PM_1
Invoking prereq "checkconflictagainstohwithdetail"
Prereq "checkConflictAgainstOHWithDetail" passed.
OPatch succeeded.

Apply patch using "opatch" utility 

[oracle@z1p12c 22291127]$ opatch apply
Oracle Interim Patch Installer version 12.1.0.1.10
Copyright (c) 2016, Oracle Corporation.  All rights reserved.
Oracle Home       : /u01/app/oracle/product/12.1.0/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
OPatch version    : 12.1.0.1.10
OUI version       : 12.1.0.2.0
Log file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2016-06-08_16-19-00PM_1.log
Verifying environment and performing prerequisite checks...
Prerequisite check "CheckActiveFilesAndExecutables" failed.
The details are:
Following executables are active :
/u01/app/oracle/product/12.1.0/dbhome_1/bin/oracle
/u01/app/oracle/product/12.1.0/dbhome_1/lib/libclntsh.so.12.1
UtilSession failed: Prerequisite check "CheckActiveFilesAndExecutables" failed.
Log file location: /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2016-06-08_16-19-00PM_1.log 

Opatch failed and found root cause in logfile generated  

[oracle@z1p12c 22291127]$ tail -100f /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2016-06-19-00PM_1.log
[Jun 8, 2016 4:19:06 PM]     Ignoring file "/u01/stage/22291127/README.txt" in the patch directory.
[Jun 8, 2016 4:19:07 PM]     Checking conflict among patches...
[Jun 8, 2016 4:19:07 PM]     checkConflictAmongPatchesWithDetail start -> Wed Jun 08 16:19:07 EDT 2016
[Jun 8, 2016 4:19:07 PM]     Wed Jun 08 16:19:07 EDT 2016
[Jun 8, 2016 4:19:08 PM]     Adapter can go list: 19769480 20299023 20831110 21359755 21948354 22291127
[Jun 8, 2016 4:19:08 PM]     Adapter not need list:
[Jun 8, 2016 4:19:08 PM]     Adapter auto-rollback list:
[Jun 8, 2016 4:19:08 PM]     checkConflictAmongPatchesWithDetail end -> Wed Jun 08 16:19:08 EDT 2016
[Jun 8, 2016 4:19:08 PM]     Intra-Conflict check PASSED
[Jun 8, 2016 4:19:08 PM]     Checking if Oracle Home has components required by patches...
[Jun 8, 2016 4:19:08 PM]     Check if patch "19769480"  is a no-op patch.
[Jun 8, 2016 4:19:08 PM]     Check if patch "20299023"  is a no-op patch.
[Jun 8, 2016 4:19:08 PM]     Opt. comp. not in OH inventory: oracle.has.crs, 12.1.0.2.0
[Jun 8, 2016 4:19:08 PM]     [ oracle.has.crs, 12.1.0.2.0 ]
[Jun 8, 2016 4:19:08 PM]     Check if patch "20831110"  is a no-op patch.
[Jun 8, 2016 4:19:08 PM]     Check if patch "21359755"  is a no-op patch.
[Jun 8, 2016 4:19:08 PM]     Check if patch "21948354"  is a no-op patch.
[Jun 8, 2016 4:19:08 PM]     Check if patch "22291127"  is a no-op patch.
[Jun 8, 2016 4:19:08 PM]     Opt. comp. not in OH inventory: oracle.oid.client, 12.1.0.2.0
[Jun 8, 2016 4:19:08 PM]     [ oracle.oid.client, 12.1.0.2.0 ]
[Jun 8, 2016 4:19:08 PM]     CheckMissingComp: hash doesn't match, re-build FVs
[Jun 8, 2016 4:19:08 PM]     Checking conflicts against Oracle Home...
[Jun 8, 2016 4:19:08 PM]     checkConflictAgainstOHWithDetail start -> Wed Jun 08 16:19:08 EDT 2016
[Jun 8, 2016 4:19:08 PM]     Wed Jun 08 16:19:08 EDT 2016
[Jun 8, 2016 4:19:08 PM]     Adapter can go list: 19769480 20299023 20831110 21359755 21948354 22291127
[Jun 8, 2016 4:19:08 PM]     Adapter not need list:
[Jun 8, 2016 4:19:08 PM]     Adapter auto-rollback list:
[Jun 8, 2016 4:19:08 PM]     checkConflictAgainstOHWithDetail end -> Wed Jun 08 16:19:08 EDT 2016
[Jun 8, 2016 4:19:08 PM]     Running prerequisite checks...
[Jun 8, 2016 4:19:08 PM]     Space Needed : 760.355MB
[Jun 8, 2016 4:19:09 PM]     Prereq checkPatchApplicableOnCurrentPlatform Passed for patch : 19769480
[Jun 8, 2016 4:19:09 PM]     Prereq checkPatchApplicableOnCurrentPlatform Passed for patch : 20299023
[Jun 8, 2016 4:19:09 PM]     Prereq checkPatchApplicableOnCurrentPlatform Passed for patch : 20831110
[Jun 8, 2016 4:19:09 PM]     Prereq checkPatchApplicableOnCurrentPlatform Passed for patch : 21359755
[Jun 8, 2016 4:19:09 PM]     Prereq checkPatchApplicableOnCurrentPlatform Passed for patch : 21948354
[Jun 8, 2016 4:19:09 PM]     Prereq checkPatchApplicableOnCurrentPlatform Passed for patch : 22291127
[Jun 8, 2016 4:19:10 PM]     Start fuser command /sbin/fuser /u01/app/oracle/product/12.1.0/dbhome_1/bin/ora Wed Jun 08 16:19:10 EDT 2016
[Jun 8, 2016 4:19:10 PM]     Finish fuser command /sbin/fuser /u01/app/oracle/product/12.1.0/dbhome_1/bin/ort Wed Jun 08 16:19:10 EDT 2016
[Jun 8, 2016 4:19:10 PM]     Files in use by a process: /u01/app/oracle/product/12.1.0/dbhome_1/bin/oracle P823 )
[Jun 8, 2016 4:19:10 PM]     Start fuser command /sbin/fuser /u01/app/oracle/product/12.1.0/dbhome_1/bin/rmaed Jun 08 16:19:10 EDT 2016
[Jun 8, 2016 4:19:10 PM]     Finish fuser command /sbin/fuser /u01/app/oracle/product/12.1.0/dbhome_1/bin/rmWed Jun 08 16:19:10 EDT 2016
[Jun 8, 2016 4:19:10 PM]     Start fuser command /sbin/fuser /u01/app/oracle/product/12.1.0/dbhome_1/bin/kfeed Jun 08 16:19:10 EDT 2016
[Jun 8, 2016 4:19:10 PM]     Finish fuser command /sbin/fuser /u01/app/oracle/product/12.1.0/dbhome_1/bin/kfWed Jun 08 16:19:10 EDT 2016
[Jun 8, 2016 4:19:10 PM]     Start fuser command /sbin/fuser /u01/app/oracle/product/12.1.0/dbhome_1/bin/amded Jun 08 16:19:10 EDT 2016
[Jun 8, 2016 4:19:10 PM]     Finish fuser command /sbin/fuser /u01/app/oracle/product/12.1.0/dbhome_1/bin/amWed Jun 08 16:19:10 EDT 2016
[Jun 8, 2016 4:19:10 PM]     Start fuser command /sbin/fuser /u01/app/oracle/product/12.1.0/dbhome_1/lib/lib.so.12.1 at Wed Jun 08 16:19:10 EDT 2016
[Jun 8, 2016 4:19:10 PM]     Finish fuser command /sbin/fuser /u01/app/oracle/product/12.1.0/dbhome_1/lib/lih.so.12.1 at Wed Jun 08 16:19:10 EDT 2016
[Jun 8, 2016 4:19:10 PM]     Files in use by a process: /u01/app/oracle/product/12.1.0/dbhome_1/lib/libclnts2.1 PID(  5366 8817 )
[Jun 8, 2016 4:19:10 PM]     Start fuser command /sbin/fuser /u01/app/oracle/product/12.1.0/dbhome_1/lib/libtsh12.so at Wed Jun 08 16:19:10 EDT 2016
[Jun 8, 2016 4:19:10 PM]     Finish fuser command /sbin/fuser /u01/app/oracle/product/12.1.0/dbhome_1/lib/lintsh12.so at Wed Jun 08 16:19:10 EDT 2016

workaround # 
As oracle Home Owner  #
$ /sbin/fuser /u01/app/oracle/product/12.1.0/dbhome_1/lib/libclntsh.so.12.1
$  ps -ef | grep oracle 

kill the process which are still accessing from oracle home and apply patch again  


oracle@z1p12c 22291127]$ opatch apply
Oracle Interim Patch Installer version 12.1.0.1.10
Copyright (c) 2016, Oracle Corporation.  All rights reserved.

Oracle Home       : /u01/app/oracle/product/12.1.0/dbhome_1
Central Inventory : /u01/app/oraInventory
   from           : /u01/app/oracle/product/12.1.0/dbhome_1/oraInst.loc
OPatch version    : 12.1.0.1.10
OUI version       : 12.1.0.2.0
Log file location : /u01/app/oracle/product/12.1.0/dbhome_1/cfgtoollogs/opatch/opatch2016-06-08_16-32-35PM_1.log
Verifying environment and performing prerequisite checks...
OPatch continues with these patches:   19769480  20299023  20831110  21359755  21948354  22291127

Do you want to proceed? [y|n]
..continue .....

Before Applying patch to the oracle_home 

STOP AGENT
STOP LISTENER
STOP DATABASES 
make sure none of the services use Oracle_home

--Nikhil Tatineni--
--12c Patching-- 




Querys to monitor RAC

following few  Query's will help to find out culprits-  Query to check long running transaction from last 8 hours  Col Sid Fo...