opatchauto failed again, apply of GI+PSU tested on 1 RAC node

The Problem is in the PSU, it was replaced with a fix. But it’s interresting anyway so let’s check it..

[oracle@standby1 ~]$ df -h
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/vg_test-lv_root   44G   18G   24G  44% /
tmpfs                        1.5G   72K  1.5G   1% /dev/shm
/dev/sda1                    477M   55M  397M  13% /boot
/dev/sdc1                     50G   19G   29G  39% /products
install                      985G  505G  480G  52% /media/sf_install

I have enough space for the patches and it’s not under root, then it would fill up the filesystem and crash

[root@standby1 ~]# . oraenv
ORACLE_SID = [orcl1] ? +ASM1
The Oracle base remains unchanged with value /products/oracle
[root@standby1 ~]# /products/grid/12.1.0.2/OPatch/opatchauto apply /media/sf_install/patches/23273629 -analyze -ocmrf /products/ocm.rsp
OPatch Automation Tool
Copyright (c)2014, Oracle Corporation. All rights reserved.

OPatchauto Version : 12.1.0.1.10
OUI Version        : 12.1.0.2.0
Running from       : /products/grid/12.1.0.2

opatchauto log file: /products/grid/12.1.0.2/cfgtoollogs/opatchauto/23273629/opatch_gi_2016-07-22_10-07-29_analyze.log

NOTE: opatchauto is running in ANALYZE mode. There will be no change to your system.

OCM RSP file has been ignored in analyze mode. 

Parameter Validation: Successful

Configuration Validation: Successful

Patch Location: /media/sf_install/patches/23273629
Grid Infrastructure Patch(es): 21436941 23054246 23054327 23054341
DB Patch(es): 23054246 23054327 

The following patch(es) are duplicate patches with patches installed in the Oracle Home.
 [ 20299023]
You have already installed same patch(es) with same UPI(s) or same version(s).
These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /products/grid/12.1.0.2

The following patch(es) are duplicate patches with patches installed in the Oracle Home.
 [ 20831110]
You have already installed same patch(es) with same UPI(s) or same version(s).
These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /products/grid/12.1.0.2

The following patch(es) are duplicate patches with patches installed in the Oracle Home.
 [ 19769480]
You have already installed same patch(es) with same UPI(s) or same version(s).
These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /products/grid/12.1.0.2

The following patch(es) are duplicate patches with patches installed in the Oracle Home.
 [ 20299023]
You have already installed same patch(es) with same UPI(s) or same version(s).
These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /products/oracle/12.1.0.2

The following patch(es) are duplicate patches with patches installed in the Oracle Home.
 [ 20831110]
You have already installed same patch(es) with same UPI(s) or same version(s).
These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /products/oracle/12.1.0.2

The following patch(es) are duplicate patches with patches installed in the Oracle Home.
 [ 19769480]
You have already installed same patch(es) with same UPI(s) or same version(s).
These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /products/oracle/12.1.0.2
Patch Validation: Successful
Grid Infrastructure home:
/products/grid/12.1.0.2
DB home(s):
/products/oracle/12.1.0.2

Analyzing patch(es) on "/products/oracle/12.1.0.2" ...
Patch "/media/sf_install/patches/23273629/23054246" analyzed on "/products/oracle/12.1.0.2" with warning for apply.
Patch "/media/sf_install/patches/23273629/23054327" analyzed on "/products/oracle/12.1.0.2" with warning for apply.

Analyzing patch(es) on "/products/grid/12.1.0.2" ...
Patch "/media/sf_install/patches/23273629/21436941" analyzed on "/products/grid/12.1.0.2" with warning for apply.
Patch "/media/sf_install/patches/23273629/23054246" analyzed on "/products/grid/12.1.0.2" with warning for apply.
Patch "/media/sf_install/patches/23273629/23054327" analyzed on "/products/grid/12.1.0.2" with warning for apply.
Patch "/media/sf_install/patches/23273629/23054341" analyzed on "/products/grid/12.1.0.2" with warning for apply.

[WARNING] The local database instance 'orcl1' from '/products/oracle/12.1.0.2' is not running. SQL changes, if any,  will not be analyzed. Please refer to the log file for more details.

Apply Summary:

opatchauto ran into some warnings during analyze (Please see log file for details):
GI Home: /products/grid/12.1.0.2: 21436941,23054246,23054327,23054341
DB Home: /products/oracle/12.1.0.2: 23054246,23054327

opatchauto completed with warnings. 

OK. so I have warnings, we know warnings are not Errors.

- the analyze.log
2016-07-22_10-09-39 :
Executing command:
/bin/bash -c 'ORACLE_HOME=/products/oracle/12.1.0.2 ORACLE_SID=orcl1 /products/oracle/12.1.0.2/OPatch/datapatch -prereq'

[WARNING] The local database instance 'orcl1' from '/products/oracle/12.1.0.2' is not running. SQL changes, if any,  will not be analyzed.
To analyze the SQL changes, bring up the database instance and run the above command manually from any one node (run as oracle). 

the analyze.debug.log looks good too..

- the opatch.log for grid:

[Jul 22, 2016 10:09:37 AM]   Composite patch 23054246 successfully applied.
[Jul 22, 2016 10:09:37 AM]   Patches 21436941,23054327,23054341 successfully applied.
[Jul 22, 2016 10:09:37 AM]   UtilSession: N-Apply done.
[Jul 22, 2016 10:09:37 AM]   --------------------------------------------------------------------------------
[Jul 22, 2016 10:09:37 AM]   The following warnings have occurred during OPatch execution:
[Jul 22, 2016 10:09:37 AM]   1) OUI-67303:
                             Patches [   20831113   20299018   19872484 ] will be rolled back.
[Jul 22, 2016 10:09:37 AM]   --------------------------------------------------------------------------------
[Jul 22, 2016 10:09:37 AM]   OUI-67008:OPatch Session completed with warnings.
[Jul 22, 2016 10:09:37 AM]   Finishing UtilSession at Fri Jul 22 10:09:37 CEST 2016

will be rolled back.. ok, the end of the opatch.log for grid looks nice

- the opatch.log for oracle home:

[Jul 22, 2016 10:09:09 AM]   Composite patch 23054246 successfully applied.
[Jul 22, 2016 10:09:09 AM]   Patch 23054327 successfully applied.
[Jul 22, 2016 10:09:09 AM]   UtilSession: N-Apply done.
[Jul 22, 2016 10:09:09 AM]   --------------------------------------------------------------------------------
[Jul 22, 2016 10:09:09 AM]   The following warnings have occurred during OPatch execution:
[Jul 22, 2016 10:09:09 AM]   1) OUI-67303:
                             Patches [   20831113 ] will be rolled back.
[Jul 22, 2016 10:09:09 AM]   --------------------------------------------------------------------------------
[Jul 22, 2016 10:09:09 AM]   OUI-67008:OPatch Session completed with warnings.

– looks good…

at this point I am sure everything is correct for the apply, let’s apply

[root@standby1 opatch]# /products/grid/12.1.0.2/OPatch/opatchauto apply /media/sf_install/patches/23273629 -ocmrf /products/ocm.rsp
OPatch Automation Tool
Copyright (c)2014, Oracle Corporation. All rights reserved.

OPatchauto Version : 12.1.0.1.10
OUI Version        : 12.1.0.2.0
Running from       : /products/grid/12.1.0.2

opatchauto log file: /products/grid/12.1.0.2/cfgtoollogs/opatchauto/23273629/opatch_gi_2016-07-22_10-22-16_deploy.log

Parameter Validation: Successful

Configuration Validation: Successful

Patch Location: /media/sf_install/patches/23273629
Grid Infrastructure Patch(es): 21436941 23054246 23054327 23054341
DB Patch(es): 23054246 23054327 

The following patch(es) are duplicate patches with patches installed in the Oracle Home.
 [ 20299023]
You have already installed same patch(es) with same UPI(s) or same version(s).
These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /products/grid/12.1.0.2

The following patch(es) are duplicate patches with patches installed in the Oracle Home.
 [ 20831110]
You have already installed same patch(es) with same UPI(s) or same version(s).
These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /products/grid/12.1.0.2

The following patch(es) are duplicate patches with patches installed in the Oracle Home.
 [ 19769480]
You have already installed same patch(es) with same UPI(s) or same version(s).
These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /products/grid/12.1.0.2

The following patch(es) are duplicate patches with patches installed in the Oracle Home.
 [ 20299023]
You have already installed same patch(es) with same UPI(s) or same version(s).
These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /products/oracle/12.1.0.2

The following patch(es) are duplicate patches with patches installed in the Oracle Home.
 [ 20831110]
You have already installed same patch(es) with same UPI(s) or same version(s).
These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /products/oracle/12.1.0.2

The following patch(es) are duplicate patches with patches installed in the Oracle Home.
 [ 19769480]
You have already installed same patch(es) with same UPI(s) or same version(s).
These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /products/oracle/12.1.0.2
Patch Validation: Successful
Grid Infrastructure home:
/products/grid/12.1.0.2
DB home(s):
/products/oracle/12.1.0.2

Performing prepatch operations on CRS Home... Successful

Applying patch(es) to "/products/oracle/12.1.0.2" ...
Command "/products/oracle/12.1.0.2/OPatch/opatch napply -phBaseFile /tmp/OraDB12Home1_oracle_patchList -local  -invPtrLoc /products/grid/12.1.0.2/oraInst.loc -oh /products/oracle/12.1.0.2 -silent -ocmrf /products/ocm.rsp" execution failed:
OPatch failed to restore OH '/products/oracle/12.1.0.2'. Consult OPatch document to restore the home manually before proceeding.
UtilSession failed: Re-link fails on target "isqora".

Log file Location for the failed command: /products/oracle/12.1.0.2/cfgtoollogs/opatch/opatch2016-07-22_10-25-57AM_1.log

For more details, please refer to the log file "/products/grid/12.1.0.2/cfgtoollogs/opatchauto/23273629/opatch_gi_2016-07-22_10-22-16_deploy.debug.log".

Apply Summary:
Following patch(es) are successfully installed:
DB Home: /products/oracle/12.1.0.2:

Following patch(es) failed to be installed:
GI Home: /products/grid/12.1.0.2: 21436941,23054246,23054327,23054341
DB Home: /products/oracle/12.1.0.2: 23054246,23054327
Command failure exception

opatchauto failed with error code 2.
[root@standby1 opatch]# 

Failed again… OK so check the log…

[Jul 22, 2016 10:28:57 AM]   OUI-67200:Make failed to invoke "/usr/bin/make -f ins_odbc.mk isqora ORACLE_HOME=/products/oracle/12.1.0.2"....'/usr/bin/ld: cannot find -lodbcinst
                             collect2: ld returned 1 exit status
                             make: *** [/products/oracle/12.1.0.2/odbc/lib/libsqora.so.12.1] Error 1
                             '
[Jul 22, 2016 10:28:57 AM]   Stack Description: java.lang.RuntimeException: /usr/bin/ld: cannot find -lodbcinst
                             collect2: ld returned 1 exit status
                             make: *** [/products/oracle/12.1.0.2/odbc/lib/libsqora.so.12.1] Error 1
[Jul 22, 2016 10:28:57 AM]   StackTrace: oracle.opatch.MakeAction.apply(MakeAction.java:530)
[Jul 22, 2016 10:28:57 AM]   StackTrace: oracle.opatch.OPatchSession.runMake(OPatchSession.java:730)
[Jul 22, 2016 10:28:57 AM]   StackTrace: oracle.opatch.MergedPatchObject.invokeConsolidatedMake(MergedPatchObject.java:1784)
[Jul 22, 2016 10:28:57 AM]   StackTrace: oracle.opatch.opatchutil.NApply.process(NApply.java:1268)
[Jul 22, 2016 10:28:57 AM]   StackTrace: oracle.opatch.opatchutil.OUSession.napply(OUSession.java:1089)
[Jul 22, 2016 10:28:57 AM]   StackTrace: sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[Jul 22, 2016 10:28:57 AM]   StackTrace: sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[Jul 22, 2016 10:28:57 AM]   StackTrace: sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[Jul 22, 2016 10:28:57 AM]   StackTrace: java.lang.reflect.Method.invoke(Method.java:597)
[Jul 22, 2016 10:28:57 AM]   StackTrace: oracle.opatch.UtilSession.process(UtilSession.java:317)
[Jul 22, 2016 10:28:57 AM]   StackTrace: oracle.opatch.OPatchSession.main(OPatchSession.java:2585)
[Jul 22, 2016 10:28:57 AM]   StackTrace: oracle.opatch.OPatch.main(OPatch.java:634)
[Jul 22, 2016 10:28:57 AM]   OUI-67050:Running make for target install_srvm
[Jul 22, 2016 10:28:57 AM]   Start invoking 'make' at Fri Jul 22 10:28:57 CEST 2016Fri Jul 22 10:28:57 CEST 2016
[Jul 22, 2016 10:28:57 AM]   Finish invoking 'make' at Fri Jul 22 10:28:57 CEST 2016
[Jul 22, 2016 10:28:57 AM]   OUI-67050:Running make for target install
[Jul 22, 2016 10:28:57 AM]   Start invoking 'make' at Fri Jul 22 10:28:57 CEST 2016Fri Jul 22 10:28:57 CEST 2016
[Jul 22, 2016 10:28:57 AM]   Finish invoking 'make' at Fri Jul 22 10:28:57 CEST 2016
[Jul 22, 2016 10:28:57 AM]   The following make actions have failed :
[Jul 22, 2016 10:28:57 AM]   OUI-67124:Re-link fails on target "isqora".
[Jul 22, 2016 10:28:57 AM]   Do you want to proceed? [y|n]
[Jul 22, 2016 10:29:00 AM]   N (auto-answered by -silent)
[Jul 22, 2016 10:29:00 AM]   User Responded with: N
[Jul 22, 2016 10:29:30 AM]   Re-link fails on target "isqora".
[Jul 22, 2016 10:29:30 AM]   Running make for target "install_srvm"
[Jul 22, 2016 10:29:30 AM]   Start invoking 'make' at Fri Jul 22 10:29:30 CEST 2016Fri Jul 22 10:29:30 CEST 2016
[Jul 22, 2016 10:29:30 AM]   Finish invoking 'make' at Fri Jul 22 10:29:30 CEST 2016
[Jul 22, 2016 10:29:30 AM]   Running make for target "install"
[Jul 22, 2016 10:29:30 AM]   Start invoking 'make' at Fri Jul 22 10:29:30 CEST 2016Fri Jul 22 10:29:30 CEST 2016
[Jul 22, 2016 10:29:30 AM]   Finish invoking 'make' at Fri Jul 22 10:29:30 CEST 2016
[Jul 22, 2016 10:29:30 AM]   --------------------------------------------------------------------------------
                             Failed to run make commands. Please contact Oracle Support.
[Jul 22, 2016 10:29:30 AM]   OUI-67115:OPatch failed to restore OH '/products/oracle/12.1.0.2'. Consult OPatch document to restore the home manually before proceeding.
[Jul 22, 2016 10:29:30 AM]   OUI-67124:
                             NApply was not able to restore the home.  Please invoke the following scripts:
                               - restore.[sh,bat]
                               - make.txt (Unix only)
                             to restore the ORACLE_HOME.  They are located under
                             "/products/oracle/12.1.0.2/.patch_storage/NApply/2016-07-22_10-25-57AM"
[Jul 22, 2016 10:29:30 AM]   OUI-67073:UtilSession failed: Re-link fails on target "isqora".
[Jul 22, 2016 10:29:30 AM]   --------------------------------------------------------------------------------
[Jul 22, 2016 10:29:30 AM]   The following warnings have occurred during OPatch execution:
[Jul 22, 2016 10:29:30 AM]   1) OUI-67303:
                             Patches [   20831113 ] will be rolled back.
[Jul 22, 2016 10:29:30 AM]   2) OUI-67200:Make failed to invoke "/usr/bin/make -f ins_odbc.mk isqora ORACLE_HOME=/products/oracle/12.1.0.2"....'/usr/bin/ld: cannot find -lodbcinst
                             collect2: ld returned 1 exit status
                             make: *** [/products/oracle/12.1.0.2/odbc/lib/libsqora.so.12.1] Error 1
                             '
[Jul 22, 2016 10:29:30 AM]   3) OUI-67124:Re-link fails on target "isqora".
[Jul 22, 2016 10:29:30 AM]   4) OUI-67200:Make failed to invoke "/usr/bin/make -f ins_odbc.mk isqora ORACLE_HOME=/products/oracle/12.1.0.2"....'/usr/bin/ld: cannot find -lodbcinst
                             collect2: ld returned 1 exit status
                             make: *** [/products/oracle/12.1.0.2/odbc/lib/libsqora.so.12.1] Error 1
                             '
[Jul 22, 2016 10:29:30 AM]   5) OUI-67124:
                             NApply was not able to restore the home.  Please invoke the following scripts:
                               - restore.[sh,bat]
                               - make.txt (Unix only)
                             to restore the ORACLE_HOME.  They are located under
                             "/products/oracle/12.1.0.2/.patch_storage/NApply/2016-07-22_10-25-57AM"
[Jul 22, 2016 10:29:30 AM]   --------------------------------------------------------------------------------

ok, let’s fix my Oracle Home

[oracle@standby1 ~]$ cd /products/oracle/12.1.0.2/.patch_storage/NApply/2016-07-22_10-25-57AM
[oracle@standby1 2016-07-22_10-25-57AM]$ ll
total 16
drwxr-xr-x. 16 oracle oinstall 4096 Jul 22 10:26 backup
-rw-r--r--.  1 oracle oinstall 2094 Jul 22 10:28 make.txt
-rw-r--r--.  1 oracle oinstall  976 Jul 22 10:28 patchlist.txt
-rwx--x---.  1 oracle oinstall 1601 Jul 22 10:26 restore.sh
[oracle@standby1 2016-07-22_10-25-57AM]$ cat restore.sh
#!/bin/sh
# Copyright (c) 2016, Oracle Corporation.  All rights reserved

echo This script is going to restore the Oracle Home to the previous state.
echo It does not perform any of the following:
echo - Running init/pre/post scripts
echo - Oracle binary re-link
echo - Customized steps performed manually by user
echo Please use this script with supervision from Oracle Technical Support.

# Get ORACLE_HOME from environment variable "ORACLE_HOME"
OH=${ORACLE_HOME}

# Error out if OH is not set
if [ "$OH" = "" ]; then
    echo Oracle Home is not set.
    echo Please set env. variable ORACLE_HOME and try again.
    echo Script failed to proceed.
    exit 1
fi

echo "About to modify Oracle Home( $OH )"
echo "Do you want to proceed? [Y/N]"
if [ "$1" = "-silent" ]; then
   response="Y"
else
    read response;
fi

if [ $response = "y" ] || [ $response = "Y" ]; then
    echo "User responded with : Y"
else
    echo "User responded with : $response"
    exit 0
fi

echo Date   Time : `date` >> $OH/cfgtoollogs/opatch/opatch_history.txt
echo Oracle Home : $OH >> $OH/cfgtoollogs/opatch/opatch_history.txt
echo Command     : restore.sh >> $OH/cfgtoollogs/opatch/opatch_history.txt

\rm -rf $OH/.patch_storage/NApply/2016-07-22_10-25-57AM/files

cd /products/oracle/12.1.0.2/inventory/oneoffs
\rm -rf 21359755

\rm -rf 21948354

\rm -rf 22291127

\rm -rf 23054246

\rm -rf 23054327

cd $OH/.patch_storage/NApply/2016-07-22_10-25-57AM/backup
for dname in "`\ls -1A`" ; do
    \cp -fR $dname $OH ;
done

\rm -rf $OH/.patch_storage/NApply/2016-07-22_10-25-57AM/files

echo Restore script completed.
[oracle@standby1 2016-07-22_10-25-57AM]$ . oraenv
ORACLE_SID = [oracle] ? orcl1
The Oracle base has been set to /products/oracle
[oracle@standby1 2016-07-22_10-25-57AM]$ ./restore.sh
This script is going to restore the Oracle Home to the previous state.
It does not perform any of the following:
- Running init/pre/post scripts
- Oracle binary re-link
- Customized steps performed manually by user
Please use this script with supervision from Oracle Technical Support.
About to modify Oracle Home( /products/oracle/12.1.0.2 )
Do you want to proceed? [Y/N]
y
User responded with : Y
Restore script completed.
[oracle@standby1 2016-07-22_10-25-57AM]$ ls -ltr
total 16
-rwx--x---.  1 oracle oinstall 1601 Jul 22 10:26 restore.sh
drwxr-xr-x. 16 oracle oinstall 4096 Jul 22 10:26 backup
-rw-r--r--.  1 oracle oinstall  976 Jul 22 10:28 patchlist.txt
-rw-r--r--.  1 oracle oinstall 2094 Jul 22 10:28 make.txt
[oracle@standby1 2016-07-22_10-25-57AM]$ vi make.txt
[oracle@standby1 2016-07-22_10-25-57AM]$ cat make.txt
cd $ORACLE_HOME/rdbms/lib;/usr/bin/make -f ins_rdbms.mk iextjobo ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/rdbms/lib;/usr/bin/make -f ins_rdbms.mk iextjob ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/rdbms/lib;/usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/rdbms/lib;/usr/bin/make -f ins_rdbms.mk libasmclntsh12.ohso ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/rdbms/lib;/usr/bin/make -f ins_rdbms.mk client_sharedlib ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/rdbms/lib;/usr/bin/make -f ins_rdbms.mk ikfed ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/network/lib;/usr/bin/make -f ins_net_client.mk client_sharedlib ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/rdbms/lib;/usr/bin/make -f ins_rdbms.mk iorion ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/ldap/lib;/usr/bin/make -f ins_ldap.mk ldapmoddn ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/ldap/lib;/usr/bin/make -f ins_ldap.mk ldapmodifymt ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/ldap/lib;/usr/bin/make -f ins_ldap.mk ldapcompare ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/ldap/lib;/usr/bin/make -f ins_ldap.mk ldapbind ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/ldap/lib;/usr/bin/make -f ins_ldap.mk ldapsearch ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/ldap/lib;/usr/bin/make -f ins_ldap.mk ldapaddmt ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/ldap/lib;/usr/bin/make -f ins_ldap.mk ldapmodify ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/ldap/lib;/usr/bin/make -f ins_ldap.mk ldapdelete ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/ldap/lib;/usr/bin/make -f ins_ldap.mk ldapadd ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/rdbms/lib;/usr/bin/make -f ins_rdbms.mk patchset_opt_all ioracle ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/rdbms/lib;/usr/bin/make -f ins_rdbms.mk irman ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/ctx/lib;/usr/bin/make -f ins_ctx.mk ictxhx ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/odbc/lib;/usr/bin/make -f ins_odbc.mk isqora ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/srvm/lib;/usr/bin/make -f ins_srvm.mk install_srvm ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/racg/lib;/usr/bin/make -f ins_has.mk install ORACLE_HOME=$ORACLE_HOME
[oracle@standby1 2016-07-22_10-25-57AM]$ chmod 755 make.txt
[oracle@standby1 2016-07-22_10-25-57AM]$ ./make.txt

I will not paste it here, it’s a relink like I said

[oracle@standby1 12.1.0.2]$ setasmgid oracle_binary_path=/products/oracle/12.1.0.2/bin/oracle
KFSG-00302: group '' not found in operating system dictionary
[oracle@standby1 12.1.0.2]$ su - grid
Password:
[grid@standby1 ~]$ . oraenv
ORACLE_SID = [grid] ? +ASM1
The Oracle base has been set to /products/oracle
[grid@standby1 ~]$ setasmgid oracle_binary_path=/products/oracle/12.1.0.2/bin/oracle
KFSG-00304: this executable must run as SETUID root
[grid@standby1 ~]$ su
Password:
[root@standby1 grid]# . oraenv
ORACLE_SID = [+ASM1] ?
The Oracle base remains unchanged with value /products/oracle
[root@standby1 grid]# setasmgid oracle_binary_path=/products/oracle/12.1.0.2/bin/oracle
[root@standby1 grid]# ls -l /products/oracle/12.1.0.2/bin/oracle
-rwsr-s--x. 1 oracle asmadmin 323791685 Jul 22 10:35 /products/oracle/12.1.0.2/bin/oracle

as the patch process unlocks the grid infrastructure and later fails I have to do the post tasks at patching manually

[root@standby1 grid]# cd /products/grid/12.1.0.2/rdbms/install/
[root@standby1 install]# ll
total 72
-rwxr-x---. 1 grid oinstall  7864 Feb 26  2012 clonedb.pl
-rw-r--r--. 1 grid oinstall  1694 Dec 19  2003 config_filemap.sbs
-rw-r--r--. 1 grid oinstall  1714 May  2 13:42 filemap.ora
-rw-r--r--. 1 grid oinstall   231 Jun  6  2013 install.excl
drwxr-xr-x. 2 grid oinstall  4096 May  2 13:39 rdbms
-rwxr--r--. 1 grid oinstall  1092 May  2 13:42 rootadd_filemap.sh
-rwxr-x---. 1 grid oinstall  2988 May  2 13:41 rootadd_rdbms.sh
-rwxr-x---. 1 grid oinstall  5658 Jun 17  2011 s7x_workflow.akwf
drwxr-xr-x. 2 grid oinstall  4096 May  2 13:40 sbs
-rwxr-x---. 1 grid oinstall 25387 Oct 23  2013 setup_processor_group.sh
[root@standby1 install]# ./rootadd_rdbms.sh
[root@standby1 install]# cd ../..
[root@standby1 12.1.0.2]# cd crs/
auth/     config/   demo/     init/     install/  lib/      log/      mesg/     profile/  public/   sbs/      script/   template/ trace/    utl/
[root@standby1 12.1.0.2]# cd crs/install/
[root@standby1 install]# ./rootcrs.sh -patch
Using configuration parameter file: /products/grid/12.1.0.2/crs/install/crsconfig_params
2016/07/22 10:40:14 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.

2016/07/22 10:40:27 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.

CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'standby1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'standby1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'standby1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'standby1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'standby1'
CRS-2672: Attempting to start 'ora.evmd' on 'standby1'
CRS-2676: Start of 'ora.mdnsd' on 'standby1' succeeded
CRS-2676: Start of 'ora.evmd' on 'standby1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'standby1'
CRS-2676: Start of 'ora.gpnpd' on 'standby1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'standby1'
CRS-2676: Start of 'ora.gipcd' on 'standby1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'standby1'
CRS-2676: Start of 'ora.cssdmonitor' on 'standby1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'standby1'
CRS-2672: Attempting to start 'ora.diskmon' on 'standby1'
CRS-2676: Start of 'ora.diskmon' on 'standby1' succeeded
CRS-2676: Start of 'ora.cssd' on 'standby1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'standby1'
CRS-2672: Attempting to start 'ora.ctssd' on 'standby1'
CRS-2676: Start of 'ora.ctssd' on 'standby1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'standby1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'standby1'
CRS-2676: Start of 'ora.asm' on 'standby1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'standby1'
CRS-2676: Start of 'ora.storage' on 'standby1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'standby1'
CRS-2676: Start of 'ora.crf' on 'standby1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'standby1'
CRS-2676: Start of 'ora.crsd' on 'standby1' succeeded
CRS-6017: Processing resource auto-start for servers: standby1
CRS-2672: Attempting to start 'ora.net1.network' on 'standby1'
CRS-2676: Start of 'ora.net1.network' on 'standby1' succeeded
CRS-2672: Attempting to start 'ora.ons' on 'standby1'
CRS-2673: Attempting to stop 'ora.standby1.vip' on 'standby2'
CRS-2677: Stop of 'ora.standby1.vip' on 'standby2' succeeded
CRS-2672: Attempting to start 'ora.standby1.vip' on 'standby1'
CRS-2676: Start of 'ora.ons' on 'standby1' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'standby2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'standby2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'standby2'
CRS-2676: Start of 'ora.standby1.vip' on 'standby1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'standby1'
CRS-2677: Stop of 'ora.scan1.vip' on 'standby2' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'standby1'
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'standby1' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'standby1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'standby1'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'standby1' succeeded
CRS-6016: Resource auto-start has completed for server standby1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [459864538].

voila, the node is again running … and opatchauto does not work, because something looks to miss

Advertisements

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s