Posted in 2016

opatchauto works (?)

Hi, as I have changed my position, now working in Vienna with a RAC (I am so happy), I have started to focus my work on RAC administration. As a DBA I have to patch. Well, in single instances it was simple, stop everything, opatch apply, start everything. In a RAC it’s much more, finally it looks that opatchauto in OPatchauto Version : 12.1.0.1.10 works and you don’t need to patch manually as written in Doc ID 1591616.1.

The database was running and all I had to do is to start this command (if you don’t have ocm.rsp generate it)

[root@rac2-pub oracle]# opatchauto apply /media/sf_install/20996835 -ocmrf /u01/app/grid/12.1.0.2/OPatch/ocm.rsp 


OPatch Automation Tool
Copyright (c)2014, Oracle Corporation. All rights reserved.

OPatchauto Version : 12.1.0.1.10
OUI Version        : 12.1.0.2.0
Running from       : /u01/app/grid/12.1.0.2

opatchauto log file: /u01/app/grid/12.1.0.2/cfgtoollogs/opatchauto/20996835/opatch_gi_2016-04-25_16-54-20_deploy.log

Parameter Validation: Successful

Configuration Validation: Successful

Patch Location: /media/sf_install/20996835
Grid Infrastructure Patch(es): 20831110 19872484 20299018 20831113 
DB Patch(es): 20831110 20831113 

Patch Validation: Successful
Grid Infrastructure home:
/u01/app/grid/12.1.0.2
DB home(s):
/u01/app/oracle/product/12.1.0.2


Performing prepatch operations on CRS Home... Successful

Applying patch(es) to "/u01/app/oracle/product/12.1.0.2" ...
Patch "/media/sf_install/20996835/20831110" successfully applied to "/u01/app/oracle/product/12.1.0.2".
Patch "/media/sf_install/20996835/20831113" successfully applied to "/u01/app/oracle/product/12.1.0.2".

Applying patch(es) to "/u01/app/grid/12.1.0.2" ...
Patch "/media/sf_install/20996835/20831110" successfully applied to "/u01/app/grid/12.1.0.2".
Patch "/media/sf_install/20996835/19872484" successfully applied to "/u01/app/grid/12.1.0.2".
Patch "/media/sf_install/20996835/20299018" successfully applied to "/u01/app/grid/12.1.0.2".
Patch "/media/sf_install/20996835/20831113" successfully applied to "/u01/app/grid/12.1.0.2".

Performing postpatch operations on CRS Home... Successful

SQL changes, if any, are applied successfully on the following database(s): orcl

Apply Summary:
Following patch(es) are successfully installed:
GI Home: /u01/app/grid/12.1.0.2: 20831110,19872484,20299018,20831113
DB Home: /u01/app/oracle/product/12.1.0.2: 20831110,20831113

opatchauto succeeded.

As you can see it works and datapatch is called too… Does it look like opatchauto works?
So I wanted to test an apply of the newest PSU as it’s new and my current patchlist is what’s in production.
But it filled all of my 11GB free space in my root directory and that is definifely NOT OK, too much of .patch_storage saved I think. The PSU has only 1,2GB as I can remember and if you don’t have 8,5GB free on filesystem the first prechecks fail, the script fails at the start, you don’t get to the “Performing prepatch operations on CRS Home…” operation.

Now after reset to my snapshot something is wrong, I did this snapshot after my database was installed, no patching was ever done in that state.

On both nodes:

[root@rac2-pub ~]# crsctl query crs activeversion -f
Oracle Clusterware active version on the cluster is [12.1.0.2.0]. 
The cluster upgrade state is [ROLLING PATCH]. 
The cluster active patch level is [459864538].

And because of that my reapply fails (I want to have Apr2015 PSU) so I call:

[root@rac1-pub ~]# opatchauto apply /media/sf_install/patches/20996835 -ocmrf ./ocm.rsp 
OPatch Automation Tool
Copyright (c)2014, Oracle Corporation. All rights reserved.

OPatchauto Version : 12.1.0.1.10
OUI Version        : 12.1.0.2.0
Running from       : /u01/app/grid/12.1.0.2

opatchauto log file: /u01/app/grid/12.1.0.2/cfgtoollogs/opatchauto/20996835/opatch_gi_2016-04-26_14-48-59_deploy.log

Parameter Validation: Successful

Configuration Validation: Successful

Patch Location: /media/sf_install/patches/20996835
Grid Infrastructure Patch(es): 20831110 19872484 20299018 20831113 
DB Patch(es): 20831110 20831113 

Patch Validation: Successful
Grid Infrastructure home:
/u01/app/grid/12.1.0.2
DB home(s):
/u01/app/oracle/product/12.1.0.2


Performing prepatch operations on CRS Home... Failed
Command "/u01/app/grid/12.1.0.2/perl/bin/perl -I/u01/app/grid/12.1.0.2/perl/lib 
-I/u01/app/grid/12.1.0.2/OPatch/opatchautotemp_rac1-pub/patchwork/crs/install 
/u01/app/grid/12.1.0.2/OPatch/opatchautotemp_rac1-pub/patchwork/crs/install/rootcrs.pl -prepatch" execution failed: 
Died at /u01/app/grid/12.1.0.2/OPatch/opatchautotemp_rac1-pub/patchwork/crs/install/crspatch.pm line 773.


For more details, please refer to the log file 
"/u01/app/grid/12.1.0.2/cfgtoollogs/opatchauto/20996835/opatch_gi_2016-04-26_14-48-59_deploy.debug.log".

Apply Summary:

Following patch(es) failed to be installed:
GI Home: /u01/app/grid/12.1.0.2: 20831110,19872484,20299018,20831113
DB Home: /u01/app/oracle/product/12.1.0.2: 20831110,20831113
Command failure exception

opatchauto failed with error code 2.

In the deploy log I found some specific errors which took me to:

CRS-1153: There was an error setting Oracle Clusterware to rolling patch mode. (Doc ID 1943498.1)

BUT the workaround is not working, now I have my software in an unknown (for me) state, I don’t know what to do

[root@rac1-pub ~]# crsctl stop rollingpatch
CRS-1170: Rejecting rolling patch mode change because the patch has not been applied yet. 
The software patch level [459864538] on this node is not the same as the expected patch level [0].
CRS-4000: Command Stop failed, or completed with errors.

Maybe the problem is at storage level, because I have iscsi initiators and all these internal scripts maybe check something in the database or somewhere only Oracle knows (because noone checks all the scripts). Maybe that’s the reason. But I like to install my RAC 10000 times, so I will resize my root mount point to 100GB (from 44GB)

Advertisements