Posted in 2016

My DNS for a RAC

zone "example.com" IN {
	type master;
	file "example.zone";
};

zone "1.168.192.in-addr.arpa" IN {
	type master;
	file "reverse-1.rev";
};

zone "56.168.192.in-addr.arpa" IN {
        type master;
        file "reverse-56.rev";
};

[root@test ~]# cat /var/named/example.zone 
$TTL 1D
@	IN SOA	@ example.com. (
					100	; serial
					1D	; refresh
					1H	; retry
					1W	; expire
					3H )	; minimum
	NS	test
	A	192.168.56.254
	AAAA	::1

test		IN A 192.168.56.254
rac1-pub	IN A 192.168.56.101
rac2-pub	IN A 192.168.56.102
rac1-priv	IN A 192.168.1.101
rac2-priv	IN A 192.168.1.102
rac1-vip	IN A 192.168.56.201
rac2-vip	IN A 192.168.56.202
rac-scan	IN A 192.168.56.10
rac-scan	IN A 192.168.56.20
rac-scan	IN A 192.168.56.30

[root@test ~]# cat /var/named/reverse-1.rev 
$TTL 3D
1.168.192.in-addr.arpa. IN SOA test.example.com. example.com. ( 
	1 ;Serial number 
	10800 ;Refresh 
	3600 ;Retry 
	604800 ;Expire 
	86400) ;Minimum TTL

	NS example.com.
101 	IN PTR rac1-priv.example.com.
102 	IN PTR rac2-priv.example.com.

[root@test ~]# cat /var/named/reverse-56.rev 
$TTL 3D
56.168.192.in-addr.arpa. IN SOA test.example.com. example.com. ( 
	1 ;Serial number 
	10800 ;Refresh 
	3600 ;Retry 
	604800 ;Expire 
	86400) ;Minimum TTL

	NS example.com.
10 IN PTR rac-scan.example.com.
20 IN PTR rac-scan.example.com.
30 IN PTR rac-scan.example.com.
101 IN PTR rac1-pub.example.com.
102 IN PTR rac2-pub.example.com.
201 IN PTR rac1-vip.example.com.
202 IN PTR rac2-vip.example.com.
254 IN PTR test.example.com.
Advertisements
Posted in 2016

opatchauto failed again, apply of GI+PSU tested on 1 RAC node

The Problem is in the PSU, it was replaced with a fix. But it’s interresting anyway so let’s check it..

[oracle@standby1 ~]$ df -h
Filesystem                   Size  Used Avail Use% Mounted on
/dev/mapper/vg_test-lv_root   44G   18G   24G  44% /
tmpfs                        1.5G   72K  1.5G   1% /dev/shm
/dev/sda1                    477M   55M  397M  13% /boot
/dev/sdc1                     50G   19G   29G  39% /products
install                      985G  505G  480G  52% /media/sf_install

I have enough space for the patches and it’s not under root, then it would fill up the filesystem and crash

[root@standby1 ~]# . oraenv
ORACLE_SID = [orcl1] ? +ASM1
The Oracle base remains unchanged with value /products/oracle
[root@standby1 ~]# /products/grid/12.1.0.2/OPatch/opatchauto apply /media/sf_install/patches/23273629 -analyze -ocmrf /products/ocm.rsp
OPatch Automation Tool
Copyright (c)2014, Oracle Corporation. All rights reserved.

OPatchauto Version : 12.1.0.1.10
OUI Version        : 12.1.0.2.0
Running from       : /products/grid/12.1.0.2

opatchauto log file: /products/grid/12.1.0.2/cfgtoollogs/opatchauto/23273629/opatch_gi_2016-07-22_10-07-29_analyze.log

NOTE: opatchauto is running in ANALYZE mode. There will be no change to your system.

OCM RSP file has been ignored in analyze mode. 

Parameter Validation: Successful

Configuration Validation: Successful

Patch Location: /media/sf_install/patches/23273629
Grid Infrastructure Patch(es): 21436941 23054246 23054327 23054341
DB Patch(es): 23054246 23054327 

The following patch(es) are duplicate patches with patches installed in the Oracle Home.
 [ 20299023]
You have already installed same patch(es) with same UPI(s) or same version(s).
These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /products/grid/12.1.0.2

The following patch(es) are duplicate patches with patches installed in the Oracle Home.
 [ 20831110]
You have already installed same patch(es) with same UPI(s) or same version(s).
These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /products/grid/12.1.0.2

The following patch(es) are duplicate patches with patches installed in the Oracle Home.
 [ 19769480]
You have already installed same patch(es) with same UPI(s) or same version(s).
These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /products/grid/12.1.0.2

The following patch(es) are duplicate patches with patches installed in the Oracle Home.
 [ 20299023]
You have already installed same patch(es) with same UPI(s) or same version(s).
These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /products/oracle/12.1.0.2

The following patch(es) are duplicate patches with patches installed in the Oracle Home.
 [ 20831110]
You have already installed same patch(es) with same UPI(s) or same version(s).
These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /products/oracle/12.1.0.2

The following patch(es) are duplicate patches with patches installed in the Oracle Home.
 [ 19769480]
You have already installed same patch(es) with same UPI(s) or same version(s).
These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /products/oracle/12.1.0.2
Patch Validation: Successful
Grid Infrastructure home:
/products/grid/12.1.0.2
DB home(s):
/products/oracle/12.1.0.2

Analyzing patch(es) on "/products/oracle/12.1.0.2" ...
Patch "/media/sf_install/patches/23273629/23054246" analyzed on "/products/oracle/12.1.0.2" with warning for apply.
Patch "/media/sf_install/patches/23273629/23054327" analyzed on "/products/oracle/12.1.0.2" with warning for apply.

Analyzing patch(es) on "/products/grid/12.1.0.2" ...
Patch "/media/sf_install/patches/23273629/21436941" analyzed on "/products/grid/12.1.0.2" with warning for apply.
Patch "/media/sf_install/patches/23273629/23054246" analyzed on "/products/grid/12.1.0.2" with warning for apply.
Patch "/media/sf_install/patches/23273629/23054327" analyzed on "/products/grid/12.1.0.2" with warning for apply.
Patch "/media/sf_install/patches/23273629/23054341" analyzed on "/products/grid/12.1.0.2" with warning for apply.

[WARNING] The local database instance 'orcl1' from '/products/oracle/12.1.0.2' is not running. SQL changes, if any,  will not be analyzed. Please refer to the log file for more details.

Apply Summary:

opatchauto ran into some warnings during analyze (Please see log file for details):
GI Home: /products/grid/12.1.0.2: 21436941,23054246,23054327,23054341
DB Home: /products/oracle/12.1.0.2: 23054246,23054327

opatchauto completed with warnings. 

OK. so I have warnings, we know warnings are not Errors.

- the analyze.log
2016-07-22_10-09-39 :
Executing command:
/bin/bash -c 'ORACLE_HOME=/products/oracle/12.1.0.2 ORACLE_SID=orcl1 /products/oracle/12.1.0.2/OPatch/datapatch -prereq'

[WARNING] The local database instance 'orcl1' from '/products/oracle/12.1.0.2' is not running. SQL changes, if any,  will not be analyzed.
To analyze the SQL changes, bring up the database instance and run the above command manually from any one node (run as oracle). 

the analyze.debug.log looks good too..

- the opatch.log for grid:

[Jul 22, 2016 10:09:37 AM]   Composite patch 23054246 successfully applied.
[Jul 22, 2016 10:09:37 AM]   Patches 21436941,23054327,23054341 successfully applied.
[Jul 22, 2016 10:09:37 AM]   UtilSession: N-Apply done.
[Jul 22, 2016 10:09:37 AM]   --------------------------------------------------------------------------------
[Jul 22, 2016 10:09:37 AM]   The following warnings have occurred during OPatch execution:
[Jul 22, 2016 10:09:37 AM]   1) OUI-67303:
                             Patches [   20831113   20299018   19872484 ] will be rolled back.
[Jul 22, 2016 10:09:37 AM]   --------------------------------------------------------------------------------
[Jul 22, 2016 10:09:37 AM]   OUI-67008:OPatch Session completed with warnings.
[Jul 22, 2016 10:09:37 AM]   Finishing UtilSession at Fri Jul 22 10:09:37 CEST 2016

will be rolled back.. ok, the end of the opatch.log for grid looks nice

- the opatch.log for oracle home:

[Jul 22, 2016 10:09:09 AM]   Composite patch 23054246 successfully applied.
[Jul 22, 2016 10:09:09 AM]   Patch 23054327 successfully applied.
[Jul 22, 2016 10:09:09 AM]   UtilSession: N-Apply done.
[Jul 22, 2016 10:09:09 AM]   --------------------------------------------------------------------------------
[Jul 22, 2016 10:09:09 AM]   The following warnings have occurred during OPatch execution:
[Jul 22, 2016 10:09:09 AM]   1) OUI-67303:
                             Patches [   20831113 ] will be rolled back.
[Jul 22, 2016 10:09:09 AM]   --------------------------------------------------------------------------------
[Jul 22, 2016 10:09:09 AM]   OUI-67008:OPatch Session completed with warnings.

– looks good…

at this point I am sure everything is correct for the apply, let’s apply

[root@standby1 opatch]# /products/grid/12.1.0.2/OPatch/opatchauto apply /media/sf_install/patches/23273629 -ocmrf /products/ocm.rsp
OPatch Automation Tool
Copyright (c)2014, Oracle Corporation. All rights reserved.

OPatchauto Version : 12.1.0.1.10
OUI Version        : 12.1.0.2.0
Running from       : /products/grid/12.1.0.2

opatchauto log file: /products/grid/12.1.0.2/cfgtoollogs/opatchauto/23273629/opatch_gi_2016-07-22_10-22-16_deploy.log

Parameter Validation: Successful

Configuration Validation: Successful

Patch Location: /media/sf_install/patches/23273629
Grid Infrastructure Patch(es): 21436941 23054246 23054327 23054341
DB Patch(es): 23054246 23054327 

The following patch(es) are duplicate patches with patches installed in the Oracle Home.
 [ 20299023]
You have already installed same patch(es) with same UPI(s) or same version(s).
These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /products/grid/12.1.0.2

The following patch(es) are duplicate patches with patches installed in the Oracle Home.
 [ 20831110]
You have already installed same patch(es) with same UPI(s) or same version(s).
These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /products/grid/12.1.0.2

The following patch(es) are duplicate patches with patches installed in the Oracle Home.
 [ 19769480]
You have already installed same patch(es) with same UPI(s) or same version(s).
These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /products/grid/12.1.0.2

The following patch(es) are duplicate patches with patches installed in the Oracle Home.
 [ 20299023]
You have already installed same patch(es) with same UPI(s) or same version(s).
These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /products/oracle/12.1.0.2

The following patch(es) are duplicate patches with patches installed in the Oracle Home.
 [ 20831110]
You have already installed same patch(es) with same UPI(s) or same version(s).
These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /products/oracle/12.1.0.2

The following patch(es) are duplicate patches with patches installed in the Oracle Home.
 [ 19769480]
You have already installed same patch(es) with same UPI(s) or same version(s).
These patch(es) will be skipped.

Opatchauto skipped installing the above patch in /products/oracle/12.1.0.2
Patch Validation: Successful
Grid Infrastructure home:
/products/grid/12.1.0.2
DB home(s):
/products/oracle/12.1.0.2

Performing prepatch operations on CRS Home... Successful

Applying patch(es) to "/products/oracle/12.1.0.2" ...
Command "/products/oracle/12.1.0.2/OPatch/opatch napply -phBaseFile /tmp/OraDB12Home1_oracle_patchList -local  -invPtrLoc /products/grid/12.1.0.2/oraInst.loc -oh /products/oracle/12.1.0.2 -silent -ocmrf /products/ocm.rsp" execution failed:
OPatch failed to restore OH '/products/oracle/12.1.0.2'. Consult OPatch document to restore the home manually before proceeding.
UtilSession failed: Re-link fails on target "isqora".

Log file Location for the failed command: /products/oracle/12.1.0.2/cfgtoollogs/opatch/opatch2016-07-22_10-25-57AM_1.log

For more details, please refer to the log file "/products/grid/12.1.0.2/cfgtoollogs/opatchauto/23273629/opatch_gi_2016-07-22_10-22-16_deploy.debug.log".

Apply Summary:
Following patch(es) are successfully installed:
DB Home: /products/oracle/12.1.0.2:

Following patch(es) failed to be installed:
GI Home: /products/grid/12.1.0.2: 21436941,23054246,23054327,23054341
DB Home: /products/oracle/12.1.0.2: 23054246,23054327
Command failure exception

opatchauto failed with error code 2.
[root@standby1 opatch]# 

Failed again… OK so check the log…

[Jul 22, 2016 10:28:57 AM]   OUI-67200:Make failed to invoke "/usr/bin/make -f ins_odbc.mk isqora ORACLE_HOME=/products/oracle/12.1.0.2"....'/usr/bin/ld: cannot find -lodbcinst
                             collect2: ld returned 1 exit status
                             make: *** [/products/oracle/12.1.0.2/odbc/lib/libsqora.so.12.1] Error 1
                             '
[Jul 22, 2016 10:28:57 AM]   Stack Description: java.lang.RuntimeException: /usr/bin/ld: cannot find -lodbcinst
                             collect2: ld returned 1 exit status
                             make: *** [/products/oracle/12.1.0.2/odbc/lib/libsqora.so.12.1] Error 1
[Jul 22, 2016 10:28:57 AM]   StackTrace: oracle.opatch.MakeAction.apply(MakeAction.java:530)
[Jul 22, 2016 10:28:57 AM]   StackTrace: oracle.opatch.OPatchSession.runMake(OPatchSession.java:730)
[Jul 22, 2016 10:28:57 AM]   StackTrace: oracle.opatch.MergedPatchObject.invokeConsolidatedMake(MergedPatchObject.java:1784)
[Jul 22, 2016 10:28:57 AM]   StackTrace: oracle.opatch.opatchutil.NApply.process(NApply.java:1268)
[Jul 22, 2016 10:28:57 AM]   StackTrace: oracle.opatch.opatchutil.OUSession.napply(OUSession.java:1089)
[Jul 22, 2016 10:28:57 AM]   StackTrace: sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
[Jul 22, 2016 10:28:57 AM]   StackTrace: sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
[Jul 22, 2016 10:28:57 AM]   StackTrace: sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
[Jul 22, 2016 10:28:57 AM]   StackTrace: java.lang.reflect.Method.invoke(Method.java:597)
[Jul 22, 2016 10:28:57 AM]   StackTrace: oracle.opatch.UtilSession.process(UtilSession.java:317)
[Jul 22, 2016 10:28:57 AM]   StackTrace: oracle.opatch.OPatchSession.main(OPatchSession.java:2585)
[Jul 22, 2016 10:28:57 AM]   StackTrace: oracle.opatch.OPatch.main(OPatch.java:634)
[Jul 22, 2016 10:28:57 AM]   OUI-67050:Running make for target install_srvm
[Jul 22, 2016 10:28:57 AM]   Start invoking 'make' at Fri Jul 22 10:28:57 CEST 2016Fri Jul 22 10:28:57 CEST 2016
[Jul 22, 2016 10:28:57 AM]   Finish invoking 'make' at Fri Jul 22 10:28:57 CEST 2016
[Jul 22, 2016 10:28:57 AM]   OUI-67050:Running make for target install
[Jul 22, 2016 10:28:57 AM]   Start invoking 'make' at Fri Jul 22 10:28:57 CEST 2016Fri Jul 22 10:28:57 CEST 2016
[Jul 22, 2016 10:28:57 AM]   Finish invoking 'make' at Fri Jul 22 10:28:57 CEST 2016
[Jul 22, 2016 10:28:57 AM]   The following make actions have failed :
[Jul 22, 2016 10:28:57 AM]   OUI-67124:Re-link fails on target "isqora".
[Jul 22, 2016 10:28:57 AM]   Do you want to proceed? [y|n]
[Jul 22, 2016 10:29:00 AM]   N (auto-answered by -silent)
[Jul 22, 2016 10:29:00 AM]   User Responded with: N
[Jul 22, 2016 10:29:30 AM]   Re-link fails on target "isqora".
[Jul 22, 2016 10:29:30 AM]   Running make for target "install_srvm"
[Jul 22, 2016 10:29:30 AM]   Start invoking 'make' at Fri Jul 22 10:29:30 CEST 2016Fri Jul 22 10:29:30 CEST 2016
[Jul 22, 2016 10:29:30 AM]   Finish invoking 'make' at Fri Jul 22 10:29:30 CEST 2016
[Jul 22, 2016 10:29:30 AM]   Running make for target "install"
[Jul 22, 2016 10:29:30 AM]   Start invoking 'make' at Fri Jul 22 10:29:30 CEST 2016Fri Jul 22 10:29:30 CEST 2016
[Jul 22, 2016 10:29:30 AM]   Finish invoking 'make' at Fri Jul 22 10:29:30 CEST 2016
[Jul 22, 2016 10:29:30 AM]   --------------------------------------------------------------------------------
                             Failed to run make commands. Please contact Oracle Support.
[Jul 22, 2016 10:29:30 AM]   OUI-67115:OPatch failed to restore OH '/products/oracle/12.1.0.2'. Consult OPatch document to restore the home manually before proceeding.
[Jul 22, 2016 10:29:30 AM]   OUI-67124:
                             NApply was not able to restore the home.  Please invoke the following scripts:
                               - restore.[sh,bat]
                               - make.txt (Unix only)
                             to restore the ORACLE_HOME.  They are located under
                             "/products/oracle/12.1.0.2/.patch_storage/NApply/2016-07-22_10-25-57AM"
[Jul 22, 2016 10:29:30 AM]   OUI-67073:UtilSession failed: Re-link fails on target "isqora".
[Jul 22, 2016 10:29:30 AM]   --------------------------------------------------------------------------------
[Jul 22, 2016 10:29:30 AM]   The following warnings have occurred during OPatch execution:
[Jul 22, 2016 10:29:30 AM]   1) OUI-67303:
                             Patches [   20831113 ] will be rolled back.
[Jul 22, 2016 10:29:30 AM]   2) OUI-67200:Make failed to invoke "/usr/bin/make -f ins_odbc.mk isqora ORACLE_HOME=/products/oracle/12.1.0.2"....'/usr/bin/ld: cannot find -lodbcinst
                             collect2: ld returned 1 exit status
                             make: *** [/products/oracle/12.1.0.2/odbc/lib/libsqora.so.12.1] Error 1
                             '
[Jul 22, 2016 10:29:30 AM]   3) OUI-67124:Re-link fails on target "isqora".
[Jul 22, 2016 10:29:30 AM]   4) OUI-67200:Make failed to invoke "/usr/bin/make -f ins_odbc.mk isqora ORACLE_HOME=/products/oracle/12.1.0.2"....'/usr/bin/ld: cannot find -lodbcinst
                             collect2: ld returned 1 exit status
                             make: *** [/products/oracle/12.1.0.2/odbc/lib/libsqora.so.12.1] Error 1
                             '
[Jul 22, 2016 10:29:30 AM]   5) OUI-67124:
                             NApply was not able to restore the home.  Please invoke the following scripts:
                               - restore.[sh,bat]
                               - make.txt (Unix only)
                             to restore the ORACLE_HOME.  They are located under
                             "/products/oracle/12.1.0.2/.patch_storage/NApply/2016-07-22_10-25-57AM"
[Jul 22, 2016 10:29:30 AM]   --------------------------------------------------------------------------------

ok, let’s fix my Oracle Home

[oracle@standby1 ~]$ cd /products/oracle/12.1.0.2/.patch_storage/NApply/2016-07-22_10-25-57AM
[oracle@standby1 2016-07-22_10-25-57AM]$ ll
total 16
drwxr-xr-x. 16 oracle oinstall 4096 Jul 22 10:26 backup
-rw-r--r--.  1 oracle oinstall 2094 Jul 22 10:28 make.txt
-rw-r--r--.  1 oracle oinstall  976 Jul 22 10:28 patchlist.txt
-rwx--x---.  1 oracle oinstall 1601 Jul 22 10:26 restore.sh
[oracle@standby1 2016-07-22_10-25-57AM]$ cat restore.sh
#!/bin/sh
# Copyright (c) 2016, Oracle Corporation.  All rights reserved

echo This script is going to restore the Oracle Home to the previous state.
echo It does not perform any of the following:
echo - Running init/pre/post scripts
echo - Oracle binary re-link
echo - Customized steps performed manually by user
echo Please use this script with supervision from Oracle Technical Support.

# Get ORACLE_HOME from environment variable "ORACLE_HOME"
OH=${ORACLE_HOME}

# Error out if OH is not set
if [ "$OH" = "" ]; then
    echo Oracle Home is not set.
    echo Please set env. variable ORACLE_HOME and try again.
    echo Script failed to proceed.
    exit 1
fi

echo "About to modify Oracle Home( $OH )"
echo "Do you want to proceed? [Y/N]"
if [ "$1" = "-silent" ]; then
   response="Y"
else
    read response;
fi

if [ $response = "y" ] || [ $response = "Y" ]; then
    echo "User responded with : Y"
else
    echo "User responded with : $response"
    exit 0
fi

echo Date   Time : `date` >> $OH/cfgtoollogs/opatch/opatch_history.txt
echo Oracle Home : $OH >> $OH/cfgtoollogs/opatch/opatch_history.txt
echo Command     : restore.sh >> $OH/cfgtoollogs/opatch/opatch_history.txt

\rm -rf $OH/.patch_storage/NApply/2016-07-22_10-25-57AM/files

cd /products/oracle/12.1.0.2/inventory/oneoffs
\rm -rf 21359755

\rm -rf 21948354

\rm -rf 22291127

\rm -rf 23054246

\rm -rf 23054327

cd $OH/.patch_storage/NApply/2016-07-22_10-25-57AM/backup
for dname in "`\ls -1A`" ; do
    \cp -fR $dname $OH ;
done

\rm -rf $OH/.patch_storage/NApply/2016-07-22_10-25-57AM/files

echo Restore script completed.
[oracle@standby1 2016-07-22_10-25-57AM]$ . oraenv
ORACLE_SID = [oracle] ? orcl1
The Oracle base has been set to /products/oracle
[oracle@standby1 2016-07-22_10-25-57AM]$ ./restore.sh
This script is going to restore the Oracle Home to the previous state.
It does not perform any of the following:
- Running init/pre/post scripts
- Oracle binary re-link
- Customized steps performed manually by user
Please use this script with supervision from Oracle Technical Support.
About to modify Oracle Home( /products/oracle/12.1.0.2 )
Do you want to proceed? [Y/N]
y
User responded with : Y
Restore script completed.
[oracle@standby1 2016-07-22_10-25-57AM]$ ls -ltr
total 16
-rwx--x---.  1 oracle oinstall 1601 Jul 22 10:26 restore.sh
drwxr-xr-x. 16 oracle oinstall 4096 Jul 22 10:26 backup
-rw-r--r--.  1 oracle oinstall  976 Jul 22 10:28 patchlist.txt
-rw-r--r--.  1 oracle oinstall 2094 Jul 22 10:28 make.txt
[oracle@standby1 2016-07-22_10-25-57AM]$ vi make.txt
[oracle@standby1 2016-07-22_10-25-57AM]$ cat make.txt
cd $ORACLE_HOME/rdbms/lib;/usr/bin/make -f ins_rdbms.mk iextjobo ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/rdbms/lib;/usr/bin/make -f ins_rdbms.mk iextjob ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/rdbms/lib;/usr/bin/make -f ins_rdbms.mk ioracle ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/rdbms/lib;/usr/bin/make -f ins_rdbms.mk libasmclntsh12.ohso ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/rdbms/lib;/usr/bin/make -f ins_rdbms.mk client_sharedlib ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/rdbms/lib;/usr/bin/make -f ins_rdbms.mk ikfed ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/network/lib;/usr/bin/make -f ins_net_client.mk client_sharedlib ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/rdbms/lib;/usr/bin/make -f ins_rdbms.mk iorion ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/ldap/lib;/usr/bin/make -f ins_ldap.mk ldapmoddn ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/ldap/lib;/usr/bin/make -f ins_ldap.mk ldapmodifymt ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/ldap/lib;/usr/bin/make -f ins_ldap.mk ldapcompare ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/ldap/lib;/usr/bin/make -f ins_ldap.mk ldapbind ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/ldap/lib;/usr/bin/make -f ins_ldap.mk ldapsearch ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/ldap/lib;/usr/bin/make -f ins_ldap.mk ldapaddmt ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/ldap/lib;/usr/bin/make -f ins_ldap.mk ldapmodify ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/ldap/lib;/usr/bin/make -f ins_ldap.mk ldapdelete ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/ldap/lib;/usr/bin/make -f ins_ldap.mk ldapadd ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/rdbms/lib;/usr/bin/make -f ins_rdbms.mk patchset_opt_all ioracle ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/rdbms/lib;/usr/bin/make -f ins_rdbms.mk irman ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/ctx/lib;/usr/bin/make -f ins_ctx.mk ictxhx ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/odbc/lib;/usr/bin/make -f ins_odbc.mk isqora ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/srvm/lib;/usr/bin/make -f ins_srvm.mk install_srvm ORACLE_HOME=$ORACLE_HOME
cd $ORACLE_HOME/racg/lib;/usr/bin/make -f ins_has.mk install ORACLE_HOME=$ORACLE_HOME
[oracle@standby1 2016-07-22_10-25-57AM]$ chmod 755 make.txt
[oracle@standby1 2016-07-22_10-25-57AM]$ ./make.txt

I will not paste it here, it’s a relink like I said

[oracle@standby1 12.1.0.2]$ setasmgid oracle_binary_path=/products/oracle/12.1.0.2/bin/oracle
KFSG-00302: group '' not found in operating system dictionary
[oracle@standby1 12.1.0.2]$ su - grid
Password:
[grid@standby1 ~]$ . oraenv
ORACLE_SID = [grid] ? +ASM1
The Oracle base has been set to /products/oracle
[grid@standby1 ~]$ setasmgid oracle_binary_path=/products/oracle/12.1.0.2/bin/oracle
KFSG-00304: this executable must run as SETUID root
[grid@standby1 ~]$ su
Password:
[root@standby1 grid]# . oraenv
ORACLE_SID = [+ASM1] ?
The Oracle base remains unchanged with value /products/oracle
[root@standby1 grid]# setasmgid oracle_binary_path=/products/oracle/12.1.0.2/bin/oracle
[root@standby1 grid]# ls -l /products/oracle/12.1.0.2/bin/oracle
-rwsr-s--x. 1 oracle asmadmin 323791685 Jul 22 10:35 /products/oracle/12.1.0.2/bin/oracle

as the patch process unlocks the grid infrastructure and later fails I have to do the post tasks at patching manually

[root@standby1 grid]# cd /products/grid/12.1.0.2/rdbms/install/
[root@standby1 install]# ll
total 72
-rwxr-x---. 1 grid oinstall  7864 Feb 26  2012 clonedb.pl
-rw-r--r--. 1 grid oinstall  1694 Dec 19  2003 config_filemap.sbs
-rw-r--r--. 1 grid oinstall  1714 May  2 13:42 filemap.ora
-rw-r--r--. 1 grid oinstall   231 Jun  6  2013 install.excl
drwxr-xr-x. 2 grid oinstall  4096 May  2 13:39 rdbms
-rwxr--r--. 1 grid oinstall  1092 May  2 13:42 rootadd_filemap.sh
-rwxr-x---. 1 grid oinstall  2988 May  2 13:41 rootadd_rdbms.sh
-rwxr-x---. 1 grid oinstall  5658 Jun 17  2011 s7x_workflow.akwf
drwxr-xr-x. 2 grid oinstall  4096 May  2 13:40 sbs
-rwxr-x---. 1 grid oinstall 25387 Oct 23  2013 setup_processor_group.sh
[root@standby1 install]# ./rootadd_rdbms.sh
[root@standby1 install]# cd ../..
[root@standby1 12.1.0.2]# cd crs/
auth/     config/   demo/     init/     install/  lib/      log/      mesg/     profile/  public/   sbs/      script/   template/ trace/    utl/
[root@standby1 12.1.0.2]# cd crs/install/
[root@standby1 install]# ./rootcrs.sh -patch
Using configuration parameter file: /products/grid/12.1.0.2/crs/install/crsconfig_params
2016/07/22 10:40:14 CLSRSC-4015: Performing install or upgrade action for Oracle Trace File Analyzer (TFA) Collector.

2016/07/22 10:40:27 CLSRSC-4003: Successfully patched Oracle Trace File Analyzer (TFA) Collector.

CRS-4123: Oracle High Availability Services has been started.
CRS-2791: Starting shutdown of Oracle High Availability Services-managed resources on 'standby1'
CRS-2673: Attempting to stop 'ora.drivers.acfs' on 'standby1'
CRS-2677: Stop of 'ora.drivers.acfs' on 'standby1' succeeded
CRS-2793: Shutdown of Oracle High Availability Services-managed resources on 'standby1' has completed
CRS-4133: Oracle High Availability Services has been stopped.
CRS-4123: Starting Oracle High Availability Services-managed resources
CRS-2672: Attempting to start 'ora.mdnsd' on 'standby1'
CRS-2672: Attempting to start 'ora.evmd' on 'standby1'
CRS-2676: Start of 'ora.mdnsd' on 'standby1' succeeded
CRS-2676: Start of 'ora.evmd' on 'standby1' succeeded
CRS-2672: Attempting to start 'ora.gpnpd' on 'standby1'
CRS-2676: Start of 'ora.gpnpd' on 'standby1' succeeded
CRS-2672: Attempting to start 'ora.gipcd' on 'standby1'
CRS-2676: Start of 'ora.gipcd' on 'standby1' succeeded
CRS-2672: Attempting to start 'ora.cssdmonitor' on 'standby1'
CRS-2676: Start of 'ora.cssdmonitor' on 'standby1' succeeded
CRS-2672: Attempting to start 'ora.cssd' on 'standby1'
CRS-2672: Attempting to start 'ora.diskmon' on 'standby1'
CRS-2676: Start of 'ora.diskmon' on 'standby1' succeeded
CRS-2676: Start of 'ora.cssd' on 'standby1' succeeded
CRS-2672: Attempting to start 'ora.cluster_interconnect.haip' on 'standby1'
CRS-2672: Attempting to start 'ora.ctssd' on 'standby1'
CRS-2676: Start of 'ora.ctssd' on 'standby1' succeeded
CRS-2676: Start of 'ora.cluster_interconnect.haip' on 'standby1' succeeded
CRS-2672: Attempting to start 'ora.asm' on 'standby1'
CRS-2676: Start of 'ora.asm' on 'standby1' succeeded
CRS-2672: Attempting to start 'ora.storage' on 'standby1'
CRS-2676: Start of 'ora.storage' on 'standby1' succeeded
CRS-2672: Attempting to start 'ora.crf' on 'standby1'
CRS-2676: Start of 'ora.crf' on 'standby1' succeeded
CRS-2672: Attempting to start 'ora.crsd' on 'standby1'
CRS-2676: Start of 'ora.crsd' on 'standby1' succeeded
CRS-6017: Processing resource auto-start for servers: standby1
CRS-2672: Attempting to start 'ora.net1.network' on 'standby1'
CRS-2676: Start of 'ora.net1.network' on 'standby1' succeeded
CRS-2672: Attempting to start 'ora.ons' on 'standby1'
CRS-2673: Attempting to stop 'ora.standby1.vip' on 'standby2'
CRS-2677: Stop of 'ora.standby1.vip' on 'standby2' succeeded
CRS-2672: Attempting to start 'ora.standby1.vip' on 'standby1'
CRS-2676: Start of 'ora.ons' on 'standby1' succeeded
CRS-2673: Attempting to stop 'ora.LISTENER_SCAN1.lsnr' on 'standby2'
CRS-2677: Stop of 'ora.LISTENER_SCAN1.lsnr' on 'standby2' succeeded
CRS-2673: Attempting to stop 'ora.scan1.vip' on 'standby2'
CRS-2676: Start of 'ora.standby1.vip' on 'standby1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER.lsnr' on 'standby1'
CRS-2677: Stop of 'ora.scan1.vip' on 'standby2' succeeded
CRS-2672: Attempting to start 'ora.scan1.vip' on 'standby1'
CRS-2676: Start of 'ora.LISTENER.lsnr' on 'standby1' succeeded
CRS-2676: Start of 'ora.scan1.vip' on 'standby1' succeeded
CRS-2672: Attempting to start 'ora.LISTENER_SCAN1.lsnr' on 'standby1'
CRS-2676: Start of 'ora.LISTENER_SCAN1.lsnr' on 'standby1' succeeded
CRS-6016: Resource auto-start has completed for server standby1
CRS-6024: Completed start of Oracle Cluster Ready Services-managed resources
CRS-4123: Oracle High Availability Services has been started.
Oracle Clusterware active version on the cluster is [12.1.0.2.0]. The cluster upgrade state is [NORMAL]. The cluster active patch level is [459864538].

voila, the node is again running … and opatchauto does not work, because something looks to miss

Posted in 2016

Most important thing in IT – test system

Today I wanted to patch my Oracle Homes with the new PSU July 2016 (DB+GI without OJVM). I hoped opatchauto will work… well, it worked not.. again.

it failed on a napply and relink of isqora. OK. As it’s only my personal test RAC I was not sad, because there was information too how to fix it.. run restore.sh to restore the db home and make.txt which had arround 30 make commands, basically all linking commands you see, rman, listener, basically a relink all. The log had written that these commands should be executed only under advisory of Oracle Support. If this would happen on the production server at night, well I would be really sad mad 🙂

Always test on your test systems, if you cannot patch a test system with documented notes, do not patch on production.

Posted in 2016

Ansible: Playbooks – Basics

Automation in ansible is about playbooks. A playbook is a simple file in yaml syntax with basically the same commands as in a command line syntax. More info here: Ansible Playbooks

[root@hackbook ansible]# cat first.yaml
---
- hosts: agents
  tasks:
  - name: Check uptime
    command: uptime
  - name: Create a file
    file: dest=/tmp/oracle state=touch
  - name: Check them
    command: ls -l /tmp/oracle

and to play this simple playbook you can run:

[root@hackbook ansible]# ansible-playbook first.yaml -v
Using /etc/ansible/ansible.cfg as config file

PLAY [agents] ******************************************************************

TASK [setup] *******************************************************************
ok: [hack-a1.example.com]
ok: [hack-a2.example.com]

TASK [Check uptime] ************************************************************
changed: [hack-a1.example.com] => {"changed": true, "cmd": ["uptime"], "delta": "0:00:00.003360", "end": "2016-07-20 15:24:36.256354", "rc": 0, "start": "2016-07-20 15:24:36.252994", "stderr": "", "stdout": " 15:24:36 up 40 min,  1 user,  load average: 0.00, 0.01, 0.05", "stdout_lines": [" 15:24:36 up 40 min,  1 user,  load average: 0.00, 0.01, 0.05"], "warnings": []}
changed: [hack-a2.example.com] => {"changed": true, "cmd": ["uptime"], "delta": "0:00:00.003419", "end": "2016-07-20 15:24:30.403320", "rc": 0, "start": "2016-07-20 15:24:30.399901", "stderr": "", "stdout": " 15:24:30 up 40 min,  1 user,  load average: 0.00, 0.01, 0.05", "stdout_lines": [" 15:24:30 up 40 min,  1 user,  load average: 0.00, 0.01, 0.05"], "warnings": []}

TASK [Create a file] ***********************************************************
changed: [hack-a2.example.com] => {"changed": true, "dest": "/tmp/oracle", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:user_tmp_t:s0", "size": 0, "state": "file", "uid": 0}
changed: [hack-a1.example.com] => {"changed": true, "dest": "/tmp/oracle", "gid": 0, "group": "root", "mode": "0644", "owner": "root", "secontext": "unconfined_u:object_r:user_tmp_t:s0", "size": 0, "state": "file", "uid": 0}

TASK [Check the file] **********************************************************
changed: [hack-a1.example.com] => {"changed": true, "cmd": ["ls", "-l", "/tmp/oracle"], "delta": "0:00:00.003528", "end": "2016-07-20 15:24:36.937009", "rc": 0, "start": "2016-07-20 15:24:36.933481", "stderr": "", "stdout": "-rw-r--r--. 1 root root 0 Jul 20 15:24 /tmp/oracle", "stdout_lines": ["-rw-r--r--. 1 root root 0 Jul 20 15:24 /tmp/oracle"], "warnings": []}
changed: [hack-a2.example.com] => {"changed": true, "cmd": ["ls", "-l", "/tmp/oracle"], "delta": "0:00:00.003405", "end": "2016-07-20 15:24:31.084435", "rc": 0, "start": "2016-07-20 15:24:31.081030", "stderr": "", "stdout": "-rw-r--r--. 1 root root 0 Jul 20 15:24 /tmp/oracle", "stdout_lines": ["-rw-r--r--. 1 root root 0 Jul 20 15:24 /tmp/oracle"], "warnings": []}

PLAY RECAP *********************************************************************
hack-a1.example.com        : ok=4    changed=3    unreachable=0    failed=0
hack-a2.example.com        : ok=4    changed=3    unreachable=0    failed=0

To install/deinstall a webserver with a proxy you can use these playbooks:

[root@hackbook ansible]# cat webserver.yaml
---
- hosts: agents
  environment:
    http_proxy: http://x.x.x.x:8080
  tasks:
  - name: copy resolv.conf to nodes
    copy:
      dest=/etc/resolv.conf
      src=/etc/resolv.conf
  - name: copy /etc/yum.repos.d/public-yum-ol7.repo
    copy:
      dest=/etc/yum.repos.d/public-yum-ol7.repo
      src=/etc/yum.repos.d/public-yum-ol7.repo
  - name: install httpd related packages
    yum:
      name: httpd
      state: installed
  - name: ensure apache is running
    service: name=httpd state=started
[root@hackbook ansible]# cat remove_webserver.yaml
---
- hosts: agents
  tasks:
  - name: stop httpd
    service:
      name: httpd
      state: stopped
  - name: remove httpd
    yum:
      name: httpd
      state: absent
  - name: remove httpd-tools
    yum:
      name: httpd-tools
      state: absent
[root@hackbook ansible]#
Posted in 2016

Automation with ansible for dba 1.0

I don’t have any knowledge how to automate, I never cared about it, because I was a DBA, now my new colleague is a geeky system administrator and now I see how things are done. Let me introduce:

[oracle@hackbook scripts]$ rpm -qa | grep ansible
ansible-2.1.0.0-1.el7.noarch

I have created 2 VMs to play with them.. they are copies of a template used for my hackbook VM.
-f is for parallel execution, default are 5 parallel processes

  1. Check all hosts
    [root@hackbook ~]# ansible all -m ping
    hack-a1.example.com | SUCCESS => {
        "changed": false,
        "ping": "pong"
    }
    hack-a2.example.com | SUCCESS => {
        "changed": false,
        "ping": "pong"
    }
  2. File transfer
    [root@hackbook ~]# ansible agents -m copy -a "src=/home/oracle/Downloads/workapp.zip dest=/tmp"
    hack-a1.example.com | SUCCESS => {
        "changed": true,
        "checksum": "3ee1dfe5609e1767d50d9758b9f945a92ea9b046",
        "dest": "/tmp/workapp.zip",
        "gid": 0,
        "group": "root",
        "md5sum": "9a01fd1b33b361304c9c3150f64c9232",
        "mode": "0644",
        "owner": "root",
        "secontext": "unconfined_u:object_r:admin_home_t:s0",
        "size": 43338,
        "src": "/root/.ansible/tmp/ansible-tmp-1469011514.56-11092828015532/source",
        "state": "file",
        "uid": 0
    }
    hack-a2.example.com | SUCCESS => {
        "changed": true,
        "checksum": "3ee1dfe5609e1767d50d9758b9f945a92ea9b046",
        "dest": "/tmp/workapp.zip",
        "gid": 0,
        "group": "root",
        "md5sum": "9a01fd1b33b361304c9c3150f64c9232",
        "mode": "0644",
        "owner": "root",
        "secontext": "unconfined_u:object_r:admin_home_t:s0",
        "size": 43338,
        "src": "/root/.ansible/tmp/ansible-tmp-1469011514.57-237433087750369/source",
        "state": "file",
        "uid": 0
    }
  3. Shell execution
    [root@hackbook ~]# ansible all -m shell -a "chown oracle:dba /tmp/work*" -f 1
    hack-a1.example.com | SUCCESS | rc=0 >>
    
    
    hack-a2.example.com | SUCCESS | rc=0 >>
    
    
    [root@hackbook ~]# ansible all -m shell -a "ls -l /tmp/work*" -f 1
    hack-a1.example.com | SUCCESS | rc=0 >>
    -rw-r--r--. 1 oracle dba 43338 Jul 20 12:47 /tmp/workapp.zip
    
    hack-a2.example.com | SUCCESS | rc=0 >>
    -rw-r--r--. 1 oracle dba 43338 Jul 20 12:47 /tmp/workapp.zip
    
    
  4. Files
    [root@hackbook ~]# ansible all -m file -a "dest=/tmp/oracle state=directory"
    hack-a2.example.com | SUCCESS => {
        "changed": true,
        "gid": 0,
        "group": "root",
        "mode": "0755",
        "owner": "root",
        "path": "/tmp/oracle",
        "secontext": "unconfined_u:object_r:user_tmp_t:s0",
        "size": 6,
        "state": "directory",
        "uid": 0
    }
    hack-a1.example.com | SUCCESS => {
        "changed": true,
        "gid": 0,
        "group": "root",
        "mode": "0755",
        "owner": "root",
        "path": "/tmp/oracle",
        "secontext": "unconfined_u:object_r:user_tmp_t:s0",
        "size": 6,
        "state": "directory",
        "uid": 0
    }
    [root@hackbook ~]# ansible all -m file -a "dest=/tmp/oracle state=absent"
    hack-a2.example.com | SUCCESS => {
        "changed": true,
        "path": "/tmp/oracle",
        "state": "absent"
    }
    hack-a1.example.com | SUCCESS => {
        "changed": true,
        "path": "/tmp/oracle",
        "state": "absent"
    }
Posted in 2016

pfile or spfile?

In my old work there were strange things on daily business. I don’t know why, but for instance a dataguard configuration was a nonstandard configuration for Oracle Support. We had only shipping configured and scripts took care of switchover/failover and the gap check. Do you know since when this concept exists? ..since 8i, well, that’s around year 2002, last year was 2015 and the “standard” is still in use. I understand that dinosaurs are incredible, but we should focus on the future.

So why should someone use an spfile, when you can use a pfile? A pfile has so much benefits.. you can write to it in every editor, so cool and that’s it, no more benefits.

With an spfile you have so much negative aspects, you cannot write to it and well that’s it.
So the strong thing on a pfile is it’s simplicity to write to it, or?
Spfile is even simpler:
1. you can write with sqlplus to it, alter system set parameter=value scope=spfile ;
and oracle checks if you can add it or not.
2. if you start an instance with an spfile, you can change dynamic parameters and save them in the spfile with “scope=both”, you don’t need to open the file in an editor and make mistakes, typos, like humans do, you know.
3. You can add comments to the parameters that can be selected, they are not so cool like the pfile hashtag comments (twitter friends like to hashtag everything)
4. The most important thing, you can make a backup of this file either to disk or tape.
Imagine, you are a cool pfile tuner, restart several times, change the pfile and suddenly a hacker removes your file and reboots the machine.. the database is lost. (OK, it’s not lost, but we can pretend it’s lost)

I don’t say, people who work with Oracle longer don’t know what they do, but there are sometimes “new” features which should be adopted. If you believe in someone skills, do not forget, he is a human, he can be wrong, like I can, but it’s better to choose from 2 possibilities than from 1, because if you choose from 1, you are like the Slovak government, you have 1 contract, you pay for an advisor let’s say 1 million (it’s on the internet, the correct number) to choose the best contract and you cheer him afterwards for doing a great job.

I dedicate this article the greatest DBA woman in Slovakia.

It’s so bad when there…

It’s so bad when there are more automation tools… puppet, chef, salt and now ansible. You can be either a DBA or a System Administrator.. or a Network admin.. if you do everything, you cannot be really good at all of them, you may understand all the networking stuff, but implementing iptables is not administration of networks or updating an OS and basic tasks, it’s much more… with a database come always new features, as a System admin you have also much work with automation, monitoring, installation of 10000000 of packages and everything has it’s own guide.. It’s not possible to become a DBA 2.0.