Exadata Bundle Patch 5 Gotcha's

A couple of months ago we became proud parents of a bouncing baby Exadata V2, (1/4 rack). I had about two weeks to configure it and get up to speed on the high points before we were to begin work on our first proof of concept (POC) for a local client. I had just enough time to patch the storage cells up to version 11.2.1.2.6 configure storage and create the databases. Well we’ve been working it pretty hard for the last couple of months with customer POC’s and our own testing but today I finally got a chance to get caught up on our patchsets for Exadata. The most current bundle patch releases are 4 and 5. After hearing about widespread problems with bundle patch 4 I decided BP5 was probably the best fit for us. I did a few searches and didn’t find any serious complications from BP5. However during the installation I did come across a couple of issues I thought I’d share here.

The first issue…
The first issue in the install process involved DBFS. We use DBFS for a high performance clustered file system to stage and load data. When we ran the "opatch apply" command we got the following error message after a few screens of output:

—- —– —– —– —– —– —– —–
The following actions have failed: Copy failed from ‘/home/oracle/stage/9870547/files/bin/dbfs_client’ to ‘/u01/app/oracle/product/11.2.0/dbhome_1/bin/dbfs_client’…
—- —– —– —– —– —– —– —–

We checked permissions on the file and directory but that didn’t appear to be the problem. So we renamed the file and retried the install.

—- —– —– —– —– —– —– —–
mv /u01/app/oracle/product/11.2.0/dbhome_1/bin/dbfs_client \
   /u01/app/oracle/product/11.2.0/dbhome_1/bin/dbfs_client.orig
—- —– —– —– —– —– —– —–

Once the file was out of the way the installatiton of BP5 continued without error.

The second issue…
You might classify the second issue as documentation problem. The instructions in the README file appears to recommend turning on RDS at this juncture. Here’s an excerpt from the README.

—- —– —– —– —– —– —– —–
**************
SPECIAL NOTE :
**************
The patch will activate RDS by default, if the applying system is using UDP,
user need to switch manually by doing:

cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk ipc_g
—- —– —– —– —– —– —– —–

Now it may be just the way I read this note but appear to be two problems with this "special note". The first thing I noticed was that this was not the command to activate RDS. It was in fact the command to configure it back to UDP. See MOS Doc ID 751343.1. So maybe the instructions meant for me to turn off RDS for the remainder of the patch install. This is more likely than not what was meant. But if that was the case why were there no instructions to turn RDS back on at the end of the install? Anyway I knew from experience that I would have to turn off RDS in order to startup the database in upgrade mode. You see a few weeks ago I went through the process of configuring RDS for the database home and the ASM home. I don’t know if its a bug or a feature or if I simply missed something when I configured RDS (don’t think so) after that I found I could no longer startup my databases using SQL*Plus. When I did I got this error:

—- —– —– —– —– —– —– —–
SQL> startup ORA-27504: IPC error creating OSD context ORA-27300: OS system dependent operation:bind-rds failed with status: 99 ORA-27301: OS failure message: Cannot assign requested address ORA-27302: failure occurred at: skgxpvaddr14 ORA-27303: additional information: Could not bind RDS socket to 192.168.8.201. Check interfaces specified for Oracle IPC (RAC/Exadata). [pid: 24249]
—- —– —– —– —– —– —– —–

Normally this wouldn’t be a problem because I can supply the -o option to srvctl to pass in startup options like "mount" or "force". Unfortunately "upgrade" is not one of the valid options for "srvctl start database". Consequently to run the post patch scripts in upgrade mode you must first switch back to UDP. Your database needs to be down during this process. At this point in the patching process your databases are all offlineat least on the node being patched (for rolling upgrades). Per the README the following command will reconfigure your database home for UDP:

—- —– —– —– —– —– —– —–
cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk ipc_g
—- —– —– —– —– —– —– —–

After this is done you can start the database in upgrade mode and run the post patch scripts. The README states that you should only have one instance of the database up for this process and that you should have the cluster_database parameter set to ‘FALSE’. Since this is something you can complete from just one node of your cluster you don’t have to reconfigure RDS/UDP on all nodes. You need only run the post patch scripts from one running instance each of your databases. Of course after the patch installation is complete you will want to rebuild the IPC library for RDS using the following command:

—- —– —– —– —– —– —– —–
cd $ORACLE_HOME/rdbms/lib
make -f ins_rdbms.mk ipc_rds ioracle
—- —– —– —– —– —– —– —–

Aside from these two relatively minor issues BP5 installed cleanly. There are several pages of bug fixes in this patch and from what I’ve seen so far I can’t think of any technical reason not to go ahead and install it. By the way one of the first things you will be instructed to do is unlock the GridInfrastructure home using the "$GI_HOME/crs/install/rootcrs.pl -unlock" command. This process will also shutdown your cluster and all the Grid Infrastructure resources (databases, listeners, SCAN’s, etc.). Once the bundle patch is applied (and before you run the post patch scripts) you will run the "$GI_HOME/crs/install/rootcrs.pl -patch" command which will restart the entire Oracle stack.