How it Works

It should work with 10gR2-11gR2 without changes. The first time is run, it will attempt to identify the Grid Infrastructure / CRS home and the version that is being used (CRS/HAS must be up for it to gather)- it takes that information and writes the info to a crsstat.env located in the same location as the script. It only gathers the information the first time, so if you upgrade the Oracle software you will want to remove the crsstat.env and allow the script to re-gather the information.

The output of the script assumes a term width of 132 and attempts to highlight possible issues, by showing in RED those resources with a target of ONLINE and a state of OFFLINE. You can also pass in “search criteria” as the first and only parameter.

Also of note, per Oracle support document 1086563.1, the “State Details” output in 11gR2 could show incorrect information, such as, “Instance Shutdown” even though it is up and running correctly.

Under the covers, the script is utilizing the “crs_stat -t” in 10gR2/11gR1 and “crsctl status resource -t” and then awks their output to create the tabular format. It has been used on AIX and Linux (SuSE, Red Hat, Oracle).

How to Install

Pre-requisite to Installation:

1. CRS/HAS must be running
2. The location of the crsstat script must be writable by the user that will run the installation (either root or your oracle user)

Installation Location:

Generally, it is practical to put the crsstat script in the /usr/local/bin directory, so everyone will have access (in most users path). Make sure the crsstat.env file can be created in the /usr/local/bin directory the first time the script is run. If it is preferred to put it an alternate location, choose a location that is accessible by your Unix users that should be able to list the CRS or HAS resources (i.e. in their PATH).

Installation Process:

1) Copy the installation script from this link http://blog.enkitec.com/wp-content/uploads/2011/10/crsstat.zip
2) Do a “cp crsstat.zip /usr/local/bin/”
3) As root run the crsstat and a “crsstat.env” file is created:

# cd /usr/local/bin
# unzip crsstat.zip
# chmod 755 crsstat
# ./crsstat

Output like example below…

# chmod 644 crsstat.env

4) Verify that the crsstat.env file has two entries like the following:
GRID_HOME=/u01/app/oracle/product/11.2.0/grid
GRID_VER=11.2

Example Usage

Basic Usage (note that the dbfs filesystem is offline when it should be online, thus it is highlight in red):

$ crsstat

Resource Name                            Resource Type  Target       State        Node            State Details
---------------------------------------- -------------- ------------ ------------ --------------- --------------
ora.DATA.dg                              diskgroup      ONLINE       ONLINE       tstdb01
ora.DATA.dg                              diskgroup      ONLINE       ONLINE       tstdb02
ora.LISTENER.lsnr                        Listener       ONLINE       ONLINE       tstdb01
ora.LISTENER.lsnr                        Listener       ONLINE       ONLINE       tstdb02
ora.LISTENER_SCAN1.lsnr                  SCAN Listener  ONLINE       ONLINE       tstdb02
ora.LISTENER_SCAN2.lsnr                  SCAN Listener  ONLINE       ONLINE       tstdb01
ora.LISTENER_SCAN3.lsnr                  SCAN Listener  ONLINE       ONLINE       tstdb01
ora.RECO.dg                              diskgroup      ONLINE       ONLINE       tstdb01
ora.RECO.dg                              diskgroup      ONLINE       ONLINE       tstdb02
ora.SYSTEM.dg                            diskgroup      ONLINE       ONLINE       tstdb01
ora.SYSTEM.dg                            diskgroup      ONLINE       ONLINE       tstdb02
ora.asm                                  ASM            ONLINE       ONLINE       tstdb01
ora.asm                                  ASM            ONLINE       ONLINE       tstdb02
ora.cvu                                  cvu            ONLINE       ONLINE       tstdb01
ora.dbfs.db                              database       ONLINE       ONLINE       tstdb01         Open
ora.dbfs.db                              database       ONLINE       ONLINE       tstdb02         Open
ora.dbfs.filesystem                                     ONLINE       OFFLINE
ora.dbfs.filesystem                                     ONLINE       OFFLINE
ora.demo.db                              database       ONLINE       ONLINE       tstdb01         Open
ora.demo.db                              database       ONLINE       ONLINE       tstdb02         Open
ora.demo.workshop.svc                    service        ONLINE       ONLINE       tstdb01
ora.demo.workshop.svc                    service        ONLINE       ONLINE       tstdb02
ora.tstdb01.vip                          Cluster VIP    ONLINE       ONLINE       tstdb01
ora.tstdb02.vip                          Cluster VIP    ONLINE       ONLINE       tstdb02
ora.net1.network                         Network (VIP)  ONLINE       ONLINE       tstdb01
ora.net1.network                         Network (VIP)  ONLINE       ONLINE       tstdb02
ora.ons                                  Ora Notif Svc  ONLINE       ONLINE       tstdb01
ora.ons                                  Ora Notif Svc  ONLINE       ONLINE       tstdb02
ora.registry.acfs                        registry       ONLINE       ONLINE       tstdb01
ora.registry.acfs                        registry       ONLINE       ONLINE       tstdb02
ora.scan1.vip                            SCAN VIP       ONLINE       ONLINE       tstdb02
ora.scan2.vip                            SCAN VIP       ONLINE       ONLINE       tstdb01
ora.scan3.vip                            SCAN VIP       ONLINE       ONLINE       tstdb01
ora.test.db                              database       OFFLINE      OFFLINE
ora.test.db                              database       OFFLINE      OFFLINE

A couple of examples using the search key:

$ crsstat service

Resource Name                            Resource Type  Target       State        Node            State Details
---------------------------------------- -------------- ------------ ------------ --------------- --------------
ora.demo.workshop.svc                    service        ONLINE       ONLINE       tstdb01
ora.demo.workshop.svc                    service        ONLINE       ONLINE       tstdb02

$ crsstat demo

Resource Name                            Resource Type  Target       State        Node            State Details
---------------------------------------- -------------- ------------ ------------ --------------- --------------
ora.demo.db                              database       ONLINE       ONLINE       enkdb01         Open
ora.demo.db                              database       ONLINE       ONLINE       enkdb02         Open
ora.demo.workshop.svc                    service        ONLINE       ONLINE       enkdb01
ora.demo.workshop.svc                    service        ONLINE       ONLINE       enkdb02

Another example, as sometimes when there are many databases it is useful to pipe the output to grep:

$ crsstat database | grep -i offline

ora.test.db                              database       OFFLINE      OFFLINE
ora.test.db                              database       OFFLINE      OFFLINE

Let’s revisit a use case I alluded to in my earlier blog entry: incorporating LDAP data into a query. Suppose your task is to code a custom authorization scheme for an enterprise application reporting system. You want row access to be determined by a user’s group memberships in the company’s Active Directory server. If a user is assigned to “the Company A group” in Active Directory, he gets to see Company A’s data. You can configure that kind of restriction at a database level using Oracle’s Virtual Private Database feature. Configuring VPD is beyond the scope of this blog entry, but ultimately you will be associating a table to a portion of a SQL WHERE clause that gets appended to every query written against that table. You can find more documentation on VPD on the OTN site. In this case, let’s assume you know that another team has written a PL/SQL database package to query data from Active Directory, and you want to make use of that code. The only problem is that all the functions in that package return collections of various types, and you need to be able to access that information from the WHERE clause you are building with VPD. This is where table functions come in handy.

Let’s begin by setting up our example schema. We’ll be looking at client sales data. First, let’s set up our clients.

CREATE TABLE clients (
  client_id NUMBER,
  client_name VARCHAR2(50));
 
INSERT INTO clients VALUES (100, 'Timco Industries');
INSERT INTO clients VALUES (200, 'Moneybros');
INSERT INTO clients VALUES (300, 'The Monacle Corporation');
INSERT INTO clients VALUES (400, 'Schmapple');

We also need application users who will be logging in to view the report data. We’ll assume each application user is uniquely linked to a user in Active Directory via the external_id column.

CREATE TABLE users (
  sequence_number NUMBER,
  user_name VARCHAR2(20),
  external_id VARCHAR2(20),
  first_name VARCHAR2(30),
  last_name VARCHAR2(30));
 
INSERT INTO users VALUES (1, 'JACKA', 'jaugusti', 'Jack', 'Augustine');
INSERT INTO users VALUES (2, 'KENKI', 'kkidd', 'Ken', 'Kidd');
INSERT INTO users VALUES (3, 'BRIAH', 'bhill', 'Brian', 'Hill');
INSERT INTO users VALUES (4, 'KEOVO', 'kvongkas', 'Keo', 'Vongkaseum');

Finally, we need a data table on which to report.

CREATE TABLE quarterly_sales_data (
  sequence_number NUMBER,
  client_id NUMBER,
  sales_year NUMBER,
  sales_quarter NUMBER,
  sales_amount NUMBER);
 
INSERT INTO quarterly_sales_data VALUES (1, 100, 2011, 1, 20000);
INSERT INTO quarterly_sales_data VALUES (2, 100, 2011, 2, 17500);
INSERT INTO quarterly_sales_data VALUES (3, 100, 2011, 3, 33000);
INSERT INTO quarterly_sales_data VALUES (4, 200, 2011, 1, 4500);
INSERT INTO quarterly_sales_data VALUES (5, 200, 2011, 2, 7000);
INSERT INTO quarterly_sales_data VALUES (6, 200, 2011, 3, 6300);
INSERT INTO quarterly_sales_data VALUES (7, 300, 2011, 1, 11100);
INSERT INTO quarterly_sales_data VALUES (8, 300, 2011, 2, 12800);
INSERT INTO quarterly_sales_data VALUES (9, 300, 2011, 3, 14000);
INSERT INTO quarterly_sales_data VALUES (10, 400, 2011, 1, 55600);
INSERT INTO quarterly_sales_data VALUES (11, 400, 2011, 2, 80000);
INSERT INTO quarterly_sales_data VALUES (12, 400, 2011, 3, 78500);

Our basic report query will look something like this:

SELECT client_name, sales_year, sales_quarter, sales_amount
  FROM quarterly_sales_data NATURAL JOIN clients;

Our Active Directory groups will be TIMCO_GRP, MONEYBROS_GRP, MONACLE_GRP, and SCHMAPPLE_GRP. If a user belongs to one or more of these groups, he has permission to view the respective client’s data.

Now we need a function that returns data from Active Directory. Just to keep it simple we will create a package that hardcodes the group memberships and returns them as collections of various types. We will then look at how table functions can be used to transform each one of those collections into an aggregate result set. (NOTE: querying LDAP directories from PL/SQL is not too difficult, and might serve as the basis for a future blog entry.)

Here’s our function. We’ve placed it in a package called LDAP_UTIL.

CREATE OR REPLACE PACKAGE ldap_util AS
 
  TYPE ldap_entry_typ IS RECORD (
    cn VARCHAR2(30),
    description VARCHAR2(1000),
    distinguished_name VARCHAR2(200),
    name VARCHAR2(30),
    samaccountname VARCHAR2(30));
 
  TYPE ldap_groups_nt IS TABLE OF ldap_entry_typ;
  TYPE ldap_groups_aa IS TABLE OF ldap_entry_typ INDEX BY PLS_INTEGER;
  TYPE ldap_groups_va IS VARRAY(100) of ldap_entry_typ;
 
  FUNCTION get_user_groups_nt (p_user VARCHAR2) RETURN ldap_groups_nt;
  FUNCTION get_user_groups_aa (p_user VARCHAR2) RETURN ldap_groups_aa;
  FUNCTION get_user_groups_va (p_user VARCHAR2) RETURN ldap_groups_va;
 
END ldap_util;
/
SHOW ERRORS
 
CREATE OR REPLACE PACKAGE BODY ldap_util AS
 
  FUNCTION get_user_groups_nt (p_user VARCHAR2) RETURN ldap_groups_nt IS
    l_groups ldap_groups_nt := ldap_groups_nt();
  BEGIN
    CASE p_user
      WHEN 'jaugusti' THEN
 
        l_groups.extend;
        l_groups(1).cn := 'TIMCO_GRP';
        l_groups(1).description := 'Employees working on the Timco Industries account.';
        l_groups(1).distinguished_name := 'CN=TIMCO_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(1).name := 'TIMCO_GRP';
        l_groups(1).samaccountname := 'TIMCO_GRP';
 
      WHEN 'kkidd' THEN
 
        l_groups.extend;
        l_groups(1).cn := 'MONEYBROS_GRP';
        l_groups(1).description := 'Employees working on the Moneybros account.';
        l_groups(1).distinguished_name := 'CN=MONEYBROS_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(1).name := 'MONEYBROS_GRP';
        l_groups(1).samaccountname := 'MONEYBROS_GRP';
 
      WHEN 'bhill' THEN
 
        l_groups.extend;
        l_groups(1).cn := 'TIMCO_GRP';
        l_groups(1).description := 'Employees working on the Timco Industries account.';
        l_groups(1).distinguished_name := 'CN=TIMCO_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(1).name := 'TIMCO_GRP';
        l_groups(1).samaccountname := 'TIMCO_GRP';
 
        l_groups.extend;
        l_groups(2).cn := 'MONACLE_GRP';
        l_groups(2).description := 'Employees working on the Monacle Corporation account.';
        l_groups(2).distinguished_name := 'CN=MONACLE_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(2).name := 'MONACLE_GRP';
        l_groups(2).samaccountname := 'MONACLE_GRP';
 
      WHEN 'kvongkas' THEN
 
        l_groups.extend;
        l_groups(1).cn := 'MONEYBROS_GRP';
        l_groups(1).description := 'Employees working on the Moneybros account.';
        l_groups(1).distinguished_name := 'CN=MONEYBROS_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(1).name := 'MONEYBROS_GRP';
        l_groups(1).samaccountname := 'MONEYBROS_GRP';
 
        l_groups.extend;
        l_groups(2).cn := 'MONACLE_GRP';
        l_groups(2).description := 'Employees working on the Monacle Corporation account.';
        l_groups(2).distinguished_name := 'CN=MONACLE_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(2).name := 'MONACLE_GRP';
        l_groups(2).samaccountname := 'MONACLE_GRP';
 
        l_groups.extend;
        l_groups(3).cn := 'SCHMAPPLE_GRP';
        l_groups(3).description := 'Employees working on the Schmapple account.';
        l_groups(3).distinguished_name := 'CN=SCHMAPPLE_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(3).name := 'SCHMAPPLE_GRP';
        l_groups(3).samaccountname := 'SCHMAPPLE_GRP';
 
    END CASE;
    RETURN l_groups;
  END get_user_groups_nt;
 
  FUNCTION get_user_groups_aa (p_user VARCHAR2) RETURN ldap_groups_aa IS
    l_groups ldap_groups_aa;
  BEGIN
    CASE p_user
      WHEN 'jaugusti' THEN
 
        l_groups(1).cn := 'TIMCO_GRP';
        l_groups(1).description := 'Employees working on the Timco Industries account.';
        l_groups(1).distinguished_name := 'CN=TIMCO_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(1).name := 'TIMCO_GRP';
        l_groups(1).samaccountname := 'TIMCO_GRP';
 
      WHEN 'kkidd' THEN
 
        l_groups(1).cn := 'MONEYBROS_GRP';
        l_groups(1).description := 'Employees working on the Moneybros account.';
        l_groups(1).distinguished_name := 'CN=MONEYBROS_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(1).name := 'MONEYBROS_GRP';
        l_groups(1).samaccountname := 'MONEYBROS_GRP';
 
      WHEN 'bhill' THEN
 
        l_groups(1).cn := 'TIMCO_GRP';
        l_groups(1).description := 'Employees working on the Timco Industries account.';
        l_groups(1).distinguished_name := 'CN=TIMCO_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(1).name := 'TIMCO_GRP';
        l_groups(1).samaccountname := 'TIMCO_GRP';
 
        l_groups(2).cn := 'MONACLE_GRP';
        l_groups(2).description := 'Employees working on the Monacle Corporation account.';
        l_groups(2).distinguished_name := 'CN=MONACLE_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(2).name := 'MONACLE_GRP';
        l_groups(2).samaccountname := 'MONACLE_GRP';
 
      WHEN 'kvongkas' THEN
 
        l_groups(1).cn := 'MONEYBROS_GRP';
        l_groups(1).description := 'Employees working on the Moneybros account.';
        l_groups(1).distinguished_name := 'CN=MONEYBROS_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(1).name := 'MONEYBROS_GRP';
        l_groups(1).samaccountname := 'MONEYBROS_GRP';
 
        l_groups(2).cn := 'MONACLE_GRP';
        l_groups(2).description := 'Employees working on the Monacle Corporation account.';
        l_groups(2).distinguished_name := 'CN=MONACLE_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(2).name := 'MONACLE_GRP';
        l_groups(2).samaccountname := 'MONACLE_GRP';
 
        l_groups(3).cn := 'SCHMAPPLE_GRP';
        l_groups(3).description := 'Employees working on the Schmapple account.';
        l_groups(3).distinguished_name := 'CN=SCHMAPPLE_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(3).name := 'SCHMAPPLE_GRP';
        l_groups(3).samaccountname := 'SCHMAPPLE_GRP';
 
    END CASE;
    RETURN l_groups;
  END get_user_groups_aa;
 
  FUNCTION get_user_groups_va (p_user VARCHAR2) RETURN ldap_groups_va IS
    l_groups ldap_groups_va := ldap_groups_va();
  BEGIN
    CASE p_user
      WHEN 'jaugusti' THEN
 
        l_groups.extend;
        l_groups(1).cn := 'TIMCO_GRP';
        l_groups(1).description := 'Employees working on the Timco Industries account.';
        l_groups(1).distinguished_name := 'CN=TIMCO_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(1).name := 'TIMCO_GRP';
        l_groups(1).samaccountname := 'TIMCO_GRP';
 
      WHEN 'kkidd' THEN
 
        l_groups.extend;
        l_groups(1).cn := 'MONEYBROS_GRP';
        l_groups(1).description := 'Employees working on the Moneybros account.';
        l_groups(1).distinguished_name := 'CN=MONEYBROS_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(1).name := 'MONEYBROS_GRP';
        l_groups(1).samaccountname := 'MONEYBROS_GRP';
 
      WHEN 'bhill' THEN
 
        l_groups.extend;
        l_groups(1).cn := 'TIMCO_GRP';
        l_groups(1).description := 'Employees working on the Timco Industries account.';
        l_groups(1).distinguished_name := 'CN=TIMCO_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(1).name := 'TIMCO_GRP';
        l_groups(1).samaccountname := 'TIMCO_GRP';
 
        l_groups.extend;
        l_groups(2).cn := 'MONACLE_GRP';
        l_groups(2).description := 'Employees working on the Monacle Corporation account.';
        l_groups(2).distinguished_name := 'CN=MONACLE_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(2).name := 'MONACLE_GRP';
        l_groups(2).samaccountname := 'MONACLE_GRP';
 
      WHEN 'kvongkas' THEN
 
        l_groups.extend;
        l_groups(1).cn := 'MONEYBROS_GRP';
        l_groups(1).description := 'Employees working on the Moneybros account.';
        l_groups(1).distinguished_name := 'CN=MONEYBROS_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(1).name := 'MONEYBROS_GRP';
        l_groups(1).samaccountname := 'MONEYBROS_GRP';
 
        l_groups.extend;
        l_groups(2).cn := 'MONACLE_GRP';
        l_groups(2).description := 'Employees working on the Monacle Corporation account.';
        l_groups(2).distinguished_name := 'CN=MONACLE_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(2).name := 'MONACLE_GRP';
        l_groups(2).samaccountname := 'MONACLE_GRP';
 
        l_groups.extend;
        l_groups(3).cn := 'SCHMAPPLE_GRP';
        l_groups(3).description := 'Employees working on the Schmapple account.';
        l_groups(3).distinguished_name := 'CN=SCHMAPPLE_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(3).name := 'SCHMAPPLE_GRP';
        l_groups(3).samaccountname := 'SCHMAPPLE_GRP';
 
    END CASE;
    RETURN l_groups;
  END get_user_groups_va;
 
END ldap_util;
/
SHOW ERRORS

As you can see, LDAP_UTIL has three functions that return the three kinds of collections in PL/SQL: nested tables, associative arrays, and varrays.

  FUNCTION get_user_groups_nt (p_user VARCHAR2) RETURN ldap_groups_nt;
  FUNCTION get_user_groups_aa (p_user VARCHAR2) RETURN ldap_groups_aa;
  FUNCTION get_user_groups_va (p_user VARCHAR2) RETURN ldap_groups_va;

The collections consist of a group of records representing “group” data from Active Directory.

  TYPE ldap_entry_typ IS RECORD (
    cn VARCHAR2(30),
    description VARCHAR2(1000),
    distinguished_name VARCHAR2(200),
    name VARCHAR2(30),
    samaccountname VARCHAR2(30));

The p_user parameter would be the user’s network id, corresponding to the sAMAccountName attribute in Active Directory; since we’re hardcoding group data this is not important to us at the moment. The point is you pass in a username and you get a collection of groups.

Using the same technique we demonstrated in part one, we now create a wrapper package of pipelined functions to transform the collection data into something we can query – an aggregate result set.

CREATE OR REPLACE PACKAGE ldap_util_pipelined AS
 
  FUNCTION get_user_groups_nt (p_user VARCHAR2) RETURN ldap_util.ldap_groups_nt PIPELINED;
  FUNCTION get_user_groups_aa (p_user VARCHAR2) RETURN ldap_util.ldap_groups_nt PIPELINED;
  FUNCTION get_user_groups_va (p_user VARCHAR2) RETURN ldap_util.ldap_groups_va PIPELINED;
 
END ldap_util_pipelined;
/
SHOW ERRORS
 
CREATE OR REPLACE PACKAGE BODY ldap_util_pipelined AS
 
  FUNCTION get_user_groups_nt (p_user VARCHAR2) 
  RETURN ldap_util.ldap_groups_nt PIPELINED IS
    l_groups ldap_util.ldap_groups_nt;
  BEGIN
    l_groups := ldap_util.get_user_groups_nt(p_user);
    FOR i IN 1 .. l_groups.count LOOP
      PIPE ROW (l_groups(i));
    END LOOP;
    RETURN;
  END get_user_groups_nt;
 
  FUNCTION get_user_groups_aa (p_user VARCHAR2) 
  RETURN ldap_util.ldap_groups_nt PIPELINED IS
    l_groups ldap_util.ldap_groups_aa;
  BEGIN
    l_groups := ldap_util.get_user_groups_aa(p_user);
    FOR i IN 1 .. l_groups.count LOOP
      PIPE ROW (l_groups(i));
    END LOOP;
    RETURN;
  END get_user_groups_aa;
 
  FUNCTION get_user_groups_va (p_user VARCHAR2) 
  RETURN ldap_util.ldap_groups_va PIPELINED IS
    l_groups ldap_util.ldap_groups_va;
  BEGIN
    l_groups := ldap_util.get_user_groups_va(p_user);
    FOR i IN 1 .. l_groups.count LOOP
      PIPE ROW (l_groups(i));
    END LOOP;
    RETURN;
  END get_user_groups_va;
 
END ldap_util_pipelined;
/
SHOW ERRORS

Note the return types of the three functions in ldap_util_pipelined.

  FUNCTION get_user_groups_nt (p_user VARCHAR2) RETURN ldap_util.ldap_groups_nt PIPELINED;
  FUNCTION get_user_groups_aa (p_user VARCHAR2) RETURN ldap_util.ldap_groups_nt PIPELINED;
  FUNCTION get_user_groups_va (p_user VARCHAR2) RETURN ldap_util.ldap_groups_va PIPELINED;

The first two return a nested table collection, while the third returns a VARRAY. These two collections can be pipelined because they correspond to native SQL datatypes in Oracle; not so for associative arrays. In order to pipeline the associative array returned from ldap_util.get_user_groups_aa, we need to map that collection to one of the other types. Here, we map it to a nested table. The iterative logic is the same, though, as the other two functions.

Now we can verify that our approach worked by issuing some simple queries, supported by the SQL “TABLE” construct.

SQL> select * from table(ldap_util_pipelined.get_user_groups_nt('jaugusti'));
 
CN        DESCRIPTION                                        DISTINGUISHED_NAME                            NAME      SAMACCOUN
--------- -------------------------------------------------- --------------------------------------------- --------- ---------
TIMCO_GRP Employees working on the Timco Industries account. CN=TIMCO_GRP,OU=Global Groups,DC=acme,DC=com  TIMCO_GRP TIMCO_GRP

As you can see, the individual attributes of the record structure form the columns of our new table, and we can reference those columns directly, if we wish.

SQL> select cn from table(ldap_util_pipelined.get_user_groups_nt('kvongkas'));
 
CN
---------------
MONEYBROS_GRP
MONACLE_GRP
SCHMAPPLE_GRP

All that’s left for us at this point is to construct a WHERE clause condition to use with Oracle’s Virtual Private Database feature to limit data returned from queries against the table QUARTERLY_SALES_DATA. That condition could look something like this, assuming that our application connects to the database as the logged in user.

client_id IN (SELECT CASE cn
                       WHEN 'TIMCO_GRP' THEN 100
                       WHEN 'MONEYBROS_GRP' THEN 200
                       WHEN 'MONACLE_GRP' THEN 300
                       WHEN 'SCHMAPPLE_GRP' THEN 400
                     END
                FROM TABLE(ldap_util_pipelined.get_user_groups_nt(USER)))

Here the mapping between AD group name and client ID is hardcoded in the CASE expression. A more flexible way would be to create a mapping table. Also, this is probably not a case study in writing a well performing VPD clause, but the real point here is that using SQL table functions and PL/SQL pipelined functions, we can translate data retrieved from an API, LDAP directory, or web service into table-like structures that we can query, join, and otherwise manipulate just like real tables.

Tweet

Making the assumption that you are using RMAN, you have already configured your backups to run a weekly full on each database to tape once per week, incrementals daily, and a reasonable amount of archivelogs being generated daily to the Fast Recovery Area, we can proceed to estimate your immediate needs.

If you are running Oracle and you are using RMAN, you can get the sizing numbers you need for your tape requirements somewhat easily if you are already taking backups to your Fast Recovery Area. There are a number of different views that can provide information about your backups, but I have found the most beneficial information can be gathered from the views used here. The helpful script that follows can be run to collect data on all the different types of backups you are currently running and the sizes (this script allows you to feed a list of SIDs):

#!/bin/bash

. /home/oracle/.bash_profile

ORAENV_ASK=NO

for DB in `echo “$DB_LIST”`; do
export ORACLE_SID=$DB
. oraenv $DB >/dev/null

echo “Database: $DB”

sqlplus -s \/ as sysdba<<EOF
set serveroutput on
set linesize 150
set pagesize 300

select ctime “Date”,

decode(backup_type, ‘L’, ‘Archive Log’, ‘D’, ‘Full’, ‘Incremental’) backup_type,bsize “Size GB” from (select trunc(bp.completion_time) ctime, backup_type, round(sum(bp.bytes/1024/1024/1024),2) bsize

from v\$backup_set bs,

v\$backup_piece bp where bs.set_stamp = bp.set_stamp and bs.set_count  = bp.set_count and bp.status = ‘A’ group by trunc(bp.completion_time), backup_type) order by 1, 2;

exit;
EOF
done

You can then look at the history of your current backups and right-size your tape requirements.

You might also check how many archive logs you are generating in a week under full production load:

col gbytes form 999,999.99

select sum(blocks*block_size)/1024/1024/1024 gbytes

from v$archived_log

where next_time > sysdate-8;

Some things to consider when you decide on your tape solution:

The type of tape drives

Encryption needs

The amount of compression

The number of tape drives

The type of backup software (OSB vs Netbackup vs Tivoli)

Your vaulting schedule (number of available media slots)

How fast can your channel send the data to the tape drives

Assuming you already have a tape SAN and you are not given enough money in your budget or enough lead time to get the gear in place before an audit, you might consider looking at what you have now in place and checking to see if you are doing too many backups, you have tuned your backups incorrectly, or if you have a number of backups that would benefit from VTL or dedup technologies in order to free space on your existing tape infrastructure.

Tweet

CREATE OR REPLACE PACKAGE example AS
 
     TYPE nested_table_typ IS TABLE OF VARCHAR2(100);
 
     FUNCTION get_nested_table_data RETURN nested_table_typ PIPELINED;
 
END example;
/
 
CREATE OR REPLACE PACKAGE BODY example AS
 
     FUNCTION get_nested_table_data RETURN nested_table_typ PIPELINED IS
     BEGIN
          PIPE ROW ('one');
          PIPE ROW ('two');
          PIPE ROW ('three');
          RETURN;
     END get_nested_table_data;
 
END example;
/

The package function example.get_nested_table_data returns a PL/SQL collection — in this case, a nested table. The PIPELINED directive instructs the function to return data to the calling query as rows are populated by the PIPE ROW command. That way, you don’t need to wait for the function to finish before you get results from your query.

The function can be invoked from the SELECT clause of a query in the usual way, and will return a valid result.

SELECT example.get_nested_table_data FROM dual;
 
GET_NESTED_TABLE_DATA
-------------------------------
SCHEMA.SYS_PLSQL_166672_9_1(one,two,three)

This is possible because nested tables and VARRAYs are supported as native types in SQL. The other type of PL/SQL collection, associative arrays, are not. Therefore you cannot use an associative array as the return type of a pipelined table function. However, it’s pretty trivial to create a wrapper function that translates an associative array into a nested table. You wouldn’t necessarily take this approach in your own code, but it comes in handy when you’re transforming data coming from a third party package or function.

But what if you need the contents of the collection presented as a table, with one record per entry in the collection? This is where the TABLE operator comes in handy.

SELECT column_value FROM TABLE(example.get_nested_table_data);
 
COLUMN_VALUE
-------------------
one
two
three

The TABLE operator casts a collection type as an aggregated result set, meaning it allows you to query it as if it were a table.

Why would you need to do this?

One reason might be because you want to query or join data retrieved from a web service, or from an LDAP server. Another reason might be to simplify a complex query with lots of unions, case statements, and subqueries by fetching, manipulating, and combining your data inside a PL/SQL function. You can leverage the ease and maintainability of building your collection in a procedural manner, while still having access to the collection data from SQL. There is some overhead associated with that approach, though, so you’ll want to be careful about applying it in performance-critical situations. Still, though, it’s worth considering when the complexity of SQL constructs gets out of hand.

Tweet

SELECT * FROM RECYCLEBIN;

Dropped tables and any associated objects such as indexes, constraints, nested tables, are not removed when the table is dropped and still occupy space in the tablespace where they originated. They continue to count against user space quotas, until specifically purged from the recyclebin either manually or by the database because of tablespace space constraints.

When you drop a tablespace including its contents, the objects in the tablespace are NOT placed in the recycle bin AND the database purges any entries in the recycle bin for objects located in that tablespace.

When you drop a user, any objects belonging to the user are not placed in the recycle bin and any objects in the recycle bin are purged that belong to that user.

When you drop a cluster, its member tables are not placed in the recycle bin and any former member tables in the recycle bin are purged that belong to that cluster.

When you drop a type, any dependent objects such as subtypes are not placed in the recycle bin and any former dependent objects in the recycle bin are purged.

Recyclebin Settings
In 10g release 1, recyclebin was enabled by default and there was only an undocumented parameter to disable it. This parameter was _recyclebin which defaulted to a value of TRUE. The recyclebin initialization parameter was introduced in 10g release 2 and had a default setting of ‘on’. It was a dynamic parameter and could be enabled or disabled with either an ‘alter session’ or an ‘alter system’ command. Oracle 11g release 1 made no changes to the basic feature, except to add the ‘deferred’ option to the ‘alter session’ command to allow subsequent, but not the current, session to be affected. Release 2 remained unchanged from release 1.
To find out if the recyclebin is enabled, except for 10g release 1, you can execute ‘show parameter recyclebin’ from a SQL prompt as a user with sysdba privileges.
Sample output:

SQL> show parameter recyclebin
 
NAME                                 TYPE                             VALUE
------------------------------------ -------------------------------- ------------------------------
recyclebin                           string                           on

I would recommend keeping the recyclebin enabled at all times, except during an upgrade, to add an extra layer of recovery options to the existing environment.

Using Recyclebin

Once recyclebin is enabled, you don’t have to give much thought to its use. Every time a ‘drop table’ command is executed, the table and its associated objects (indexes, constraints, nested tables) are placed in the recyclebin and renamed with a system-generated naming standard. Because objects age out of the recyclebin based on demands for space, it is always a good idea to back up a table with an export or a create table as select (CTAS) option before dropping it. This is especially true for dropping very large tables which may not fit in the recyclebin.

In the event that you do not want a table to be preserved in the recyclebin, the addition of the ‘purge’ option at the end of the ‘drop table’ statement ensures that the recyclebin bin is bypassed and the table as well as the associated objects are dropped from the database.

Purging the recyclebin can be performed at both the user and system level. As stated earlier, each named account in the database will automatically have a recyclebin assigned to it. A user can simply execute the ‘purge recyclebin’ command to purge their individual recyclebin. Users with the sysdba privilege can purse the entire recyclebin with the ‘purge dba_recyclebin’ command. Be careful, though, as this command also purges the users’ individual recyclebins as well. Purging the entire reyclebin should be reserved for upgrade situations, or unless otherwise directed by Oracle Support.

You can also perform selective purges of the recyclebin. For example ‘purge tablespace tmpdata’ only purges those objects in the recyclebin belonging to the tmpdata tablespace. Also, ‘purge tablespace tmpdata user jon’ will purge only those objects in the recyclebin belonging to the tmpdata tablespace owned by the named user jon. You can also purge individual tables from the recyclebin with the command ‘purge table
’. The table name in this command can be either the system-generated recyclebin name or the original name of the table.

To recover a table from the recyclebin, you need to know if it is available. Use the following query to get this information:

select owner, object_name, original_name, droptime
from dba_recyclebin
where owner = ‘CMN’;
 
OWNER                          OBJECT_NAME                    ORIGINAL_NAME                    DROPTIME
------------------------------ ------------------------------ -------------------------------- -------------------
CMN                            BIN$o5lg4yWjH3ngRAAUT0rhuQ==$0 CMN_BASE_CONTACT                 2011-05-18:23:12:42
CMN                            BIN$o5lg4yWiH3ngRAAUT0rhuQ==$0 CMN_BASE_CONTACT_PK              2011-05-18:23:12:42
CMN                            BIN$o5icV6rtByXgRAAUT0rhuQ==$0 CMN_BASE_CONTACT                 2011-05-18:22:51:07
CMN                            BIN$o5icV6rfByXgRAAUT0rhuQ==$0 CMN_BASE_CONTACT                 2011-05-18:22:17:45
CMN                            BIN$o5lg4yWvH3ngRAAUT0rhuQ==$0 CMN_BASE_CONTACT                 2011-05-18:23:15:11
CMN                            BIN$o5icV6rgByXgRAAUT0rhuQ==$0 EXECUTE_TABLE                    2011-05-18:22:49:58
CMN                            BIN$o5mU5fo6KF/gRAAUT0rhuQ==$0 EXECUTE_TABLE                    2011-05-18:23:30:15
CMN                            BIN$o5icV6ruByXgRAAUT0rhuQ==$0 EXECUTE_TABLE                    2011-05-18:22:53:23
CMN                            BIN$o5mU5foqKF/gRAAUT0rhuQ==$0 CMN_BASE_CONTACT                 2011-05-18:23:27:15
CMN                            BIN$o5mU5fo5KF/gRAAUT0rhuQ==$0 CMN_BASE_CONTACT                 2011-05-18:23:30:08
CMN                            BIN$o5mU5fo4KF/gRAAUT0rhuQ==$0 CMN_BASE_CONTACT_PK              2011-05-18:23:30:08
CMN                            BIN$o5mU5fpGKF/gRAAUT0rhuQ==$0 CMN_BASE_CONTACT                 2011-05-18:23:32:02
CMN                            BIN$o5mU5fpHKF/gRAAUT0rhuQ==$0 EXECUTE_TABLE                    2011-05-18:23:32:02

NOTE – The owner clause in this query was only given to limit the size of the output.

From this output, you can see which tables are present in the recyclebin and when they were dropped. As you can see, there can be several versions of the same table in the recyclebin.

This brings us to the purpose of the recyclebin – recovering an accidentally dropped table. What you may or may not know is that the recyclebin tables can be queried like normal tables. You can query the recyclebin copies of the table until you find the one that contains the data you are interested in…

select count(*)
from cmn."BIN$o5mU5fpGKF/gRAAUT0rhuQ==$0";
 
  COUNT(*)
----------
        89

Note that it is important to use both the owner identifier (CMN) and double-quotation marks because of the special characters in the recyclebin table name.
Once the table you want to recover has been identified, there are a few different methods by which you can recover the table or the individual data. The ‘flashback table
to before drop’ command will recover the latest copy of the table in the recyclebin. You can also specify the ‘rename to ’ at the end of the command to recover the table to a different table name. In the case where there are several copies of the table in the recyclebin and you are not interested in the latest copy, you can do a simple create table as select (CTAS) operation using the recyclebin table name, not the original name. Remember to use double-quotation marks around the recyclebin table name due to the special characters. You should also be able to recover select rows from the recyclebin table using the same method.

Summary
In situations where the code we deploy is beyond our control, this feature may prove invaluable. It is one of those features that you can set and forget, but are very thankful when it is available and you need it.

By default, crontab runs scripts in the user’s home directory, as specified in /etc/passwd. For user “Tim” with home directory /home/tim, a sample crontab might look something like the following:

0 0 * * * runme.sh

Every night at midnight, cron will try to run the file /home/tim/runme.sh. However, if runme.sh is actually in a sub-directory under ~tim, you will need to tell cron how to reach the file. One option would be to qualify the path in crontab, like so:

0 0 * * * ~/bin/runme.sh

Now cron can find the file. However, if the script makes references to other files inside of /bin in a path-relative way, those references will be broken.

#!/bin/bash
#runme.sh
./runthis.sh      # can't find this file when running under cron
./runthat.sh      # can't find this file when running under cron

That’s because to cron ./ is not the directory in which the script is running, but rather whatever is defined as the base directory; in the default case, the user’s home directory.

It is possible to specify the base directory in the crontab file itself by defining the special variable HOME.

HOME=/home/tim/bin
0 0 * * * runme.sh

However, you may not always have control over how your crontab is configured. Perhaps a system administrator is in charge of all the schedules on your server. Or you may organize your scripts in several different directories.

An easy solution is to start your bash scripts with the following command:

cd $(dirname $0)

The program dirname returns the directory portion of the fully qualified path name sent to it. The $0 argument in this context is name of the running script. Basically, this code ensures that the current working directory is the same as the directory in which your script is running. That way all your path-relative references are correct whether running manually or through a scheduler.

Tweet

One problem with this approach is how to deal with “stale data”. In other words, what happens when data is changing in the database? How does the developer ensure that the cached data is current?

I ran into this issue recently when consulting with one of our clients on a third-party J2EE application that used Toplink for its persistence layer. What we noticed was that at a certain point in the business process the application was clearly displaying stale data. The “status” column on a certain record was being changed in the database (as could be verified by a SQL query), but the application still displayed the prior status. Refreshing the screen and requerying the data had no effect. The problem was that the status was being changed through a custom stored procedure which performed direct updates on the screen’s underlying table. This approach was technically against the directives of the third-party application vendor, who insisted that all such changes to the database be made via calls to its own web services. That was for good reason. The web services they provided as an API for custom development were the same ones used by the screens to query data. They used the same Toplink Session Cache. Changes made through the web services, therefore, would have updated the cache, thereby avoiding the problem. As it was, the client had unintentionally created out-of-sync scenario between the cache and the database, and the status information displayed on the screen was invalid.

The most obvious solution would have been to modify the custom procedure to make use of the provided APIs. However, we didn’t have an API in this instance that provided exactly the required functionality. Having the software vendor modify or extend the API for our purposes would have been prohibitively expensive, in both time and money.

Another possibility we considered was to have the vendor make a smaller modification to bypass the Toplink cache for queries on that screen. It is possible to disable caching per table, through the global configuration file persistence.xml, or as a property of an individual query. However, the vendor was not predisposed to introduce what they viewed as a “hack” to accommodate our custom functionality.

Ultimately we decided to implement a tacked-on solution that would arrive at the same goal and eliminate the need for an additional iteration through the vendor’s dev/test cycle. We created a custom servlet, installed in the third-party application itself, that invalidated the Toplink Session Cache for a specified table or, as an additional method, for all tables. Invalidating the cache means that the next time a query is executed against that table, the cache is bypassed and data is retrieved directly from the database. To make it easier for the non-Java in-house developers to use, we created a custom wrapper package to mask the call to the servlet. A call to this package was added to the client’s custom code just after the record in question was updated and commited.

The solution consisted of two Java classes, JAR’ed and deployed as a library in the third-party application’s WEB-INF/lib directory.

CacheInvalidator.java
CacheInvalidatorServlet.java

The web application configuration file web.xml was also modified to add a reference to the new “cacheinvalidator” servlet.

The code was adapted from the following documentation, available on Oracle Tech Net:

Database Change Notification and TopLink Cache Invalidation
http://www.oracle.com/technetwork/middleware/ias/index-097190.html

Oracle TopLink Cache Invalidation
http://www.oracle.com/technetwork/middleware/ias/index-100679.html

The two methods of CacheInvalidator that do the actual work are shown here. The first takes tablename as an argument, so that the invalidation can be targeted. The second invalidates, or effectively wipes, the entire cache.

public void invalidateClass(Session session, String tableName) throws Exception {
    if(tableName == null || tableName.trim().equals("")) {
        throw new Exception("tableName cannot be null");
    }
    Class baseClass = (Class)tableNameToClass.get(tableName);
    if(baseClass == null) {
        throw new Exception("table \"" + tableName + "\" not found in cache");
    }
 
    // invalidate in TopLink cache the object corresponding to the baseClass
    session.getIdentityMapAccessor().invalidateClass(baseClass);
}
 
// invalidate all objects in the cache
public void invalidateAllClasses(Session session) {
    // invalidate in TopLink cache the object corresponding to the baseClass
    session.getIdentityMapAccessor().invalidateAll();
}

The CacheInvalidatorServlet takes a table name or the special keyword “ALL” as a parameter and calls the corresponding method in CacheInvalidator.

String oName = request.getParameter("objectName");
String responseString = "";
 
try {
    SessionManager manager = SessionManager.getManager();
    //SESSIONNAME defined in sessions.xml
    Server server = (Server)manager.getSession("SESSIONNAME");
    Session session = server.getActiveSession();
 
    CacheInvalidator invalidator = new CacheInvalidator(session);
    if (oName.equals("ALL")) {
        invalidator.invalidateAllClasses(session);
        responseString = "successfully invalidated toplink cache for all tables.";
    } else {
        invalidator.invalidateClass(session, oName);
        responseString = "successfully invalidated toplink cache for table: "+oName;
    }
} catch (Exception e) {
    responseString = e.getMessage();
}

Caling a servlet from PL/SQL is not particularly complicated, but in our case involved the lookup of server information stored in the database. The wrapper package allowed us to provide the developers with a simple, one-line invocation.

p_servlet_response := utl_http.request('http://'||l_app_server_host||':'
          ||l_app_server_port||'/jedi/cacheinvalidatorservlet?objectName='
          ||p_obj_to_invalidate);

After making a call to our cacheinvalidator package subsequent to updating the status record, the stale data problem was eliminated in the application.

Tweet

We will start with a typical query you would execute to find out who is accessing
certain objects in the database: select distinct owner
from v$access;

Our test system: Oracle 11.2.0.1.0 64-bit on Sun SPARC Solaris version 10

SQL> set timing on
SQL> select distinct owner
  2  from v$access;
 
 
from v$access
     *
ERROR at line 2:
ORA-01013: user requested cancel of current operation
 
Elapsed: 00:08:13.62

Not exactly the performance we expect from this Oracle database version. What’s going on?

NOTE – SQL_ID = f9u1sd8tq9rcm (obtained from querying v$session during execution)

Let’s look at the explain plan for this query. Since we want to get all the information
possible, we use the following alter session command and a call to the dbms_xplan package.

alter session set statistics_level=ALL;
 
Session altered.
 
--We execute the query again with the gather_plan_statistics hint to generate extended statistics.
 
SQL> select /*+ gather_plan_statistics */ distinct owner from v$access;
We know the query probably won't finish, so we kill it with a  after a minute.

NOTE – SQL_ID = dmfdcmvwkgfqh (obtained from querying v$session during execution)
The SQL_ID changed because of the hint.

select * from table(dbms_xplan.display_cursor('&sql_id','&child_no','allstats'))
/
 
Enter value for sql_id: dmfdcmvwkgfqh
Enter value for child_no: 0
old   1: select * from table(dbms_xplan.display_cursor('&sql_id','&child_no','allstats'))
new   1: select * from table(dbms_xplan.display_cursor('dmfdcmvwkgfqh','0','allstats'))
 
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL_ID  dmfdcmvwkgfqh, child number 0
-------------------------------------
select /*+ gather_plan_statistics */ distinct owner from v$access
 
Plan hash value: 3850776806
 
-------------------------------------------------------------------------------------------------------------------------
| Id  | Operation                  | Name            | Starts | E-Rows | A-Rows |   A-Time   |  OMem |  1Mem |  O/1/M   |
-------------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT           |                 |      1 |        |      0 |00:00:00.01 |       |       |          |
|   1 |  HASH UNIQUE               |                 |      1 |      1 |      0 |00:00:00.01 |   741K|   741K|          |
|   2 |   NESTED LOOPS             |                 |      1 |      1 |     32 |00:00:04.04 |       |       |          |
|   3 |    NESTED LOOPS            |                 |      1 |      1 |     32 |00:00:04.04 |       |       |          |
|   4 |     MERGE JOIN CARTESIAN   |                 |      1 |    100 |   2291K|00:00:11.60 |       |       |          |
|*  5 |      FIXED TABLE FULL      | X$KSUSE         |      1 |      1 |     47 |00:00:00.01 |       |       |          |
|   6 |      BUFFER SORT           |                 |     47 |    100 |   2291K|00:00:08.38 |  3454K|   810K|     1/0/0|
|   7 |       FIXED TABLE FULL     | X$KGLDP         |      1 |    100 |  48886 |00:00:00.33 |       |       |          |
|*  8 |     FIXED TABLE FIXED INDEX| X$KGLLK (ind:1) |   2291K|      1 |     32 |00:01:13.38 |       |       |          |
|*  9 |    FIXED TABLE FIXED INDEX | X$KGLOB (ind:1) |     32 |      1 |     32 |00:00:00.01 |       |       |          |
-------------------------------------------------------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
   5 - filter("S"."INST_ID"=USERENV('INSTANCE'))
   8 - filter(("L"."KGLLKUSE"="S"."ADDR" AND "L"."KGLLKHDL"="D"."KGLHDADR" AND "L"."KGLNAHSH"="D"."KGLNAHSH"))
   9 - filter(("O"."KGLNAHSH"="D"."KGLRFHSH" AND "O"."KGLHDADR"="D"."KGLRFHDL"))
 
28 rows selected.

A ‘merge join cartesian’ operation is at the heart of this query. This operation was common in a 10g environment, and is not uncommon in 11g, apparently. This operation mixed with bad cardinality estimates can really hang a query.
The bad estimates I refer too can be seen in this plan as an enormous difference between what the optimizer estimates (E-Rows) the number of rows it will need to access compared to the actual (A-Rows) number of rows that will need to
be accessed. A typical cause of this disparity is statistics that are severely out of date or non-existent. We can check the number of fixed tables that have statistics with the following query (for a 10gR2 database, it should return about 790 rows):

SQL> select count(*) from sys.tab_stats$;
 
  COUNT(*)
----------
         0

If no rows are returned (as was the case in this system), then no fixed object statistics are present. This could be the cause of the problem querying the v$access view. We can check this assumption further by trying to remove the optimizer from the equation altogether with the use of a query hint:

SQL> select /*+ RULE */ distinct owner from v$access;
 
OWNER
----------------------------------------------------------------
0
ATG_ADMIN
ATG_CATALOG1
DBSNMP
JRADAMS
PUBLIC
STOREORD
SYS
SYSTEM
TCSWEB
WEBJMS
 
12 rows selected.
 
Elapsed: 00:00:00.55

As you can see, the query returned in less than a second. Let’s look at the explain plan (we won’t bother trying to look at all the stats since we bypassed the optimizer):

SQL> set autotrace traceonly explain
SQL> select /*+ RULE */ distinct owner from v$access;
Elapsed: 00:00:00.00
 
Execution Plan
----------------------------------------------------------
Plan hash value: 868229819
 
-------------------------------------------
| Id  | Operation               | Name    |
-------------------------------------------
|   0 | SELECT STATEMENT        |         |
|   1 |  SORT UNIQUE            |         |
|   2 |   MERGE JOIN            |         |
|   3 |    SORT JOIN            |         |
|   4 |     MERGE JOIN          |         |
|   5 |      SORT JOIN          |         |
|   6 |       MERGE JOIN        |         |
|   7 |        SORT JOIN        |         |
|   8 |         FIXED TABLE FULL| X$KGLLK |
|*  9 |        SORT JOIN        |         |
|  10 |         FIXED TABLE FULL| X$KGLDP |
|* 11 |      SORT JOIN          |         |
|  12 |       FIXED TABLE FULL  | X$KGLOB |
|* 13 |    SORT JOIN            |         |
|* 14 |     FIXED TABLE FULL    | X$KSUSE |
-------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
   9 - access("L"."KGLLKHDL"="D"."KGLHDADR" AND
              "L"."KGLNAHSH"="D"."KGLNAHSH")
       filter("L"."KGLNAHSH"="D"."KGLNAHSH" AND
              "L"."KGLLKHDL"="D"."KGLHDADR")
  11 - access("O"."KGLNAHSH"="D"."KGLRFHSH" AND
              "O"."KGLHDADR"="D"."KGLRFHDL")
       filter("O"."KGLHDADR"="D"."KGLRFHDL" AND
              "O"."KGLNAHSH"="D"."KGLRFHSH")
  13 - access("L"."KGLLKUSE"="S"."ADDR")
       filter("L"."KGLLKUSE"="S"."ADDR")
  14 - filter("S"."INST_ID"=USERENV('INSTANCE'))
 
Note
-----
   - rule based optimizer used (consider using cbo)

Although there are several merge joins as well as several full table scans present,
there is no sign of the ‘merge join cartesian’ from the earlier optimizer explain plan. This should be enough proof that the optimizer does not have the information it needs to produce the best results.

The RULE hint is a definite work around for this problem, but Oracle has been stating
that the rule-based optimizer has been deprecated for years, so there is no guarantee how much longer this hint will be effective as a workaround. We should try to solve this issue working with the optimizer, and not bypassing it, because improving the optimizer’s function will bring benefits to the entire system

Generating new fixed object statistics is a database-wide change though that needs to be approached with caution. Even though there is some evidence that it would
make a positive impact in this situation, it could have a negative impact on other
processes that have higher importance, especially if this is an established production
system where performance is at an expected level.

In the case where we cannot readily generate new fixed object statistics, there are
optimizer-based workarounds in the form of these hints:

_optimizer_cartesian_enabled
_optimizer_mjc_enabled

The _optimizer_cartesian_enabled parameter was introduced in 10gR2 as a means to stop the merge join cartesian operation. This parameter is set to TRUE by default. It can be set at the session or system level. Let’s set it for this session, then try our test query again:

SQL> set autotrace off;
 
SQL> alter session set "_optimizer_cartesian_enabled"=FALSE;
 
Session altered.
 
SQL> col pd_name format a50
SQL> col pd_value format a50
SQL> select n.ksppinm pd_name, c.ksppstvl pd_value
from x$ksppi n, x$ksppcv c
where n.indx=c.indx
and (
   lower(n.ksppinm) like lower('_optimizer_cartesian_enabled')
   or lower(n.ksppdesc) like lower('_optimizer_cartesian_enabled')
)
/
 
PD_NAME                                  PD_VALUE
---------------------------------------- --------------------
_optimizer_cartesian_enabled             FALSE
 
SQL> set timing on
SQL> select distinct owner
  2  from v$access;
 
OWNER
----------------------------------------------------------------
 
PUBLIC
SYSTEM
SYS
WEBJMS
STOREORD
0
JRADAMS
ATG_ADMIN
ATG_CATALOG1
TCSWEB
DBSNMP
 
12 rows selected.
 
Elapsed: 00:00:03.57

Looks like it worked in this particular situation. As for the other hint, _optimizer_mjc_enabled, which is TRUE by default also, there are references online that point to setting this parameter to FALSE as well in order to have the optimizer avoid cartesian merge joins as well. MOS note 457058.1 outlines the use of this parameter. We can try it here to test the theory:

We will set the _optimizer_cartesian_enabled parameter back to its default value first:

SQL> alter session set "_optimizer_cartesian_enabled"=TRUE;
 
Session altered.
 
SQL> select n.ksppinm pd_name, c.ksppstvl pd_value
from x$ksppi n, x$ksppcv c
where n.indx=c.indx
and (
   lower(n.ksppinm) like lower('_optimizer_cartesian_enabled')
   or lower(n.ksppdesc) like lower('_optimizer_cartesian_enabled')
)
/ 
 
PD_NAME                                  PD_VALUE
---------------------------------------- --------------------
_optimizer_cartesian_enabled             TRUE
 
--Now we can set the _optimizer_mjc_enabled parameter:
 
SQL> alter session set "_optimizer_mjc_enabled"=FALSE;
 
Session altered.
 
SQL> select n.ksppinm pd_name, c.ksppstvl pd_value
from x$ksppi n, x$ksppcv c
where n.indx=c.indx
and (
   lower(n.ksppinm) like lower('_optimizer_mjc_enabled')
   or lower(n.ksppdesc) like lower('_optimizer_mjc_enabled')
)
/  
 
PD_NAME                                  PD_VALUE
---------------------------------------- --------------------
_optimizer_mjc_enabled                   FALSE
 
--Now for the query:
 
SQL> select distinct owner
from v$access;  
 
 
ERROR at line 1:
ORA-01013: user requested cancel of current operation
 
Elapsed: 00:03:09.75

What does the explain plan look like?

Don’t forget to run the query again with the gather_plan_statistics hint

select /*+ gather_plan_statistics */ distinct owner from v$access;
 
select * from table(dbms_xplan.display_cursor('&sql_id','&child_no','allstats'))
/
 
Enter value for sql_id: dmfdcmvwkgfqh
Enter value for child_no: 0
old   1: select * from table(dbms_xplan.display_cursor('&sql_id','&child_no','allstats'))
new   1: select * from table(dbms_xplan.display_cursor('dmfdcmvwkgfqh','0','allstats'))
 
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL_ID  dmfdcmvwkgfqh, child number 0
-------------------------------------
select /*+ gather_plan_statistics */ distinct owner from v$access
 
Plan hash value: 3850776806
 
-------------------------------------------------------------------------------------------------------------------------
| Id  | Operation                  | Name            | Starts | E-Rows | A-Rows |   A-Time   |  OMem |  1Mem |  O/1/M   |
-------------------------------------------------------------------------------------------------------------------------
|   0 | SELECT STATEMENT           |                 |      1 |        |      0 |00:00:00.01 |       |       |          |
|   1 |  HASH UNIQUE               |                 |      1 |      1 |      0 |00:00:00.01 |   741K|   741K|          |
|   2 |   NESTED LOOPS             |                 |      1 |      1 |     32 |00:00:04.04 |       |       |          |
|   3 |    NESTED LOOPS            |                 |      1 |      1 |     32 |00:00:04.04 |       |       |          |
|   4 |     MERGE JOIN CARTESIAN   |                 |      1 |    100 |   2291K|00:00:11.60 |       |       |          |
|*  5 |      FIXED TABLE FULL      | X$KSUSE         |      1 |      1 |     47 |00:00:00.01 |       |       |          |
|   6 |      BUFFER SORT           |                 |     47 |    100 |   2291K|00:00:08.38 |  3454K|   810K|     1/0/0|
|   7 |       FIXED TABLE FULL     | X$KGLDP         |      1 |    100 |  48886 |00:00:00.33 |       |       |          |
|*  8 |     FIXED TABLE FIXED INDEX| X$KGLLK (ind:1) |   2291K|      1 |     32 |00:01:13.38 |       |       |          |
|*  9 |    FIXED TABLE FIXED INDEX | X$KGLOB (ind:1) |     32 |      1 |     32 |00:00:00.01 |       |       |          |
-------------------------------------------------------------------------------------------------------------------------
 
Predicate Information (identified by operation id):
---------------------------------------------------
 
   5 - filter("S"."INST_ID"=USERENV('INSTANCE'))
   8 - filter(("L"."KGLLKUSE"="S"."ADDR" AND "L"."KGLLKHDL"="D"."KGLHDADR" AND "L"."KGLNAHSH"="D"."KGLNAHSH"))
   9 - filter(("O"."KGLNAHSH"="D"."KGLRFHSH" AND "O"."KGLHDADR"="D"."KGLRFHDL"))
 
28 rows selected.

It looks like the _optimizer_mjc_enabled parameter is not as affective in stopping the merge join cartesian operations as the _optimizer_cartesian_enabled parameter.

I would suggest enabling hints like this at the session level first to verify behavior against certain operations, then only enabling them for those operations. In my opinion, disabling merge join cartesian operations altogether would introduce behavior into a system whose consequences should be thoroughly tested beforehand. Don’t get caught up in the ‘silver bullet’ mentality when you try to apply a fix for one process against the entire system thinking it is a cure for all your database woes.

So now that we have a work around, we don’t need to worry about this situation anymore, right? I would think that most of us want to fix the underlying issue and not ‘patch it and go’. I want to know what happens if we do generate the new fixed table statistics we want. Fortunately, I was lucky enough to have a test system where I was able to recreate this problem. Here is what happened when fixed table statistics were generated:

First, make sure the hints we were working with have been set back to their default values.

SQL> alter session set "_optimizer_cartesian_enabled"=TRUE;
 
Session altered.
 
SQL> alter session set "_optimizer_mjc_enabled"=TRUE;
 
Session altered.
 
SQL> select n.ksppinm pd_name, c.ksppstvl pd_value
from x$ksppi n, x$ksppcv c
where n.indx=c.indx
and (
   lower(n.ksppinm) like lower('_optimizer%enabled')
   or lower(n.ksppdesc) like lower('_optimizer%enabled')
)
/
 
PD_NAME                                  PD_VALUE
---------------------------------------- --------------------
_optimizer_mjc_enabled                   TRUE
_optimizer_cartesian_enabled             TRUE

Note – There were more records in the actual output, but I abreviated the output
to avoid confusion.

Now generate new fixed table statistics:

********************************************************************************

NOTE – THERE IS A POSSIBILITY OF HITTING A BUG NATIVE TO 11.2.0.1 WHEN TRYING TO GENENERATE FIXED OBJECT STATISTICS.

SQL> exec dbms_stats.gather_fixed_objects_stats;
BEGIN dbms_stats.gather_fixed_objects_stats; END;
 
*
ERROR at line 1:
ORA-20011: Approximate NDV failed: ORA-00439: feature not enabled: Data Guard
Broker
ORA-06512: at “SYS.DBMS_STATS”, line 20508
ORA-06512: at “SYS.DBMS_STATS”, line 20951
ORA-06512: at “SYS.DBMS_STATS”, line 21498
ORA-06512: at line 1

MOS note 9471411.8 outlines bug 9471411, which is the cause of this error.
This bug is fixed in 11.2.0.2, 11.2.0.1.3, and 12.1 (future release).

If by chance you encounter this bug, there will still be fixed objects that
will have stats generated. Retry the query after the bug is encountered.

********************************************************************************

SQL> exec dbms_stats.gather_fixed_objects_stats;
BEGIN dbms_stats.gather_fixed_objects_stats; END;
 
PL/SQL procedure successfully completed.
 
SQL> select count(*) from sys.tab_stats$;
 
  COUNT(*)
----------
       798
 
Elapsed: 00:00:00.01

Now let’s flush the old SQL out of the shared pool to ensure we get fresh explain plans:

SQL> alter system flush shared_pool;
 
System altered.
 
Elapsed: 00:00:03.38
 
SQL> select distinct owner
  2  from v$access;
 
OWNER
----------------------------------------------------------------
SYSTEM
PUBLIC
SYS
TKCSOWNER
SUPRKRON
0
 
6 rows selected.
 
Elapsed: 00:00:00.28

This is what we should have been getting in terms of performance from the beginning.

What does the explain plan say?

select /*+ gather_plan_statistics */ distinct owner from v$access;
 
select * from table(dbms_xplan.display_cursor('&sql_id','&child_no','allstats'))
/
 
Enter value for sql_id: dmfdcmvwkgfqh
Enter value for child_no: 0
old   1: select * from table(dbms_xplan.display_cursor('&sql_id','&child_no','allstats'))
new   1: select * from table(dbms_xplan.display_cursor('dmfdcmvwkgfqh','0','allstats'))
 
PLAN_TABLE_OUTPUT
--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
SQL_ID  dmfdcmvwkgfqh, child number 0
-------------------------------------
select /*+ gather_plan_statistics */ distinct owner from v$access
 
Plan hash value: 174934893
 
-------------------------------------------------------------------------------------------------------------------------
| Id  | Operation                  | Name            | Starts | E-Rows | A-Rows |   A-Time   |  OMem |  1Mem |  O/1/M   |
-------------------------------------------------------------------------------------------------------------------------
|   1 |  HASH UNIQUE               |                 |      1 |     40 |      6 |00:00:00.44 |  1207K|  1207K|     1/0/0|
|   2 |   NESTED LOOPS             |                 |      1 |   2327 |    500 |00:00:00.39 |       |       |          |
|   3 |    NESTED LOOPS            |                 |      1 |   2327 |    500 |00:00:00.34 |       |       |          |
|   4 |     HASH JOIN              |                 |      1 |   2327 |   2400 |00:00:00.26 |  1236K|  1236K|     1/0/0|
|   5 |      FIXED TABLE FULL      | X$KSUSE         |      1 |   1105 |   1105 |00:00:00.01 |       |       |          |
|   6 |      FIXED TABLE FULL      | X$KGLLK         |      1 |   2327 |   2400 |00:00:00.02 |       |       |          |
|   7 |     FIXED TABLE FIXED INDEX| X$KGLDP (ind:1) |   2400 |      1 |    500 |00:00:00.08 |       |       |          |
|   8 |    FIXED TABLE FIXED INDEX | X$KGLOB (ind:1) |    500 |      1 |    500 |00:00:00.04 |       |       |          |
-------------------------------------------------------------------------------------------------------------------------
 
19 rows selected.

No merge join cartesian operations and the number of estimated rows (E-Rows) is a lot closer to the number of actual rows (A-Rows) than the previous optimizer-based plan.

Summary

We have seen that fixed object statistics are important to the optimizer. Fixed objects are used in all kinds of recursive statements that can affect user performance, so enabling the optimizer to access them effectively is a benefit that can outweigh most any possible risk of gathering them in a database where they do not exist at all.

When considering jobs to generate statistics, don’t forget the fixed tables in this plan. As these objects change less frequently than most objects in a production system, once a month should be a sufficient starting point. Of course, you wouls want to re-gather these statistics when the load caracteristics of your system change, after patches, after hardware upgrades or migrations. Think about gathering statistics on your dictionary objects (dbms_stats.gather_dictionary_stats) as well when considering the fixed object stats. Once a month should be a sufficient starting point as well for these statistics.

Was the tightening of space sudden or gradual?

The answer was that the space was being eaten up gradually. At that point, I reviewed the ‘usual suspects’ when looking to clear space from an Oracle database server. These include:

• Overly-large alert.log or listener.log files
• Archived copies of the alert.log or listener.log files (seen it done).
• Excessive trace files or core dumps (this would also prompt me to look for errors in the alert.log).
• Excessive .aud files under $ORACLE_HOME/rdbms/audit.
• Executing ‘sudo du -ak | sort –nr > ./out.log’ in a Unix environment at the top of the file system in question will generate a list of space hogs in a file call out.log in the current directory. You may find some old exports to remove. If removal is not an option, at least compress them.

Once the usual suspects were covered, space was still tight, so the removal of the old 8i Oracle software seemed a good way to free up some space.

I wanted to use the 10g Oracle Universal Installar (OUI) and the next problem I was facing was a lack of an X-window capability on the server and no one present to tell me if it was even configured. From previous experience, setting up X-window capabilities is time-consuming and often frustrating; whether you are using Cygwin, VNCServer, or another piece of client software.
Wouldn’t it be good to be able to run one command silently and have Oracle remove the old home without having to go through the graphical, interactive OUI? The OUI has had a silent method of execution ever since OUI first appeared in version 8i. It can also perform silent de-installs. You just have to perform a few pre-checks before using this method.

Do you have a newer version of OUI to work with?

In this case I did as there was a newer 10g installation in the same file system. If not, I could have used a version of OUI from the 10g distribution media, or downloaded a copy of the distribution media from Oracle Technet (http://www.oracle.com/technetwork/index.html). If I had to download a copy, I would probably download the latest version available for my platform.

What is the name and location of the Oracle home we want to remove?

The quickest way to determine this information is with the use of the opatch utility. This utility is located under $ORACLE_HOME/OPatch. You should use the highest version of the utility present on the system. In my case, it was the one present under the 10g Oracle home. If this utility is not present, the latest version can be downloaded from My Oracle Support (MOS), path number 6880880. It is a compressed file that is designed to be decompressed directly into the ORACLE_HOME where you are going to install it. It will replace the existing opatch utility if it exists.
Execute ‘opatch lsinventory –all’ to get a listing similar to this one:

opatch lsinventory -all
Invoking OPatch 10.2.0.2.2
 
Oracle interim Patch Installer version 10.2.0.2.2
Copyright (c) 2005, Oracle Corporation.  All rights reserved..
 
Oracle Home       : /u01/app/oracle/product/10.2.0
Central Inventory : /u01/app/oracle/oraInventory
   from           : /var/opt/oracle/oraInst.loc
OPatch version    : 10.2.0.2.2
OUI version       : 10.2.0.1.0
OUI location      : /u01/app/oracle/product/10.2.0/oui
Log file location : /u01/app/oracle/product/10.2.0/cfgtoollogs/opatch/opatch2011-03-22_19-50-00PM.log
 
Lsinventory Output file location : /u01/app/oracle/product/10.2.0/cfgtoollogs/opatch/lsinv/lsinventory2011-03-22_19-50-00PM.txt
 
--------------------------------------------------------------------------------
List of Oracle Homes:
  Name          Location
   OraDb10g_home1         /u01/app/oracle/product/10.2.0.4ee/db_1
   agent10g         	/u01/app/oracle/product/agent10g
   Oracle10gR1         /u01/app/oracle/product/10.1.0se
   EKUASDV         /u01/app/oracle/product/10.2.0
   EKUASQA         /u01/app/oracle/product/10.2.0
 
Installed Top-level Products (2):
 
Oracle Database 10g                                                  10.2.0.1.0
Oracle Database 10g Products                                         10.2.0.1.0
There are 2 products installed in this Oracle Home.
 
Interim patches (1) :
 
Patch  4375798      : applied on Wed Sep 12 18:54:13 GMT 2007
   Created on 31 Oct 2005, 23:17:10 hrs US/Pacific
   Bugs fixed:
     4375798
--------------------------------------------------------------------------------
 
OPatch succeeded.

As you can see, all the Oracle homes present on the system are listed, including the names and locations. Please note that this listing was not from the actual system I was working on and the names have been changed to protect the parties involved.
Is this ORACLE_HOME still being used?

After making sure that the entry for the old Oracle home is either gone or commented out of the oratab file, there are several commands you can use to find out if any active processes are using files out of the old ORACLE_HOME. The simplest is ‘ps –ef | grep ’ where is the location of the old ORACLE_HOME. To be absolutely sure, you can use the ‘lsof | grep ’, but it would probably require an OS account with root privileges. If the lsof command does not return any values, then it is probably safe to remove the old Oracle home.

Removing the ORACLE_HOME

After going through the previous considerations and finding out the name and location of the ORACLE_HOME I want to remove, I can use the OUI to remove it. The OUI can be executed from the $ORACLE_HOME/oui/bin directory or from the distribution media. The syntax I need is:

/runInstaller -deinstall -silent REMOVE_HOMES={“/app/oracle/OraHome_1″}

The REMOVE_HOMES argument requires the exact path of the ORACLE_HOME to be removed. If the actual directory where the software is located is still present after running the OUI command above, you can safely remove it with OS commands.

Summary
In all honesty, I had previously ignored the silent aspect of the OUI to perform installs (and de-installs). I guess I was addicted to the attractive Java-type OUI interface with its active progress bar and animations. The silent method of OUI does come in handy for performing multiple installations on several of the same platforms, such as when you deploy OEM agent software. Bottom line is that it might be beneficial to take a quick look at other methods of execution of your most common tools.

Addendum
The runInstaller -deinstall option is not available in 11.2.0.2, but was tested fine in 11.2.0.1. The test consisted of using the 11.2.0.1 runInstaller to remove a previous installation of the 11.2.0.2 software where the 11.2.0.2 Oracle home had been registered in the central inventory, but the 11.2.0.2 binaries had been removed. It was necessary to remove the home before proceeding with a new installation of the 11.2.0.2 software.