Fix for Page Load Issues in Safari 5.1.7

Easy Fix for High CPU Usage in SQL Developer on OS X

Run Oracle Forms 11g Applications on OS X Lion

Here is what the memory allocation on the machine looked like; 10G in PageTables.

$ cat /proc/meminfo
MemTotal: 264251332 kB
MemFree: 356300 kB
Buffers: 954760 kB
Cached: 245170884 kB
SwapCached: 1758320 kB
Active: 175702124 kB
Inactive: 75100644 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 264251332 kB
LowFree: 356300 kB
SwapTotal: 4194296 kB
SwapFree: 504648 kB
Dirty: 52 kB
Writeback: 56 kB
AnonPages: 2918272 kB
Mapped: 43409016 kB
Slab: 2252952 kB
PageTables: 10498900 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 136319960 kB
Committed_AS: 67219780 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 279684 kB
VmallocChunk: 34359458295 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB

Their server had 256G of RAM and a SGA that was already 100G; a good candidate for HugePages due to their sizes.

From Oracle documentation, here are some major advantages of using HugePages.

* HugePages is crucial for faster Oracle database performance on Linux if you have a large RAM and SGA.

* HugePages are not swappable. Therefore there is no page-in/page-out mechanism overhead. HugePages are universally regarded as pinned.

Implementing HugePages has become common practice with Oracle 11g and is fairly well documented in MOS Note 361468.1.
The basics steps are as follows:

* Set the memlock ulimit for the oracle user.
* Disable Automatic Memory Managment if necesary as it is incompatible with HugePages.
* Run the Oracle supplied hugepages_settings.sh script to calculate the recommended value for the vm.nr_hugepages kernel parameter.
* Edit /etc/sysctl.conf with the vm.nr_hugepages with the recommeneded setting.
* Reboot the server

Unfortunately, The database we were working with was 10g. As it turns out, there are some differences between
Oracle 10 and 11, mainly that there is no HugePage logging in the alert log on version 10. See MOS Note: 1392543.1
This makes troubleshooting difficult because HugePages does not take effect until after it is configured.

Another noticable difference is the fact that the initialization parameter use_large_pages was not added until version 11.2.0.2.
In the event that you have multiple databases on one machine, this parameter allows you to control which databases
use HugePages. In addition, it can prevent a database from starting up if not enough HugePages can be allocated at instance startup.
See “USE_LARGE_PAGES To Enable HugePages In 11.2 [ID 1392497.1]” for more detailed info on the values for this parameter.

Similar across both versions is the initialization parameter PRE_PAGE_SGA, which determines whether Oracle
will read the entire SGA into memory at instance startup or not. However, it is not
specific to HugePages as it works with the default OS pages as well. Also, there is an environment variable DISABLE_HUGETLBFS
that can be used in 10g to accomplish the same thing as the use_large_pages parameter in 11g. If this variable is set, it
will prevent Oracle from using HugePages. So presumably you could have startup scripts that set or unset this variable prior
to starting databases which would give you ability to have some use huge pages while others do not, even in 10g.

Getting back to the story, we successfully got HugePages working on their QA box, but then it did not
work in production. HupePages were created by the OS after the reboot as seen in /proc/meminfo;
however, Oracle didn’t use them. In comparing the two environments, two notable differences were found:

1. memlock settings in /etc/security/limits.conf ( Can be seen by running ulimit ) was unlimited in QA, but not production

> ulimit -l
98566144

> cat /etc/security/limits.conf
# /etc/security/limits.conf
#
…
oracle soft memlock 98566144
oracle hard memlock 98566144
# End of file

2. SHMMAX was 90G on QA, but 190G on production

Oracle’s recommendation for the memlock setting ( from the MOS note 1392543.1 ) is stated as follows:
“Set the value (in KB) slightly smaller than installed RAM.”
The customer had decided to set it to 94% of their 100G SGA.

After some searching, we found the MOS note “Oracle Not Utilizing Hugepages [ID 803238.1]” which
gives a formula for calculating the value. This value is the minimum required in order for the database
to successfully startup using HugePages. The memlock setting is specified in KB and must match the
memory size of the number of HugePages that Oracle should be able to allocate.

memlock = ( HugePages * Hugepagesize )
where HugePages is the output given using script from MOS note 401749.1 and Hugepagesize is value from
in /proc/meminfo (on Linux Hugepagesize is normally 2048).

memlock = ( 51203 * 2048 ) = 104863744
This number is higher than what was set for memlock in /etc/security/limits.conf .

Now, what about SHMMAX? According to MOS note 567506.1, Oracle Global Customer Support officially recommends
a maximum for SHMMAX of “1/2 of physical RAM”. Since the customer’s value was higher than this,
SHMMAX was lowered from 190G to 90G to match that of QA.

After making these two changes, the database successfuly started up using HugePages.
It is worth mentioning here that if HugePages had not been successfully configured, the vm.nr_hugepages
parameter change in /etc/sysctl.conf should be backed out as the space allocated to HugePages would be
wasted if Oracle cannot use it.

So, was it memlock or SHMMAX that was causing the issue? Or was it both?

Since 11g writes a lot of information to the alert log regarding hugepage usage I decided to see what kind
of messages would be emitted when the same situation occurred on an 11g database.
As it turns out, the messages in the alert log are very informational in pointing you in the right direction.

Here is a snippet from the 11g alert log on a clean startup utilizing HugePages:

Starting ORACLE instance (normal)
****************** Large Pages Information *****************
 
Total Shared Global Region in Large Pages = 806 MB (100%)
 
Large Pages used by this instance: 403 (806 MB)
Large Pages unused system wide = 97 (194 MB) (alloc incr 4096 KB)
Large Pages configured system wide = 500 (1000 MB)
Large Page size = 2048 KB
***********************************************************

In the following example, I lowered the memlock setting to below its recommended value.
Note the Memlock messages saying it is too small.
The “recommendation” part seems a bit odd as it suggests to add 0 pages!
We’ll look at that one next.

Starting ORACLE instance (normal)
Memlock limit too small: 32768 to accommodate segment size: 4194304
Memlock limit too small: 32768 to accommodate segment size: 8388608
Memlock limit too small: 32768 to accommodate segment size: 4194304
Memlock limit too small: 32768 to accommodate segment size: 4194304
...
Memlock limit too small: 32768 to accommodate segment size: 4194304
Memlock limit too small: 32768 to accommodate segment size: 4194304
Memlock limit too small: 32768 to accommodate segment size: 2097152
****************** Large Pages Information *****************
 
Total Shared Global Region in Large Pages = 0 KB (0%)
 
Large Pages used by this instance: 0 (0 KB)
Large Pages unused system wide = 500 (1000 MB) (alloc incr 4096 KB)
Large Pages configured system wide = 500 (1000 MB)
Large Page size = 2048 KB
 
RECOMMENDATION:
Total Shared Global Region size is 810 MB. For optimal performance,
prior to the next instance restart increase the number
of unused Large Pages by atleast 0 2048 KB Large Pages (0 KB)
system wide to get 100% of the Shared
Global Region allocated with Large pages
***********************************************************

In the following example, the memlock value and hugepages are both too small.
As you can see from the alert log below, messages about both are mentioned, good!

Starting ORACLE instance (normal)
Memlock limit too small: 32768 to accommodate segment size: 4194304
Memlock limit too small: 32768 to accommodate segment size: 8388608
Memlock limit too small: 32768 to accommodate segment size: 4194304
...
Memlock limit too small: 32768 to accommodate segment size: 4194304
Memlock limit too small: 32768 to accommodate segment size: 4194304
Memlock limit too small: 32768 to accommodate segment size: 2097152
****************** Large Pages Information *****************
 
Total Shared Global Region in Large Pages = 0 KB (0%)
 
Large Pages used by this instance: 0 (0 KB)
Large Pages unused system wide = 100 (200 MB) (alloc incr 4096 KB)
Large Pages configured system wide = 100 (200 MB)
Large Page size = 2048 KB
 
RECOMMENDATION:
Total Shared Global Region size is 810 MB. For optimal performance,
prior to the next instance restart increase the number
of unused Large Pages by atleast 305 2048 KB Large Pages (610 MB)
system wide to get 100% of the Shared
Global Region allocated with Large pages
***********************************************************

Let’s take a look at the kernel parameter SHMMAX now.
On the Oracle VM where I was testing this, the default SHMMAX was set to 64GB. Much bigger
than the 2GB of RAM allocated to the VM. In trying values between 1GB and 64Gb, the
database always started up using HugePages without any issue. However, the value of SHMALL
is one that needs to be set correctly.

In the next example, I lowered SHMALL to below its recommended value.
You will see the output below from SQL*Plus, but nothing was shown in the alert log.
After reading a couple of Documents on MOS, you should
choose the higher of the two values below for SHMALL, it will depend on your database memory sizes.
From MOS note 169706.1 : SHMALL = SHMMAX / OS PAGE_SIZE
From MOS note 301830.1 : SHMALL = sum(SGAs)/ OS PAGE_SIZE

SQL > startup
ORA-27102: out of memory
Linux-x86_64 Error: 28: No space left on device

As I mentioned earlier, HugePages is not compatible with Automatic Memory Mangement.
If you try to start the database using AMM, here is what the entry from the alert log will look like.
Note that you can still use the sga_target parameter, just not memory_target, memory_max_target.
Note also that you can’t use the use_indirect data._buffers parameter.

Starting ORACLE instance (normal)
******************************************************************
Large Pages are not compatible with specified SGA memory parameters
use_large_pages = "ONLY" cannot be used with memory_target,
memory_max_target, or use_indirect_data_buffers parameters
Large Pages are compatible with sga_target and shared_pool_size
******************************************************************

Since the alert log in Oracle 10g does not log any HugePage information, we have to be a little more creative
to uncover clues as to why huge pages are not used . There is a note on MOS that recommends using strace
to help diagnose this type of issue. So let’s use strace on a
sqlplus session when starting the Oracle 10 database to see if that will provide
any useful troubleshooting information. I intentionally lowered the SHMALL parameter in this example to 56000.
Notice here that we are trying to allocate more memory that we have allotted for with the SHMALL kernel parameter.

> strace -f -o output.txt sqlplus / as sysdba
 
SQL*Plus: Release 10.2.0.5.0 - Production
 
Copyright (c) 1982, 2010, Oracle. All Rights Reserved.
 
Connected to an idle instance.
 
SYS:ORA10>startup
ORA-27102: out of memory
Linux-x86_64 Error: 28: No space left on device

In the output.txt file, you should see output similar to the following showing the shmget calls.

9624  shmget(0x22675a08, 232783872, IPC_CREAT|IPC_EXCL|SHM_HUGETLB|0660) = -1 ENOSPC (No space left on device)

In the next example, I lowered the ulimit setting. Here the database successfully started; however, HugePages was
not used as seen in /proc/meminfo.

>strace -f -o output.txt sqlplus / as sysdba
 
SQL*Plus: Release 10.2.0.5.0 - Production
 
Copyright (c) 1982, 2010, Oracle. All Rights Reserved.
 
Connected to an idle instance.
 
SYS:ORA10>;startup
ORACLE instance started.
 
Total System Global Area 230686720 bytes
Fixed Size 2095344 bytes
Variable Size 117442320 bytes
Database Buffers 104857600 bytes
Redo Buffers 6291456 bytes
Database mounted.
Database opened.
 
SYS:ORA10>;exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
 
> cat /proc/meminfo | grep -i huge
HugePages_Total: 520
HugePages_Free: 520
HugePages_Rsvd: 0
Hugepagesize: 2048 kB

From output.txt, you can see that the database tried to use huge pages, but failed and then started up using
the OS standard pages.

10783 shmget(0x22675a08, 232783872, IPC_CREAT|IPC_EXCL|SHM_HUGETLB|0660) = -1 ENOMEM (Cannot allocate memory)
...
10783 shmget(0x22675a08, 232783872, IPC_CREAT|IPC_EXCL|0660) = 12353549

If you do not want an 11g database to startup without HugePages, remember to set the use_large_pages initilization
parameter mentioned previously. When set to “only”, the database will fail to start. You will need to be at least
on version 11.2.0.2 though.

For Oracle 10g, if you want to ensure that your database is using HugePages,
you can use the equation from MOS note 1392543.1 to calculate the
“Large Pages used by this instance” as seen in 11g alert log.
If you have multiple databases, you will need to monitor /proc/meminfo between instance startups.
Note that if HugePages_Total = HugePages_Free, then you at NOT using HugePages.

Large Pages used by this instance (#pages) = HugePages_Total – HugePages_Free + HugePages_Rsvd

 > cat /proc/meminfo | grep -i huge
HugePages_Total: 520
HugePages_Free: 409
HugePages_Rsvd: 0
Hugepagesize: 2048 kB

#pages = 520 – 409 + 0 = 111

Here’s an example using Oracle 11g showing the correlation between the equation and the alert log.

DBM>startup
ORACLE instance started.
 
Total System Global Area 839282688 bytes
Fixed Size 2233000 bytes
Variable Size 780143960 bytes
Database Buffers 50331648 bytes
Redo Buffers 6574080 bytes
Database mounted.
Database opened.
 
[el5-5-64:oracle:ORA10] /home/oracle
> cat /proc/meminfo
HugePages_Total: 520
HugePages_Free: 403
HugePages_Rsvd: 286
Hugepagesize: 2048 kB

#pages = 520 – 403 + 286 = 403, which matches the value that is shown in the alert log snippet below.

Starting ORACLE instance (normal)
****************** Large Pages Information *****************
Parameter use_large_pages = ONLY
 
Total Shared Global Region in Large Pages = 806 MB (100%)
 
Large Pages used by this instance: 403 (806 MB)
Large Pages unused system wide = 117 (234 MB) (alloc incr 4096 KB)
Large Pages configured system wide = 520 (1040 MB)
Large Page size = 2048 KB
***********************************************************

Please see MOS notes Oracle Not Utilizing Hugepages [ID 803238.1] and
Upon startup of Linux database get ORA-27102: out of memory Linux-X86_64 Error: 28: No space left on device [ID 301830.1]
for more troubleshooting examples.

Finally, it is important to remember is that the HugePage confguration is not dynamic. If you make changes to the SGA size, add another database, etc.,
you should revisit the HugePage configuration.

Read on…

I ran into an interesting problem recently when attempting to publish an MS Access application with Citrix XenApp. The application pulled data from an Oracle database, so an ODBC connection was created on the Citrix server. What we found was that when a non-privileged user tried to run the application, the connection failed. When the Citrix server admin ran it, however, it worked. Furthermore, as long as the admin was logged in, anybody else could run the application successfully; when he logged out, the users’ connections broke again.

Since Access is not my area of expertise, we finally contacted Microsoft tech support after trying a few suggestions we dug up in our own research. The support analyst clued us in to the following solution, which I pulled from an Oracle Primavera setup document.

From what I’ve seen on different support forums, this seems to be a pretty standard setup task when running Oracle on Windows over Citrix.

If using an Oracle database platform, confirm [that] the Create Global Objects Windows Security Policy is set up on the Terminal Servers and add the domain Users or Groups of the end-users who will be accessing the published application. If the local security policy is not setup, the following steps can be used to add the policy:

1. Click Start, Programs, Administrative Tools, and then click Local Security Policy.
2. Expand Local Policies, and then click User Rights Assignments.
3. In the right pane, double-click Create Global Objects.
4. In the Local Security Policy Setting dialogbox, click Add.
5. In the Select Users or Group dialog box, select the user group that these end users are a part of, click Add, and then click OK.
6. Click OK.

If using an Oracle database platform, the Oracle client software also requires that you give the Authenticated User Group, Read and Execute privilege to the Oracle Home Directory and ALL its sub directories and files on the Terminal Servers.

Read on…

VMSTAT is a tool on Unix type plaforms that provides an impressive amount of resource information, but is formatted in such a way that it can be difficult to read much less determine when the snapshot occurred if looking at historic snapshots. Here is an example

[root@new-host ~]# vmstat 2
procs ———–memory———- —swap– —–io—- –system– —–cpu——
r b swpd free buff cache si so bi bo in cs us sy id wa st
0 0 0 1916352 39376 906944 0 0 186 39 1024 1139 1 2 93 4 0
1 0 0 1916352 39384 906968 0 0 0 76 1015 1203 0 2 95 4 0
0 0 0 1916260 39392 906976 0 0 0 58 1014 1118 0 2 98 0 0

Through some simple Unix scripting commands, you can get more user-friendly information from this tool.

The vmstat_tool.sh script below works for most Linux versions:

#!/bin/sh
#vmstat_tool.sh
#vmstat parser script. Makes the information easily readable.

#Ensure that the number of loops and delay parameters
#were passed to this script
if [ $# -ne 2 ]
then
echo “Usage: $0 ”
exit
fi

export LPCNT=$1
export DLAY=$2

c=0

while [ "$c" -lt $LPCNT ]
do
date
vmstat | tail -n 1 | awk ‘{printf(“%d processes are waiting\n%d processes are blocked\n%d virtual memory used, %d idle memory, %d buffer memory, %d cache memory\n%d memory swapped in from disk, %d memory swapped out to disk\n%d blocks per second written\n%d blocks per second read\n%d interrupts per second\n%d context switches per second\nCPU usage: %d%% (user, non-kernel), %d%% (system, kernel), %d%% idle, %d%% wait\n\n”,$1,$2,$3,$4,$5,$6,$7,$8,$9,$10,$11,$12,$13,$14,$15,$16)}’

sleep $DLAY
c=$((c+1))
done

Sample Output (Single Loop):

Sat Feb 18 11:15:48 PST 2012
1 processes are waiting
0 processes are blocked
0 virtual memory used, 1990308 idle memory, 32964 buffer memory, 843564 cache memory
0 memory swapped in from disk, 0 memory swapped out to disk
928 blocks per second written
75 blocks per second read
1062 interrupts per second
1128 context switches per second
CPU usage: 4% (user, non-kernel), 2% (system, kernel), 77% idle, 17% wait

As you can see. the output is much more readable than the regular vmstat output.

Should you need to do more than just a cursory examination of the VMSTAT output, you can use these scripts to generate an output file for examination with the following method:

nohup vmstat_tool.sh 288 300 > ./vmstat.log &

In the above example, you are using the native nohup command to run the vmstat_tool.sh script to loop every 5 minutes (300 seconds) for 24 hours (288 5-minute loops). The output will to the vmstat.log script in the local directory and will run in the background by using ‘&’ at the end of the command. Examining the output is a simple matter of using the native cat and grep commands in the following manner:

cat ./vmstat.log | grep “processes are waiting” | more

Sample Output:

0 processes are waiting 0 processes are blocked
0 processes are waiting 0 processes are blocked
0 processes are waiting 0 processes are blocked
0 processes are waiting 0 processes are blocked
0 processes are waiting 0 processes are blocked
0 processes are waiting 0 processes are blocked
0 processes are waiting 0 processes are blocked
0 processes are waiting 0 processes are blocked

You can now see the output of the first line of the VMSTAT tool scrolled one screen at a time. If you see a line that has some abnormal values, you can then use the vi utility to zero in on that particular value and get the timestamp from it.

How it Works

It should work with 10gR2-11gR2 without changes. The first time is run, it will attempt to identify the Grid Infrastructure / CRS home and the version that is being used (CRS/HAS must be up for it to gather)- it takes that information and writes the info to a crsstat.env located in the same location as the script. It only gathers the information the first time, so if you upgrade the Oracle software you will want to remove the crsstat.env and allow the script to re-gather the information.

The output of the script assumes a term width of 132 and attempts to highlight possible issues, by showing in RED those resources with a target of ONLINE and a state of OFFLINE. You can also pass in “search criteria” as the first and only parameter.

Also of note, per Oracle support document 1086563.1, the “State Details” output in 11gR2 could show incorrect information, such as, “Instance Shutdown” even though it is up and running correctly.

Under the covers, the script is utilizing the “crs_stat -t” in 10gR2/11gR1 and “crsctl status resource -t” and then awks their output to create the tabular format. It has been used on AIX and Linux (SuSE, Red Hat, Oracle).

How to Install

Pre-requisite to Installation:

1. CRS/HAS must be running
2. The location of the crsstat script must be writable by the user that will run the installation (either root or your oracle user)

Installation Location:

Generally, it is practical to put the crsstat script in the /usr/local/bin directory, so everyone will have access (in most users path). Make sure the crsstat.env file can be created in the /usr/local/bin directory the first time the script is run. If it is preferred to put it an alternate location, choose a location that is accessible by your Unix users that should be able to list the CRS or HAS resources (i.e. in their PATH).

Installation Process:

1) Copy the installation script from this link http://blog.enkitec.com/wp-content/uploads/2012/07/crsstat.v1.2.2.zip
2) Do a “cp crsstat.v1.2.1.zip /usr/local/bin/”
3) As root run the crsstat and a “crsstat.env” file is created:

# cd /usr/local/bin
# unzip crsstat.v1.2.1.zip
# chmod 755 crsstat
# ./crsstat

Output like example below…

# chmod 644 crsstat.env

4) Verify that the crsstat.env file has two entries like the following:
GRID_HOME=/u01/app/oracle/product/11.2.0/grid
GRID_VER=11.2

Example Usage

Basic Usage (note that the dbfs filesystem is offline when it should be online, thus it is highlight in red):

$ crsstat

Resource Name                            Resource Type  Target       State        Node            State Details
---------------------------------------- -------------- ------------ ------------ --------------- --------------
ora.DATA.dg                              diskgroup      ONLINE       ONLINE       tstdb01
ora.DATA.dg                              diskgroup      ONLINE       ONLINE       tstdb02
ora.LISTENER.lsnr                        Listener       ONLINE       ONLINE       tstdb01
ora.LISTENER.lsnr                        Listener       ONLINE       ONLINE       tstdb02
ora.LISTENER_SCAN1.lsnr                  SCAN Listener  ONLINE       ONLINE       tstdb02
ora.LISTENER_SCAN2.lsnr                  SCAN Listener  ONLINE       ONLINE       tstdb01
ora.LISTENER_SCAN3.lsnr                  SCAN Listener  ONLINE       ONLINE       tstdb01
ora.RECO.dg                              diskgroup      ONLINE       ONLINE       tstdb01
ora.RECO.dg                              diskgroup      ONLINE       ONLINE       tstdb02
ora.SYSTEM.dg                            diskgroup      ONLINE       ONLINE       tstdb01
ora.SYSTEM.dg                            diskgroup      ONLINE       ONLINE       tstdb02
ora.asm                                  ASM            ONLINE       ONLINE       tstdb01
ora.asm                                  ASM            ONLINE       ONLINE       tstdb02
ora.cvu                                  cvu            ONLINE       ONLINE       tstdb01
ora.dbfs.db                              database       ONLINE       ONLINE       tstdb01         Open
ora.dbfs.db                              database       ONLINE       ONLINE       tstdb02         Open
ora.dbfs.filesystem                                     ONLINE       OFFLINE
ora.dbfs.filesystem                                     ONLINE       OFFLINE
ora.demo.db                              database       ONLINE       ONLINE       tstdb01         Open
ora.demo.db                              database       ONLINE       ONLINE       tstdb02         Open
ora.demo.workshop.svc                    service        ONLINE       ONLINE       tstdb01
ora.demo.workshop.svc                    service        ONLINE       ONLINE       tstdb02
ora.tstdb01.vip                          Cluster VIP    ONLINE       ONLINE       tstdb01
ora.tstdb02.vip                          Cluster VIP    ONLINE       ONLINE       tstdb02
ora.net1.network                         Network (VIP)  ONLINE       ONLINE       tstdb01
ora.net1.network                         Network (VIP)  ONLINE       ONLINE       tstdb02
ora.ons                                  Ora Notif Svc  ONLINE       ONLINE       tstdb01
ora.ons                                  Ora Notif Svc  ONLINE       ONLINE       tstdb02
ora.registry.acfs                        registry       ONLINE       ONLINE       tstdb01
ora.registry.acfs                        registry       ONLINE       ONLINE       tstdb02
ora.scan1.vip                            SCAN VIP       ONLINE       ONLINE       tstdb02
ora.scan2.vip                            SCAN VIP       ONLINE       ONLINE       tstdb01
ora.scan3.vip                            SCAN VIP       ONLINE       ONLINE       tstdb01
ora.test.db                              database       OFFLINE      OFFLINE
ora.test.db                              database       OFFLINE      OFFLINE

A couple of examples using the search key:

$ crsstat service

Resource Name                            Resource Type  Target       State        Node            State Details
---------------------------------------- -------------- ------------ ------------ --------------- --------------
ora.demo.workshop.svc                    service        ONLINE       ONLINE       tstdb01
ora.demo.workshop.svc                    service        ONLINE       ONLINE       tstdb02

$ crsstat demo

Resource Name                            Resource Type  Target       State        Node            State Details
---------------------------------------- -------------- ------------ ------------ --------------- --------------
ora.demo.db                              database       ONLINE       ONLINE       enkdb01         Open
ora.demo.db                              database       ONLINE       ONLINE       enkdb02         Open
ora.demo.workshop.svc                    service        ONLINE       ONLINE       enkdb01
ora.demo.workshop.svc                    service        ONLINE       ONLINE       enkdb02

Another example, as sometimes when there are many databases it is useful to pipe the output to grep:

$ crsstat database | grep -i offline

ora.test.db                              database       OFFLINE      OFFLINE
ora.test.db                              database       OFFLINE      OFFLINE

Let’s revisit a use case I alluded to in my earlier blog entry: incorporating LDAP data into a query. Suppose your task is to code a custom authorization scheme for an enterprise application reporting system. You want row access to be determined by a user’s group memberships in the company’s Active Directory server. If a user is assigned to “the Company A group” in Active Directory, he gets to see Company A’s data. You can configure that kind of restriction at a database level using Oracle’s Virtual Private Database feature. Configuring VPD is beyond the scope of this blog entry, but ultimately you will be associating a table to a portion of a SQL WHERE clause that gets appended to every query written against that table. You can find more documentation on VPD on the OTN site. In this case, let’s assume you know that another team has written a PL/SQL database package to query data from Active Directory, and you want to make use of that code. The only problem is that all the functions in that package return collections of various types, and you need to be able to access that information from the WHERE clause you are building with VPD. This is where table functions come in handy.

Let’s begin by setting up our example schema. We’ll be looking at client sales data. First, let’s set up our clients.

CREATE TABLE clients (
  client_id NUMBER,
  client_name VARCHAR2(50));
 
INSERT INTO clients VALUES (100, 'Timco Industries');
INSERT INTO clients VALUES (200, 'Moneybros');
INSERT INTO clients VALUES (300, 'The Monacle Corporation');
INSERT INTO clients VALUES (400, 'Schmapple');

We also need application users who will be logging in to view the report data. We’ll assume each application user is uniquely linked to a user in Active Directory via the external_id column.

CREATE TABLE users (
  sequence_number NUMBER,
  user_name VARCHAR2(20),
  external_id VARCHAR2(20),
  first_name VARCHAR2(30),
  last_name VARCHAR2(30));
 
INSERT INTO users VALUES (1, 'JACKA', 'jaugusti', 'Jack', 'Augustine');
INSERT INTO users VALUES (2, 'KENKI', 'kkidd', 'Ken', 'Kidd');
INSERT INTO users VALUES (3, 'BRIAH', 'bhill', 'Brian', 'Hill');
INSERT INTO users VALUES (4, 'KEOVO', 'kvongkas', 'Keo', 'Vongkaseum');

Finally, we need a data table on which to report.

CREATE TABLE quarterly_sales_data (
  sequence_number NUMBER,
  client_id NUMBER,
  sales_year NUMBER,
  sales_quarter NUMBER,
  sales_amount NUMBER);
 
INSERT INTO quarterly_sales_data VALUES (1, 100, 2011, 1, 20000);
INSERT INTO quarterly_sales_data VALUES (2, 100, 2011, 2, 17500);
INSERT INTO quarterly_sales_data VALUES (3, 100, 2011, 3, 33000);
INSERT INTO quarterly_sales_data VALUES (4, 200, 2011, 1, 4500);
INSERT INTO quarterly_sales_data VALUES (5, 200, 2011, 2, 7000);
INSERT INTO quarterly_sales_data VALUES (6, 200, 2011, 3, 6300);
INSERT INTO quarterly_sales_data VALUES (7, 300, 2011, 1, 11100);
INSERT INTO quarterly_sales_data VALUES (8, 300, 2011, 2, 12800);
INSERT INTO quarterly_sales_data VALUES (9, 300, 2011, 3, 14000);
INSERT INTO quarterly_sales_data VALUES (10, 400, 2011, 1, 55600);
INSERT INTO quarterly_sales_data VALUES (11, 400, 2011, 2, 80000);
INSERT INTO quarterly_sales_data VALUES (12, 400, 2011, 3, 78500);

Our basic report query will look something like this:

SELECT client_name, sales_year, sales_quarter, sales_amount
  FROM quarterly_sales_data NATURAL JOIN clients;

Our Active Directory groups will be TIMCO_GRP, MONEYBROS_GRP, MONACLE_GRP, and SCHMAPPLE_GRP. If a user belongs to one or more of these groups, he has permission to view the respective client’s data.

Now we need a function that returns data from Active Directory. Just to keep it simple we will create a package that hardcodes the group memberships and returns them as collections of various types. We will then look at how table functions can be used to transform each one of those collections into an aggregate result set. (NOTE: querying LDAP directories from PL/SQL is not too difficult, and might serve as the basis for a future blog entry.)

Here’s our function. We’ve placed it in a package called LDAP_UTIL.

CREATE OR REPLACE PACKAGE ldap_util AS
 
  TYPE ldap_entry_typ IS RECORD (
    cn VARCHAR2(30),
    description VARCHAR2(1000),
    distinguished_name VARCHAR2(200),
    name VARCHAR2(30),
    samaccountname VARCHAR2(30));
 
  TYPE ldap_groups_nt IS TABLE OF ldap_entry_typ;
  TYPE ldap_groups_aa IS TABLE OF ldap_entry_typ INDEX BY PLS_INTEGER;
  TYPE ldap_groups_va IS VARRAY(100) of ldap_entry_typ;
 
  FUNCTION get_user_groups_nt (p_user VARCHAR2) RETURN ldap_groups_nt;
  FUNCTION get_user_groups_aa (p_user VARCHAR2) RETURN ldap_groups_aa;
  FUNCTION get_user_groups_va (p_user VARCHAR2) RETURN ldap_groups_va;
 
END ldap_util;
/
SHOW ERRORS
 
CREATE OR REPLACE PACKAGE BODY ldap_util AS
 
  FUNCTION get_user_groups_nt (p_user VARCHAR2) RETURN ldap_groups_nt IS
    l_groups ldap_groups_nt := ldap_groups_nt();
  BEGIN
    CASE p_user
      WHEN 'jaugusti' THEN
 
        l_groups.extend;
        l_groups(1).cn := 'TIMCO_GRP';
        l_groups(1).description := 'Employees working on the Timco Industries account.';
        l_groups(1).distinguished_name := 'CN=TIMCO_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(1).name := 'TIMCO_GRP';
        l_groups(1).samaccountname := 'TIMCO_GRP';
 
      WHEN 'kkidd' THEN
 
        l_groups.extend;
        l_groups(1).cn := 'MONEYBROS_GRP';
        l_groups(1).description := 'Employees working on the Moneybros account.';
        l_groups(1).distinguished_name := 'CN=MONEYBROS_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(1).name := 'MONEYBROS_GRP';
        l_groups(1).samaccountname := 'MONEYBROS_GRP';
 
      WHEN 'bhill' THEN
 
        l_groups.extend;
        l_groups(1).cn := 'TIMCO_GRP';
        l_groups(1).description := 'Employees working on the Timco Industries account.';
        l_groups(1).distinguished_name := 'CN=TIMCO_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(1).name := 'TIMCO_GRP';
        l_groups(1).samaccountname := 'TIMCO_GRP';
 
        l_groups.extend;
        l_groups(2).cn := 'MONACLE_GRP';
        l_groups(2).description := 'Employees working on the Monacle Corporation account.';
        l_groups(2).distinguished_name := 'CN=MONACLE_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(2).name := 'MONACLE_GRP';
        l_groups(2).samaccountname := 'MONACLE_GRP';
 
      WHEN 'kvongkas' THEN
 
        l_groups.extend;
        l_groups(1).cn := 'MONEYBROS_GRP';
        l_groups(1).description := 'Employees working on the Moneybros account.';
        l_groups(1).distinguished_name := 'CN=MONEYBROS_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(1).name := 'MONEYBROS_GRP';
        l_groups(1).samaccountname := 'MONEYBROS_GRP';
 
        l_groups.extend;
        l_groups(2).cn := 'MONACLE_GRP';
        l_groups(2).description := 'Employees working on the Monacle Corporation account.';
        l_groups(2).distinguished_name := 'CN=MONACLE_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(2).name := 'MONACLE_GRP';
        l_groups(2).samaccountname := 'MONACLE_GRP';
 
        l_groups.extend;
        l_groups(3).cn := 'SCHMAPPLE_GRP';
        l_groups(3).description := 'Employees working on the Schmapple account.';
        l_groups(3).distinguished_name := 'CN=SCHMAPPLE_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(3).name := 'SCHMAPPLE_GRP';
        l_groups(3).samaccountname := 'SCHMAPPLE_GRP';
 
    END CASE;
    RETURN l_groups;
  END get_user_groups_nt;
 
  FUNCTION get_user_groups_aa (p_user VARCHAR2) RETURN ldap_groups_aa IS
    l_groups ldap_groups_aa;
  BEGIN
    CASE p_user
      WHEN 'jaugusti' THEN
 
        l_groups(1).cn := 'TIMCO_GRP';
        l_groups(1).description := 'Employees working on the Timco Industries account.';
        l_groups(1).distinguished_name := 'CN=TIMCO_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(1).name := 'TIMCO_GRP';
        l_groups(1).samaccountname := 'TIMCO_GRP';
 
      WHEN 'kkidd' THEN
 
        l_groups(1).cn := 'MONEYBROS_GRP';
        l_groups(1).description := 'Employees working on the Moneybros account.';
        l_groups(1).distinguished_name := 'CN=MONEYBROS_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(1).name := 'MONEYBROS_GRP';
        l_groups(1).samaccountname := 'MONEYBROS_GRP';
 
      WHEN 'bhill' THEN
 
        l_groups(1).cn := 'TIMCO_GRP';
        l_groups(1).description := 'Employees working on the Timco Industries account.';
        l_groups(1).distinguished_name := 'CN=TIMCO_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(1).name := 'TIMCO_GRP';
        l_groups(1).samaccountname := 'TIMCO_GRP';
 
        l_groups(2).cn := 'MONACLE_GRP';
        l_groups(2).description := 'Employees working on the Monacle Corporation account.';
        l_groups(2).distinguished_name := 'CN=MONACLE_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(2).name := 'MONACLE_GRP';
        l_groups(2).samaccountname := 'MONACLE_GRP';
 
      WHEN 'kvongkas' THEN
 
        l_groups(1).cn := 'MONEYBROS_GRP';
        l_groups(1).description := 'Employees working on the Moneybros account.';
        l_groups(1).distinguished_name := 'CN=MONEYBROS_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(1).name := 'MONEYBROS_GRP';
        l_groups(1).samaccountname := 'MONEYBROS_GRP';
 
        l_groups(2).cn := 'MONACLE_GRP';
        l_groups(2).description := 'Employees working on the Monacle Corporation account.';
        l_groups(2).distinguished_name := 'CN=MONACLE_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(2).name := 'MONACLE_GRP';
        l_groups(2).samaccountname := 'MONACLE_GRP';
 
        l_groups(3).cn := 'SCHMAPPLE_GRP';
        l_groups(3).description := 'Employees working on the Schmapple account.';
        l_groups(3).distinguished_name := 'CN=SCHMAPPLE_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(3).name := 'SCHMAPPLE_GRP';
        l_groups(3).samaccountname := 'SCHMAPPLE_GRP';
 
    END CASE;
    RETURN l_groups;
  END get_user_groups_aa;
 
  FUNCTION get_user_groups_va (p_user VARCHAR2) RETURN ldap_groups_va IS
    l_groups ldap_groups_va := ldap_groups_va();
  BEGIN
    CASE p_user
      WHEN 'jaugusti' THEN
 
        l_groups.extend;
        l_groups(1).cn := 'TIMCO_GRP';
        l_groups(1).description := 'Employees working on the Timco Industries account.';
        l_groups(1).distinguished_name := 'CN=TIMCO_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(1).name := 'TIMCO_GRP';
        l_groups(1).samaccountname := 'TIMCO_GRP';
 
      WHEN 'kkidd' THEN
 
        l_groups.extend;
        l_groups(1).cn := 'MONEYBROS_GRP';
        l_groups(1).description := 'Employees working on the Moneybros account.';
        l_groups(1).distinguished_name := 'CN=MONEYBROS_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(1).name := 'MONEYBROS_GRP';
        l_groups(1).samaccountname := 'MONEYBROS_GRP';
 
      WHEN 'bhill' THEN
 
        l_groups.extend;
        l_groups(1).cn := 'TIMCO_GRP';
        l_groups(1).description := 'Employees working on the Timco Industries account.';
        l_groups(1).distinguished_name := 'CN=TIMCO_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(1).name := 'TIMCO_GRP';
        l_groups(1).samaccountname := 'TIMCO_GRP';
 
        l_groups.extend;
        l_groups(2).cn := 'MONACLE_GRP';
        l_groups(2).description := 'Employees working on the Monacle Corporation account.';
        l_groups(2).distinguished_name := 'CN=MONACLE_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(2).name := 'MONACLE_GRP';
        l_groups(2).samaccountname := 'MONACLE_GRP';
 
      WHEN 'kvongkas' THEN
 
        l_groups.extend;
        l_groups(1).cn := 'MONEYBROS_GRP';
        l_groups(1).description := 'Employees working on the Moneybros account.';
        l_groups(1).distinguished_name := 'CN=MONEYBROS_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(1).name := 'MONEYBROS_GRP';
        l_groups(1).samaccountname := 'MONEYBROS_GRP';
 
        l_groups.extend;
        l_groups(2).cn := 'MONACLE_GRP';
        l_groups(2).description := 'Employees working on the Monacle Corporation account.';
        l_groups(2).distinguished_name := 'CN=MONACLE_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(2).name := 'MONACLE_GRP';
        l_groups(2).samaccountname := 'MONACLE_GRP';
 
        l_groups.extend;
        l_groups(3).cn := 'SCHMAPPLE_GRP';
        l_groups(3).description := 'Employees working on the Schmapple account.';
        l_groups(3).distinguished_name := 'CN=SCHMAPPLE_GRP,OU=Global Groups,DC=acme,DC=com';
        l_groups(3).name := 'SCHMAPPLE_GRP';
        l_groups(3).samaccountname := 'SCHMAPPLE_GRP';
 
    END CASE;
    RETURN l_groups;
  END get_user_groups_va;
 
END ldap_util;
/
SHOW ERRORS

As you can see, LDAP_UTIL has three functions that return the three kinds of collections in PL/SQL: nested tables, associative arrays, and varrays.

  FUNCTION get_user_groups_nt (p_user VARCHAR2) RETURN ldap_groups_nt;
  FUNCTION get_user_groups_aa (p_user VARCHAR2) RETURN ldap_groups_aa;
  FUNCTION get_user_groups_va (p_user VARCHAR2) RETURN ldap_groups_va;

The collections consist of a group of records representing “group” data from Active Directory.

  TYPE ldap_entry_typ IS RECORD (
    cn VARCHAR2(30),
    description VARCHAR2(1000),
    distinguished_name VARCHAR2(200),
    name VARCHAR2(30),
    samaccountname VARCHAR2(30));

The p_user parameter would be the user’s network id, corresponding to the sAMAccountName attribute in Active Directory; since we’re hardcoding group data this is not important to us at the moment. The point is you pass in a username and you get a collection of groups.

Using the same technique we demonstrated in part one, we now create a wrapper package of pipelined functions to transform the collection data into something we can query – an aggregate result set.

CREATE OR REPLACE PACKAGE ldap_util_pipelined AS
 
  FUNCTION get_user_groups_nt (p_user VARCHAR2) RETURN ldap_util.ldap_groups_nt PIPELINED;
  FUNCTION get_user_groups_aa (p_user VARCHAR2) RETURN ldap_util.ldap_groups_nt PIPELINED;
  FUNCTION get_user_groups_va (p_user VARCHAR2) RETURN ldap_util.ldap_groups_va PIPELINED;
 
END ldap_util_pipelined;
/
SHOW ERRORS
 
CREATE OR REPLACE PACKAGE BODY ldap_util_pipelined AS
 
  FUNCTION get_user_groups_nt (p_user VARCHAR2) 
  RETURN ldap_util.ldap_groups_nt PIPELINED IS
    l_groups ldap_util.ldap_groups_nt;
  BEGIN
    l_groups := ldap_util.get_user_groups_nt(p_user);
    FOR i IN 1 .. l_groups.count LOOP
      PIPE ROW (l_groups(i));
    END LOOP;
    RETURN;
  END get_user_groups_nt;
 
  FUNCTION get_user_groups_aa (p_user VARCHAR2) 
  RETURN ldap_util.ldap_groups_nt PIPELINED IS
    l_groups ldap_util.ldap_groups_aa;
  BEGIN
    l_groups := ldap_util.get_user_groups_aa(p_user);
    FOR i IN 1 .. l_groups.count LOOP
      PIPE ROW (l_groups(i));
    END LOOP;
    RETURN;
  END get_user_groups_aa;
 
  FUNCTION get_user_groups_va (p_user VARCHAR2) 
  RETURN ldap_util.ldap_groups_va PIPELINED IS
    l_groups ldap_util.ldap_groups_va;
  BEGIN
    l_groups := ldap_util.get_user_groups_va(p_user);
    FOR i IN 1 .. l_groups.count LOOP
      PIPE ROW (l_groups(i));
    END LOOP;
    RETURN;
  END get_user_groups_va;
 
END ldap_util_pipelined;
/
SHOW ERRORS

Note the return types of the three functions in ldap_util_pipelined.

  FUNCTION get_user_groups_nt (p_user VARCHAR2) RETURN ldap_util.ldap_groups_nt PIPELINED;
  FUNCTION get_user_groups_aa (p_user VARCHAR2) RETURN ldap_util.ldap_groups_nt PIPELINED;
  FUNCTION get_user_groups_va (p_user VARCHAR2) RETURN ldap_util.ldap_groups_va PIPELINED;

The first two return a nested table collection, while the third returns a VARRAY. These two collections can be pipelined because they correspond to native SQL datatypes in Oracle; not so for associative arrays. In order to pipeline the associative array returned from ldap_util.get_user_groups_aa, we need to map that collection to one of the other types. Here, we map it to a nested table. The iterative logic is the same, though, as the other two functions.

Now we can verify that our approach worked by issuing some simple queries, supported by the SQL “TABLE” construct.

SQL> select * from table(ldap_util_pipelined.get_user_groups_nt('jaugusti'));
 
CN        DESCRIPTION                                        DISTINGUISHED_NAME                            NAME      SAMACCOUN
--------- -------------------------------------------------- --------------------------------------------- --------- ---------
TIMCO_GRP Employees working on the Timco Industries account. CN=TIMCO_GRP,OU=Global Groups,DC=acme,DC=com  TIMCO_GRP TIMCO_GRP

As you can see, the individual attributes of the record structure form the columns of our new table, and we can reference those columns directly, if we wish.

SQL> select cn from table(ldap_util_pipelined.get_user_groups_nt('kvongkas'));
 
CN
---------------
MONEYBROS_GRP
MONACLE_GRP
SCHMAPPLE_GRP

All that’s left for us at this point is to construct a WHERE clause condition to use with Oracle’s Virtual Private Database feature to limit data returned from queries against the table QUARTERLY_SALES_DATA. That condition could look something like this, assuming that our application connects to the database as the logged in user.

client_id IN (SELECT CASE cn
                       WHEN 'TIMCO_GRP' THEN 100
                       WHEN 'MONEYBROS_GRP' THEN 200
                       WHEN 'MONACLE_GRP' THEN 300
                       WHEN 'SCHMAPPLE_GRP' THEN 400
                     END
                FROM TABLE(ldap_util_pipelined.get_user_groups_nt(USER)))

Here the mapping between AD group name and client ID is hardcoded in the CASE expression. A more flexible way would be to create a mapping table. Also, this is probably not a case study in writing a well performing VPD clause, but the real point here is that using SQL table functions and PL/SQL pipelined functions, we can translate data retrieved from an API, LDAP directory, or web service into table-like structures that we can query, join, and otherwise manipulate just like real tables.

Tweet

Making the assumption that you are using RMAN, you have already configured your backups to run a weekly full on each database to tape once per week, incrementals daily, and a reasonable amount of archivelogs being generated daily to the Fast Recovery Area, we can proceed to estimate your immediate needs.

If you are running Oracle and you are using RMAN, you can get the sizing numbers you need for your tape requirements somewhat easily if you are already taking backups to your Fast Recovery Area. There are a number of different views that can provide information about your backups, but I have found the most beneficial information can be gathered from the views used here. The helpful script that follows can be run to collect data on all the different types of backups you are currently running and the sizes (this script allows you to feed a list of SIDs):

#!/bin/bash

. /home/oracle/.bash_profile

ORAENV_ASK=NO

for DB in `echo “$DB_LIST”`; do
export ORACLE_SID=$DB
. oraenv $DB >/dev/null

echo “Database: $DB”

sqlplus -s \/ as sysdba<<EOF
set serveroutput on
set linesize 150
set pagesize 300

select ctime “Date”,

decode(backup_type, ‘L’, ‘Archive Log’, ‘D’, ‘Full’, ‘Incremental’) backup_type,bsize “Size GB” from (select trunc(bp.completion_time) ctime, backup_type, round(sum(bp.bytes/1024/1024/1024),2) bsize

from v\$backup_set bs,

v\$backup_piece bp where bs.set_stamp = bp.set_stamp and bs.set_count  = bp.set_count and bp.status = ‘A’ group by trunc(bp.completion_time), backup_type) order by 1, 2;

exit;
EOF
done

You can then look at the history of your current backups and right-size your tape requirements.

You might also check how many archive logs you are generating in a week under full production load:

col gbytes form 999,999.99

select sum(blocks*block_size)/1024/1024/1024 gbytes

from v$archived_log

where next_time > sysdate-8;

Some things to consider when you decide on your tape solution:

The type of tape drives

Encryption needs

The amount of compression

The number of tape drives

The type of backup software (OSB vs Netbackup vs Tivoli)

Your vaulting schedule (number of available media slots)

How fast can your channel send the data to the tape drives

Assuming you already have a tape SAN and you are not given enough money in your budget or enough lead time to get the gear in place before an audit, you might consider looking at what you have now in place and checking to see if you are doing too many backups, you have tuned your backups incorrectly, or if you have a number of backups that would benefit from VTL or dedup technologies in order to free space on your existing tape infrastructure.