My crsstat script: Improved Formatting of crs_stat on 10g and 11g
My crsstat shows a tabular format of CRS or HAS resources (i.e. Cluster 10gR2-11gR2 or ASM/Single instance scenario 11gR2). It has been a work in progress for a while. When I first started working with 10gR2 RAC, I quickly discovered that I didn’t quite like the formatting of the default crs_stat -t output, so I wrote my first version of the crsstat script, which consisted of a korn shell script and an awk script (which gave pretty close to the same output as the current version). I enhanced it at some point to show a different color, if the state didn’t match the target state. Then 11gR2 came out and I really didn’t like the output, so I sat down and wrote the awk script to format the 11gR2 output and added a variable at the top to choose which awk script. One of my colleagues saw it and liked the output, so I shared it; however, he asked if I could simplify the installation, as at that time you had to set the location of the script, whether it was 10g or 11g, and the location of the CRS_HOME. Finally, I sat down and decided to make it as universal and easy to use, as I could by adding the awk scripts as variables within the ksh script and adding a section to attempt to determine the CRS or HAS home automatically. It seems to work fairly well and several of my colleagues have been using it, so I thought I would share it more universally. Hope everyone finds it useful!
How it Works
It should work with 10gR2-11gR2 without changes. The first time is run, it will attempt to identify the Grid Infrastructure / CRS home and the version that is being used (CRS/HAS must be up for it to gather)- it takes that information and writes the info to a crsstat.env located in the same location as the script. It only gathers the information the first time, so if you upgrade the Oracle software you will want to remove the crsstat.env and allow the script to re-gather the information.
The output of the script assumes a term width of 132 and attempts to highlight possible issues, by showing in RED those resources with a target of ONLINE and a state of OFFLINE. You can also pass in “search criteria” as the first and only parameter.
Also of note, per Oracle support document 1086563.1, the “State Details” output in 11gR2 could show incorrect information, such as, “Instance Shutdown” even though it is up and running correctly.
Under the covers, the script is utilizing the “crs_stat -t” in 10gR2/11gR1 and “crsctl status resource -t” and then awks their output to create the tabular format. It has been used on AIX and Linux (SuSE, Red Hat, Oracle).
How to Install
Pre-requisite to Installation:
1. CRS/HAS must be running
2. The location of the crsstat script must be writable by the user that will run the installation (either root or your oracle user)
Installation Location:
Generally, it is practical to put the crsstat script in the /usr/local/bin directory, so everyone will have access (in most users path). Make sure the crsstat.env file can be created in the /usr/local/bin directory the first time the script is run. If it is preferred to put it an alternate location, choose a location that is accessible by your Unix users that should be able to list the CRS or HAS resources (i.e. in their PATH).
Installation Process:
1) Copy the installation script from this link http://blog.enkitec.com/wp-content/uploads/2011/10/crsstat.zip
2) Do a “cp crsstat.zip /usr/local/bin/”
3) As root run the crsstat and a “crsstat.env” file is created:
# cd /usr/local/bin # unzip crsstat.zip # chmod 755 crsstat # ./crsstat
Output like example below…
# chmod 644 crsstat.env
4) Verify that the crsstat.env file has two entries like the following:
GRID_HOME=/u01/app/oracle/product/11.2.0/grid
GRID_VER=11.2
Example Usage
Basic Usage (note that the dbfs filesystem is offline when it should be online, thus it is highlight in red):
$ crsstat
Resource Name Resource Type Target State Node State Details
---------------------------------------- -------------- ------------ ------------ --------------- --------------
ora.DATA.dg diskgroup ONLINE ONLINE tstdb01
ora.DATA.dg diskgroup ONLINE ONLINE tstdb02
ora.LISTENER.lsnr Listener ONLINE ONLINE tstdb01
ora.LISTENER.lsnr Listener ONLINE ONLINE tstdb02
ora.LISTENER_SCAN1.lsnr SCAN Listener ONLINE ONLINE tstdb02
ora.LISTENER_SCAN2.lsnr SCAN Listener ONLINE ONLINE tstdb01
ora.LISTENER_SCAN3.lsnr SCAN Listener ONLINE ONLINE tstdb01
ora.RECO.dg diskgroup ONLINE ONLINE tstdb01
ora.RECO.dg diskgroup ONLINE ONLINE tstdb02
ora.SYSTEM.dg diskgroup ONLINE ONLINE tstdb01
ora.SYSTEM.dg diskgroup ONLINE ONLINE tstdb02
ora.asm ASM ONLINE ONLINE tstdb01
ora.asm ASM ONLINE ONLINE tstdb02
ora.cvu cvu ONLINE ONLINE tstdb01
ora.dbfs.db database ONLINE ONLINE tstdb01 Open
ora.dbfs.db database ONLINE ONLINE tstdb02 Open
ora.dbfs.filesystem ONLINE OFFLINE
ora.dbfs.filesystem ONLINE OFFLINE
ora.demo.db database ONLINE ONLINE tstdb01 Open
ora.demo.db database ONLINE ONLINE tstdb02 Open
ora.demo.workshop.svc service ONLINE ONLINE tstdb01
ora.demo.workshop.svc service ONLINE ONLINE tstdb02
ora.tstdb01.vip Cluster VIP ONLINE ONLINE tstdb01
ora.tstdb02.vip Cluster VIP ONLINE ONLINE tstdb02
ora.net1.network Network (VIP) ONLINE ONLINE tstdb01
ora.net1.network Network (VIP) ONLINE ONLINE tstdb02
ora.ons Ora Notif Svc ONLINE ONLINE tstdb01
ora.ons Ora Notif Svc ONLINE ONLINE tstdb02
ora.registry.acfs registry ONLINE ONLINE tstdb01
ora.registry.acfs registry ONLINE ONLINE tstdb02
ora.scan1.vip SCAN VIP ONLINE ONLINE tstdb02
ora.scan2.vip SCAN VIP ONLINE ONLINE tstdb01
ora.scan3.vip SCAN VIP ONLINE ONLINE tstdb01
ora.test.db database OFFLINE OFFLINE
ora.test.db database OFFLINE OFFLINE
A couple of examples using the search key:
$ crsstat service
Resource Name Resource Type Target State Node State Details
---------------------------------------- -------------- ------------ ------------ --------------- --------------
ora.demo.workshop.svc service ONLINE ONLINE tstdb01
ora.demo.workshop.svc service ONLINE ONLINE tstdb02
$ crsstat demo
Resource Name Resource Type Target State Node State Details
---------------------------------------- -------------- ------------ ------------ --------------- --------------
ora.demo.db database ONLINE ONLINE enkdb01 Open
ora.demo.db database ONLINE ONLINE enkdb02 Open
ora.demo.workshop.svc service ONLINE ONLINE enkdb01
ora.demo.workshop.svc service ONLINE ONLINE enkdb02
Another example, as sometimes when there are many databases it is useful to pipe the output to grep:
$ crsstat database | grep -i offline
ora.test.db database OFFLINE OFFLINE
ora.test.db database OFFLINE OFFLINE
This script is pretty nifty. We used a similar script on 10g Clusterware since the output was completely unreadable using crs_stat. But this is better. It worked on OL5.6 with 11.2.0.2.0 Grid Infrastructure.
Is it possible to separate the resources based on whether they are local or cluster (similar to the crsctl stat res -t output).
Hi Leighton,
Yes, this could be done. I will plan to add an optional switch that could be used for this purpose.
Nice one!!! Thanks for sharing.
Hi Paul,
nice, little, useful script. I just tested this on Solaris 10 with Oracle 10g and 11.2 and it works almost 100%. The awk script for 11.2 uses ‘gsub’ which is not implemented in awk on Solaris. You can comment out those two lines or call ‘nawk’ instead and it works fine.
Bjoern
Hi Bjoern,
Thank you for the information on Solaris. I will incorporate the use of nawk for Solaris platform and will get out a new version in the near future.
can you tell me , how do i make the script work in solaris .. because it is giving error as follows
Resource Name Resource Type Target State Node State Details
—————————————- ————– ———— ———— ————– —————
awk: syntax error near line 83
awk: illegal statement near line 83
awk: syntax error near line 84
awk: illegal statement near line 84
$
Very nice! Just gave it a whirl on 11.2.0.3 Oracle Linux and ran like a charm. I too have been underwhelmed by the output of crs_stat -t, namely the chopping of the resource name, so well done!
Hi Paul,
I noticed that on 10.2.0.4 Clusterware it lists all the resources as being of the type ‘application’. Also the state details column is null for all columns.
tibora10t[tibtst1]-/home/oracle/bin >./crsstat
Resource Name Resource Type Target State Node State Details
—————————————- ————– ———— ———— ————— —————
ora.tibora10t.ASM1.asm application ONLINE ONLINE tibora10t
ora.tibora10t.LISTENER_TIBORA10T.lsnr application ONLINE ONLINE tibora10t
ora.tibora10t.gsd application ONLINE ONLINE tibora10t
ora.tibora10t.ons application ONLINE ONLINE tibora10t
ora.tibora10t.vip application ONLINE ONLINE tibora10t
ora.tibora11t.ASM2.asm application ONLINE ONLINE tibora11t
ora.tibora11t.LISTENER_TIBORA11T.lsnr application ONLINE ONLINE tibora11t
ora.tibora11t.gsd application ONLINE ONLINE tibora11t
ora.tibora11t.ons application ONLINE ONLINE tibora11t
ora.tibora11t.vip application ONLINE ONLINE tibora11t
ora.tibtst.db application ONLINE ONLINE tibora10t
ora.tibtst.tibtst1.inst application ONLINE ONLINE tibora10t
ora.tibtst.tibtst2.inst application ONLINE ONLINE tibora11t
ora.tibtst.tibtst_apollo.cs application ONLINE ONLINE tibora10t
ora.tibtst.tibtst_apollo.tibtst2.srv application ONLINE ONLINE tibora11t
ora.tibtst.tibtst_applog.cs application ONLINE ONLINE tibora10t
ora.tibtst.tibtst_applog.tibtst1.srv application ONLINE ONLINE tibora11t
ora.tibtst.tibtst_basic.cs application ONLINE ONLINE tibora10t
ora.tibtst.tibtst_basic.tibtst2.srv application ONLINE ONLINE tibora11t
ora.tibtst.tibtst_smap.cs application ONLINE ONLINE tibora10t
ora.tibtst.tibtst_smap.tibtst1.srv application ONLINE ONLINE tibora10t
ora.tibtst.tibtst_smap.tibtst2.srv application ONLINE ONLINE tibora11t
Hi Leighton,
I don’t have 10g clusterware handy right now, but my recollection is that there wasn’t a “type” or “state details” in the 10g (all types were “application”). I could interpret the resource name to specify a “pseudo type” in the type field that aligns as closely as possible with the 11g version- I will look to do this in the near future. I don’t think there is anything to be done with the “state details” in 10g that wouldn’t be intrusive. I will double-check when I have access to 10g clusterware.
this script is not working for me in solaris 10 . giving error like follows ,
Resource Name Resource Type Target State Node State Details
—————————————- ————– ———— ———— ————— —————
awk: syntax error near line 83
awk: illegal statement near line 83
awk: syntax error near line 84
awk: illegal statement near line 84
can you please help me ?
Hi,
Here is a link to a new version.
I haven’t gotten some of the more involved suggested changes done yet, but it has been updated to use nawk on Solaris:
http://blog.enkitec.com/wp-content/uploads/2011/12/crsstat.zip
I have tested your script on solaris . it is awesome…… Working Fine ……
Thanks for your quick response…..