My crsstat script: Improved Formatting of crs_stat on 10g and 11g
My crsstat shows a tabular format of CRS or HAS resources (i.e. Cluster 10gR2-11gR2 or ASM/Single instance scenario 11gR2). It has been a work in progress for a while. When I first started working with 10gR2 RAC, I quickly discovered that I didn’t quite like the formatting of the default crs_stat -t output, so I wrote my first version of the crsstat script, which consisted of a korn shell script and an awk script (which gave pretty close to the same output as the current version). I enhanced it at some point to show a different color, if the state didn’t match the target state. Then 11gR2 came out and I really didn’t like the output, so I sat down and wrote the awk script to format the 11gR2 output and added a variable at the top to choose which awk script. One of my colleagues saw it and liked the output, so I shared it; however, he asked if I could simplify the installation, as at that time you had to set the location of the script, whether it was 10g or 11g, and the location of the CRS_HOME. Finally, I sat down and decided to make it as universal and easy to use, as I could by adding the awk scripts as variables within the ksh script and adding a section to attempt to determine the CRS or HAS home automatically. It seems to work fairly well and several of my colleagues have been using it, so I thought I would share it more universally. Hope everyone finds it useful!
How it Works
It should work with 10gR2-11gR2 without changes. The first time is run, it will attempt to identify the Grid Infrastructure / CRS home and the version that is being used (CRS/HAS must be up for it to gather)- it takes that information and writes the info to a crsstat.env located in the same location as the script. It only gathers the information the first time, so if you upgrade the Oracle software you will want to remove the crsstat.env and allow the script to re-gather the information.
The output of the script assumes a term width of 132 and attempts to highlight possible issues, by showing in RED those resources with a target of ONLINE and a state of OFFLINE. You can also pass in “search criteria” as the first and only parameter.
Also of note, per Oracle support document 1086563.1, the “State Details” output in 11gR2 could show incorrect information, such as, “Instance Shutdown” even though it is up and running correctly.
Under the covers, the script is utilizing the “crs_stat -t” in 10gR2/11gR1 and “crsctl status resource -t” and then awks their output to create the tabular format. It has been used on AIX and Linux (SuSE, Red Hat, Oracle).
How to Install
Pre-requisite to Installation:
1. CRS/HAS must be running
2. The location of the crsstat script must be writable by the user that will run the installation (either root or your oracle user)
Installation Location:
Generally, it is practical to put the crsstat script in the /usr/local/bin directory, so everyone will have access (in most users path). Make sure the crsstat.env file can be created in the /usr/local/bin directory the first time the script is run. If it is preferred to put it an alternate location, choose a location that is accessible by your Unix users that should be able to list the CRS or HAS resources (i.e. in their PATH).
Installation Process:
1) Copy the installation script from this link http://blog.enkitec.com/wp-content/uploads/2012/07/crsstat.v1.2.2.zip
2) Do a “cp crsstat.v1.2.1.zip /usr/local/bin/”
3) As root run the crsstat and a “crsstat.env” file is created:
# cd /usr/local/bin # unzip crsstat.v1.2.1.zip # chmod 755 crsstat # ./crsstat
Output like example below…
# chmod 644 crsstat.env
4) Verify that the crsstat.env file has two entries like the following:
GRID_HOME=/u01/app/oracle/product/11.2.0/grid
GRID_VER=11.2
Example Usage
Basic Usage (note that the dbfs filesystem is offline when it should be online, thus it is highlight in red):
$ crsstat
Resource Name Resource Type Target State Node State Details
---------------------------------------- -------------- ------------ ------------ --------------- --------------
ora.DATA.dg diskgroup ONLINE ONLINE tstdb01
ora.DATA.dg diskgroup ONLINE ONLINE tstdb02
ora.LISTENER.lsnr Listener ONLINE ONLINE tstdb01
ora.LISTENER.lsnr Listener ONLINE ONLINE tstdb02
ora.LISTENER_SCAN1.lsnr SCAN Listener ONLINE ONLINE tstdb02
ora.LISTENER_SCAN2.lsnr SCAN Listener ONLINE ONLINE tstdb01
ora.LISTENER_SCAN3.lsnr SCAN Listener ONLINE ONLINE tstdb01
ora.RECO.dg diskgroup ONLINE ONLINE tstdb01
ora.RECO.dg diskgroup ONLINE ONLINE tstdb02
ora.SYSTEM.dg diskgroup ONLINE ONLINE tstdb01
ora.SYSTEM.dg diskgroup ONLINE ONLINE tstdb02
ora.asm ASM ONLINE ONLINE tstdb01
ora.asm ASM ONLINE ONLINE tstdb02
ora.cvu cvu ONLINE ONLINE tstdb01
ora.dbfs.db database ONLINE ONLINE tstdb01 Open
ora.dbfs.db database ONLINE ONLINE tstdb02 Open
ora.dbfs.filesystem ONLINE OFFLINE
ora.dbfs.filesystem ONLINE OFFLINE
ora.demo.db database ONLINE ONLINE tstdb01 Open
ora.demo.db database ONLINE ONLINE tstdb02 Open
ora.demo.workshop.svc service ONLINE ONLINE tstdb01
ora.demo.workshop.svc service ONLINE ONLINE tstdb02
ora.tstdb01.vip Cluster VIP ONLINE ONLINE tstdb01
ora.tstdb02.vip Cluster VIP ONLINE ONLINE tstdb02
ora.net1.network Network (VIP) ONLINE ONLINE tstdb01
ora.net1.network Network (VIP) ONLINE ONLINE tstdb02
ora.ons Ora Notif Svc ONLINE ONLINE tstdb01
ora.ons Ora Notif Svc ONLINE ONLINE tstdb02
ora.registry.acfs registry ONLINE ONLINE tstdb01
ora.registry.acfs registry ONLINE ONLINE tstdb02
ora.scan1.vip SCAN VIP ONLINE ONLINE tstdb02
ora.scan2.vip SCAN VIP ONLINE ONLINE tstdb01
ora.scan3.vip SCAN VIP ONLINE ONLINE tstdb01
ora.test.db database OFFLINE OFFLINE
ora.test.db database OFFLINE OFFLINE
A couple of examples using the search key:
$ crsstat service
Resource Name Resource Type Target State Node State Details
---------------------------------------- -------------- ------------ ------------ --------------- --------------
ora.demo.workshop.svc service ONLINE ONLINE tstdb01
ora.demo.workshop.svc service ONLINE ONLINE tstdb02
$ crsstat demo
Resource Name Resource Type Target State Node State Details
---------------------------------------- -------------- ------------ ------------ --------------- --------------
ora.demo.db database ONLINE ONLINE enkdb01 Open
ora.demo.db database ONLINE ONLINE enkdb02 Open
ora.demo.workshop.svc service ONLINE ONLINE enkdb01
ora.demo.workshop.svc service ONLINE ONLINE enkdb02
Another example, as sometimes when there are many databases it is useful to pipe the output to grep:
$ crsstat database | grep -i offline
ora.test.db database OFFLINE OFFLINE
ora.test.db database OFFLINE OFFLINE
This script is pretty nifty. We used a similar script on 10g Clusterware since the output was completely unreadable using crs_stat. But this is better. It worked on OL5.6 with 11.2.0.2.0 Grid Infrastructure.
Is it possible to separate the resources based on whether they are local or cluster (similar to the crsctl stat res -t output).
Hi Leighton,
Yes, this could be done. I will plan to add an optional switch that could be used for this purpose.
Nice one!!! Thanks for sharing.
Hi Paul,
nice, little, useful script. I just tested this on Solaris 10 with Oracle 10g and 11.2 and it works almost 100%. The awk script for 11.2 uses ‘gsub’ which is not implemented in awk on Solaris. You can comment out those two lines or call ‘nawk’ instead and it works fine.
Bjoern
Hi Bjoern,
Thank you for the information on Solaris. I will incorporate the use of nawk for Solaris platform and will get out a new version in the near future.
can you tell me , how do i make the script work in solaris .. because it is giving error as follows
Resource Name Resource Type Target State Node State Details
—————————————- ————– ———— ———— ————– —————
awk: syntax error near line 83
awk: illegal statement near line 83
awk: syntax error near line 84
awk: illegal statement near line 84
$
Very nice! Just gave it a whirl on 11.2.0.3 Oracle Linux and ran like a charm. I too have been underwhelmed by the output of crs_stat -t, namely the chopping of the resource name, so well done!
Hi Paul,
I noticed that on 10.2.0.4 Clusterware it lists all the resources as being of the type ‘application’. Also the state details column is null for all columns.
tibora10t[tibtst1]-/home/oracle/bin >./crsstat
Resource Name Resource Type Target State Node State Details
—————————————- ————– ———— ———— ————— —————
ora.tibora10t.ASM1.asm application ONLINE ONLINE tibora10t
ora.tibora10t.LISTENER_TIBORA10T.lsnr application ONLINE ONLINE tibora10t
ora.tibora10t.gsd application ONLINE ONLINE tibora10t
ora.tibora10t.ons application ONLINE ONLINE tibora10t
ora.tibora10t.vip application ONLINE ONLINE tibora10t
ora.tibora11t.ASM2.asm application ONLINE ONLINE tibora11t
ora.tibora11t.LISTENER_TIBORA11T.lsnr application ONLINE ONLINE tibora11t
ora.tibora11t.gsd application ONLINE ONLINE tibora11t
ora.tibora11t.ons application ONLINE ONLINE tibora11t
ora.tibora11t.vip application ONLINE ONLINE tibora11t
ora.tibtst.db application ONLINE ONLINE tibora10t
ora.tibtst.tibtst1.inst application ONLINE ONLINE tibora10t
ora.tibtst.tibtst2.inst application ONLINE ONLINE tibora11t
ora.tibtst.tibtst_apollo.cs application ONLINE ONLINE tibora10t
ora.tibtst.tibtst_apollo.tibtst2.srv application ONLINE ONLINE tibora11t
ora.tibtst.tibtst_applog.cs application ONLINE ONLINE tibora10t
ora.tibtst.tibtst_applog.tibtst1.srv application ONLINE ONLINE tibora11t
ora.tibtst.tibtst_basic.cs application ONLINE ONLINE tibora10t
ora.tibtst.tibtst_basic.tibtst2.srv application ONLINE ONLINE tibora11t
ora.tibtst.tibtst_smap.cs application ONLINE ONLINE tibora10t
ora.tibtst.tibtst_smap.tibtst1.srv application ONLINE ONLINE tibora10t
ora.tibtst.tibtst_smap.tibtst2.srv application ONLINE ONLINE tibora11t
Hi Leighton,
I don’t have 10g clusterware handy right now, but my recollection is that there wasn’t a “type” or “state details” in the 10g (all types were “application”). I could interpret the resource name to specify a “pseudo type” in the type field that aligns as closely as possible with the 11g version- I will look to do this in the near future. I don’t think there is anything to be done with the “state details” in 10g that wouldn’t be intrusive. I will double-check when I have access to 10g clusterware.
this script is not working for me in solaris 10 . giving error like follows ,
Resource Name Resource Type Target State Node State Details
—————————————- ————– ———— ———— ————— —————
awk: syntax error near line 83
awk: illegal statement near line 83
awk: syntax error near line 84
awk: illegal statement near line 84
can you please help me ?
Hi,
Here is a link to a new version.
I haven’t gotten some of the more involved suggested changes done yet, but it has been updated to use nawk on Solaris:
http://blog.enkitec.com/wp-content/uploads/2011/12/crsstat.zip
I have tested your script on solaris . it is awesome…… Working Fine ……
Thanks for your quick response…..
Here is the latest version of the crsstat script:
http://blog.enkitec.com/wp-content/uploads/2012/04/crsstat.v1.2.1.zip
It is updated with the following enhancements and fixes—
Usage with -h or invalid switch.
Added fields: cluster/local resource, failure count FC, and restart count RC.
Added switch (-c) for outputing Cluster and Local resources in different sections.
Fixed issue with resetting colors back.
Thank you for sharing this excellent script. It is much clear than crs_stat on 11gR2.
One thing that I noticed when issue “crsstat database”, “Resource Name” only show database name instead of instance name. When we need to switch instance back to original node after one node eviction, it is difficult to distinguish instance. Can you explain how can I tell instance name?
It would take pulling information out of the runtime and static information. The runtime output from crsctl status resource -v contains an “ID” which identifies the resource. If you then get the static output from crsctl status resource -p (or -f), the “USR_ORA_INST_NAME” is the instance name for the corresponding “ID”. I like the idea and can look at adding this, but not sure when I might get a chance to get the code in the script.
Very useful script – thanks a lot for sharing!
Just one thing: It seems to me that the Resource Type is shown in the opposite way: e.g.. a Scan Listener is a Cluster Resource, but it shows as L
The output is just showing the Cluster versus Local as defined by Oracle– similar to the division as shown when doing a “crsctl show resource -t”.
No it doesn’t. At least not the version that I downloaded today in the afternoon. crsctl stat res -t shows correctly e.g. a Scan Listener as CLUSTER resource whereas your script shows it with the flag L which stands – as I suppose – for LOCAL resource. In other words it shows this in the opposite (and therefore incorrect) way than crsctl stat res -t does. Although this can be easily corrected, I suggest that you fix that in the script that you offer for download.
You are correct, the one offered for download is showing exactly opposite of what it should. I will get an updated version out there at my earliest opportunity.
Thank you – also again for this very handy script! I have recommended it
Excellent, excellent tool.
I tweaked just a little to remove the white-on-black display forced by the printf command. We use black on white terminals and I wanted the output to match it. It’s just a personal preference.
I further modified the script to have another flag “-o” to show only those resources that are supposed to be online but offline now. This is very useful when you want to quickly find out the resources needing attention. The -o flag also works with search string. For instance:
crsstat -o : shows all the resources offline but target=online
crssat -o diskgroup : same output but for diskgroups only
Here is the latest version of the crsstat script:
http://blog.enkitec.com/wp-content/uploads/2012/07/crsstat.v1.2.2.zip
The bug noted by Uwe Hesse: local resources were marked as “Cluster Resources” or with “C” and cluster resources were marked as “Local Resources” or with “L” has been corrected in version 1.2.2. Resources now show the correct indicator for Cluster or Local.
Hi Paul,
thanks for the sharing.
I got an error out when run the script under a normal directory,
root # pwd
/orabackup/workarea/yl
root # ksh crsstat.sh
crsstat.sh[125]: ./crsctl: not found
crsstat.sh[128]: ./crsctl: not found
The CRS or HAS version could not be determined.
Commands to query for either failed.
Exiting….
seems the error is with the code below:
CSSD_DIR=`dirname $( ps -eo args | grep ocssd.bin | grep -v grep | ${AWK} ‘{print $1}’)`
if [[ ! -z ${CSSD_DIR} || ${CSSD_DIR} == "." ]] ; then
CSSD_DIR=$(dirname $( ps -eo args | grep hasd.bin | grep -v grep | ${AWK} ‘{print $1}’))
fi
here is a trace with ksh -x
+ dirname /oracle/product/10.2.0/crs_1/bin/ocssd.bin
CSSD_DIR=/oracle/product/10.2.0/crs_1/bin
+ [[ ! -z /reuters/app/oracle/product/10.2.0/crs_1/bin ]]
+ + ps -eo args
+ grep hasd.bin
+ /usr/bin/nawk {print $1}
+ grep -v grep
+ dirname
CSSD_DIR=.
+ [[ ! -z . ]]
+ + /usr/bin/nawk {print substr( $1, 1, length( $1)-4)}
+ echo .
GRID_HOME=
+ + ./crsctl query crs softwareversion
crsstat.sh[125]: ./crsctl: not found
retval=
Could you please help on this?
Hi, The directory from which the script runs is not important; however, CRS/HAS must be running for it to determine the Grid Home automatically. Alternatively, you could manually set the location and version of CRS/HAS:
cd {location of crsstat script}
vi crsstat.env
and put in the following 2 variables:
GRID_HOME=/u01/app/11.2.0.3/grid
GRID_VER=11.2
Hope this helps…