My crsstat script: Improved Formatting of crs_stat on 10g and 11g

My crsstat shows a tabular format of CRS or HAS resources (i.e. Cluster 10gR2-11gR2 or ASM/Single instance scenario 11gR2). It has been a work in progress for a while. When I first started working with 10gR2 RAC, I quickly discovered that I didn’t quite like the formatting of the default crs_stat -t output, so I wrote my first version of the crsstat script, which consisted of a korn shell script and an awk script (which gave pretty close to the same output as the current version). I enhanced it at some point to show a different color, if the state didn’t match the target state. Then 11gR2 came out and I really didn’t like the output, so I sat down and wrote the awk script to format the 11gR2 output and added a variable at the top to choose which awk script. One of my colleagues saw it and liked the output, so I shared it; however, he asked if I could simplify the installation, as at that time you had to set the location of the script, whether it was 10g or 11g, and the location of the CRS_HOME. Finally, I sat down and decided to make it as universal and easy to use, as I could by adding the awk scripts as variables within the ksh script and adding a section to attempt to determine the CRS or HAS home automatically. It seems to work fairly well and several of my colleagues have been using it, so I thought I would share it more universally. Hope everyone finds it useful!

How it Works

It should work with 10gR2-11gR2 without changes. The first time is run, it will attempt to identify the Grid Infrastructure / CRS home and the version that is being used (CRS/HAS must be up for it to gather)- it takes that information and writes the info to a crsstat.env located in the same location as the script. It only gathers the information the first time, so if you upgrade the Oracle software you will want to remove the crsstat.env and allow the script to re-gather the information.

The output of the script assumes a term width of 132 and attempts to highlight possible issues, by showing in RED those resources with a target of ONLINE and a state of OFFLINE. You can also pass in “search criteria” as the first and only parameter.

Also of note, per Oracle support document 1086563.1, the “State Details” output in 11gR2 could show incorrect information, such as, “Instance Shutdown” even though it is up and running correctly.

Under the covers, the script is utilizing the “crs_stat -t” in 10gR2/11gR1 and “crsctl status resource -t” and then awks their output to create the tabular format. It has been used on AIX and Linux (SuSE, Red Hat, Oracle).

How to Install

Pre-requisite to Installation:

1. CRS/HAS must be running
2. The location of the crsstat script must be writable by the user that will run the installation (either root or your oracle user)

Installation Location:

Generally, it is practical to put the crsstat script in the /usr/local/bin directory, so everyone will have access (in most users path). Make sure the crsstat.env file can be created in the /usr/local/bin directory the first time the script is run. If it is preferred to put it an alternate location, choose a location that is accessible by your Unix users that should be able to list the CRS or HAS resources (i.e. in their PATH).

Installation Process:

1) Copy the installation script from this link http://blog.enkitec.com/wp-content/uploads/2012/07/crsstat.v1.2.2.zip
2) Do a “cp crsstat.v1.2.1.zip /usr/local/bin/”
3) As root run the crsstat and a “crsstat.env” file is created:

# cd /usr/local/bin
# unzip crsstat.v1.2.1.zip
# chmod 755 crsstat
# ./crsstat

Output like example below…

# chmod 644 crsstat.env

4) Verify that the crsstat.env file has two entries like the following:
GRID_HOME=/u01/app/oracle/product/11.2.0/grid
GRID_VER=11.2

Example Usage

Basic Usage (note that the dbfs filesystem is offline when it should be online, thus it is highlight in red):

$ crsstat

Resource Name                            Resource Type  Target       State        Node            State Details
---------------------------------------- -------------- ------------ ------------ --------------- --------------
ora.DATA.dg                              diskgroup      ONLINE       ONLINE       tstdb01
ora.DATA.dg                              diskgroup      ONLINE       ONLINE       tstdb02
ora.LISTENER.lsnr                        Listener       ONLINE       ONLINE       tstdb01
ora.LISTENER.lsnr                        Listener       ONLINE       ONLINE       tstdb02
ora.LISTENER_SCAN1.lsnr                  SCAN Listener  ONLINE       ONLINE       tstdb02
ora.LISTENER_SCAN2.lsnr                  SCAN Listener  ONLINE       ONLINE       tstdb01
ora.LISTENER_SCAN3.lsnr                  SCAN Listener  ONLINE       ONLINE       tstdb01
ora.RECO.dg                              diskgroup      ONLINE       ONLINE       tstdb01
ora.RECO.dg                              diskgroup      ONLINE       ONLINE       tstdb02
ora.SYSTEM.dg                            diskgroup      ONLINE       ONLINE       tstdb01
ora.SYSTEM.dg                            diskgroup      ONLINE       ONLINE       tstdb02
ora.asm                                  ASM            ONLINE       ONLINE       tstdb01
ora.asm                                  ASM            ONLINE       ONLINE       tstdb02
ora.cvu                                  cvu            ONLINE       ONLINE       tstdb01
ora.dbfs.db                              database       ONLINE       ONLINE       tstdb01         Open
ora.dbfs.db                              database       ONLINE       ONLINE       tstdb02         Open
ora.dbfs.filesystem                                     ONLINE       OFFLINE
ora.dbfs.filesystem                                     ONLINE       OFFLINE
ora.demo.db                              database       ONLINE       ONLINE       tstdb01         Open
ora.demo.db                              database       ONLINE       ONLINE       tstdb02         Open
ora.demo.workshop.svc                    service        ONLINE       ONLINE       tstdb01
ora.demo.workshop.svc                    service        ONLINE       ONLINE       tstdb02
ora.tstdb01.vip                          Cluster VIP    ONLINE       ONLINE       tstdb01
ora.tstdb02.vip                          Cluster VIP    ONLINE       ONLINE       tstdb02
ora.net1.network                         Network (VIP)  ONLINE       ONLINE       tstdb01
ora.net1.network                         Network (VIP)  ONLINE       ONLINE       tstdb02
ora.ons                                  Ora Notif Svc  ONLINE       ONLINE       tstdb01
ora.ons                                  Ora Notif Svc  ONLINE       ONLINE       tstdb02
ora.registry.acfs                        registry       ONLINE       ONLINE       tstdb01
ora.registry.acfs                        registry       ONLINE       ONLINE       tstdb02
ora.scan1.vip                            SCAN VIP       ONLINE       ONLINE       tstdb02
ora.scan2.vip                            SCAN VIP       ONLINE       ONLINE       tstdb01
ora.scan3.vip                            SCAN VIP       ONLINE       ONLINE       tstdb01
ora.test.db                              database       OFFLINE      OFFLINE
ora.test.db                              database       OFFLINE      OFFLINE

A couple of examples using the search key:

$ crsstat service

Resource Name                            Resource Type  Target       State        Node            State Details
---------------------------------------- -------------- ------------ ------------ --------------- --------------
ora.demo.workshop.svc                    service        ONLINE       ONLINE       tstdb01
ora.demo.workshop.svc                    service        ONLINE       ONLINE       tstdb02

$ crsstat demo

Resource Name                            Resource Type  Target       State        Node            State Details
---------------------------------------- -------------- ------------ ------------ --------------- --------------
ora.demo.db                              database       ONLINE       ONLINE       enkdb01         Open
ora.demo.db                              database       ONLINE       ONLINE       enkdb02         Open
ora.demo.workshop.svc                    service        ONLINE       ONLINE       enkdb01
ora.demo.workshop.svc                    service        ONLINE       ONLINE       enkdb02

Another example, as sometimes when there are many databases it is useful to pipe the output to grep:

$ crsstat database | grep -i offline

ora.test.db                              database       OFFLINE      OFFLINE
ora.test.db                              database       OFFLINE      OFFLINE

About Paul Elbow

Comments

32 Responses to “My crsstat script: Improved Formatting of crs_stat on 10g and 11g”
  1. Leighton says:

    This script is pretty nifty. We used a similar script on 10g Clusterware since the output was completely unreadable using crs_stat. But this is better. It worked on OL5.6 with 11.2.0.2.0 Grid Infrastructure.
    Is it possible to separate the resources based on whether they are local or cluster (similar to the crsctl stat res -t output).

    • Paul Elbow says:

      Hi Leighton,
      Yes, this could be done. I will plan to add an optional switch that could be used for this purpose.

  2. Anand says:

    Nice one!!! Thanks for sharing.

  3. Bjoern Rost says:

    Hi Paul,

    nice, little, useful script. I just tested this on Solaris 10 with Oracle 10g and 11.2 and it works almost 100%. The awk script for 11.2 uses ‘gsub’ which is not implemented in awk on Solaris. You can comment out those two lines or call ‘nawk’ instead and it works fine.

    Bjoern

    • Paul Elbow says:

      Hi Bjoern,

      Thank you for the information on Solaris. I will incorporate the use of nawk for Solaris platform and will get out a new version in the near future.

    • newuser says:

      can you tell me , how do i make the script work in solaris .. because it is giving error as follows

      Resource Name Resource Type Target State Node State Details
      —————————————- ————– ———— ———— ————– —————
      awk: syntax error near line 83
      awk: illegal statement near line 83
      awk: syntax error near line 84
      awk: illegal statement near line 84

      $

  4. Andy Klock says:

    Very nice! Just gave it a whirl on 11.2.0.3 Oracle Linux and ran like a charm. I too have been underwhelmed by the output of crs_stat -t, namely the chopping of the resource name, so well done!

  5. Leighton says:

    Hi Paul,
    I noticed that on 10.2.0.4 Clusterware it lists all the resources as being of the type ‘application’. Also the state details column is null for all columns.
    tibora10t[tibtst1]-/home/oracle/bin >./crsstat

    Resource Name Resource Type Target State Node State Details
    —————————————- ————– ———— ———— ————— —————
    ora.tibora10t.ASM1.asm application ONLINE ONLINE tibora10t
    ora.tibora10t.LISTENER_TIBORA10T.lsnr application ONLINE ONLINE tibora10t
    ora.tibora10t.gsd application ONLINE ONLINE tibora10t
    ora.tibora10t.ons application ONLINE ONLINE tibora10t
    ora.tibora10t.vip application ONLINE ONLINE tibora10t
    ora.tibora11t.ASM2.asm application ONLINE ONLINE tibora11t
    ora.tibora11t.LISTENER_TIBORA11T.lsnr application ONLINE ONLINE tibora11t
    ora.tibora11t.gsd application ONLINE ONLINE tibora11t
    ora.tibora11t.ons application ONLINE ONLINE tibora11t
    ora.tibora11t.vip application ONLINE ONLINE tibora11t
    ora.tibtst.db application ONLINE ONLINE tibora10t
    ora.tibtst.tibtst1.inst application ONLINE ONLINE tibora10t
    ora.tibtst.tibtst2.inst application ONLINE ONLINE tibora11t
    ora.tibtst.tibtst_apollo.cs application ONLINE ONLINE tibora10t
    ora.tibtst.tibtst_apollo.tibtst2.srv application ONLINE ONLINE tibora11t
    ora.tibtst.tibtst_applog.cs application ONLINE ONLINE tibora10t
    ora.tibtst.tibtst_applog.tibtst1.srv application ONLINE ONLINE tibora11t
    ora.tibtst.tibtst_basic.cs application ONLINE ONLINE tibora10t
    ora.tibtst.tibtst_basic.tibtst2.srv application ONLINE ONLINE tibora11t
    ora.tibtst.tibtst_smap.cs application ONLINE ONLINE tibora10t
    ora.tibtst.tibtst_smap.tibtst1.srv application ONLINE ONLINE tibora10t
    ora.tibtst.tibtst_smap.tibtst2.srv application ONLINE ONLINE tibora11t

    • Paul Elbow says:

      Hi Leighton,
      I don’t have 10g clusterware handy right now, but my recollection is that there wasn’t a “type” or “state details” in the 10g (all types were “application”). I could interpret the resource name to specify a “pseudo type” in the type field that aligns as closely as possible with the 11g version- I will look to do this in the near future. I don’t think there is anything to be done with the “state details” in 10g that wouldn’t be intrusive. I will double-check when I have access to 10g clusterware.

  6. newuser says:

    this script is not working for me in solaris 10 . giving error like follows ,

    Resource Name Resource Type Target State Node State Details
    —————————————- ————– ———— ———— ————— —————
    awk: syntax error near line 83
    awk: illegal statement near line 83
    awk: syntax error near line 84
    awk: illegal statement near line 84

    can you please help me ?

  7. newuser says:

    I have tested your script on solaris . it is awesome…… Working Fine ……

    Thanks for your quick response…..

  8. Paul Elbow says:

    Here is the latest version of the crsstat script:

    http://blog.enkitec.com/wp-content/uploads/2012/04/crsstat.v1.2.1.zip

    It is updated with the following enhancements and fixes—

    Usage with -h or invalid switch.
    Added fields: cluster/local resource, failure count FC, and restart count RC.
    Added switch (-c) for outputing Cluster and Local resources in different sections.
    Fixed issue with resetting colors back.

  9. Kevin says:

    Thank you for sharing this excellent script. It is much clear than crs_stat on 11gR2.
    One thing that I noticed when issue “crsstat database”, “Resource Name” only show database name instead of instance name. When we need to switch instance back to original node after one node eviction, it is difficult to distinguish instance. Can you explain how can I tell instance name?

    • Paul Elbow says:

      It would take pulling information out of the runtime and static information. The runtime output from crsctl status resource -v contains an “ID” which identifies the resource. If you then get the static output from crsctl status resource -p (or -f), the “USR_ORA_INST_NAME” is the instance name for the corresponding “ID”. I like the idea and can look at adding this, but not sure when I might get a chance to get the code in the script.

  10. Uwe Hesse says:

    Very useful script – thanks a lot for sharing!

  11. Uwe Hesse says:

    Just one thing: It seems to me that the Resource Type is shown in the opposite way: e.g.. a Scan Listener is a Cluster Resource, but it shows as L

    • Paul Elbow says:

      The output is just showing the Cluster versus Local as defined by Oracle– similar to the division as shown when doing a “crsctl show resource -t”.

      • Uwe Hesse says:

        No it doesn’t. At least not the version that I downloaded today in the afternoon. crsctl stat res -t shows correctly e.g. a Scan Listener as CLUSTER resource whereas your script shows it with the flag L which stands – as I suppose – for LOCAL resource. In other words it shows this in the opposite (and therefore incorrect) way than crsctl stat res -t does. Although this can be easily corrected, I suggest that you fix that in the script that you offer for download.

        • Paul Elbow says:

          You are correct, the one offered for download is showing exactly opposite of what it should. I will get an updated version out there at my earliest opportunity.

  12. Arup Nanda says:

    Excellent, excellent tool.

    I tweaked just a little to remove the white-on-black display forced by the printf command. We use black on white terminals and I wanted the output to match it. It’s just a personal preference.

  13. Arup Nanda says:

    I further modified the script to have another flag “-o” to show only those resources that are supposed to be online but offline now. This is very useful when you want to quickly find out the resources needing attention. The -o flag also works with search string. For instance:

    crsstat -o : shows all the resources offline but target=online
    crssat -o diskgroup : same output but for diskgroups only

  14. Paul Elbow says:

    Here is the latest version of the crsstat script:

    http://blog.enkitec.com/wp-content/uploads/2012/07/crsstat.v1.2.2.zip

    The bug noted by Uwe Hesse: local resources were marked as “Cluster Resources” or with “C” and cluster resources were marked as “Local Resources” or with “L” has been corrected in version 1.2.2. Resources now show the correct indicator for Cluster or Local.

  15. Robert says:

    Hi Paul,

    thanks for the sharing.
    I got an error out when run the script under a normal directory,
    root # pwd
    /orabackup/workarea/yl
    root # ksh crsstat.sh
    crsstat.sh[125]: ./crsctl: not found
    crsstat.sh[128]: ./crsctl: not found
    The CRS or HAS version could not be determined.
    Commands to query for either failed.
    Exiting….

    seems the error is with the code below:

    CSSD_DIR=`dirname $( ps -eo args | grep ocssd.bin | grep -v grep | ${AWK} ‘{print $1}’)`
    if [[ ! -z ${CSSD_DIR} || ${CSSD_DIR} == "." ]] ; then
    CSSD_DIR=$(dirname $( ps -eo args | grep hasd.bin | grep -v grep | ${AWK} ‘{print $1}’))
    fi

    here is a trace with ksh -x

    + dirname /oracle/product/10.2.0/crs_1/bin/ocssd.bin
    CSSD_DIR=/oracle/product/10.2.0/crs_1/bin
    + [[ ! -z /reuters/app/oracle/product/10.2.0/crs_1/bin ]]
    + + ps -eo args
    + grep hasd.bin
    + /usr/bin/nawk {print $1}
    + grep -v grep
    + dirname
    CSSD_DIR=.
    + [[ ! -z . ]]
    + + /usr/bin/nawk {print substr( $1, 1, length( $1)-4)}
    + echo .
    GRID_HOME=
    + + ./crsctl query crs softwareversion
    crsstat.sh[125]: ./crsctl: not found
    retval=

    Could you please help on this?

    • Paul Elbow says:

      Hi, The directory from which the script runs is not important; however, CRS/HAS must be running for it to determine the Grid Home automatically. Alternatively, you could manually set the location and version of CRS/HAS:

      cd {location of crsstat script}
      vi crsstat.env
      and put in the following 2 variables:

      GRID_HOME=/u01/app/11.2.0.3/grid
      GRID_VER=11.2

      Hope this helps…

  16. David A. Riskind says:

    Any plans to add an option to parse output of crsctl version 12c? I really appreciate the formatting of the tool by the way!

  17. Rajee says:

    Hi
    I tried running the script on HP-UX and getting the error as below (even after creating the env file in the same directory)

    ./crsstat.v1.2.2[112]: syntax error at line 113 : `${env_file}’ unexpected.. Please advise

  18. Jorge Donato says:

    Paul,

    awesome script, thank you very much for sharing this one.

    Jorge Donato

  19. Michael Sesuraj says:

    Hi Paul,
    Greate script, work like a Gem. We have couple of Exadata with 11.2.0.4, crs_stat no longer supported, this script does more than crs_stat did in the past.

    Well done!

    Michael Sesuraj
    Australia.

  20. Sanjay says:

    Sorry found that it shows RED if the Target is ONLINE but state is OFFLINE.

    Sanjay

Trackbacks

Check out what others are saying about this post...


Speak Your Mind

Tell us what you're thinking...
and oh, if you want a pic to show with your comment, go get a gravatar!