Collect Diagnostic Data during a database hang.

The following set of diagnostic data collection routines must be run when database is not responding to TNS connection request and queries to views like v$session, etc hang.

The symptoms of this hang are
  • Unable to connect using tnsnames to local instance. E.g. if ssh on node 01A, then cannot sqlplus to instance hosted on that server. E.g. sqlplus username/password@connectstring
  • Unable to connect using tnsname .e.g sqlplus username/password@connectstring_active
  • Bequeath connections are successful. E.g. sqlplus / as sysdba
  • Querying certain V$ views causes the session to hang.

Diagnostic Data set 1:
  SQL> set lines 250 pages 1000
  SQL> spool xle_xkclfx.out
  SQL> select * from x$le;
  SQL> select * from x$kclfx;
  SQL> exec dbms_lock.sleep(60);
  SQL> select * from x$le;
  SQL> select * from x$kclfx;
  SQL> exec dbms_lock.sleep(60);
  SQL> select * from x$le;
  SQL> select * from x$kclfx;
  SQL> exec dbms_lock.sleep(60);
  SQL> select * from x$le;
  SQL> select * from x$kclfx;
  SQL> exec dbms_lock.sleep(60);
  SQL> select * from x$le;
  SQL> select * from x$kclfx;
  SQL> spool off;

Diagnostic Data set 2:
  Fetch the OSPID for all the running lms processes using the following command
$ ps -ef | grep -i lms | grep -v grep  | awk '{print $2}'

   For each of the PID returned by the above command run the following
  SQL> oradebug setospid <OS PID of LMS process>
  SQL> oradebug unlimit
  SQL> oradebug tracefile_name
  SQL> oradebug dump errorstack 3
  SQL> !sleep 180s
  SQL> oradebug dump errorstack 3
  SQL> !sleep 180s
  SQL> oradebug dump errorstack 3
  SQL> exit;

Diagnostic Data set 3:
  SQL> oradebug setmypid
  SQL> oradebug tracefile_name
  SQL> oradebug unlimit
  SQL> oradebug -g all hanganalyze 3
  SQL> oradebug -g all dump systemstate 266
  SQL> !sleep 180s
  SQL> oradebug -g all dump systemstate 266
  SQL> !sleep 180s
  SQL> oradebug -g all dump systemstate 266

Diagnostic Data set 4:
Run the following to obtain the PID of the LMS processes
  $ ps -ef | grep -i lms | grep -v grep  | awk '{print $2}'

  For each of the PID returned by the above command
    SQL> oradebug setospid <OS PID of LMS process>
  SQL> oradebug unlimit
  SQL> oradebug tracefile_name
  SQL> oradebug event 10708 trace name context forever, level 15
  SQL> !sleep 180s

Reboot the cluster now.

Post reboot the files collected by in the above steps must be uploaded to Oracle.

Comments

Post a Comment

Popular posts from this blog

Recover Standby database from ORA-00332: archived log is too small

When a standby database crashes and upon instance recovery gets stuck with error ORA-00332: archived log is too small - may be incompletely archived, then following set of actions must be carried out to restore the service. 1. Check that the instance is in MOUNTED. (select status from v$instance) 2. If not mounted, then start the database in mounted mode SQL> startup mount; 3. Search the alert log to identify the archive log that is corrupt,  Sample lines from the alert log Errors in file /u01/app/oracle/diag/rdbms/fxfgsby/FXFGSBY1/trace/FXFGSBY1_ora_19463.trc: ORA-00332: archived log is too small - may be incompletely archived ORA-00334: archived log: '+FLASH/fxfgsby/archivelog/2013_08_19/thread_1_seq_173831.5998.823855747' Shutting down recovery slaves due to error 332 4. In the above example, the corrupt log file belongs to Thread#=1 and Sequence#=173831 5. Log on to the Primary database cluster and take a copy of the file from ASM onto the local file system. Not...
Read more

Check Active Dataguard Replication Status

In order to check if ADG is up-to-date, one has to tail the alert log on both the Primary and Physical Standby databases, followed by a manual check to look for the log status of (in-transit) against each of the threads. While this method is perfect it can be bit tedious to perform. You can now do the same using one of the attached SQL from the Primary database. Deployment and usage steps - Open a vi editor on the Primary database. - Copy of the below SQL into the editor - Complete the  database link creation command based on the database    you are checking. - Save the text file as check_adg_status.sql - execute the sql file as sys user. e.g. sqlplus -s  / as sysdba @check_adg_status.sql  Note : These commands cannot be run from the Physical standby as the query relies on using a temporary database link that cannot be created on a read-only database. Troubleshooting 1. Some database clusters may report the following error when invoked   ...
Read more

Top CPU consuming oracle sessions

Use the shell script from  here  to obtain the details of the Oracle sessions using high CPU.  Usage: Log in as oracle user and navigate to the folder where the script has been copied into. Do not forget to grant execute permissions to the script prior to invoking it. $ ./top_oracle_sessions.sh [cpu_threshold]  e.g.1  $ ./top_oracle_sessions.sh 90       This will gather information if overall CPU usage is above 90% e.g.1  $ ./top_oracle_sessions.sh        This will gather information if overall CPU usage is above 50%. This is the default setting. Notes: Use the attached script to obtain the details of the Oracle sessions using high CPU. The script works by follows Takes 2 snapshots using Linux "top" command with a interval of 2 seconds between snaps Uses the data from the second snapshot to determine if the overall CPU usage is above a set threshold. The threshold is a conf...
Read more