Category Archives: ADDM

Oracle 12c – Perform Emergency Monitoring and Real-time ADDM

Emergency Monitoring feature of EM Cloud allows for DBA to connect to non-responding database by way of proprietary resources for diagnosis of problem.  The EM agent connects directly to SGA, bypassing the SQL retrieval layer.   The page displays ASH data and Hang Analysis table, displaying blocking sessions and allows for killing this blocks.  Emergency Monitoring page is accessible from Performance Menu of EM Cloud.  EM Express has no diagnostic connection.

 

Real-Time ADDM – introduced in EM Cloud 12c, much of functionality is available in EM Express.  ADDM is to help with analysis of problem which cause unresponsive behavior without restarting DB.  ADDM checks for problems based on set of predefined criteria, then suggests solutions.

Two Ways ADDM connects:

  1. Normal Connection – normal JDBC connection, used for extensive performance analysis.
  2. Diagnostic Connection – latchless connection for hung situations, where JDBC connection is not available.

 

Differences between REAL-Time ADDM and ADDM

  1. Real-Time ADDM performs similar analysis to ADDM by does not use AWR snapshots, rather it uses ASH data from SGA.
  2. Real-Time ADDM using in-memory real time performance data.
  3. Real-Time ADDM analysis is automatic when problem is detected.
  4. RT ADDM scans every 3 seconds using MMON process.
  5. RT ADDM does not use lock or latch.
  6. Things that would trigger RT ADDM analysis:
    1. High load – active session > 3x number of CPU
    2. I/O bound – I/O of active sessions concentrated on a single block read.
    3. CPU bound – Active sessions > 10% of all sessions and CPU > 50% utilization.
    4. Over-allocated memory – memory allocated > 95% of physical memory.
    5. Interconnect bound – Single block continually used.
    6. Session limit – near 100%
    7. Process limit – near 100%
    8. Hung session – hung session > 10% of all sessions.
    9. Deadlock detected – Any deadlock.
  7. MMON slave process will store ADDM reports created by AWR, which is accessible by DBA_HIST_REPORTS view.
  8. RT ADDM controls level analysis resource usage by:
    1. Duration between reports – RT ADDM will not produce new report if it has been less than 5 min since last report.
    2. Oracle RAC control – Only one RAC instance can create ADDM report at any given time.
    3. Repeated Triggers – A unique issue can only trigger an analysis if the impact is 100% or greater previous impact within 45 min.
    4. Newly identified issues – If new issue is identified, an analysis will be performed regardless of active session load.

 

 

Larry Catt

OCP

Oracle 12c – Monitor performance

There are several methods to monitor database performance, but EM is probably the most frequent used today.  Oracle Enterprise Manager Cloud Control is designed to manage the entire Oracle environment: database, server, middleware, and more.  OEM Cloud control requires additional licensing, however EM Express is provide free with Enterprise editions.

 

EM Express can be used to manage a single Oracle 12c database and it is actually built into the database.  EM Express is a light weight management console and provides basic administrative tasks with minimal impact to database operations.  EM Express has no background processes and it gathers information from already existing database stores.  The Performance Hub of Express provides the following abilities:

  1. ASH Analytics.
  2. SQL Monitor
  3. ADDM
  4. Workload metrics.
  5. Resource usage
  6. See real-time and historic data.
  7. In historic mode AWR (Automatic Workload Repository) data is used.
  8. Performance Hub tabs are
    1. Summary – real-time mode
    2. Activity – ASH analytics both real-time and historic.
    3. Workload – info about Top SQL
    4. RAC – RAC specific metrics.
    5. Monitor SQL – current and historic SQL executions.
    6. ADDM – ADDM reports both real-time and historic
    7. Current ADDM findings – performance analysis of last 5 minutes.

 

 

Larry Catt

OCP

Oracle 12c – Generate ADDM

In Oracle 12c, one needs to generate ADDM reports for comparison of Normal database operations and Periods performance is thought to be sub-par.   This article covers the generation of ADDM for normal operation.   There are three methods to generate ADDM reports:  EM Cloud control, ADDMRPT.SQL script, and DBMS_ADVISOR package.

 

EM Cloud Control:

  1. Logon to EM.
  2. Click the Advisor Central Link.
  3. Select ADDM link.
  4. From here you can start/stop snapshots; create ADDM tasks and display results.
  5. You can view ADDM analysis from the Performance Analysis section of the Home page.

 

ADDMRPT.SQL

  1. This script is located under ORACLE_HOME/rdbms/admin directory and can be executed at any time.
  2. Logon to SQLPLUS with sysdba privileges.

 

[oracle@linux2 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Thu Jun 29 09:04:21 2016

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

 

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 – 64bit Production

With the Partitioning, OLAP, Advanced Analytics, Real Application Testing

and Unified Auditing options

SQL>

 

  1. Execute the script and select snapshots to generate reports.

 

SQL> @$ORACLE_HOME/rdbms/admin/addmrpt.sql

 

Current Instance

~~~~~~~~~~~~~~~~

   DB Id    DB Name      Inst Num Instance

———– ———— ——– ————

 1457738340 ORCL                1 orcl

 

Instances in this Workload Repository schema

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

   DB Id     Inst Num DB Name      Instance     Host

———— ——– ———— ———— ————

* 1457738340        1 ORCL         orcl         linux2.local

                                                domain

Using 1457738340 for database Id

Using          1 for instance number

 

 

Specify the number of days of snapshots to choose from

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Entering the number of days (n) will result in the most recent

(n) days of snapshots being listed.  Pressing <return> without

specifying a number lists all completed snapshots.

 

Listing the last 3 days of Completed Snapshots

 

                                                        Snap

Instance     DB Name        Snap Id    Snap Started    Level

———— ———— ——— —————— —–

orcl         ORCL               388 27 Jun 2016 00:00      1

                                389 27 Jun 2016 01:00      1

                                390 27 Jun 2016 02:00      1

Specify the Begin and End Snapshot Ids

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Enter value for begin_snap:

 

 

  1. Enter a starting snapshot id:

 

Specify the Begin and End Snapshot Ids

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~

Enter value for begin_snap: 389

Begin Snapshot Id specified: 389

 

Enter value for end_snap:

 

 

  1. Enter a Ending snapshot id:

 

Enter value for end_snap: 390

End   Snapshot Id specified: 390

 

 

  1. Enter a report name or use default with enter.

 

Specify the Report Name

~~~~~~~~~~~~~~~~~~~~~~~

The default report file name is addmrpt_1_389_390.txt.  To use this name,

press <return> to continue, otherwise enter an alternative.

 

Enter value for report_name:

 

  1. The report will be generated and placed in current directory.

 

End of Report

Report written to addmrpt_1_389_390.txt

SQL>

 

 

DBMS_ADVISOR package

 

  1. Logon to SQLPLUS with sysdba privileges.

 

[oracle@linux2 ~]$ sqlplus / as sysdba

SQL*Plus: Release 12.1.0.2.0 Production on Thu Jun 29 09:04:21 2016

Copyright (c) 1982, 2014, Oracle.  All rights reserved.

 

Connected to:

Oracle Database 12c Enterprise Edition Release 12.1.0.2.0 – 64bit Production

With the Partitioning, OLAP, Advanced Analytics, Real Application Testing

and Unified Auditing options

SQL>

 

  1. Create a ADDM task.

 

Begin

  DBMS_ADVISOR.create_task (

    advisor_name      => ‘ADDM’,

    task_name         => ‘389_390_AWR_SNAPSHOT’,

    task_desc         => ‘Advisor for snapshots 389 to 390.’);

End;

/

 

  1. Set start and end parameters for task.

 

Begin

  — Set the start and end snapshots.

  DBMS_ADVISOR.set_task_parameter (

    task_name => ‘389_390_AWR_SNAPSHOT’,

    parameter => ‘START_SNAPSHOT’,

    value     => 389);

 

  DBMS_ADVISOR.set_task_parameter (

    task_name => ‘389_390_AWR_SNAPSHOT’,

    parameter => ‘END_SNAPSHOT’,

    value     => 390);

End;

/

 

  1. Execute task

 

Begin

  — Execute the task.

  DBMS_ADVISOR.execute_task(task_name => ‘389_390_AWR_SNAPSHOT’);

End;

/

 

  1. View report.

 

— Display the report.

SET LONG 1000000 LONGCHUNKSIZE 1000000

SET LINESIZE 1000 PAGESIZE 0

SET TRIM ON TRIMSPOOL ON

SET ECHO OFF FEEDBACK OFF

 

SELECT DBMS_ADVISOR.get_task_report(‘389_390_AWR_SNAPSHOT’) AS report

FROM   dual;

 

  1. This completes generation for ADDM reports.

 

 

 

 

 

Larry Catt

OCP

Oracle 12c – Explain ADR Enhancements

Oracle 12c Automatic Diagnostic Repository (ADR) provides files to store diagnostic information about the health of the database.   This release has added two new log files to add in administration of the RDBMS suite:  DDL Log File and Debug Log File.

 

NEW DDL Log file – The new DDL log file has the same format and behavior as the alert log.  When the ENABLE_DDL_LOGGING parameter is set to TRUE, DDL statements are written out to the DDL log file.  Two log files are maintained:  XML and plain text.  Log files are stored in the /log/ddl directory of the ADR home and are included in IPS incident packages.

 

Debug Log – the debug log records unusual incidents that do not impact normal operations.  They are occurrences which do not warrant an entry in alert log or incident report.  The debug log has the same format as alert log.

 

 

Larry Catt

OCP

Oracle 12c – Analyze and identify performance issues

The Oracle Automatic Database Diagnostic Monitor (ADDM) is a utility which automatically detects and reports performance issues.  ADDM is based on snapshots take by AWR (Automatic Workload Repository). This data is analyzed and displayed as ADDM findings on Database Home Page of EM.   The DBMS_ADDM package can execute ADDM reports outside of EM but the user executing DBMS_ADDM must have ADVISOR privilege.  ADDM allows administrators to quickly identify performance problems which may not be readily noticeable.  Every ADDM finding will have one or more recommendations to reduce the impact.

 

AWR takes snapshots are taken once an hour and stored for 8 days by default.  It is recommended that snapshots are stored for 30 days.  A new snapshot will result in an ADDM analysis being performed with the following steps:

 

  1. Locates the root cause of performance issue.
  2. Provides recommendation to correct.
  3. Quantifies the expected benefits.
  4. Identifies areas where no action is needed.

 

ADDM analysis is done top down through DB time statistics this is an iterative process and once one problem is solved another will become the bottleneck.  ADDM recommendation may include:

 

  1. Hardware changes – add CPU, memory, change I/O subsystem.
  2. Database configuration – change init parameters.
  3. Schema changes – could include hash partitioning table or index or using ASSM automatic segment space management.
  4. Application changes – cache options or use of bind variables.
  5. Using other advisors – Running SQL Tuning Advisor or Segment Advisor.

 

ADDM is enabled in database by default and controlled by init parameters CONTROL_MANAGEMENT_PACK_ACCESS.    ADDM is enabled when CONTROL_MANAGEMENT_PACK_ACCESS parameter is set to DIAGNOSTIC+TUNING (default) or DIAGNOSTIC.    ADDM is disabled when CONTROL_MANAGEMENT_PACK_ACCESS is setting to NONE.

Additionally the initialization parameter of STATISTICS_LEVEL must be set to TYPICAL (default) or ALL to enable ADDM, but if it is set to BASIC ADDM will be disable.

 

The parameter DBIO_EXPECTED of the DBMS_ADVISOR package is required to perform analysis of I/O performance. It is set to 7000 microseconds with the following package call:

 

EXECUTE DBMS_ADVISOR.SET_DEFAULT_TASK_PARAMETER(‘ ADDM’, ‘DBIO_EXPECTED’, 7000);

 

An ADDM analysis results come in one of three classes:

  • PROBLEM – root cause of database performance issues.
  • SYMPTOM – finding that’s info leads to one or more problem findings.
  • INFORMATION – Findings that are used to report areas of system that do not have performance issues.

 

If one problem has multiple findings, ADDM may report them separately.   Problem will be shown largest impact first.

 

Larry Catt

OCP

Oracle 12c – Diagnose performance issues using ASH enhancements

Conventional ADDM uses AWR which are based on reports taken during intervals.  Some changes will not appear on this conventional ADDM because changes where not great enough to trigger new AWR analysis.  This is also not a real-time comparison of performance.   ASH Active Session History is real-time and had the following characteristics:

 

  1. Active Session History is a collection of real-time data stored in the SGA based on active session
  2. Can be viewed via the V$ACTIVE_SESSION_HISTORY view.
  3. EM Cloud has the ability to run ASH reports from:
    1. HOME PAGE à Performance Menu/Home à Logon with admin privilege à Under Average Active Sessions click ASH Reports à Enter Start and Stop time of report à click Generate.
  4. Reports are placed under Run ASH Report Page.
  5. ASH Reports are divided into the sections
    1. Top Events – top wait events
    2. Load Profile – Describes load during report period.
    3. Top SQL – order by amount of resource consumed by SQL statements.
    4. Top Sessions – order by sessions waiting for wait events.
    5. Top DB Objects/Files/Latches – order by top objects being used/accessed.
    6. Activity Over Time – Shows picks and values in performance over reporting period.
  6. EM Cloud also includes new ASH Analytics page, providing graphical view of ASH data.

 

Larry Catt

OCP