Oracle Clusterware Architecture: The Foundation of Oracle RAC
In the previous article, we introduced the main components of Oracle RAC and saw how multiple database instances work together to provide high availability and scalability.
At the center of every RAC environment is Oracle Clusterware. While the database instances process SQL statements and user transactions, Clusterware is responsible for keeping the cluster healthy. It continuously monitors nodes, starts and stops resources, detects failures, and automatically recovers services whenever possible.
If Oracle RAC is the engine, Oracle Clusterware is the operating system that keeps every component running in the correct order.
In this article, we’ll explore the architecture of Oracle Clusterware, understand its main processes, and learn how they work together.
What is Oracle Clusterware?
Oracle Clusterware is a collection of software components that manage all resources within an Oracle RAC cluster.
Its responsibilities include:
- Managing cluster membership
- Starting and stopping cluster resources
- Detecting failures
- Restarting failed resources
- Managing node communication
- Coordinating failover
- Monitoring cluster health
Unlike the Oracle database, Clusterware starts before any database instance.
Oracle Clusterware Architecture
A simplified Oracle RAC cluster looks like this:
Oracle RAC Cluster
+-------------------------------------------------------+
| Oracle Clusterware |
| |
| OHASD CRSD CSSD EVMD ORAAGENT ASM |
+-------------------------------------------------------+
| |
| |
Shared Storage Private Interconnect
Every node runs its own Clusterware stack.
Oracle High Availability Services (OHAS)
The first process to start is Oracle High Availability Services (OHAS).
It is responsible for starting all other Clusterware components.
Startup sequence:
Linux Boot
|
v
OHASD
|
+--------------------------+
| |
v v
CSSD CRSD
|
v
EVMD
|
v
ASM
|
v
Database
If OHASD is not running, none of the cluster services will be available.
Main Clusterware Components
The Clusterware stack consists of several important daemons.
| Process | Purpose |
|---|---|
| OHASD | Starts and monitors Clusterware |
| CSSD | Cluster synchronization |
| CRSD | Resource management |
| EVMD | Event notification |
| ORAAGENT | Resource execution |
| ASM | Shared storage management |
Each daemon has a dedicated responsibility.
OHASD (Oracle High Availability Services Daemon)
OHASD is the parent process for the Clusterware stack.
Responsibilities:
- Starts Clusterware
- Restarts failed Clusterware processes
- Starts ASM
- Starts listeners
- Starts VIPs
- Starts database instances
Check its status:
ps -ef | grep ohasd
Example:
root 2315 1 0 ? ohasd.bin
CSSD (Cluster Synchronization Services Daemon)
CSSD is one of the most critical Clusterware processes.
Its responsibilities include:
- Cluster membership
- Heartbeat monitoring
- Voting Disk management
- Split-brain prevention
Communication occurs through:
- Private interconnect
- Voting Disks
Simplified heartbeat:
Node1 <-------------> Node2
Heartbeat
Voting Disk
If a node stops sending heartbeats, CSSD determines whether it should remain part of the cluster.
Split-Brain Protection
One of CSSD’s most important functions is preventing a split-brain scenario.
Split-brain occurs when two nodes lose communication but both continue operating independently.
Example:
Network Failure Node1 X----------------X Node2 Both think they own the database
Without protection, this could corrupt shared data.
Voting Disks help determine which node (or group of nodes) remains active. Nodes that lose quorum are evicted from the cluster to protect data integrity.
CRSD (Cluster Ready Services Daemon)
CRSD manages cluster resources.
Examples of resources:
- ASM
- Database instances
- Listeners
- Services
- VIP addresses
- SCAN listeners
CRSD continuously checks whether these resources are online.
If a database instance stops unexpectedly:
Database Instance
DOWN
|
CRSD detects failure
|
Restart Instance
This automatic recovery reduces downtime.
Oracle Agent (ORAAGENT)
ORAAGENT executes the actions requested by CRSD.
Typical actions include:
- Start a listener
- Stop ASM
- Restart a database
- Relocate a service
Think of CRSD as the manager and ORAAGENT as the technician carrying out the work.
EVMD (Event Manager Daemon)
EVMD handles cluster events.
Examples:
- Node joins
- Node leaves
- Resource failures
- Resource relocation
These events can be used for:
- Notifications
- Monitoring tools
- Troubleshooting
Oracle Local Registry (OLR)
Each node maintains an Oracle Local Registry (OLR).
Unlike the Oracle Cluster Registry (OCR), which is shared across the cluster, the OLR contains node-specific information required during startup.
Node1
OLR
|
Starts OHAS
Without the OLR, Clusterware cannot initialize correctly on that node.
Oracle Cluster Registry (OCR)
The OCR stores cluster-wide configuration.
Examples include:
- Cluster nodes
- Databases
- Services
- VIPs
- Listeners
- ASM resources
Whenever a resource is added or modified, the OCR is updated.
Cluster Resources
Everything managed by Clusterware is called a resource.
Common resources include:
ora.asm ora.LISTENER.lsnr ora.DATA.dg ora.db19c.db ora.scan1.vip ora.node1.vip
View resources:
crsctl stat res -t
Example output:
-------------------------------------------------------------------------------- Name Target State Server -------------------------------------------------------------------------------- ora.asm ONLINE ONLINE node1 ora.DATA.dg ONLINE ONLINE node1 ora.db19c.db ONLINE ONLINE node1 ora.LISTENER.lsnr ONLINE ONLINE node1 ora.node1.vip ONLINE ONLINE node1
This command is one of the first tools a RAC administrator should know.
Cluster Startup Sequence
When a node starts, Clusterware follows a defined order:
Linux | OHASD | CSSD | CRSD | EVMD | ASM | Listener | Database | Services
Each component depends on the previous one being available.
Checking Cluster Status
Verify the overall cluster:
crsctl check cluster
Example:
************************************************************** node1: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online node2: CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online **************************************************************
Check the local stack:
crsctl check crs
Example:
CRS-4638: Oracle High Availability Services is online CRS-4537: Cluster Ready Services is online CRS-4529: Cluster Synchronization Services is online CRS-4533: Event Manager is online
Common Clusterware Problems
CSSD Offline
Possible causes:
- Voting Disk unavailable.
- Private interconnect failure.
- Storage issues.
CRSD Does Not Start
Possible causes:
- OCR corruption.
- OCR inaccessible.
- Disk group not mounted.
Resources Continuously Restart
Possible causes:
- Incorrect dependencies.
- Listener configuration issues.
- Database startup errors.
Node Evictions
Possible causes:
- Private interconnect latency.
- Network packet loss.
- Storage delays.
- CPU starvation.
Node eviction protects the cluster from data corruption and should be investigated carefully.
Best Practices
- Use redundant private interconnects.
- Protect OCR and Voting Disks with ASM redundancy.
- Monitor Clusterware logs regularly.
- Review resource status after maintenance.
- Keep Grid Infrastructure and Oracle Database Release Updates aligned.
- Test failover procedures in a non-production environment before implementing major changes.
DBA Tip
One of the most useful habits when troubleshooting RAC is to start at the Clusterware layer.
Instead of immediately checking the database, verify the health of the cluster:
- Is OHAS running?
- Is CSSD online?
- Is CRSD online?
- Are ASM disk groups mounted?
- Are cluster resources online?
- Only then investigate the database instance.
This top-down approach often saves time because many database startup issues are actually caused by underlying Clusterware or storage problems.
In Oracle RAC, the database depends on Clusterware. If the Clusterware stack is unhealthy, the database is only showing the symptom—not the root cause.
Conclusion
Oracle Clusterware is the backbone of every RAC environment. It manages node membership, monitors critical resources, detects failures, and automates recovery to keep the cluster available.
Understanding the responsibilities of OHASD, CSSD, CRSD, EVMD, ORAAGENT, the OLR, and the OCR gives you the foundation needed to troubleshoot RAC effectively and understand why resources start, stop, or fail over.
Google+
Facebook
