Part 17 focused on GES, locks, blocking sessions, and deadlocks.
Now we move to the other half of RAC:
GCS — Global Cache Service
If GES is primarily about global locking, GCS is about global access to database blocks.
This is one of the most important areas to master if you want to troubleshoot Oracle RAC at an advanced level.
1. The Core RAC Problem
Imagine a 2-node RAC:
RAC
|
+--------+--------+
| |
ORCL1 ORCL2
| |
SGA SGA
| |
Buffer Cache Buffer Cache
Both instances have their own buffer cache.
But they access the same database.
So what happens when:
ORCL1 needs a block
that is currently cached and modified in:
ORCL2
Oracle doesn’t simply read the block from disk.
Instead, RAC coordinates the block between instances through Cache Fusion.
2. Cache Fusion
The basic idea:
ORCL1 Buffer Cache
|
| Request block
↓
Interconnect
|
↓
ORCL2 Buffer Cache
The block can be transferred through the RAC interconnect.
This is called:
Cache Fusion
It is one of the fundamental technologies that makes Oracle RAC work.
3. Why Cache Fusion Matters
Without Cache Fusion, RAC would have to constantly write blocks to disk and reread them.
That would be extremely expensive.
Instead:
Instance 1
|
| memory-to-memory
↓
RAC Interconnect
|
↓
Instance 2
The block can move directly between buffer caches.
This is much faster than:
Buffer Cache
↓
Disk
↓
Disk
↓
Buffer Cache
But Cache Fusion isn’t free.
Heavy cross-instance block transfers can become a significant source of RAC overhead.
4. Local vs Remote Block Access
Suppose SQL running on ORCL1 needs block 100.
Local
ORCL1 | ↓ Buffer Cache | ↓ Block 100
Very efficient.
Remote
ORCL1 | ↓ Interconnect | ↓ ORCL2 Buffer Cache | ↓ Block 100
Now the request involves:
- GCS
- RAC interconnect
- Remote instance
- Global resource coordination
This is where GC wait events appear.
5. The Most Important GC Waits
For Oracle RAC performance tuning, memorize these:
gc cr request gc current request gc current block busy gc cr block busy gc buffer busy
You don’t need to memorize them blindly.
Understand what they mean.
6. gc cr request
Example:
gc cr request
This generally means a session is requesting a consistent-read version of a block through the global cache.
Think:
SQL ↓ Needs CR block ↓ Block is remote ↓ Request through Cache Fusion
High values can indicate significant remote consistent-read activity.
7. gc current request
This is particularly important.
It generally indicates that the session is requesting the current version of a block from another RAC instance.
Conceptually:
ORCL1 | | Need CURRENT block ↓ ORCL2 | | Owns current version ↓ Transfer through interconnect
Heavy gc current request activity deserves investigation.
8. CR vs CURRENT
This distinction is essential.
CR = Consistent Read
Used when Oracle needs a version of the block consistent with the query’s read-consistency requirements.
CURRENT
The current version of the block.
Simplified:
SELECT ↓ CR block UPDATE ↓ CURRENT block
This is simplified, but it’s a useful mental model.
9. gc current block busy
This event deserves particular attention.
It indicates that the requested current block is not immediately available because the remote instance is busy with that block.
Conceptually:
ORCL1 | | Request current block ↓ ORCL2 | | Block busy ↓ Wait
Possible causes include:
- Hot blocks
- Heavy concurrent modification
- High cross-instance activity
- Block contention
10. gc cr block busy
Similar concept, but for a consistent-read request.
gc cr block busy
can indicate that the requested CR block isn’t immediately available.
Again, don’t jump directly to:
“The interconnect is slow.”
The interconnect may be perfectly healthy.
The workload itself may be generating excessive block contention.
11. gc buffer busy
This one often appears in RAC investigations.
A session wants to access a buffer, but the buffer is currently busy with another operation.
It can be associated with:
- Hot blocks
- Concurrent access
- Block transfers
- Cache Fusion activity
You need to investigate the object and SQL rather than treating the event as the root cause.
12. The Most Important Principle
If you see:
gc current request
don’t immediately conclude:
RAC network problem.
Instead ask:
Why does this session need a block from another instance?
That’s the expert question.
13. Example
Imagine:
ORCL1 Sales application
and:
ORCL2 Batch application
Both update:
ORDERS
The workload becomes:
ORCL1 | | UPDATE ORDERS ↓ Block A ORCL2 | | UPDATE ORDERS ↓ Block A
The block keeps moving between instances.
This is sometimes called:
Block pinging
14. Block Pinging
Conceptually:
ORCL1 owns block
↓
ORCL2 requests block
↓
Block moves
↓
ORCL1 requests block
↓
Block moves back
↓
ORCL2 requests block
↓
Block moves again
You get:
ORCL1 ↔ ORCL2
repeatedly.
This can cause significant RAC overhead.
15. The Real Problem Is Often Workload Placement
Suppose:
ORCL1 → SALES ORCL2 → SALES
and both instances heavily update the same objects.
You may have:
SALES
|
+-----+-----+
| |
ORCL1 ORCL2
| |
UPDATE UPDATE
| |
+-----+-----+
|
Cache Fusion
A better architecture might be:
ORCL1 SALES ORCL2 REPORTING
depending on the application’s requirements and service architecture.
This is why RAC services are a performance tool, not just a connection-routing feature.
16. Cache Fusion and Services
A good RAC design tries to keep workloads that frequently access the same data closer together.
For example:
SERVICE: SALES_APP Preferred: ORCL1 Available: ORCL2
and:
SERVICE: REPORTING Preferred: ORCL2 Available: ORCL1
The objective isn’t simply:
“50% of connections on each node.”
The objective is:
Efficient workload placement.
17. Measuring GC Waits
Start with:
SELECT inst_id,
event,
total_waits,
time_waited_micro / 1000000 AS time_waited_sec
FROM gv$system_event
WHERE event LIKE 'gc %'
ORDER BY time_waited_micro DESC;
This gives you an overview of GC waits by instance.
18. Current GC Waits
During an active incident:
SELECT inst_id,
event,
COUNT(*) AS sessions
FROM gv$session
WHERE status = 'ACTIVE'
AND event LIKE 'gc %'
GROUP BY inst_id, event
ORDER BY sessions DESC;
Example:
INST EVENT SESSIONS ---- ----------------------- -------- 1 gc current request 12 2 gc current block busy 8 2 gc cr request 15
Now you know which instance is experiencing the most activity.
19. Historical GC Analysis With ASH
If the incident happened earlier:
SELECT instance_number,
event,
COUNT(*) AS samples
FROM dba_hist_active_sess_history
WHERE sample_time BETWEEN
TIMESTAMP '2026-08-08 09:00:00'
AND TIMESTAMP '2026-08-08 09:30:00'
AND event LIKE 'gc %'
GROUP BY instance_number, event
ORDER BY samples DESC;
This is much more useful than looking only at current statistics.
20. Find the SQL Generating GC Waits
Now drill down:
SELECT sql_id,
event,
COUNT(*) AS samples
FROM dba_hist_active_sess_history
WHERE sample_time BETWEEN
TIMESTAMP '2026-08-08 09:00:00'
AND TIMESTAMP '2026-08-08 09:30:00'
AND event LIKE 'gc %'
AND sql_id IS NOT NULL
GROUP BY sql_id, event
ORDER BY samples DESC
FETCH FIRST 30 ROWS ONLY;
Example:
SQL_ID EVENT ----------- ----------------------- 8a12abc gc current request 8a12abc gc current block busy
Now we have a SQL candidate.
21. Find the Service
Add service information:
SELECT service_name,
event,
COUNT(*) AS samples
FROM dba_hist_active_sess_history
WHERE sample_time BETWEEN
TIMESTAMP '2026-08-08 09:00:00'
AND TIMESTAMP '2026-08-08 09:30:00'
AND event LIKE 'gc %'
GROUP BY service_name, event
ORDER BY samples DESC;
Example:
SERVICE EVENT ------------ ----------------------- SALES_APP gc current request BATCH gc current request
Now you can correlate the RAC wait with the workload.
22. Find the Instance
SELECT instance_number,
service_name,
sql_id,
event,
COUNT(*) samples
FROM dba_hist_active_sess_history
WHERE sample_time BETWEEN
TIMESTAMP '2026-08-08 09:00:00'
AND TIMESTAMP '2026-08-08 09:30:00'
AND event LIKE 'gc %'
GROUP BY instance_number,
service_name,
sql_id,
event
ORDER BY samples DESC;
This gives you:
Instance + Service + SQL + Wait
That’s already a very strong diagnostic picture.
23. Find the Object
ASH contains object-related information for many sampled activities.
Start with:
SELECT current_obj#,
COUNT(*) samples
FROM dba_hist_active_sess_history
WHERE sample_time BETWEEN
TIMESTAMP '2026-08-08 09:00:00'
AND TIMESTAMP '2026-08-08 09:30:00'
AND event LIKE 'gc %'
AND current_obj# > 0
GROUP BY current_obj#
ORDER BY samples DESC;
Then resolve the object:
SELECT owner,
object_name,
object_type
FROM dba_objects
WHERE object_id = :object_id;
Now:
GC Wait ↓ SQL ↓ Object
24. Find Hot Objects
Suppose you discover:
OWNER = SALES OBJECT = ORDERS TYPE = TABLE
Then investigate:
- Which SQL modifies it?
- Which instances access it?
- Is it partitioned?
- Is there a hot index?
- Are there sequential inserts?
- Are transactions too large?
- Is batch processing overlapping with OLTP?
25. Hot Index Blocks
Indexes are particularly interesting.
Consider:
ORDER_ID
with continuously increasing values:
100001 100002 100003 100004 ...
A conventional B-tree index can have heavy activity around the right-hand side of the index.
If multiple RAC instances insert concurrently:
ORCL1 → right-most index block ORCL2 → right-most index block ORCL1 → right-most index block ORCL2 → right-most index block
you can create significant contention.
26. But Don’t Automatically Blame the Index
A high index activity doesn’t automatically mean:
“Drop the index.”
The index may be necessary.
You need to determine:
- Is it actually hot?
- Is contention caused by inserts?
- Is the index badly designed?
- Is workload distribution causing the problem?
- Would partitioning help?
- Is the sequence/application design contributing?
27. The Importance of SQL Design
Consider:
UPDATE orders SET status = 'COMPLETE' WHERE customer_id = :1;
If thousands of sessions repeatedly update the same small set of rows or blocks, you can create hot-block contention.
Now compare with a workload that distributes updates across many blocks.
Same table.
Different concurrency pattern.
Therefore:
RAC performance depends heavily on data-access patterns, not just hardware.
28. Global Cache Efficiency
A useful concept is the relationship between:
Logical work
and:
Global cache transfers
A system can have powerful hardware but poor RAC performance if too much work crosses instances.
Think:
Local access
↓
Cheap
Remote block access
↓
More expensive
Repeated remote block pinging
↓
Potentially very expensive
29. AWR RAC Sections
When reading a RAC AWR report, pay particular attention to sections related to:
- Global Cache
- Cluster statistics
- Global Cache Transfer Statistics
- Wait Events
- Instance Activity
- SQL by cluster-related metrics
The exact report layout can vary by Oracle version and report generation mode.
30. Cache Fusion Diagnostic Workflow
When GC waits are high:
GC Wait ↓ Which instance? ↓ Which service? ↓ Which SQL? ↓ Which object? ↓ Which block? ↓ Why remote? ↓ Why frequently transferred? ↓ Can workload be localized?
This is the workflow I recommend.
31. Example Production Incident
Let’s say users report:
“The SALES application becomes slow every day around 18:00.”
AWR shows:
DB Time ↑
Top event:
gc current request
ASH:
INSTANCE 2 SERVICE SALES_APP SQL_ID 8a12abc
Object:
SALES.ORDERSThen you discover:
ORCL1 → nightly batch ORCL2 → sales OLTP
The batch modifies:
ORDERS
while users are simultaneously updating:
ORDERS
Now the picture is:
ORDERS
|
+---------+---------+
| |
ORCL1 ORCL2
| |
BATCH OLTP
| |
+--------+----------+
|
Cache Fusion
|
GC waits ↑
32. First Possible Solution
Move the batch workload to the instance where the majority of the relevant data activity occurs, or otherwise redesign service placement.
For example:
ORCL1 | +-- SALES_APP ORCL2 | +-- BATCH
But don’t blindly move services.
You need to measure the workload first.
33. Second Possible Solution
Reduce the amount of data being modified.
For example, review:
UPDATE orders SET status = 'X' WHERE ...
Ask:
- Can the predicate be improved?
- Is the correct index available?
- Is the transaction too large?
- Can the batch be divided into smaller units?
- Can the processing window be changed?
34. Third Possible Solution: Partitioning
If the workload naturally separates by:
DATE REGION CUSTOMER BUSINESS_UNIT
partitioning can sometimes reduce contention and improve workload locality.
For example:
ORDERS | +-- PARTITION_2026_07 +-- PARTITION_2026_08 +-- PARTITION_2026_09
The right partitioning strategy depends entirely on access patterns.
Partitioning should never be applied simply because:
“RAC is slow.”
35. RAC Interconnect
Now we need to investigate the network.
Remember:
ORCL1 | | Cache Fusion ↓ RAC Interconnect | ↓ ORCL2
If the interconnect has:
- Packet loss
- High latency
- Errors
- Saturation
- NIC problems
Cache Fusion performance can suffer.
36. But Separate Two Problems
This is an important troubleshooting rule:
Problem A
Bad workload ↓ Too many block transfers
Problem B
Bad interconnect ↓ Block transfers are slow
They can produce similar symptoms.
Your job is to determine which one you actually have.
37. Check RAC Interconnect Configuration
On the database:
SELECT inst_id,
name,
ip_address,
is_public,
source
FROM gv$cluster_interconnects
ORDER BY inst_id;
This allows you to see the interconnect configuration from the Oracle perspective.
38. Check OS-Level Network Health
On Linux:
ip -s link
or:
ip -s link show <interface>
Look for:
RX errors TX errors dropped overruns
Also:
ethtool <interface>
Check:
Speed Duplex Link detected Errors
The exact commands depend on your OS and network architecture.
39. Check Interface Statistics
For example:
cat /proc/net/dev
Look for abnormal:
errors dropped
You can also monitor traffic with:
sar -n DEV 1 10
and:
sar -n EDEV 1 10
40. Network Latency Matters
A RAC interconnect isn’t just about bandwidth.
Latency matters enormously.
Conceptually:
Low latency ↓ Fast block transfer ↓ Good Cache Fusion performance
versus:
High latency ↓ Slower block transfer ↓ Longer GC waits
But again:
Don’t diagnose the network until you’ve demonstrated that the workload actually requires excessive remote block transfers.
41. A Practical RAC GC Report
Here is a useful starting script:
SET LINESIZE 220
SET PAGESIZE 100
PROMPT =========================================
PROMPT GLOBAL CACHE WAIT EVENTS
PROMPT =========================================
SELECT inst_id,
event,
total_waits,
ROUND(time_waited_micro / 1000000,2) wait_seconds
FROM gv$system_event
WHERE event LIKE 'gc %'
ORDER BY time_waited_micro DESC;
PROMPT =========================================
PROMPT CURRENT GLOBAL CACHE WAITS
PROMPT =========================================
SELECT inst_id,
event,
COUNT(*) sessions
FROM gv$session
WHERE status = 'ACTIVE'
AND event LIKE 'gc %'
GROUP BY inst_id, event
ORDER BY sessions DESC;
PROMPT =========================================
PROMPT RAC INTERCONNECTS
PROMPT =========================================
SELECT inst_id,
name,
ip_address,
is_public,
source
FROM gv$cluster_interconnects
ORDER BY inst_id;
This is a good baseline for your RAC toolkit.
42. Advanced ASH Query
Now let’s combine everything:
SELECT instance_number,
service_name,
sql_id,
event,
current_obj#,
COUNT(*) samples
FROM dba_hist_active_sess_history
WHERE sample_time BETWEEN
TIMESTAMP '2026-08-08 09:00:00'
AND TIMESTAMP '2026-08-08 09:30:00'
AND event LIKE 'gc %'
GROUP BY instance_number,
service_name,
sql_id,
event,
current_obj#
ORDER BY samples DESC
FETCH FIRST 50 ROWS ONLY;
This is an extremely useful query during a RAC performance investigation.
It can give you:
Instance Service SQL Wait Object Samples
all in one result.
43. Turning the Object ID Into a Name
You can then join to DBA_OBJECTS.
For example:
SELECT ash.instance_number,
ash.service_name,
ash.sql_id,
ash.event,
o.owner,
o.object_name,
o.object_type,
COUNT(*) samples
FROM dba_hist_active_sess_history ash
LEFT JOIN dba_objects o
ON o.object_id = ash.current_obj#
WHERE ash.sample_time BETWEEN
TIMESTAMP '2026-08-08 09:00:00'
AND TIMESTAMP '2026-08-08 09:30:00'
AND ash.event LIKE 'gc %'
GROUP BY ash.instance_number,
ash.service_name,
ash.sql_id,
ash.event,
o.owner,
o.object_name,
o.object_type
ORDER BY samples DESC
FETCH FIRST 50 ROWS ONLY;
Now your output can become:
INSTANCE SERVICE SQL_ID EVENT OBJECT -------- ---------- ---------- --------------------- ---------------- 2 SALES_APP 8a12abc gc current request SALES.ORDERS 2 SALES_APP 8a12abc gc current block busy SALES.ORDERS 1 BATCH 4xyz789 gc current request SALES.ORDERS
That is a very strong indication of a workload-locality problem.
44. The RAC Performance Triangle
When you see GC waits, investigate three things:
SQL
/ \
/ \
/ \
OBJECT ---- INSTANCE
\ /
\ /
SERVICE
More practically:
SQL ↓ Object ↓ Which instance? ↓ Which service? ↓ Why is another instance accessing it?
45. The Most Common Mistake
A junior DBA sees:
gc current request
and says:
“The RAC interconnect needs tuning.”
An experienced DBA asks:
“Which SQL is requesting these blocks?”
Then:
“Which object?”
Then:
“Which instance owns the current version?”
Then:
“Why is this workload distributed across instances?”
Only after answering those questions should the interconnect become a primary suspect.
46. RAC Cache Fusion Troubleshooting Checklist
Waits
gc cr requestgc current requestgc cr block busygc current block busygc buffer busy
SQL
- Top SQL by GC samples
- Execution plan
- SQL execution frequency
- Logical reads
- Physical reads
Objects
- Hot table
- Hot index
- Object access pattern
- Partitioning
- Block contention
RAC
- Instance distribution
- Service distribution
- Cache Fusion traffic
- Workload locality
Network
- Interconnect
- Latency
- Packet drops
- NIC errors
- Bandwidth
47. A Senior DBA’s Decision Tree
When you see high GC waits:
GC waits high
|
↓
Which GC event?
|
+-----------+-----------+
| |
CR CURRENT
| |
↓ ↓
Read workload Modification workload
| |
+-----------+-----------+
|
↓
Which SQL?
|
↓
Which object?
|
↓
Which instance?
|
↓
Which service?
|
↓
Why remote access?
|
+--------+--------+
| |
Workload issue Network issue
| |
↓ ↓
Tune workload Tune infrastructure
This is the approach I want you to internalize.
48. Part 18 Summary
We’ve now covered the other major side of RAC internals:
Architecture
- GCS
- Cache Fusion
- CR vs CURRENT blocks
- Local vs remote blocks
Wait events
gc cr requestgc current requestgc cr block busygc current block busygc buffer busy
Troubleshooting
- ASH analysis
- SQL identification
- Object identification
- Service analysis
- Instance analysis
- Hot blocks
- Block pinging
Infrastructure
- RAC interconnect
- Network latency
- NIC errors
- Network drops
Architecture optimization
- Service placement
- Workload locality
- Batch vs OLTP separation
- Partitioning considerations
The Key Lesson
Remember this:
High GC waits
≠
Bad RAC
≠
Bad Network
Instead:
High GC waits
↓
Find SQL
↓
Find Object
↓
Find Instance
↓
Find Service
↓
Understand Workload
↓
Determine Why Blocks Move
↓
Then decide what to fix
That’s the difference between RAC monitoring and RAC performance engineering.
Google+
Facebook
