Part 18 covered Cache Fusion and GCS. We saw that RAC performance depends heavily on the private interconnect.
Now we’ll investigate the infrastructure underneath it.
The goal of this part is not simply to check whether the network is “up”.
The goal is to answer:
Is the RAC interconnect correctly configured, redundant, fast, stable, and suitable for Cache Fusion and Clusterware traffic?
1. RAC Network Architecture
A typical Oracle RAC environment has several network roles:
CLIENTS
|
Public Network
|
+-----------+-----------+
| |
RAC1 RAC2
| |
Public Public
VIP VIP
| |
+-----------------------+
Private Network
|
RAC Interconnect
|
+--------+--------+
| |
RAC1 RAC2
| |
Private Private
IP IP
You should think of the RAC network as multiple logical functions, not simply “one network.”
2. The Main Network Components
For RAC 19c, you should understand:
SCAN VIP Public IP Private IP Interconnect DNS GNS (where applicable)
They have different purposes.
3. Public IP
The public IP is used for normal client-facing communication.
Example:
RAC1 192.168.10.101 RAC2 192.168.10.102
Applications normally connect through:
SCAN
rather than directly connecting to individual node addresses.
4. VIP
Each RAC node normally has a VIP.
Example:
RAC1 Public IP = 192.168.10.101 VIP = 192.168.10.111 RAC2 Public IP = 192.168.10.102 VIP = 192.168.10.112
The VIP provides fast client failure handling.
If RAC1 fails:
RAC1 X | ↓ VIP relocates | ↓ RAC2
The client doesn’t need to wait for a long TCP timeout against the failed node.
5. SCAN
SCAN stands for:
Single Client Access Name
For example:
racprod-scan.wadhahdaouehi.tn
It normally resolves to multiple IP addresses.
Example:
racprod-scan.wadhahdaouehi.tn
|
+-- 192.168.10.201
+-- 192.168.10.202
+-- 192.168.10.203
This allows Oracle RAC to provide connection distribution and failover capabilities.
6. Private Interconnect
Now we reach the most important network for this part.
The private network is used for internal RAC communication.
Examples include:
- Cache Fusion
- Global Cache Service
- Cluster synchronization
- Clusterware communication
- Other internal RAC traffic
Conceptually:
RAC1
|
| 10.10.10.101
|
+====================+
|
Private LAN
|
+====================+
|
| 10.10.10.102
|
RAC2
7. Why the Interconnect Is Critical
If the interconnect becomes unstable:
Cache Fusion
↓
Delayed
↓
GC waits
↓
Performance degradation
But it can become much more serious:
Interconnect failure
↓
Cluster communication failure
↓
Node eviction risk
↓
RAC availability incident
Therefore the private network is both:
A performance component
and:
A cluster availability component.
8. First Check — OIFCFG
On RAC, use:
oifcfg getif
Example:
eth0 192.168.10.0 global public eth1 10.10.10.0 global cluster_interconnect
This is one of the first commands I run when reviewing a RAC installation.
9. What oifcfg getif Tells You
You want to understand:
Interface Network Scope Role
For example:
eth0 192.168.10.0 global public
and:
eth1 10.10.10.0 global cluster_interconnect
This tells you Oracle’s view of the network configuration.
10. Check the Linux Interfaces
On every RAC node:
ip addr
or:
ip -br addr
Example:
eth0 UP 192.168.10.101/24 eth1 UP 10.10.10.101/24
Compare this against:
oifcfg getif
You want consistency.
11. Check Interface State
Run:
ip link
You want:
state UP
For example:
2: eth0: <BROADCAST,MULTICAST,UP,LOWER_UP> 3: eth1: <BROADCAST,MULTICAST,UP,LOWER_UP>
Pay attention to:
UP LOWER_UP
The second indicates the physical link is detected.
12. Check Interface Errors
This is one of the most important commands:
ip -s link show eth1
Look for:
RX:
errors
dropped
overruns
TX:
errors
dropped
For example:
RX: bytes packets errors dropped
... ... 0 0
TX: bytes packets errors dropped
... ... 0 0
Ideally, these should remain clean.
13. Why Drops Matter
Suppose:
RX dropped = 125000
You shouldn’t immediately conclude:
“RAC is broken.”
Instead investigate:
NIC ↓ Driver ↓ OS ↓ Switch ↓ VLAN ↓ MTU ↓ Traffic saturation
Network errors require correlation.
14. Check NIC Hardware
Use:
ethtool eth1
Example:
Speed: 25000Mb/s Duplex: Full Auto-negotiation: off Link detected: yes
The exact speed depends on your infrastructure.
The important thing is to verify that the NIC is operating as expected.
15. Check Driver
Run:
ethtool -i eth1
You’ll see information such as:
driver: version: firmware-version: bus-info:
This becomes important when troubleshooting:
- NIC instability
- Driver bugs
- Firmware problems
- Unexpected link resets
16. Check MTU
Run:
ip link show eth1
Example:
mtu 9000
or:
mtu 1500
Both can be valid.
The important rule is:
The MTU configuration must be consistent across the communication path.
17. Jumbo Frames
A common configuration is:
MTU = 9000
This is often called:
Jumbo frames.
The idea is to reduce packet-processing overhead by allowing larger Ethernet frames.
But there is a critical warning:
Don’t enable jumbo frames on one side and assume everything will work.
The complete path must support the configuration.
18. Test MTU Properly
For a 9000-byte MTU, use:
ping -M do -s 8972 10.10.10.102
Why 8972?
Because:
9000 - 20 bytes IP header - 8 bytes ICMP header = 8972
If the network supports this end-to-end, you should receive replies without fragmentation.
19. Test Standard MTU
For a normal 1500-byte MTU:
ping -M do -s 1472 10.10.10.102
Again:
1500 -20 -8 =1472
This is a useful diagnostic test.
20. Important: Test Both Directions
Don’t only test:
RAC1 → RAC2
Also test:
RAC2 → RAC1
For example:
# RAC1 ping -M do -s 8972 10.10.10.102
and:
# RAC2 ping -M do -s 8972 10.10.10.101
Network problems can be asymmetric.
21. Test Packet Loss
Basic test:
ping -c 100 10.10.10.102
You want:
100 packets transmitted 100 received 0% packet loss
But don’t rely exclusively on ping.
RAC traffic is not simply ICMP traffic.
22. Test Latency
Example:
ping -c 100 10.10.10.102
Look at:
min/avg/max/mdev
Example:
0.10/0.15/0.30/0.03 ms
The exact acceptable latency depends on your environment.
The important thing is:
Consistency matters.
A stable low-latency interconnect is what you want.
23. Why Jitter Matters
Imagine:
Request 1 → 0.10 ms Request 2 → 0.12 ms Request 3 → 5.00 ms Request 4 → 0.11 ms
Average latency might still look reasonable.
But the occasional spike can affect sensitive cluster communication.
So don’t only look at the average.
Investigate:
- max
- variance
- packet loss
- errors
- retransmissions
24. Check Routing
Use:
ip route
You should understand how the private network is routed.
For a simple private RAC network:
10.10.10.0/24 dev eth1
You don’t want unexpected routing through the public interface.
25. Test the Path
Use:
tracepath 10.10.10.102
or:
traceroute 10.10.10.102
For a properly designed private network, the path should be predictable.
26. RAC Interconnect Discovery
Oracle provides:
SELECT inst_id,
name,
ip_address,
is_public,
source
FROM gv$cluster_interconnects
ORDER BY inst_id;
This is useful because you’re now checking:
Linux view
+
Oracle view
Both should make sense together.
27. Check Clusterware Configuration
Use:
olsnodes -n -s
Example:
rac1 1 Active rac2 2 Active
This gives you a quick overview of cluster membership.
28. Check CRS Resources
Run:
crsctl stat res -t
You want to see:
ONLINE
for the expected resources.
Pay particular attention to:
ora.cssd ora.crsd ora.evmd
and the VIP/SCAN resources.
29. Check Cluster Status
A useful command:
crsctl check cluster -all
You want the cluster services to report healthy status on all nodes.
30. Check VIPs
Use:
srvctl status vip
Example:
VIP 192.168.10.111 is enabled VIP 192.168.10.111 is running on node rac1
and:
srvctl status vip -node rac1
depending on your exact GI version and command syntax.
31. Check SCAN
Run:
srvctl status scan
and:
srvctl status scan_listener
Also verify DNS:
nslookup racprod-scan.wadhahdaouehi.tn
You want all expected SCAN addresses returned.
32. DNS Is Critical
A RAC installation can appear healthy while DNS is incorrectly configured.
Check:
nslookup racprod-scan.wadhahdaouehi.tn
Then reverse resolution:
nslookup 192.168.10.201
The exact expected reverse DNS configuration depends on your environment.
The important principle:
RAC networking should have consistent forward and reverse name resolution where required.
33. Check Hostname Resolution
On each node:
hostname hostname -f
Then:
getent hosts rac1 getent hosts rac2
And:
getent hosts racprod-scan
This is often more useful than testing with nslookup alone because applications and OS components may use the system resolver configuration.
34. Network Redundancy
A production RAC should not depend on one physical network component if the business requires high availability.
A conceptual design:
RAC1
/ \
NIC1 NIC2
\ /
Bond
|
Switch Infrastructure
|
Bond
/ \
NIC1 NIC2
\ /
RAC2
The exact design depends on:
- Linux version
- Oracle Grid Infrastructure version
- NIC hardware
- Switch architecture
- Vendor recommendations
35. Bonding
Check:
cat /proc/net/bonding/bond0
You may see:
Bonding Mode MII Status Slave Interface Link Failure Count
You want to understand whether both links are healthy and whether the configured bonding mode matches your switch configuration.
36. Don’t Assume Bonding Is Always Better
This is important.
More NICs do not automatically mean:
Better RAC performance.
You must consider:
- Switch configuration
- LACP support
- Failover mode
- Driver support
- Oracle certification/support
- Linux configuration
- Traffic patterns
Always validate the architecture rather than copying a generic configuration from another environment.
37. Network Failure Scenario
Imagine:
RAC1 | eth1 X
If there is no redundancy:
Interconnect
X
|
Cluster communication lost
This can potentially lead to:
CSS problems
↓
Node eviction
That is why network redundancy is an availability requirement in serious RAC environments.
38. The CSS Perspective
Oracle Cluster Synchronization Services, CSS, is critical for cluster membership.
Conceptually:
RAC1 <------> RAC2
CSS
If nodes cannot maintain reliable cluster communication, Oracle must protect the cluster.
This is where node eviction can occur.
39. Node Eviction
A simplified scenario:
RAC1 | | Cannot communicate reliably X | RAC2
Oracle may determine that one node is no longer safely participating in the cluster.
The purpose of eviction is:
Protect the integrity of the cluster and database.
This is why RAC networking is not merely a performance concern.
40. What to Check After an Eviction
Don’t immediately blame the database.
Investigate:
1. Alert log 2. CRS logs 3. CSS logs 4. OS logs 5. NIC logs 6. Switch logs 7. Network errors 8. Link flaps 9. Packet loss 10. Interface resets
For example:
dmesg | egrep -i 'link|eth|nic|network'
and:
journalctl -k
depending on your Linux distribution.
41. Check Link Flaps
A link that repeatedly changes:
UP DOWN UP DOWN
is dangerous.
Check:
dmesg | grep -i "link"
or:
journalctl -k | grep -i "link"
Look for:
NIC reset link down link up carrier lost
42. Switch-Side Investigation
When troubleshooting RAC interconnect problems, involve the network team.
Ask them to check:
- Port errors
- CRC errors
- Drops
- Interface resets
- MTU
- VLAN configuration
- LACP
- Speed/duplex
- Port flapping
- Switch CPU
- Congestion
A DBA cannot fully diagnose a physical network problem from the database server alone.
43. CRC Errors
CRC errors are particularly interesting.
For example:
RX CRC errors
can indicate physical/network-layer problems.
Possible causes include:
- Bad cable
- Bad transceiver
- Faulty NIC
- Switch port issue
- Duplex mismatch
- Hardware problem
The exact cause requires correlation.
44. MTU Mismatch Example
Suppose:
RAC1 eth1 = MTU 9000 RAC2 eth1 = MTU 1500
You can have strange behavior.
Small packets:
Works
Large packets:
Fails / fragments / drops
This is why a simple:
ping 10.10.10.102
is not enough.
Test with the correct packet size and DF behavior.
45. Practical MTU Test
For 9000:
ping -M do -s 8972 10.10.10.102
If that fails, progressively reduce:
ping -M do -s 8000 10.10.10.102 ping -M do -s 7000 10.10.10.102 ping -M do -s 6000 10.10.10.102
This can help identify the maximum supported payload.
But remember:
This is a diagnostic test, not a replacement for validating the entire network configuration.
46. Monitor the Interface During Load
One snapshot isn’t enough.
Run:
sar -n DEV 1 60
and:
sar -n EDEV 1 60
You can monitor:
- packets/sec
- throughput
- errors
- drops
during the actual RAC workload.
47. Advanced Monitoring With ethtool
Useful commands include:
ethtool -S eth1
This can expose driver/NIC counters.
Look for counters related to:
errors drops missed packets CRC timeouts restarts
The exact counter names vary by NIC driver.
48. Check CPU Impact
Don’t forget the host CPU.
High network interrupt processing can affect the system.
Check:
mpstat -P ALL 1 10
and:
sar -n DEV 1 10
You may discover that the issue isn’t bandwidth but CPU/interrupt pressure.
49. RAC Interconnect Health-Check Script
Let’s build a simple baseline script.
Run as root:
#!/bin/bash echo "==========================================" echo "RAC INTERCONNECT HEALTH CHECK" echo "==========================================" echo echo "HOSTNAME" hostname -f echo echo "INTERFACES" ip -br addr echo echo "ROUTES" ip route echo echo "OIFCFG" su - oracle -c "oifcfg getif" echo echo "CLUSTER NODES" su - oracle -c "olsnodes -n -s" echo echo "CRS STATUS" su - oracle -c "crsctl check cluster -all" echo echo "INTERFACE STATISTICS" ip -s link echo echo "NETWORK DEVICES" nmcli device status 2>/dev/null || true echo echo "KERNEL NETWORK ERRORS" dmesg | egrep -i 'link|nic|network|eth|bond' | tail -50 echo echo "==========================================" echo "END" echo "=========================================="
This is a starting point, not a replacement for Oracle’s supported diagnostics.
50. Database-Level Health Check
Now add SQL.
SET LINESIZE 220
SET PAGESIZE 100
PROMPT =====================================
PROMPT RAC INTERCONNECT
PROMPT =====================================
SELECT inst_id,
name,
ip_address,
is_public,
source
FROM gv$cluster_interconnects
ORDER BY inst_id;
PROMPT =====================================
PROMPT GLOBAL CACHE WAITS
PROMPT =====================================
SELECT inst_id,
event,
total_waits,
ROUND(time_waited_micro/1000000,2) wait_seconds
FROM gv$system_event
WHERE event LIKE 'gc %'
ORDER BY time_waited_micro DESC;
PROMPT =====================================
PROMPT ACTIVE GC WAITS
PROMPT =====================================
SELECT inst_id,
event,
COUNT(*) sessions
FROM gv$session
WHERE status = 'ACTIVE'
AND event LIKE 'gc %'
GROUP BY inst_id,event
ORDER BY sessions DESC;
51. Combine Infrastructure and Database Evidence
This is where the investigation becomes interesting.
Imagine:
Database
gc current request ↑
Linux
No errors No drops Low latency
Switch
No errors No congestion
Then the network is probably not your first suspect.
Investigate:
SQL Object Workload locality Service placement
52. Another Scenario
Database:
gc current request ↑ gc current block busy ↑
Linux:
RX drops CRC errors
Switch:
CRC errors Port flapping
Now you have a very different situation.
The network becomes a strong suspect.
This is why:
Correlation is more important than any single metric.
53. RAC Interconnect Troubleshooting Matrix
| Symptom | First Investigation |
|---|---|
gc current request | SQL/object/workload |
gc current block busy | Hot blocks/contention |
| High GC waits + clean network | Workload locality |
| High GC waits + packet drops | Network |
| CRC errors | NIC/switch/cabling |
| MTU test failure | Network path |
| Link flapping | NIC/switch |
| Node eviction | CSS + network + OS |
| VIP problems | Public network |
| SCAN connection problems | DNS/SCAN/listener |
| Cross-instance contention | Services/workload placement |
54. Production Case Study
Let’s take a realistic scenario.
You receive:
“The production RAC became slow at 14:30.”
You run:
SELECT inst_id,
event,
COUNT(*)
FROM gv$session
WHERE status='ACTIVE'
GROUP BY inst_id,event
ORDER BY COUNT(*) DESC;
You discover:
ORCL2 gc current request
Next:
SELECT sql_id,
event,
COUNT(*)
FROM dba_hist_active_sess_history
WHERE sample_time BETWEEN
TIMESTAMP '2026-08-08 14:20:00'
AND TIMESTAMP '2026-08-08 14:40:00'
AND event LIKE 'gc %'
GROUP BY sql_id,event
ORDER BY COUNT(*) DESC;
You find:
SQL_ID = 8a12abc
Then object analysis shows:
SALES.ORDERS
You inspect services:
SALES_APP → ORCL2 BATCH → ORCL1
Both heavily modify:
SALES.ORDERS
Then OS network checks show:
0% packet loss 0 CRC errors 0 interface drops
Conclusion:
The interconnect is functioning correctly. The primary problem is workload distribution and cross-instance block contention.
That’s a much stronger diagnosis than:
“RAC network is slow.”
55. Another Case: Real Network Problem
Now suppose:
gc current request ↑
and:
ip -s link show eth1
shows:
RX errors: 15231 RX dropped: 9482
Then:
ethtool -S eth1
shows increasing hardware errors.
Switch team reports:
CRC errors
Now your conclusion becomes:
The RAC interconnect is experiencing network-layer errors that are affecting Cache Fusion communication.
This is an infrastructure problem.
56. The Expert RAC Network Workflow
When investigating RAC network problems:
INCIDENT
|
↓
Database symptoms
|
+------+------+
| |
GC waits Cluster errors
| |
↓ ↓
ASH/AWR CRS/CSS logs
| |
+------+------+
|
↓
Oracle network view
|
↓
Linux network
|
↓
NIC / driver
|
↓
Switch
|
↓
Physical layer
Don’t jump directly to the bottom.
Start with the symptom and work downward.
57. What You Should Memorize
For Oracle RAC network troubleshooting, these commands should become familiar:
Oracle
oifcfg getif olsnodes -n -s crsctl check cluster -all crsctl stat res -t srvctl status vip srvctl status scan srvctl status scan_listener
Linux
ip -br addr ip -s link ip route ethtool eth1 ethtool -S eth1 sar -n DEV 1 10 sar -n EDEV 1 10 ping tracepath
Database
GV$CLUSTER_INTERCONNECTS GV$SYSTEM_EVENT GV$SESSION GV$ACTIVE_SESSION_HISTORY DBA_HIST_ACTIVE_SESS_HISTORY
58. Part 19 — Final Checklist
Before declaring a RAC interconnect healthy, verify:
Oracle
oifcfgis correct- Correct private interface
- Correct public interface
- All nodes see the expected interconnect
- Clusterware healthy
Linux
- Interface UP
- Link UP
- Correct IP
- Correct MTU
- No significant errors
- No unexpected drops
- No link flapping
- Correct speed/duplex
Network
- Correct VLAN
- Correct MTU end-to-end
- No packet loss
- No CRC errors
- No congestion
- Redundant paths where required
RAC workload
- GC waits understood
- Hot objects identified
- Service placement reviewed
- Cross-instance traffic understood
The Key Lesson
The most important concept from Part 19 is this:
Never troubleshoot the RAC interconnect in isolation.
A RAC performance problem often follows this chain:
Application
↓
SQL
↓
Object
↓
Block
↓
Cache Fusion
↓
Interconnect
↓
NIC
↓
Switch
And the reverse is also true:
Network problem
↓
Cache Fusion latency
↓
GC waits
↓
SQL response time
↓
Application slowdown
An expert RAC DBA needs to be comfortable moving up and down this entire stack.
Google+
Facebook
