Real Application Cluster (RAC) - Overview, Advantages,Services, Background Processes and Commands

Real Application Cluster:

RAC is a database clustering(array or bundle) solution that allows more than one instance to mount and open an Oracle database. RAC can only be used on special clustered systems with shared disk storage and a private network interconnect.

In contrast, RAC allows multiple instances on different computer (nodes in a cluster) to access the same database files simultaneously and the clustered nodes are communicate over the Private interconnect. 

Real Application Clusters (RAC) introduced in Oracle 9i which has expanded from 8i's Oracle Parallel Server (OPS).
   

Benefits (Advantages):

High Availability - If some nodes fail, the remaining nodes will still be available for processing user requests. Failover support is available from Oracle 8 with Oracle's Transparent Application Failover (TAF) technology and from Oracle 10g, client/server side load balancing.

Performance (increased transaction response time) - RAC normally adds some overhead. However, for some applications one can achieve increased performance by running operations in parallel (mainly for Data Warehouse(DSS) applications).

Scale-up (increased transaction volume) - RAC can be used to provide increased application scalability (mainly used for OLTP applications). You can add or delete node as per DB load trends.For Eg: To cater Xmas load you can add node and later you can remove node from cluster.

Load Balancing - From Oracle 10g, you can use client/server side load balancing to distribute the load across the nodes. You will be using VIP or SCAN(11gR2) and service name(with preferred and available node specification) for load distribution, this will ensure the equal distribution of load across the cluster nodes.

Storage Options for RAC:

The database's data and control files are shared between the instances. However, each instance must have its own UNDO and REDO:
-UNDO: Each instance must have its own UNDO_TABLESPACE
-REDO: Each instance must have its own REDO LOG files (called as a thread)

Shared files can be stored on:

-RAW volumes

-Clustered file systems (like OCFS(Oracle introduced this in 10gR1), VCFS etc)

-ASM (Automatic Storage Management, introduced in 10gR2)

-ACFS (ASM Cluster File System, introduced in 11gR2)

What is Clusterware?

Oracle Clusterware provides Cluster Management Services and High Availability Services to Oracle and 3rd party software that wants to link into it. For example, if you kill database mandatory background process, CRS will detect the process failure and automatically restart the database.

Oracle clusterware eliminates the need for 3rd party clusterware software like Veritas Cluster, SUN Clusters, IBM HACMP and HP Omniguard etc. Oracle clusterware is provided at no additional cost with the 10g,11g and 12C database.

Oracle clusterware needs to be installed on all nodes of the cluster before installing the database(RDBMS) software. During installation, you will be prompted to configure a virtual IP, voting disk and cluster registry. You should configure shared storage, shared file systems,public/private/virtual/scan IPs before clusterware installation, also ssh need to be configured for remote software installation (You will be installing clusterware s/w from one(first) node of the cluster, it uses SSH settings to install s/w on remote nodes from the first node). 

Clusterware Components:

Oracle Cluster Registry(OCR):

The Cluster Registry (OCR) is used to store cluster wide settings and status information. For example: node names, IP and VIP addresses, voting disk locations, node applications, database names, instance names, listener names, etc.

The Oracle Cluster Registry (OCR) is a binary file that is maintained by the CRS daemon. The OCR can be stored in a raw partition or regular file in a clustered filesystem or in ASM(only in 11gR2) .

Voting Disk:

A Voting Disk is a shared disk device or file used to determine node availability and resolve split-brain scenarios.All instances write to the voting disk (check in) to indicate that they are still active. This is required as instances may not always be able to communicate across the network with each other. CSS will register/vote the node membership information in voting disk.

The voting disk, like the OCR, should be multiplexed and backed-up to protect them against media failures. Always implement 3 or more (an uneven number) of voting disks. Voting disks can be stored in a raw partition, regular file in a clustered filesystem (like OCFS, or in ASM (preferred).

Clusterware processes:

When clusterware is started, the following processes will be running. These are node specific daemons. Runs on all the nodes of the cluster.

- CRSD    -   Cluster Resource Services Daemon

(Run as root user. Failure of this process will restart the CRSD automatically without restarting the node) 

- CSSD    -   Cluster Synchronization Services Daemon 
(Run as Oracle user. Failure of this  process will restart the node ) 

- EVMD    -  Event Manager Daemon
(Run as Oracle user, Failure of this process will restart the EVMD automatically without restarting the node)

- OPROCD - Oracle Process Daemon 
( Run as root User. Failure of this  process will restart the node.This process runs only if your are using oracle clusterware, this process does not exists in 11gR2)

- OCLSOMON - Oracle CSSD Monitor
(Run as root User.Failure of this  process will restart the node. This process does not exists in 11gR2)

Corresponded executable are located in $CRS_HOME/bin/

How to stop and Start the clusterware services?

On unix there are rc scripts in place to start and stop CRS when the machine starts or shutdown.All are node specific commands.

/etc/init.d/init.crs start          
/etc/init.d/init.crs stop
/etc/init.d/init.crs enable       ->    Start the CRS during the node reboot. 
/etc/init.d/init.crs disable       ->    To disable the CRS re-start during the node reboot.
As a root we can manually start, stop, enable or disable the services with below commands. All are node specific commands.
# crsctl start crs
# crsctl stop crs

# crsctl check crs                 ->  to verify the status of crs processes   

# crsctl enable crs               ->   Start the CRS during the node restart 
# crsctl disable crs               ->  To disable the CRS re-start during the node reboot.

In 11gR2, below command will start the CRS on all the nodes 

# crsctl start cluster

# crsctl check cluster            -> to verify the status of crs processes  

Node Application Services: 


CRSD will pull the below services up. These services are registered in OCR and node specific.
- GSD - Global Service Daemon. Without this daemon, srvctl commands will not function in 10gR2. Dependency has been removed in 10gR2. CRS will take care of srvctl.
- ONS - Oracle Notification services. This service is required for publish the load notification to the application. 
This is one of the prerequisite for FAN(Fast Application Notification) configuration.
- VIP  - Virtual IP. This IP prevent the connections from TCP/IP timeouts. This IP address will fail-over to another surviving node in the event of node failure. This will allow  connected sessions to (almost) immediately fail-over to another surviving node when a problem is experienced. VIP will switch back to its original node when the original node joins the cluster, but connections (sessions) will not switch back to original node.
- LISTENER  - Required for database connectivity and server side load balancing. Listener knows the load of all other nodes in the cluster under the server side load balancing configuration



How to stop and Start the Node Application services?

- To stop node applications running on a node, enter the following

$ORACLE_HOME/bin/srvctl stop nodeapps -n <node>
- To start node applications running on a node, enter the following command, where node is the name of the node where the applications are  command, where node is the name of the node where the applications are runningrunning

$ORACLE_HOME/bin/srvctl start nodeapps -n <node>
- To verify the status of nodeapps
$ORACLE_HOME/bin/srvctl status nodeapps -n <node>



RAC Database Parameters:



Some of the INIT.ORA/SPFILE parameters required for RAC:
CLUSTER DATABASE=TRUE - Start the database in RAC mode
INSTANCE_NUMBER=n  - A unique instance number
THREAD=n                  - Each instance must have its own thread of redo logs
UNDO_TABLESPACE=<Tablespace Name>  - Each instance must have its own undo tablespace 
Etc.





RAC Background Processes

DIAG: Diagnosability Daemon – Monitors the health of the instance and captures the data for instance process failures.
LCKx - This process manages the global enqueue requests and the cross-instance broadcast. Workload is automatically shared and balanced when there are multiple Global Cache Service Processes (LMSx).

LMON - The Global Enqueue Service Monitor (LMON) monitors the entire cluster to manage the global enqueues and the resources. LMON manages instance and process failures and the associated recovery for the Global Cache Service (GCS) and Global Enqueue Service (GES). In particular, LMON handles the part of recovery associated with
global resources. LMON-provided services are also known as cluster group services (CGS)

LMDx - The Global Enqueue Service Daemon (LMD) is the lock agent process that manages enqueue manager service requests for Global Cache Service enqueues
to control access to global enqueues and resources. The LMD process also requests are the requests originating from another instance.
handles deadlock detection and remote enqueue requests. Remote resource 

LMSx - The Global Cache Service (GCS) messages. Real Application Clusters software provides for up to 10 Global Cache Service Processes. The number of LMSx varies depending on the amount of messaging traffic among nodes in the cluster. 

The LMSx handles the acquisition interrupt and blocking interrupt requests from the remote instances for Global Cache Service resources. For cross-instance consistent read requests, the LMSx requesting instance. The LMSx also controls the flow of messages to remote will create a consistent read version of the block and send it to the instances.

The LMSn processes handle the blocking interrupts from the remote instance for the Global Cache Service resources by:   
Managing the resource requests and cross-instance call operations for the shared resources. Building a list of invalid lock elements and validating the lock elements during recovery. Handling the  global lock deadlock detection and Monitoring for the lock conversion timeouts