Showing posts with label RAC. Show all posts
Showing posts with label RAC. Show all posts

Wednesday, February 15, 2017

Steps to Create RAC Service on Windows


Add the service using the below command

>srvctl add service -d UKTILDB -s TIL_UKTILDB_TIBCO_UK -r UKTILDB1 -a UKTILDB2

The added service will be listed as offline since we are adding the service as our own user.

>crs_stat -t
Name           Type           Target    State     Host
------------------------------------------------------------
ora....DB2.srv application    ONLINE    ONLINE    ukti...b01a
ora....LING.cs application    ONLINE    ONLINE    ukti...b01a
ora....DB1.srv application    ONLINE    ONLINE    ukti...b01a
ora....IBCO.cs application    ONLINE    ONLINE    ukti...b01a
ora....DB1.srv application    OFFLINE   OFFLINE
ora....O_UK.cs application    OFFLINE   OFFLINE
ora....B1.inst application    ONLINE    ONLINE    ukti...b01a
ora....B2.inst application    ONLINE    ONLINE    ukti...b01b
ora.UKTILDB.db application    ONLINE    ONLINE    ukti...b01a
ora....SM1.asm application    ONLINE    ONLINE    ukti...b01a
ora....1A.lsnr application    ONLINE    ONLINE    ukti...b01a
ora....01a.gsd application    ONLINE    ONLINE    ukti...b01a
ora....01a.ons application    ONLINE    ONLINE    ukti...b01a
ora....01a.vip application    ONLINE    ONLINE    ukti...b01a
ora....SM2.asm application    ONLINE    ONLINE    ukti...b01b
ora....1B.lsnr application    ONLINE    ONLINE    ukti...b01b
ora....01b.gsd application    ONLINE    ONLINE    ukti...b01b
ora....01b.ons application    ONLINE    ONLINE    ukti...b01b
ora....01b.vip application    ONLINE    ONLINE    ukti...b01b


NAME=ora.UKTILDB.TIL_UKTILDB_TIBCO_UK.UKTILDB1.srv
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE

NAME=ora.UKTILDB.TIL_UKTILDB_TIBCO_UK.cs
TYPE=application
TARGET=OFFLINE
STATE=OFFLINE

Login as Service account user and add the service account user to remote desktop user Still it will show the service  status offline.

>crs_stat -t

Name           Type           Target    State     Host
------------------------------------------------------------
ora....DB2.srv application    ONLINE    ONLINE    ukti...b01a
ora....LING.cs application    ONLINE    ONLINE    ukti...b01a
ora....DB1.srv application    ONLINE    ONLINE    ukti...b01a
ora....IBCO.cs application    ONLINE    ONLINE    ukti...b01a
ora....DB1.srv application    OFFLINE   OFFLINE
ora....O_UK.cs application    OFFLINE   OFFLINE
ora....B1.inst application    ONLINE    ONLINE    ukti...b01a
ora....B2.inst application    ONLINE    ONLINE    ukti...b01b
ora.UKTILDB.db application    ONLINE    ONLINE    ukti...b01a
ora....SM1.asm application    ONLINE    ONLINE    ukti...b01a
ora....1A.lsnr application    ONLINE    ONLINE    ukti...b01a
ora....01a.gsd application    ONLINE    ONLINE    ukti...b01a
ora....01a.ons application    ONLINE    ONLINE    ukti...b01a
ora....01a.vip application    ONLINE    ONLINE    ukti...b01a
ora....SM2.asm application    ONLINE    ONLINE    ukti...b01b
ora....1B.lsnr application    ONLINE    ONLINE    ukti...b01b
ora....01b.gsd application    ONLINE    ONLINE    ukti...b01b
ora....01b.ons application    ONLINE    ONLINE    ukti...b01b
ora....01b.vip application    ONLINE    ONLINE    ukti...b01b


Check the permission of the user

>crs_getperm ora.UKTILDB.TIL_UKTILDB_TIBCO_UK.UKTILDB1.srv
Name: ora.UKTILDB.TIL_UKTILDB_TIBCO_UK.UKTILDB1.srv
owner:system:rwx,pgrp::---,other::r--,

>crs_getperm ora.UKTILDB.TIL_UKTILDB_TIBCO_UK.cs
Name: ora.UKTILDB.TIL_UKTILDB_TIBCO_UK.cs
owner:system:rwx,pgrp::---,other::r--,

Then change the permission  to the service account owner.

>crs_setperm ora.UKTILDB.TIL_UKTILDB_TIBCO_UK.UKTILDB1.srv -o TIL\uktil02ighsorldb01


>crs_setperm ora.UKTILDB.TIL_UKTILDB_TIBCO_UK.cs -o TIL\uktil02ighsorldb01

>crs_getperm ora.UKTILDB.TIL_UKTILDB_TIBCO_UK.UKTILDB1.srv
Name: ora.UKTILDB.TIL_UKTILDB_TIBCO_UK.UKTILDB1.srv

>crs_getperm ora.UKTILDB.TIL_UKTILDB_TIBCO_UK.UKTILDB1.srv
Name: ora.UKTILDB.TIL_UKTILDB_TIBCO_UK.UKTILDB1.srv
owner:TIL\uktil02ighsorldb01:rwx,pgrp::---,other::r--,

Then start the service which was created earlier.

>srvctl start service -d UKTILDB -s TIL_UKTILDB_TIBCO_UK

>crs_stat -t

Name           Type           Target    State     Host
------------------------------------------------------------
ora....DB2.srv application    ONLINE    ONLINE    ukti...b01a
ora....LING.cs application    ONLINE    ONLINE    ukti...b01a
ora....DB1.srv application    ONLINE    ONLINE    ukti...b01a
ora....IBCO.cs application    ONLINE    ONLINE    ukti...b01a
ora....DB1.srv application    ONLINE    ONLINE    ukti...b01a
ora....O_UK.cs application    ONLINE    ONLINE    ukti...b01a
ora....B1.inst application    ONLINE    ONLINE    ukti...b01a
ora....B2.inst application    ONLINE    ONLINE    ukti...b01b
ora.UKTILDB.db application    ONLINE    ONLINE    ukti...b01a
ora....SM1.asm application    ONLINE    ONLINE    ukti...b01a
ora....1A.lsnr application    ONLINE    ONLINE    ukti...b01a
ora....01a.gsd application    ONLINE    ONLINE    ukti...b01a
ora....01a.ons application    ONLINE    ONLINE    ukti...b01a
ora....01a.vip application    ONLINE    ONLINE    ukti...b01a
ora....SM2.asm application    ONLINE    ONLINE    ukti...b01b
ora....1B.lsnr application    ONLINE    ONLINE    ukti...b01b
ora....01b.gsd application    ONLINE    ONLINE    ukti...b01b
ora....01b.ons application    ONLINE    ONLINE    ukti...b01b
ora....01b.vip application    ONLINE    ONLINE    ukti...b01b

======================================================



GPNP Profile - Oracle Clusterware 11g Release 2

WHAT IS GPNP PROFILE?

The GPnP profile is a small XML file located in GRID_HOME/gpnp/<hostname>/profiles/peer under the name profile.xml. It is used to establish the correct global personality of a node. Each node maintains a local copy of the GPnP Profile and is maintained by the GPnP Deamon (GPnPD) .

WHAT DOES GPNP PROFILE CONTAIN?

GPnP Profile  is used to store necessary information required for the startup of Oracle Clusterware like  SPFILE location,ASM DiskString  etc.

It contains various attributes defining node personality.
- Cluster name
Network classifications (Public/Private)
Storage to be used for CSS
Storage to be used for ASM : SPFILE location,ASM DiskString  etc
- Digital signature information : The profile is security sensitive. It might identify the storage to be used as the root partition of a machine.  Hence, it contains digital signature information of the provisioning authority.

Here is the GPnP profile of our RAC setup. gpnptool can be  used  for reading/editing the gpnp profile.

[root@host01 peer]# gpnptool get
<?xml version=”1.0″ encoding=”UTF-8″?><gpnp:GPnP-Profile Version=”1.0″ xmlns=”http://www.grid-pnp.org/2005/11/gpnp-profile” xmlns:gpnp=”http://www.grid-pnp.org/2005/11/gpnp-profile” xmlns:orcl=”http://www.oracle.com/gpnp/2005/11/gpnp-profile” xmlns:xsi=”http://www.w3.org/2001/XMLSchema-instance” xsi:schemaLocation=”http://www.grid-pnp.org/2005/11/gpnp-profile gpnp-profile.xsd” ProfileSequence=”7″ ClusterUId=”14cddaccc0464f92bfc703ec1004a386″ ClusterName=”cluster01″ PALocation=””><gpnp:Network-Profile><gpnp:HostNetwork id=”gen” HostName=”*”><gpnp:Network id=”net1″ IP=”192.9.201.0″ Adapter=”eth0″ Use=”public”/><gpnp:Network id=”net2″ IP=”10.0.0.0″ Adapter=”eth1″ Use=”cluster_interconnect”/></gpnp:HostNetwork></gpnp:Network-Profile><orcl:CSS-Profile id=”css” DiscoveryString=”+asm” LeaseDuration=”400″/><orcl:ASM-Profile id=”asm” DiscoveryString=”” SPFile=”+DATA/cluster01/asmparameterfile/registry.253.783619911″/><ds:Signature xmlns:ds=”http://www.w3.org/2000/09/xmldsig#”><ds:SignedInfo><ds:CanonicalizationMethod Algorithm=”http://www.w3.org/2001/10/xml-exc-c14n#”/><ds:SignatureMethod Algorithm=”http://www.w3.org/2000/09/xmldsig#rsa-sha1″/><ds:Reference URI=””><ds:Transforms><ds:Transform Algorithm=”http://www.w3.org/2000/09/xmldsig#enveloped-signature”/><ds:Transform Algorithm=”http://www.w3.org/2001/10/xml-exc-c14n#”> <InclusiveNamespaces xmlns=”http://www.w3.org/2001/10/xml-exc-c14n#” PrefixList=”gpnp orcl xsi”/></ds:Transform></ds:Transforms><ds:DigestMethod Algorithm=”http://www.w3.org/2000/09/xmldsig#sha1″/><ds:DigestValue>4VMorzxVNa+FeOx2SCk1unVBpfU=</ds:DigestValue></ds:Reference></ds:SignedInfo><ds:SignatureValue>bbzV04n2zSGTtUEvqqB+pjw1vH7i8MOEUqkhXAyloX0a41T2FkDEA++ksc0BafndAk7tR+6LGdppE1aOsaJUtYxQqaHJdpVsJF+sj2jN7LPJlT5NBt+K7b08TLjDID92Se6vEiDAeeKlEbpVWKMUIvQvp6LrYK8cDB/YjUnXuGU=</ds:SignatureValue></ds:Signature></gpnp:GPnP-Profile>

WHO UPDATES GPNP PROFILE?
GPnPd daemon replicates changes to the profile during
  • - installation,
  • - system boot or
  • - when updated
Profile is updated Whenever changes are made to a cluster with configuration tools like
  • . oifcfg (Change network),
  • . crsctl (change location of voting disk),
  • . asmcmd (change ASM_DISKSTRING, SPfile location) etc.
HOW IS GPNP PROFILE USED BY CLUSTERWARE?

To start clusterware, voting disk needs to be accessed. If voting disk is on ASM, this information (that voting disk is on ASM) is read from GPnP profile (<orcl:CSS-Profile id=”css” DiscoveryString=”+asm” LeaseDuration=”400″/>).   The voting disk is read using kfed utility  even if ASM is not up.
Next,  the clusterware checks if all the nodes have the updated GPnP profile and the node joins the cluster based on the GPnP configuration . Whenver a node is started/added to the cluster, the clusterware software on the starting node starts a GPnP agent.
  • - If the node is already part of the cluster, the GPnP agent reads the existing profile on that node.
  • - If the node is being added to the cluster, GPnP agent locates agent on another existing node using multicast protocol (provided by mDNS) and gets the profile from that agent.
Next CRSD needs to read OCR to startup various resources on the node and hence update it as status of resources changes. Since OCR is also on ASM, location of ASM SPfile should be known.
The order of searching the ASM SPfile is
  • - GPnP profile
  • - ORACLE_HOME/dbs/spfile<sid.ora>
  • - ORACLE_HOME/dbs/init<sid.ora>
In cluster environment, the location of  SPfile for ASMread from GPnP profile.

[grid@host01 peer]$ gpnptool getpval -asm_spf
Warning: some command line parameters were defaulted. Resulting command line:
         /u01/app/11.2.0/grid/bin/gpnptool.bin getpval -asm_spf -p=profile.xml -o-
+DATA/cluster01/asmparameterfile/registry.253.793721441
The oputput of the query shows that SPfile is on ASM in DATA diskgroup. 

To find out the location of ASM disks, following query is issued :
[root@host01 peer]# gpnptool getpval -asm_dis
ASM-Profile id=”asm” DiscoveryString=””
The  device headers of every device in the disk string returned by the above query are scanned  (if configured by you at ASM initial setup time). Here Discovery String is blank is as ASMDISKSTRINGS parameter has not been set. Hence, headers of all the ASM disks are scanned .
Here, I have shown the output of the query only on the disk which contains SPfile.(spfflg is not null)

[root@host01 ~]#  kfed read /dev/sdb3 | grep -E ‘spf|ausize’
kfdhdb.ausize:                  1048576 ; 0x0bc: 0x00100000
kfdhdb.spfile:                       16 ; 0x0f4: 0x00000010
kfdhdb.spfflg:                        1 ; 0x0f8: 0x00000001
In the output above, we see that
     the device /dev/sdb3 contains a copy of the ASM spfile (spfflg=1).
     The ASM spfile location starts at the disk offset of 16 (spfile=16)

Considering the allocation unit size (kfdhdb.ausize = 1M), let’s dump the ASM spfile from the device:
[root@host01 ~]#  dd if=/dev/sdb3 of=spfileASM_Copy2.ora skip=16  bs=1M count=1
1+0 records in
1+0 records out
1048576 bytes (1.0 MB) copied, 0.170611 seconds, 6.1 MB/s
[root@host01 ~]# strings spfileASM_Copy2.ora
+ASM1.__oracle_base=’/u01/app/grid’#ORACLE_BASE set from in memory value
+ASM2.__oracle_base=’/u01/app/grid’#ORACLE_BASE set from in memory value
+ASM3.__oracle_base=’/u01/app/grid’#ORACLE_BASE set from in memory value
+ASM3.asm_diskgroups=’FRA’#Manual Mount
+ASM2.asm_diskgroups=’FRA’#Manual Mount
+ASM1.asm_diskgroups=’FRA’#Manual Mount
*.asm_power_limit=1
*.diagnostic_dest=’/u01/app/grid’
*.instance_type=’asm’
*.large_pool_size=12M
*.remote_login_passwordfile=’EXCLUSIVE’
Using the parameters in SPfile, ASM is started.
Once ASM is up, OCR is read by CRSD and various resources on the node are started.
Each node reads network information in GPnP profile and using GNS,  negotiates appropriate network identity for itself . Hence, nodes can be dynamically added/deleted.
What happens if GPnP profile is lost?
To know please click  here.
————————————————————
GPNPTOOL COMMAND REFERENCE:

How to read the profile
[root@inssc3 bin]# ./gpnptool get

How to find GPnP Deamons are running on the local node
[root@host01 peer]# gpnptool lfind
Success. Local gpnpd found.

How to find the location of ASM spfile if the ASM is down
[root@host01 peer]# gpnptool getpval -asm_spf
+DATA/cluster01/asmparameterfile/registry.253.783619911

How to find all RD-discoverable resources of given type
[root@host01 peer]# gpnptool find
Found 3 instances of service ‘gpnp’.
        mdns:service:gpnp._tcp.local.://host03:18015/agent=gpnpd,cname=cluster01,host=host03,pid=5066/gpnpd h:host03 c:cluster01
        mdns:service:gpnp._tcp.local.://host02:17637/agent=gpnpd,cname=cluster01,host=host02,pid=5236/gpnpd h:host02 c:cluster01
        mdns:service:gpnp._tcp.local.://host01:16633/agent=gpnpd,cname=cluster01,host=host01,pid=5206/gpnpd h:host01 c:cluster01

Tuesday, October 25, 2016

Diagnosing RAC Instance eviction issue

An instance eviction occurs when a member was evicted from the group by another member of the cluster database for one of several reasons, which may include a communications error in the cluster, failure to issue a heartbeat to the control file, and other reasons.  This mechanism is in place to prevent problems from occuring that would affect the entire database.
 For example,instead of allowing a cluster-wide hang to occur, Oracle will evict the problematic instance(s) from the cluster.  When an ORA-29740 error occurs, a surviving instance will remove the problem instance(s) from the cluster.  When the problem is detected the instances 'race' to get a lock on the control file (Results Record lock) for updating.  The instance that obtains the lock tallies the votes of the instances to decide membership.  A member is evicted if:

a) A communications link is down
b) There is a split-brain (more than 1 subgroup) and the member is not in the largest subgroup
c) The member is perceived to be inactive


The various reasons for instance eviction :-

Reason 0 = No reconfiguration  
Reason 1 = The Node Monitor generated the reconfiguration.
Reason 2 = An instance death was detected.
Reason 3 = Communications Failure
Reason 4 = Reconfiguration after suspend


To determine the reason behind your instance eviction-

a) Review each instance's alert log
b) Review instance's LMON trace file
c) Review OS Watcher logs if used. If OSWATCHER is not installed in all the nodes of your RAC clusterware, request you to please install it . Refer Note 301137.1 for  procedure of installing and how to use OS watcher
d) Review the CKPT process trace file of the evicted instance
e) Other bdump or udump files generated at the exact time of the instance eviction
f) Review each node's syslog or messages file

Configure ASM mirroring for OCR and Voting Disk

1.Create another disk group OCR_VOTE_NORM with three asm disks with normal redundancy.

2.Replaced the current OCR_VOTE disk group (external redundancy) with OCR_VOTE_NORM
./crsctl replace votedisk +OCR_VOTE_NORM

3. Added OCR_VOTE_NORM disk group to OCR
./ocrconfig -add +OCR_VOTE_NORM

4. There are two copies of OCR files and three copies of Voting disks. 

Follow Oracle Note 428681.1 -OCR / Vote disk Maintenance Operations: (ADD/REMOVE/REPLACE/MOVE) to move OCR file from normal redundancy DG to external redundancy DG.

gc [current/cr] [2/3]-way Wait Event

A current or cr block was requested and received after 2 or 3 network hops. 
The request was processed immediately.

 If these events consume lot of time then check..

 - Network bandwidth saturation
 - Private Interconnect configuration
 - Socket send and receive buffer
 - Run queue length and system CPU usage

 Tuning
- System and load tuning to reduce latencies
- SQL and schema tuning to minimize IO and remote cache references 

Migrating Oracle Cluster Registry(OCR) location

Complete the following procedure:

1.Use the OCRCHECK utility to verify that a copy of OCR other than the one you are going to replace is online, using the following command:

$ ocrcheck

OCRCHECK displays all OCR locations that are registered and whether they are available (online). If an OCR location suddenly becomes unavailable, then it might take a short period for Oracle Clusterware to show the change in status.

Note:   The OCR location that you are replacing can be either online or offline.

2. Use the following command to verify that Oracle Clusterware is running on the node on which the you are going to perform the replace operation:

$ crsctl check crs

3. Run the following command as root to replace the current OCR location using either destination_file or +ASM_disk_group to indicate the current and target OCR locations:

# ocrconfig -replace <current_OCR_location> -replacement <new_OCR_location>

If you have only one OCR location, then use the following commands:

# ocrconfig -add +new_storage_disk_group
# ocrconfig -delete +current(old)_disk_group

Note:
If your cluster configuration changes while the node on which OCR resides is stopped, and the Oracle Clusterware stack is running on the other nodes, then OCR detects configuration changes and self-corrects the configuration by changing the contents of the ocr.loc file.

Migrating voting disks from one disk group to other

To migrate voting disks to other Oracle ASM diskgroup , specify the new Oracle ASM disk group name in the following command:

$ crsctl replace votedisk +asm_disk_group

You can run this command on any node in the cluster.

Verifying the voting disk location :-

After moving the voting disk, verify the voting disk location, as follows:

$ crsctl query css votedisk

Thursday, October 20, 2016

Apache Ambari

What is Apache Ambari ?

Apache Ambari is a web-based tool for provisioning, managing, and monitoring Apache Hadoop clusters. Ambari provides a dashboard for viewing cluster health such as heatmaps and ability to view MapReduce, Pig and Hive applications visually along with features to diagnose their performance characteristics in a user-friendly manner. It has a very simple and interactive UI to install various tools and perform various management, configuring and monitoring tasks. 
Apache Ambari, as part of the Hortonworks Data Platform, allows enterprises to plan, install and securely configure Hadoop making it easier to provide ongoing cluster maintenance and management, no matter the size of the cluster.

Ambari's Extensibility Features 

- Ambari Stacks
- Ambari Blueprints
- Ambari Views 

What Ambari Does?

Ambari makes Hadoop management simpler by providing a consistent, secure platform for operational control. Ambari provides an intuitive Web UI as well as a robust REST API, which is particularly useful for automating cluster operations. With Ambari, Hadoop operators get the following core benefits:

  • Simplified Installation, Configuration and Management. Easily and efficiently create, manage and monitor clusters at scale. Takes the guesswork out of configuration with Smart Configs and Cluster Recommendations.  Enables repeatable, automated cluster creation with Ambari Blueprints.
  • Centralized Security Setup. Reduce the complexity to administer and configure cluster security across the entire platform. Helps automate the setup and configuration of advanced cluster security capabilities such as Kerberos and Apache Ranger.
  • Full Visibility into Cluster Health. Ensure your cluster is healthy and available with a holistic approach to monitoring. Configures predefined alerts — based on operational best practices — for cluster monitoring. Captures and visualizes critical operational metrics — using Grafana — for analysis and troubleshooting. Integrated with Hortonworks SmartSense for proactive issue prevention and resolution.
  • Highly Extensible and Customizable. Fit Hadoop seamlessly into your enterprise environment. Highly extensible with Ambari Stacks for bringing custom services under management, and with Ambari Views for customizing the Ambari Web UI.
Ambari have two components
  1. Ambari server – This is the master process which communicates with Ambari agents installed on each node participating in the cluster. This has postgres database instance which is used to maintain all cluster related metadata.
  2. Ambari Agent – These are acting agents for Ambari on each node. Each agent periodically sends his own health status along with different metrics, installed services status and many more things. According master decides on next action and conveys back to the agent to act.
Ambari System Architecture
























Ambari Server Architecture



Wednesday, October 19, 2016

Oracle Local Registry (OLR) - New Component of Oracle Clusterware 11g Release 2

In Oracle Clusterware 11g Release2 an additional component related to the OCR called the Oracle Local Registry (OLR) is installed on each node in the cluster. The OLR is a local registry for node specific resources. THe OLR is not shared by other nodes in the cluster. It is installed and configured when Oracle clusterware is installed.
 
Purpose of OLR

It is the very first file that is accessed by OHASD service to startup 11gR2 clusterware when OCR is stored on ASM. OCR should be accessible to find out the resources which need to be started on a node. If OCR is on ASM, it can’t be read until ASM (which itself is a resource for the node and this information is stored in OCR) is up. To resolve this problem, information about the resources which need to be started on a node is stored in an operating system  file which is called Oracle Local Registry or OLR. Since OLR is a file an operating system file, it can be accessed by various processes on the node for read/write irrespective of the status of the clusterware (up/down). Hence, when  a node joins the cluster,  OLR on that node is read, various resources ,including ASM  are started on the node  . Once ASM is up , OCR is accessible and is used henceforth to manage all the clusterware resources. If OLR is missing or corrupted, clusterware can’t be started on that node!
 
Where is OLR located?

The OLR file is located in the grid_home/cdata/<hostname>.olr 
The location of OLR is stored in /etc/oracle/olr.loc.and used by OHASD .
 
What does OLR contain?

The OLR stores data about
  • ORA_CRS_HOME
  • localhost version
  • active version
  • GPnP details
  • OCR latest backup time and location
  • information about OCR daily, weekly backup location
  •  node name etc.
This information stored in the OLR is needed by OHASD to start or join a cluster.
A quick  peek at the backup of the olr shows the resources that are being maintained.
[root@server01 ~]# ocrconfig -local -manualbackup

Server01     2016/12/12 11:12:27     /u01/app/11.2.0/grid/cdata/host01/backup_20161212_110527.olr

[root@
Server01~]#strings /u01/app/11.2.0/grid/cdata/host01/backup_20161212_110527.olr |grep -v type |grep ora!

ora!drivers!acfs
ora!crsd
ora!asm
ora!evmd
ora!ctssd
ora!cssd
ora!cssdmonitor
ora!diskmon
ora!gpnpd
ora!gipcd
ora!mdnsd
 
OLR administration

You can view the status of the OLR file on each node by using the ocrcheck command with the –local parameter as seen here:
#ocrcheck -local
 
ocrdump can be used to  dump the contents of the OLR to tthe text terminal:
#ocrdump -local -stdout
 
You can use the ocrconfig command to export and import the OLR as seen in these examples:
#ocrconfig -local -export <export file name >
#ocrconfig -local -import <file name>
 
And you can repair the OLR file should it become corrupted with the ocrconfig command as seen in this example:
#ocrconfig -local -repair olr <file name>
 
The OLR is backed up at the end of an installation or an upgrade. After that time, you can only manually back up the OLR. Automatic backups are not supported for the OLR.

To manually back up OLR:
# ocrconfig –local –manualbackup

To view the contents of the OLR backup file:
#ocrdump -local -backupfile olr_backup_file_name

To change the OLR backup location
#ocrconfig -local -backuploc new_olr_backup_path

To restore OLR: 
# crsctl stop crs
# ocrconfig -local -restore file_name 
# ocrcheck -local 
# crsctl start crs 
$ cluvfy comp olr