Methods are available:
1. Rolling Method - Assuming that RDBMS & ASM instances are UP and enabling Write-Back Flash Cache in One Cell Server at a time
2. Non-Rolling Method - Assuming that RDBMS & ASM instances are DOWN while enabling Write-Back Flash Cache
Note: Before performing the below steps, Perform the following check as root from one of the compute nodes:
Check all griddisk “asmdeactivationoutcome” and “asmmodestatus” to ensure that all griddisks on all cells are “Yes” and “ONLINE” respectively.
# dcli -g cell_group -l root cellcli -e list griddisk attributes asmdeactivationoutcome, asmmodestatus
Check that all of the flashcache are in the “normal” state and that no flash disks are in a degraded or critical state:
# dcli -g cell_group -l root cellcli -e list flashcache detail
exadata01cell01: WriteThrough
exadata01cell02: WriteThrough
exadata01cell03: WriteThrough
1. Rolling Method:
(Assuming that RDBMS & ASM instances are UP and enabling Write-Back Flash Cache in One Cell Server at a time)
Login to Cell Server:
Step 1. Drop the flash cache on that cell
#cellcli –e drop flashcache
Flash cache exadata01cell01_FLASHCACHE successfully dropped
Step 2. Check the status of ASM if the grid disks go OFFLINE. The following command should return 'Yes' for the grid disks being listed:
# cellcli -e list griddisk attributes name,asmmodestatus,asmdeactivationoutcome
DATAC1_CD_00_exadata01cell01 OFFLINE Yes
DATAC1_CD_01_exadata01cell01 OFFLINE Yes
DATAC1_CD_02_exadata01cell01 OFFLINE Yes
DATAC1_CD_03_exadata01cell01 OFFLINE Yes
DATAC1_CD_04_exadata01cell01 OFFLINE Yes
DATAC1_CD_05_exadata01cell01 OFFLINE Yes
DBFS_DG_CD_02_exadata01cell01 OFFLINE Yes
DBFS_DG_CD_03_exadata01cell01 OFFLINE Yes
DBFS_DG_CD_04_exadata01cell01 OFFLINE Yes
DBFS_DG_CD_05_exadata01cell01 OFFLINE Yes
RECOC1_CD_00_exadata01cell01 OFFLINE Yes
RECOC1_CD_01_exadata01cell01 OFFLINE Yes
RECOC1_CD_02_exadata01cell01 OFFLINE Yes
RECOC1_CD_03_exadata01cell01 OFFLINE Yes
RECOC1_CD_04_exadata01cell01 OFFLINE Yes
RECOC1_CD_05_exadata01cell01 OFFLINE Yes
Step 3. Inactivate the griddisk on the cell
# cellcli –e alter griddisk all inactive
Step 4. Shut down cellsrv service
# cellcli -e alter cell shutdown services cellsrv
Stopping CELLSRV services...
The SHUTDOWN of CELLSRV services was successful.
Step 5. Set the cell flashcache mode to writeback
# cellcli -e "alter cell flashCacheMode=writeback"
Cell exadata01cell01 successfully altered
Step 6. Restart the cellsrv service
# cellcli -e alter cell startup services cellsrv
Starting CELLSRV services...
The STARTUP of CELLSRV services was successful.
Step 7. Reactivate the griddisks on the cell
# cellcli –e alter griddisk all active
GridDisk DATAC1_CD_00_exadata01cell03 successfully altered
GridDisk DATAC1_CD_01_exadata01cell03 successfully altered
GridDisk DATAC1_CD_02_exadata01cell03 successfully altered
GridDisk DATAC1_CD_03_exadata01cell03 successfully altered
GridDisk DATAC1_CD_04_exadata01cell03 successfully altered
GridDisk DATAC1_CD_05_exadata01cell03 successfully altered
GridDisk DBFS_DG_CD_02_exadata01cell03 successfully altered
GridDisk DBFS_DG_CD_03_exadata01cell03 successfully altered
GridDisk DBFS_DG_CD_04_exadata01cell03 successfully altered
GridDisk DBFS_DG_CD_05_exadata01cell03 successfully altered
GridDisk RECOC1_CD_00_exadata01cell03 successfully altered
GridDisk RECOC1_CD_01_exadata01cell03 successfully altered
GridDisk RECOC1_CD_02_exadata01cell03 successfully altered
GridDisk RECOC1_CD_03_exadata01cell03 successfully altered
GridDisk RECOC1_CD_04_exadata01cell03 successfully altered
GridDisk RECOC1_CD_05_exadata01cell03 successfully altered
Step 8. Verify all grid disks have been successfully put online using the following command:
# cellcli -e list griddisk attributes name, asmmodestatus
DATAC1_CD_00_exadata01cell02 ONLINE Yes
DATAC1_CD_01_exadata01cell02 ONLINE Yes
DATAC1_CD_02_exadata01cell02 ONLINE Yes
DATAC1_CD_03_exadata01cell02 ONLINE Yes
DATAC1_CD_04_exadata01cell02 ONLINE Yes
DATAC1_CD_05_exadata01cell02 ONLINE Yes
DBFS_DG_CD_02_exadata01cell02 ONLINE Yes
DBFS_DG_CD_03_exadata01cell02 ONLINE Yes
DBFS_DG_CD_04_exadata01cell02 ONLINE Yes
DBFS_DG_CD_05_exadata01cell02 ONLINE Yes
RECOC1_CD_00_exadata01cell02 ONLINE Yes
RECOC1_CD_01_exadata01cell02 ONLINE Yes
RECOC1_CD_02_exadata01cell02 ONLINE Yes
RECOC1_CD_03_exadata01cell02 ONLINE Yes
RECOC1_CD_04_exadata01cell02 ONLINE Yes
RECOC1_CD_05_exadata01cell02 ONLINE Yes
Step 9. Recreate the flash cache
# cellcli -e create flashcache all
Flash cache exadata01cell01_FLASHCACHE successfully created
If the flash disk is used for flash cache, then the effective cache size increases. If the flash disk is used for grid disks, then the grid disks are re-created on the new flash disk. If those gird disks were part of an Oracle ASM disk group, then they are added back to the disk group, and the data is rebalanced on them based on the disk group redundancy and ASM_POWER_LIMIT parameter.
Step 10. Check the status of the cell to confirm that it's now in WriteBack mode:
# cellcli -e list cell detail | grep flashCacheMode
flashCacheMode: WriteBack
Step 11. Repeat these same steps again on the next cell to the FINAL cell. However, before taking another storage server offline, execute the following making sure 'asmdeactivationoutcome' displays YES:
# cellcli -e list griddisk attributes name,asmmodestatus, asmdeactivationoutcome
DATAC1_CD_00_exadata01cell01 ONLINE Yes
DATAC1_CD_01_exadata01cell01 ONLINE Yes
DATAC1_CD_02_exadata01cell01 ONLINE Yes
DATAC1_CD_03_exadata01cell01 ONLINE Yes
DATAC1_CD_04_exadata01cell01 ONLINE Yes
DATAC1_CD_05_exadata01cell01 ONLINE Yes
DBFS_DG_CD_02_exadata01cell01 ONLINE Yes
DBFS_DG_CD_03_exadata01cell01 ONLINE Yes
DBFS_DG_CD_04_exadata01cell01 ONLINE Yes
DBFS_DG_CD_05_exadata01cell01 ONLINE Yes
RECOC1_CD_00_exadata01cell01 ONLINE Yes
RECOC1_CD_01_exadata01cell01 ONLINE Yes
RECOC1_CD_02_exadata01cell01 ONLINE Yes
RECOC1_CD_03_exadata01cell01 ONLINE Yes
RECOC1_CD_04_exadata01cell01 ONLINE Yes
RECOC1_CD_05_exadata01cell01 ONLINE Yes
After changing the flashcache modes on all cells, check if flashcache modes are changed to write-back for all cells.
CellCLI> dcli -g ~/cell_group -l root cellcli -e "list cell attributes flashcachemode"
exadata01cell01: WriteBack
exadata01cell02: WriteBack
exadata01cell03: WriteBack
2. Non-Rolling Method:
(Assuming that RDBMS & ASM instances are DOWN while enabling Write-Back Flash Cache)
Step 1. Drop the flash cache on that cell
# cellcli -e drop flashcache
Step 2. Shut down cellsrv service
# cellcli -e alter cell shutdown services cellsrv
Step 3. Set the cell flashcache mode to writeback
# cellcli -e "alter cell flashCacheMode=writeback"
Step 4. Restart the cellsrv service
# cellcli -e alter cell startup services cellsrv
Step 5. Recreate the flash cache
# cellcli -e create flashcache all
Write-Back Flash Cache Not Required for DiskGroup:
Note: We can disable Write-Back Flash Cache diskgroups like RECO not requiring this feature. This can save space in the flash cache.
CACHINGPOLICY could be used to change the flash cache policy of the griddisk.
Before changing the cache policy from default to none, ensure there is no cached data in flash cache for the grid disk:
CellCLI> create griddisk all harddisk prefix=RECO, size=1006, cachingPolicy="none“;
OR
CELLCLI>ALTER GRIDDISK grid_disk_name FLUSH;
CELLCLI>ALTER GRIDDISK grid_disk_name CACHINGPOLICY="none";
Flushing the data from Flash Cache to Disk – Manual Method:
The data which is not been synchronized with griddisk can be synchronized using the FLUSH option.
CELLCLI>ALTER GRIDDISK grid_disk_name FLUSH
Use the following command to check the progress of this activity:
CELLCLI>LIST GRIDDISK ATTRIBUTES name, flushstatus, flusherr
Reinstating WriteThrough FlashCache:
1. To reinstate Writethrough caching, FlashCache must first be flushed.
2. FlashCache must then be dropped and cellsrv stopped.
Step 1. CELLCLI> alter flashcache all flush
Step 2. CELLCLI> drop flashcache
Step 3. CELLCLI> alter cell shutdown services cellsrv
Step 4. CELLCLI> alter cell flashCacheMode = WriteThrough
Step 5. CELLCLI> alter cell startup services cellsrv
Monitoring Flash Cache Usage:
CELLCLI> list metricdefinition attributes name, description where name like '.*_DIRTY‘
|
Number of unflushed bytes cached in FLASHCACHE on a cell disk
|
|
Number of unflushed bytes in FlashCache
|
|
Number of unflushed bytes in FlashCache which cannot be flushed. Because cached disks are not accessible
|
|
Number of unflushed bytes cached in FLASHCACHE for a grid disk
|