Virtualization of NFS storage: Create and Manage a tablespace that uses NFS mounted file system file

Direct NFS is a new feature introduced with Oracle 11g and is an optimized NFS (Network File System) client that provides faster and more scalable access to NFS storage located on NAS storage devices (accessible over TCP/IP). Direct NFS is built directly into the database kernel - just like ASM which is mainly used when using DAS or SAN storage. Direct NFS provides faster performance that what can be provided by the operating system's NFS driver as Oracle bypasses the operating system and generates exactly the requests it needs (no user configuration or tuning required). Data is cached just once in user space, which saves memory (no second copy in kernel space). Performance is further improved by load balancing across multiple network interfaces (if available). Direct NFS is provided as part of the database kernel, and is thus available on all supported database platforms.

In this post I plan to practice to set up a data file on NFS.

Create NFS Volume and Share

For this practice I chosen Open Source Storage - FreeNAS (http://www.freenas.org)

Install and setup FreeNAS on the Linux platform

For this virtual simulation I have chosen VirtualBox (https://www.virtualbox.org) where I have created two Hard Disks:

• FreeNAS.vdi - 4GB
• NFSDATA.vdi - 4GB

I have created new Virtual Machine with following options:

• Name: FreeNAS
• Operating system: BSD
• Version: FreeBSD or FreeBSD (64 bit)
• Memory: 1024MB
• Virtual Hard Disk: Create new hard disk
• File type: I have chosen VDI
• Virtual disk storage details: Dynamically allocated
• Location: it's up to you
• Size: again it's up to you

Additionally is necessary to create new disk, which will be shared for other virtual machines. In settings for new created virtual machine (FreeNAS):

• Storage tab: add new IDE controller - Create new disk (the same steps as was in the previous step for creating new disk. Be aware to have this disk with the size 4GB or bigger. I have chosen the name NFSDATA.vdi)

Installation
Start new created virtual machine FreeNAS with two hard disks (FreeNAS.vdi and NFSDATA.vdi) and boot downloaded installation of FreeNAS:

Reboot and unmount the installation disk.

Configure network interface

• Select first option: 1) Configure Network Interfaces
  
• Select an interface: 1)em0
  
• Delete existing config? n
• Configure interface for DHCP? n
• Configure IPv4? y
• Interface name: lan1
• IPv4 Address: 192.168.12.101/24
• configure IPv6? n
• GUI interface for configuration of FreeNAS is on the following IP: 192.168.12.101.
  

Create disc volumes

• Open the GUI interface. IP address is in the previous picture - last two lines - 192.168.12.101.
  
  
• In the top right you see that red "Alert" light blinking. Click it now.
  
  It means that you should change the password for the admin user.
• Create group:
  In the left menu expand the "Groups" and click on the "Add Group"
  
  In the case, that you want to simulate NFS mounted filesystem with the Oracle database in the other Virtual machine, the best solution is to choose the same uid/gid as you have for the user/group oracle/dba or oinstall (user and group under who you have installed the Oracle software).
• Create user:
  In the left menu expand the "Users" and click on the "Add User".
  
• Set up disks and file sharing:
  FreeNAS supports two types of file systems and three sharing protocols:
  • UFS - An old stand-by, the Unix File System. Don't get us wrong, it's a solid system; we just don't want to use it here. For one thing, it lacks the volume management, RAID and pooling options we can get with our second option, ZFS.
  • ZFS - The hot new kid on the block. This is the file system we want to use, mostly because of ZFS' data integrity, which promises safer, more resilient storage. Not an insignificant factor if you're trying to preserve a lifetime's worth of photos, or hundreds of tracks that cost you $0.99 apiece. We could spend a great deal of time talking about ZFS, but that's a topic for another day. For now, let's get on with it and talk about FreeNAS' sharing protocols.
  FreeNAS supports three sharing methods:
  • CIFS/SAMBA - Open-source implementation of Microsoft's SMB (Now mainly referred to as CIFS)
  • AFP - Apple Filing Protocol
  • NFS - Network File System
  Now set up our disk volumes: in the left menu expand "Storage" -> "Volumes" and click on the "Create Volume"
  
  After creating new volume, you should see:
  
• Change permission:
  In the left menu expand "Storage" -> "Volumes" ->"/mnt/nfs_share_1" and click on the "Change permission"
  
  Here you can specify the owner of the volume with the read-write permissions.
• Enable services:
  In the left menu expand the "Services" and click on the "Control Services". Here you can enable NFS and SSH.
  
• Make Sharing: In the left menu expand "Sharing" -> "NFS Shares" click on the "Add NFS Share".
  

Now we have prepared the NFS file system. To verify it we can try with "exportfs -v"

Mounting NFS on the client node

• To be able to mount the NFS export, as the root user create an empty directory on each cluster node named /u01/app/nfs_share
• Add the hostname of the shared storage to the /etc/hosts:

  [root@oralin1 ~]# cat /etc/hosts 192.168.12.101 freenas-nas

• Make sure the NFS export is mounted on the cluster nodes during boot time by adding the following line to the "/etc/fstab" file on each cluster node
  

  freenas-nas:/mnt/nfs_share_1 /u01/app/nfs_share nfs rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,vers=3,timeo=600 0 0

• Mount the NFS export by executing the mount /u01/app/nfs_share command on each server
• Check if the NFS export is correctly mounted with the "mount" command
  

  freenas-nas:/mnt/nfs_share_1 on /u01/app/nfs_share type nfs (rw,bg,hard,nointr,rsize=32768,wsize=32768,tcp,actimeo=0,nfsvers=3,timeo=600,addr=192.168.12.101)

Configure Direct NFS Client (DNFS)

• Configure oranfstab file
  For reference, Direct NFS searches for mount entries in the following order:
  1. $ORACLE_HOME/dbs/oranfstab
  2. /etc/oranfstab
  3. /etc/mtab
  Configure the oranfstab as like below:

  [oracle@oralin1 ~]$ cat /u01/app/oracle/product/11.1.0/db_1/dbs/oranfstab server: freenas-nas path: 192.168.12.101 local: 192.168.12.11 export: /mnt/nfs_share_1 mount: /u01/app/nfs_share

• Enable the Direct NFS Client ODM Library

  cd $ORACLE_HOME/lib mv libodm11.so libodm11.so_bak ln -s libnfsodm11.so libodm11.so

• Re-start the Database instance
  alert log shows Direct NFS client is enabled by checking the below message

  Oracle instance running with ODM: Oracle Direct NFS ODM Library Version 2.0

• Connect to DB instance and move existing datafiles to NFS drive or create tablespace on NFS drive to check DNFS is working or not

  SYS@ORCL> select * from v$dnfs_servers; ID SVRNAME DIRNAME MNTPORT NFSPORT WTMAX RTMAX ---------- -------------------- -------------------- ---------- ---------- ---------- ---------- 1 freenas-nas /mnt/nfs_share_1 958 2049 32768 32768 SYS@ORCL> create tablespace nfs_test datafile '/u01/app/nfs_share/nfs_test_1.dbf' size 5m; Tablespace created. SYS@ORCL> select * from v$dnfs_files; FILENAME FILESIZE PNUM SVR_ID -------------------------------------------------- ---------- ---------- ---------- /u01/app/nfs_share/nfs_test_1.dbf 5251072 9 1