How to Configure NVMe over TCP (NVMe/TCP) on Oracle Linux 9

This blog on configuring NVMe over TCP on Oracle Linux 9 originally appeared on Ron Ekins’s blog. It has been republished with the author’s credit and consent.

Over the recent months, I’ve had an increasing number of discussions around the use of Non-volatile Memory Express over TCP (NVMe/TCP) storage and Oracle, so I thought it was time that I documented how to configure NVMe/TCP on Oracle Linux 9 for Oracle Database 19c.

Is this NVMe over TCP supported?

Yes, Oracle supports NVMe over Fabrics (NVMe-oF) storage devices for Oracle Database 19c binaries, datafiles, and recovery files.

See supported storage options for Oracle Database 19c for the latest information.

Note: In Oracle Database 19c, NVMe-oF uses the kernel initiator, which exposes the disks as block devices. This changes with Oracle AI Database 26ai, so expect another blog post on the GA of on-premises 26ai.

For this blog, I’ll use a two-node Oracle Database 19c RAC Cluster running Oracle Linux 9 with NVMe storage presented over TCP from a Everpure FlashArray™ system.

Linux Server

Let’s start by confirming the version of Oracle Linux we’re using:

Install NVMe-CLI

On each RAC node, install the nvme-cli NVMe CLI utility:

# dnf install nvme-cli

Identify Host NQN

Identify the host NVMe Qualified Name (NQN) for each Oracle Database RAC node using nvme show-hostnqn. For example:

Alternatively, you can grab the NQN in the /etc/nvme/hostnqn file created when you installed nvme-cli.

Loading NVMe-TCP Kernel Module

Install and confirm the NVMe-TCP module is installed on each RAC node:

To ensure the NVMe-TCP module is loaded automatically after a host, reboot and add nvme_tcp to /etc/modules-load.d/nvme-tcp.conf:

Everpure FlashArray

Log in to the FlashArray WebUI and create a new host by navigating to Storage > Hosts, and then click + to create a host.

Create Host

Now, click on Configure NQNs from the Hosts Ports panel.

Provide the NQN and details previously obtained using nvme show-hostnqn, and repeat for all RAC nodes.

Configure NVMe-oF NQNs

Now, create a host group and add member hosts, in this example, the Oracle RAC nodes.

From the above, we can see the interface is reported as NVMe-oF.

Create required volumes, providing meaningful names and size.

Return to Storage > Hosts, select Host Group, and connect the newly created volumes.

FlashArray NVMe/TCP Volumes

Log in to the FlashArray CLI and use purenetwork etc list –service nvme-tcp to list the FlashArray system’s NVMe/TCP addresses. For example:

Use purevol list <volume name> –human-readable –total to list the volumes and serial numbers. For example:     

Linux NVMe Configuration

NVMe Discover

Returning to the Oracle Database server, perform nvme discover using the IPs obtained with purenetwork eth list –service nvme-tcp to discover the available subsystems on the NVMe controller.

For example:

# nvme discover –transport tcp –traddr 192.168.130.10 –trsvcid 4420 | grep -E ‘traddr|subnqn|Entry’

NVMe Connect

Now that we have the confirmed connectivity, we can connect using nvme connect or nvme connect-all.

NVMe connect example:

Alternatively, here’s an example using NVMe connect-all:

Verification

Verify that the expected paths to the array have been established with nvme list-subsys. For example:

NVMe List Controllers

Verify that the volumes are exposed on each of the paths using nvme list. For example:

Use nvme id-ctrl to send an identify controller command to a specified NVMe device to return controller capabilities, features, and identification data, such as serial number (sn), model number (mn), and firmware revision (fr). For example:

NVMe Persistence

Create a file /opt/nvme_connect.sh, providing the nvme connect commands previously used.

And now create a file /etc/systemd/system/nvme_fabrics.persistent.service:

Remember to chmod (755) the above to make the files executable before moving on.

NVMe Persistence Test

Disconnect the current nvme devices using the nvme-cli command nvme disconnect. For example:

Now, try to start the new service with systemctl start:

Confirm the service started ok with systemctl status:

If OK, enable the service with systemctl enable.

Multipathing

My Oracle RAC lab servers are currently configured to use device-mapper (DM) multipathing rather than native multipathing. So to avoid any multipathing issues, I’ll disable native multipathing.

Let’s check to see if native NVMe multipath is enabled by checking /sys/module/nvme_core/parameters/multipath. If “Y,” it’s enabled.

If enabled “Y,” disable using grubby. The change also needs to be copied to the boot filesystem so that the parameter will be set when nvme_core is loaded at boot time.

Before we reboot, check that the above has been successful using grubby –info=DEFAULT.

How to Map an EUI to FlashArray UID

The way to map an EUI to a FlashArray NVMe-oF-based volume is by using the Namespace Globally Unique Identifier (NGUID) for every volume presented to our database servers.

After a reboot, perform a multipath -l and identify the NVMe disks. For example:

From the above, we can see that NVMe devices return an EUI-64 (Extended Unique Identifier) value rather than WWID (World Wide Identifier).

Note: This does not follow the WWID format of including the Everpure vendor prefix of “3624a9370.”

For the Everpure FlashArray, an NGUID is broken into three parts, per the NVM Express Base Specification:

• Identifier Extension (First 7 bytes of the FlashArray ID)
• IEEE Company_id (Everpure unique ID)
• Vendor Specific Extension Identifier (Last 5 bytes of the FlashArray volume)

Breaking down: eui.006c1b16ce1c034d24a9371c05569b29

Everpure ID

The first 8 bytes will contain a leading “00,” followed by the first 7 bytes of FlashArray ID. This can be seen using the Pure CLI purearray list command. For example:

Or alternatively, you can use Pure UI or the purevol list <volume name> command. For example:

Array ID

The next 3 bytes of the string are the Unique Company ID for Everpure (24a937).

FA SN / UUID

The last 5 bytes of the string will be the unique identifier of the individual volume on the FlashArray system.

We can obtain this by again using the purevol list <volume name> command. For example:

Giving us:

Example: eui.006c1b16ce1c034d24a9371c05569b29

DM Multipath

Before we label disks for Oracle ASM with ASMLib v3.1, let’s configure multipathing by adding aliases for our FlashArray volume names for each eui device in /etc/multipath.conf

First, create a device entry within the devices section.

Confirm FlashArray volume names and serial numbers. For example:

And add them to the multipaths section /etc/multipath.conf. For example:

Repeat the above for every node in the Oracle RAC cluster. Then reload the multipath configuration using multipath -r or systemctl restart multipathd.

If we repeat the multipath -l:

Summary

In this post, I’ve shared how to configure an Oracle Linux 9.6 server to use NVMe block storage over TCP (NVMe/TCP) presented from a Everpure FlashArray system.

I’ve also shared how to map the NVMe devices back to FlashArray volumes and how to configure device-mapper (DM) multipath alias names to provide descriptive, meaningful device names.

If you have completed the steps above and are looking to use your new NVMe FlashArray volumes with Oracle Database 19c, check out my blog post: “Installation and Configuration of the New Oracle ASMLib v3.1 on Oracle Linux 9.”