How Disk Multipath works on Linux

Linux Multipath command is used to manage storage SAN (storage area network) disks on OS side. Linux multipath provides a way of organizing the I/O paths logically, by creating a single multipath device on top of the underlying devices.

• Start multipath on Linux
• List multipath devices on Linux
• Get Disk WWID ( SCSI ID ) on Linux
• List of disks mapped via HBA on Linux box
• How multipath works in Linux?

Start multipath on Linux

We need to install the following two rpms.

device-mapper-multipath
device-mapper

start multipathd service with service command or systemctl command

service multipathd start

To make multipath work, we need make sure below module is loaded

# lsmod | grep dm_multipath
dm_multipath 27427 4 dm_round_robin,dm_service_time
# modinfo dm_multipath
filename: /lib/modules/3.10.0-693.21.1.el7.x86_64/kernel/drivers/md/dm-multipath.ko.xz
license: GPL
author: Sistina Software
description: device-mapper multipath target
retpoline: Y
rhelversion: 7.4
srcversion: 063067F9F167E7B653A4773
depends: dm-mod
intree: Y
vermagic: 3.10.0-693.21.1.el7.x86_64 SMP mod_unload modversions
signer: Red Hat Enterprise Linux kernel signing key
sig_key: B1:C3:31:09:FB:DA:94:AD:2F:E1:E8:E5:C1:E5:52:BD:22:57:60:FE
sig_hashalgo: sha256

List multipath devices on Linux

We can use this command to list all the multipath devices on Linux.

# multipath -l

Each multipath device has a World Wide Identifier (WWID), which is guaranteed to be globally unique and unchanging.

We can get the WWID using below commands.

# multipath -l | grep dm
3624a937079ebbf3c903141360001149b dm-5 PURE ,FlashArray
3624a937079ebbf3c903141360001149a dm-14 PURE ,FlashArray

OR using below command

# ls -ld /dev/disk/by-id/scsi-*
lrwxrwxrwx 1 root root 9 May 22 18:29 /dev/disk/by-id/scsi-3600c0ff00013b88f9058e35a01000000 -> ../../sde
lrwxrwxrwx 1 root root 9 May 22 18:29 /dev/disk/by-id/scsi-3600c0ff00013b88f9c58e35a01000000 -> ../../sdd

Get the WWID ( SCSI ID ) of disks on Linux

Each LUN is mapped to a disk on Linux. There is a unique WWID generated for each disk. This ID can be used for representing them in the configuration as the device name may change post reboot while the WWID will not change and are reboot persistent.

# lsscsi –scsi_id
3624a937079ebbf3c903141360001149c dm-2 PURE ,FlashArray
size=1.0T features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’queue-length 0′ prio=1 status=active
|- 13:0:0:249 sdo 8:224 active ready running
|- 11:0:0:249 sdr 65:16 active ready running
|- 11:0:1:249 sdas 66:192 active ready running
`- 13:0:1:249 sdan 66:112 active ready running

To get the WWID of the disks we can also use below command

# /usr/lib/udev/scsi_id -g -u -d /dev/sda
3600c0ff00013b88f9058e35a01000000

List of disks mapped via HBA on Linux box

A node with two HBAs attached to a storage controller with two ports sees four devices: /dev/sda, /dev/sdb, dev/sdc, and /dev/sdd. DM Multipath creates a single device with a unique WWID that reroutes I/O to those four underlying devices according to the multipath configuration.

In our case, we have 1 LUN on the storage side connected to the Linux box hence we see 4 devices

# ls -ld /sys/block/sd*/device
lrwxrwxrwx 1 root root 0 May 22 18:29 /sys/block/sda/device -> ../../../0:0:0:0
lrwxrwxrwx 1 root root 0 May 22 18:29 /sys/block/sdb/device -> ../../../0:0:0:1
lrwxrwxrwx 1 root root 0 May 22 18:29 /sys/block/sdc/device -> ../../../0:0:1:0
lrwxrwxrwx 1 root root 0 May 22 18:29 /sys/block/sdd/device -> ../../../0:0:1:1

we can also get the underlying disks this way.

# ls -l /dev/disk/by-path/
total 0
lrwxrwxrwx 1 root root 9 May 22 18:29 pci-0000:04:00.2-fc-0x207000c0ff13d3a7-lun-0 -> ../../sda
lrwxrwxrwx 1 root root 9 May 22 18:29 pci-0000:04:00.2-fc-0x207000c0ff13d3a7-lun-1 -> ../../sdb
lrwxrwxrwx 1 root root 9 May 22 18:29 pci-0000:04:00.2-fc-0x247000c0ff13d3a7-lun-0 -> ../../sdc
lrwxrwxrwx 1 root root 9 May 22 18:29 pci-0000:04:00.2-fc-0x247000c0ff13d3a7-lun-1 -> ../../sdd

How multipath works in Linux?

In the multipath output, we can see that this device dm-9 has four underlying disks.

# multipath -l
3624a937079ebbf3c903141360001149c dm-9 PURE ,FlashArray
size=1.0T features=’0′ hwhandler=’0′ wp=rw
`-+- policy=’queue-length 0′ prio=1 status=active
|- 11:0:0:249 sdo 8:224 active ready running
|- 11:0:1:249 sdao 66:128 active ready running
|- 13:0:0:249 sdr 65:16 active ready running
`- 13:0:1:249 sdas 66:192 active ready running

From the output, we can see that all the disks are active and ready. That means all the disks are available for IO on OS side. That also means that the storage we use is an active-active product. 11/13 means the HBA number on OS side. 0 1 means the controller ID on the storage side. 249 is the LUN ID on the storage side.

Below value explains what each field means

1:0:0:1

^ ^ ^ ^

| | | |

H C T L

Where H is the HBA number, C is the channel on the HBA, T is the SCSI target ID, and L is the LUN from the Storage.