HP-UX 11i v3 DS2500 write performance optimization

From Wiki-UX.info
Jump to: navigation, search


Abstract[edit]

On direct attached Fibre Channel storage subsystems like HP StorageWorks 2500 Disk System (DS2500) with two initiators, therefore two Lunpath H/W Path, can manifest heavy write delays, specially where LVM extent-based distribution is used, because HP-UX 11i v3 I/O load balancing policy defaults to round-robin, that is bias to performe better on active / active initiators.

This article present results obtained on a real-live system with configuration details and basic performance measurement methodology. Results may differ between different systems and configurations, but works as a starting point for evaluating the problem.

Configuration[edit]

The HP-UX 11i v3 (11.31) server is an rx4640 with 2x Intel(R) Itanium 2 processors (1.6 GHz, 9 MB). Volume group configuration is made out of 4 disk using default 4 MB extent size. There are two PVG groups made out of two disk each, that corresponds to the extent distribution and it's mirrors. The logical volume /dev/vg01/lvol1 has been lvsplit-ed with the purpose of measuring I/O read and write rates per distribution set and across the sets.

The system has 2x HP Tachyon XL2 Fibre Channel Mass Storage Adapters, on a private loop with a HP StorageWorks 2500 Disk System (DS2500).

UFC kernel parameters

  • filecache_max: 2035108862 (25%)
  • filecache_min: 81404354 (1%)

LVM Configuration[edit]

1. Volume group configuration

# vgdisplay -v /dev/vg01
--- Volume groups ---
VG Name                     /dev/vg01
VG Write Access             read/write
VG Status                   available
Max LV                      255
Cur LV                      2
Open LV                     2
Max PV                      16
Cur PV                      4
Act PV                      4
Max PE per PV               17501
VGDA                        8
PE Size (Mbytes)            4
Total PE                    70004
Alloc PE                    70004
Free PE                     0
Total PVG                   2
Total Spare PVs             0
Total Spare PVs in use      0
VG Version                  1.0
VG Max Size                 1120064m
VG Max Extents              280016

   --- Logical volumes ---
   LV Name                     /dev/vg01/lvol1
   LV Status                   available/syncd
   LV Size (Mbytes)            140008
   Current LE                  35002
   Allocated PE                35002
   Used PV                     2

   LV Name                     /dev/vg01/lvol1b
   LV Status                   available/syncd
   LV Size (Mbytes)            140008
   Current LE                  35002
   Allocated PE                35002
   Used PV                     2


   --- Physical volumes ---
   PV Name                     /dev/disk/disk22
   PV Status                   available
   Total PE                    17501
   Free PE                     0
   Autoswitch                  On
   Proactive Polling           On

   PV Name                     /dev/disk/disk23
   PV Status                   available
   Total PE                    17501
   Free PE                     0
   Autoswitch                  On
   Proactive Polling           On

   PV Name                     /dev/disk/disk26
   PV Status                   available
   Total PE                    17501
   Free PE                     0
   Autoswitch                  On
   Proactive Polling           On

   PV Name                     /dev/disk/disk27
   PV Status                   available
   Total PE                    17501
   Free PE                     0
   Autoswitch                  On
   Proactive Polling           On


   --- Physical volume groups ---
   PVG Name                    PVG1
   PV Name                     /dev/disk/disk22
   PV Name                     /dev/disk/disk23

   PVG Name                    PVG2
   PV Name                     /dev/disk/disk26
   PV Name                     /dev/disk/disk27

2. Extent-based distributed logical volume

# lvidisplay -v /dev/vg01/lvol1
--- Logical volumes ---
LV Name                     /dev/vg01/lvol1
VG Name                     /dev/vg01
LV Permission               read/write
LV Status                   available/syncd
Mirror copies               0
Consistency Recovery        MWC
Schedule                    parallel
LV Size (Mbytes)            140008
Current LE                  35002
Allocated PE                35002
Stripes                     0
Stripe Size (Kbytes)        0
Bad block                   on
Allocation                  PVG-strict/distributed
IO Timeout (Seconds)        default

   --- Distribution of logical volume ---
   PV Name                 LE on PV  PE on PV
   /dev/disk/disk22        17501     17501
   /dev/disk/disk23        17501     17501

   --- Logical extents ---
   LE    PV1                     PE1   Status 1
   00000 /dev/disk/disk22        00000 current
   00001 /dev/disk/disk23        00000 current
   00002 /dev/disk/disk22        00001 current
   00003 /dev/disk/disk23        00001 current
   00004 /dev/disk/disk22        00002 current
   00005 /dev/disk/disk23        00002 current
...

I/O Configuration[edit]

# ioscan -funC fc
Class     I  H/W Path  Driver S/W State   H/W Type     Description
===============================================================
fc        0  0/3/1/0   td   CLAIMED     INTERFACE    HP Tachyon XL2 Fibre Channel Mass Storage Adapter
                      /dev/td0
fc        1  0/3/2/0   td   CLAIMED     INTERFACE    HP Tachyon XL2 Fibre Channel Mass Storage Adapter
                      /dev/td1
# fcmsutil /dev/td0

                           Vendor ID is = 0x00103c
                           Device ID is = 0x001029
                XL2 Chip Revision No is = 2.3
            PCI Sub-system Vendor ID is = 0x00103c
                   PCI Sub-system ID is = 0x00128c
                               Topology = PRIVATE_LOOP
                             Link Speed = 2Gb
                     Local N_Port_id is = 0x000001
                       Local Loop_id is = 125
            N_Port Node World Wide Name = 0x50060b000011a7b5
            N_Port Port World Wide Name = 0x50060b000011a7b4
                           Driver state = ONLINE
                       Hardware Path is = 0/3/1/0
                 Number of Assisted IOs = 4014135
        Number of Active Login Sessions = 8
                   Dino Present on Card = NO
                     Maximum Frame Size = 2048
                         Driver Version = @(#) td - HP Fibre Channel Tachyon TL/TS/XL2 Driver
B.11.31.0809 /ux/core/isu/TL/kern/src/common/td/td_drvinit.c: Jun 15 2008, 22:41:01
# fcmsutil /dev/td1

                           Vendor ID is = 0x00103c
                           Device ID is = 0x001029
                XL2 Chip Revision No is = 2.3
            PCI Sub-system Vendor ID is = 0x00103c
                   PCI Sub-system ID is = 0x00128c
                               Topology = PRIVATE_LOOP
                             Link Speed = 2Gb
                     Local N_Port_id is = 0x000001
                       Local Loop_id is = 125
            N_Port Node World Wide Name = 0x50060b0000116671
            N_Port Port World Wide Name = 0x50060b0000116670
                           Driver state = ONLINE
                       Hardware Path is = 0/3/2/0
                 Number of Assisted IOs = 4014016
        Number of Active Login Sessions = 8
                   Dino Present on Card = NO
                     Maximum Frame Size = 2048
                         Driver Version = @(#) td - HP Fibre Channel Tachyon TL/TS/XL2 Driver
B.11.31.0809 /ux/core/isu/TL/kern/src/common/td/td_drvinit.c: Jun 15 2008, 22:41:01
# ioscan -m dsf
Persistent DSF           Legacy DSF(s)
========================================
...
/dev/rdisk/disk22        /dev/rdsk/c4t6d0
                         /dev/rdsk/c6t6d0
/dev/rdisk/disk23        /dev/rdsk/c4t7d0
                         /dev/rdsk/c6t7d0
...
/dev/rdisk/disk26        /dev/rdsk/c4t13d0
                         /dev/rdsk/c6t13d0
/dev/rdisk/disk27        /dev/rdsk/c4t14d0
                         /dev/rdsk/c6t14d0
...
# ioscan -NfnkC disk
Class     I  H/W Path  Driver S/W State   H/W Type     Description
===================================================================
...
disk     22  64000/0xfa00/0x3  esdisk   CLAIMED     DEVICE       HPQ     BF07258243
                      /dev/disk/disk22   /dev/rdisk/disk22
disk     23  64000/0xfa00/0x4  esdisk   CLAIMED     DEVICE       HPQ     BF07258243
                      /dev/disk/disk23   /dev/rdisk/disk23
...
disk     26  64000/0xfa00/0x7  esdisk   CLAIMED     DEVICE       HPQ     BF07258243
                      /dev/disk/disk26   /dev/rdisk/disk26
disk     27  64000/0xfa00/0x8  esdisk   CLAIMED     DEVICE       HPQ     BF07258243
                      /dev/disk/disk27   /dev/rdisk/disk27
# ioscan -fnkC disk
Class     I  H/W Path       Driver S/W State   H/W Type     Description
=======================================================================
disk     56  0/3/1/0.8.0.255.4.6.0   sdisk   CLAIMED     DEVICE       HPQ     BF07258243
                           /dev/dsk/c4t6d0   /dev/rdsk/c4t6d0
disk     54  0/3/1/0.8.0.255.4.7.0   sdisk   CLAIMED     DEVICE       HPQ     BF07258243
                           /dev/dsk/c4t7d0   /dev/rdsk/c4t7d0
...
disk      7  0/3/1/0.8.0.255.4.13.0  sdisk   CLAIMED     DEVICE       HPQ     BF07258243
                           /dev/dsk/c4t13d0   /dev/rdsk/c4t13d0
disk      8  0/3/1/0.8.0.255.4.14.0  sdisk   CLAIMED     DEVICE       HPQ     BF07258243
                           /dev/dsk/c4t14d0   /dev/rdsk/c4t14d0
...
disk     11  0/3/2/0.8.0.255.4.6.0   sdisk   CLAIMED     DEVICE       HPQ     BF07258243
                           /dev/dsk/c6t6d0   /dev/rdsk/c6t6d0
disk     12  0/3/2/0.8.0.255.4.7.0   sdisk   CLAIMED     DEVICE       HPQ     BF07258243
                           /dev/dsk/c6t7d0   /dev/rdsk/c6t7d0
...
disk     15  0/3/2/0.8.0.255.4.13.0  sdisk   CLAIMED     DEVICE       HPQ     BF07258243
                           /dev/dsk/c6t13d0   /dev/rdsk/c6t13d0
disk     16  0/3/2/0.8.0.255.4.14.0  sdisk   CLAIMED     DEVICE       HPQ     BF07258243
                           /dev/dsk/c6t14d0   /dev/rdsk/c6t14d0
...
# ioscan -m hwpath
Lun H/W Path      Lunpath H/W Path                 Legacy H/W Path
====================================================================
64000/0xfa00/0x3
                  0/3/1/0.0x21000011c69e905f.0x0   0/3/1/0.8.0.255.4.6.0
                  0/3/2/0.0x22000011c69e905f.0x0   0/3/2/0.8.0.255.4.6.0
64000/0xfa00/0x4
                  0/3/1/0.0x21000014c3a0e2a2.0x0   0/3/1/0.8.0.255.4.7.0
                  0/3/2/0.0x22000014c3a0e2a2.0x0   0/3/2/0.8.0.255.4.7.0
...
64000/0xfa00/0x7
                  0/3/1/0.0x21000011c69e731a.0x0   0/3/1/0.8.0.255.4.13.0
                  0/3/2/0.0x22000011c69e731a.0x0   0/3/2/0.8.0.255.4.13.0
64000/0xfa00/0x8
                  0/3/1/0.0x21000011c6ba31e1.0x0   0/3/1/0.8.0.255.4.14.0
                  0/3/2/0.0x22000011c6ba31e1.0x0   0/3/2/0.8.0.255.4.14.0
...

Performance test methodology[edit]

1. Test file creation time without modifiying the multi-path policies. Create a script /tmp/createfiles.shcontaining:

# cat /tmp/createfiles.sh
dd if=/dev/zero of=/data/TCFCU/TRAN.EXT/dat001 bs=64k count=16384
dd if=/dev/zero of=/data/TCFCU/TRAN.EXT/dat002 bs=64k count=16384
dd if=/dev/zero of=/data/TCFCU/TRAN.EXT/dat003 bs=64k count=16384
dd if=/dev/zero of=/data/TCFCU/TRAN.EXT/dat004 bs=64k count=12171
dd if=/dev/zero of=/data/TCFCU/TRAN.EXT/over001 bs=64k count=16384
dd if=/dev/zero of=/data/TCFCU/TRAN.EXT/over002 bs=64k count=10192
dd if=/dev/zero of=/data/TCFCU/TRAN.EXT/over003 bs=64k count=5693
dd if=/dev/zero of=/data/TCFCU/TRAN.EXT/over004 bs=64k count=5494
dd if=/dev/zero of=/data/TCFCU/TRAN.EXT/over005 bs=64k count=5803

2. Execute the script for a first run to determinate how fast can the files been created without reading involved.

# timex sh /tmp/createfiles.sh

3. Test file read access time. Create a script /tmp/readfiles.sh containing:

for file in $(ls /data/TCFCU/TRAN.EXT/*)
do
   echo "Reading $file"
   dd if=$file of=/dev/null bs=64k 2> /dev/null
done

4. Execute the script for a first run to determinate how fast can the files been readed without writing involved.

5. Change the access policy to preferred_path to bias agaist one of the FC HBA per disk:

64000/0xfa00/0x3
                 [0/3/1/0.0x21000011c69e905f.0x0]  0/3/1/0.8.0.255.4.6.0  <-- Preferred_path
                  0/3/2/0.0x22000011c69e905f.0x0   0/3/2/0.8.0.255.4.6.0
64000/0xfa00/0x4
                  0/3/1/0.0x21000014c3a0e2a2.0x0   0/3/1/0.8.0.255.4.7.0
                 [0/3/2/0.0x22000014c3a0e2a2.0x0]  0/3/2/0.8.0.255.4.7.0  <-- Preferred_path
# scsimgr save_attr -D /dev/rdisk/disk22 \
-a load_bal_policy=preferred_path \
-a preferred_path=0/3/1/0.0x21000011c69e905f.0x0
# scsimgr save_attr -D /dev/rdisk/disk23 \
-a load_bal_policy=preferred_path \
-a preferred_path=0/3/2/0.0x22000014c3a0e2a2.0x0

4. Execute both the create and read scripts for a second run to determinate how fast can the files with the policies changed.

# timex sh /tmp/createfiles.sh
# timex sh /tmp/readfiles.sh


Default Roud-robin policy results[edit]

# timex sh createfiles.sh
16384+0 records in
16384+0 records out
16384+0 records in
16384+0 records out
16384+0 records in
16384+0 records out
12171+0 records in
12171+0 records out
16384+0 records in
16384+0 records out
10192+0 records in
10192+0 records out
5693+0 records in
5693+0 records out
5494+0 records in
5494+0 records out
5803+0 records in
5803+0 records out

real     3:36.20
user        0.10
sys         7.51
# timex sh readfiles.sh
Reading /data/TCFCU/TRAN.EXT/dat001
Reading /data/TCFCU/TRAN.EXT/dat002
Reading /data/TCFCU/TRAN.EXT/dat003
Reading /data/TCFCU/TRAN.EXT/dat004
Reading /data/TCFCU/TRAN.EXT/over001
Reading /data/TCFCU/TRAN.EXT/over002
Reading /data/TCFCU/TRAN.EXT/over003
Reading /data/TCFCU/TRAN.EXT/over004
Reading /data/TCFCU/TRAN.EXT/over005

real       10.82
user        0.09
sys         6.60

Preferred path policy[edit]

# scsimgr save_attr -D /dev/rdisk/disk22 -a load_bal_policy=preferred_path -a preferred_path=0/3/1/0.0x21000011c69e905f.0x0
Value of attribute load_bal_policy saved successfully
Value of attribute preferred_path saved successfully

# scsimgr save_attr -D /dev/rdisk/disk23 -a load_bal_policy=preferred_path -a preferred_path=0/3/2/0.0x22000014c3a0e2a2.0x0
Value of attribute load_bal_policy saved successfully
Value of attribute preferred_path saved successfully
# timex sh createfiles.sh
16384+0 records in
16384+0 records out
16384+0 records in
16384+0 records out
16384+0 records in
16384+0 records out
12171+0 records in
12171+0 records out
16384+0 records in
16384+0 records out
10192+0 records in
10192+0 records out
5693+0 records in
5693+0 records out
5494+0 records in
5494+0 records out
5803+0 records in
5803+0 records out

real       38.84
user        0.10
sys         7.94
# timex sh readfiles.sh
Reading /data/TCFCU/TRAN.EXT/dat001
Reading /data/TCFCU/TRAN.EXT/dat002
Reading /data/TCFCU/TRAN.EXT/dat003
Reading /data/TCFCU/TRAN.EXT/dat004
Reading /data/TCFCU/TRAN.EXT/over001
Reading /data/TCFCU/TRAN.EXT/over002
Reading /data/TCFCU/TRAN.EXT/over003
Reading /data/TCFCU/TRAN.EXT/over004
Reading /data/TCFCU/TRAN.EXT/over005

real       19.58
user        0.09
sys         6.33

Conclusions[edit]

  1. I/O load balancing policy has a major impact in the write and read performance of direct attached fibre channel JBOD (like DS2500), specially under extent-based distributions were alternate extents can be directed associated with alternate fibre channel adapters and their paths (preferred path).
  2. Write performance can be substantially improved using the preferred_path policy. Testing shows that defaults configuration requires 3:36.20 minutes , that can be decreased to 38.84 seconds. This may play a major impact on applications with a higher write to read relationship.
  3. Read performance may be impacted using the preferred_path policy. Testing shows 10.82 seconds increased to 19.58 seconds. This may play a minor impact on applications with a higher read to write relationship. Read data is not conclusive, since these low values are mostly affected by the size of the Unified File Cache (UFC).

Reference[edit]

Authors[edit]