From Wiki-UX.info

Wiki-UX / Fibre Channel / HP-UX 11i v3 DS2500 write performance optimization
Jump to: navigation, search

HP-UX 11i v3 DS2500 write performance optimization

Abstract

On direct attached Fibre Channel storage subsystems like HP StorageWorks 2500 Disk System (DS2500) with two initiators, therefore two Lunpath H/W Path, can manifest heavy write delays, specially where LVM extent-based distribution is used, because HP-UX 11i v3 I/O load balancing policy defaults to round-robin, that is bias to performe better on active / active initiators.

This article present results obtained on a real-live system with configuration details and basic performance measurement methodology. Results may differ between different systems and configurations, but works as a starting point for evaluating the problem.

Contents


Configuration

The HP-UX 11i v3 (11.31) server is an rx4640 with 2x Intel(R) Itanium 2 processors (1.6 GHz, 9 MB). Volume group configuration is made out of 4 disk using default 4 MB extent size. There are two PVG groups made out of two disk each, that corresponds to the extent distribution and it's mirrors. The logical volume /dev/vg01/lvol1 has been lvsplit-ed with the purpose of measuring I/O read and write rates per distribution set and across the sets.

The system has 2x HP Tachyon XL2 Fibre Channel Mass Storage Adapters, on a private loop with a HP StorageWorks 2500 Disk System (DS2500).

UFC kernel parameters

  • filecache_max: 2035108862 (25%)
  • filecache_min: 81404354 (1%)

LVM Configuration

1. Volume group configuration

# vgdisplay -v /dev/vg01
--- Volume groups ---
VG Name                     /dev/vg01
VG Write Access             read/write
VG Status                   available
Max LV                      255
Cur LV                      2
Open LV                     2
Max PV                      16
Cur PV                      4
Act PV                      4
Max PE per PV               17501
VGDA                        8
PE Size (Mbytes)            4
Total PE                    70004
Alloc PE                    70004
Free PE                     0
Total PVG                   2
Total Spare PVs             0
Total Spare PVs in use      0
VG Version                  1.0
VG Max Size                 1120064m
VG Max Extents              280016
 
   --- Logical volumes ---
   LV Name                     /dev/vg01/lvol1
   LV Status                   available/syncd
   LV Size (Mbytes)            140008
   Current LE                  35002
   Allocated PE                35002
   Used PV                     2
 
   LV Name                     /dev/vg01/lvol1b
   LV Status                   available/syncd
   LV Size (Mbytes)            140008
   Current LE                  35002
   Allocated PE                35002
   Used PV                     2
 
 
   --- Physical volumes ---
   PV Name                     /dev/disk/disk22
   PV Status                   available
   Total PE                    17501
   Free PE                     0
   Autoswitch                  On
   Proactive Polling           On
 
   PV Name                     /dev/disk/disk23
   PV Status                   available
   Total PE                    17501
   Free PE                     0
   Autoswitch                  On
   Proactive Polling           On
 
   PV Name                     /dev/disk/disk26
   PV Status                   available
   Total PE                    17501
   Free PE                     0
   Autoswitch                  On
   Proactive Polling           On
 
   PV Name                     /dev/disk/disk27
   PV Status                   available
   Total PE                    17501
   Free PE                     0
   Autoswitch                  On
   Proactive Polling           On
 
 
   --- Physical volume groups ---
   PVG Name                    PVG1
   PV Name                     /dev/disk/disk22
   PV Name                     /dev/disk/disk23
 
   PVG Name                    PVG2
   PV Name                     /dev/disk/disk26
   PV Name                     /dev/disk/disk27

2. Extent-based distributed logical volume

# lvidisplay -v /dev/vg01/lvol1
--- Logical volumes ---
LV Name                     /dev/vg01/lvol1
VG Name                     /dev/vg01
LV Permission               read/write
LV Status                   available/syncd
Mirror copies               0
Consistency Recovery        MWC
Schedule                    parallel
LV Size (Mbytes)            140008
Current LE                  35002
Allocated PE                35002
Stripes                     0
Stripe Size (Kbytes)        0
Bad block                   on
Allocation                  PVG-strict/distributed
IO Timeout (Seconds)        default
 
   --- Distribution of logical volume ---
   PV Name                 LE on PV  PE on PV
   /dev/disk/disk22        17501     17501
   /dev/disk/disk23        17501     17501
 
   --- Logical extents ---
   LE    PV1                     PE1   Status 1
   00000 /dev/disk/disk22        00000 current
   00001 /dev/disk/disk23        00000 current
   00002 /dev/disk/disk22        00001 current
   00003 /dev/disk/disk23        00001 current
   00004 /dev/disk/disk22        00002 current
   00005 /dev/disk/disk23        00002 current
...

I/O Configuration

# ioscan -funC fc
Class     I  H/W Path  Driver S/W State   H/W Type     Description
===============================================================
fc        0  0/3/1/0   td   CLAIMED     INTERFACE    HP Tachyon XL2 Fibre Channel Mass Storage Adapter
                      /dev/td0
fc        1  0/3/2/0   td   CLAIMED     INTERFACE    HP Tachyon XL2 Fibre Channel Mass Storage Adapter
                      /dev/td1
# fcmsutil /dev/td0
 
                           Vendor ID is = 0x00103c
                           Device ID is = 0x001029
                XL2 Chip Revision No is = 2.3
            PCI Sub-system Vendor ID is = 0x00103c
                   PCI Sub-system ID is = 0x00128c
                               Topology = PRIVATE_LOOP
                             Link Speed = 2Gb
                     Local N_Port_id is = 0x000001
                       Local Loop_id is = 125
            N_Port Node World Wide Name = 0x50060b000011a7b5
            N_Port Port World Wide Name = 0x50060b000011a7b4
                           Driver state = ONLINE
                       Hardware Path is = 0/3/1/0
                 Number of Assisted IOs = 4014135
        Number of Active Login Sessions = 8
                   Dino Present on Card = NO
                     Maximum Frame Size = 2048
                         Driver Version = @(#) td - HP Fibre Channel Tachyon TL/TS/XL2 Driver
B.11.31.0809 /ux/core/isu/TL/kern/src/common/td/td_drvinit.c: Jun 15 2008, 22:41:01
# fcmsutil /dev/td1
 
                           Vendor ID is = 0x00103c
                           Device ID is = 0x001029
                XL2 Chip Revision No is = 2.3
            PCI Sub-system Vendor ID is = 0x00103c
                   PCI Sub-system ID is = 0x00128c
                               Topology = PRIVATE_LOOP
                             Link Speed = 2Gb
                     Local N_Port_id is = 0x000001
                       Local Loop_id is = 125
            N_Port Node World Wide Name = 0x50060b0000116671
            N_Port Port World Wide Name = 0x50060b0000116670
                           Driver state = ONLINE
                       Hardware Path is = 0/3/2/0
                 Number of Assisted IOs = 4014016
        Number of Active Login Sessions = 8
                   Dino Present on Card = NO
                     Maximum Frame Size = 2048
                         Driver Version = @(#) td - HP Fibre Channel Tachyon TL/TS/XL2 Driver
B.11.31.0809 /ux/core/isu/TL/kern/src/common/td/td_drvinit.c: Jun 15 2008, 22:41:01
# ioscan -m dsf
Persistent DSF           Legacy DSF(s)
========================================
...
/dev/rdisk/disk22        /dev/rdsk/c4t6d0
                         /dev/rdsk/c6t6d0
/dev/rdisk/disk23        /dev/rdsk/c4t7d0
                         /dev/rdsk/c6t7d0
...
/dev/rdisk/disk26        /dev/rdsk/c4t13d0
                         /dev/rdsk/c6t13d0
/dev/rdisk/disk27        /dev/rdsk/c4t14d0
                         /dev/rdsk/c6t14d0
...
# ioscan -NfnkC disk
Class     I  H/W Path  Driver S/W State   H/W Type     Description
===================================================================
...
disk     22  64000/0xfa00/0x3  esdisk   CLAIMED     DEVICE       HPQ     BF07258243
                      /dev/disk/disk22   /dev/rdisk/disk22
disk     23  64000/0xfa00/0x4  esdisk   CLAIMED     DEVICE       HPQ     BF07258243
                      /dev/disk/disk23   /dev/rdisk/disk23
...
disk     26  64000/0xfa00/0x7  esdisk   CLAIMED     DEVICE       HPQ     BF07258243
                      /dev/disk/disk26   /dev/rdisk/disk26
disk     27  64000/0xfa00/0x8  esdisk   CLAIMED     DEVICE       HPQ     BF07258243
                      /dev/disk/disk27   /dev/rdisk/disk27
# ioscan -fnkC disk
Class     I  H/W Path       Driver S/W State   H/W Type     Description
=======================================================================
disk     56  0/3/1/0.8.0.255.4.6.0   sdisk   CLAIMED     DEVICE       HPQ     BF07258243
                           /dev/dsk/c4t6d0   /dev/rdsk/c4t6d0
disk     54  0/3/1/0.8.0.255.4.7.0   sdisk   CLAIMED     DEVICE       HPQ     BF07258243
                           /dev/dsk/c4t7d0   /dev/rdsk/c4t7d0
...
disk      7  0/3/1/0.8.0.255.4.13.0  sdisk   CLAIMED     DEVICE       HPQ     BF07258243
                           /dev/dsk/c4t13d0   /dev/rdsk/c4t13d0
disk      8  0/3/1/0.8.0.255.4.14.0  sdisk   CLAIMED     DEVICE       HPQ     BF07258243
                           /dev/dsk/c4t14d0   /dev/rdsk/c4t14d0
...
disk     11  0/3/2/0.8.0.255.4.6.0   sdisk   CLAIMED     DEVICE       HPQ     BF07258243
                           /dev/dsk/c6t6d0   /dev/rdsk/c6t6d0
disk     12  0/3/2/0.8.0.255.4.7.0   sdisk   CLAIMED     DEVICE       HPQ     BF07258243
                           /dev/dsk/c6t7d0   /dev/rdsk/c6t7d0
...
disk     15  0/3/2/0.8.0.255.4.13.0  sdisk   CLAIMED     DEVICE       HPQ     BF07258243
                           /dev/dsk/c6t13d0   /dev/rdsk/c6t13d0
disk     16  0/3/2/0.8.0.255.4.14.0  sdisk   CLAIMED     DEVICE       HPQ     BF07258243
                           /dev/dsk/c6t14d0   /dev/rdsk/c6t14d0
...
# ioscan -m hwpath
Lun H/W Path      Lunpath H/W Path                 Legacy H/W Path
====================================================================
64000/0xfa00/0x3
                  0/3/1/0.0x21000011c69e905f.0x0   0/3/1/0.8.0.255.4.6.0
                  0/3/2/0.0x22000011c69e905f.0x0   0/3/2/0.8.0.255.4.6.0
64000/0xfa00/0x4
                  0/3/1/0.0x21000014c3a0e2a2.0x0   0/3/1/0.8.0.255.4.7.0
                  0/3/2/0.0x22000014c3a0e2a2.0x0   0/3/2/0.8.0.255.4.7.0
...
64000/0xfa00/0x7
                  0/3/1/0.0x21000011c69e731a.0x0   0/3/1/0.8.0.255.4.13.0
                  0/3/2/0.0x22000011c69e731a.0x0   0/3/2/0.8.0.255.4.13.0
64000/0xfa00/0x8
                  0/3/1/0.0x21000011c6ba31e1.0x0   0/3/1/0.8.0.255.4.14.0
                  0/3/2/0.0x22000011c6ba31e1.0x0   0/3/2/0.8.0.255.4.14.0
...

Performance test methodology

1. Test file creation time without modifiying the multi-path policies. Create a script /tmp/createfiles.shcontaining:

# cat /tmp/createfiles.sh
dd if=/dev/zero of=/data/TCFCU/TRAN.EXT/dat001 bs=64k count=16384
dd if=/dev/zero of=/data/TCFCU/TRAN.EXT/dat002 bs=64k count=16384
dd if=/dev/zero of=/data/TCFCU/TRAN.EXT/dat003 bs=64k count=16384
dd if=/dev/zero of=/data/TCFCU/TRAN.EXT/dat004 bs=64k count=12171
dd if=/dev/zero of=/data/TCFCU/TRAN.EXT/over001 bs=64k count=16384
dd if=/dev/zero of=/data/TCFCU/TRAN.EXT/over002 bs=64k count=10192
dd if=/dev/zero of=/data/TCFCU/TRAN.EXT/over003 bs=64k count=5693
dd if=/dev/zero of=/data/TCFCU/TRAN.EXT/over004 bs=64k count=5494
dd if=/dev/zero of=/data/TCFCU/TRAN.EXT/over005 bs=64k count=5803

2. Execute the script for a first run to determinate how fast can the files been created without reading involved.

# timex sh /tmp/createfiles.sh

3. Test file read access time. Create a script /tmp/readfiles.sh containing:

for file in $(ls /data/TCFCU/TRAN.EXT/*)
do
   echo "Reading $file"
   dd if=$file of=/dev/null bs=64k 2> /dev/null
done

4. Execute the script for a first run to determinate how fast can the files been readed without writing involved.

5. Change the access policy to preferred_path to bias agaist one of the FC HBA per disk:

64000/0xfa00/0x3
                 [0/3/1/0.0x21000011c69e905f.0x0]  0/3/1/0.8.0.255.4.6.0  <-- Preferred_path
                  0/3/2/0.0x22000011c69e905f.0x0   0/3/2/0.8.0.255.4.6.0
64000/0xfa00/0x4
                  0/3/1/0.0x21000014c3a0e2a2.0x0   0/3/1/0.8.0.255.4.7.0
                 [0/3/2/0.0x22000014c3a0e2a2.0x0]  0/3/2/0.8.0.255.4.7.0  <-- Preferred_path
# scsimgr save_attr -D /dev/rdisk/disk22 \
-a load_bal_policy=preferred_path \
-a preferred_path=0/3/1/0.0x21000011c69e905f.0x0
# scsimgr save_attr -D /dev/rdisk/disk23 \
-a load_bal_policy=preferred_path \
-a preferred_path=0/3/2/0.0x22000014c3a0e2a2.0x0

4. Execute both the create and read scripts for a second run to determinate how fast can the files with the policies changed.

# timex sh /tmp/createfiles.sh
# timex sh /tmp/readfiles.sh


Default Roud-robin policy results

# timex sh createfiles.sh
16384+0 records in
16384+0 records out
16384+0 records in
16384+0 records out
16384+0 records in
16384+0 records out
12171+0 records in
12171+0 records out
16384+0 records in
16384+0 records out
10192+0 records in
10192+0 records out
5693+0 records in
5693+0 records out
5494+0 records in
5494+0 records out
5803+0 records in
5803+0 records out
 
real     3:36.20
user        0.10
sys         7.51
# timex sh readfiles.sh
Reading /data/TCFCU/TRAN.EXT/dat001
Reading /data/TCFCU/TRAN.EXT/dat002
Reading /data/TCFCU/TRAN.EXT/dat003
Reading /data/TCFCU/TRAN.EXT/dat004
Reading /data/TCFCU/TRAN.EXT/over001
Reading /data/TCFCU/TRAN.EXT/over002
Reading /data/TCFCU/TRAN.EXT/over003
Reading /data/TCFCU/TRAN.EXT/over004
Reading /data/TCFCU/TRAN.EXT/over005
 
real       10.82
user        0.09
sys         6.60

Preferred path policy

# scsimgr save_attr -D /dev/rdisk/disk22 -a load_bal_policy=preferred_path -a preferred_path=0/3/1/0.0x21000011c69e905f.0x0
Value of attribute load_bal_policy saved successfully
Value of attribute preferred_path saved successfully
 
# scsimgr save_attr -D /dev/rdisk/disk23 -a load_bal_policy=preferred_path -a preferred_path=0/3/2/0.0x22000014c3a0e2a2.0x0
Value of attribute load_bal_policy saved successfully
Value of attribute preferred_path saved successfully
# timex sh createfiles.sh
16384+0 records in
16384+0 records out
16384+0 records in
16384+0 records out
16384+0 records in
16384+0 records out
12171+0 records in
12171+0 records out
16384+0 records in
16384+0 records out
10192+0 records in
10192+0 records out
5693+0 records in
5693+0 records out
5494+0 records in
5494+0 records out
5803+0 records in
5803+0 records out
 
real       38.84
user        0.10
sys         7.94
# timex sh readfiles.sh
Reading /data/TCFCU/TRAN.EXT/dat001
Reading /data/TCFCU/TRAN.EXT/dat002
Reading /data/TCFCU/TRAN.EXT/dat003
Reading /data/TCFCU/TRAN.EXT/dat004
Reading /data/TCFCU/TRAN.EXT/over001
Reading /data/TCFCU/TRAN.EXT/over002
Reading /data/TCFCU/TRAN.EXT/over003
Reading /data/TCFCU/TRAN.EXT/over004
Reading /data/TCFCU/TRAN.EXT/over005
 
real       19.58
user        0.09
sys         6.33

Conclusions

  1. I/O load balancing policy has a major impact in the write and read performance of direct attached fibre channel JBOD (like DS2500), specially under extent-based distributions were alternate extents can be directed associated with alternate fibre channel adapters and their paths (preferred path).
  2. Write performance can be substantially improved using the preferred_path policy. Testing shows that defaults configuration requires 3:36.20 minutes , that can be decreased to 38.84 seconds. This may play a major impact on applications with a higher write to read relationship.
  3. Read performance may be impacted using the preferred_path policy. Testing shows 10.82 seconds increased to 19.58 seconds. This may play a minor impact on applications with a higher read to write relationship. Read data is not conclusive, since these low values are mostly affected by the size of the Unified File Cache (UFC).

Reference

Authors

This page was last modified on 15 July 2010, at 03:54. This page has been accessed 2,129 times.