From Wiki-UX.info

Wiki-UX / Logical Volume Manager / How to migrate a LVM volume group to a different backing storage
Jump to: navigation, search

How to migrate a LVM volume group to a different backing storage

Contents

Abstract

This article explains three methods to migrate a LVM volume group to a different backing storage subsystem.

Moving data to different internal disks, disk system (JBOD) or storage attached network (SAN) can represent a serious challange, especially because the size of the current volume group can easily be in the multi-terabyte realms and even larger in the newest storage networks.

Moving data may require hours or even days to complete the task.

Background

Three methods to migrate file system (vxfs, hfs) allocated on LVM volume groups are presented in this article. Migration of logical volumes used as RAW devices is restricted to the first method, because the logical volume size cannot change during migration.

The first method uses Mirror-UX commands to replicate LVM logical volumes to the LUNS in the new storage subsystem. The main advantage of this approach is that access to data structures (filesystems, raw devices) on the logical volumes are not interrupted during the data replication. Nonetheless, performance impact during the data replication should be taken into account because that can lead to intolerable application response time. The main caveat is that all the LVM restrictions (MAX PVs per VG, MAX PEs per PV, PE Size) of the source volume group is inheritated by the new volume group. That can even make imposible data replication in case that MAX PVs per VG are exhausted or not enough to allow extending the volume group.

The second approach is to replicate the data to a new volume group, export both volume groups and import the destiny volume group in place of the the old one. The main advantages of this approach is that the new volume group could be layout with new LVM restrictions and the logical volumes and filesystem can be resized previously to moving the data. All the data is accesible in read / write mode until your start to copy the data and in read only mode during the data replication. After the volume group is exported, it can always be imported back in case of problem with the the data subsystem.

The main caveat is that enough down time should be schedule to actually copy the data from one system to another. This may be unresonable in systems where a small number of large filesystems has been created instead of medium sized filesystems.

The third method is a variation of the second procedure, and is the only alternative when dealing with raw devices. Using the "dd" command (or any other tool that can copy raw data) the data it copy directly between logical volumes instead of file systems. This approach has similar limitations that the previous method.

Using LVM Mirroring

  1. Present the physical volumes (LUNs) in the new SAN / JBOD to the system.
  2. Format physical volumes to receive LVM information.
  3. Extend the volume group to the new LUNs.
  4. Mirror the logical volumes to the new physical volumes.
  5. Reduce mirror from original physical volume.
  6. Reduce physical volumes out of the volume group.
  7. Shutdown and disconnect cables of the old SAN / JBOD from the system.

Example:

In this example, volume group “vgorig” is formed by 16 physical volumes (LUNs) on external SCSI attached JBOD. The volume group contains 4 logical volumes using VxFS filesystem, each using 4 physical volumes.

The filesystems will be moved to 4 new LUNs in a SAN device. Configure the new disks using the same size of the old disks.

Current system configuration:

# strings /etc/lvmtab
/dev/vgorig
/dev/dsk/c8t0d1
/dev/dsk/c8t0d2
/dev/dsk/c8t0d3
/dev/dsk/c8t0d4

The filesystems and mount points on /dev/vgorig are.

# grep vgorig /etc/fstab
/dev/vgorig/lvol1 /fs1 vxfs delaylog,largefiles 0 1
/dev/vgorig/lvol2 /fs2 vxfs delaylog,largefiles 0 1
/dev/vgorig/lvol3 /fs3 vxfs delaylog,largefiles 0 1
/dev/vgorig/lvol4 /fs4 vxfs delaylog,largefiles 0 1

This configuration can be represented by the following diagram.

+------------+-------+---------+------------+
|    fs1     |  fs2  |   fs3   |    fs4     |
+------------+-------+---------+------------+
|   lvol1    | lvol2 |  lvol3  |   lvol4    |
+------------+-------+---------+------------+
|               /dev/vgorig                 |
+-------------------------------------------+
|  c8t0d1  |  c8t0d2  |  c8t0d3  |  c8t0d4  |
+----------+----------+----------+----------+

The new LUNs available to the system will be.

/dev/dsk/c10t0d1
/dev/dsk/c10t0d2
/dev/dsk/c10t0d3
/dev/dsk/c10t0d4


1. Present the physical volumes (LUNs) in the new SAN / JBOD to the system. Use ioscan to identify the new disk names.

# ioscan –funC disk
disk      2   0/0/1/0/0.0.1   sdisk    CLAIMED     DEVICE       HP 36.4GST336607FC
                           /dev/dsk/c10t0d1   /dev/rdsk/c10t0d1
disk      2   0/0/1/0/0.0.2   sdisk    CLAIMED     DEVICE       HP 36.4GST336607FC
                           /dev/dsk/c10t0d2   /dev/rdsk/c10t0d2
disk      2   0/0/1/0/0.0.3   sdisk    CLAIMED     DEVICE       HP 36.4GST336607FC
                           /dev/dsk/c10t0d3   /dev/rdsk/c10t0d3
disk      2   0/0/1/0/0.0.4   sdisk    CLAIMED     DEVICE       HP 36.4GST336607FC
                           /dev/dsk/c10t0d4   /dev/rdsk/c10t0d4

2. Format physical volumes to receive LVM information.

# pvcreate /dev/rdsk/c10t0d1
# pvcreate /dev/rdsk/c10t0d2
# pvcreate /dev/rdsk/c10t0d3
# pvcreate /dev/rdsk/c10t0d4

3. Extend the volume group to the new LUNs (Add new LUNs).

# vgextend /dev/vgorig /dev/dsk/c10t0d1
# vgextend /dev/vgorig /dev/dsk/c10t0d2
# vgextend /dev/vgorig /dev/dsk/c10t0d3
# vgextend /dev/vgorig /dev/dsk/c10t0d4

4. Mirror the logical volumes to the new physical volumes. It's very important to specify the target disk.

# lvextend -m 1 /dev/vgorig/lvol1 /dev/dsk/c10t0d1
# lvextend -m 1 /dev/vgorig/lvol2 /dev/dsk/c10t0d2
# lvextend -m 1 /dev/vgorig/lvol3 /dev/dsk/c10t0d3
# lvextend -m 1 /dev/vgorig/lvol4 /dev/dsk/c10t0d4

The previous steps changes can be represented by the following diagram.

+------------+-------+---------+------------+
|    fs1     |  fs2  |   fs3   |    fs4     |
+------------+-------+---------+------------+
|   lvol1    | lvol2 |  lvol3  |   lvol4    |
+------------+-------+---------+------------+
|               /dev/vgorig                 |
+-------------------------------------------+
| c8t0d1   |  c8t0d2  |  c8t0d3  |  c8t0d4  |
| c10t0d1  | c10t0d2  | c10t0d3  | c10t0d4  |
+----------+----------+----------+----------+

Verify the logical volume configuration. Mirror copies line in the lvdisplay command should be 1.

# lvdisplay -v /dev/vgorig/lvol1 | more
# lvdisplay -v /dev/vgorig/lvol2 | more
# lvdisplay -v /dev/vgorig/lvol3 | more
# lvdisplay -v /dev/vgorig/lvol4 | more

5. Reduce mirror from original physical volume (Remove copy out of the old PVs).

# lvreduce -m 0 /dev/vgorig/lvol1 /dev/dsk/c8t0d1
# lvreduce -m 0 /dev/vgorig/lvol2 /dev/dsk/c8t0d2
# lvreduce -m 0 /dev/vgorig/lvol3 /dev/dsk/c8t0d3
# lvreduce -m 0 /dev/vgorig/lvol4 /dev/dsk/c8t0d4

6. Reduce physical volumes out of the volume group.

# vgreduce /dev/vgorig /dev/dsk/c8t0d1
# vgreduce /dev/vgorig /dev/dsk/c8t0d2
# vgreduce /dev/vgorig /dev/dsk/c8t0d3
# vgreduce /dev/vgorig /dev/dsk/c8t0d4

7. Shutdown and disconnect cables of the old SAN / JBOD from the system.

The final configuration is represented by the following diagram.

+------------+-------+---------+------------+
|    fs1     |  fs2  |   fs3   |    fs4     |
+------------+-------+---------+------------+
|   lvol1    | lvol2 |  lvol3  |   lvol4    |
+------------+-------+---------+------------+
|               /dev/vgorig                 |
+-------------------------------------------+
| c10t0d1  | c10t0d2  | c10t0d3  | c10t0d4  |
+----------+----------+----------+----------+

Creating a new volume group and replicate the file systems

This is the list of steps necessary to perform the migration.

1. Present the new LUNS to the system.

2. Create the volume group.

3. Create the logical volumes and filesystems.

4. Copy the data.

5. Umount the original filesystems, disable the volume group.

6. Export the original and the new volume groups.

7. Import the new volume group in the place of the original volume group.

8. Mount the new filesystems.

Example:

In this example, we have a volume group called "vgorig" formed by 16 physical volumes (LUNs) on external SCSI Attached JBOD. The volume group contains 4 logical volumes using VxFS filesystem.

We are moving the filesystems to the "vgnew" volume group formed by 4 LUNs. The new system will contain the same number of logical volumes. Neither the new VG PE size, logical volumes size or filesystem size are discuse, because is out of the scope of the document.

# strings /etc/lvmtab

/dev/vgorig
/dev/dsk/c8t0d1
/dev/dsk/c8t0d2
/dev/dsk/c8t0d3
/dev/dsk/c8t0d4
/dev/dsk/c8t0d5
/dev/dsk/c8t0d6
/dev/dsk/c8t0d7
/dev/dsk/c8t0d8
/dev/dsk/c8t0d9
/dev/dsk/c8t0d10
/dev/dsk/c8t0d11
/dev/dsk/c8t0d12
/dev/dsk/c8t0d13
/dev/dsk/c8t0d14
/dev/dsk/c8t0d15
/dev/dsk/c8t0d16

The filesystems and mount points on /dev/vgorig are.

# grep vgorig /etc/fstab

/dev/vgorig/lvol1 /fs1 vxfs delaylog,largefiles 0 1
/dev/vgorig/lvol2 /fs2 vxfs delaylog,largefiles 0 1
/dev/vgorig/lvol3 /fs3 vxfs delaylog,largefiles 0 1
/dev/vgorig/lvol4 /fs4 vxfs delaylog,largefiles 0 1

The new SAN present 4 new LUNs to the system.

/dev/dsk/c10t0d1
/dev/dsk/c10t0d2
/dev/dsk/c10t0d3
/dev/dsk/c10t0d4

The following command sintaxis of the pvcreate, vgcreate, lvcreate commands are symplified for demostration purposes. The proper flags for any desired options should be provied. Specially, take care of using custom –e and –s options to define the Max PE per PV Physical Extent and PE Size.

1. Prepare the disk to become LVM physical volumes.

# pvcreate /dev/dsk/c10t0d1
# pvcreate /dev/dsk/c10t0d2
# pvcreate /dev/dsk/c10t0d3
# pvcreate /dev/dsk/c10t0d4

2. Create the new LVM volume group.

# vgcreate /dev/vgnew /dev/dsk/c10t0d1 /dev/dsk/c10t0d2 /dev/dsk/c10t0d3 /dev/dsk/c10t0d4

3. Create the required logical volumes.

# lvcreate –l <SIZE> –n lvol1 /dev/vgnew
# lvcreate –l <SIZE> –n lvol2 /dev/vgnew
# lvcreate –l <SIZE> –n lvol3 /dev/vgnew
# lvcreate –l <SIZE> –n lvol4 /dev/vgnew

4. Initializate the filesystems.

# newfs -o largefiles /dev/vgnew/rlvol1
# newfs -o largefiles /dev/vgnew/rlvol2
# newfs -o largefiles /dev/vgnew/rlvol3
# newfs -o largefiles /dev/vgnew/rlvol4

5. Create the required mountpoints and mount the new filesystems.

mkdir /newfs1
mkdir /newfs2
mkdir /newfs3
mkdir /newfs4

mount /dev/vgtarget/lvol1 /newfs1
mount /dev/vgtarget/lvol2 /newfs2
mount /dev/vgtarget/lvol3 /newfs3
mount /dev/vgtarget/lvol4 /newfs4

6. Transfer the information from the source filesystems to the target filesystems using your prefered tool. Normally using fbackup / frecover provied the safest procedure. Each individual filesystem should be parked during the data migration.

# cd /fs1
# fbackup -0i ./ -f - | ( cd /newfs1; frecover -rvf - )
# cd /fs2
# fbackup -0i ./ -f - | ( cd /newfs2; frecover -rvf - )
# cd /fs3
# fbackup -0i ./ -f - | ( cd /newfs3; frecover -rvf - )
# cd /fs4
# fbackup -0i ./ -f - | ( cd /newfs4; frecover -rvf - )

7. Umount the target and the source filesystems.

# umount /newfs1
# umount /newfs2
# umount /newfs3
# umount /fs1
# umount /fs2
# umount /fs3

8. Create a backup of the actual volume group information.

# ll /dev/vgorig/group
crw-r----- 1 root sys 64 0x000000 Jun 6 2006 /dev/vgorig/group

# cp /etc/lvmtab /lvmtab.bak
# tar cvf /vgorig.tar /dev/vgorig

9. Remove the source and target volume group using "vgexport".

# vgchange -a n /dev/vgtarget
# vgchange -a n /dev/vgorig

# vgexport –s -m /vgorig.map /dev/vgorig
# vgexport –s -m /vgtarget.map /dev/vgtarget
# cat /vgnew.map
# cat /vgorig.map

10. Import back the target volume group using the original volume group name.

# mkdir /dev/vgorig
# mknod /dev/vgorig/group c 64 0x0#0000
# vgimport –s -m /dev/vgtarget.map /dev/vgorig

11. Check the volume group and reactivate if the logical volume where created correctly.

# ll /dev/vgorig
# vgchange -a y vgorig
# vgcfgbackup -u vgorig

12. Remove backup information.

# rm /vgorig.tar /vgtarget.map /vgorig.map

13. Remove the new directories.

# rmdir /newfs1 /newfs2 /newfs3

Creating a new volume group and replicate the logical volumes

This is the list of steps necessary to perform the migration.

  1. Present the new LUNS to the system.
  2. Create the volume group.
  3. Create the logical volumes.
  4. Unmount the filesystems and copy the data using the "dd" command.
  5. Verify the filesystem integrity.
  6. Umount the original filesystems, disable the volume group.
  7. Export the original and the new volume groups.
  8. Import the new volume group in the place of the original volume group.
  9. Extend the logical volumes and filesystems if required.
  10. Check the volume group and reactivate if the logical volume where created correctly.
  11. Mount the new filesystems.
  12. Remove backup information.

Example:

In this example, we have a “vgorig” formed by 16 physical volumes (LUNs) on external a scsi attached JBOD. The volume group contains 4 logical volumes using VxFS filesystems.

We are moving the filesystems to a “vgnew” formed by 4 LUNs. The new system will contain the same number of logical volumes. Neither the new VG PE size, logical volumes size or filesystem size are discuse, because is out of the scope of the document.

# strings /etc/lvmtab

/dev/vgorig
/dev/dsk/c8t0d1
/dev/dsk/c8t0d2
/dev/dsk/c8t0d3
/dev/dsk/c8t0d4
/dev/dsk/c8t0d5
/dev/dsk/c8t0d6
/dev/dsk/c8t0d7
/dev/dsk/c8t0d8
/dev/dsk/c8t0d9
/dev/dsk/c8t0d10
/dev/dsk/c8t0d11
/dev/dsk/c8t0d12
/dev/dsk/c8t0d13
/dev/dsk/c8t0d14
/dev/dsk/c8t0d15
/dev/dsk/c8t0d16

The filesystems and mount points on /dev/vgorig are.

# grep vgorig /etc/fstab

/dev/vgorig/lvol1 /fs1 vxfs delaylog,largefiles 0 1
/dev/vgorig/lvol2 /fs2 vxfs delaylog,largefiles 0 1
/dev/vgorig/lvol3 /fs3 vxfs delaylog,largefiles 0 1
/dev/vgorig/lvol4 /fs4 vxfs delaylog,largefiles 0 1

The new SAN present 4 new LUNs to the system.

/dev/dsk/c10t0d1
/dev/dsk/c10t0d2
/dev/dsk/c10t0d3
/dev/dsk/c10t0d4

The command sintaxis of the pvcreate, vgcreate, lvcreate commands are symplified for demostration. The proper flags for the desired options should be provied. Specially, take care of the the vgcreate –e –s and lvcreate –l | -L flags.

1. Prepare the disk to become LVM physical volumes.

# pvcreate /dev/dsk/c10t0d1
# pvcreate /dev/dsk/c10t0d2
# pvcreate /dev/dsk/c10t0d3
# pvcreate /dev/dsk/c10t0d4

2. Create the new LVM volume group.

# vgcreate /dev/vgnew /dev/dsk/c10t0d1 /dev/dsk/c10t0d2 /dev/dsk/c10t0d3 /dev/dsk/c10t0d4

3. Create the required logical volumes.

# lvcreate –l <SIZE> –n lvol1 /dev/vgnew
# lvcreate –l <SIZE> –n lvol2 /dev/vgnew
# lvcreate –l <SIZE> –n lvol3 /dev/vgnew
# lvcreate –l <SIZE> –n lvol4 /dev/vgnew

4. Copy the data using dd command.

# dd if=/dev/vgorig/rlvol1 of=/dev/vgnew/rlvol1 bs=4096k
# dd if=/dev/vgorig/rlvol2 of=/dev/vgnew/rlvol2 bs=4096k
# dd if=/dev/vgorig/rlvol3 of=/dev/vgnew/rlvol3 bs=4096k
# dd if=/dev/vgorig/rlvol4 of=/dev/vgnew/rlvol4 bs=4096k

5. Verify the filesystem integrity.

# fsck -F vxfs /dev/vgnew/rlvol1
# fsck -F vxfs /dev/vgnew/rlvol2
# fsck -F vxfs /dev/vgnew/rlvol3
# fsck -F vxfs /dev/vgnew/rlvol4

6. Umount the original filesystems, disable the volume group.

umount /dev/vgorig/lvol1 /fs1
umount /dev/vgorig/lvol2 /fs2
umount /dev/vgorig/lvol3 /fs3
umount /dev/vgorig/lvol4 /fs4


7. Export the original and the new volume groups.

# vgchange -a n /dev/vgtarget
# vgchange -a n /dev/vgorig

# vgexport –s -m /vgsource.map /dev/vgsource
# vgexport –s -m /vgtarget.map /dev/vgtarget
# cat /vgnew.map
# cat /vgorig.map

8. Import the new volume group in the place of the original volume group.

# mkdir /dev/vgorig
# mknod /dev/vgorig/group c 64 0x0#0000
# vgimport –s -m /dev/vgtarget.map /dev/vgorig

9. Extend the logical volumes and filesystems if required.

10. Check the volume group and reactivate if the logical volume where created correctly.

# ll /dev/vgsource
# vgchange -a y vgorig
# vgcfgbackup -u vgorig

11. Mount the new filesystems.

# mount /dev/vgorig/lvol1 /fs1
# mount /dev/vgorig/lvol2 /fs2
# mount /dev/vgorig/lvol3 /fs3
# mount /dev/vgorig/lvol4 /fs4

12. Remove backup information.

Reference

Authors

This page was last modified on 3 August 2011, at 19:15. This page has been accessed 8,488 times.