From Wiki-UX.info

Wiki-UX / How to set the size of the buffer file cache
Jump to: navigation, search

How to set the size of the buffer file cache

Contents

Abstract

This article explains how to set the size of the VxFS Buffer Cache using the correspondent kernel parameters. Historically, the maximum and minimum values are set to 50% and 5% of the total physical memory. For current environments, specially when backing storage is provided by SAN devices that provided their own cache memory, those values may not provided a good balance of resources.

The article presents some general guidelines on values that can be used to better balance the available system memory with the buffer cache. Those values are consider starting point and measurement and judgment must be employed for each system deployment.

Always consult your Independent Software Vendor for recommendations on the optimal parameters for their application. Use performance measurement tools like Glance or Sar to check the effect on the system of reducing or increasing the values.

HP-UX 11i v1 (11.11)

In this release, the VxFS buffer cache is manage by the dbc_max_pct and the dbc_min_pct. kernel parameters.

# kmtune -q dbc_max_pct -q dbc_min_pct
Parameter             Current Dyn Planned                    Module     Version
===============================================================================
dbc_max_pct                50  -  50
dbc_min_pct                 5  -  5

HP-UX 11i v2 (11.23)

In this release, the VxFS buffer cache is manage by the dbc_max_pct and the dbc_min_pct. kernel parameters.

# kctune -q dbc_max_pct -q dbc_min_pct
Tunable      Value  Expression  Changes
dbc_max_pct     50  50          Immed
dbc_min_pct      5  Default     Immed

HP-UX 11i v3 (11.31)

According to the "The Unified File Cache (UFC) in HP-UX 11i v3" white paper, HP-UX 11i v3 integrates the Buffer Cache and the Page Cache to provide coherence for file access. As part of the integration project, kernel interfaces have been implemented to facilitate porting of third-party file systems.

It has created a new twist to the wide known issue of HP-UX systems with physical memory footprints larger that 4 GB, "the defaults VxFS file cache are set too high".

This problem can be perceive in two ways:

  1. Glance reports that memory usage is higher that 90%, even in systems with low or no application usage.
  2. Glance shows that the ksyncer_daemon "trash" the I/O subsystem to 100% for several seconds.

On page 8 of the "The Unified File Cache (UFC) in HP-UX 11i v3" under the Flexible cache flushing the ksyncer_daemon operation is explained. Verbatim:

"The ksyncer daemon flushes modified data from the cache to the files at a default interval of 30 seconds. A process can use the fcntl() and fadvise() system calls to override the default and tune the flush interval at the file level. The ksyncer daemon can be configured (enabled or disabled) by an entry in /etc/rc.config.d/syncer. However, it is strongly recommended to enable the ksyncer daemon (use the default entry) in order to minimize the risk of data loss in the event of a system failure."
The ksyncer daemon only flushes pages that were modified by the write() access type. A process can use the msync() system call to flush the pages that were modified by the mmap() access type, if it needs to ensure that those modifications are written to the file at a certain point in time - otherwise the modifications are written to the file at a point in time that is unknown to the process, for example when the pages are reclaimed and freed by the System Pager daemon."

It means that if large amounts of data are in the File Cache memory waiting the be flushed into non-volatile storage (disk drives or SAN luns), every time the "ksyncer" interval is reach, that data will be moved to the actual storage, producing an intermittent High write request over the I/O subsystem.

Reducing the size of the file cache from the defaults 50% maximum / 5% minimum to more conservative values, will not only reclaim that memory for the system process and reduce the maximum overall size of data that needs to be flushed from the cache to non-volatile storage, but will also reduce the risk of data lost due to failure to flush the data in a disaster scenario, like a power outage.

To determine the current file cache size, use the "kctune" command:

# kctune -q filecache_max -q filecache_min
Tunable             Value  Expression  Changes
filecache_max  1020264448  Default     Auto
filecache_min   102023168  Default     Auto

Recommended Buffer Cache Limits

The following table can provides a starting point to replace the default values.

Physical Memory Maximum Minimum
<= 4GB 10% 5%
> 4GB <= 8GB 5% 3%
> 8GB <= 16 GB 3% 2%
32+ GB 2% 1%


Is a system reboot required?

A system reboot is not strictly required on HP-UX 11i v2 and 11i v3, since on these releases these parameters can be dynamically modified. Nonetheless, releasing physical memory already used by the VxFS Buffer Cache can take several hours depending on the amount of data already cached and the system memory request patterns, specially if the system is not requesting more memory for user land applications. Occasionally, due to coalescence problem of the cache memory pages (memory fragmentation), the Buffer cache memory will not been return to the memory pool in a timely manner. This may be of special consideration on system with 4 GB of system memory where total available memory stresses the memory allocation routines.

Is always a good recommendation to schedule down time to apply changes to the Buffer Cache kernel parameters and reboot the system. That way, newer measures can be taken on a clean slate that allows to establish a current baseline for further measurements.

Change default values

HP-UX 11i v1

Use the kctune command, "SAM" to set new kernel parameters values. After these changes are done, reboot the system to immediately reclaim the memory and start a measurement baseline. System administrators are require to evaluate the cache hit ration (sar -b) do determinate if additional memory can be release or need to be use to improve I/O performance.

HP-UX 11i v2 / v3

Use the kctune command, "SAM" or "System Management Homepage" to set new kernel parameters values. After these changes are done, reboot the system to immediately reclaim the memory and start a measurement baseline. System administrators are require to evaluate the cache hit ration (sar -b) do determinate if additional memory can be release or need to be use to improve I/O performance.

Glance and VxFS Buffer Cache Usage

Glance information should be taken with a grain of salt when looking for memory utilization. When displaying memory utilization, Glance take into account the memory assigned to the Buffer Cache. This can lead to misreadings in some systems with constant I/O usage of VxFS file systems, where the buffer cache holds all the memory provided as their upper limit.

Example:

B3692A Glance C.04.70.001       11:35:04    delta 9000/800    Current  Avg  High
--------------------------------------------------------------------------------
CPU  Util   SA                                                 |  3%    3%    3%
Disk Util                                                      |  0%    0%    0%
Mem  Util   S                   SU         U                   | 65%   65%   65% (1)(2)
Swap Util   U        UR    R                                   | 31%   31%   31%
--------------------------------------------------------------------------------
                                  PROCESS LIST                      Users=   10
                         User      CPU %    Thrd Disk        Memory       Block
Process Name     PID     Name   ( 200% max) Cnt  IOrate      RSS/VSS      On
--------------------------------------------------------------------------------
glance             26717 root          6.0     1   0.0     1.5mb    5.6mb SLEEP
scopeux             1750 root          0.0     1   0.0   120.1mb  125.3mb SLEEP
midaemon            1768 root          0.0     2   0.0    72.6mb  104.8mb SLEEP
 
 
 
 
 
 
 
 
      C - cum/interval toggle                                       Page 1 of 1
ProcList CPU Rpt  Mem Rpt  Disk Rpt         NextKeys SlctProc   Help     Exit
  1. Observe the Mem Util section. The letters stand for S: System, U; User; B: Buffer. Glance list the buffer cache as part of the total memory usage. That memory can be released from the buffer cache by the kernel very quickly when user process request additional memory from the heap.
  2. Is very common to see the "current" and "high" values to reach 90% or above. Always remove Buffer Cache or File Cache total from that value. To check for that values, press the "m" key on the Glance TUI and that will lead you to the memory section. More details on memory usage will be displayed.
B3692A Glance C.04.70.001       11:35:45    delta 9000/800    Current  Avg  High
--------------------------------------------------------------------------------
CPU  Util   S                                                  |  1%    2%    4%
Disk Util   F                                                  |  2%    1%    2%
Mem  Util   S                   SU         U                   | 65%   65%   65%
Swap Util   U        UR    R                                   | 31%   31%   31%
--------------------------------------------------------------------------------
                                 MEMORY REPORT                      Users=   10
Event         Current   Cumulative   Current Rate   Cum Rate   High Rate
--------------------------------------------------------------------------------
Page Faults         0          111         0.0        2.6        60.6
Page In             0            0         0.0        0.0         5.3
Page Out            0            0         0.0        0.0         0.0
KB Paged In       0kb          0kb         0.0        0.0         0.0
KB Paged Out      0kb          0kb         0.0        0.0         0.0
Reactivations       0            0         0.0        0.0         0.0
Deactivations       0            0         0.0        0.0         0.0
KB Deactivated    0kb          0kb         0.0        0.0         0.0
VM Reads            0            0         0.0        0.0         0.0
VM Writes           0            0         0.0        0.0         0.0
Total VM :   615mb   Sys Mem  :   868mb   User Mem:   432mb   Phys Mem :   2.0gb (1)
Active VM:   443mb   Buf Cache:     4mb   Free Mem:   710mb   FileCache:    33mb
MemFS Blk Cnt:                0   MemFS Swp Cnt:                0   Page 1 of 1
ProcList CPU Rpt  Mem Rpt  Disk Rpt         NextKeys SlctProc   Help     Exit
  1. The last lines of the output display overall system memory utilization with higher detail. Keep in mind that Buf Cache (File Cache) is part of Sys Mem.

Buffer cache and kmeminfo

The WTEC tool kmeminfo can be very useful to explore the relationships between the different system memory areas. On system when the default limits and constant I/O will present up to 50% of memory assigned to the Buffer Cache.

Example:

tool: kmeminfo 9.14 - libp4 9.366 - libhpux 1.258 - HP CONFIDENTIAL
unix: /stand/current/vmunix 11.23 64bit IA64 on host "camccscbbur04"
core: /dev/kmem live
link: Fri Nov 09 15:32:57 CST 2007
boot: Mon Oct 19 14:05:05 2009
time: Wed Nov  4 10:43:14 2009
nbpg: 4096 bytes
 
 
 
----------------------------------------------------------------------
Physical memory usage summary (in page/byte/percent):
 
Physical memory       =  1041841    4.0g 100%                           (1)
Free memory           =    23333   91.1m   2%                           (2)
User processes        =   161271  630.0m  15%  details with -user
System                =   849626    3.2g  82%  
  Kernel              =   328706    1.3g  32%  kernel text and data
    Dynamic Arenas    =   117272  458.1m  11%  details with -arena
      vx_global_kmcac =    28043  109.5m   3%  
      vx_buffer_kmcac =    17856   69.8m   2%  
      spinlock        =    15145   59.2m   1%  
      ALLOCB_MBLK_LM  =     5440   21.2m   1%  
      M_DYNAMIC       =     5186   20.3m   0%  
      Other arenas    =    45602  178.1m   4%  details with -arena
    Super page pool   =    87939  343.5m   8%  details with -kas
    Emergency pool    =     2075    8.1m   0%  system critical reserve
    UAREA's           =     7392   28.9m   1%  
    Static Tables     =    94475  369.0m   9%  details with -static
      nbuf            =    44896  175.4m   4%  bufcache headers
      pfdat           =    24418   95.4m   2%  
      vhpt            =     8192   32.0m   1%  
      text            =     8084   31.6m   1%  vmunix text section
      bufhash         =     2048    8.0m   0%  bufcache hash headers
      Other tables    =     6836   26.7m   1%  details with -static
  Buffer cache        =   520920    2.0g  50%  details with -bufcache   (3)
  1. The Physical memory is 4 Gb.
  2. The Free memory is 91 Mb.
  3. The Buffer Cache occupies 2 Gb (50%) of the available system memory, this system will have a major impact of reducing the size of the Buffer cache. Setting the value to a 10% will release 1.5 Gb of the system memory.

Reference

Authors

This page was last modified on 4 November 2009, at 12:23. This page has been accessed 11,161 times.