How to set the size of the buffer file cache
Contents
Abstract
This article explains how to set the size of the VxFS Buffer Cache using the correspondent kernel parameters. Historically, the maximum and minimum values are set to 50% and 5% of the total physical memory. For current environments, specially when backing storage is provided by SAN devices that provided their own cache memory, those values may not provided a good balance of resources.
The article presents some general guidelines on values that can be used to better balance the available system memory with the buffer cache. Those values are consider starting point and measurement and judgment must be employed for each system deployment.
Always consult your Independent Software Vendor for recommendations on the optimal parameters for their application. Use performance measurement tools like Glance or Sar to check the effect on the system of reducing or increasing the values.
HP-UX 11i v1 (11.11)
In this release, the VxFS buffer cache is manage by the dbc_max_pct and the dbc_min_pct. kernel parameters.
# kmtune -q dbc_max_pct -q dbc_min_pct Parameter Current Dyn Planned Module Version =============================================================================== dbc_max_pct 50 - 50 dbc_min_pct 5 - 5
HP-UX 11i v2 (11.23)
In this release, the VxFS buffer cache is manage by the dbc_max_pct and the dbc_min_pct. kernel parameters.
# kctune -q dbc_max_pct -q dbc_min_pct Tunable Value Expression Changes dbc_max_pct 50 50 Immed dbc_min_pct 5 Default Immed
HP-UX 11i v3 (11.31)
According to the "The Unified File Cache (UFC) in HP-UX 11i v3" white paper, HP-UX 11i v3 integrates the Buffer Cache and the Page Cache to provide coherence for file access. As part of the integration project, kernel interfaces have been implemented to facilitate porting of third-party file systems.
It has created a new twist to the wide known issue of HP-UX systems with physical memory footprints larger that 4 GB, "the defaults VxFS file cache are set too high".
This problem can be perceive in two ways:
- Glance reports that memory usage is higher that 90%, even in systems with low or no application usage.
- Glance shows that the ksyncer_daemon "trash" the I/O subsystem to 100% for several seconds.
On page 8 of the "The Unified File Cache (UFC) in HP-UX 11i v3" under the Flexible cache flushing the ksyncer_daemon operation is explained. Verbatim:
- "The ksyncer daemon flushes modified data from the cache to the files at a default interval of 30 seconds. A process can use the fcntl() and fadvise() system calls to override the default and tune the flush interval at the file level. The ksyncer daemon can be configured (enabled or disabled) by an entry in /etc/rc.config.d/syncer. However, it is strongly recommended to enable the ksyncer daemon (use the default entry) in order to minimize the risk of data loss in the event of a system failure."
- The ksyncer daemon only flushes pages that were modified by the write() access type. A process can use the msync() system call to flush the pages that were modified by the mmap() access type, if it needs to ensure that those modifications are written to the file at a certain point in time - otherwise the modifications are written to the file at a point in time that is unknown to the process, for example when the pages are reclaimed and freed by the System Pager daemon."
It means that if large amounts of data are in the File Cache memory waiting the be flushed into non-volatile storage (disk drives or SAN luns), every time the "ksyncer" interval is reach, that data will be moved to the actual storage, producing an intermittent High write request over the I/O subsystem.
Reducing the size of the file cache from the defaults 50% maximum / 5% minimum to more conservative values, will not only reclaim that memory for the system process and reduce the maximum overall size of data that needs to be flushed from the cache to non-volatile storage, but will also reduce the risk of data lost due to failure to flush the data in a disaster scenario, like a power outage.
To determine the current file cache size, use the "kctune" command:
# kctune -q filecache_max -q filecache_min
Tunable Value Expression Changes
filecache_max 1020264448 Default Auto
filecache_min 102023168 Default Auto
Recommended Buffer Cache Limits
The following table can provides a starting point to replace the default values.
| Physical Memory | Maximum | Minimum |
| <= 4GB | 10% | 5% |
| > 4GB <= 8GB | 5% | 3% |
| > 8GB <= 16 GB | 3% | 2% |
| 32+ GB | 2% | 1% |
Is a system reboot required?
A system reboot is not strictly required on HP-UX 11i v2 and 11i v3, since on these releases these parameters can be dynamically modified. Nonetheless, releasing physical memory already used by the VxFS Buffer Cache can take several hours depending on the amount of data already cached and the system memory request patterns, specially if the system is not requesting more memory for user land applications. Occasionally, due to coalescence problem of the cache memory pages (memory fragmentation), the Buffer cache memory will not been return to the memory pool in a timely manner. This may be of special consideration on system with 4 GB of system memory where total available memory stresses the memory allocation routines.
Is always a good recommendation to schedule down time to apply changes to the Buffer Cache kernel parameters and reboot the system. That way, newer measures can be taken on a clean slate that allows to establish a current baseline for further measurements.
Change default values
HP-UX 11i v1
Use the kctune command, "SAM" to set new kernel parameters values. After these changes are done, reboot the system to immediately reclaim the memory and start a measurement baseline. System administrators are require to evaluate the cache hit ration (sar -b) do determinate if additional memory can be release or need to be use to improve I/O performance.
HP-UX 11i v2 / v3
Use the kctune command, "SAM" or "System Management Homepage" to set new kernel parameters values. After these changes are done, reboot the system to immediately reclaim the memory and start a measurement baseline. System administrators are require to evaluate the cache hit ration (sar -b) do determinate if additional memory can be release or need to be use to improve I/O performance.
Glance and VxFS Buffer Cache Usage
Glance information should be taken with a grain of salt when looking for memory utilization. When displaying memory utilization, Glance take into account the memory assigned to the Buffer Cache. This can lead to misreadings in some systems with constant I/O usage of VxFS file systems, where the buffer cache holds all the memory provided as their upper limit.
Example:
B3692A Glance C.04.70.001 11:35:04 delta 9000/800 Current Avg High
--------------------------------------------------------------------------------
CPU Util SA | 3% 3% 3%
Disk Util | 0% 0% 0%
Mem Util S SU U | 65% 65% 65% (1)(2)
Swap Util U UR R | 31% 31% 31%
--------------------------------------------------------------------------------
PROCESS LIST Users= 10
User CPU % Thrd Disk Memory Block
Process Name PID Name ( 200% max) Cnt IOrate RSS/VSS On
--------------------------------------------------------------------------------
glance 26717 root 6.0 1 0.0 1.5mb 5.6mb SLEEP
scopeux 1750 root 0.0 1 0.0 120.1mb 125.3mb SLEEP
midaemon 1768 root 0.0 2 0.0 72.6mb 104.8mb SLEEP
C - cum/interval toggle Page 1 of 1
ProcList CPU Rpt Mem Rpt Disk Rpt NextKeys SlctProc Help Exit
- Observe the Mem Util section. The letters stand for S: System, U; User; B: Buffer. Glance list the buffer cache as part of the total memory usage. That memory can be released from the buffer cache by the kernel very quickly when user process request additional memory from the heap.
- Is very common to see the "current" and "high" values to reach 90% or above. Always remove Buffer Cache or File Cache total from that value. To check for that values, press the "m" key on the Glance TUI and that will lead you to the memory section. More details on memory usage will be displayed.
B3692A Glance C.04.70.001 11:35:45 delta 9000/800 Current Avg High
--------------------------------------------------------------------------------
CPU Util S | 1% 2% 4%
Disk Util F | 2% 1% 2%
Mem Util S SU U | 65% 65% 65%
Swap Util U UR R | 31% 31% 31%
--------------------------------------------------------------------------------
MEMORY REPORT Users= 10
Event Current Cumulative Current Rate Cum Rate High Rate
--------------------------------------------------------------------------------
Page Faults 0 111 0.0 2.6 60.6
Page In 0 0 0.0 0.0 5.3
Page Out 0 0 0.0 0.0 0.0
KB Paged In 0kb 0kb 0.0 0.0 0.0
KB Paged Out 0kb 0kb 0.0 0.0 0.0
Reactivations 0 0 0.0 0.0 0.0
Deactivations 0 0 0.0 0.0 0.0
KB Deactivated 0kb 0kb 0.0 0.0 0.0
VM Reads 0 0 0.0 0.0 0.0
VM Writes 0 0 0.0 0.0 0.0
Total VM : 615mb Sys Mem : 868mb User Mem: 432mb Phys Mem : 2.0gb (1)
Active VM: 443mb Buf Cache: 4mb Free Mem: 710mb FileCache: 33mb
MemFS Blk Cnt: 0 MemFS Swp Cnt: 0 Page 1 of 1
ProcList CPU Rpt Mem Rpt Disk Rpt NextKeys SlctProc Help Exit
- The last lines of the output display overall system memory utilization with higher detail. Keep in mind that Buf Cache (File Cache) is part of Sys Mem.
Buffer cache and kmeminfo
The WTEC tool kmeminfo can be very useful to explore the relationships between the different system memory areas. On system when the default limits and constant I/O will present up to 50% of memory assigned to the Buffer Cache.
Example:
tool: kmeminfo 9.14 - libp4 9.366 - libhpux 1.258 - HP CONFIDENTIAL
unix: /stand/current/vmunix 11.23 64bit IA64 on host "camccscbbur04"
core: /dev/kmem live
link: Fri Nov 09 15:32:57 CST 2007
boot: Mon Oct 19 14:05:05 2009
time: Wed Nov 4 10:43:14 2009
nbpg: 4096 bytes
----------------------------------------------------------------------
Physical memory usage summary (in page/byte/percent):
Physical memory = 1041841 4.0g 100% (1)
Free memory = 23333 91.1m 2% (2)
User processes = 161271 630.0m 15% details with -user
System = 849626 3.2g 82%
Kernel = 328706 1.3g 32% kernel text and data
Dynamic Arenas = 117272 458.1m 11% details with -arena
vx_global_kmcac = 28043 109.5m 3%
vx_buffer_kmcac = 17856 69.8m 2%
spinlock = 15145 59.2m 1%
ALLOCB_MBLK_LM = 5440 21.2m 1%
M_DYNAMIC = 5186 20.3m 0%
Other arenas = 45602 178.1m 4% details with -arena
Super page pool = 87939 343.5m 8% details with -kas
Emergency pool = 2075 8.1m 0% system critical reserve
UAREA's = 7392 28.9m 1%
Static Tables = 94475 369.0m 9% details with -static
nbuf = 44896 175.4m 4% bufcache headers
pfdat = 24418 95.4m 2%
vhpt = 8192 32.0m 1%
text = 8084 31.6m 1% vmunix text section
bufhash = 2048 8.0m 0% bufcache hash headers
Other tables = 6836 26.7m 1% details with -static
Buffer cache = 520920 2.0g 50% details with -bufcache (3)
- The Physical memory is 4 Gb.
- The Free memory is 91 Mb.
- The Buffer Cache occupies 2 Gb (50%) of the available system memory, this system will have a major impact of reducing the size of the Buffer cache. Setting the value to a 10% will release 1.5 Gb of the system memory.
Reference
- The Unified File Cache (UFC) in HP-UX 11i v3
- How to change kernel parameters using CLI
- Modify kernel using kconfig in HPUX 11i v3
- How to bypass the VxFS file system cache
- WFM: 3605653307 - 3605653307 - 4606041542
