Ever wondered all the precious memory installed on your server has gone? Among many other reasons, if you are running Solaris 10 and use ZFS file system then there may be your answer.
ZFS Adaptive Replacement Cache (ARC) tends to use up to 75% of the installed physical memory on servers with 4GB or less and upto everything except 1GB of memory on servers with more than 4GB of memory to cache data in a bid to improve performance.
This can significantly affect performance on mission critical servers running Databases etc.
To identify how much memory uses:
# kstat -m zfs | grep size
data_size 18935877120
hdr_size 66041496
l2_hdr_size 0
l2_size 0
other_size 11310112
size 19013228728
Here “19013228728” (approx 18G) indicates the total memory used by ZFS.
Alternatively, the following mdb command show ZFS ARC usage:
# echo "::arc" | mdb -k|grep size size = 2048 MB hdr_size = 12493584 data_size = 2048608256 other_size = 86475456 l2_size = 0 l2_hdr_size = 0
It makes sense to cap the maximum ZFS ARC can use on servers where memory requirement for other services is more.
To set the maximum limit for ZFS ARC, edit /etc/system file and add the following line
set zfs:zfs_arc_max=2147483648
where 2147483648 restricts the usage to a maximum of 2GB physical memory. Unfortunately, this requires a reboot for the setting to take effect and cannot be dynamically changed.