Symptom:
As part of a database tuning effort you increase the SGA/PGA sizes; and Oracle greets with anORA-27102: out of memory
error message. The system had enough free memory to serve the needs of Oracle.
SQL> startup
ORA-27102: out of memory
SVR4 Error: 22: Invalid argument
Diagnosis
$ oerr ORA 27102
27102, 00000, "out of memory"
// \*Cause: Out of memory
// \*Action: Consult the trace file for details
Not so helpful. Let’s look the alert log for some clues.
% tail -2 alert.log
WARNING: EINVAL creating segment of size 0x000000028a006000
fix shm parameters in /etc/system or equivalent
Oracle is trying to create a 10G shared memory segment (depends on SGA/PGA sizes), but operating system (Solaris in this example) responded with an invalid argument (EINVAL) error message. There is a little hint about setting shm parameters in/etc/system
.
Prior to Solaris 10,shmsys:shminfo_shmmax
parameter has to be set in/etc/system
with maximum memory segment value that can be created. 8M is the default value on Solaris 9 and prior versions; where as 1/4th of the physical memory is the default on Solaris 10 and later. On a Solaris 10 (or later) system, it can be verified as shown below:
% prtconf | grep Mem
Memory size: 32760 Megabytes
% id -p
uid=59008(oracle) gid=10001(dba) projid=3(default)
% prctl -n project.max-shm-memory -i project 3
project: 3: default
NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT
project.max-shm-memory
privileged 7.84GB - deny -
system 16.0EB max deny -
Now it is clear that the system is using the default value of 8G in this scenario, where as the application (Oracle) is trying to create a memory segment (10G) larger than 8G. Hence the failure.
So, the solution is to configure the system with a value large enough for the shared segment being created, so Oracle succeeds in starting up the database instance.
On Solaris 9 and prior releases, it can be done by adding the following line to/etc/system
, followed by a reboot for the system to pick up the new value.
set shminfo_shmmax = 0x000000028a006000
Howevershminfo_shmmax
parameter was obsoleted with the release of Solaris 10; and Sun doesn’t recommend setting this parameter in/etc/system
even though it works as expected.
On Solaris 10 and later, this value can be changed dynamically on a per project basis with the help of resource control facilities . This is how we do it on Solaris 10 and later:
% prctl -n project.max-shm-memory -r -v 10G -i project 3
% prctl -n project.max-shm-memory -i project 3
project: 3: default
NAME PRIVILEGE VALUE FLAG ACTION RECIPIENT
project.max-shm-memory
privileged 10.0GB - deny -
system 16.0EB max deny -
Note that changes made with theprctl
command on a running system are temporary, and will be lost when the system is rebooted. To make the changes permanent, create a project withprojadd
command and associate it with the user account as shown below:
% projadd -p 3 -c 'eBS benchmark' -U oracle -G dba -K 'project.max-shm-memory=(privileged,10G,deny)' OASB
% usermod -K project=OASB oracle
Finally make sure the project is created withprojects -l
orcat /etc/project
commands.
% projects -l
...
...
OASB
projid : 3
comment: "eBS benchmark"
users : oracle
groups : dba
attribs: project.max-shm-memory=(privileged,10737418240,deny)
% cat /etc/project
...
...
OASB:3:eBS benchmark:oracle:dba:project.max-shm-memory=(privileged,10737418240,deny)
With these changes, Oracle would start the database up normally.
SQL> startup
ORACLE instance started.
Total System Global Area 1.0905E+10 bytes
Fixed Size 1316080 bytes
Variable Size 4429966096 bytes
Database Buffers 6442450944 bytes
Redo Buffers 31457280 bytes
Database mounted.
Database opened.
Addendum : Oracle RAC settings
Anonymous Bob suggested the following settings for Oracle RAC in the form of a comment for the benefit of others who run into similar issue(s) when running Oracle RAC. I’m pasting the comment as is (Disclaimer: I have not verified these settings):
Thanks for a great explanation, I would like to add one comment that will help those with an Oracle RAC installation. Modifying the default project covers oracle processes great and is all that is needed for a single instance DB. In RAC however, the CRS process starts the DB and it is a root owned process and root does not use the default project. To fix ORA-27102 issue for RAC I added the following lines to an init script that runs before the init.crs script fires.
# Recommended Oracle RAC system params
ndd -set /dev/udp udp_xmit_hiwat 65536
ndd -set /dev/udp udp_recv_hiwat 65536
# For root processes like crsd
prctl -n project.max-shm-memory -r -v 8G -i project system
prctl -n project.max-shm-ids -r -v 512 -i project system
# For oracle processes like sqlplus
prctl -n project.max-shm-memory -r -v 8G -i project default
prctl -n project.max-shm-ids -r -v 512 -i project default
So simple yet it took me a week working with Oracle and SUN to come up with that answer…Hope that helps someone out.