HugePages Configuration and Monitoring

I was recently working with a customer who was experiencing some memory pressure on their Linux database server.
They were running Oracle 10.2.0.5 on Oracle Enterprise Server 5.7.
The buffer cache was 50G and they wanted to increase it to about 100G.

Here is what the memory allocation on the machine looked like; 10G in PageTables.

$ cat /proc/meminfo
MemTotal: 264251332 kB
MemFree: 356300 kB
Buffers: 954760 kB
Cached: 245170884 kB
SwapCached: 1758320 kB
Active: 175702124 kB
Inactive: 75100644 kB
HighTotal: 0 kB
HighFree: 0 kB
LowTotal: 264251332 kB
LowFree: 356300 kB
SwapTotal: 4194296 kB
SwapFree: 504648 kB
Dirty: 52 kB
Writeback: 56 kB
AnonPages: 2918272 kB
Mapped: 43409016 kB
Slab: 2252952 kB
PageTables: 10498900 kB
NFS_Unstable: 0 kB
Bounce: 0 kB
CommitLimit: 136319960 kB
Committed_AS: 67219780 kB
VmallocTotal: 34359738367 kB
VmallocUsed: 279684 kB
VmallocChunk: 34359458295 kB
HugePages_Total: 0
HugePages_Free: 0
HugePages_Rsvd: 0
Hugepagesize: 2048 kB

Their server had 256G of RAM and a SGA that was already 100G; a good candidate for HugePages due to their sizes.

From Oracle documentation, here are some major advantages of using HugePages.

* HugePages is crucial for faster Oracle database performance on Linux if you have a large RAM and SGA.

* HugePages are not swappable. Therefore there is no page-in/page-out mechanism overhead. HugePages are universally regarded as pinned.

Implementing HugePages has become common practice with Oracle 11g and is fairly well documented in MOS Note 361468.1.
The basics steps are as follows:

* Set the memlock ulimit for the oracle user.
* Disable Automatic Memory Managment if necesary as it is incompatible with HugePages.
* Run the Oracle supplied hugepages_settings.sh script to calculate the recommended value for the vm.nr_hugepages kernel parameter.
* Edit /etc/sysctl.conf with the vm.nr_hugepages with the recommeneded setting.
* Reboot the server

Unfortunately, The database we were working with was 10g. As it turns out, there are some differences between
Oracle 10 and 11, mainly that there is no HugePage logging in the alert log on version 10. See MOS Note: 1392543.1
This makes troubleshooting difficult because HugePages does not take effect until after it is configured.

Another noticable difference is the fact that the initialization parameter use_large_pages was not added until version 11.2.0.2.
In the event that you have multiple databases on one machine, this parameter allows you to control which databases
use HugePages. In addition, it can prevent a database from starting up if not enough HugePages can be allocated at instance startup.
See “USE_LARGE_PAGES To Enable HugePages In 11.2 [ID 1392497.1]” for more detailed info on the values for this parameter.

Similar across both versions is the initialization parameter PRE_PAGE_SGA, which determines whether Oracle
will read the entire SGA into memory at instance startup or not. However, it is not
specific to HugePages as it works with the default OS pages as well. Also, there is an environment variable DISABLE_HUGETLBFS
that can be used in 10g to accomplish the same thing as the use_large_pages parameter in 11g. If this variable is set, it
will prevent Oracle from using HugePages. So presumably you could have startup scripts that set or unset this variable prior
to starting databases which would give you ability to have some use huge pages while others do not, even in 10g.

Getting back to the story, we successfully got HugePages working on their QA box, but then it did not
work in production. HupePages were created by the OS after the reboot as seen in /proc/meminfo;
however, Oracle didn’t use them. In comparing the two environments, two notable differences were found:

1. memlock settings in /etc/security/limits.conf ( Can be seen by running ulimit ) was unlimited in QA, but not production

> ulimit -l
98566144

> cat /etc/security/limits.conf
# /etc/security/limits.conf
#

oracle soft memlock 98566144
oracle hard memlock 98566144
# End of file

2. SHMMAX was 90G on QA, but 190G on production

Oracle’s recommendation for the memlock setting ( from the MOS note 1392543.1 ) is stated as follows:
“Set the value (in KB) slightly smaller than installed RAM.”
The customer had decided to set it to 94% of their 100G SGA.

After some searching, we found the MOS note “Oracle Not Utilizing Hugepages [ID 803238.1]” which
gives a formula for calculating the value. This value is the minimum required in order for the database
to successfully startup using HugePages. The memlock setting is specified in KB and must match the
memory size of the number of HugePages that Oracle should be able to allocate.

memlock = ( HugePages * Hugepagesize )
where HugePages is the output given using script from MOS note 401749.1 and Hugepagesize is value from
in /proc/meminfo (on Linux Hugepagesize is normally 2048).

memlock = ( 51203 * 2048 ) = 104863744
This number is higher than what was set for memlock in /etc/security/limits.conf .

Now, what about SHMMAX? According to MOS note 567506.1, Oracle Global Customer Support officially recommends
a maximum for SHMMAX of “1/2 of physical RAM”. Since the customer’s value was higher than this,
SHMMAX was lowered from 190G to 90G to match that of QA.

After making these two changes, the database successfuly started up using HugePages.
It is worth mentioning here that if HugePages had not been successfully configured, the vm.nr_hugepages
parameter change in /etc/sysctl.conf should be backed out as the space allocated to HugePages would be
wasted if Oracle cannot use it.

So, was it memlock or SHMMAX that was causing the issue? Or was it both?

Since 11g writes a lot of information to the alert log regarding hugepage usage I decided to see what kind
of messages would be emitted when the same situation occurred on an 11g database.
As it turns out, the messages in the alert log are very informational in pointing you in the right direction.

Here is a snippet from the 11g alert log on a clean startup utilizing HugePages:

Starting ORACLE instance (normal)
****************** Large Pages Information *****************
 
Total Shared Global Region in Large Pages = 806 MB (100%)
 
Large Pages used by this instance: 403 (806 MB)
Large Pages unused system wide = 97 (194 MB) (alloc incr 4096 KB)
Large Pages configured system wide = 500 (1000 MB)
Large Page size = 2048 KB
***********************************************************

In the following example, I lowered the memlock setting to below its recommended value.
Note the Memlock messages saying it is too small.
The “recommendation” part seems a bit odd as it suggests to add 0 pages!
We’ll look at that one next.

Starting ORACLE instance (normal)
Memlock limit too small: 32768 to accommodate segment size: 4194304
Memlock limit too small: 32768 to accommodate segment size: 8388608
Memlock limit too small: 32768 to accommodate segment size: 4194304
Memlock limit too small: 32768 to accommodate segment size: 4194304
...
Memlock limit too small: 32768 to accommodate segment size: 4194304
Memlock limit too small: 32768 to accommodate segment size: 4194304
Memlock limit too small: 32768 to accommodate segment size: 2097152
****************** Large Pages Information *****************
 
Total Shared Global Region in Large Pages = 0 KB (0%)
 
Large Pages used by this instance: 0 (0 KB)
Large Pages unused system wide = 500 (1000 MB) (alloc incr 4096 KB)
Large Pages configured system wide = 500 (1000 MB)
Large Page size = 2048 KB
 
RECOMMENDATION:
Total Shared Global Region size is 810 MB. For optimal performance,
prior to the next instance restart increase the number
of unused Large Pages by atleast 0 2048 KB Large Pages (0 KB)
system wide to get 100% of the Shared
Global Region allocated with Large pages
***********************************************************

In the following example, the memlock value and hugepages are both too small.
As you can see from the alert log below, messages about both are mentioned, good!

Starting ORACLE instance (normal)
Memlock limit too small: 32768 to accommodate segment size: 4194304
Memlock limit too small: 32768 to accommodate segment size: 8388608
Memlock limit too small: 32768 to accommodate segment size: 4194304
...
Memlock limit too small: 32768 to accommodate segment size: 4194304
Memlock limit too small: 32768 to accommodate segment size: 4194304
Memlock limit too small: 32768 to accommodate segment size: 2097152
****************** Large Pages Information *****************
 
Total Shared Global Region in Large Pages = 0 KB (0%)
 
Large Pages used by this instance: 0 (0 KB)
Large Pages unused system wide = 100 (200 MB) (alloc incr 4096 KB)
Large Pages configured system wide = 100 (200 MB)
Large Page size = 2048 KB
 
RECOMMENDATION:
Total Shared Global Region size is 810 MB. For optimal performance,
prior to the next instance restart increase the number
of unused Large Pages by atleast 305 2048 KB Large Pages (610 MB)
system wide to get 100% of the Shared
Global Region allocated with Large pages
***********************************************************

Let’s take a look at the kernel parameter SHMMAX now.
On the Oracle VM where I was testing this, the default SHMMAX was set to 64GB. Much bigger
than the 2GB of RAM allocated to the VM. In trying values between 1GB and 64Gb, the
database always started up using HugePages without any issue. However, the value of SHMALL
is one that needs to be set correctly.

In the next example, I lowered SHMALL to below its recommended value.
You will see the output below from SQL*Plus, but nothing was shown in the alert log.
After reading a couple of Documents on MOS, you should
choose the higher of the two values below for SHMALL, it will depend on your database memory sizes.
From MOS note 169706.1 : SHMALL = SHMMAX / OS PAGE_SIZE
From MOS note 301830.1 : SHMALL = sum(SGAs)/ OS PAGE_SIZE

SQL > startup
ORA-27102: out of memory
Linux-x86_64 Error: 28: No space left on device

As I mentioned earlier, HugePages is not compatible with Automatic Memory Mangement.
If you try to start the database using AMM, here is what the entry from the alert log will look like.
Note that you can still use the sga_target parameter, just not memory_target, memory_max_target.
Note also that you can’t use the use_indirect data._buffers parameter.

Starting ORACLE instance (normal)
******************************************************************
Large Pages are not compatible with specified SGA memory parameters
use_large_pages = "ONLY" cannot be used with memory_target,
memory_max_target, or use_indirect_data_buffers parameters
Large Pages are compatible with sga_target and shared_pool_size
******************************************************************

Since the alert log in Oracle 10g does not log any HugePage information, we have to be a little more creative
to uncover clues as to why huge pages are not used . There is a note on MOS that recommends using strace
to help diagnose this type of issue. So let’s use strace on a
sqlplus session when starting the Oracle 10 database to see if that will provide
any useful troubleshooting information. I intentionally lowered the SHMALL parameter in this example to 56000.
Notice here that we are trying to allocate more memory that we have allotted for with the SHMALL kernel parameter.

> strace -f -o output.txt sqlplus / as sysdba
 
SQL*Plus: Release 10.2.0.5.0 - Production
 
Copyright (c) 1982, 2010, Oracle. All Rights Reserved.
 
Connected to an idle instance.
 
SYS:ORA10>startup
ORA-27102: out of memory
Linux-x86_64 Error: 28: No space left on device

In the output.txt file, you should see output similar to the following showing the shmget calls.

9624  shmget(0x22675a08, 232783872, IPC_CREAT|IPC_EXCL|SHM_HUGETLB|0660) = -1 ENOSPC (No space left on device)

In the next example, I lowered the ulimit setting. Here the database successfully started; however, HugePages was
not used as seen in /proc/meminfo.

>strace -f -o output.txt sqlplus / as sysdba
 
SQL*Plus: Release 10.2.0.5.0 - Production
 
Copyright (c) 1982, 2010, Oracle. All Rights Reserved.
 
Connected to an idle instance.
 
SYS:ORA10>;startup
ORACLE instance started.
 
Total System Global Area 230686720 bytes
Fixed Size 2095344 bytes
Variable Size 117442320 bytes
Database Buffers 104857600 bytes
Redo Buffers 6291456 bytes
Database mounted.
Database opened.
 
SYS:ORA10>;exit
Disconnected from Oracle Database 10g Enterprise Edition Release 10.2.0.5.0 - 64bit Production
With the Partitioning, OLAP, Data Mining and Real Application Testing options
 
> cat /proc/meminfo | grep -i huge
HugePages_Total: 520
HugePages_Free: 520
HugePages_Rsvd: 0
Hugepagesize: 2048 kB

From output.txt, you can see that the database tried to use huge pages, but failed and then started up using
the OS standard pages.

10783 shmget(0x22675a08, 232783872, IPC_CREAT|IPC_EXCL|SHM_HUGETLB|0660) = -1 ENOMEM (Cannot allocate memory)
...
10783 shmget(0x22675a08, 232783872, IPC_CREAT|IPC_EXCL|0660) = 12353549

If you do not want an 11g database to startup without HugePages, remember to set the use_large_pages initilization
parameter mentioned previously. When set to “only”, the database will fail to start. You will need to be at least
on version 11.2.0.2 though.

For Oracle 10g, if you want to ensure that your database is using HugePages,
you can use the equation from MOS note 1392543.1 to calculate the
“Large Pages used by this instance” as seen in 11g alert log.
If you have multiple databases, you will need to monitor /proc/meminfo between instance startups.
Note that if HugePages_Total = HugePages_Free, then you at NOT using HugePages.

Large Pages used by this instance (#pages) = HugePages_Total – HugePages_Free + HugePages_Rsvd

 > cat /proc/meminfo | grep -i huge
HugePages_Total: 520
HugePages_Free: 409
HugePages_Rsvd: 0
Hugepagesize: 2048 kB

#pages = 520 – 409 + 0 = 111

Here’s an example using Oracle 11g showing the correlation between the equation and the alert log.

DBM>startup
ORACLE instance started.
 
Total System Global Area 839282688 bytes
Fixed Size 2233000 bytes
Variable Size 780143960 bytes
Database Buffers 50331648 bytes
Redo Buffers 6574080 bytes
Database mounted.
Database opened.
 
[el5-5-64:oracle:ORA10] /home/oracle
> cat /proc/meminfo
HugePages_Total: 520
HugePages_Free: 403
HugePages_Rsvd: 286
Hugepagesize: 2048 kB

#pages = 520 – 403 + 286 = 403, which matches the value that is shown in the alert log snippet below.

Starting ORACLE instance (normal)
****************** Large Pages Information *****************
Parameter use_large_pages = ONLY
 
Total Shared Global Region in Large Pages = 806 MB (100%)
 
Large Pages used by this instance: 403 (806 MB)
Large Pages unused system wide = 117 (234 MB) (alloc incr 4096 KB)
Large Pages configured system wide = 520 (1040 MB)
Large Page size = 2048 KB
***********************************************************

Please see MOS notes Oracle Not Utilizing Hugepages [ID 803238.1] and
Upon startup of Linux database get ORA-27102: out of memory Linux-X86_64 Error: 28: No space left on device [ID 301830.1]
for more troubleshooting examples.

Finally, it is important to remember is that the HugePage confguration is not dynamic. If you make changes to the SGA size, add another database, etc.,
you should revisit the HugePage configuration.

About Rick Miners

Comments

7 Responses to “HugePages Configuration and Monitoring”
  1. Ankur says:

    Hi,

    Nice post, I am following the same from long time, but recently I came across a customer who is running 8 instances on one box having 96GB of Physical RAM.

    Out of that one major Database SGA size is 30GB. other databases are of 5-10GB range.

    what should be the setting of SHMMAX and memlock in this case?

    Thanks,
    Ankur

    • Rick Miners says:

      Hi Ankur,

      In order to get these settings, you will need to know the number of huge pages required for the running instances.
      This will be the value calculated by Oracle’s hugepages_settings.sh script.
      This script calculates the number of pages required for the instances currently running on the machine.

      Once you have this, you can then use the formula shown below to calcualte the value of memlock.
      See Oracle Not Utilizing Hugepages [ID 803238.1] for additional info.
      ( HugePages * Hugepagesize )
      So, for a snippet of code you could put in a bash script, you could do the calculation as follows for the memlock settings.
      MEMLOCK=`echo “$HUGE_PAGES*$HUGE_PAGE_SIZE” | bc -q`
      echo “oracle soft memlock >= $MEMLOCK”
      echo “oracle hard memlock >= $MEMLOCK”

      For the SHMMAX setting, let’s look at the following MOS note, Maximum SHMMAX values for Linux x86 and x86-64 [ID 567506.1] .
      ——————————————————————————————————–
      What is the maximum value of SHMMAX for a 64-bit (x86-64) Linux system?
      Oracle Global Customer Support officially recommends a ” maximum” for SHMMAX of “1/2 of physical RAM”.
      ——————————————————————————————————–
      Of course, you can set it higher as needed at your own discretion, but I would use only as much as you could and leave the rest to the OS for its own.
      The note then goes on to explain the theoritical and physical limits.

  2. Anthony says:

    Thank you – this post was clear and full of detail – saved me today when my system was not using hugepages and swapping like crazy.
    Anthony

  3. Binoy says:

    Thanks for the explanation and code on huge pages.

  4. Google says:

    You’ve made some really good points there. I looked on the
    internet to learn more about the issue and found most individuals will go along with your views on this web
    site.

Trackbacks

Check out what others are saying about this post...
  1. [...] from an engineering background, my profs would be marking many wrong for missing units. I went to Enkitec’s blog by Rick Miners and many of the Oracle Notes. Thought I understood it perfectly only to find I needed to do a bit [...]

  2. [...] here to see what to look for and how to troubleshoot HugePages implementation. Share [...]



Speak Your Mind

Tell us what you're thinking...
and oh, if you want a pic to show with your comment, go get a gravatar!