| |
1. Introduction
System process is accessing memory through virtual addresses, but the CPU must convert it away physical memory address can actually access memory. In order to improve the conversion efficiency, CPU caches the latest mapping between virtual memory addresses and physical memory address by the CPU and stored in a mapping table maintained. In order to maximize access speed of memory, you need to save as many mappings in the mapping table.
In Linux, the memory pages are in the form of division, by default each page is 4K, which means that if a lot of physical memory, the entry of the mapping table will very much affect the retrieval efficiency of the CPU. Because the memory size is fixed, in order to reduce entry map can be taken only way to increase the size of the page.
2. HugePages Profile
2.1 Related Concepts
HugePages in Linux2.6 kernel is introduced, mainly to provide a 4k page and larger page selection.
concept
Concept note
page table
page table is a data structure model virtual memory system for the operating system for storing correspondence between virtual addresses and physical addresses.
When we access memory page table is accessed first, then Linux through the page table mapping to access the real physical memory (ram + swap)
TLB
A Translation Lookaside Buffer (TLB)
TLB is allocated in the cpu in a fixed-size buffer (or cache), to save part of the page table, the CPU faster access and address translation.
hugetlb
hugetlb is TLB entry in the record and point Hugepages.
hugetlbfs
This is a new 2.6 kernel based on the memory file system like tmpfs.
In the TLB by hugetlb to point hugepage. These are allocated hugepage as a memory file system hugetlbfs (similar tmpfs) to the processes.
2.2 HugePages significance
HugePages is a feature of the Linux kernel, it can use hugepage with a larger memory pages to replace the traditional 4K pages. Use HugePage mainly the following benefits:
1. HugePages at system startup, direct distribution and to retain the size of the corresponding memory area.
2. HugePages after the startup, if there is no administrator intervention is not going to release and change.
3. There is no swap.
Notswappable: HugePages are not swappable Therefore thereis no page-in / page-outmechanism overhead.HugePages are universally regarded aspinned..
4. greatly improves the CPU cache stored in the page table covered memory size, thereby improving the TLB hit rate.
Virtual memory addresses to connect to the process page table and then connect to the physical memory. So when accessing memory page tables need to get access to the mapping between the virtual and physical memory, and then access to physical memory.
CPU cache TLB in part used to store part of the page table to improve the conversion speed. Because the page size bigger, so the same size of the TLB, covered memory size is also larger. Improve TLB hit rate, but also improve the speed of address translation.
5. To reduce the page table load.
When XXX system performance test, if no HugePages, pagetable size on the database server is about 5G (which should also lead to the database server memory performance test when the main reason for lack of):
node74: / home / Oracle # cat / proc / meminfo
MemTotal: 16323732 kB
PageTables: 5442384kB
After configuring HugePages, pagetable size is only 124M (Performance Test when memory usage stable at around 80%):
node74: / home / oracle # cat / proc / meminfo
MemTotal: 16323732 kB
PageTables: 127384 kB
Eliminated page tablelookup overhead: because hugepage is not swappable, and all will be no page table lookups.
Faster overall memory performance: Because the virtual memory requires two steps to correspond to the actual physical memory addresses, so fewer pages, reduce the heat to access the page table, the page table to avoid hot bottlenecks.
6. improve memory performance and reduce CPU load, the principle above
2.3 HugePages need attention
1. Hugepages allocation will be set aside after, which is the sum of all SGA size than some instances on the server to a large, almost not possible.
For example Hugepages set to 8G, oracle SGA to 9G, then the oracle at startup will not use this to 8G Hugepages. This 8G wasted. So when setting Hugepages To calculate the size of the SGA, we will give back a script to calculate.
2. The other processes can not use Hugepages memory, so do not set too large, slightly bigger than the SGA SGA can be used to ensure hugepages enough.
3. meminfo and Hugepage related, there are four:
HugePages_Total: 4611
HugePages_Free: 474
HugePages_Rsvd: 467
Hugepagesize: 2048 kB
The number of pages HugePages_Total to the assigned, and Hugepagesize is obtained by multiplying the allocated memory size. 4611 * 2/1024 about 9GB
HugePages_Free the number Hugepages I never have been used before. Even oraclesga this memory has been allocated, but if not actually written, then see or Free of. This is easily misunderstood.
HugePages_Rsvd reserve has been allocated for the page number but not yet used. When Oracle has just started, most of the memory are Reserved and should be Free, with the oracle SGA use, Reserved and Free will continue to decrease.
HugePages_Free-HugePages_Rsvd this part is not being used to memory, if no other oracle instance, this memory may never be used to, which is being wasted.
4. HugePages and oracle AMM (Automatic Memory Management) are mutually exclusive, so use HugePages must set the memory parameters MEMORY_TARGET / MEMORY_MAX_TARGET zero.
3. Configure HugePages
3.1 modify kernel parameters memlock
Modify kernel parameters memlock, the unit is KB, if the memory is 16G, slightly smaller than the size of memlock of physical memory. Program lock 12GB memory size. Parameter is set to greater than SGA is no harm.
Log in as root two database servers, file editing limits.conf:
node74: ~ # vi /etc/security/limits.conf
Add the following two lines:
* Soft memlock 12582912
* Hard memlock 12582912
3.2 verify memlock limit
Root and oracle users to log in again, check memlocklimit
node74: ~ # ulimit -l
12582912
oracle @ node74: ~> ulimit -l
12582912
3.3 Disable AMM
If you are using 11G and later, AMM has been enabled by default, but the AMM and Hugepages are not compatible, you must first disable AMM. Disable AMM, follow these steps:
3.3.1 Close database instance
Oracle user has two database servers, through sqlplus closed two database instances.
oracle @ node74: ~> sqlplus / as sysdba
SQL> shutdown immediate
3.3.2 Create pfile
Users log on to the oracle one of the hosts, execute the following command to create a pfile:
oracle @ node74: ~> sqlplus / as sysdba
SQL> create pfile = '/ home / oracle / pfile.ora' fromspfile = '+ DG_ORA / orcl / spfileorcl.ora';
3.3.3 Editing pfile
Edit pfile, and delete memory_max_target memory_target parameters:
oracle @ node74: ~> vi /home/oracle/pfile.ora
Delete the following lines:
orcl1.memory_max_target = 11114905600
orcl2.memory_max_target = 11114905600
* .memory_max_target = 0
orcl1.memory_target = 11114905600
orcl2.memory_target = 11114905600
* .memory_target = 0
Save the modified file.
3.3.4 Create spfile
Execute the following command to create the spfile:
oracle @ node74: ~> sqlplus / as sysdba
SQL> create spfile = '+ DG_ORA / orcl / spfileorcl.ora'from pfile =' / home / oracle / pfile.ora ';
3.3.5 modify the system parameters kernel.shmall
Kernel.shmall maximum shared memory size is a system that can be used. Units are page (4KB). After disabling AMM, you need to modify the system parameters kernel.shmall, this parameter is set too low, it may cause the database startup failure ORA-27102 (see Appendix 4.2).
ORACLE recommended that it is set to the sum of all SGA database instance system. For example SGA the sum of 9GB, you need to set kernel.shmall = 9 * 1024 * 1024/4 = 2359296.
Log in as root two database servers, edit sysctl.conf file.
node74: ~ # vi /etc/sysctl.conf
Kernel.shmall modify parameters:
kernel.shmall = 2359296
Execute sysctl -p to validate the configuration:
node74: ~ # sysctl -p
3.3.6 Start the database instance
Users log on to the oracle two database servers, start two instances of the database through sqlplus.
oracle @ node74: ~> sqlplus / as sysdba
SQL> startup
3.4 Calculation need to use hugepage page size
Ensure that all instances have been started (including ASM), then run hugepages_settings.sh (script content Appendix 4.1) to evaluate the need to set Hugepages size to root user.
node74: / home / oracle # ./hugepages_settings.sh
This script is provided by Doc ID 401749.1 from MyOracle Support
(Http://support.oracle.com) where it is intended tocompute values for
the recommended HugePages / HugeTLB configuration forthe current shared
memory segments. Before proceeding with the executionplease make sure
that:
* OracleDatabase instance (s) are up and running
* OracleDatabase 11g Automatic Memory Management (AMM) is not setup
(See Doc ID749851.1)
* The sharedmemory segments can be listed by command:
# Ipcs -m
Press Enter toproceed ...
---- Press the Enter key
Recommended setting: vm.nr_hugepages = 4611
You can also manually calculate:
nr_hugepages> = SGA_Target / Hugepagesize
= 9G * 1024M / 2M
4608 =
Take a slightly larger than the 4608 value.
3.5 modify the parameters vm.nr_hugepages
Log in as root two database servers, edit /etc/sysctl.conf:
node74: ~ # vi /etc/sysctl.conf
Modify vm.nr_hugepages parameter value calculated from the previous step:
vm.nr_hugepages = 4611
Execute sysctl -p to validate the configuration:
node74: ~ # sysctl -p
3.6 Stop the database instance, restart the OS
Stop all database instances, restart the operating system. (Theoretically without rebooting the operating system, it is recommended to restart)
3.7 Check the settings have taken effect
After rebooting the system, start the whole database, using the following command to check the configuration to take effect:
node74: ~ # grep HugePages / proc / meminfo
HugePages_Total: 4611
HugePages_Free: 2394
HugePages_Rsvd: 2387
HugePages_Surp: 0
HugePages_Free
4. Appendix
4.1 Script hugepages_settings.sh
#! / Bin / bash
#
# Hugepages_settings.sh
#
# Linux bash script to compute values for the
# Recommended HugePages / HugeTLB configuration
#
# Note: This script does calculation for all shared memory
# Segments available when the script is run, no matter it
# Is an Oracle RDBMS shared memory segment or not.
#
# This script is provided by Doc ID 401749.1 from My Oracle Support
# Http://support.oracle.com
# Welcome text
echo "
This script is provided by Doc ID 401749.1 from My Oracle Support
(Http://support.oracle.com) where it is intended to compute values for
the recommended HugePages / HugeTLB configuration for the current shared
memory segments. Before proceeding with the execution please make sure
that:
* Oracle Database instance (s) are up and running
* Oracle Database 11g Automatic Memory Management (AMM) is not setup
(See Doc ID 749851.1)
* The shared memory segments can be listed by command:
# Ipcs -m
Press Enter to proceed ... "
read
# Check for the kernel version
. KERN = `uname -r | awk -F '{printf ("% d% d \ n. ", $ 1, $ 2);}'`
# Find out the HugePage size
HPG_SZ = `grep Hugepagesize / proc / meminfo | awk '{print $ 2}'`
# Initialize the counter
NUM_PG = 0
# Cumulative number of pages required to handle the running shared memory segments
for SEG_BYTES in `ipcs -m | awk '{print $ 5}' | grep" [0-9] [0-9] * "`
do
MIN_PG = `echo" $ SEG_BYTES / ($ HPG_SZ * 1024) "| bc -q`
if [$ MIN_PG -gt 0]; then
NUM_PG = `echo" $ NUM_PG + $ MIN_PG + 1 "| bc -q`
fi
done
RES_BYTES = `echo" $ NUM_PG * $ HPG_SZ * 1024 "| bc -q`
# An SGA less than 100MB does not make sense
# Bail out if that is the case
if [$ RES_BYTES -lt 100000000]; then
echo "***********"
echo "** ERROR **"
echo "***********"
echo "Sorry! There are not enough total of shared memory segments allocated for
HugePages configuration. HugePages can only be used for shared memory segments
that you can list by command:
# Ipcs -m
. Of a size that can match an Oracle Database SGA Please make sure that:
* Oracle Database instance is up and running
* Oracle Database 11g Automatic Memory Management (AMM) is not configured "
exit 1
fi
# Finish with results
case $ KERN in
'2.4') HUGETLB_POOL = `echo" $ NUM_PG * $ HPG_SZ / 1024 "| bc -q`;
echo "Recommended setting: vm.hugetlb_pool = $ HUGETLB_POOL" ;;
'2.6') echo "Recommended setting: vm.nr_hugepages = $ NUM_PG" ;;
*) Echo "Unrecognized kernel version $ KERN. Exiting." ;;
esac
# End
4.2 database startup error ORA-27102
Upon startup of Linux database getORA-27102: out of memory Linux-X86_64 Error: 28: No space left on device
Subject:
Upon startup of Linux database getORA-27102: out of memory Linux-X86_64 Error: 28: No space left on device
Doc ID:
Note: 301830.1
Type:
PROBLEM
Last Revision Date:
31-OCT-2008
Status:
PUBLISHED
In this Document
Symptoms
Changes
Cause
Solution
References
-------------------------------------------------- ------------------------------
Applies to:
OracleServer- Enterprise Edition - Version: 9.2.0.4 to 11.1.0.6
UnitedLinux x86-64
Red Hat Enterprise Linux Advanced Serverx86-64 (AMD Opetron Architecture)
x86 64 bit
Symptoms
When trying to increase the SGA to approachhalf available RAM with an Oracle 64bit version on a Linux 64bit operatingsystem, even though shmmax is set to match half the amount of RAM, youget the following error when trying to start the instance:
SQL> startup nomount
ORA-27102: out of memory
Linux-x86_64 Error: 28: No space left ondevice
Changes
shmall is too small, most likely is set tothe default setting of 2097152
$ Cat / proc / sys / kernel / shmall
2097152
Cause
shmall is the total amount of sharedmemory, in pages, that the system can use at one time.
Solution
Set shmall equal to the sum of all the SGAson the system, divided by the page size.
The page size can be determined using thefollowing command:
$ Getconf PAGE_SIZE
4096
For example, if the sum of all the SGAs onthe system is 16Gb and the result of '$ getconf PAGE_SIZE' is 4096 (4Kb) then set shmall to 4194304 (4Mb)
As the root user set the shmall to 4194304in the /etc/sysctl.conf file:
kernel.shmall = 4194304
then run the following command:
# Sysctl -p
# Cat / proc / sys / kernel / shmall
4194304
NOTE:
The above command loads the new value and areboot is not necessary
Switch back to being the oracle user andretry the startup command.
References
Note 169706.1 - Oracle Database onAIX, HP-UX, Linux, Mac OS X, Solaris, Tru64 Unix Operating SystemsInstallation and Configuration Requirements Quick Reference (8.0.5 to 11.1)
Keywords
NO ~ SPACE ~ LEFT ~ ON ~ DEVICE; START ~ INSTANCE; OUT ~ OF ~ MEMORY; 64BIT; |
|
|
|