<?xml version="1.0" encoding="UTF-8"?>
|
<!--
|
! CCPL HEADER START
|
!
|
! This work is licensed under the Creative Commons
|
! Attribution-NonCommercial-NoDerivs 3.0 Unported License.
|
! To view a copy of this license, visit
|
! http://creativecommons.org/licenses/by-nc-nd/3.0/
|
! or send a letter to Creative Commons, 444 Castro Street,
|
! Suite 900, Mountain View, California, 94041, USA.
|
!
|
! You can also obtain a copy of the license at legal-notices/CC-BY-NC-ND.txt.
|
! See the License for the specific language governing permissions
|
! and limitations under the License.
|
!
|
! If applicable, add the following below this CCPL HEADER, with the fields
|
! enclosed by brackets "[]" replaced with your own identifying information:
|
! Portions Copyright [yyyy] [name of copyright owner]
|
!
|
! CCPL HEADER END
|
!
|
! Copyright 2011-2015 ForgeRock AS.
|
!
|
-->
|
<chapter xml:id='chap-tuning'
|
xmlns='http://docbook.org/ns/docbook' version='5.0' xml:lang='en'
|
xmlns:xsi='http://www.w3.org/2001/XMLSchema-instance'
|
xsi:schemaLocation='http://docbook.org/ns/docbook
|
http://docbook.org/xml/5.0/xsd/docbook.xsd'
|
xmlns:xlink='http://www.w3.org/1999/xlink'
|
xmlns:xinclude='http://www.w3.org/2001/XInclude'>
|
<title>Tuning Servers For Performance</title>
|
<indexterm><primary>Performance tuning</primary></indexterm>
|
|
<para>Server tuning refers to the art of adjusting server, JVM, and system
|
configuration to meet the service level performance requirements of directory
|
clients. In the optimal case you achieve service level performance
|
requirements without much tuning at all, perhaps only setting JVM runtime
|
options when installing OpenDJ.</para>
|
|
<para>If you are reading this chapter, however, you are probably not
|
facing an optimal situation. Instead you are looking for trade offs that
|
maximize performance for clients given the constraints of your deployment.
|
This chapter therefore aims to provide suggestions on how to measure and
|
to improve directory service performance for better trade offs.</para>
|
|
<section xml:id="perf-define-starting-points">
|
<title>Defining Performance Requirements & Constraints</title>
|
|
<para>Your key performance requirement is most likely to satisfy your
|
users or customers with the resources available to you. Before you can
|
solve potential performance problems, define what those users or customers
|
expect, and determine what resources you will have to satisfy their
|
expectations.</para>
|
|
<section xml:id="perf-sla">
|
<title>Service-Level Agreements</title>
|
|
<para>Service-level agreement (SLA) is a formal name for what directory
|
client applications and the people who run them expect from your service in
|
terms of performance.</para>
|
|
<para>SLAs might cover many aspects of the directory service. Whether or not
|
your SLA is formally defined, you ought to know what is expected, or at least
|
what you provide, in the following four areas.</para>
|
|
<itemizedlist>
|
<listitem>
|
<para>Directory service <firstterm>response times</firstterm></para>
|
|
<para>Directory service response times range from less than a
|
millisecond on average across a low latency connection on the same
|
network to however long it takes your network to deliver the response.
|
More important than average or best response times is the response time
|
distribution, because applications set timeouts based on worst case
|
scenarios. For example, a response time performance requirement might
|
be defined as, "Directory response times must average less than 10
|
milliseconds for all operations except searches returning more than 10
|
entries, with 99.9% of response times under 40 milliseconds."</para>
|
</listitem>
|
<listitem>
|
<para>Directory service <firstterm>throughput</firstterm></para>
|
|
<indexterm>
|
<primary>Replication</primary>
|
<secondary>Write throughput</secondary>
|
</indexterm>
|
<para>Directory service throughput can range up to many thousands of
|
operations per second. In fact there is no upper limit for read operations
|
such as searches, because only write operations must be replicated. To
|
increase read throughput, simply add additional replicas. More important
|
than average throughput is peak throughput. You might have peak write
|
throughput in the middle of the night when batch jobs update entries in
|
bulk, and peak binds for a special event or first thing Monday morning.
|
For example, a throughput performance requirement might be expressed as,
|
"The directory service must sustain a mix of 5,000 operations per second
|
made up of 70% reads, 25% modifies, 3% adds, and 2% deletes."</para>
|
|
<para>Even better is to mimic the behavior of key operations for
|
performance testing, so that you understand the patterns of operations
|
in the throughput you need to provide.</para>
|
</listitem>
|
<listitem>
|
<para>Directory service <firstterm>availability</firstterm></para>
|
|
<para>OpenDJ is designed to let you build directory services that are
|
basically available, including during maintenance and even upgrade of
|
individual servers. Yet, in order to reach very high levels of
|
availability, you must make sure not only that the software is
|
designed for availability, but also that your operations execute in
|
such a way as to preserve availability. Availability requirements
|
can be as lax as best effort, or as stringent as 99.999% or more
|
uptime.</para>
|
|
<para>Replication is the OpenDJ feature that allows you to build a
|
highly available directory service.</para>
|
</listitem>
|
<listitem>
|
<para>Directory service administrative support</para>
|
|
<para>Do not forget to make sure you understand and set expectations
|
about how you support your users when they run into trouble. Directory
|
services can perhaps help you turn password management into a self-service
|
visit to a web site, but some users no doubt still need to know what they
|
can expect if they need your help.</para>
|
</listitem>
|
</itemizedlist>
|
|
<para>Writing down the SLA, even if your first version consists of
|
guesses, helps you reduce performance tuning from an open-ended project
|
to a clear set of measurable goals for a manageable project with a definite
|
outcome.</para>
|
</section>
|
|
<section xml:id="perf-constraints">
|
<title>Available Resources</title>
|
|
<para>With your SLA in hand, take inventory of the server, networks,
|
storage, people, and other resources at your disposal. Now is the time to
|
estimate whether it is possible to meet the requirements at all.</para>
|
|
<para>If for example you are expected to serve more throughput than the
|
network can transfer, maintain high availability with only one physical
|
machine, store 100 GB of backups on a 50 GB partition, or provide 24/7
|
support all alone, no amount of tweaking available resources is likely to
|
fix the problem.</para>
|
|
<para>When checking that the resources you have at least theoretically
|
suffice to meet your requirements, do not forget that high availability in
|
particular requires at least two of everything to avoid single points
|
of failure. Be sure to list the resources you expect to have, when and how
|
long you expect to have them, and why you need them. Also make note of
|
what is missing and why.</para>
|
|
<para>
|
In addition to the suggestions in this section, also read the section on
|
<link xlink:href="release-notes#prerequisites-hardware" xlink:show="new"
|
xlink:role="http://docbook.org/xlink/role/olink"><citetitle
|
>Hardware</citetitle></link> in the <citetitle>Release Notes</citetitle>.
|
</para>
|
|
<section xml:id="perf-hardware">
|
<title>Server Hardware Recommendations</title>
|
|
<para>Concerning server hardware, OpenDJ runs on systems with Java support,
|
and is therefore quite portable. That said, OpenDJ tends to perform best on
|
single-board, x86 systems due to low memory latency.</para>
|
</section>
|
|
<section xml:id="perf-storage">
|
<title>Storage Recommendations</title>
|
|
<para>OpenDJ is designed to work with local storage for the database,
|
not for network file systems such as NFS.</para>
|
|
<para>High performance storage is essential if you need to handle high
|
write throughput.</para>
|
|
<para condition="local-db">The Berkeley Java Edition DB works well with traditional disks as
|
long as the database cache size allows the DB to stay fully cached in
|
memory. This is the case because the database transaction log is append
|
only. When the DB is too big to stay cached in memory, however, then
|
cache misses lead to random disk access, slowing OpenDJ performance.</para>
|
|
<para condition="local-db">You might mitigate this effect by using solid-state disks for
|
persistent storage, or for file system cache.</para>
|
|
<para>Regarding database size on disk, if you have sustained write traffic
|
then the database grows to about twice its initial size on disk. This is
|
normal, and due to the way the database manages its logs. The size on disk
|
does not impact the DB cache size requirements.</para>
|
|
<xinclude:include href="../shared/para-write-barriers.xml" />
|
</section>
|
</section>
|
</section>
|
|
<section xml:id="perf-testing">
|
<title>Testing Performance</title>
|
|
<para>Even if you do not need high availability, you still need two of
|
everything, because your test environment needs to mimic your production
|
environment as closely as possible if you want to avoid nasty
|
surprises.</para>
|
|
<para>In your test environment, you set up OpenDJ as you will later in
|
production, and then conduct experiments to determine how best to meet
|
the requirements defined in the SLA.</para>
|
|
<para>Use <link xlink:show="new" xlink:href="reference#make-ldif-1"
|
xlink:role="http://docbook.org/xlink/role/olink">make-ldif</link> to generate
|
sample data that match what you expect to find in production.</para>
|
|
<para>
|
The OpenDJ LDAP Toolkit provides command-line tools
|
to help with basic performance testing.
|
</para>
|
|
<itemizedlist>
|
<listitem>
|
<para>
|
The
|
<link
|
xlink:show="new"
|
xlink:href="${coreDocBase}dev-guide/#addrate-1"
|
>addrate</link> command measures add and delete throughput and response time.
|
</para>
|
</listitem>
|
<listitem>
|
<para>The <link xlink:show="new"
|
xlink:href="${coreDocBase}dev-guide/#authrate-1">authrate</link> command
|
measures bind throughput and response time.</para>
|
</listitem>
|
<listitem>
|
<para>The <link xlink:show="new"
|
xlink:href="${coreDocBase}dev-guide/#modrate-1">modrate</link> command
|
measures modification throughput and response time.</para>
|
</listitem>
|
<listitem>
|
<para>The <link xlink:show="new"
|
xlink:href="${coreDocBase}dev-guide/#searchrate-1">searchrate</link> command
|
measures search throughput and response time.</para>
|
</listitem>
|
</itemizedlist>
|
|
<para>
|
All these commands show you information about the response time distributions,
|
and allow you to perform tests at specific levels of throughput.
|
</para>
|
|
<para>
|
If you need additional precision when evaluating response times,
|
use the global configuration setting <literal>etime-resolution</literal>,
|
to change elapsed processing time resolution
|
from milliseconds (default) to nanoseconds.
|
</para>
|
|
<screen>
|
$ <userinput>dsconfig \
|
set-global-configuration-prop \
|
--port 4444 \
|
--hostname opendj.example.com \
|
--bindDN "cn=Directory Manager" \
|
--bindPassword password \
|
--set etime-resolution:nanoseconds \
|
--no-prompt</userinput>
|
</screen>
|
</section>
|
|
<section xml:id="perf-tweaking">
|
<title>Tweaking OpenDJ Performance</title>
|
|
<para>When your tests show that OpenDJ performance is lacking even though
|
you have the right underlying network, hardware, storage, and system
|
resources in place, you can tweak OpenDJ performance in a number of ways.
|
This section mentions the most common tweaks.</para>
|
|
<xinclude:include href="../shared/sec-prerequisites-file-descriptors.xml" />
|
|
<section xml:id="perf-java">
|
<title>Java Settings</title>
|
|
<para>Default Java settings let you evaluate OpenDJ using limited system
|
resources. If you need high performance for production system, test with
|
the following JVM options. These apply to the Sun/Oracle JVM.</para>
|
|
<tip>
|
<para>
|
To apply JVM settings for your server,
|
edit <filename>config/java.properties</filename>,
|
and apply the changes with the
|
<link
|
xlink:show="new"
|
xlink:href="reference#dsjavaproperties-1"
|
xlink:role="http://docbook.org/xlink/role/olink"
|
><command>dsjavaproperties</command></link> command.
|
</para>
|
</tip>
|
|
<variablelist>
|
<varlistentry>
|
<term><option>-server</option></term>
|
<listitem>
|
<para>Use the C2 compiler and optimizer.</para>
|
</listitem>
|
</varlistentry>
|
<varlistentry>
|
<term><option>-d64</option></term>
|
<listitem>
|
<para>To use a heap larger than about 3.5 GB on a 64-bit system, use
|
this option.</para>
|
</listitem>
|
</varlistentry>
|
<varlistentry>
|
<term><option>-Xms, -Xmx</option></term>
|
<listitem>
|
<para>Set both minimum and maximum heap size to the same value to avoid
|
resizing. Leave space for the entire DB cache and more.</para>
|
</listitem>
|
</varlistentry>
|
<varlistentry>
|
<term><option>-Xmn</option></term>
|
<listitem>
|
<para>Set the new generation size between 1-4 GB for high throughput
|
deployments, but leave enough overall JVM heap to avoid overlaps with
|
the space used for DB cache.</para>
|
</listitem>
|
</varlistentry>
|
<varlistentry>
|
<term><option>-XX:MaxTenuringThreshold=1</option></term>
|
<listitem>
|
<para>Force OpenDJ to create only objects that have either a short
|
lifetime, or a long lifetime.</para>
|
</listitem>
|
</varlistentry>
|
<varlistentry>
|
<term><option>-XX:+UseConcMarkSweepGC</option></term>
|
<listitem>
|
<para>The CMS garbage collector tends to give the best performance
|
characteristics. You might also consider the G1 garbage collector.</para>
|
</listitem>
|
</varlistentry>
|
<varlistentry>
|
<term><option>-XX:+PrintGCDetails</option></term>
|
<term><option>-XX:+PrintGCTimeStamps</option></term>
|
<listitem>
|
<para>Use these when diagnosing JVM tuning problems. You can turn them
|
off when everything is running smoothly.</para>
|
</listitem>
|
</varlistentry>
|
<varlistentry>
|
<term><option>-XX:+UseCompressedOops</option></term>
|
<listitem>
|
<para>Java object pointers normally have the same size as native machine
|
pointers. If you run a small, but 64-bit JVM, then compressed object
|
pointers can save space. Set this option when you have a 64-bit JVM,
|
<option>-Xmx</option> less than 32 GB, and Java SE 6u23 or later.</para>
|
</listitem>
|
</varlistentry>
|
</variablelist>
|
</section>
|
|
<section xml:id="perf-data-storage">
|
<title>Data Storage Settings</title>
|
|
<para>By default, OpenDJ compresses attribute descriptions and object class
|
sets to reduce data size. This is called compact encoding.</para>
|
|
<para>
|
By default, OpenDJ does not however compress entries
|
stored in its backend database.
|
If your entries hold values that compress well —
|
such as text, and not JPEG photos or MP3 audio —
|
you can gain space by setting the backend property
|
<literal>entries-compressed</literal>,
|
to <literal>true</literal> before you (re-)import data from LDIF.
|
With <literal>entries-compressed: true</literal>
|
OpenDJ compresses entries before writing them to the database.<footnote>
|
<para>
|
OpenDJ does not proactively rewrite all entries in the database
|
after you change the settings.
|
Instead, to force OpenDJ to compress all entries,
|
import the data from LDIF.
|
</para>
|
</footnote>
|
</para>
|
|
<screen>
|
$ <userinput>dsconfig \
|
set-backend-prop \
|
--port 4444 \
|
--hostname opendj.example.com \
|
--bindDN "cn=Directory Manager" \
|
--bindPassword password \
|
--backend-name userRoot \
|
--set entries-compressed:true \
|
--trustAll \
|
--no-prompt</userinput>
|
|
$ <userinput>import-ldif \
|
--port 4444 \
|
--hostname opendj.example.com \
|
--bindDN "cn=Directory Manager" \
|
--bindPassword password \
|
--ldifFile /path/to/Example.ldif \
|
--backendID userRoot \
|
--includeBranch dc=example,dc=com \
|
--start 0</userinput>
|
<computeroutput>Import task 20120917100628767 scheduled to start Sep 17, 2012 10:06:28 AM CEST</computeroutput>
|
</screen>
|
</section>
|
|
<section xml:id="perf-import">
|
<title>LDIF Import Settings</title>
|
<indexterm>
|
<primary>Importing data</primary>
|
<secondary>Performance</secondary>
|
</indexterm>
|
|
<para>You can tweak OpenDJ to speed up import of large LDIF files.</para>
|
|
<para>
|
By default, the temporary directory used for scratch files is
|
<filename>import-tmp</filename> under the directory
|
where you installed OpenDJ.
|
Use the
|
<link
|
xlink:show="new"
|
xlink:href="reference#import-ldif-1"
|
xlink:role="http://docbook.org/xlink/role/olink"
|
><command>import-ldif</command></link> command
|
with the <option>--tmpdirectory</option> option
|
to set this directory to a <literal>tmpfs</literal> file system,
|
such as <filename>/tmp</filename>.
|
</para>
|
|
<para>In some cases, you can improve performance by using the
|
<option>--threadCount</option> option with the
|
<command>import-ldif</command> command to set the thread count larger than
|
the default, which is twice the number of CPUs.</para>
|
|
<para>If you are certain your LDIF contains only valid entries with
|
correct syntax, because the LDIF was exported from OpenDJ with all checks
|
active for example, you can skip schema and DN validation. Use the
|
<option>--skipSchemaValidation</option> and
|
<option>--skipDNValidation</option> options with the
|
<command>import-ldif</command> command to skip validation.</para>
|
</section>
|
|
<section xml:id="perf-db-cache">
|
<title>Database Cache Settings</title>
|
|
<para>
|
Database cache size is, by default, set as a percentage of the JVM heap
|
by using the backend property <literal>db-cache-percent</literal>.
|
Alternatively, you use the backend property <literal>db-cache-size</literal>,
|
to set the size.
|
If you set up multiple database backends,
|
the total percent of JVM heap used must remain less than 100,
|
and must leave space for other uses.
|
Default settings work for servers
|
with one user data backend JVM heaps up to 2 GB.
|
For heaps larger than 2 GB,
|
you can allocate a larger percentage of heap space to DB cache.
|
</para>
|
|
<para>Depending on the size of your database, you have a choice to make
|
about database cache settings.</para>
|
|
<para>
|
By caching the entire database in the JVM heap,
|
you can get more deterministic response times and limit disk I/O.
|
Yet, caching the whole DB can require a very large JVM,
|
which you must pre-load on startup,
|
and which can result in long garbage collections
|
and a difficult-to-manage JVM.
|
Test database pre-load on startup by setting
|
the <literal>preload-time-limit</literal> for the backend.
|
</para>
|
|
<screen>
|
$ <userinput>dsconfig \
|
set-backend-prop \
|
--port 4444 \
|
--hostname opendj.example.com \
|
--bindDN "cn=Directory Manager" \
|
--bindPassword password \
|
--backend-name userRoot \
|
--set preload-time-limit:30m \
|
--trustAll \
|
--no-prompt</userinput>
|
</screen>
|
|
<para>Database pre-load is single-threaded, and loads each database one
|
at a time.</para>
|
|
<para>By allowing file system cache to hold the portion of database that
|
does not fit in DB cache, you trade less deterministic and slightly slower
|
response times for not having to pre-load the DB and not having garbage
|
collection pauses with large JVMs. How you configure the file system cache
|
depends on your operating system.</para>
|
</section>
|
|
<section xml:id="perf-entry-cache">
|
<title>Caching Large, Frequently Used Entries</title>
|
|
<para>
|
OpenDJ implements an entry cache designed for deployments
|
with a few large entries that are regularly updated or accessed.
|
The common use case is a deployment with a few large static groups
|
that are updated or accessed regularly.
|
An entry cache is used to keep such groups in memory in a format that
|
avoids the need to constantly read and deserialize the large entries.
|
</para>
|
|
<para>
|
When configuring an entry cache,
|
take care to include only the entries that need to be cached
|
by using the configuration properties
|
<literal>include-filter</literal> and <literal>exclude-filter</literal>.
|
The memory devoted to the entry cache is not available for other purposes.
|
</para>
|
|
<para>
|
The following example adds a Soft Reference entry cache
|
to hold entries that match the filter
|
<literal>(ou=Large Static Groups)</literal>.
|
A Soft Reference entry cache allows cached entries to be released
|
if the JVM is running low on memory.
|
A Soft Reference entry cache has no maximum size setting,
|
so the number of entries cached is limited only by the
|
<literal>include-filter</literal> and <literal>exclude-filter</literal> settings.
|
</para>
|
|
<screen>
|
$ <userinput>dsconfig \
|
create-entry-cache \
|
--port 4444 \
|
--hostname opendj.example.com \
|
--bindDN "cn=Directory Manager" \
|
--bindPassword password \
|
--cache-name "Large Group Entry Cache" \
|
--type soft-reference \
|
--set cache-level:1 \
|
--set include-filter:"(ou=Large Static Groups)" \
|
--set enabled:true \
|
--trustAll \
|
--no-prompt</userinput>
|
</screen>
|
|
<para>
|
The entry cache configuration takes effect when the entry cache is enabled.
|
</para>
|
</section>
|
|
<section xml:id="perf-logging">
|
<title>Logging Settings</title>
|
|
<para>Debug logs trace the internal workings of OpenDJ, and therefore
|
generally should be used sparingly, especially in high performance
|
deployments.</para>
|
|
<para>In general leave other logs active for production environments to
|
help troubleshoot any issues that arise.</para>
|
|
<para>For OpenDJ servers handling very high throughput, however, such as
|
100,000 operations per second or more, the access log constitute a
|
performance bottleneck, as each client request results in multiple access log
|
messages. Consider disabling the access log in such cases.</para>
|
|
<screen>
|
$ <userinput>dsconfig \
|
set-log-publisher-prop \
|
--port 4444 \
|
--hostname opendj.example.com \
|
--bindDN "cn=Directory Manager" \
|
--bindPassword password \
|
--publisher-name "File-Based Access Logger" \
|
--set enabled:false \
|
--trustAll \
|
--no-prompt</userinput>
|
</screen>
|
</section>
|
</section>
|
</chapter>
|
