Indexes OpenDJ provides several indexing schemes to speed up searches. When a client requests a directory search operation, the client sends the server a filter expression such as (&(uid=*jensen*)(l=Stavanger)). The server then uses applicable indexes to find entries with attribute values likely to match the search. If no indexes are applicable, then the server potentially has to go through all entries to look for candidate matches. Looking through all entries is resource-intensive for large directories. For this reason, the unindexed-search privilege, allowing users to request searches for which no applicable index exists, is reserved for the directory root user by default. Rather than granting the unindexed-search privilege to more users and client applications, you configure indexes to correspond to the searches that clients need to perform. This chapter first describes index types, then demonstrates how to index attribute values. This chapter also lists the default indexing configuration for OpenDJ directory server.

OpenDJ provides several different index types, each corresponding to a different type of search.

Indexes Approximate An approximate index is used to match values that "sound like" those provided in the filter. An approximate index on cn allows clients to find people even when they misspell names as in the following example. $ ldapsearch --port 1389 --baseDN dc=example,dc=com "(cn~=Babs Jansen)" cn dn: uid=bjensen,ou=People,dc=example,dc=com cn: Barbara Jensen cn: Babs Jensen

Indexes Equality An equality index is used to match values that correspond exactly (though generally without case sensitivity) to the value provided in the search filter. An equality index requires clients to match values without wildcards or misspellings. $ ldapsearch --port 1389 --baseDN dc=example,dc=com "(uid=bjensen)" mail dn: uid=bjensen,ou=People,dc=example,dc=com mail: bjensen@example.com

Indexes Ordering An ordering index is used to match values for a filter that specifies a range. The ds-sync-hist has an ordering index by default because searches on that attribute often seek entries with changes more recent than the last time a search was performed. The following example shows a search that specifies ranges. $ ldapsearch --port 1389 --baseDN dc=example,dc=com "(&(uidNumber>=1120)(roomNumber>=4500))" uid dn: uid=charvey,ou=People,dc=example,dc=com uid: charvey dn: uid=eward,ou=People,dc=example,dc=com uid: eward dn: uid=mvaughan,ou=People,dc=example,dc=com uid: mvaughan dn: uid=pchassin,ou=People,dc=example,dc=com uid: pchassin

Indexes Presence A presence index is used to match the fact that an attribute is present on the entry, regardless of the value. The aci attribute is indexed for presence by default to allow quick retrieval of entries with ACIs. $ ldapsearch --port 1389 --baseDN dc=example,dc=com "(aci=*)" - dn: dc=example,dc=com dn: ou=People,dc=example,dc=com

Indexes Substring A substring index is used to match values specified with wildcards in the filter. Substring indexes can be expensive to maintain, especially for large attribute values. $ ldapsearch --port 1389 --baseDN dc=example,dc=com "(cn=Barb*)" cn dn: uid=bfrancis,ou=People,dc=example,dc=com cn: Barbara Francis dn: uid=bhal2,ou=People,dc=example,dc=com cn: Barbara Hall dn: uid=bjablons,ou=People,dc=example,dc=com cn: Barbara Jablonski dn: uid=bjensen,ou=People,dc=example,dc=com cn: Barbara Jensen cn: Babs Jensen dn: uid=bmaddox,ou=People,dc=example,dc=com cn: Barbara Maddox

Indexes Virtual list view (browsing) A VLV or browsing index are designed to help the server respond to client applications that need virtual list view results, for example to browse through a long list in a GUI. They also help the server respond to clients that request server-side sorting of the search results. VLV indexes correspond to particular searches. Configure your VLV indexes using the Control Panel, and copy the command-line equivalent from the Details pane for the operation, if necessary.

Indexes Extensible matching rule In some cases you need an index for a matching rule other than those described above. For example, OpenDJ supports generalized time based matching so applications can search for all times later than, or earlier than a specified time.

Indexes Configuring You modify index configurations using the dsconfig command. The configuration changes then take effect after you rebuild the index according to the new configuration, using the rebuild-index. The dsconfig --help-database command lists subcommands for creating, reading, updating, and deleting index configuration. Indexes are per directory backend rather than per suffix. To maintain separate indexes for different suffixes on the same directory server, put the suffixes in different backends.

You can configure standard indexes from the Control Panel, and also on the command line using the dsconfig command. After you finish configuring the index, you must rebuild the index for the changes to take effect. The following example creates a new substring index for description. $ dsconfig create-local-db-index --port 4444 --hostname opendj.example.com --bindDN "cn=Directory Manager" --bindPassword password --backend-name userRoot --index-name description --set index-type:substring --trustAll --no-prompt Indexes Approximate The following example configures an approximate index for cn (common name). $ dsconfig set-local-db-index-prop --port 4444 --hostname opendj.example.com --bindDN "cn=Directory Manager" --bindPassword password --backend-name userRoot --index-name cn --set index-type:approximate --trustAll --no-prompt Indexes Extensible matching rule The OpenDJ Control Panel New Index window does not help you set up extensible matching rule indexes. Use the dsconfig command instead. The following example configures an extensible matching rule index for "later than" and "earlier than" generalized time matching on a lastLoginTime attribute. $ dsconfig create-local-db-index --port 4444 --hostname opendj.example.com --bindDN "cn=Directory Manager" --bindPassword password --backend-name userRoot --set index-type:extensible --set index-extensible-matching-rule:1.3.6.1.4.1.26027.1.4.5 --set index-extensible-matching-rule:1.3.6.1.4.1.26027.1.4.6 --index-name lastLoginTime --trustAll --no-prompt

Indexes Virtual list view (browsing) In the OpenDJ Control Panel, select Manage Indexes > New VLV Index..., and then set up your VLV index using the New VLV Index window. After you finish configuring your index and click OK, the Control Panel prompts you to make the additional changes necessary to complete the VLV index configuration, and then to build the index. You can also create the equivalent index configuration using the dsconfig command. $ dsconfig create-local-db-vlv-index --port 4444 --hostname opendj.example.com --bindDn "cn=Directory Manager" --bindPassword password --backend-name userRoot --index-name people-by-last-name --set base-dn:ou=People,dc=example,dc=com --set filter:"(|(givenName=*)(sn=*))" --set scope:single-level --set sort-order:"+sn +givenName" --trustAll --no-prompt When referring to a virtual list view (VLV) index after creation, you must add vlv. as a prefix. In other words, if you named the VLV index people-by-last-name, you refer to it as vlv.people-by-last-name when rebuilding indexes, changing index properties such as the index entry limit, or verifying indexes.

Indexes Rebuilding After you change an index configuration, or when you find that an index is corrupt, you can rebuild the index. When you rebuild indexes, you specify the base DN of the data to index, and either the list of indexes to rebuild or --rebuildAll. You can rebuild indexes while the server is offline, or while the server is online. If you rebuild the index while the server is online, then you must schedule the rebuild process as a task. The following example rebuilds the cn index immediately with the server online. $ rebuild-index --port 4444 --hostname opendj.example.com --bindDN "cn=Directory Manager" --bindPassword password --baseDN dc=example,dc=com --index cn --start 0 Rebuild Index task 20110607171639867 scheduled to start Jun 7, 2011 5:16:39 PM The following example rebuilds degraded indexes immediately with the server online. $ rebuild-index --port 4444 --hostname opendj.example.com --bindDN "cn=Directory Manager" --bindPassword password --baseDN dc=example,dc=com --rebuildDegraded ... [31/Jan/2012:16:43:25 +0100] severity="NOTICE" msgCount=7 msgID=8847510 message="Due to changes in the configuration, index dc_example_dc_com_description is currently operating in a degraded state and must be rebuilt before it can be used" [31/Jan/2012:16:43:25 +0100] severity="NOTICE" msgCount=8 msgID=8847591 message="Rebuild of all degraded indexes started with 160 total entries to process" ... [31/Jan/2012:16:43:25 +0100] severity="NOTICE" msgCount=10 msgID=8847493 message="Rebuild complete. Processed 160 entries in 0 seconds (average rate 1860.5/sec)" ... Rebuild Index task 20120131164324838 has been successfully completed When you add a new attribute as described in Updating Directory Schema, and then create indexes for the new attribute, the new indexes appear as degraded, even though the attribute has not yet been used, and so indexes are sure to be empty, rather than degraded. In this special case, you can safely use the rebuild-index command --clearDegradedState option to avoid having to scan the entire directory backend to rebuild the new, unused index. This is shown in the following example, where an index has just been created for newUnusedAttribute. $ dbtest list-index-status --backendID userRoot --baseDN dc=example,dc=com | grep newUnusedAttribute newUnusedAttribute.equality Index ...newUnusedAttribute.equality false... newUnusedAttribute.presence Index ...newUnusedAttribute.presence false... newUnusedAttribute.substring Index ...newUnusedAttribute.substring false... $ rebuild-index --port 4444 --hostname opendj.example.com --bindDN "cn=Directory Manager" --bindPassword password --baseDN dc=example,dc=com --clearDegradedState --index newUnusedAttribute --start 0 Rebuild Index task 20130211175925012 scheduled to start Feb 11, 2013 5:59:25 PM CET $ dbtest list-index-status --backendID userRoot --baseDN dc=example,dc=com | grep newUnusedAttribute newUnusedAttribute.equality Index ...newUnusedAttribute.equality true... newUnusedAttribute.presence Index ...newUnusedAttribute.presence true... newUnusedAttribute.substring Index ...newUnusedAttribute.substring true... If the newly indexed attribute has already been used, rebuild indexes instead.

Indexes Entry limits As the number of entries in your directory grows, it can make sense not to maintain indexes for particular values. For example, every entry in the directory has the value top for the objectClass attribute, so maintaining a list of entries that match the filter (objectClass=top) is not a reasonable use of resources. In a very, very large directory, the same can be true for (givenName=John) and (sn=Smith). In an index, each index key points to a list of entries that are candidates to match. For the objectClass index key that corresponds to =top, the list of entries can include every entry in the directory. OpenDJ directory server therefore defines an index entry limit. When the number of entries that an index key points to exceeds the index entry limit, OpenDJ stops maintaining the list of entries for that index key. The default index entry limit value is 4000. 4000 is usually plenty large for all index keys, except for objectClass indexes. If you have clients performing searches with filters such as (objectClass=person), you might suggest that they adjust the search to be more specific, such as (&(mail=username@maildomain.net)(objectClass=person)), so that the server can use an index, in this case equality for mail, to limit the number of candidate entries to check for matches. You can change the index entry limit on a per index basis. The following example changes the index entry limit for the objectClass index, and then rebuilds the index for the configuration change to take effect. $ dsconfig set-local-db-index-prop --port 4444 --hostname opendj.example.com --bindDN "cn=Directory Manager" --bindPassword password --backend-name userRoot --index-name objectClass --set index-entry-limit:5000 --trustAll --no-prompt $ rebuild-index --port 4444 --hostname opendj.example.com --bindDN "cn=Directory Manager" --bindPassword password --baseDN dc=example,dc=com --index objectclass --start 0 Rebuild Index task 20110607160349596 scheduled to start Jun 7, 2011 4:03:49 PM Alternatively, you can configure the index entry limit for all indexes stored in a backend by using the dsconfig set-backend-prop command with the --backend-name backendName --set index-entry-limit:limitValue options.

Indexes Verifying You can verify that indexes correspond to current directory data, and that indexes do not contain errors using the verify-index command. The following example verifies the cn (common name) index for completeness and for errors. $ verify-index --baseDN dc=example,dc=com --index cn --clean --countErrors [07/Jun/2011:16:06:50 +0200] category=BACKEND severity=INFORMATION msgID=9437595 msg=Local DB backend userRoot does not specify the number of lock tables: defaulting to 97 [07/Jun/2011:16:06:50 +0200] category=BACKEND severity=INFORMATION msgID=9437594 msg=Local DB backend userRoot does not specify the number of cleaner threads: defaulting to 24 threads [07/Jun/2011:16:06:51 +0200] category=JEB severity=NOTICE msgID=8847461 msg=Checked 1316 records and found 0 error(s) in 0 seconds (average rate 2506.7/sec) [07/Jun/2011:16:06:51 +0200] category=JEB severity=INFORMATION msgID=8388710 msg=Number of records referencing more than one entry: 315 [07/Jun/2011:16:06:51 +0200] category=JEB severity=INFORMATION msgID=8388711 msg=Number of records that exceed the entry limit: 0 [07/Jun/2011:16:06:51 +0200] category=JEB severity=INFORMATION msgID=8388712 msg=Average number of entries referenced is 1.58/record [07/Jun/2011:16:06:51 +0200] category=JEB severity=INFORMATION msgID=8388713 msg=Maximum number of entries referenced by any record is 32 Ignore the messages regarding lock tables and cleaner threads. The important information is whether any errors are found in the indexes.

Indexes Debugging searches When searching, you can improve performance by making sure your search is indexed as you expect. One way of checking is to request the debugsearchindex attribute in your results. $ ldapsearch --port 1389 --baseDN dc=example,dc=com "(uid=bjensen)" debugsearchindex dn: cn=debugsearch debugsearchindex: filter=(uid=bjensen)[INDEX:uid.equality][COUNT:1] final=[COUNT:1] When you request the debugsearchindex attribute, instead of performing the search, OpenDJ returns debug information indicating how it would process the search operation. In the example above you notice OpenDJ hits the equality index for uid right away. A less exact search requires more work from OpenDJ. In the following example OpenDJ would have to return 160 entries. $ ldapsearch --port 1389 --baseDN dc=example,dc=com "(uid=*)" debugsearchindex dn: cn=debugsearch debugsearchindex: filter=(uid=*)[NOT-INDEXED] scope=wholeSubtree[COUNT:160] final=[COUNT:160] By default OpenDJ rejects unindexed searches when the number of candidate entries goes beyond the search or look-though limit.

Indexes Default settings When you first install OpenDJ directory server and import your data from LDIF, the following indexes are configured. Index Approx. Equality Ordering Presence Substring Entry Limit aci - - - Yes - 4000 cn - Yes - - Yes 4000 dn2id Non-configurable internal index ds-sync-conflict - Yes - - - 4000 ds-sync-hist - - Yes - - 4000 entryUUID - Yes - - - 4000 givenName - Yes - - Yes 4000 id2children Non-configurable internal index id2subtree Non-configurable internal index mail - Yes - - Yes 4000 member - Yes - - - 4000 objectClass - Yes - - - 4000 sn - Yes - - Yes 4000 telephone­Number - Yes - - Yes 4000 uid - Yes - - - 4000 unique­Member - Yes - - - 4000 When you create a JE backend using the dsconfig command, OpenDJ creates the aci presence, ds-sync-conflict equality, ds-sync-hist ordering, entryUUID equality, and objectClass equality indexes automatically.