Indexes OpenDJ provides several indexing schemes to speed up searches. When a client requests a directory search operation, the client sends the server a filter expression such as (&(uid=*jensen*)(l=Stavanger)). The server then uses applicable indexes to find entries with attribute values likely to match the search. If no indexes are applicable, then the server potentially has to go through all entries to look for candidate matches. Looking through all entries is resource-intensive for large directories. For this reason, the unindexed-search privilege, allowing users to request searches for which no applicable index exists, is reserved for the directory root user by default. Rather than granting the unindexed-search privilege to more users and client applications, you configure indexes to correspond to the searches that clients need to perform. See for details. This chapter first describes index types, and demonstrates how to index attribute values. This chapter also lists the default indexing configuration for OpenDJ directory server.

OpenDJ provides several different index types, each corresponding to a different type of search.

Indexes Approximate An approximate index is used to match values that "sound like" those provided in the filter. An approximate index on cn allows clients to find people even when they misspell names as in the following example. $ ldapsearch --port 1389 --baseDN dc=example,dc=com "(cn~=Babs Jansen)" cn dn: uid=bjensen,ou=People,dc=example,dc=com cn: Barbara Jensen cn: Babs Jensen

Indexes Equality An equality index is used to match values that correspond exactly (though generally without case sensitivity) to the value provided in the search filter. An equality index requires clients to match values without wildcards or misspellings. $ ldapsearch --port 1389 --baseDN dc=example,dc=com "(uid=bjensen)" mail dn: uid=bjensen,ou=People,dc=example,dc=com mail: bjensen@example.com

Indexes Ordering An ordering index is used to match values for a filter that specifies a range. The ds-sync-hist has an ordering index by default because searches on that attribute often seek entries with changes more recent than the last time a search was performed. The following example shows a search that specifies ranges. $ ldapsearch --port 1389 --baseDN dc=example,dc=com \ "(&(uidNumber>=1120)(roomNumber>=4500))" uid dn: uid=charvey,ou=People,dc=example,dc=com uid: charvey dn: uid=eward,ou=People,dc=example,dc=com uid: eward dn: uid=mvaughan,ou=People,dc=example,dc=com uid: mvaughan dn: uid=pchassin,ou=People,dc=example,dc=com uid: pchassin

Indexes Presence A presence index is used to match the fact that an attribute is present on the entry, regardless of the value. The aci attribute is indexed for presence by default to allow quick retrieval of entries with ACIs. $ ldapsearch --port 1389 --baseDN dc=example,dc=com "(aci=*)" - dn: dc=example,dc=com dn: ou=People,dc=example,dc=com

Indexes Substring A substring index is used to match values specified with wildcards in the filter. Substring indexes can be expensive to maintain, especially for large attribute values. $ ldapsearch --port 1389 --baseDN dc=example,dc=com "(cn=Barb*)" cn dn: uid=bfrancis,ou=People,dc=example,dc=com cn: Barbara Francis dn: uid=bhal2,ou=People,dc=example,dc=com cn: Barbara Hall dn: uid=bjablons,ou=People,dc=example,dc=com cn: Barbara Jablonski dn: uid=bjensen,ou=People,dc=example,dc=com cn: Barbara Jensen cn: Babs Jensen dn: uid=bmaddox,ou=People,dc=example,dc=com cn: Barbara Maddox

Indexes Virtual list view (browsing) A VLV or browsing index are designed to help the server respond to client applications that need virtual list view results, for example to browse through a long list in a GUI. They also help the server respond to clients that request server-side sorting of the search results. VLV indexes correspond to particular searches. Configure your VLV indexes using the Control Panel, and copy the command-line equivalent from the Details pane for the operation, if necessary.

Indexes Extensible matching rule In some cases you need an index for a matching rule other than those described above. For example, OpenDJ supports generalized time based matching so applications can search for all times later than, or earlier than a specified time.

Indexes Debugging searches OpenDJ search performance depends on indexes. As mentioned above, unindexed searches are so resource intensive that by default OpenDJ refuses to perform unindexed searches. This is because, in order to find candidate matches for an unindexed search, OpenDJ has to scan the entire directory database. Most searches should therefore use indexes. A simple way of checking the indexes that match a search is to request the debugsearchindex attribute in your results. $ ldapsearch \ --port 1389 \ --baseDN dc=example,dc=com \ --bindDN "cn=Directory Manager" \ --bindPassword password \ "(uid=user.1000)" \ debugsearchindex dn: cn=debugsearch debugsearchindex: filter=(uid=user.1000)[INDEX:uid.equality][COUNT:1] final=[COU NT:1] When you request the debugsearchindex attribute, instead of performing the search, OpenDJ returns debug information indicating how it would process the search operation. In the example above you notice OpenDJ hits the equality index for uid right away. A less exact search requires more work from OpenDJ. In the following example OpenDJ would have to return over 10,000 entries. $ ldapsearch \ --port 1389 \ --baseDN dc=example,dc=com \ --bindDN "cn=Directory Manager" \ --bindPassword password \ "(uid=*)" \ debugsearchindex dn: cn=debugsearch debugsearchindex: filter=(uid=*)[NOT-INDEXED] scope=wholeSubtree[LIMIT-EXCEEDED: 10002] final=[NOT-INDEXED] By default OpenDJ rejects unindexed searches when the number of candidate entries goes beyond the search or look-though limit. $ ldapsearch --port 1389 --baseDN dc=example,dc=com "(uid=*)" SEARCH operation failed Result Code: 50 (Insufficient Access Rights) Additional Information: You do not have sufficient privileges to perform an unindexed search When an unindexed search is performed, it shows up in the access log with the unindexed label. ...SEARCH RES ... result=50 message="You do not have sufficient privileges to perform an unindexed search" nentries=0 unindexed etime=1 If directory users tell you their client applications are getting this error, then you can work with them either to help them make their search filter specific enough to use existing indexes, or to index attributes they need indexed in order to perform their searches. For example, if a directory client application is having trouble performing a search with a filters such as (objectClass=person), you can suggest that they adjust the search to be more specific, such as (&(mail=username@maildomain.net)(objectClass=person)), so that the server can use an index, in this case equality for mail, to limit the number of candidate entries to check for matches. You can view and edit what is indexed through OpenDJ Control Panel, Indexes > Manage Indexes. Alternatively you can manage indexes using the command-line tools demonstrated in . If an index already exists, but you suspect it is not working properly, see , too. If you do need to allow some applications to perform unindexed searches, because they need to retrieve very large numbers of entries for example, then you can assign them the unindexed-search privilege. See Configuring Privileges for details. A successful unindexed search also shows up in the access log with the label unindexed, usually with a large etime as well. ...SEARCH RES conn=11 op=1 msgID=2 result=0 nentries=10000 unindexed etime=1129 There is a trade off between the cost of maintaining an index and the value the index has in speeding up searches. Although monitoring index use is not something to leave active in production due to the additional cost and memory needed to maintain the statistics, in a test environment you can activate index analysis using the dsconfig set-backend-prop command. $ dsconfig \ set-backend-prop \ --port 4444 \ --hostname opendj.example.com \ --bindDN "cn=Directory Manager" \ --bindPassword password \ --backend-name userRoot \ --set index-filter-analyzer-enabled:true \ --no-prompt \ --trustAll The command causes OpenDJ to analyze filters used and keep the results in memory, so that you can read them through the cn=monitor interface. $ ldapsearch \ --port 1389 \ --baseDN "cn=userRoot Database Environment,cn=monitor" \ --bindDN "cn=Directory Manager" \ --bindPassword password \ "(objectclass=*)" \ filter-use dn: cn=userRoot Database Environment,cn=monitor filter-use: (mail=aa*@maildomain.net) hits:1 maxmatches:0 message: filter-use: (objectClass=*) hits:1 maxmatches:-1 message:presence index type is disabled for the objectClass attribute filter-use: (uid=user.1000) hits:2 maxmatches:1 message: filter-use: (uid=user.1001) hits:1 maxmatches:1 message: filter-use: (cn=aa*) hits:1 maxmatches:10 message: filter-use: (cn=b*) hits:1 maxmatches:834 message: The filter-use values consist of the filter, followed by hits being the number of times the filter was used, followed by maxmatches being the number of matches found for the filter, followed by a message. You can turn off index analysis with the dsconfig set-backend-prop command as well. $ dsconfig \ set-backend-prop \ --port 4444 \ --hostname opendj.example.com \ --bindDN "cn=Directory Manager" \ --bindPassword password \ --backend-name userRoot \ --set index-filter-analyzer-enabled:false \ --no-prompt \ --trustAll

Indexes Configuring You modify index configurations by using the dsconfig command. The subcommands to use depend on the backend type, as shown in the examples that follow. The configuration changes then take effect after you rebuild the index according to the new configuration, using the rebuild-index command. The dsconfig --help-database command lists subcommands for creating, reading, updating, and deleting index configuration. Indexes are per directory backend rather than per suffix. To maintain separate indexes for different suffixes on the same directory server, put the suffixes in different backends.

You can configure standard indexes from the Control Panel, and also on the command line using the dsconfig command. After you finish configuring the index, you must rebuild the index for the changes to take effect. The following example creates a new substring index for the description attribute in a backend of type local-db. $ dsconfig \ create-local-db-index \ --port 4444 \ --hostname opendj.example.com \ --bindDN "cn=Directory Manager" \ --bindPassword password \ --backend-name userRoot \ --index-name description \ --set index-type:substring \ --trustAll \ --no-prompt The following example creates a new equality index for the cn (common name) attribute in a backend of type persistit named myData. $ dsconfig \ create-backend-index \ --port 4444 \ --hostname opendj.example.com \ --bindDN "cn=Directory Manager" \ --bindPassword password \ --backend-name myData \ --index-name cn \ --set index-type:equality \ --trustAll \ --no-prompt Indexes Approximate The following example configures an approximate index for the cn (common name) attribute in a backend of type local-db. $ dsconfig \ set-local-db-index-prop \ --port 4444 \ --hostname opendj.example.com \ --bindDN "cn=Directory Manager" \ --bindPassword password \ --backend-name userRoot \ --index-name cn \ --set index-type:approximate \ --trustAll \ --no-prompt The following example configures an approximate index for the cn (common name) attribute in a backend of type persistit named myData. $ dsconfig \ set-backend-index-prop \ --port 4444 \ --hostname opendj.example.com \ --bindDN "cn=Directory Manager" \ --bindPassword password \ --backend-name myData \ --index-name cn \ --set index-type:approximate \ --trustAll \ --no-prompt Indexes Extensible matching rule The OpenDJ Control Panel New Index window does not help you set up extensible matching rule indexes. Use the dsconfig command instead. The following example configures an extensible matching rule index for "later than" and "earlier than" generalized time matching on a lastLoginTime attribute in a backend of type local-db. $ dsconfig \ create-local-db-index \ --port 4444 \ --hostname opendj.example.com \ --bindDN "cn=Directory Manager" \ --bindPassword password \ --backend-name userRoot \ --set index-type:extensible \ --set index-extensible-matching-rule:1.3.6.1.4.1.26027.1.4.5 \ --set index-extensible-matching-rule:1.3.6.1.4.1.26027.1.4.6 \ --index-name lastLoginTime \ --trustAll \ --no-prompt The following example configures an extensible matching rule index for "later than" and "earlier than" generalized time matching on a lastLoginTime attribute in a backend of type persistit named myData. $ dsconfig \ create-backend-index \ --port 4444 \ --hostname opendj.example.com \ --bindDN "cn=Directory Manager" \ --bindPassword password \ --backend-name myData \ --set index-type:extensible \ --set index-extensible-matching-rule:1.3.6.1.4.1.26027.1.4.5 \ --set index-extensible-matching-rule:1.3.6.1.4.1.26027.1.4.6 \ --index-name lastLoginTime \ --trustAll \ --no-prompt

Indexes Virtual list view (browsing) In the OpenDJ Control Panel, select Manage Indexes > New VLV Index, and then set up your VLV index using the New VLV Index window. New VLV Index window The New VLV Index window helps you to configure a browsing index. After you finish configuring your index and click OK, the Control Panel prompts you to make the additional changes necessary to complete the VLV index configuration, and then to build the index. You can also create the equivalent index configuration by using the dsconfig command. The following example shows how to create the VLV index for a backend of type local-db. $ dsconfig \ create-local-db-vlv-index \ --port 4444 \ --hostname opendj.example.com \ --bindDn "cn=Directory Manager" \ --bindPassword password \ --backend-name userRoot \ --index-name people-by-last-name \ --set base-dn:ou=People,dc=example,dc=com \ --set filter:"(|(givenName=*)(sn=*))" \ --set scope:single-level \ --set sort-order:"+sn +givenName" \ --trustAll \ --no-prompt When referring to a virtual list view (VLV) index after creation, you must add vlv. as a prefix. In other words, if you named the VLV index people-by-last-name, you refer to it as vlv.people-by-last-name when rebuilding indexes, changing index properties such as the index entry limit, or verifying indexes. The following example shows how to create the VLV index for a backend of type persistit named myData serving dc=example,dc=net. $ dsconfig \ create-backend-vlv-index \ --port 4444 \ --hostname opendj.example.com \ --bindDn "cn=Directory Manager" \ --bindPassword password \ --backend-name myData \ --index-name people-by-last-name \ --set base-dn:ou=People,dc=example,dc=net \ --set filter:"(|(givenName=*)(sn=*))" \ --set scope:single-level \ --set sort-order:"+sn +givenName" \ --trustAll \ --no-prompt

Indexes Rebuilding After you change an index configuration, or when you find that an index is corrupt, you can rebuild the index. When you rebuild indexes, you specify the base DN of the data to index, and either the list of indexes to rebuild or --rebuildAll. You can rebuild indexes while the server is offline, or while the server is online. If you rebuild the index while the server is online, then you must schedule the rebuild process as a task. The following example rebuilds the cn index immediately with the server online. $ rebuild-index \ --port 4444 \ --hostname opendj.example.com \ --bindDN "cn=Directory Manager" \ --bindPassword password \ --baseDN dc=example,dc=com \ --index cn \ --start 0 \ --trustAll Rebuild Index task 20150219181540575 scheduled to start Feb 19, 2015 6:15:40 The following example rebuilds degraded indexes immediately with the server online. $ rebuild-index \ --port 4444 \ --hostname opendj.example.com \ --bindDN "cn=Directory Manager" \ --bindPassword password \ --baseDN dc=example,dc=com \ --rebuildDegraded ... [31/Jan/2012:16:43:25 +0100] severity="NOTICE" msgCount=7 msgID=8847510 message="Due to changes in the configuration, index dc_example_dc_com_description is currently operating in a degraded state and must be rebuilt before it can be used" [31/Jan/2012:16:43:25 +0100] severity="NOTICE" msgCount=8 msgID=8847591 message="Rebuild of all degraded indexes started with 160 total entries to process" ... [31/Jan/2012:16:43:25 +0100] severity="NOTICE" msgCount=10 msgID=8847493 message="Rebuild complete. Processed 160 entries in 0 seconds (average rate 1860.5/sec)" ... Rebuild Index task 20120131164324838 has been successfully completed When you add a new attribute as described in Updating Directory Schema, and then create indexes for the new attribute, the new indexes appear as degraded, even though the attribute has not yet been used, and so indexes are sure to be empty, rather than degraded. In this special case, you can safely use the rebuild-index command --clearDegradedState option to avoid having to scan the entire directory backend to rebuild the new, unused index. This is shown in the following example, where an index has just been created for newUnusedAttribute. Start by testing the index status by using the dbtest command. The final column show in the output is the Index Valid column, false before the rebuild, true after. The dbtest list-index-status command can take a long time to complete, as it reads all indexes for all backends. $ dbtest \ list-index-status \ --backendID userRoot \ --baseDN dc=example,dc=com \ | grep newUnusedAttribute newUnusedAttribute.equality Index ...newUnusedAttribute.equality false... newUnusedAttribute.presence Index ...newUnusedAttribute.presence false... newUnusedAttribute.substring Index ...newUnusedAttribute.substring false... $ rebuild-index \ --port 4444 \ --hostname opendj.example.com \ --bindDN "cn=Directory Manager" \ --bindPassword password \ --baseDN dc=example,dc=com \ --clearDegradedState \ --index newUnusedAttribute \ --start 0 Rebuild Index task 20130211175925012 scheduled to start Feb 11, 2013 5:59:25 PM CET $ dbtest \ list-index-status \ --backendID userRoot \ --baseDN dc=example,dc=com \ | grep newUnusedAttribute newUnusedAttribute.equality Index ...newUnusedAttribute.equality true... newUnusedAttribute.presence Index ...newUnusedAttribute.presence true... newUnusedAttribute.substring Index ...newUnusedAttribute.substring true... If the newly indexed attribute has already been used, rebuild indexes instead.

Indexes Entry limits As the number of entries in your directory grows, it can make sense not to maintain indexes for particular values. For example, every entry in the directory has the value top for the objectClass attribute, so maintaining a list of entries that match the filter (objectClass=top) is not a reasonable use of resources. In a very, very large directory, the same can be true for (givenName=John) and (sn=Smith). In an index, each index key points to a list of entries that are candidates to match. For the objectClass index key that corresponds to =top, the list of entries can include every entry in the directory. OpenDJ directory server therefore defines an index entry limit. When the number of entries that an index key points to exceeds the index entry limit, OpenDJ stops maintaining the list of entries for that index key. The default index entry limit value is 4000. 4000 is intended to be large enough for most index keys, though it prevents OpenDJ from maintaining indexes at any cost. You can use the dbtest command to evaluate how well attributes are indexed, and consider whether to change the index entry limit. Non-zero values in the "Undefined" column indicate the number of index keys that have reached the limit and are no longer maintained. The "Undefined keys" are then listed below. $ dbtest list-index-status --backendID userRoot --baseDN dc=example,dc=com Index Name Index Type JE Database Name Index Valid Record Count Undefined 95% 90% 85% --------------------------------------------------------------------------------------------------------------------------------------- id2children Index dc_example_dc_com_id2children true 2 1 0 0 0 id2subtree Index dc_example_dc_com_id2subtree true 2 2 0 0 0 uid.equality Index dc_example_dc_com_uid.equality true 10000 0 0 0 0 aci.presence Index dc_example_dc_com_aci.presence true 0 0 0 0 0 ds-sync-conflict.equality Index dc_example_dc_com_ds-sync-conflict.equality true 0 0 0 0 0 givenName.equality Index dc_example_dc_com_givenName.equality true 8605 0 0 0 0 givenName.substring Index dc_example_dc_com_givenName.substring true 19629 0 0 0 0 objectClass.equality Index dc_example_dc_com_objectClass.equality true 6 4 0 0 0 member.equality Index dc_example_dc_com_member.equality true 0 0 0 0 0 uniqueMember.equality Index dc_example_dc_com_uniqueMember.equality true 0 0 0 0 0 cn.equality Index dc_example_dc_com_cn.equality true 10000 0 0 0 0 cn.substring Index dc_example_dc_com_cn.substring true 86040 0 0 0 0 sn.equality Index dc_example_dc_com_sn.equality true 10000 0 0 0 0 sn.substring Index dc_example_dc_com_sn.substring true 32217 0 0 0 0 telephoneNumber.equality Index dc_example_dc_com_telephoneNumber.equality true 10000 0 0 0 0 telephoneNumber.substring Index dc_example_dc_com_telephoneNumber.substring true 73235 0 0 0 0 ds-sync-hist.ordering Index dc_example_dc_com_ds-sync-hist.ordering true 0 0 0 0 0 mail.equality Index dc_example_dc_com_mail.equality true 10000 0 0 0 0 mail.substring Index dc_example_dc_com_mail.substring true 31235 15 0 0 0 entryUUID.equality Index dc_example_dc_com_entryUUID.equality true 10002 0 0 0 0 Total: 20 Index: objectClass.equality Undefined keys: [inetorgperson] [organizationalperson] [person] [top] Index: id2children Undefined keys: [2] Index: mail.substring Undefined keys: [.net] [@maild] [aildom] [ain.ne] [domain] [et] [ildoma] [in.net] [ldomai] [maildo] [main.n] [n.net] [net] [omain.] [t] Index: id2subtree Undefined keys: [1] [2] In this case (for a directory with only about 10,000 entries) the list of undefined keys is perfectly reasonable. Every user entry has the object classes listed, and every user entry has a mail address ending in @maildomain.net, so those values are not specific enough to be used in search filters. The id2children and id2subtree are for OpenDJ's internal use. For an explanation of the output of the dbtest list-index-status command, see dbtest(1). If you do find the limit is too low for a certain key, you can change the index entry limit on a per index basis. The following example changes the index entry limit for the objectClass index, and then rebuilds the index for the configuration change to take effect. The example is contrived, but the steps are the same for any other index. Changing the index entry limit significantly can result in serious performance degradation. Be prepared to test performance thoroughly before you roll out an index entry limit change in production. The following example uses the dsconfig set-local-db-index-prop command, and works with a backend of type local-db. For other indexed backend types, use the dsconfig set-backend-index-prop command. $ dsconfig \ set-local-db-index-prop \ --port 4444 \ --hostname opendj.example.com \ --bindDN "cn=Directory Manager" \ --bindPassword password \ --backend-name userRoot \ --index-name objectClass \ --set index-entry-limit:5000 \ --trustAll \ --no-prompt $ rebuild-index \ --port 4444 \ --hostname opendj.example.com \ --bindDN "cn=Directory Manager" \ --bindPassword password \ --baseDN dc=example,dc=com \ --index objectclass \ --start 0 Rebuild Index task 20110607160349596 scheduled to start Jun 7, 2011 4:03:49 PM Alternatively, you can configure the index entry limit for all indexes stored in a backend by using the dsconfig set-backend-prop command with the --backend-name backendName --set index-entry-limit:limitValue options.

Indexes Verifying You can verify that indexes correspond to current directory data, and that indexes do not contain errors by using the verify-index command. The following example verifies the cn (common name) index for completeness and for errors. $ verify-index \ --baseDN dc=example,dc=com \ --index cn \ --clean \ --countErrors [07/Jun/2011:16:06:50 +0200] category=BACKEND severity=INFORMATION msgID=9437595 msg=Local DB backend userRoot does not specify the number of lock tables: defaulting to 97 [07/Jun/2011:16:06:50 +0200] category=BACKEND severity=INFORMATION msgID=9437594 msg=Local DB backend userRoot does not specify the number of cleaner threads: defaulting to 24 threads [07/Jun/2011:16:06:51 +0200] category=JEB severity=NOTICE msgID=8847461 msg=Checked 1316 records and found 0 error(s) in 0 seconds (average rate 2506.7/sec) [07/Jun/2011:16:06:51 +0200] category=JEB severity=INFORMATION msgID=8388710 msg=Number of records referencing more than one entry: 315 [07/Jun/2011:16:06:51 +0200] category=JEB severity=INFORMATION msgID=8388711 msg=Number of records that exceed the entry limit: 0 [07/Jun/2011:16:06:51 +0200] category=JEB severity=INFORMATION msgID=8388712 msg=Average number of entries referenced is 1.58/record [07/Jun/2011:16:06:51 +0200] category=JEB severity=INFORMATION msgID=8388713 msg=Maximum number of entries referenced by any record is 32 Ignore the messages regarding lock tables and cleaner threads. The important information is whether any errors are found in the indexes.

Indexes Default settings When you first install OpenDJ directory server and import your data from LDIF, the following indexes are configured. Index Approx. Equality Ordering Presence Substring Entry Limit aci - - - Yes - 4000 cn - Yes - - Yes 4000 dn2id Non-configurable internal index ds-sync-conflict - Yes - - - 4000 ds-sync-hist - - Yes - - 4000 entryUUID - Yes - - - 4000 givenName - Yes - - Yes 4000 id2children Non-configurable internal index id2subtree Non-configurable internal index mail - Yes - - Yes 4000 member - Yes - - - 4000 objectClass - Yes - - - 4000 sn - Yes - - Yes 4000 telephone­Number - Yes - - Yes 4000 uid - Yes - - - 4000 unique­Member - Yes - - - 4000