mirror of https://github.com/OpenIdentityPlatform/OpenDJ.git

ludovicp
30.19.2010 990faf76eb34cb0f342d8a110101105d538a23dd
refs
author ludovicp <ludovicp@localhost>
Friday, July 30, 2010 14:19 +0200
committer ludovicp <ludovicp@localhost>
Friday, July 30, 2010 14:19 +0200
commit990faf76eb34cb0f342d8a110101105d538a23dd
tree bd3b56121c648d7b817d6042c972e0709faebdf5 tree | zip | gz
parent 0d5c134ec980613db96372da5a1025658fe767eb view | diff
import-ldif: Change second phase import strategy in order to better handle large LDIF files with low memory.
In first phase write buffer positions to index files instead of storing in memory and suffering OOME due to O(N) memory growth. In second phase, read buffer positions from index files and fall-back to batch import of indexes when the number of buffers for an index would cause OOME if they were all opened at once. Also, improve second phase progress statistics to report batch count, kb remaining/rate, and fix several race conditions in the statistics.
3 files modified
721 ■■■■■ changed files
opendj-sdk/opends/src/messages/messages/jeb.properties 10 ●●●●● diff | view | raw | blame | history
opendj-sdk/opends/src/server/org/opends/server/backends/jeb/importLDIF/Importer.java 676 ●●●●● diff | view | raw | blame | history
opendj-sdk/opends/src/server/org/opends/server/backends/jeb/importLDIF/IndexInputBuffer.java 35 ●●●●● diff | view | raw | blame | history