mirror of https://github.com/OpenIdentityPlatform/OpenDJ.git

ludovicp
30.19.2010 e4076b2991c9604907f6a2f5ba2d526d0072adf6
refs
author ludovicp <ludovicp@localhost>
Friday, July 30, 2010 14:19 +0200
committer ludovicp <ludovicp@localhost>
Friday, July 30, 2010 14:19 +0200
commite4076b2991c9604907f6a2f5ba2d526d0072adf6
tree 50f561c7dcbcba85974233b2fed74f98700be6dd tree | zip | gz
parent 1b5fac65c3243a723d2307323d234550301f5987 view | diff
import-ldif: Change second phase import strategy in order to better handle large LDIF files with low memory.
In first phase write buffer positions to index files instead of storing in memory and suffering OOME due to O(N) memory growth. In second phase, read buffer positions from index files and fall-back to batch import of indexes when the number of buffers for an index would cause OOME if they were all opened at once. Also, improve second phase progress statistics to report batch count, kb remaining/rate, and fix several race conditions in the statistics.
3 files modified
721 ■■■■■ changed files
opends/src/messages/messages/jeb.properties 10 ●●●●● diff | view | raw | blame | history
opends/src/server/org/opends/server/backends/jeb/importLDIF/Importer.java 676 ●●●●● diff | view | raw | blame | history
opends/src/server/org/opends/server/backends/jeb/importLDIF/IndexInputBuffer.java 35 ●●●●● diff | view | raw | blame | history