From e4076b2991c9604907f6a2f5ba2d526d0072adf6 Mon Sep 17 00:00:00 2001 From: ludovicp <ludovicp@localhost> Date: Fri, 30 Jul 2010 12:19:32 +0000 Subject: [PATCH] import-ldif: Change second phase import strategy in order to better handle large LDIF files with low memory. In first phase write buffer positions to index files instead of storing in memory and suffering OOME due to O(N) memory growth. In second phase, read buffer positions from index files and fall-back to batch import of indexes when the number of buffers for an index would cause OOME if they were all opened at once. Also, improve second phase progress statistics to report batch count, kb remaining/rate, and fix several race conditions in the statistics. --- opends/src/messages/messages/jeb.properties | 10 ++++++---- 1 files changed, 6 insertions(+), 4 deletions(-) diff --git a/opends/src/messages/messages/jeb.properties b/opends/src/messages/messages/jeb.properties index 6ef34be..8badbd1 100644 --- a/opends/src/messages/messages/jeb.properties +++ b/opends/src/messages/messages/jeb.properties @@ -278,8 +278,8 @@ the import process can start SEVERE_ERR_JEB_IMPORT_THREAD_EXCEPTION_153=An error occurred in import thread \ %s: %s. The thread can not continue -NOTICE_JEB_IMPORT_LDIF_PHASE_TWO_REPORT_154=Index %s: bytes left = %d, \ -key processed rate = %.1f/sec +NOTICE_JEB_IMPORT_LDIF_PHASE_TWO_REPORT_154=Index %s %d%% complete: \ + remaining = %d kb, rate = %d kb/s; batch %d/%d SEVERE_ERR_JEB_IMPORT_CREATE_TMPDIR_ERROR_155=Unable to create the temporary \ directory %s SEVERE_ERR_JEB_INVALID_LOGGING_LEVEL_156=The database logging level string \ @@ -399,7 +399,7 @@ SEVERE_ERR_JEB_IMPORT_UNCAUGHT_EXCEPTION_210=The following error was received \ by the uncaught exception handler: %s NOTICE_JEB_IMPORT_LDIF_INDEX_STARTED_211=Index %s phase two started processing \ -%d buffers +%d buffers in %d batches SEVERE_ERR_JEB_IMPORT_LDIF_REBUILD_INDEX_TASK_ERR_212=The following \ error was received while processing the rebuild index task: %s NOTICE_JEB_IMPORT_ADJUST_THREAD_COUNT_213=Insufficient memory to allocate \ @@ -410,4 +410,6 @@ the LDIF stream SEVERE_ERR_JEB_IMPORT_BUFFER_IO_ERROR_216=I/O error occurred while reading \ the index scratch file %s in the temporary directory - +MILD_WARN_IMPORT_LDIF_LACK_MEM_PHASE_TWO_217=Insufficient free memory (%d bytes) to \ + perform import phase 2 in a single batch. Some indexes will be imported using \ + several batches which may result in reduced performance -- Gitblit v1.10.0