Return-Path: X-Original-To: apmail-directory-dev-archive@www.apache.org Delivered-To: apmail-directory-dev-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 50E6011CEE for ; Sun, 20 Jul 2014 05:28:28 +0000 (UTC) Received: (qmail 43460 invoked by uid 500); 20 Jul 2014 05:28:27 -0000 Delivered-To: apmail-directory-dev-archive@directory.apache.org Received: (qmail 43408 invoked by uid 500); 20 Jul 2014 05:28:27 -0000 Mailing-List: contact dev-help@directory.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: "Apache Directory Developers List" Delivered-To: mailing list dev@directory.apache.org Received: (qmail 43396 invoked by uid 99); 20 Jul 2014 05:28:27 -0000 Received: from minotaur.apache.org (HELO minotaur.apache.org) (140.211.11.9) by apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 Jul 2014 05:28:27 +0000 Received: from localhost (HELO mail-wi0-f178.google.com) (127.0.0.1) (smtp-auth username kayyagari, mechanism plain) by minotaur.apache.org (qpsmtpd/0.29) with ESMTP; Sun, 20 Jul 2014 05:28:26 +0000 Received: by mail-wi0-f178.google.com with SMTP id hi2so2524378wib.17 for ; Sat, 19 Jul 2014 22:28:25 -0700 (PDT) MIME-Version: 1.0 X-Received: by 10.180.74.11 with SMTP id p11mr45073142wiv.68.1405834105185; Sat, 19 Jul 2014 22:28:25 -0700 (PDT) Received: by 10.216.121.9 with HTTP; Sat, 19 Jul 2014 22:28:25 -0700 (PDT) In-Reply-To: <53CA15A9.10302@gmail.com> References: <53CA15A9.10302@gmail.com> Date: Sun, 20 Jul 2014 10:58:25 +0530 Message-ID: Subject: Re: Mavibot bulkloader From: Kiran Ayyagari To: Apache Directory Developers List Content-Type: multipart/alternative; boundary=f46d043891c373eedd04fe9942c7 --f46d043891c373eedd04fe9942c7 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: quoted-printable On Sat, Jul 19, 2014 at 12:22 PM, Emmanuel L=C3=A9charny wrote: > Hi guys, > > I have a bit of time this week-end, I'm reviewing the bulk loader code > and will try to imrpove it. > > So far, we need to add a few imrpovements and features, AFAICS : > > - unlimited number of entries. Currently, we are memory bound, as we > load in memory all the DN we read from the file. A merged sort could > help here (still to think more about this pb) > I already have this piece implemented a while ago, can send the code if needed > - we need to inject the config in the bulkloader in order to inject the > user indexes we want to load. This is currently not done (although we do > have the list of indexes to process) > - speed up - if we can. > > If you have any other idea or suggestion, that would be very welcome ! > --=20 Kiran Ayyagari http://keydap.com --f46d043891c373eedd04fe9942c7 Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable

On Sat, Jul 19, 2014 at 12:22 PM, Emmanuel L=C3=A9charny <elech= arny@gmail.com> wrote:

Hi guys,

I have a bit of time this week-end, I'm reviewing the bulk loader code<= br> and will try to imrpove it.

So far, we need to add a few imrpovements and features, AFAICS :

- unlimited number of entries. Currently, we are memory bound, as we
load in memory all the DN we read from the file. A merged sort could
help here (still to think more about this pb)

I alread= y have this piece implemented a while ago, can send the code if needed
=

- we need to inject the config in the bulkloader in order to inject the
user indexes we want to load. This is currently not done (although we do have the list of indexes to process)
- speed up - if we can.

If you have any other idea or suggestion, that would be very welcome !

--
Kiran Ayyagari
http://keydap.com

--f46d043891c373eedd04fe9942c7--