Return-Path: X-Original-To: apmail-directory-api-archive@minotaur.apache.org Delivered-To: apmail-directory-api-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id 68E331070B for ; Wed, 16 Oct 2013 15:19:17 +0000 (UTC) Received: (qmail 55298 invoked by uid 500); 16 Oct 2013 15:19:16 -0000 Delivered-To: apmail-directory-api-archive@directory.apache.org Received: (qmail 55240 invoked by uid 500); 16 Oct 2013 15:19:07 -0000 Mailing-List: contact api-help@directory.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: api@directory.apache.org Delivered-To: mailing list api@directory.apache.org Received: (qmail 55224 invoked by uid 99); 16 Oct 2013 15:19:06 -0000 Received: from athena.apache.org (HELO athena.apache.org) (140.211.11.136) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Oct 2013 15:19:06 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS X-Spam-Check-By: apache.org Received-SPF: pass (athena.apache.org: domain of jbdundas@gmail.com designates 209.85.219.41 as permitted sender) Received: from [209.85.219.41] (HELO mail-oa0-f41.google.com) (209.85.219.41) by apache.org (qpsmtpd/0.29) with ESMTP; Wed, 16 Oct 2013 15:19:02 +0000 Received: by mail-oa0-f41.google.com with SMTP id o9so637302oag.0 for ; Wed, 16 Oct 2013 08:18:41 -0700 (PDT) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=gmail.com; s=20120113; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :cc:content-type; bh=q2GlBwWA3N0BTfKLdIh6NV0L450k22vNgd42YG+noSQ=; b=iOytNphv9BqAqbzYOsCpE1eEeRV8E9gh8nJU7/s6dzmJCkEFjHrO6KaRbzRqL3qfkn k8OHxswWzLPw4L1S8vjOfmgIkD4sjAuGAM2htaNo94XMW8baXv1J38rJBP/erRIuD2JM mAeEE2k2tpWo7JoNkKUDin8CcGN8cNLecjPs2HGcTSttzCARVCIchGaYPI0jmfd3Sasu Uk8qouYE51oQCJ9p5nU7uEhcCEy7tFX/N4UKuGm8uMnqif5ZlKVWLTfQnaYh5Sa/KrdY AnOKJ7vGPODKMXqWUfexuTdQFRY240lflLYunIqGJ1BiaCV1nmhHHuSiNWMxQX1gM/AI VHrg== MIME-Version: 1.0 X-Received: by 10.182.214.98 with SMTP id nz2mr5631864obc.37.1381936721034; Wed, 16 Oct 2013 08:18:41 -0700 (PDT) Received: by 10.76.116.163 with HTTP; Wed, 16 Oct 2013 08:18:40 -0700 (PDT) In-Reply-To: References:

Date: Wed, 16 Oct 2013 11:18:40 -0400 Message-ID: Subject: Re: Need Help with the LDAP API for large records From: jitesh dundas To: api@directory.apache.org Cc: users@directory.apache.org Content-Type: multipart/alternative; boundary=e89a8ff1cefe5c17ed04e8dd374b X-Virus-Checked: Checked by ClamAV on apache.org --e89a8ff1cefe5c17ed04e8dd374b Content-Type: text/plain; charset=ISO-8859-1 HI Kiran, Thank you for your reply. You are correct in the sense that we close the connection after every 1000 entries that are completed in the for loop and then open the connection again. *if (i%1000 == 0 || webphonePersons.size() == i){* client.closeConnection(connection); System.out.println("Completed => "+ i); Thread.sleep(4000); connection = client.getConnection(); }//end if condition We are also putting Thread.sleep() to wait and give some time for the connection to start/stop. Also, we have added the same sleep condition of 400 milliseconds after every entry that is added( to prevent overlaps). However, based on your response, I am making the change to remove this logic and keep one connection only throughout the loop. Hopefully it will work. Thanks & Regards, Jitesh Dundas Phone - 732-357-6292 On Wed, Oct 16, 2013 at 10:56 AM, Kiran Ayyagari wrote: > On Wed, Oct 16, 2013 at 8:14 PM, jitesh dundas wrote: > > > Hi Kiran, > > > > Thank you for your reply. My apologies for not being able to send the > full > > java file( As you can see that I am already doing that):- > > > > public Main(String host, int port, String userName, String password) { > > this.host = host; > > this.port = port; > > this.userName = userName; > > this.password = password; > > } > > > > public LdapConnection getConnection() throws LdapException, > IOException { > > LdapConnection connection = new LdapNetworkConnection(host, port); > > connection.bind("uid=" + userName + ",ou=system", password); > > return connection; > > } > > > > Also, I see from the updated logs now that it traverses 3000 records and > > then it starts giving these error messages. Any ideas? > > > > Maybe it is the way in which I am making the entries. > > > > if entry creation is wrong you will get a different error (instead of > bind > error) > are you opening a new connection for adding each entry? it appears so to me > after seeing the method public static void addEntryIntoSonarTestLdap(Entry > labLdapEntry) > > > Thanks & Regards, > > Jitesh Dundas > > > > Phone - 732-357-6292 > > > > > > > > > > > > On Wed, Oct 16, 2013 at 10:26 AM, Kiran Ayyagari > >wrote: > > > > > On Wed, Oct 16, 2013 at 7:51 PM, jitesh dundas > > wrote: > > > > > > > Hi All, > > > > > > > > I am a new user who is implementing Apache LDAP on one of our > > company's > > > > servers. The records in the LDAP ( entries) come from a text file ( > > LDAP > > > > dump) having 600000 records in it. > > > > > > > > I have written a utility to insert it buyt it keeps failing with > these > > > > errors:- > > > > > > > > [10:16:25] ERROR > > > > [org.apache.directory.ldap.client.api.LdapNetworkConnection] - Bind > > > failed > > > > : timeout occurred > > > > [10:16:25] ERROR > > > > [org.apache.directory.ldap.client.api.LdapNetworkConnection] - The > > > response > > > > queue has been emptied, no response was found. > > > > org.apache.directory.api.ldap.model.exception.LdapException: TimeOut > > > > occurred > > > > at > > > > > > > > > > > > > > org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1089) > > > > at > > > > > > > > > > > > > > org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:121) > > > > at > > > > > > > > > > > > > > org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:106) > > > > at Main.getConnection(Main.java:38) > > > > at Main.main(Main.java:130) > > > > at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method) > > > > at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown Source) > > > > at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown > Source) > > > > at java.lang.reflect.Method.invoke(Unknown Source) > > > > at > > > > > > > > > > > > > > org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58) > > > > > > > > > > > > Here is the code to add it:- > > > > > > > > public static void addEntryIntoSonarTestLdap(Entry labLdapEntry) > > throws > > > > LdapException, IOException { > > > > //Sonar Ldap Connection Details > > > > Main client = new Main("altv0018.aldc.att.com", 10389, > "admin", > > > > "secret"); > > > > > > > replace "admin" with "uid=admin,ou=system" and try > > > > > > > > > > > List webphonePersons = > > > WebphoneFileReader.readFile(); > > > > for(int i=6000; i > > > LdapConnection connection = client.getConnection(); > > > > connection.setTimeOut(1000000); > > > > try { > > > > > > > > > > > > //create the new entries to add. > > > > Entry sonarLdapEntry = new > > > > DefaultEntry(labLdapEntry.getDn().toString().trim());//dn of the > entry > > > goes > > > > here. > > > > sonarLdapEntry.add("objectClass","top"); > > > > sonarLdapEntry.add("objectClass","inetOrgPerson"); > > > > sonarLdapEntry.add("objectClass","person"); > > > > > > sonarLdapEntry.add("objectClass","organizationalPerson"); > > > > sonarLdapEntry.add("cn", > > > webphonePersons.get(i).getUID()); > > > > > > > > sonarLdapEntry.add("sn",webphonePersons.get(i).getFirstName() + " " + > > > > webphonePersons.get(i).getLastName()); > > > > > > > sonarLdapEntry.add("uid",webphonePersons.get(i).getUID()); > > > > > > > > > sonarLdapEntry.add("userPassword","{AES256}MT3QA5xD8c1ZNy9EJSgW/Q=="); > > > > connection.add(sonarLdapEntry); > > > > LOGGER.info("Sonar Test LDAP@Added User > > > > sonarLdapEntry=>"+sonarLdapEntry.toString()); > > > > } catch (Exception e) { > > > > // TODO Auto-generated catch block > > > > //e.printStackTrace(); > > > > System.out.println("UID FAILED => "+ > > > > webphonePersons.get(i).getAttUID()); > > > > } > > > > //closing connections after every 1000 records > added > > > and > > > > then opened again. > > > > //to prevent memory issues. > > > > if (i%1000 == 0){ > > > > client.closeConnection(connection); > > > > connection = client.getConnection(); > > > > } > > > > } > > > > > > > > } > > > > > > > > Any ideas on how to resolve it? We have the Java memory options set > to > > > > -Xms1024m -Xmx1024m > > > > > > > > Thanks & Regards, > > > > Jitesh Dundas > > > > > > > > Phone - 732-357-6292 > > > > > > > > > > > > > > > > -- > > > Kiran Ayyagari > > > http://keydap.com > > > > > > > > > -- > Kiran Ayyagari > http://keydap.com > --e89a8ff1cefe5c17ed04e8dd374b--