directory-api mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jitesh dundas <jbdun...@gmail.com>
Subject Re: Need Help with the LDAP API for large records
Date Wed, 16 Oct 2013 17:35:25 GMT
Hi Kiran,

Well it seems that this is working much better. Removing the extra
connections and giving a sleep condition as shown in my previous email has
not given the error till now.
It is still running but I guess it should be done by tonight.

Will confirm when all is done with you all on this list so that others can
get help later if needed.

Special Thanks to Kiran for his prompt reply. Appreciate it :)


Thanks & Regards,
Jitesh Dundas

Phone - 732-357-6292





On Wed, Oct 16, 2013 at 11:33 AM, jitesh dundas <jbdundas@gmail.com> wrote:

> Hi Kiran,
>
> Thanks again for the reply. Appreciate it.
>
> We are using the latest ApacheDS 2.0.0 version. I modified the method to
> look like this:-
>
>   LdapConnection connection = client.getConnection();
>       connection.setTimeOut(1000000000);
>
>       Date date = new Date();
>       String dateStr = "Started="+date.getDate() + "-" + date.getMonth() +
> "-" + date.getYear() ;
>       System.out.println("START@"+ dateStr );
>
>
>       List<WebphonePerson> webphonePersons = WebphoneFileReader.readFile();
>       System.out.println("Found Users" + webphonePersons.size());
>       for(int i=18000; i<webphonePersons.size() ; i++){
>
>
>           try {
>                   //create the new entries to add.
>                 Entry sonarLdapEntry = new DefaultEntry("cn="+
> webphonePersons.get(i).getAttUID() +",ou=people,O=ATT,C=US");//dn of the
> entry goes here.
>
>                 sonarLdapEntry.add("objectClass","top");
>                 sonarLdapEntry.add("objectClass","inetOrgPerson");
>                 sonarLdapEntry.add("objectClass","person");
>                 sonarLdapEntry.add("objectClass","organizationalPerson");
>                 sonarLdapEntry.add("cn",webphonePersons.get(i).getUID());
>
> sonarLdapEntry.add("sn",webphonePersons.get(i).getFirstName() + " " +
> webphonePersons.get(i).getLastName());
>                 sonarLdapEntry.add("uid",webphonePersons.get(i).getUID());
>
> sonarLdapEntry.add("userPassword","{AES256}MT3QA5xD8c1ZNy9EJSgW/Q==");
>                 connection.add(sonarLdapEntry);
>                 Thread.sleep(200);
>             } catch (Exception e) {
>                 //e.printStackTrace();
>                 System.out.println("ATTUID FAILED => "+
> webphonePersons.get(i).getAttUID());
>
>             }
>               if (i%1000 == 0 || webphonePersons.size() == i){
>             //client.closeConnection(connection);
>
>             System.out.println("Completed => "+ i);
>             Thread.sleep(4000);
>             //connection = client.getConnection();
>               }//end if condition
>         }//end for loop
>
> Can you please tell me what is wrong here? I would love to make it better.
>
>
> Thanks & Regards,
> Jitesh Dundas
>
> Phone - 732-357-6292
>
>
>
>
>
> On Wed, Oct 16, 2013 at 11:26 AM, Kiran Ayyagari <kayyagari@apache.org>wrote:
>
>> On Wed, Oct 16, 2013 at 8:48 PM, jitesh dundas <jbdundas@gmail.com>
>> wrote:
>>
>> > HI Kiran,
>> >
>> > Thank you for your reply. You are correct in the sense that we close the
>> > connection after every 1000 entries that are completed in the for loop
>> and
>> > then open the connection again.
>> >
>> > *if (i%1000 == 0 || webphonePersons.size() == i){*
>> >             client.closeConnection(connection);
>> >             System.out.println("Completed => "+ i);
>> >             Thread.sleep(4000);
>> >             connection = client.getConnection();
>> > }//end if condition
>> >
>> > We are also putting Thread.sleep() to wait and give some time for the
>> > connection to start/stop. Also, we have added the same sleep condition
>> of
>> > 400 milliseconds after every entry that is added( to prevent overlaps).
>> >
>> > However, based on your response, I am making the change to remove this
>> > logic and keep one connection only throughout the loop. Hopefully it
>> will
>> > work.
>> >
>> > not that your method is wrong, but one connection is enough here
>> otoh, which version of ApacheDS you are using?
>>
>> >
>> > Thanks & Regards,
>> > Jitesh Dundas
>> >
>> > Phone - 732-357-6292
>> >
>> >
>> >
>> >
>> >
>> > On Wed, Oct 16, 2013 at 10:56 AM, Kiran Ayyagari <kayyagari@apache.org
>> > >wrote:
>> >
>> > > On Wed, Oct 16, 2013 at 8:14 PM, jitesh dundas <jbdundas@gmail.com>
>> > wrote:
>> > >
>> > > > Hi Kiran,
>> > > >
>> > > > Thank you for your reply. My apologies for not being able to send
>> the
>> > > full
>> > > > java file( As you can see that I am already doing that):-
>> > > >
>> > > >  public Main(String host, int port, String userName, String
>> password) {
>> > > >     this.host = host;
>> > > >     this.port = port;
>> > > >     this.userName = userName;
>> > > >     this.password = password;
>> > > >   }
>> > > >
>> > > >   public LdapConnection getConnection() throws LdapException,
>> > > IOException {
>> > > >     LdapConnection connection = new LdapNetworkConnection(host,
>> port);
>> > > >     connection.bind("uid=" + userName + ",ou=system", password);
>> > > >     return connection;
>> > > >   }
>> > > >
>> > > > Also, I see from the updated logs now that it traverses 3000 records
>> > and
>> > > > then it starts giving these error messages. Any ideas?
>> > > >
>> > > > Maybe it is the way in which I am making the entries.
>> > > >
>> > > > if entry creation is wrong you will get a different error (instead
>> of
>> > > bind
>> > > error)
>> > > are you opening a new connection for adding each entry? it appears so
>> to
>> > me
>> > > after seeing the method public static void
>> > addEntryIntoSonarTestLdap(Entry
>> > > labLdapEntry)
>> > >
>> > > > Thanks & Regards,
>> > > > Jitesh Dundas
>> > > >
>> > > > Phone - 732-357-6292
>> > > >
>> > > >
>> > > >
>> > > >
>> > > >
>> > > > On Wed, Oct 16, 2013 at 10:26 AM, Kiran Ayyagari <
>> kayyagari@apache.org
>> > > > >wrote:
>> > > >
>> > > > > On Wed, Oct 16, 2013 at 7:51 PM, jitesh dundas <
>> jbdundas@gmail.com>
>> > > > wrote:
>> > > > >
>> > > > > > Hi All,
>> > > > > >
>> > > > > >  I am a new user who is implementing Apache LDAP on one
of our
>> > > > company's
>> > > > > > servers. The records in the LDAP ( entries) come from a
text
>> file (
>> > > > LDAP
>> > > > > > dump) having 600000 records in it.
>> > > > > >
>> > > > > > I have written a utility to insert it buyt it keeps failing
with
>> > > these
>> > > > > > errors:-
>> > > > > >
>> > > > > > [10:16:25] ERROR
>> > > > > > [org.apache.directory.ldap.client.api.LdapNetworkConnection]
-
>> Bind
>> > > > > failed
>> > > > > > : timeout occurred
>> > > > > > [10:16:25] ERROR
>> > > > > > [org.apache.directory.ldap.client.api.LdapNetworkConnection]
-
>> The
>> > > > > response
>> > > > > > queue has been emptied, no response was found.
>> > > > > > org.apache.directory.api.ldap.model.exception.LdapException:
>> > TimeOut
>> > > > > > occurred
>> > > > > >     at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1089)
>> > > > > >     at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:121)
>> > > > > >     at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:106)
>> > > > > >     at Main.getConnection(Main.java:38)
>> > > > > >     at Main.main(Main.java:130)
>> > > > > >     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
>> Method)
>> > > > > >     at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown
>> Source)
>> > > > > >     at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
>> > > Source)
>> > > > > >     at java.lang.reflect.Method.invoke(Unknown Source)
>> > > > > >     at
>> > > > > >
>> > > > > >
>> > > > >
>> > > >
>> > >
>> >
>> org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
>> > > > > >
>> > > > > >
>> > > > > > Here is the code to add it:-
>> > > > > >
>> > > > > >   public static void addEntryIntoSonarTestLdap(Entry
>> labLdapEntry)
>> > > > throws
>> > > > > > LdapException, IOException {
>> > > > > >       //Sonar Ldap Connection Details
>> > > > > >       Main client = new Main("altv0018.aldc.att.com", 10389,
>> > > "admin",
>> > > > > > "secret");
>> > > > > >
>> > > > > replace "admin" with "uid=admin,ou=system" and try
>> > > > >
>> > > > > >
>> > > > > >       List<WebphonePerson> webphonePersons =
>> > > > > WebphoneFileReader.readFile();
>> > > > > >       for(int i=6000; i<webphonePersons.size() ; i++){
>> > > > > >           LdapConnection connection = client.getConnection();
>> > > > > >           connection.setTimeOut(1000000);
>> > > > > >           try {
>> > > > > >
>> > > > > >
>> > > > > >                   //create the new entries to add.
>> > > > > >                 Entry sonarLdapEntry = new
>> > > > > > DefaultEntry(labLdapEntry.getDn().toString().trim());//dn
of the
>> > > entry
>> > > > > goes
>> > > > > > here.
>> > > > > >                 sonarLdapEntry.add("objectClass","top");
>> > > > > >
>> sonarLdapEntry.add("objectClass","inetOrgPerson");
>> > > > > >                 sonarLdapEntry.add("objectClass","person");
>> > > > > >
>> > > > sonarLdapEntry.add("objectClass","organizationalPerson");
>> > > > > >                 sonarLdapEntry.add("cn",
>> > > > > webphonePersons.get(i).getUID());
>> > > > > >
>> > > > > > sonarLdapEntry.add("sn",webphonePersons.get(i).getFirstName()
+
>> "
>> > " +
>> > > > > > webphonePersons.get(i).getLastName());
>> > > > > >
>> > > > > sonarLdapEntry.add("uid",webphonePersons.get(i).getUID());
>> > > > > >
>> > > > > >
>> > > sonarLdapEntry.add("userPassword","{AES256}MT3QA5xD8c1ZNy9EJSgW/Q==");
>> > > > > >                 connection.add(sonarLdapEntry);
>> > > > > >                 LOGGER.info("Sonar Test LDAP@Added User
>> > > > > > sonarLdapEntry=>"+sonarLdapEntry.toString());
>> > > > > >             } catch (Exception e) {
>> > > > > >                 // TODO Auto-generated catch block
>> > > > > >                 //e.printStackTrace();
>> > > > > >                 System.out.println("UID FAILED => "+
>> > > > > > webphonePersons.get(i).getAttUID());
>> > > > > >             }
>> > > > > >                   //closing connections after every 1000
records
>> > > added
>> > > > > and
>> > > > > > then opened again.
>> > > > > >                   //to prevent memory issues.
>> > > > > >                   if (i%1000 == 0){
>> > > > > >                 client.closeConnection(connection);
>> > > > > >                 connection = client.getConnection();
>> > > > > >                   }
>> > > > > >             }
>> > > > > >
>> > > > > >     }
>> > > > > >
>> > > > > > Any ideas on how to resolve it? We have the Java memory
options
>> set
>> > > to
>> > > > > > -Xms1024m -Xmx1024m
>> > > > > >
>> > > > > > Thanks & Regards,
>> > > > > > Jitesh Dundas
>> > > > > >
>> > > > > > Phone - 732-357-6292
>> > > > > >
>> > > > >
>> > > > >
>> > > > >
>> > > > > --
>> > > > > Kiran Ayyagari
>> > > > > http://keydap.com
>> > > > >
>> > > >
>> > >
>> > >
>> > >
>> > > --
>> > > Kiran Ayyagari
>> > > http://keydap.com
>> > >
>> >
>>
>>
>>
>> --
>> Kiran Ayyagari
>> http://keydap.com
>>
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message