directory-api mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From jitesh dundas <jbdun...@gmail.com>
Subject Re: Need Help with the LDAP API for large records
Date Wed, 16 Oct 2013 15:33:36 GMT
Hi Kiran,

Thanks again for the reply. Appreciate it.

We are using the latest ApacheDS 2.0.0 version. I modified the method to
look like this:-

  LdapConnection connection = client.getConnection();
      connection.setTimeOut(1000000000);

      Date date = new Date();
      String dateStr = "Started="+date.getDate() + "-" + date.getMonth() +
"-" + date.getYear() ;
      System.out.println("START@"+ dateStr );

      List<WebphonePerson> webphonePersons = WebphoneFileReader.readFile();
      System.out.println("Found Users" + webphonePersons.size());
      for(int i=18000; i<webphonePersons.size() ; i++){

          try {
                  //create the new entries to add.
                Entry sonarLdapEntry = new DefaultEntry("cn="+
webphonePersons.get(i).getAttUID() +",ou=people,O=ATT,C=US");//dn of the
entry goes here.
                sonarLdapEntry.add("objectClass","top");
                sonarLdapEntry.add("objectClass","inetOrgPerson");
                sonarLdapEntry.add("objectClass","person");
                sonarLdapEntry.add("objectClass","organizationalPerson");
                sonarLdapEntry.add("cn",webphonePersons.get(i).getUID());

sonarLdapEntry.add("sn",webphonePersons.get(i).getFirstName() + " " +
webphonePersons.get(i).getLastName());
                sonarLdapEntry.add("uid",webphonePersons.get(i).getUID());

sonarLdapEntry.add("userPassword","{AES256}MT3QA5xD8c1ZNy9EJSgW/Q==");
                connection.add(sonarLdapEntry);
                Thread.sleep(200);
            } catch (Exception e) {
                //e.printStackTrace();
                System.out.println("ATTUID FAILED => "+
webphonePersons.get(i).getAttUID());
            }
              if (i%1000 == 0 || webphonePersons.size() == i){
            //client.closeConnection(connection);
            System.out.println("Completed => "+ i);
            Thread.sleep(4000);
            //connection = client.getConnection();
              }//end if condition
        }//end for loop

Can you please tell me what is wrong here? I would love to make it better.


Thanks & Regards,
Jitesh Dundas

Phone - 732-357-6292





On Wed, Oct 16, 2013 at 11:26 AM, Kiran Ayyagari <kayyagari@apache.org>wrote:

> On Wed, Oct 16, 2013 at 8:48 PM, jitesh dundas <jbdundas@gmail.com> wrote:
>
> > HI Kiran,
> >
> > Thank you for your reply. You are correct in the sense that we close the
> > connection after every 1000 entries that are completed in the for loop
> and
> > then open the connection again.
> >
> > *if (i%1000 == 0 || webphonePersons.size() == i){*
> >             client.closeConnection(connection);
> >             System.out.println("Completed => "+ i);
> >             Thread.sleep(4000);
> >             connection = client.getConnection();
> > }//end if condition
> >
> > We are also putting Thread.sleep() to wait and give some time for the
> > connection to start/stop. Also, we have added the same sleep condition of
> > 400 milliseconds after every entry that is added( to prevent overlaps).
> >
> > However, based on your response, I am making the change to remove this
> > logic and keep one connection only throughout the loop. Hopefully it will
> > work.
> >
> > not that your method is wrong, but one connection is enough here
> otoh, which version of ApacheDS you are using?
>
> >
> > Thanks & Regards,
> > Jitesh Dundas
> >
> > Phone - 732-357-6292
> >
> >
> >
> >
> >
> > On Wed, Oct 16, 2013 at 10:56 AM, Kiran Ayyagari <kayyagari@apache.org
> > >wrote:
> >
> > > On Wed, Oct 16, 2013 at 8:14 PM, jitesh dundas <jbdundas@gmail.com>
> > wrote:
> > >
> > > > Hi Kiran,
> > > >
> > > > Thank you for your reply. My apologies for not being able to send the
> > > full
> > > > java file( As you can see that I am already doing that):-
> > > >
> > > >  public Main(String host, int port, String userName, String
> password) {
> > > >     this.host = host;
> > > >     this.port = port;
> > > >     this.userName = userName;
> > > >     this.password = password;
> > > >   }
> > > >
> > > >   public LdapConnection getConnection() throws LdapException,
> > > IOException {
> > > >     LdapConnection connection = new LdapNetworkConnection(host,
> port);
> > > >     connection.bind("uid=" + userName + ",ou=system", password);
> > > >     return connection;
> > > >   }
> > > >
> > > > Also, I see from the updated logs now that it traverses 3000 records
> > and
> > > > then it starts giving these error messages. Any ideas?
> > > >
> > > > Maybe it is the way in which I am making the entries.
> > > >
> > > > if entry creation is wrong you will get a different error (instead of
> > > bind
> > > error)
> > > are you opening a new connection for adding each entry? it appears so
> to
> > me
> > > after seeing the method public static void
> > addEntryIntoSonarTestLdap(Entry
> > > labLdapEntry)
> > >
> > > > Thanks & Regards,
> > > > Jitesh Dundas
> > > >
> > > > Phone - 732-357-6292
> > > >
> > > >
> > > >
> > > >
> > > >
> > > > On Wed, Oct 16, 2013 at 10:26 AM, Kiran Ayyagari <
> kayyagari@apache.org
> > > > >wrote:
> > > >
> > > > > On Wed, Oct 16, 2013 at 7:51 PM, jitesh dundas <jbdundas@gmail.com
> >
> > > > wrote:
> > > > >
> > > > > > Hi All,
> > > > > >
> > > > > >  I am a new user who is implementing Apache LDAP on one of our
> > > > company's
> > > > > > servers. The records in the LDAP ( entries) come from a text
> file (
> > > > LDAP
> > > > > > dump) having 600000 records in it.
> > > > > >
> > > > > > I have written a utility to insert it buyt it keeps failing
with
> > > these
> > > > > > errors:-
> > > > > >
> > > > > > [10:16:25] ERROR
> > > > > > [org.apache.directory.ldap.client.api.LdapNetworkConnection]
-
> Bind
> > > > > failed
> > > > > > : timeout occurred
> > > > > > [10:16:25] ERROR
> > > > > > [org.apache.directory.ldap.client.api.LdapNetworkConnection]
-
> The
> > > > > response
> > > > > > queue has been emptied, no response was found.
> > > > > > org.apache.directory.api.ldap.model.exception.LdapException:
> > TimeOut
> > > > > > occurred
> > > > > >     at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.directory.ldap.client.api.LdapNetworkConnection.bind(LdapNetworkConnection.java:1089)
> > > > > >     at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:121)
> > > > > >     at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.apache.directory.ldap.client.api.AbstractLdapConnection.bind(AbstractLdapConnection.java:106)
> > > > > >     at Main.getConnection(Main.java:38)
> > > > > >     at Main.main(Main.java:130)
> > > > > >     at sun.reflect.NativeMethodAccessorImpl.invoke0(Native
> Method)
> > > > > >     at sun.reflect.NativeMethodAccessorImpl.invoke(Unknown
> Source)
> > > > > >     at sun.reflect.DelegatingMethodAccessorImpl.invoke(Unknown
> > > Source)
> > > > > >     at java.lang.reflect.Method.invoke(Unknown Source)
> > > > > >     at
> > > > > >
> > > > > >
> > > > >
> > > >
> > >
> >
> org.eclipse.jdt.internal.jarinjarloader.JarRsrcLoader.main(JarRsrcLoader.java:58)
> > > > > >
> > > > > >
> > > > > > Here is the code to add it:-
> > > > > >
> > > > > >   public static void addEntryIntoSonarTestLdap(Entry
> labLdapEntry)
> > > > throws
> > > > > > LdapException, IOException {
> > > > > >       //Sonar Ldap Connection Details
> > > > > >       Main client = new Main("altv0018.aldc.att.com", 10389,
> > > "admin",
> > > > > > "secret");
> > > > > >
> > > > > replace "admin" with "uid=admin,ou=system" and try
> > > > >
> > > > > >
> > > > > >       List<WebphonePerson> webphonePersons =
> > > > > WebphoneFileReader.readFile();
> > > > > >       for(int i=6000; i<webphonePersons.size() ; i++){
> > > > > >           LdapConnection connection = client.getConnection();
> > > > > >           connection.setTimeOut(1000000);
> > > > > >           try {
> > > > > >
> > > > > >
> > > > > >                   //create the new entries to add.
> > > > > >                 Entry sonarLdapEntry = new
> > > > > > DefaultEntry(labLdapEntry.getDn().toString().trim());//dn of
the
> > > entry
> > > > > goes
> > > > > > here.
> > > > > >                 sonarLdapEntry.add("objectClass","top");
> > > > > >
> sonarLdapEntry.add("objectClass","inetOrgPerson");
> > > > > >                 sonarLdapEntry.add("objectClass","person");
> > > > > >
> > > > sonarLdapEntry.add("objectClass","organizationalPerson");
> > > > > >                 sonarLdapEntry.add("cn",
> > > > > webphonePersons.get(i).getUID());
> > > > > >
> > > > > > sonarLdapEntry.add("sn",webphonePersons.get(i).getFirstName()
+ "
> > " +
> > > > > > webphonePersons.get(i).getLastName());
> > > > > >
> > > > > sonarLdapEntry.add("uid",webphonePersons.get(i).getUID());
> > > > > >
> > > > > >
> > > sonarLdapEntry.add("userPassword","{AES256}MT3QA5xD8c1ZNy9EJSgW/Q==");
> > > > > >                 connection.add(sonarLdapEntry);
> > > > > >                 LOGGER.info("Sonar Test LDAP@Added User
> > > > > > sonarLdapEntry=>"+sonarLdapEntry.toString());
> > > > > >             } catch (Exception e) {
> > > > > >                 // TODO Auto-generated catch block
> > > > > >                 //e.printStackTrace();
> > > > > >                 System.out.println("UID FAILED => "+
> > > > > > webphonePersons.get(i).getAttUID());
> > > > > >             }
> > > > > >                   //closing connections after every 1000 records
> > > added
> > > > > and
> > > > > > then opened again.
> > > > > >                   //to prevent memory issues.
> > > > > >                   if (i%1000 == 0){
> > > > > >                 client.closeConnection(connection);
> > > > > >                 connection = client.getConnection();
> > > > > >                   }
> > > > > >             }
> > > > > >
> > > > > >     }
> > > > > >
> > > > > > Any ideas on how to resolve it? We have the Java memory options
> set
> > > to
> > > > > > -Xms1024m -Xmx1024m
> > > > > >
> > > > > > Thanks & Regards,
> > > > > > Jitesh Dundas
> > > > > >
> > > > > > Phone - 732-357-6292
> > > > > >
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Kiran Ayyagari
> > > > > http://keydap.com
> > > > >
> > > >
> > >
> > >
> > >
> > > --
> > > Kiran Ayyagari
> > > http://keydap.com
> > >
> >
>
>
>
> --
> Kiran Ayyagari
> http://keydap.com
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message