manifoldcf-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From lalit jangra <lalit.j.jan...@gmail.com>
Subject Re: Apache ManifoldCF job stuck up
Date Thu, 10 Jul 2014 09:59:46 GMT
Thanks Karl,

As you suggested, i followed below steps.

1. Stopped HSQL DB, agents & web applicaitons (by shutting down tomcat).
2. Updated properties.xml with below details for PostgreSQL & did leave all
setting as it is.


  <property name="org.apache.manifoldcf.databaseimplementationclass"
value="org.apache.manifoldcf.core.database.DBInterfacePostgreSQL"/>

  <property name="org.apache.manifoldcf.dbsuperusername" value="postgres"/>

  <property name="org.apache.manifoldcf.dbsuperuserpassword" value="admin"/>

  <property name="org.apache.manifoldcf.database.name" value="mcfdkw" />



  <property name="org.apache.manifoldcf.hsqldbdatabaseprotocol"
value="hsql"/>

  <property name="org.apache.manifoldcf.hsqldbdatabaseserver"
value="localhost"/>

  <property name="org.apache.manifoldcf.hsqldbdatabaseinstance"
value="xdb"/>

  <property name="org.apache.manifoldcf.database.maxhandles" value="500"/>

  <property name="org.apache.manifoldcf.crawler.threads" value="500"/>


3. Initialized DB with ./initialize.sh script, works fine and i can see
successful registration messages.
4. Started agents using ./start-agents.sh
5. Started Tomcat, created connections & jobs.Started crawl job.
6.  Job not taking off and still can see below error in logs.


ERROR 2014-07-10 10:51:55,690 (Agents thread) - Exception tossed: Service
'A' of type 'AGENT_org.apache.manifoldcf.crawler.system.CrawlerAgent' is
already active

org.apache.manifoldcf.core.interfaces.ManifoldCFException: Service 'A' of
type 'AGENT_org.apache.manifoldcf.crawler.system.CrawlerAgent' is already
active

Regards.









On Thu, Jul 10, 2014 at 10:12 AM, Karl Wright <daddywri@gmail.com> wrote:

> bq. So the only difference i could observe is DB which is HSQL vs
> Postgres, rest everything is same.
>
> Right, exactly.  Only this time we have some certainty that the
> configuration is otherwise correct for synchronization and processes.
>
> The next step is the following:
>
> - START with the same multiprocess-file-example directory and processes
> you verified yesterday
> - Stop all MCF processes/web applications that are using that
> multiprocess-file-example properties file:
>   - Either stop the Tomcat process, or better yet go to the "admin" tomcat
> application and stop all three MCF web applications
>   - Go to the start-agents.sh window and ^C that process
>   - Go to the start-database.sh window and ^C that process
> - Change the properties.xml file under multiprocess-file-example to point
> to the PostgresDB instance you want it to use.  This instance CANNOT be in
> use by any other MCF cluster or process.
>   - Replace the following THREE properties in properties.xml with your
> PostgreSQL equivalent, as follows:
>
> >>>>>>
>   <property name="org.apache.manifoldcf.databaseimplementationclass"
> value="org.apache.manifoldcf.core.database.DBInterfacePostgreSQL"/>
>   <property name="org.apache.manifoldcf.dbsuperusername"
> value="<something>"/>
>   <property name="org.apache.manifoldcf.dbsuperuserpassword"
> value="<something>"/>
> <<<<<<
>
> You may also choose to set these properties, to avoid collision with other
> instances.  You can use whatever value you like for these, within what
> PostgreSQL allows:
>
> org.apache.manifoldcf.database.name
> org.apache.manifoldcf.database.username
> org.apache.manifoldcf.database.password
>
> - Run the initialize script:
>   - cd multiprocess-file-example ; initialize.sh .  If this works, you
> have the database configured properly.  If it fails, look at the parameters
> you provided and figure out what is wrong before proceeding
>
> - start the ManifoldCF processes and web applications:
>   - cd multiprocess-file-example ; start-agents.sh
>   - go to the tomcat instance, and start it, or start all the web
> applications
>
> - create connections and jobs, and try a crawl.  Let me know how it works.
>
> Karl
>
>
>
> On Thu, Jul 10, 2014 at 12:25 AM, lalit jangra <lalit.j.jangra@gmail.com>
> wrote:
>
>> Thanks a lot Karl,
>>
>> Yes, it works perfectly as you suggested in last mail.
>>
>> So the only difference i could observe is DB which is HSQL vs Postgres,
>> rest everything is same.
>>
>> Please suggest.
>>
>> Regards.
>>
>>
>>
>>
>> On Wed, Jul 9, 2014 at 11:35 PM, Karl Wright <daddywri@gmail.com> wrote:
>>
>>> Hi Lalit,
>>>
>>>
>>> Sorry, I thought my instructions were clear.  For this exercise, I want
>>> you to use ONLY the example under dist/multiprocess-file-example.  DO NOT
>>> use the example under dist/example.
>>>
>>> Here are the instructions again, clarified and fleshed out:
>>>
>>>
>>> - Start with a fresh build of MCF
>>> - Go to the multiprocess-file example and start that in Jetty, with NO
>>> changes:
>>>   - cd multiprocess-file-example
>>>   - start-database.sh
>>>   - initialize.sh
>>>   - start-agents.sh
>>>   - in a separate window: cd multiprocess-file-example ; start-webapps.sh
>>>   - create connections and job, see if it runs
>>>  - If that doesn't work, open a ticket.  If that DOES work, proceed as
>>> follows:
>>> - Shut down the webapps process:
>>>   - go to the window where start-webapps.sh is running, and type ^C
>>>  - Set up a fresh tomcat instance
>>> - Configure tomcat to point to the multiprocess-file example
>>> properties.xml:
>>>   - Modify the tomcat startup scripts to include -D
>>> org.apache.manifoldcf.configfile=<path_to_dist/multiprocess-file-example/properties.xml>
>>>   - Start the new instance of tomcat
>>> - Deploy the wars to that tomcat (except combined war)
>>>   - Use the Tomcat admin tool to deploy mcf-crawler-ui.war,
>>> mcf-authority-service.war, and mcf-api-service.war
>>> - Go to the UI as served by Tomcat.  You should see your connections,
>>> job, etc just as before.  Run the job.  Does it work, yes or no?
>>>
>>>
>>> Thanks,
>>> Karl
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Jul 9, 2014 at 5:58 PM, lalit jangra <lalit.j.jangra@gmail.com>
>>> wrote:
>>>
>>>> Thanks Karl,
>>>>
>>>> Indeed i really appreciate your help and promptness.
>>>>
>>>> I tried the way you suggest and below are results.
>>>>
>>>> 1. Took a fresh MCF, started it using start.jar under dist/example
>>>> using jetty.
>>>> 2. Under multiprocess-file-example, initialized DB & started agent
>>>> using .start-agents.sh.
>>>> 3. Created all connection and jobs. Started job & it worked fine.
>>>> 4. Next i shutdown jetty as started in step1.
>>>> 5. Setup a fresh tomcat and pointed to
>>>> multiprocess-file-example/properties.xml in its setenv.sh using MCF_HOME.
>>>> 6. Started tomcat & deployed all but combined war.
>>>> 7. Accessed mcf-crawler-ui war & accessed job.
>>>> 8. On Job tab i get connection refused error.
>>>> 9. If i start DB using ./start-databse.sh, i get "Error getting
>>>> connection:invalid authorization exception-not found:manifoldcf" message.
>>>>
>>>> Really embarrassing for me as its not working.
>>>>
>>>> Regards.
>>>>
>>>>
>>>> On Wed, Jul 9, 2014 at 6:52 PM, Karl Wright <daddywri@gmail.com> wrote:
>>>>
>>>>> Hi Lalit,
>>>>>
>>>>> Since I'm quite busy at work today, can you do the following:
>>>>>
>>>>> - Start with a fresh build of MCF
>>>>> - Go to the multiprocess-file example and start that in Jetty, with NO
>>>>> changes
>>>>> - If that doesn't work, open a ticket.  If that DOES work, proceed as
>>>>> follows:
>>>>> - Shut down the webapps process in the example
>>>>> - Set up a fresh tomcat instance
>>>>> - Configure tomcat to point to the multiprocess-file example
>>>>> properties.xml.
>>>>> - Deploy the wars to that tomcat (except combined war)
>>>>> - Once again, try your job.  Let me know the result.
>>>>>
>>>>> Thanks,
>>>>> Karl
>>>>>
>>>>>
>>>>>
>>>>> On Wed, Jul 9, 2014 at 1:47 PM, Karl Wright <daddywri@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> Hi lalit,
>>>>>>
>>>>>> I don't see anything wrong with your environment.  I suspect your
>>>>>> configuration is what is wrong.
>>>>>>
>>>>>> Karl
>>>>>>
>>>>>>
>>>>>> On Wed, Jul 9, 2014 at 1:46 PM, lalit jangra <
>>>>>> lalit.j.jangra@gmail.com> wrote:
>>>>>>
>>>>>>> Thanks Karl,
>>>>>>>
>>>>>>> I agree with you. I am using Red Hat 6.4 with Postgres 9.2 & tomcat
>>>>>>> 7.
>>>>>>>
>>>>>>> Can this be an issue ?
>>>>>>>
>>>>>>> Regards.
>>>>>>>
>>>>>>>
>>>>>>> On Wed, Jul 9, 2014 at 6:43 PM, Karl Wright <daddywri@gmail.com>
>>>>>>> wrote:
>>>>>>>
>>>>>>>> Hi Lalit,
>>>>>>>>
>>>>>>>> I tried the zookeeper example here this morning and it worked
>>>>>>>> perfectly.  I am happy to try the file system example too, when I have a
>>>>>>>> moment, but I very much doubt there is any problem with that code
>>>>>>>> specifically.
>>>>>>>>
>>>>>>>> Remember that this is NOT a database issue.  It is a
>>>>>>>> synchronization issue ONLY.  So it does not matter what database instance
>>>>>>>> you point it to.
>>>>>>>>
>>>>>>>> Karl
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> On Wed, Jul 9, 2014 at 1:32 PM, lalit jangra <
>>>>>>>> lalit.j.jangra@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> Thanks Karl,
>>>>>>>>>
>>>>>>>>> I did it as you mentioned but still getting same error. This time
>>>>>>>>> also i setup new DB and tried. Also i could see syncharea folder empty
>>>>>>>>> after lock cleaning.
>>>>>>>>>
>>>>>>>>> Regards.
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Wed, Jul 9, 2014 at 6:21 PM, Karl Wright <daddywri@gmail.com>
>>>>>>>>> wrote:
>>>>>>>>>
>>>>>>>>>> Hi Lalit,
>>>>>>>>>>
>>>>>>>>>> If you were using the multiprocess-file-example before, and you
>>>>>>>>>> did not shut things down cleanly, you should do: ./lock-clean.sh first.
>>>>>>>>>>
>>>>>>>>>> Karl
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>> On Wed, Jul 9, 2014 at 1:10 PM, lalit jangra <
>>>>>>>>>> lalit.j.jangra@gmail.com> wrote:
>>>>>>>>>>
>>>>>>>>>>> Hi Karl,
>>>>>>>>>>>
>>>>>>>>>>> I think its my bad luck today. :(
>>>>>>>>>>>
>>>>>>>>>>> I tried the way you just suggested but still getting same error
>>>>>>>>>>> and job not able to run. What i did.
>>>>>>>>>>>
>>>>>>>>>>> 1. Created a brand new DB schema using ./executecommand.sh
>>>>>>>>>>> org.apache.manifoldcf.core.DBCreate username password
>>>>>>>>>>> 2. Initialized DB schema using ./initialize.sh
>>>>>>>>>>> 3. Start agent process A using ./start-agents.sh
>>>>>>>>>>> 4. Updated tomcat/bin/setenv.sh file to add
>>>>>>>>>>> "-Dorg.apache.manifoldcf.configfile" parameter to point to properties.xml
>>>>>>>>>>> under dist/multiprocess-file-example.
>>>>>>>>>>> 5. Started Tomcat.
>>>>>>>>>>> 6. Deployed all wars except combined war.
>>>>>>>>>>> 7. Started job
>>>>>>>>>>>
>>>>>>>>>>> Job still not able to take off with same error in logs.
>>>>>>>>>>>
>>>>>>>>>>> ERROR 2014-07-08 13:32:19,823 (Agents thread) - Exception
>>>>>>>>>>> tossed:  Service 'A' of type
>>>>>>>>>>>  'AGENT_org.apache.manifoldcf.crawler.system.CrawlerAgent' is already
>>>>>>>>>>> active
>>>>>>>>>>>
>>>>>>>>>>> I think its not my day today.
>>>>>>>>>>>
>>>>>>>>>>> Regards.
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> On Wed, Jul 9, 2014 at 4:02 PM, Karl Wright <daddywri@gmail.com>
>>>>>>>>>>> wrote:
>>>>>>>>>>>
>>>>>>>>>>>> Hi Lalit,
>>>>>>>>>>>>
>>>>>>>>>>>> In theory the combined war will run with external locking, it's
>>>>>>>>>>>> just that nobody ever thought that would be how people would use it, and
>>>>>>>>>>>> therefore it hasn't been tested etc.
>>>>>>>>>>>>
>>>>>>>>>>>> I would suggest that for a development setup, you use the Jetty
>>>>>>>>>>>> simple example.  For a production setup, you should adopt the standard
>>>>>>>>>>>> distributed arrangement as follows:
>>>>>>>>>>>>
>>>>>>>>>>>> - a deployed mcf-crawler-ui war
>>>>>>>>>>>> - a deployed mcf-authority-service war
>>>>>>>>>>>> - an (optional) deployed mcf-api-service war
>>>>>>>>>>>> - one or more agents processes
>>>>>>>>>>>>
>>>>>>>>>>>> This is how it is done in both kinds of multiprocess examples,
>>>>>>>>>>>> and is described thoroughly in the how-to-build-and-deploy document online.
>>>>>>>>>>>>
>>>>>>>>>>>> Thanks,
>>>>>>>>>>>> Karl
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>> On Wed, Jul 9, 2014 at 10:56 AM, lalit jangra <
>>>>>>>>>>>> lalit.j.jangra@gmail.com> wrote:
>>>>>>>>>>>>
>>>>>>>>>>>>> Thanks Karl,
>>>>>>>>>>>>>
>>>>>>>>>>>>> My idea was as its name suggests mcf-combined-war as i was
>>>>>>>>>>>>> trying to take advantage of agent process inside war. My assumption is
>>>>>>>>>>>>> wrong here as i can see here.
>>>>>>>>>>>>>
>>>>>>>>>>>>> In this case, which war should i use - mcf-crawler-ui.war? If
>>>>>>>>>>>>> i use this or any of other wars, then should i start agent process?
>>>>>>>>>>>>>
>>>>>>>>>>>>> Please suggest.
>>>>>>>>>>>>>
>>>>>>>>>>>>> Regards.
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> On Wed, Jul 9, 2014 at 3:48 PM, Karl Wright <
>>>>>>>>>>>>> daddywri@gmail.com> wrote:
>>>>>>>>>>>>>
>>>>>>>>>>>>>> Hi Lalit,
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Probably it is not a database issue, but how locking is
>>>>>>>>>>>>>> configured.  Typically the combined war is run in a non-clustered
>>>>>>>>>>>>>> environment with no external synchronization required (as it is in the
>>>>>>>>>>>>>> example I pointed you at).  We don't currently have an example that uses
>>>>>>>>>>>>>> combined war with external locking -- that would not make much sense.  Can
>>>>>>>>>>>>>> you tell me why you are trying to do it that way?
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>> On Wed, Jul 9, 2014 at 10:46 AM, lalit jangra <
>>>>>>>>>>>>>> lalit.j.jangra@gmail.com> wrote:
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Hi Karl,
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> if i try ./start-combined.sh, the job runs and i could see
>>>>>>>>>>>>>>> it finishing gracefully with updating index in solr.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> I assume there is some DB issue here? Any suggestions?
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> Regards.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> On Wed, Jul 9, 2014 at 3:17 PM, Karl Wright <
>>>>>>>>>>>>>>> daddywri@gmail.com> wrote:
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Hi Lalit,
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Please do the following: In an unmodified dist/example
>>>>>>>>>>>>>>>> directory, run the start-combined script.  Open the UI, create the
>>>>>>>>>>>>>>>> connections, and try to run the job.  Does it work for you?
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>> On Wed, Jul 9, 2014 at 10:09 AM, lalit jangra <
>>>>>>>>>>>>>>>> lalit.j.jangra@gmail.com> wrote:
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Thanks Karl,
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> I have reconfigured everything and this time i am starting
>>>>>>>>>>>>>>>>> tomcat and deploying mcf-combined-war. I created output connection,
>>>>>>>>>>>>>>>>> repository connection & job. Still when i start my job, it does not go
>>>>>>>>>>>>>>>>> ahead and stuck at starting up status.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> Regards.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> On Wed, Jul 9, 2014 at 12:49 PM, Karl Wright <
>>>>>>>>>>>>>>>>> daddywri@gmail.com> wrote:
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Hi Lalit,
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> You do not use combined.war AND an agents process at the
>>>>>>>>>>>>>>>>>> same time.  The combined war INCLUDES the agents process.
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>> On Wed, Jul 9, 2014 at 6:18 AM, lalit jangra <
>>>>>>>>>>>>>>>>>> lalit.j.jangra@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Hi Karl,
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> As per your suggestion, i built a new MCF 1.5.1
>>>>>>>>>>>>>>>>>>> instance, con from source , configured it and pointed to local postgresql
>>>>>>>>>>>>>>>>>>> DB in properties.xml  under /multiprocess-file-example. Then i tool a brand
>>>>>>>>>>>>>>>>>>> new tomcat and configured for MCF in it in setenv.sh for properties.xml
>>>>>>>>>>>>>>>>>>> using "-Dorg.apache.manifoldcf.configfile" argument and deployed
>>>>>>>>>>>>>>>>>>> mcf-combined.war using app manager.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> First i initialized DB using ./initialize.sh command &
>>>>>>>>>>>>>>>>>>> the moment i started agent using ./start-agents.sh command under
>>>>>>>>>>>>>>>>>>> /multiprocess-file-example, i got the error, even when tomcat is not
>>>>>>>>>>>>>>>>>>> started at this time. I created a job which never runs & get stuck.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> ERROR 2014-07-08 13:32:19,823 (Agents thread) -
>>>>>>>>>>>>>>>>>>> Exception tossed:  Service 'A' of type
>>>>>>>>>>>>>>>>>>>  'AGENT_org.apache.manifoldcf.crawler.system.CrawlerAgent' is already
>>>>>>>>>>>>>>>>>>>   active
>>>>>>>>>>>>>>>>>>> org.apache.manifoldcf.core.interfaces.ManifoldCFException:
>>>>>>>>>>>>>>>>>>> Service 'A'  of type
>>>>>>>>>>>>>>>>>>> 'AGENT_org.apache.manifoldcf.crawler.system.CrawlerAgent' is  already active
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> But if i try to run jetty bundle using start.jar under
>>>>>>>>>>>>>>>>>>> example, no such error but it does not fulfil my requirements.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> Regards.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> On Wed, Jul 9, 2014 at 10:03 AM, Karl Wright <
>>>>>>>>>>>>>>>>>>> daddywri@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> So, lalit, if you run the multiprocess example without
>>>>>>>>>>>>>>>>>>>> any changes, do
>>>>>>>>>>>>>>>>>>>> you see this?  I don't.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Karl
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Sent from my Windows Phone
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>>>>>>>> From: lalit jangra
>>>>>>>>>>>>>>>>>>>> Sent: 7/9/2014 12:17 AM
>>>>>>>>>>>>>>>>>>>> To: Karl Wright
>>>>>>>>>>>>>>>>>>>> Cc: user@manifoldcf.apache.org
>>>>>>>>>>>>>>>>>>>> Subject: Re: Apache ManifoldCF job stuck up
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks Karl,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I am currently running only single agent process on
>>>>>>>>>>>>>>>>>>>> single machine
>>>>>>>>>>>>>>>>>>>> without clustering. I have two environments and i could
>>>>>>>>>>>>>>>>>>>> see this issue
>>>>>>>>>>>>>>>>>>>> coming up at both places.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> While trying to start agent, i could see below error.
>>>>>>>>>>>>>>>>>>>> When i am trying
>>>>>>>>>>>>>>>>>>>> to start agent, it says below error and exits. But
>>>>>>>>>>>>>>>>>>>> there is no agent
>>>>>>>>>>>>>>>>>>>> process already running.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [root@server1 multiprocess-file-example]#
>>>>>>>>>>>>>>>>>>>> ./start-agents.sh &
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [1] 5020
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [root@server1 multiprocess-file-example]# Running...
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Configuration file successfully read
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> org.apache.manifoldcf.core.interfaces.ManifoldCFException:
>>>>>>>>>>>>>>>>>>>> Service 'A'
>>>>>>>>>>>>>>>>>>>> of type 'AGENT' is already active
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>>>>>>> org.apache.manifoldcf.core.lockmanager.BaseLockManager.registerServiceBeginServiceActivity(BaseLockManager.java:156)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>>>>>>> org.apache.manifoldcf.core.lockmanager.BaseLockManager.registerServiceBeginServiceActivity(BaseLockManager.java:120)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>>>>>>> org.apache.manifoldcf.core.lockmanager.LockManager.registerServiceBeginServiceActivity(LockManager.java:69)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>>>>>>> org.apache.manifoldcf.agents.AgentRun.doExecute(AgentRun.java:54)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>>>>>>> org.apache.manifoldcf.agents.BaseAgentsInitializationCommand.execute(BaseAgentsInitializationCommand.java:37)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>>>>>>> org.apache.manifoldcf.agents.AgentRun.main(AgentRun.java:93)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> [1]+  Exit 1                  ./start-agents.sh
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> If i am able to successfully start agent using
>>>>>>>>>>>>>>>>>>>> ./start-agents.sh,
>>>>>>>>>>>>>>>>>>>> still it throws same error but i have no other process
>>>>>>>>>>>>>>>>>>>> running.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Regards.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Tue, Jul 8, 2014 at 2:38 PM, Karl Wright <
>>>>>>>>>>>>>>>>>>>> daddywri@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi lalit,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> This occurs when you have more than one agents process
>>>>>>>>>>>>>>>>>>>> with the same
>>>>>>>>>>>>>>>>>>>>  process id using the same shared file system directory
>>>>>>>>>>>>>>>>>>>> / zookeeper
>>>>>>>>>>>>>>>>>>>>  cluster.  There is no other way it can occur.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>  Karl
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Sent from my Windows Phone
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> -----Original Message-----
>>>>>>>>>>>>>>>>>>>>  From: lalit jangra
>>>>>>>>>>>>>>>>>>>>  Sent: 7/8/2014 8:38 AM
>>>>>>>>>>>>>>>>>>>>  To: user@manifoldcf.apache.org
>>>>>>>>>>>>>>>>>>>>  Subject: Re: Apache ManifoldCF job stuck up
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks Karl,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I have tried steps you suggested and it worked on one
>>>>>>>>>>>>>>>>>>>> instance.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> But on another instance i am still not able to resolve
>>>>>>>>>>>>>>>>>>>> this issue.
>>>>>>>>>>>>>>>>>>>>  Along with steps you mentioned, i tried recreating DB
>>>>>>>>>>>>>>>>>>>> instance again,
>>>>>>>>>>>>>>>>>>>>  setting up new MCF instance , cleaning locks and then
>>>>>>>>>>>>>>>>>>>> starting agents
>>>>>>>>>>>>>>>>>>>>  first & then tomcat. But issue still persists.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> If i try to run ./start-agents.sh, i get this error for
>>>>>>>>>>>>>>>>>>>> agent A.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> ERROR 2014-07-08 13:32:19,823 (Agents thread) -
>>>>>>>>>>>>>>>>>>>> Exception tossed:
>>>>>>>>>>>>>>>>>>>>  Service 'A' of type
>>>>>>>>>>>>>>>>>>>>  'AGENT_org.apache.manifoldcf.crawler.system.CrawlerAgent'
>>>>>>>>>>>>>>>>>>>> is already
>>>>>>>>>>>>>>>>>>>>  active
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> org.apache.manifoldcf.core.interfaces.ManifoldCFException:
>>>>>>>>>>>>>>>>>>>> Service 'A'
>>>>>>>>>>>>>>>>>>>>  of type
>>>>>>>>>>>>>>>>>>>> 'AGENT_org.apache.manifoldcf.crawler.system.CrawlerAgent' is
>>>>>>>>>>>>>>>>>>>>  already active
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>>>>>>> org.apache.manifoldcf.core.lockmanager.BaseLockManager.registerServiceBeginServiceActivity(BaseLockManager.java:156)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>>>>>>> org.apache.manifoldcf.core.lockmanager.BaseLockManager.registerServiceBeginServiceActivity(BaseLockManager.java:120)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>>>>>>> org.apache.manifoldcf.core.lockmanager.LockManager.registerServiceBeginServiceActivity(LockManager.java:69)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>>>>>>> org.apache.manifoldcf.agents.system.AgentsDaemon.checkAgents(AgentsDaemon.java:270)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> But if i try to run ./start-agents-2.sh, i see similar
>>>>>>>>>>>>>>>>>>>> error but agent B
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> ERROR 2014-07-08 13:32:19,823 (Agents thread) -
>>>>>>>>>>>>>>>>>>>> Exception tossed:
>>>>>>>>>>>>>>>>>>>>  Service 'B' of type
>>>>>>>>>>>>>>>>>>>>  'AGENT_org.apache.manifoldcf.crawler.system.CrawlerAgent'
>>>>>>>>>>>>>>>>>>>> is already
>>>>>>>>>>>>>>>>>>>>  active
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> org.apache.manifoldcf.core.interfaces.ManifoldCFException:
>>>>>>>>>>>>>>>>>>>> Service 'B'
>>>>>>>>>>>>>>>>>>>>  of type
>>>>>>>>>>>>>>>>>>>> 'AGENT_org.apache.manifoldcf.crawler.system.CrawlerAgent' is
>>>>>>>>>>>>>>>>>>>>  already active
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>>>>>>> org.apache.manifoldcf.core.lockmanager.BaseLockManager.registerServiceBeginServiceActivity(BaseLockManager.java:156)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>>>>>>> org.apache.manifoldcf.core.lockmanager.BaseLockManager.registerServiceBeginServiceActivity(BaseLockManager.java:120)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>>>>>>> org.apache.manifoldcf.core.lockmanager.LockManager.registerServiceBeginServiceActivity(LockManager.java:69)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>>>>>>> org.apache.manifoldcf.agents.system.AgentsDaemon.checkAgents(AgentsDaemon.java:270)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Regards.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Mon, Jul 7, 2014 at 4:55 PM, Karl Wright <
>>>>>>>>>>>>>>>>>>>> daddywri@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi Lalit,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> If you are using file synchronization, you cannot
>>>>>>>>>>>>>>>>>>>> expect MCF to clean
>>>>>>>>>>>>>>>>>>>>  up itself unless you shut it down cleanly.  You should
>>>>>>>>>>>>>>>>>>>> be either using
>>>>>>>>>>>>>>>>>>>>  ^C or plain kill, NEVER kill -9.  kill -9 will leave
>>>>>>>>>>>>>>>>>>>> dangling locks.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> To clean up dangling locks:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> - shut ALL manifoldcf processes and web apps down
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> - run the lock-clean script
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> - start up the processes again
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>  Zookeeper synchronization, by the way, does not have
>>>>>>>>>>>>>>>>>>>> this kind of problem.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Thanks,
>>>>>>>>>>>>>>>>>>>>  Karl
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> On Mon, Jul 7, 2014 at 11:49 AM, lalit jangra <
>>>>>>>>>>>>>>>>>>>> lalit.j.jangra@gmail.com> wrote:
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Hi,
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I configured MCF 1.5.1 to run with Postgresql DB and
>>>>>>>>>>>>>>>>>>>> tomcat 7.
>>>>>>>>>>>>>>>>>>>>  Initially i created all connections, created a an
>>>>>>>>>>>>>>>>>>>> alfresco job and it
>>>>>>>>>>>>>>>>>>>>  all worked fine.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Next for updates, i stopped, tomcat and agent process
>>>>>>>>>>>>>>>>>>>> running. Then i
>>>>>>>>>>>>>>>>>>>>  updated CmisRepositoryConnector.java with my own code
>>>>>>>>>>>>>>>>>>>> and run "ant
>>>>>>>>>>>>>>>>>>>>  build" at root of MCF. It updated all code and jars
>>>>>>>>>>>>>>>>>>>> file.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Also properties.xml is reset under
>>>>>>>>>>>>>>>>>>>> /dist/multiprocess-file/example
>>>>>>>>>>>>>>>>>>>>  which i updated to connect to Postgresql DB and logging
>>>>>>>>>>>>>>>>>>>>  configurations.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I started tomcat then agent process.  Finally i started
>>>>>>>>>>>>>>>>>>>> job to crawl
>>>>>>>>>>>>>>>>>>>>  alfresco but it got stuck and moving on. I checked into
>>>>>>>>>>>>>>>>>>>>  /dist/multiprocess-file/example/logs/manifoldcf.log
>>>>>>>>>>>>>>>>>>>> file and could see
>>>>>>>>>>>>>>>>>>>>  below error.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> ERROR 2014-07-07 16:09:04,936 (Agents thread) -
>>>>>>>>>>>>>>>>>>>> Exception tossed:
>>>>>>>>>>>>>>>>>>>>  Service '' of type
>>>>>>>>>>>>>>>>>>>>  'AGENT_org.apache.manifoldcf.crawler.system.CrawlerAgent'
>>>>>>>>>>>>>>>>>>>> is already
>>>>>>>>>>>>>>>>>>>>  active
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> org.apache.manifoldcf.core.interfaces.ManifoldCFException:
>>>>>>>>>>>>>>>>>>>> Service ''
>>>>>>>>>>>>>>>>>>>>  of type
>>>>>>>>>>>>>>>>>>>> 'AGENT_org.apache.manifoldcf.crawler.system.CrawlerAgent' is
>>>>>>>>>>>>>>>>>>>>  already active
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>>>>>>> org.apache.manifoldcf.core.lockmanager.BaseLockManager.registerServiceBeginServiceActivity(BaseLockManager.java:156)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>>>>>>> org.apache.manifoldcf.core.lockmanager.BaseLockManager.registerServiceBeginServiceActivity(BaseLockManager.java:120)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>>>>>>> org.apache.manifoldcf.core.lockmanager.LockManager.registerServiceBeginServiceActivity(LockManager.java:69)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>>>>>>> org.apache.manifoldcf.agents.system.AgentsDaemon.checkAgents(AgentsDaemon.java:270)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>         at
>>>>>>>>>>>>>>>>>>>> org.apache.manifoldcf.agents.system.AgentsDaemon$AgentsThread.run(AgentsDaemon.java:208)
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> I created another job but that got stuck too.Is DB got
>>>>>>>>>>>>>>>>>>>> corrupt due to rebuild?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Also is this the right way to build MCF (I hope its
>>>>>>>>>>>>>>>>>>>> correct way). Now
>>>>>>>>>>>>>>>>>>>>  what should i do to fix this issue?
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Please help.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>>  Lalit Jangra.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>>  Regards,
>>>>>>>>>>>>>>>>>>>>  Lalit Jangra.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>>> Lalit Jangra.
>>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>>>> Lalit Jangra.
>>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>>>> Lalit Jangra.
>>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>> --
>>>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>>>> Lalit Jangra.
>>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>>
>>>>>>>>>>>>> --
>>>>>>>>>>>>> Regards,
>>>>>>>>>>>>> Lalit Jangra.
>>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>>
>>>>>>>>>>> --
>>>>>>>>>>> Regards,
>>>>>>>>>>> Lalit Jangra.
>>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Regards,
>>>>>>>>> Lalit Jangra.
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Regards,
>>>>>>> Lalit Jangra.
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> Regards,
>>>> Lalit Jangra.
>>>>
>>>
>>>
>>
>>
>> --
>> Regards,
>> Lalit Jangra.
>>
>
>


-- 
Regards,
Lalit Jangra.

Mime
View raw message