zookeeper-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Junqueira <fpjunque...@yahoo.com.INVALID>
Subject Re: Getting errors in zookeeper logs
Date Tue, 16 Sep 2014 10:59:56 GMT
Ok, so does it work now?

-Flavio


On Tuesday, September 16, 2014 11:57 AM, lalit jangra <lalit.j.jangra@gmail.com> wrote:
 

>
>
>Thanks Flavio,
>
>I have two agents which produce multiple threads to index content in ManifoldCF.For 'Too
many connections' i updated maxClientCnxns=0 in zoo.cfg in all nodes after which i could not
see this error. Previously it was maxClientCnxns=60.
>
>I have passed connection string i have put below configurations in ManifoldCF proerty
file.
>
>
><property name="org.apache.manifoldcf.lockmanagerclass" value="org.apache.manifoldcf.core.lockmanager.ZooKeeperLockManager"/>
>  <property name="org.apache.manifoldcf.zookeeper.connectstring" value="iwdc1preecma03:2181,iwdc1preecma03:2182,iwdc1preecma03:2183,iwdc2preecma04:2181,iwdc2preecma04:2182"/>
<property name="org.apache.manifoldcf.
>zookeeper.sessiontimeout" value="3000"/>
>Regards.
>
>
>
>On Tue, Sep 16, 2014 at 4:15 PM, Flavio Junqueira <fpjunqueira@yahoo.com.invalid>
wrote:
>
>There are applications level errors when trying to create those znodes. However, the important
one at this point seems to be this one:
>>
>>2014-09-16 10:37:58,073 [myid:5] - WARN  [NIOServerCxn.Factory:
>>0.0.0.0/0.0.0.0:2182:NIOServerCnxnFactory@193] - Too many connections from /
>>10.231.72.24 - max is 60
>>
>>
>>Do you have many clients or are you inadvertently creating too many handles? Also,
you need to make sure that you're passing a connect string with all servers so that you balance
client connections across servers.
>>
>>-Flavio
>>
>>
>>
>>
>>On Tuesday, September 16, 2014 11:38 AM, lalit jangra <lalit.j.jangra@gmail.com>
wrote:
>>
>>
>>>
>>>
>>>Hello Flavio,
>>>
>>
>>>Below are snapshot from logs i can see. I could see  multiple errors while
>>>zookeeper is running.
>>>
>>>
>>>2014-09-16 10:56:50,078 [myid:3] - INFO  [ProcessThread(sid:3
>>>cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when
>>>processing sessionid:0x3487ddf57dd004f type:create cxid:0x1080
>>>zxid:0xd0000a3ab txntype:-1 reqpath:n/a Error
>>>Path:/org.apache.manifoldcf.locks-Solr:3F512EBFCFCC30070AC6A034726530577B7712B9
>>>Error:KeeperErrorCode = NoNode for
>>>/org.apache.manifoldcf.locks-Solr:3F512EBFCFCC30070AC6A034726530577B7712B9
>>>
>>>2014-09-16 10:56:50,120 [myid:3] - INFO  [ProcessThread(sid:3
>>>cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when
>>>processing sessionid:0x3487ddf57dd0071 type:create cxid:0x79e
>>>zxid:0xd0000a3b8 txntype:-1 reqpath:n/a Error
>>>Path:/org.apache.manifoldcf.locks-Solr:7FE7720ABD4211729605B45374AA3FE87A0E379E
>>>Error:KeeperErrorCode = NoNode for
>>>/org.apache.manifoldcf.locks-Solr:7FE7720ABD4211729605B45374AA3FE87A0E379E
>>>
>>>2014-09-16 10:56:50,154 [myid:3] - INFO  [ProcessThread(sid:3
>>>cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when
>>>processing sessionid:0x3487ddf57dd0054 type:create cxid:0xaa
>>>zxid:0xd0000a3c8 txntype:-1 reqpath:n/a Error Path:/or
>>>
>>>
>>>2014-09-16 10:37:57,590 [myid:5] - INFO  [ProcessThread(sid:5
>>>cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when
>>>processing sessionid:0x1487dce96820026 type:create cxid:0x2fb
>>>zxid:0xb00000864 txntype:-1 reqpath:n/a Error
>>>Path:/org.apache.manifoldcf.locks-Solr:CEF2F7E4592DDB1786561160C16149CF68428D51
>>>Error:KeeperErrorCode = NoNode for
>>>/org.apache.manifoldcf.locks-Solr:CEF2F7E4592DDB1786561160C16149CF68428D51
>>>
>>>2014-09-16 10:37:57,594 [myid:5] - INFO  [ProcessThread(sid:5
>>>cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when
>>>processing sessionid:0x1487dce9682001f type:create cxid:0x67
>>>zxid:0xb00000865 txntype:-1 reqpath:n/a Error
>>>Path:/org.apache.manifoldcf.locks-Solr:CFCC6187B00BA4D0D7B45329CC1F5CFE0F54363B
>>>Error:KeeperErrorCode = NoNode for
>>>/org.apache.manifoldcf.locks-Solr:CFCC6187B00BA4D0D7B45329CC1F5CFE0F54363B
>>>
>>>2014-09-16 10:37:57,642 [myid:5] - INFO  [ProcessThread(sid:5
>>>cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when
>>>processing sessionid:0x3487dd037aa0006 type:create cxid:0x259
>>>zxid:0xb00000872 txntype:-1 reqpath:n/a Error
>>>Path:/org.apache.manifoldcf.locks-Solr:6EC9E05AA96792DBE9E1653076042355EA92F5AA
>>>Error:KeeperErrorCode = NoNode for
>>>/org.apache.manifoldcf.locks-Solr:6EC9E05AA96792DBE9E1653076042355EA92F5AA
>>>
>>>2014-09-16 10:37:58,060 [myid:5] - WARN  [NIOServerCxn.Factory:
>>>0.0.0.0/0.0.0.0:2182:NIOServerCnxnFactory@193] - Too many connections from /
>>>10.231.72.24 - max is 60
>>>
>>>2014-09-16 10:37:58,073 [myid:5] - WARN  [NIOServerCxn.Factory:
>>>0.0.0.0/0.0.0.0:2182:NIOServerCnxnFactory@193] - Too many connections from /
>>>10.231.72.24 - max is 60
>>>
>>>2014-09-16 10:37:58,232 [myid:5] - INFO  [ProcessThread(sid:5
>>>cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when
>>>processing sessionid:0x3487dd037aa0006 type:create cxid:0x279
>>>zxid:0xb0000089c txntype:-1 reqpath:n/a Error
>>>Path:/org.apache.manifoldcf.locks-Solr:AD44DF3352B10CCB5790683E2E2332F7CD42D913
>>>Error:KeeperErrorCode = NoNode for
>>>/org.apache.manifoldcf.locks-Solr:AD44DF3352B10CCB5790683E2E2332F7CD42D913
>>>
>>>2014-09-16 10:37:58,253 [myid:5] - INFO  [ProcessThread(sid:5
>>>cport:-1)::PrepRequestProcessor@645] - Got user-level KeeperException when
>>>processing
>>>
>>>On Tue, Sep 16, 2014 at 2:23 PM, Flavio Junqueira <
>>>fpjunqueira@yahoo.com.invalid> wrote:
>>>
>>>> It seems fine to limit the amount of memory, but I'd rather get the error
>>>> that is preventing the servers from starting/making progress first.
>>>>
>>>> -Flavio
>>>>
>>>>
>>>> On Tuesday, September 16, 2014 9:48 AM, lalit jangra <
>>>> lalit.j.jangra@gmail.com> wrote:
>>>>
>>>>
>>>> >
>>>> >
>>>> >Thanks Flavio,
>>>> >
>>>> >I will try and update. Can you confirm if i add java.env under conf folder
>>>> >with JVM settings as "-Xms 1024m -Xmx1024m" , it will help to limit memory
>>>> >size of zookeeper till 1 G only?
>>>> >
>>>> >Regards.
>>>> >
>>>> >On Tue, Sep 16, 2014 at 2:05 PM, Flavio Junqueira <
>>>> >fpjunqueira@yahoo.com.invalid> wrote:
>>>> >
>>>> >> What if you use 'zkServer.sh start-foreground' to debug?
>>>> >>
>>>> >> -Flavio
>>>> >>
>>>> >>
>>>> >> On Tuesday, September 16, 2014 5:20 AM, lalit jangra <
>>>> >> lalit.j.jangra@gmail.com> wrote:
>>>> >>
>>>> >>
>>>> >> >
>>>> >> >
>>>> >> >Hello Flavio,
>>>> >> >
>>>> >> >I am using 'zkServer.sh start' command to start zookeeper nodes.
I also
>>>> >> >could see logs in log folders in have specified but these logs
are in a
>>>> >> >form which is difficult to understand.
>>>> >> >
>>>> >> >Also  regarding to using 6 zookeeper nodes (3+3), is it fine
to handle
>>>> >> >failures as per 50% rule as if 3 are down my cluster should
work or
>>>> should
>>>> >> >i move to having odd numbers such as 5 or 7 here?
>>>> >> >
>>>> >> >Regards.
>>>> >> >
>>>> >> >On Tue, Sep 16, 2014 at 4:26 AM, Flavio Junqueira <
>>>> >> >fpjunqueira@yahoo.com.invalid> wrote:
>>>> >> >
>>>> >> >> Instead of guessing, I think it is best if we understand
what's going
>>>> >> >> wrong with the servers, you need to look at the server
logs. If you
>>>> >> don't
>>>> >> >> know how to get it, could you please share the command
you're using
>>>> to
>>>> >> >> start servers?
>>>> >> >>
>>>> >> >> -Flavio
>>>> >> >>
>>>> >> >>
>>>> >> >>
>>>> >> >> On Monday, September 15, 2014 3:30 PM, lalit jangra <
>>>> >> >> lalit.j.jangra@gmail.com> wrote:
>>>> >> >>
>>>> >> >>
>>>> >> >> >
>>>> >> >> >
>>>> >> >> >Hello Flavio,
>>>> >> >> >
>>>> >> >> >Can this issue arise from system not having enough
RAM for Java Heap
>>>> >> as i
>>>> >> >> >could see  my system is running on top of its RAM?
>>>> >> >> >
>>>> >> >> >Also is there any way to assign memory to zookeeper
nodes?
>>>> >> >> >
>>>> >> >> >Regards.
>>>> >> >> >
>>>> >> >> >On Mon, Sep 15, 2014 at 7:37 PM, lalit jangra <
>>>> >> lalit.j.jangra@gmail.com>
>>>> >> >> >wrote:
>>>> >> >> >
>>>> >> >> >> Thanks Flavio,
>>>> >> >> >>
>>>> >> >> >> I am having 3+3 zookeeper nodes on two servers
MCF1 & MCF2. Also i
>>>> >> could
>>>> >> >> >> see same error on both nodes. For logs into servers,
i am not
>>>> able to
>>>> >> >> read
>>>> >> >> >> anything from these, how can i read and interpret
from zookeeper
>>>> >> servers
>>>> >> >> >> what is wrong?
>>>> >> >> >>
>>>> >> >> >> I have put different log & data directories
for each of zookeeper,
>>>> >> may
>>>> >> >> be
>>>> >> >> >> i should elaborate a bit more. I am deciding on
names of logs &
>>>> data
>>>> >> >> >> directory as per myid (ranging from 1 to 6).
>>>> >> >> >>
>>>> >> >> >> ZK1 -> Data.1 -> Logs.1
>>>> >> >> >> ZK2 -> Data.2 -> Logs.2
>>>> >> >> >> ZK3 -> Data.3 -> Logs.3
>>>> >> >> >> ZK4 -> Data.4 -> Logs.4
>>>> >> >> >> ZK5 -> Data.5 -> Logs.5
>>>> >> >> >> ZK6 -> Data.6 -> Logs.6
>>>> >> >> >>
>>>> >> >> >> As i have two servers only and i need to make
it running on these
>>>> two
>>>> >> >> only
>>>> >> >> >> so i chose this architecture. Also i am trying
to make even for
>>>> >> scenario
>>>> >> >> >> where one node is down, i have only 3 zookeepers
down so still
>>>> >> second is
>>>> >> >> >> working. If i have odd numbers say 5 or 7, if
server with more
>>>> >> numbers
>>>> >> >> of
>>>> >> >> >> zookeeper is down, its gone.
>>>> >> >> >>
>>>> >> >> >> Regards.
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >> On Mon, Sep 15, 2014 at 7:29 PM, Flavio Junqueira
<
>>>> >> >> >> fpjunqueira@yahoo.com.invalid> wrote:
>>>> >> >> >>
>>>> >> >> >>> I believe you have shared just the client-side
errors, and I was
>>>> >> >> >>> wondering what's going on with the servers.
One problem I could
>>>> spot
>>>> >> >> with
>>>> >> >> >>> the configuration is with the values of dataDir
and dataLogDir.
>>>> It
>>>> >> >> looks
>>>> >> >> >>> like the processes on the same node are writing
to the same
>>>> >> directory,
>>>> >> >> >>> which should be confusing the servers.
>>>> >> >> >>>
>>>> >> >> >>> A couple of things about your setting. I'm
not sure what your
>>>> >> >> motivation
>>>> >> >> >>> is to put multiple servers on the same node.
It will induce
>>>> >> correlated
>>>> >> >> >>> crashes for the servers on the same node.
Also, we in general
>>>> >> >> recommend to
>>>> >> >> >>> use an odd number of servers (5 or 7 for your
case).
>>>> >> >> >>>
>>>> >> >> >>> -Flavio
>>>> >> >> >>>
>>>> >> >> >>> On Wednesday, September 10, 2014 6:29 AM,
lalit jangra <
>>>> >> >> >>> lalit.j.jangra@gmail.com> wrote:
>>>> >> >> >>>
>>>> >> >> >>>
>>>> >> >> >>> >
>>>> >> >> >>> >
>>>> >> >> >>> >Hi,
>>>> >> >> >>> >
>>>> >> >> >>> >I am running cluster of two Apache ManifoldCF
nodes on two
>>>> separate
>>>> >> >> >>> >machines each of which having 3 zookeeper
instances (total 6
>>>> >> >> instances in
>>>> >> >> >>> >cluster). When i am running up manifoldCF
agents, i see below
>>>> >> warning
>>>> >> >> >>> >during startup.
>>>> >> >> >>> >
>>>> >> >> >>> >[http-bio-80-exec-2-SendThread(iwdc1preecma03.iwater.ie:2181)]
>>>> >> INFO
>>>> >> >> >>> >org.apache.zookeeper.ClientCnxn - Unable
to read additional data
>>>> >> from
>>>> >> >> >>> >server sessionid 0x0, likely server has
closed socket, closing
>>>> >> socket
>>>> >> >> >>> >connection and attempting reconnect
>>>> >> >> >>> >
>>>> >> >> >>> >[http-bio-80-exec-2-SendThread(iwdc2preecma04.iwater.ie:2182)]
>>>> >> INFO
>>>> >> >> >>> >org.apache.zookeeper.ClientCnxn - Opening
socket connection to
>>>> >> server
>>>> >> >> >>> >iwdc2preecma04.iwater.ie/10.231.72.25:2182.
Will not attempt to
>>>> >> >> >>> >authenticate using SASL (unknown error)
>>>> >> >> >>> >
>>>> >> >> >>> >
>>>> >> >> >>> >Also i could see below error in logs in
while agents are
>>>> running.
>>>> >> >> >>> >
>>>> >> >> >>> >[localhost-startStop-1-SendThread(iwdc1preecma03.iwater.ie:2183
>>>> )]
>>>> >> >> WARN
>>>> >> >> >>> >org.apache.zookeeper.ClientCnxn - Session
0x6485a8006060079 for
>>>> >> server
>>>> >> >> >>> >iwdc1preecma03.iwater.ie/10.231.72.24:2183,
unexpected error,
>>>> >> closing
>>>> >> >> >>> >socket connection and attempting reconnect
>>>> >> >> >>> >
>>>> >> >> >>> >java.io.IOException: Connection reset
by peer
>>>> >> >> >>> >
>>>> >> >> >>> >        at sun.nio.ch.FileDispatcherImpl.read0(Native
Method)
>>>> >> >> >>> >
>>>> >> >> >>> >        at
>>>> >> sun.nio.ch.SocketDispatcher.read(SocketDispatcher.java:39)
>>>> >> >> >>> >
>>>> >> >> >>> >        at
>>>> sun.nio.ch.IOUtil.readIntoNativeBuffer(IOUtil.java:225)
>>>> >> >> >>> >
>>>> >> >> >>> >        at sun.nio.ch.IOUtil.read(IOUtil.java:193)
>>>> >> >> >>> >
>>>> >> >> >>> >        at
>>>> >> >> sun.nio.ch.SocketChannelImpl.read(SocketChannelImpl.java:375)
>>>> >> >> >>> >
>>>> >> >> >>> >        at
>>>> >> >> >>>
>>>> >> >> >>>
>>>> >> >>
>>>> >>
>>>> >org.apache.zookeeper.ClientCnxnSocketNIO.doIO(ClientCnxnSocketNIO.java:68)
>>>> >> >> >>> >
>>>> >> >> >>> >        at
>>>> >> >> >>>
>>>> >> >> >>>
>>>> >> >>
>>>> >>
>>>> >org.apache.zookeeper.ClientCnxnSocketNIO.doTransport(ClientCnxnSocketNIO.java:355)
>>>> >> >> >>> >
>>>> >> >> >>> >        at
>>>> >> >> >>>
>>>> >> >org.apache.zookeeper.ClientCnxn$SendThread.run(ClientCnxn.java:1068)
>>>> >> >> >>> >
>>>> >> >> >>> >
>>>> >> >> >>> >Below are configurations for 1. zookeeper
nodes & 2. MCF nodes
>>>> for
>>>> >> >> >>> >zookeeper.
>>>> >> >> >>> >
>>>> >> >> >>> >
>>>> >> >> >>> >*zoo.cfg :  Same for all six zookeeper
nodes.*
>>>> >> >> >>> >
>>>> >> >> >>> >
>>>> >> >> >>> ># The number of milliseconds of each tick
>>>> >> >> >>> >
>>>> >> >> >>> >tickTime=2000
>>>> >> >> >>> >
>>>> >> >> >>> >dataDir=/app/IW/zookeeper/data/data.1
>>>> >> >> >>> >
>>>> >> >> >>> >dataLogDir=/app/IW/zookeeper/logs/log.1
>>>> >> >> >>> >
>>>> >> >> >>> >clientPort=2181
>>>> >> >> >>> >
>>>> >> >> >>> >server.1=iwdc1preecma03:2888:3888
>>>> >> >> >>> >
>>>> >> >> >>> >server.2=iwdc1preecma03:2889:3889
>>>> >> >> >>> >
>>>> >> >> >>> >server.3=iwdc1preecma03:2890:3890
>>>> >> >> >>> >
>>>> >> >> >>> >server.4=iwdc2preecma04:2891:3891
>>>> >> >> >>> >
>>>> >> >> >>> >server.5=iwdc2preecma04:2892:3892
>>>> >> >> >>> >
>>>> >> >> >>> >server.6=iwdc2preecma04:2893:3893
>>>> >> >> >>> >
>>>> >> >> >>> ># The number of ticks that the initial
>>>> >> >> >>> >
>>>> >> >> >>> ># synchronization phase can take
>>>> >> >> >>> >
>>>> >> >> >>> >initLimit=10
>>>> >> >> >>> >
>>>> >> >> >>> ># The number of ticks that can pass between
>>>> >> >> >>> >
>>>> >> >> >>> ># sending a request and getting an acknowledgement
>>>> >> >> >>> >
>>>> >> >> >>> >syncLimit=5
>>>> >> >> >>> >
>>>> >> >> >>> ># the directory where the snapshot is
stored.
>>>> >> >> >>> >
>>>> >> >> >>> ># do not use /tmp for storage, /tmp here
is just
>>>> >> >> >>> >
>>>> >> >> >>> ># example sakes.
>>>> >> >> >>> >
>>>> >> >> >>> >#dataDir=/tmp/zookeeper
>>>> >> >> >>> >
>>>> >> >> >>> ># the port at which the clients will connect
>>>> >> >> >>> >
>>>> >> >> >>> >#clientPort=2181
>>>> >> >> >>> >
>>>> >> >> >>> ># the maximum number of client connections.
>>>> >> >> >>> >
>>>> >> >> >>> ># increase this if you need to handle
more clients
>>>> >> >> >>> >
>>>> >> >> >>> >#maxClientCnxns=60
>>>> >> >> >>> >
>>>> >> >> >>> >#
>>>> >> >> >>> >
>>>> >> >> >>> ># Be sure to read the maintenance section
of the
>>>> >> >> >>> >
>>>> >> >> >>> ># administrator guide before turning on
autopurge.
>>>> >> >> >>> >
>>>> >> >> >>> >#
>>>> >> >> >>> >
>>>> >> >> >>> >#
>>>> >> >> >>>
>>>> >> >>
>>>> >>
>>>> http://zookeeper.apache.org/doc/current/zookeeperAdmin.html#sc_maintenance
>>>> >> >> >>> >
>>>> >> >> >>> >#
>>>> >> >> >>> >
>>>> >> >> >>> ># The number of snapshots to retain in
dataDir
>>>> >> >> >>> >
>>>> >> >> >>> >autopurge.snapRetainCount=3
>>>> >> >> >>> >
>>>> >> >> >>> ># Purge task interval in hours
>>>> >> >> >>> >
>>>> >> >> >>> ># Set to "0" to disable auto purge feature
>>>> >> >> >>> >
>>>> >> >> >>> >autopurge.purgeInterval=1
>>>> >> >> >>> >
>>>> >> >> >>> >
>>>> >> >> >>> >
>>>> >> >> >>> >*ManifoldCF configurations : same for
both ManifoldCF nodes.*
>>>> >> >> >>> >
>>>> >> >> >>> >
>>>> >> >> >>> ><property name="org.apache.manifoldcf.lockmanagerclass"
>>>> >> >> >>>
>>>> >> >value="org.apache.manifoldcf.core.lockmanager.ZooKeeperLockManager"/>
>>>> >> >> >>> >
>>>> >> >> >>> >  <property name="org.apache.manifoldcf.zookeeper.connectstring"
>>>> >> >> >>>
>>>> >> >> >>>
>>>> >> >>
>>>> >>
>>>> >value="iwdc1preecma03:2181,iwdc1preecma03:2182,iwdc1preecma03:2183,iwdc2preecma04:2181,iwdc2preecma04:2182,iwdc2preecma04:2183"/>
>>>> >> >> >>> >
>>>> >> >> >>> ><property name="org.apache.manifoldcf.zookeeper.sessiontimeout"
>>>> >> >> >>> >value="4000"/>
>>>> >> >> >>> >
>>>> >> >> >>> >
>>>> >> >> >>> >
>>>> >> >> >>> >*I want to know if due to above warnings/errors,
will zookeeper
>>>> >> stop
>>>> >> >> >>> >working or will zookeeper will work and
these are non-failing
>>>> >> >> messages,
>>>> >> >> >>> >because ManifoldCF jobs are stuck while
i can see these errors.*
>>>> >> >> >>> >
>>>> >> >> >>> >Please suggest.
>>>> >> >> >>> >
>>>> >> >> >>> >Regards,
>>>> >> >> >>> >Lalit.
>>>
>>>> >
>>>> >> >
>>>> >> >> >
>>>> >> >> >>> >
>>>> >> >> >>> >
>>>> >> >> >>> >
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >>
>>>> >> >> >> --
>>>> >> >> >> Regards,
>>>> >> >> >> Lalit.
>>>> >> >> >>
>>>> >> >> >
>>>> >> >> >
>>>> >> >> >
>>>> >> >> >--
>>>> >> >> >Regards,
>>>> >> >> >Lalit.
>>>> >> >> >
>>>> >> >> >
>>>> >> >> >
>>>> >> >>
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> >--
>>>> >> >Regards,
>>>> >> >Lalit.
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >>
>>>> >
>>>> >
>>>> >
>>>> >--
>>>> >Regards,
>>>> >Lalit.
>>>> >
>>>> >
>>>> >
>>>>
>>>
>>>
>>>
>>>--
>>>Regards,
>>>Lalit.
>>>
>>>
>>>
>
>
>-- 
>
>Regards,
>Lalit.
>
>
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message