hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jay Vyas <jayunit...@gmail.com>
Subject Re: PleaseHoldException when Master is clearly running as JPS
Date Wed, 22 May 2013 15:38:19 GMT
Okay I finally fixed and the problem...!!!!

The reason it took long is because it's tricky to know if hmaster is failing due to network errors vs file system or other lower level errors.

In my case, it was failing because of both, and i was just restarting the master, which masked the error (second time I started master, I would get a "PleaseWaitException", whereas first time, I would get a "couldn't find master" error).  The trickier part is that there are alot of (okay) connect exceptions thrown in region servers which do not necessarily indicate failure (found this after some googleing) thus creating many red-herrings which made it tricky to tell wether the errors occuring in a run of hbase-start.sh are on the region end, or the master end, or simply network related.

Now - i dont fully recall wether this was the exact case - but im willing to venture to say that Using telnet it became clear to me that in fact the master ports were accessible, but oddly, that the master wasn't working correctly .
********

Maybe this could be changed?  That is - maybe the master startup could be an all-or-nothing task which had a clear end result... Maybe a two part JIRA --

1) if start-hbase.sh cannot start the master, start-hbase.sh should fail fast, or at least print a severe error message, before starting region servers.

2) the starting of master should clearly indicate as the last line in the logs wether or not it started successfully.  



On May 21, 2013, at 7:07 PM, Jay Vyas <jayunit100@gmail.com> wrote:

> Hi again ! 
> 
> Well.. to make this easier, Ive created a script which greps the hotspots in log files out and cleans them, restarting the entire cluster. 
> 
> Overall, it appears that my region servers cant see my hbase-master.... due to "closed connections", or something of that sort. 
> 
> These errors are now reproducible so hopefully someone can shed some light. ----------------->
> 
> 
> EXCEPTIONS
> hbase-master
> -------------------------------------
> tcp        0      0 :::3888                     :::*                        LISTEN      0          172888     4674/java           
> 2013-05-21 18:59:16,537 INFO org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: The identifier of this process is 4729@hbase-master
> 2013-05-21 18:59:16,562 INFO org.apache.zookeeper.ClientCnxn: Opening socket connection to server hbase-master/192.168.122.200:2222. Will not attempt to authenticate using SASL (unknown error)
> 2013-05-21 18:59:16,628 INFO org.apache.zookeeper.ClientCnxn: Socket connection established to hbase-master/192.168.122.200:2222, initiating session
> 2013-05-21 18:59:16,669 INFO org.apache.zookeeper.ClientCnxn: Unable to read additional data from server sessionid 0x0, likely server has closed socket, closing socket connection and attempting reconnect
> 2013-05-21 18:59:16,842 WARN org.apache.hadoop.hbase.zookeeper.RecoverableZooKeeper: Possibly transient ZooKeeper exception: org.apache.zookeeper.KeeperException$ConnectionLossException: KeeperErrorCode = ConnectionLoss for /hbase-master
> 2013-05-21 18:59:16,842 INFO org.apache.hadoop.hbase.util.RetryCounter: Sleeping 2000ms before retry #1...
> Tue May 21 18:59:15 EDT 2013 Starting regionserver on hbase-master
> core file size          (blocks, -c) 0
> data seg size           (kbytes, -d) unlimited
> --
> 2013-05-21 18:59:14,579 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: Notification: 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
> 2013-05-21 18:59:14,580 INFO org.apache.zookeeper.server.quorum.QuorumCnxManager: Received connection request /192.168.122.203:33665
> 2013-05-21 18:59:14,580 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: Notification: 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
> 2013-05-21 18:59:14,582 WARN org.apache.zookeeper.server.quorum.QuorumCnxManager: Interrupted while waiting for message on queue
> java.lang.InterruptedException
>     at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
>     at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2094)
>     at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:370)
>     at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:831)
> --
> 2013-05-21 18:59:16,146 INFO org.apache.zookeeper.server.quorum.Learner: Getting a diff from the leader 0x0
> 2013-05-21 18:59:16,190 INFO org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 0x0 to /tmp/hbase-root/zookeeper/version-2/snapshot.0
> 2013-05-21 18:59:16,524 INFO org.apache.zookeeper.server.persistence.FileTxnSnapLog: Snapshotting: 0x0 to /tmp/hbase-root/zookeeper/version-2/snapshot.0
> 2013-05-21 18:59:16,594 INFO org.apache.zookeeper.server.NIOServerCnxnFactory: Accepted socket connection from /192.168.122.200:40223
> 2013-05-21 18:59:16,661 WARN org.apache.zookeeper.server.NIOServerCnxn: Exception causing close of session 0x0 due to java.io.IOException: ZooKeeperServer not running
> 2013-05-21 18:59:16,661 INFO org.apache.zookeeper.server.NIOServerCnxn: Closed socket connection for client /192.168.122.200:40223 (no session established for client)
> ---------------------------------------
> hbase-regionserver1
> -------------------------------------
> tcp        0      0 :::3888                     :::*                        LISTEN      0          94882      2306/java           
> 2013-05-21 18:59:14,442 INFO org.apache.zookeeper.server.quorum.QuorumCnxManager: My election bind port: 0.0.0.0/0.0.0.0:3888
> 2013-05-21 18:59:14,451 INFO org.apache.zookeeper.server.quorum.QuorumPeer: LOOKING
> 2013-05-21 18:59:14,453 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: New election. My id =  1, proposed zxid=0x0
> 2013-05-21 18:59:14,458 WARN org.apache.zookeeper.server.quorum.QuorumCnxManager: Cannot open channel to 0 at election address hbase-master/192.168.122.200:3888
> java.net.ConnectException: Connection refused
>     at java.net.PlainSocketImpl.socketConnect(Native Method)
>     at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
>     at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
>     at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
> --
>     at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:365)
>     at java.lang.Thread.run(Thread.java:679)
> 2013-05-21 18:59:14,461 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: Notification: 1 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
> 2013-05-21 18:59:14,467 WARN org.apache.zookeeper.server.quorum.QuorumCnxManager: Cannot open channel to 2 at election address hbase-regionserver2/192.168.122.202:3888
> java.net.ConnectException: Connection refused
>     at java.net.PlainSocketImpl.socketConnect(Native Method)
>     at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
>     at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
>     at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
> --
>     at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:393)
>     at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:365)
>     at java.lang.Thread.run(Thread.java:679)
> 2013-05-21 18:59:14,468 WARN org.apache.zookeeper.server.quorum.QuorumCnxManager: Cannot open channel to 3 at election address hbase-regionserver3/192.168.122.203:3888
> java.net.ConnectException: Connection refused
>     at java.net.PlainSocketImpl.socketConnect(Native Method)
>     at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
>     at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
>     at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
> --
>     at java.lang.Thread.run(Thread.java:679)
> 2013-05-21 18:59:14,498 INFO org.apache.zookeeper.server.quorum.QuorumCnxManager: Received connection request /192.168.122.203:57959
> 2013-05-21 18:59:14,508 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: Notification: 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 3 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
> 2013-05-21 18:59:14,508 WARN org.apache.zookeeper.server.quorum.QuorumCnxManager: Cannot open channel to 0 at election address hbase-master/192.168.122.200:3888
> java.net.ConnectException: Connection refused
>     at java.net.PlainSocketImpl.socketConnect(Native Method)
>     at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
>     at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
>     at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
> --
>     at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:365)
>     at java.lang.Thread.run(Thread.java:679)
> 2013-05-21 18:59:14,509 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: Notification: 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 1 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
> 2013-05-21 18:59:14,510 WARN org.apache.zookeeper.server.quorum.QuorumCnxManager: Cannot open channel to 2 at election address hbase-regionserver2/192.168.122.202:3888
> java.net.ConnectException: Connection refused
>     at java.net.PlainSocketImpl.socketConnect(Native Method)
>     at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
>     at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
>     at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
> --
> 2013-05-21 18:59:14,801 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:user.dir=/root/hbaseinstall/hbase-0.94.7
> 2013-05-21 18:59:14,802 INFO org.apache.zookeeper.server.ZooKeeperServer: Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 180000 datadir /tmp/hbase-root/zookeeper/version-2 snapdir /tmp/hbase-root/zookeeper/version-2
> 2013-05-21 18:59:14,802 INFO org.apache.zookeeper.server.quorum.Learner: FOLLOWING - LEADER ELECTION TOOK - 349
> 2013-05-21 18:59:14,804 WARN org.apache.zookeeper.server.quorum.Learner: Unexpected exception, tries=0, connecting to hbase-regionserver3/192.168.122.203:2888
> java.net.ConnectException: Connection refused
>     at java.net.PlainSocketImpl.socketConnect(Native Method)
>     at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
>     at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
>     at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
> ---------------------------------------
> hbase-regionserver2
> -------------------------------------
> tcp        0      0 :::3888                     :::*                        LISTEN      0          95070      2364/java           
> 2013-05-21 18:59:14,511 INFO org.apache.zookeeper.server.quorum.QuorumCnxManager: My election bind port: 0.0.0.0/0.0.0.0:3888
> 2013-05-21 18:59:14,517 INFO org.apache.zookeeper.server.quorum.QuorumPeer: LOOKING
> 2013-05-21 18:59:14,518 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: New election. My id =  2, proposed zxid=0x0
> 2013-05-21 18:59:14,522 WARN org.apache.zookeeper.server.quorum.QuorumCnxManager: Cannot open channel to 0 at election address hbase-master/192.168.122.200:3888
> java.net.ConnectException: Connection refused
>     at java.net.PlainSocketImpl.socketConnect(Native Method)
>     at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
>     at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
>     at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
> --
> 2013-05-21 18:59:14,788 INFO org.apache.zookeeper.server.ZooKeeperServer: Server environment:user.dir=/root/hbaseinstall/hbase-0.94.7
> 2013-05-21 18:59:14,790 INFO org.apache.zookeeper.server.ZooKeeperServer: Created server with tickTime 3000 minSessionTimeout 6000 maxSessionTimeout 180000 datadir /tmp/hbase-root/zookeeper/version-2 snapdir /tmp/hbase-root/zookeeper/version-2
> 2013-05-21 18:59:14,790 INFO org.apache.zookeeper.server.quorum.Learner: FOLLOWING - LEADER ELECTION TOOK - 272
> 2013-05-21 18:59:14,794 WARN org.apache.zookeeper.server.quorum.Learner: Unexpected exception, tries=0, connecting to hbase-regionserver3/192.168.122.203:2888
> java.net.ConnectException: Connection refused
>     at java.net.PlainSocketImpl.socketConnect(Native Method)
>     at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
>     at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
>     at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
> ---------------------------------------
> hbase-regionserver3
> -------------------------------------
> tcp        0      0 :::3888                     :::*                        LISTEN      0          160970     2411/java           
> 2013-05-21 18:59:14,481 INFO org.apache.zookeeper.server.quorum.QuorumCnxManager: My election bind port: 0.0.0.0/0.0.0.0:3888
> 2013-05-21 18:59:14,488 INFO org.apache.zookeeper.server.quorum.QuorumPeer: LOOKING
> 2013-05-21 18:59:14,490 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: New election. My id =  3, proposed zxid=0x0
> 2013-05-21 18:59:14,495 WARN org.apache.zookeeper.server.quorum.QuorumCnxManager: Cannot open channel to 0 at election address hbase-master/192.168.122.200:3888
> java.net.ConnectException: Connection refused
>     at java.net.PlainSocketImpl.socketConnect(Native Method)
>     at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
>     at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
>     at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
> --
>     at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.process(FastLeaderElection.java:393)
>     at org.apache.zookeeper.server.quorum.FastLeaderElection$Messenger$WorkerSender.run(FastLeaderElection.java:365)
>     at java.lang.Thread.run(Thread.java:679)
> 2013-05-21 18:59:14,507 WARN org.apache.zookeeper.server.quorum.QuorumCnxManager: Cannot open channel to 2 at election address hbase-regionserver2/192.168.122.202:3888
> java.net.ConnectException: Connection refused
>     at java.net.PlainSocketImpl.socketConnect(Native Method)
>     at java.net.AbstractPlainSocketImpl.doConnect(AbstractPlainSocketImpl.java:327)
>     at java.net.AbstractPlainSocketImpl.connectToAddress(AbstractPlainSocketImpl.java:193)
>     at java.net.AbstractPlainSocketImpl.connect(AbstractPlainSocketImpl.java:180)
> --
> 2013-05-21 18:59:14,539 INFO org.apache.zookeeper.server.quorum.FastLeaderElection: Notification: 3 (n.leader), 0x0 (n.zxid), 0x1 (n.round), LOOKING (n.state), 2 (n.sid), 0x0 (n.peerEPoch), LOOKING (my state)
> 2013-05-21 18:59:14,563 INFO org.apache.zookeeper.server.quorum.QuorumCnxManager: Received connection request /192.168.122.200:42394
> 2013-05-21 18:59:14,574 INFO org.apache.zookeeper.server.quorum.QuorumCnxManager: Received connection request /192.168.122.200:42395
> 2013-05-21 18:59:14,575 WARN org.apache.zookeeper.server.quorum.QuorumCnxManager: Interrupted while waiting for message on queue
> java.lang.InterruptedException
>     at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.reportInterruptAfterWait(AbstractQueuedSynchronizer.java:2017)
>     at java.util.concurrent.locks.AbstractQueuedSynchronizer$ConditionObject.awaitNanos(AbstractQueuedSynchronizer.java:2094)
>     at java.util.concurrent.ArrayBlockingQueue.poll(ArrayBlockingQueue.java:370)
>     at org.apache.zookeeper.server.quorum.QuorumCnxManager.pollSendQueue(QuorumCnxManager.java:831)
>     at org.apache.zookeeper.server.quorum.QuorumCnxManager.access$500(QuorumCnxManager.java:62)
>     at org.apache.zookeeper.server.quorum.QuorumCnxManager$SendWorker.run(QuorumCnxManager.java:667)
> 2013-05-21 18:59:14,576 WARN org.apache.zookeeper.server.quorum.QuorumCnxManager: Send worker leaving thread
> 2013-05-21 18:59:14,576 WARN org.apache.zookeeper.server.quorum.QuorumCnxManager: Connection broken for id 0, my id = 3, error = 
> java.net.SocketException: Socket closed
>     at java.net.SocketInputStream.socketRead0(Native Method)
>     at java.net.SocketInputStream.read(SocketInputStream.java:146)
>     at java.net.SocketInputStream.read(SocketInputStream.java:199)
>     at java.io.DataInputStream.readInt(DataInputStream.java:388)
> ---------------------------------------
> 
> 
> 
> On Tue, May 21, 2013 at 5:33 PM, Jean-Daniel Cryans <jdcryans@apache.org> wrote:
>> I see:
>> 
>> 2013-05-21 17:15:07,914 DEBUG
>> org.apache.hadoop.hbase.master.AssignmentManager: Handling
>> transition=RS_ZK_REGION_FAILED_OPEN,
>> server=hbase-regionserver1,60020,1369170595340, region=70236052/-ROOT-
>> 
>> Over and over. Look in the region server logs, you should see fat
>> stack traces on why it's failing to open the -ROOT- region. Maybe
>> related to your gluster setup.
>> 
>> J-D
>> 
>> On Tue, May 21, 2013 at 2:18 PM, Jay Vyas <jayunit100@gmail.com> wrote:
>> > https://gist.github.com/anonymous/5623327 <-- all logs for starting up
>> > hbase and master
>> >
>> >
>> > On Tue, May 21, 2013 at 5:13 PM, Mohammad Tariq <dontariq@gmail.com> wrote:
>> >
>> >> No prob. I was referring to this :
>> >>
>> >> 127.0.0.1 hbase-master
>> >> 192.168.122.200 hbase-master
>> >>
>> >> I was thinking that this is your HBase master. Correct me if i'm wrong.
>> >> Could you please show me your logs?
>> >>
>> >> Warm Regards,
>> >> Tariq
>> >> cloudfront.blogspot.com
>> >>
>> >>
>> >> On Wed, May 22, 2013 at 2:36 AM, Jay Vyas <jayunit100@gmail.com> wrote:
>> >>
>> >> > Hmmm... what do you mean "have your hostname in there"?  sorry -- just
>> >> > curious about which hostname you are referring to...?
>> >> >
>> >> > Im now getting a new exception:
>> >> >
>> >> > 13/05/21 17:02:44 INFO
>> >> client.HConnectionManager$HConnectionImplementation:
>> >> > getMaster attempt 2 of 7 failed; retrying after sleep of 1002
>> >> >
>> >> >
>> >> >
>> >> > On Tue, May 21, 2013 at 4:57 PM, Mohammad Tariq <dontariq@gmail.com>
>> >> > wrote:
>> >> >
>> >> > > OK..You already have your hostname in there. But it is appearing twice.
>> >> > > Comment out "127.0.0.1 hbase-master". This might be a reason. I did not
>> >> > > notice that you are on a distributed setup. RS IPs and hostnames are
>> >> > fine.
>> >> > >
>> >> > > Warm Regards,
>> >> > > Tariq
>> >> > > cloudfront.blogspot.com
>> >> > >
>> >> > >
>> >> > > On Wed, May 22, 2013 at 2:21 AM, Jay Vyas <jayunit100@gmail.com>
>> >> wrote:
>> >> > >
>> >> > > > Hi kevin : So you don't have any region servers defined in your
>> >> > > /etc/hosts
>> >> > > > ?
>> >> > > >
>> >> > > >
>> >> > > > On Tue, May 21, 2013 at 4:46 PM, Mohammad Tariq <dontariq@gmail.com>
>> >> > > > wrote:
>> >> > > >
>> >> > > > > Sorry, my bad. By that I meant "127.0.0.1    hostname"..To me it
>> >> > seems
>> >> > > > like
>> >> > > > > HBase is not able to connect to localhost using 127.0.0.1
>> >> > > > >
>> >> > > > > Warm Regards,
>> >> > > > > Tariq
>> >> > > > > cloudfront.blogspot.com
>> >> > > > >
>> >> > > > >
>> >> > > > > On Wed, May 22, 2013 at 2:12 AM, Jay Vyas <jayunit100@gmail.com>
>> >> > > wrote:
>> >> > > > >
>> >> > > > > > Thanks, but adding 127.0.0.1 localhost to the top seems
>> >> > redundant...
>> >> > > > > > right?  I did so but still no luck :(.
>> >> > > > > >
>> >> > > > > > 1) OS? This is fedora 16.....
>> >> > > > > >
>> >> > > > > > 2) any thoughts on why the "PleaseHoldException" is being
>> >> > triggered ?
>> >> > > > > >
>> >> > > > > >
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > On Tue, May 21, 2013 at 4:32 PM, Mohammad Tariq <
>> >> > dontariq@gmail.com>
>> >> > > > > > wrote:
>> >> > > > > >
>> >> > > > > > > OS?Add "127.0.0.1   localhost" and see if it makes any
>> >> > difference.
>> >> > > > > > >
>> >> > > > > > > Warm Regards,
>> >> > > > > > > Tariq
>> >> > > > > > > cloudfront.blogspot.com
>> >> > > > > > >
>> >> > > > > > >
>> >> > > > > > > On Wed, May 22, 2013 at 1:57 AM, Jay Vyas <
>> >> jayunit100@gmail.com>
>> >> > > > > wrote:
>> >> > > > > > >
>> >> > > > > > > > #This is my /etc/hosts file ---
>> >> > > > > > > > 127.0.0.1   localhost localhost.localdomain localhost4
>> >> > > > > > > > localhost4.localdomain4
>> >> > > > > > > > #::1         localhost localhost.localdomain localhost6
>> >> > > > > > > > localhost6.localdomain6
>> >> > > > > > > > 127.0.0.1 hbase-master
>> >> > > > > > > > 192.168.122.200 hbase-master
>> >> > > > > > > > 192.168.122.201 hbase-regionserver1
>> >> > > > > > > > 192.168.122.202 hbase-regionserver2
>> >> > > > > > > > 192.168.122.203 hbase-regionserver3
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > > On Tue, May 21, 2013 at 4:25 PM, Mohammad Tariq <
>> >> > > > dontariq@gmail.com>
>> >> > > > > > > > wrote:
>> >> > > > > > > >
>> >> > > > > > > > > Hello Jay,
>> >> > > > > > > > >
>> >> > > > > > > > >       Please change the line containing "127.0.1.1" in your
>> >> > > > > > /etc/hosts
>> >> > > > > > > to
>> >> > > > > > > > > "127.0.0.1" and see if it works.
>> >> > > > > > > > >
>> >> > > > > > > > > Warm Regards,
>> >> > > > > > > > > Tariq
>> >> > > > > > > > > cloudfront.blogspot.com
>> >> > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > > > On Wed, May 22, 2013 at 1:47 AM, Jay Vyas <
>> >> > > jayunit100@gmail.com>
>> >> > > > > > > wrote:
>> >> > > > > > > > >
>> >> > > > > > > > > > Hi folks:
>> >> > > > > > > > > >
>> >> > > > > > > > > > Hope someone can shed some light on this -
>> >> > > > > > > > > >
>> >> > > > > > > > > > I cannot run "hbase shell" create commands because of the
>> >> > > > > > > > > > PleaseHoldException on a fresh install of hbase.  Im not
>> >> > > > finding
>> >> > > > > > much
>> >> > > > > > > > in
>> >> > > > > > > > > > the error logs, and all nodes appear to be up and
>> >> running,
>> >> > > > > > including
>> >> > > > > > > > > hbase
>> >> > > > > > > > > > master.
>> >> > > > > > > > > >
>> >> > > > > > > > > > Version : hbase-0.94.7
>> >> > > > > > > > > >
>> >> > > > > > > > > > Error: I am getting the "Please hold exception"... on my
>> >> > > hbase
>> >> > > > > > shell.
>> >> > > > > > > > > When
>> >> > > > > > > > > > running "create table 't1','f1'"... But...
>> >> > > > > > > > > >
>> >> > > > > > > > > > But wait :) theres more !  ... clearly, the hbase master
>> >> is
>> >> > > > > > running :
>> >> > > > > > > > > >
>> >> > > > > > > > > > [root@hbase-master ~]# jps
>> >> > > > > > > > > > 11896 HQuorumPeer
>> >> > > > > > > > > > 12914 Jps
>> >> > > > > > > > > > 9894 Main
>> >> > > > > > > > > > 5879 Main
>> >> > > > > > > > > > ******************
>> >> > > > > > > > > > 12279 HMaster
>> >> > > > > > > > > > ******************
>> >> > > > > > > > > > 5779 Main
>> >> > > > > > > > > > 11714 ZKServerTool
>> >> > > > > > > > > > 12058 HRegionServer
>> >> > > > > > > > > > 12860 Main
>> >> > > > > > > > > > 8369 Main
>> >> > > > > > > > > >
>> >> > > > > > > > > > And finally - here is a dump of the output from the shell
>> >> > ---
>> >> > > > any
>> >> > > > > > > > > thoughts?
>> >> > > > > > > > > >
>> >> > > > > > > > > > [root@hbase-master ~]# hbaseinstall/hbase-0.94.7/bin/
>> >> > > > > > > > > > hbase shell -d <<EOF
>> >> > > > > > > > > > create 't1','f1'
>> >> > > > > > > > > > EOF
>> >> > > > > > > > > >
>> >> > > > > > > > > > Setting DEBUG log level...
>> >> > > > > > > > > >
>> >> > > > > > > > > > HBase Shell; enter 'help<RETURN>' for list of supported
>> >> > > > commands.
>> >> > > > > > > > > > Type "exit<RETURN>" to leave the HBase Shell
>> >> > > > > > > > > > Version 0.94.7, r1471806, Wed Apr 24 18:48:26 PDT 2013
>> >> > > > > > > > > >
>> >> > > > > > > > > > create 't1','f1'
>> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
>> >> > > > > > > > > > environment:zookeeper.version=3.4.5-1392090, built on
>> >> > > > 09/30/2012
>> >> > > > > > > 17:52
>> >> > > > > > > > > GMT
>> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
>> >> > > environment:
>> >> > > > > > > > host.name
>> >> > > > > > > > > > =localhost
>> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
>> >> > > > > > > > > > environment:java.version=1.6.0_24
>> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
>> >> > > > > > > > > > environment:java.vendor=Sun Microsystems Inc.
>> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
>> >> > > > > > > > > >
>> >> > > > > > > >
>> >> > > > > >
>> >> > > >
>> >> > environment:java.home=/usr/lib/jvm/java-1.6.0-openjdk-1.6.0.0.x86_64/jre
>> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >> environment:java.class.path=/root/hbaseinstall/hbase-0.94.7/bin/../conf:/usr/lib/jvm/jre-1.6.0-openjdk.x86_64/lib/tools.jar:/root/hbaseinstall/hbase-0.94.7/bin/..:/root/hbaseinstall/hbase-0.94.7/bin/../hbase-0.94.7.jar:/root/hbaseinstall/hbase-0.94.7/bin/../hbase-0.94.7-tests.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/activation-1.1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/asm-3.1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/avro-1.5.3.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/avro-ipc-1.5.3.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/commons-beanutils-1.7.0.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/commons-beanutils-core-1.8.0.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/commons-cli-1.2.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/commons-codec-1.4.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/commons-collections-3.2.1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/commons-configuration-1.6.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/commons-digester-1.8.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/commons-el-1.0.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/commons-httpclient-3.1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/commons-io-2.1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/commons-lang-2.5.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/commons-logging-1.1.1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/commons-math-2.1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/commons-net-1.4.1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/core-3.1.1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/glusterfs-0.0.1-20130520-1810.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/guava-11.0.2.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/hadoop-core-1.0.4.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/high-scale-lib-1.1.1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/httpclient-4.1.2.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/httpcore-4.1.3.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jackson-core-asl-1.8.8.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jackson-jaxrs-1.8.8.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jackson-mapper-asl-1.8.8.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jackson-xc-1.8.8.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jamon-runtime-2.3.1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jasper-compiler-5.5.23.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jasper-runtime-5.5.23.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jaxb-api-2.1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jaxb-impl-2.2.3-1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jersey-core-1.8.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jersey-json-1.8.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jersey-server-1.8.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jettison-1.1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jetty-6.1.26.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jetty-util-6.1.26.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jruby-complete-1.6.5.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jsp-2.1-6.1.14.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jsp-api-2.1-6.1.14.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/jsr305-1.3.9.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/junit-4.10-HBASE-1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/libthrift-0.8.0.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/log4j-1.2.16.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/metrics-core-2.1.2.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/netty-3.2.4.Final.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/protobuf-java-2.4.0a.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/servlet-api-2.5-6.1.14.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/slf4j-api-1.4.3.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/slf4j-log4j12-1.4.3.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/snappy-java-1.0.3.2.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/stax-api-1.0.1.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/velocity-1.7.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/xmlenc-0.52.jar:/root/hbaseinstall/hbase-0.94.7/bin/../lib/zookeeper-3.4.5.jar:
>> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >> environment:java.library.path=/root/hbaseinstall/hbase-0.94.7/bin/../lib/native/Linux-amd64-64
>> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
>> >> > > > > > > > > > environment:java.io.tmpdir=/tmp
>> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
>> >> > > > > > > > > > environment:java.compiler=<NA>
>> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
>> >> > > environment:
>> >> > > > > > > os.name
>> >> > > > > > > > > > =Linux
>> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
>> >> > > > > > > > > > environment:os.arch=amd64
>> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
>> >> > > > > > > > > > environment:os.version=3.1.0-7.fc16.x86_64
>> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
>> >> > > environment:
>> >> > > > > > > > user.name
>> >> > > > > > > > > > =root
>> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
>> >> > > > > > > > > > environment:user.home=/root
>> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Client
>> >> > > > > > > > > > environment:user.dir=/root
>> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ZooKeeper: Initiating
>> >> > client
>> >> > > > > > > > connection,
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >> connectString=hbase-regionserver3:2222,hbase-regionserver2:2222,hbase-regionserver1:2222,hbase-master:2222
>> >> > > > > > > > > > sessionTimeout=180000 watcher=hconnection
>> >> > > > > > > > > > 13/05/21 15:59:08 DEBUG zookeeper.ClientCnxn:
>> >> > > > > > > > > > zookeeper.disableAutoWatchReset is false
>> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.RecoverableZooKeeper:
>> >> The
>> >> > > > > > identifier
>> >> > > > > > > > of
>> >> > > > > > > > > > this process is 12762@hbase-master
>> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ClientCnxn: Opening
>> >> socket
>> >> > > > > > > connection
>> >> > > > > > > > to
>> >> > > > > > > > > > server hbase-regionserver3/192.168.122.203:2222. Will
>> >> not
>> >> > > > > attempt
>> >> > > > > > to
>> >> > > > > > > > > > authenticate using SASL (unknown error)
>> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ClientCnxn: Socket
>> >> > > connection
>> >> > > > > > > > > established
>> >> > > > > > > > > > to hbase-regionserver3/192.168.122.203:2222, initiating
>> >> > > > session
>> >> > > > > > > > > > 13/05/21 15:59:08 DEBUG zookeeper.ClientCnxn: Session
>> >> > > > > establishment
>> >> > > > > > > > > request
>> >> > > > > > > > > > sent on hbase-regionserver3/192.168.122.203:2222
>> >> > > > > > > > > > 13/05/21 15:59:08 INFO zookeeper.ClientCnxn: Session
>> >> > > > > establishment
>> >> > > > > > > > > complete
>> >> > > > > > > > > > on server hbase-regionserver3/192.168.122.203:2222,
>> >> > > sessionid
>> >> > > > =
>> >> > > > > > > > > > 0x33ec8aac0f40002, negotiated timeout = 180000
>> >> > > > > > > > > > 13/05/21 15:59:08 DEBUG zookeeper.ClientCnxn: Reading
>> >> reply
>> >> > > > > > > > > > sessionid:0x33ec8aac0f40002, packet:: clientPath:null
>> >> > > > > > serverPath:null
>> >> > > > > > > > > > finished:false header:: 1,3  replyHeader::
>> >> 1,51539607605,0
>> >> > > > > >  request::
>> >> > > > > > > > > > '/hbase-master/hbaseid,F  response::
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >> s{17179869216,51539607572,1368812077009,1369166263798,12,0,0,0,59,0,17179869216}
>> >> > > > > > > > > >
>> >> > > > > > > > > > 13/05/21 15:59:08 DEBUG zookeeper.ClientCnxn: Reading
>> >> reply
>> >> > > > > > > > > > sessionid:0x33ec8aac0f40002, packet:: clientPath:null
>> >> > > > > > serverPath:null
>> >> > > > > > > > > > finished:false header:: 2,4  replyHeader::
>> >> 2,51539607605,0
>> >> > > > > >  request::
>> >> > > > > > > > > > '/hbase-master/hbaseid,F  response::
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >> #ffffffff0001231323237394068626173652d6d617374657234333565333230302d613533612d346634662d393838342d393735633736333966343233,s{17179869216,51539607572,1368812077009,1369166263798,12,0,0,0,59,0,17179869216}
>> >> > > > > > > > > >
>> >> > > > > > > > > > 13/05/21 15:59:08 DEBUG zookeeper.ClientCnxn: Reading
>> >> reply
>> >> > > > > > > > > > sessionid:0x33ec8aac0f40002, packet:: clientPath:null
>> >> > > > > > serverPath:null
>> >> > > > > > > > > > finished:false header:: 3,3  replyHeader::
>> >> 3,51539607605,0
>> >> > > > > >  request::
>> >> > > > > > > > > > '/hbase-master/master,T  response::
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >> s{51539607570,51539607570,1369166263477,1369166263477,0,0,0,17672084227293186,54,0,51539607570}
>> >> > > > > > > > > >
>> >> > > > > > > > > > 13/05/21 15:59:08 DEBUG zookeeper.ClientCnxn: Reading
>> >> reply
>> >> > > > > > > > > > sessionid:0x33ec8aac0f40002, packet:: clientPath:null
>> >> > > > > > serverPath:null
>> >> > > > > > > > > > finished:false header:: 4,4  replyHeader::
>> >> 4,51539607605,0
>> >> > > > > >  request::
>> >> > > > > > > > > > '/hbase-master/master,T  response::
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >> #ffffffff0001231323237394068626173652d6d6173746572006c6f63616c686f73742c36303030302c31333639313636323633313032,s{51539607570,51539607570,1369166263477,1369166263477,0,0,0,17672084227293186,54,0,51539607570}
>> >> > > > > > > > > >
>> >> > > > > > > > > > 13/05/21 15:59:08 DEBUG zookeeper.ClientCnxn: Reading
>> >> reply
>> >> > > > > > > > > > sessionid:0x33ec8aac0f40002, packet:: clientPath:null
>> >> > > > > > serverPath:null
>> >> > > > > > > > > > finished:false header:: 5,3  replyHeader::
>> >> 5,51539607605,0
>> >> > > > > >  request::
>> >> > > > > > > > > > '/hbase-master/root-region-server,T  response::
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >> s{51539607579,51539607579,1369166269726,1369166269726,0,0,0,0,52,0,51539607579}
>> >> > > > > > > > > >
>> >> > > > > > > > > > 13/05/21 15:59:08 DEBUG zookeeper.ClientCnxn: Reading
>> >> reply
>> >> > > > > > > > > > sessionid:0x33ec8aac0f40002, packet:: clientPath:null
>> >> > > > > > serverPath:null
>> >> > > > > > > > > > finished:false header:: 6,4  replyHeader::
>> >> 6,51539607605,0
>> >> > > > > >  request::
>> >> > > > > > > > > > '/hbase-master/root-region-server,T  response::
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >> #ffffffff0001231323035384068626173652d6d61737465726c6f63616c686f73742c36303032302c31333639313636323335353932,s{51539607579,51539607579,1369166269726,1369166269726,0,0,0,0,52,0,51539607579}
>> >> > > > > > > > > >
>> >> > > > > > > > > > 13/05/21 15:59:08 DEBUG zookeeper.ClientCnxn: Reading
>> >> reply
>> >> > > > > > > > > > sessionid:0x33ec8aac0f40002, packet:: clientPath:null
>> >> > > > > > serverPath:null
>> >> > > > > > > > > > finished:false header:: 7,3  replyHeader::
>> >> 7,51539607605,0
>> >> > > > > >  request::
>> >> > > > > > > > > > '/hbase-master,F  response::
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > >
>> >> >
>> >> s{17179869186,17179869186,1368811379390,1368811379390,0,88,0,0,0,12,51539607579}
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > > >
>> >> > > > > > > > > > --
>> >> > > > > > > > > > Jay Vyas
>> >> > > > > > > > > > http://jayunit100.blogspot.com
>> >> > > > > > > > > >
>> >> > > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > >
>> >> > > > > > > > --
>> >> > > > > > > > Jay Vyas
>> >> > > > > > > > http://jayunit100.blogspot.com
>> >> > > > > > > >
>> >> > > > > > >
>> >> > > > > >
>> >> > > > > >
>> >> > > > > >
>> >> > > > > > --
>> >> > > > > > Jay Vyas
>> >> > > > > > http://jayunit100.blogspot.com
>> >> > > > > >
>> >> > > > >
>> >> > > >
>> >> > > >
>> >> > > >
>> >> > > > --
>> >> > > > Jay Vyas
>> >> > > > http://jayunit100.blogspot.com
>> >> > > >
>> >> > >
>> >> >
>> >> >
>> >> >
>> >> > --
>> >> > Jay Vyas
>> >> > http://jayunit100.blogspot.com
>> >> >
>> >>
>> >
>> >
>> >
>> > --
>> > Jay Vyas
>> > http://jayunit100.blogspot.com
> 
> 
> 
> -- 
> Jay Vyas
> http://jayunit100.blogspot.com

Mime
  • Unnamed multipart/alternative (inline, 7-Bit, 0 bytes)
View raw message