hbase-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dalia Sobhy <dalia.mohso...@hotmail.com>
Subject RE: Hbase Read/Write throughput measure
Date Sun, 03 Feb 2013 22:45:02 GMT

Dear Mohamed,

It finished compiling and I have changed the hbase-site.xml of ycsb to the one from cloudera.

I am following the steps in your blog, but first I couldn't find the pom.xml in YCSB/hbase
directory. I found it only in YCSB general directory, here is the contents:

 <!-- Properties Management -->
  <properties>
    <maven.assembly.version>2.2.1</maven.assembly.version>
    <hbase.version>0.92.1</hbase.version>
    <cassandra.version>0.7.0</cassandra.version>
    <infinispan.version>7.1.0.CR1</infinispan.version>
    <openjpa.jdbc.version>2.1.1</openjpa.jdbc.version>
    <mapkeeper.version>1.0</mapkeeper.version>                               
          
    <mongodb.version>2.9.0</mongodb.version>
    <orientdb.version>1.0.1</orientdb.version>
    <redis.version>2.0.0</redis.version>
    <voldemort.version>0.81</voldemort.version>
    <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    <thrift.version>0.8.0</thrift.version>
    <hypertable.version>0.9.5.6</hypertable.version>
  </properties>

.....
....

After that I run the following from YCSB directory: mvn clean package
But here are he following errors:
/home/dalia/YCSB/hbase/src/main/java/com/yahoo/ycsb/db/HBaseClient.java:512:13: '{' should
be on the previous line.
/home/dalia/YCSB/hbase/src/main/java/com/yahoo/ycsb/db/HBaseClient.java:512:13: Must have
at least one statement.
/home/dalia/YCSB/hbase/src/main/java/com/yahoo/ycsb/db/HBaseClient.java:513: catch rcurly
at indentation level 12 not at correct indentation, 6
/home/dalia/YCSB/hbase/src/main/java/com/yahoo/ycsb/db/HBaseClient.java:514: for rcurly at
indentation level 8 not at correct indentation, 4
/home/dalia/YCSB/hbase/src/main/java/com/yahoo/ycsb/db/HBaseClient.java:515: method def child
at indentation level 8 not at correct indentation, 4
/home/dalia/YCSB/hbase/src/main/java/com/yahoo/ycsb/db/HBaseClient.java:517: Line is longer
than 80 characters.
/home/dalia/YCSB/hbase/src/main/java/com/yahoo/ycsb/db/HBaseClient.java:517: method def child
at indentation level 8 not at correct indentation, 4
/home/dalia/YCSB/hbase/src/main/java/com/yahoo/ycsb/db/HBaseClient.java:519: method def rcurly
at indentation level 4 not at correct indentation, 2
Audit done.

[INFO] There are 436 checkstyle errors.
[WARNING] Unable to locate Source XRef to link to - DISABLED
[INFO] 
[INFO] --- maven-resources-plugin:2.3:resources (default-resources) @ hbase-binding ---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/dalia/YCSB/hbase/src/main/resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:compile (default-compile) @ hbase-binding ---
[INFO] Compiling 1 source file to /home/dalia/YCSB/hbase/target/classes
[INFO] 
[INFO] --- maven-resources-plugin:2.3:testResources (default-testResources) @ hbase-binding
---
[INFO] Using 'UTF-8' encoding to copy filtered resources.
[INFO] skip non existing resourceDirectory /home/dalia/YCSB/hbase/src/test/resources
[INFO] 
[INFO] --- maven-compiler-plugin:2.3.2:testCompile (default-testCompile) @ hbase-binding ---
[INFO] No sources to compile
[INFO] 
[INFO] --- maven-surefire-plugin:2.10:test (default-test) @ hbase-binding ---
[INFO] No tests to run.
[INFO] Surefire report directory: /home/dalia/YCSB/hbase/target/surefire-reports

These errors are repeated for all the available databases ..

So any idea ?? 
> From: dontariq@gmail.com
> Date: Mon, 4 Feb 2013 02:48:25 +0530
> Subject: Re: Hbase Read/Write throughput measure
> To: user@hbase.apache.org
> 
> it may take some time depending on your connection. wait for sometime. if
> you still find some problem you can visit this
> link<http://cloudfront.blogspot.in/2013/02/how-to-benchmark-hbase-using-ycsb.html#.UQ7Tu5vS8s0>
> 
> Warm Regards,
> Tariq
> https://mtariq.jux.com/
> cloudfront.blogspot.com
> 
> 
> On Mon, Feb 4, 2013 at 2:35 AM, Dalia Sobhy <dalia.mohsobhy@hotmail.com>wrote:
> 
> >
> > Hello Mohamed,
> >
> > I am using ubuntu 12, but the problem is that I was facing some errors
> > while compiling, after solving them.
> >
> > After $git clone http://github.com/brianfrankcooper/YCSB.git
> > Cloning into 'YCSB'...
> > remote: Counting objects: 2260, done.
> > remote: Compressing objects: 100% (694/694), done.
> > Receiving objects:  86% (1952/2260), 764.00 KiB | 51 KiB/s
> >
> > It keeps hanging on 86%, so any clue??
> >
> > Thanksss :D
> >
> > > From: dontariq@gmail.com
> > > Date: Mon, 4 Feb 2013 00:17:40 +0530
> > > Subject: Re: Hbase Read/Write throughput measure
> > > To: user@hbase.apache.org
> > >
> > > Hello Dalia,
> > >
> > >           Me and Anil are talking about this :
> > >
> > > 1 - git clone http://github.com/brianfrankcooper/YCSB.git
> > > 2 - cd YCSB/
> > > 3 - mvn compile
> > >
> > > And not compiling through Java
> > >
> > > Warm Regards,
> > > Tariq
> > > https://mtariq.jux.com/
> > > cloudfront.blogspot.com
> > >
> > >
> > > On Mon, Feb 4, 2013 at 12:14 AM, Dalia Sobhy <dalia.mohsobhy@hotmail.com
> > >wrote:
> > >
> > > > Hi anil,
> > > >
> > > > How to recompile it ?
> > > >
> > > > Sent from my iPhone
> > > >
> > > > On 2013-02-03, at 8:00 PM, "anil gupta" <anilgupta84@gmail.com>
wrote:
> > > >
> > > > > Hi Dalia,
> > > > >
> > > > > Recently, i ran into same problem with YCSB. You will need to
> > recompile
> > > > > YCSB with HBase0.92. Download the maven project from yscb site and
> > > > > recompile it with HBase0.92.
> > > > >
> > > > > HTH,
> > > > > Anil Gupta
> > > > >
> > > > > On Sun, Feb 3, 2013 at 9:49 AM, Dalia Sobhy <
> > dalia.mohsobhy@hotmail.com
> > > > >wrote:
> > > > >
> > > > >>
> > > > >> Hello Mohamed,
> > > > >>
> > > > >> Yes Hbase is running good and already I have a table and and
some
> > > > records
> > > > >> there.
> > > > >>
> > > > >> I have done what you told me I copied all jar files from lib
> > directory
> > > > to
> > > > >> YSCB/db/hbase/lib/ directory. After restart and writing this
command
> > > > >> "./bin/ycsb run hbase -p columnfamily=family -P
> > workloads/workloada", I
> > > > >> receive the same error.
> > > > >>
> > > > >> But I didn't compile, because already I don't have a .java file
> > > > >>
> > > > >> ycsb-0.1.4 directory doesn't contain hbase client java file,
it
> > contains
> > > > >> only this "hbase-binding-0.1.4.jar"
> > > > >>
> > > > >> So what to do ?
> > > > >>
> > > > >>
> > > > >>> From: dontariq@gmail.com
> > > > >>> Date: Sun, 3 Feb 2013 22:54:13 +0530
> > > > >>> Subject: Re: Hbase Read/Write throughput measure
> > > > >>> To: user@hbase.apache.org
> > > > >>>
> > > > >>> Hello Dalia,
> > > > >>>
> > > > >>>        Is your Hbase running OK? Copy the hbase jar from
your
> > > > HBASE_HOME
> > > > >>> and all the jars from your HBASE_HOME/lib/ directory to your
> > > > >>> YSCB/db/hbase/lib/ directory, compile it again and restart
> > everything.
> > > > >>>
> > > > >>> Warm Regards,
> > > > >>> Tariq
> > > > >>> https://mtariq.jux.com/
> > > > >>> cloudfront.blogspot.com
> > > > >>>
> > > > >>>
> > > > >>> On Sun, Feb 3, 2013 at 10:18 PM, Dalia Sobhy <
> > > > dalia.mohsobhy@hotmail.com
> > > > >>> wrote:
> > > > >>>
> > > > >>>>
> > > > >>>> Hello Mohamed,
> > > > >>>>
> > > > >>>> I believe thats not the case.
> > > > >>>>
> > > > >>>> check this blog
> > > > >>
> > > >
> > http://stackoverflow.com/questions/11649824/hbase-error-not-a-hostport-pair
> > > > >>>>
> > > > >>>> It denotes that this error is due to incompatible Hbase
client and
> > > > >> server
> > > > >>>> versions, although I am using Hbase server :0.92 and
the server
> > is the
> > > > >>>> latest one uploaded by ycsb. So I don't know why this
is
> > happening.
> > > > >>>>
> > > > >>>> So do you have any idea?
> > > > >>>>
> > > > >>>> This is the cmd I am executing:
> > > > >>>>
> > > > >>>> root@master1:~/ycsb-0.1.4# ./bin/ycsb run hbase -p
> > > > >> columnfamily="family"
> > > > >>>> -P workloads/workloada
> > > > >>>>
> > > > >>>> java -cp
> > > > >>
> > > >
> > /home/dalia/ycsb-0.1.4/gemfire-binding/conf:/home/dalia/ycsb-0.1.4/hbase-binding/lib/hbase-binding-0.1.4.jar:/home/dalia/ycsb-0.1.4/hbase-binding/conf:/home/dalia/ycsb-0.1.4/jdbc-binding/conf:/home/dalia/ycsb-0.1.4/voldemort-binding/conf:/home/dalia/ycsb-0.1.4/infinispan-binding/conf:/home/dalia/ycsb-0.1.4/nosqldb-binding/conf:/home/dalia/ycsb-0.1.4/core/lib/core-0.1.4.jar
> > > > >>>> com.yahoo.ycsb.Client -db com.yahoo.ycsb.db.HBaseClient
-p
> > > > >>>> columnfamily=family -P workloads/workloada -t
> > > > >>>> YCSB Client 0.1
> > > > >>>> Command line: -db com.yahoo.ycsb.db.HBaseClient -p
> > columnfamily=family
> > > > >> -P
> > > > >>>> workloads/workloada -t
> > > > >>>> Loading workload...
> > > > >>>> Starting test.
> > > > >>>> 13/02/03 18:42:24 INFO zookeeper.ZooKeeper: Client
> > > > >>>> environment:zookeeper.version=3.3.2-1031432, built on
11/05/2010
> > 05:32
> > > > >> GMT
> > > > >>>> 13/02/03 18:42:24 INFO zookeeper.ZooKeeper: Client environment:
> > > > >> host.name=
> > > > >>>> master1.medcloud.com
> > > > >>>> 13/02/03 18:42:24 INFO zookeeper.ZooKeeper: Client
> > > > >>>> environment:java.version=1.6.0_31
> > > > >>>> 13/02/03 18:42:24 INFO zookeeper.ZooKeeper: Client
> > > > >>>> environment:java.vendor=Sun Microsystems Inc.
> > > > >>>> 13/02/03 18:42:24 INFO zookeeper.ZooKeeper: Client
> > > > >>>> environment:java.home=/usr/lib/jvm/j2sdk1.6-oracle/jre
> > > > >>>> 13/02/03 18:42:24 INFO zookeeper.ZooKeeper: Client
> > > > >>
> > > >
> > environment:java.class.path=/home/dalia/ycsb-0.1.4/gemfire-binding/conf:/home/dalia/ycsb-0.1.4/hbase-binding/lib/hbase-binding-0.1.4.jar:/home/dalia/ycsb-0.1.4/hbase-binding/conf:/home/dalia/ycsb-0.1.4/jdbc-binding/conf:/home/dalia/ycsb-0.1.4/voldemort-binding/conf:/home/dalia/ycsb-0.1.4/infinispan-binding/conf:/home/dalia/ycsb-0.1.4/nosqldb-binding/conf:/home/dalia/ycsb-0.1.4/core/lib/core-0.1.4.jar
> > > > >>>> 13/02/03 18:42:24 INFO zookeeper.ZooKeeper: Client
> > > > >>
> > > >
> > environment:java.library.path=/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64/server:/usr/lib/jvm/j2sdk1.6-oracle/jre/lib/amd64:/usr/lib/jvm/j2sdk1.6-oracle/jre/../lib/amd64:/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
> > > > >>>> 13/02/03 18:42:24 INFO zookeeper.ZooKeeper: Client
> > > > >>>> environment:java.io.tmpdir=/tmp
> > > > >>>> 13/02/03 18:42:24 INFO zookeeper.ZooKeeper: Client
> > > > >>>> environment:java.compiler=<NA>
> > > > >>>> 13/02/03 18:42:24 INFO zookeeper.ZooKeeper: Client environment:
> > > > os.name
> > > > >>>> =Linux
> > > > >>>> 13/02/03 18:42:24 INFO zookeeper.ZooKeeper: Client
> > > > >>>> environment:os.arch=amd64
> > > > >>>> 13/02/03 18:42:24 INFO zookeeper.ZooKeeper: Client
> > > > >>>> environment:os.version=3.2.0-23-generic
> > > > >>>> 13/02/03 18:42:24 INFO zookeeper.ZooKeeper: Client environment:
> > > > >> user.name
> > > > >>>> =root
> > > > >>>> 13/02/03 18:42:24 INFO zookeeper.ZooKeeper: Client
> > > > >>>> environment:user.home=/root
> > > > >>>> 13/02/03 18:42:24 INFO zookeeper.ZooKeeper: Client
> > > > >>>> environment:user.dir=/home/dalia/ycsb-0.1.4
> > > > >>>> 13/02/03 18:42:24 INFO zookeeper.ZooKeeper: Initiating
client
> > > > >> connection,
> > > > >>>> connectString=192.168.2.3:2181 sessionTimeout=180000
> > > > >> watcher=hconnection
> > > > >>>> 13/02/03 18:42:24 INFO zookeeper.ClientCnxn: Opening
socket
> > connection
> > > > >> to
> > > > >>>> server /192.168.2.3:2181
> > > > >>>> 13/02/03 18:42:24 INFO zookeeper.ClientCnxn: Socket connection
> > > > >> established
> > > > >>>> to master1.medcloud.com/192.168.2.3:2181, initiating
session
> > > > >>>> 13/02/03 18:42:24 INFO zookeeper.ClientCnxn: Session
establishment
> > > > >>>> complete on server master1.medcloud.com/192.168.2.3:2181,
> > sessionid =
> > > > >>>> 0x13c9b45ac70000a, negotiated timeout = 60000
> > > > >>>> java.lang.IllegalArgumentException: Not a host:port pair:
�
> > > > >>>> 1799@master1.medcloud.commaster1.medcloud.com,60020,1359814720801
> > > > >>>>
> > > > >>>>
> > > > >>>> my hbase-site.xml:
> > > > >>>>
> > > > >>>> <configuration>
> > > > >>>>  <property>
> > > > >>>>    <name>hbase.rootdir</name>
> > > > >>>>   <value>hdfs://192.168.2.3:20001/hbase</value>
> > > > >>>>
> > > > >>>> <!--   <value>hdfs://<HBASE_MASTER_SERVER>:20001/hbase</value>-->
> > > > >>>>    <description>The directory shared by region
servers.
> > > > >>>>    </description>
> > > > >>>>  </property>
> > > > >>>>  <property>
> > > > >>>>    <name>hbase.master</name>
> > > > >>>>        <value>192.168.2.3:60000</value>
> > > > >>>>    <!--<value><HBASE_MASTER_SERVER>:60000</value>-->
> > > > >>>>    <description>The host and port that the HBase
master runs at.
> > > > >>>>    </description>
> > > > >>>>  </property>
> > > > >>>>  <property>
> > > > >>>>    <name>hbase.zookeeper.quorum</name>
> > > > >>>>    <value>192.168.2.3</value>
> > > > >>>>    <!--<value><HBASE_MASTER_SERVER></value>-->
> > > > >>>>  </property>
> > > > >>>> </configuration>
> > > > >>>>
> > > > >>>> So any help please....
> > > > >>>>
> > > > >>>> Thanks,
> > > > >>>>
> > > > >>>>> From: dontariq@gmail.com
> > > > >>>>> Date: Sun, 3 Feb 2013 13:54:28 +0530
> > > > >>>>> Subject: Re: Hbase Read/Write throughput measure
> > > > >>>>> To: user@hbase.apache.org
> > > > >>>>>
> > > > >>>>> Hello Dalia,
> > > > >>>>>
> > > > >>>>>       I am sorry for late response, courtesy hectic
schedule. Add
> > > > >> the
> > > > >>>>> hbase-site.xml file to the classpath and see if it
works for you.
> > > > >>>>>
> > > > >>>>> Warm Regards,
> > > > >>>>> Tariq
> > > > >>>>> https://mtariq.jux.com/
> > > > >>>>> cloudfront.blogspot.com
> > > > >>>>>
> > > > >>>>>
> > > > >>>>> On Sun, Feb 3, 2013 at 3:50 AM, Dalia Sobhy <
> > > > >> dalia.mohsobhy@hotmail.com
> > > > >>>>> wrote:
> > > > >>>>>
> > > > >>>>>>
> > > > >>>>>> I tried yahoo bench mark I got the following
error:
> > > > >>>>>>
> > > > >>>>>> java.lang.IllegalArgumentException: Not a host:port
> > > > >>>>>>
> > > > >>>>>> So any help,
> > > > >>>>>>
> > > > >>>>>> Note that I am using hbase 0.92
> > > > >>>>>>
> > > > >>>>>> I have downloaded the lastest ycsb from github
> > > > >>>>>>
> > > > >>>>>>> From: dalia.mohsobhy@hotmail.com
> > > > >>>>>>> To: user@hbase.apache.org
> > > > >>>>>>> Subject: RE: Hbase Read/Write throughput
measure
> > > > >>>>>>> Date: Sat, 2 Feb 2013 16:47:43 +0200
> > > > >>>>>>>
> > > > >>>>>>>
> > > > >>>>>>> any help ??!!!!!!
> > > > >>>>>>>
> > > > >>>>>>>> Date: Fri, 1 Feb 2013 17:15:45 -0500
> > > > >>>>>>>> Subject: Re: Hbase Read/Write throughput
measure
> > > > >>>>>>>> From: jean-marc@spaggiari.org
> > > > >>>>>>>> To: user@hbase.apache.org
> > > > >>>>>>>>
> > > > >>>>>>>> When you will have figure how the command
is working, you will
> > > > >>>> have to
> > > > >>>>>>>> understand the output ;)
> > > > >>>>>>>>
> > > > >>>>>>>> This is how I just tried it from my HBase
directory:
> > > > >>>>>>>> bin/hbase org.apache.hadoop.hbase.PerformanceEvaluation
> > > > >> randomRead
> > > > >>>> 10
> > > > >>>>>>>>
> > > > >>>>>>>> And this is the output (last lines):
> > > > >>>>>>>> 10066326    56314
> > > > >>>>>>>> 10171183    56385
> > > > >>>>>>>> 10276040    56579
> > > > >>>>>>>> 10380897    56553
> > > > >>>>>>>>
> > > > >>>>>>>> Does ask me what it means ;) I have,
so far, no idea.
> > > > >>>>>>>>
> > > > >>>>>>>> JM
> > > > >>>>>>>>
> > > > >>>>>>>> 2013/2/1, Dalia Sobhy <dalia.mohsobhy@hotmail.com>:
> > > > >>>>>>>>> Dear Mohamed,
> > > > >>>>>>>>>
> > > > >>>>>>>>> I checked this link:
> > > > >>>>>>>>> http://wiki.apache.org/hadoop/Hbase/PerformanceEvaluation
> > > > >>>>>>>>>
> > > > >>>>>>>>> But how to use the commands there
is no clue, note that I am
> > > > >>>> using
> > > > >>>>>> cloudera
> > > > >>>>>>>>> manager.
> > > > >>>>>>>>>
> > > > >>>>>>>>> The other yahoo link I couldn't understand
how to use it.
> > > > >>>>>>>>>
> > > > >>>>>>>>> So any help please??
> > > > >>>>>>>>>
> > > > >>>>>>>>> Sent from my iPad
> > > > >>>>>>>>>
> > > > >>>>>>>>> On Feb 1, 2013, at 2:32 PM, "Mohammad
Tariq" <
> > > > >> dontariq@gmail.com
> > > > >>>>>
> > > > >>>>>> wrote:
> > > > >>>>>>>>>
> > > > >>>>>>>>>> Hello Dalia,
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>         I think the easiest way
to measure the read/write
> > > > >>>>>> throughput is
> > > > >>>>>>>>>> to use "PerformanceEvaluation"
tool that comes with the
> > > > >> Hbase
> > > > >>>>>>>>>> distribution. It spawns a map-reduce
job to do the
> > > > >> reads/writes
> > > > >>>> in
> > > > >>>>>>>>>> parallel. Apart from this there
are several other ways to
> > > > >>>> benchmark
> > > > >>>>>> your
> > > > >>>>>>>>>> Hbase cluster like YSCB <
> > > > >>>> https://github.com/brianfrankcooper/YCSB>.
> > > > >>>>>> You
> > > > >>>>>>>>>> might find this
> > > > >>>>>>>>>> link<
> > > > >> http://wiki.apache.org/hadoop/Hbase/PerformanceEvaluation>
> > > > >>>>>> useful
> > > > >>>>>>>>>> that talks about Hbase performance.
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> HTH
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> Warm Regards,
> > > > >>>>>>>>>> Tariq
> > > > >>>>>>>>>> https://mtariq.jux.com/
> > > > >>>>>>>>>> cloudfront.blogspot.com
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>
> > > > >>>>>>>>>> On Fri, Feb 1, 2013 at 5:41 PM,
Dalia Sobhy
> > > > >>>>>>>>>> <dalia.mohsobhy@hotmail.com>wrote:
> > > > >>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Dear all,
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> I want to measure the read/write
throughput for a code on a
> > > > >>>>>> cluster of
> > > > >>>>>>>>>>> 10
> > > > >>>>>>>>>>> nodes. So is there any code
or way to measure it?
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> I have seen in a cloudera-based
presentation that hbase
> > > > >>>> read/write
> > > > >>>>>>>>>>> throughput = millions queries
per second.
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> So any help please??
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Thanks
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>>
> > > > >>>>>>>>>>> Best Regards,
> > > > >>>>>>>>>>> Dalia Sobhy
> > > > >
> > > > >
> > > > >
> > > > > --
> > > > > Thanks & Regards,
> > > > > Anil Gupta
> > > >
> >
> >
 		 	   		  
Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message