flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Palle <pa...@sport.dk>
Subject Re: HBase write problem
Date Wed, 11 May 2016 14:19:24 GMT
I run the job from the cluster. I run it through the web UI.
The jar file submitted does not contain the hbase-site.xml file.

----- Original meddelelse -----

> Fra: Flavio Pompermaier <pompermaier@okkam.it>
> Til: user <user@flink.apache.org>
> Dato: Ons, 11. maj 2016 09:36
> Emne: Re: HBase write problem
> 
> Do you run the job from your IDE or from the cluster?
> 
> On Wed, May 11, 2016 at 9:22 AM, Palle < palle@sport.dk [mailto:palle@sport.dk]>
> wrote:
> 
>   Thanks for the response, but I don't think the problem is the
>   classpath - hbase-site.xml should be added. This is what it looks
>   like (hbase conf is added at the end):
> 
>   2016-05-11 09:16:45,831 INFO org.apache.zookeeper.ZooKeeper -
>   Client
>   environment:java.class.path=C:\systems\packages\flink-1.0.2\lib\flink-dist_2.11-1.0.2.jar;C:\systems\packages\flink-1.0.2\lib\flink-python_2.11-1.0.2.jar;C:\systems\packages\flink-1.0.2\lib\guava-11.0.2.jar;C:\systems\packages\flink-1.0.2\lib\hbase-annotations-1.2.1-tests.jar;C:\systems\packages\flink-1.0.2\lib\hbase-annotations-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-client-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-common-1.2.1-tests.jar;C:\systems\packages\flink-1.0.2\lib\hbase-common-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-examples-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-external-blockcache-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-hadoop-compat-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-hadoop2-compat-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-it-1.2.1-tests.jar;C:\systems\packages\flink-1.0.2\lib\hbase-it-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-prefix-tree-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-procedure-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-protocol-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-resource-bundle-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-rest-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-server-1.2.1-tests.jar;C:\systems\packages\flink-1.0.2\lib\hbase-server-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-shell-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-thrift-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\log4j-1.2.17.jar;C:\systems\packages\flink-1.0.2\lib\slf4j-log4j12-1.7.7.jar;C:\systems\master_flink\bin;C:\systems\packages\flink-1.0.2\lib;;C:\systems\packages\hbase-1.2.1\lib;C:\systems\hbase\conf;C:\systems\hbase\conf\hbase-site.xml;
>   2016-05-11 09:16:45,831 INFO org.apache.zookeeper.ZooKeeper -
>   Client
>   environment:java.library.path=C:\systems\packages\jre-1.8.0_74_x64\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\systems\master_flink\bin;C:\systems\packages\appsync-1.0.6\bin;C:\systems\packages\flink-1.0.2\bin;C:\systems\packages\jre-1.8.0_74_x64\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program
>   Files\Microsoft SQL Server\120\DTS\Binn\;C:\Program
>   Files\Microsoft SQL Server\Client
>   SDK\ODBC\110\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL
>   Server\120\Tools\Binn\;C:\Program Files\Microsoft SQL
>   Server\120\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL
>   Server\120\Tools\Binn\ManagementStudio\;C:\Program Files
>   (x86)\Microsoft SQL
>   Server\120\DTS\Binn\;C:\systems\packages\hadoop-2.7.2\bin;C:\systems\packages\jdk-1.8.0_74_x64\bin;C:\systems\packages\apache-maven-3.3.9\bin;C:\systems\packages\protoc-2.5.0-win32\;C:\systems\packages\cygwin64\bin\;C:\systems\packages\cmake-3.5.2-win32-x86\bin;C:\Program
>   Files\Microsoft Windows Performance
>   Toolkit\;C:\systems\packages\perl-5.6.0-win\bin;C:\systems\hbase\conf;.
>   ----- Original meddelelse -----
> 
>     Fra: Flavio Pompermaier < pompermaier@okkam.it [mailto:pompermaier@okkam.it]>
>     Til: user < user@flink.apache.org [mailto:user@flink.apache.org]>
>     Dato: Ons, 11. maj 2016 00:05
>     Emne: Re: HBase write problem
> 
>     Do you have the hbase-site.xml available in the classpath?
> 
>     On 10 May 2016 23:10, "Palle" < palle@sport.dk [mailto:palle@sport.dk]>
>     wrote:
> 
>       HBase write problem
> 
>       Hi all.
> 
>       I have a problem writing to HBase.
> 
>       I am using a slightly modified example of this class to
>       proof the concept:
>       https://github.com/apache/flink/blob/master/flink-batch-connectors/flink-hbase/src/test/java/org/apache/flink/addons/hbase/example/HBaseWriteExample.java
[https://github.com/apache/flink/blob/master/flink-batch-connectors/flink-hbase/src/test/java/org/apache/flink/addons/hbase/example/HBaseWriteExample.java]
> 
>       However all the HBase-specific stuff is exactly the same
>       as in the HBaseWriteExample.
> 
>       The problem I see is that the job never completes (been
>       running for more than an hour now) and it is only 13
>       key/value pairs that is to be written to HBase :-)
>       I have tested the map/reduce stuff works if I replace the
>       HBase connection stuff with just a write to a text file -
>       works OK. I have also tested that I can insert data in
>       HBase from a similar Hadoop MapReduce job.
> 
>       Here is the part of the code where I guess the problem
>       is:
> 
>       @Override
>       public Tuple2<Text, Mutation> map(Tuple2<String, Integer>
>       t) throws Exception {
>       LOG.info("Tuple2 map() called");
>       reuse.f0 = new Text(t.f0);
>       Put put = new Put(t.f0.getBytes());
>       put.add(MasterConstants.CF_SOME, MasterConstants.COUNT,
>       Bytes.toBytes(t.f1));
>       reuse.f1 = put;
>       return reuse;
>       }
>       }).output(new HadoopOutputFormat<Text, Mutation>(new
>       TableOutputFormat<Text>(), job));
> 
>       env.execute("Flink HBase Event Count Hello World Test");
> 
>       This code matches the code in the HBaseWriteExample.java
>       I should think.
> 
>       The "Tuple2" log line I see exactly the 13 times I
>       expect, and the last log line I see is this:
>       2016-05-10 21:48:42,715 INFO
>       org.apache.hadoop.hbase.mapreduce.TableOutputFormat -
>       Created table instance for event_type_count
> 
>       Any suggestions to what the problem could be?
> 
>       Thanks,
>       Palle


Mime
View raw message