flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Flavio Pompermaier <pomperma...@okkam.it>
Subject Re: HBase write problem
Date Wed, 11 May 2016 14:53:28 GMT
I can't help you with the choice of the db storage, as always the answer is
"it depends" on a lot of factors :)

For what I can tell you the problem could be that Flink support HBase 0.98,
so it could worth to update Flink connectors to a more recent version (that
should be backward compatible hopefully..) or maybe create two separte
hbase connectors (one for hbase-0.9x and one for 0.1x). Let me know about
your attempts :)

On Wed, May 11, 2016 at 4:47 PM, Palle <palle@sport.dk> wrote:

> Hadoop 2.7.2
> HBase 1.2.1
>
> I have this running from a Hadoop job, but just not from Flink.
>
> I will look into your suggestions, but would I be better off choosing
> another DB for storage? I can see that  Cassandra gets some attention in
> this mailing list. I need to store app 2 bio key value pairs consisting of
> 100 bytes for each pair.
>
> ----- Original meddelelse -----
>
> *Fra:* Flavio Pompermaier <pompermaier@okkam.it>
> *Til:* user <user@flink.apache.org>
> *Dato:* Ons, 11. maj 2016 16:29
>
> *Emne:* Re: HBase write problem
>
> And which version of HBase and Hadoop are you running?
> Did you try to put the hbase-site.xml in the jar?
> Moreover, I don't know how much reliable is at the moment the web client
> UI..my experience is that the command line client is much more reliable.
> You just need to run from the flink dir something like:
>    bin/flink  run -c  xxx.yyy.MyMainClass /path/to/shadedJar.jar
>
> On Wed, May 11, 2016 at 4:19 PM, Palle <palle@sport.dk> wrote:
>
>> I run the job from the cluster. I run it through the web UI.
>> The jar file submitted does not contain the hbase-site.xml file.
>>
>> ----- Original meddelelse -----
>>
>> *Fra:* Flavio Pompermaier <pompermaier@okkam.it>
>> *Til:* user <user@flink.apache.org>
>> *Dato:* Ons, 11. maj 2016 09:36
>>
>> *Emne:* Re: HBase write problem
>>
>> Do you run the job from your IDE or from the cluster?
>>
>> On Wed, May 11, 2016 at 9:22 AM, Palle <palle@sport.dk> wrote:
>>
>>> Thanks for the response, but I don't think the problem is the classpath
>>> - hbase-site.xml should be added. This is what it looks like (hbase conf is
>>> added at the end):
>>>
>>> 2016-05-11 09:16:45,831 INFO  org.apache.zookeeper.ZooKeeper
>>>                    - Client
>>> environment:java.class.path=C:\systems\packages\flink-1.0.2\lib\flink-dist_2.11-1.0.2.jar;C:\systems\packages\flink-1.0.2\lib\flink-python_2.11-1.0.2.jar;C:\systems\packages\flink-1.0.2\lib\guava-11.0.2.jar;C:\systems\packages\flink-1.0.2\lib\hbase-annotations-1.2.1-tests.jar;C:\systems\packages\flink-1.0.2\lib\hbase-annotations-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-client-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-common-1.2.1-tests.jar;C:\systems\packages\flink-1.0.2\lib\hbase-common-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-examples-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-external-blockcache-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-hadoop-compat-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-hadoop2-compat-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-it-1.2.1-tests.jar;C:\systems\packages\flink-1.0.2\lib\hbase-it-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-prefix-tree-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-procedure-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-protocol-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-resource-bundle-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-rest-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-server-1.2.1-tests.jar;C:\systems\packages\flink-1.0.2\lib\hbase-server-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-shell-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\hbase-thrift-1.2.1.jar;C:\systems\packages\flink-1.0.2\lib\log4j-1.2.17.jar;C:\systems\packages\flink-1.0.2\lib\slf4j-log4j12-1.7.7.jar;C:\systems\master_flink\bin;C:\systems\packages\flink-1.0.2\lib;;C:\systems\packages\hbase-1.2.1\lib;C:\systems\hbase\conf;C:\systems\hbase\conf\hbase-site.xml;
>>>
>>> 2016-05-11 09:16:45,831 INFO  org.apache.zookeeper.ZooKeeper
>>>                    - Client
>>> environment:java.library.path=C:\systems\packages\jre-1.8.0_74_x64\bin;C:\Windows\Sun\Java\bin;C:\Windows\system32;C:\Windows;C:\systems\master_flink\bin;C:\systems\packages\appsync-1.0.6\bin;C:\systems\packages\flink-1.0.2\bin;C:\systems\packages\jre-1.8.0_74_x64\bin;C:\Windows\system32;C:\Windows;C:\Windows\System32\Wbem;C:\Windows\System32\WindowsPowerShell\v1.0\;C:\Program
>>> Files\Microsoft SQL Server\120\DTS\Binn\;C:\Program Files\Microsoft SQL
>>> Server\Client SDK\ODBC\110\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL
>>> Server\120\Tools\Binn\;C:\Program Files\Microsoft SQL
>>> Server\120\Tools\Binn\;C:\Program Files (x86)\Microsoft SQL
>>> Server\120\Tools\Binn\ManagementStudio\;C:\Program Files (x86)\Microsoft
>>> SQL
>>> Server\120\DTS\Binn\;C:\systems\packages\hadoop-2.7.2\bin;C:\systems\packages\jdk-1.8.0_74_x64\bin;C:\systems\packages\apache-maven-3.3.9\bin;C:\systems\packages\protoc-2.5.0-win32\;C:\systems\packages\cygwin64\bin\;C:\systems\packages\cmake-3.5.2-win32-x86\bin;C:\Program
>>> Files\Microsoft Windows Performance
>>> Toolkit\;C:\systems\packages\perl-5.6.0-win\bin;C:\systems\hbase\conf;.
>>>
>>> ----- Original meddelelse -----
>>>
>>> *Fra:* Flavio Pompermaier <pompermaier@okkam.it>
>>> *Til:* user <user@flink.apache.org>
>>> *Dato:* Ons, 11. maj 2016 00:05
>>> *Emne:* Re: HBase write problem
>>>
>>>
>>> Do you have the hbase-site.xml available in the classpath?
>>> On 10 May 2016 23:10, "Palle" <palle@sport.dk> wrote:
>>>
>>>> HBase write problem
>>>>
>>>> Hi all.
>>>>
>>>> I have a problem writing to HBase.
>>>>
>>>> I am using a slightly modified example of this class to proof the
>>>> concept:
>>>>
>>>> https://github.com/apache/flink/blob/master/flink-batch-connectors/flink-hbase/src/test/java/org/apache/flink/addons/hbase/example/HBaseWriteExample.java
>>>>
>>>> However all the HBase-specific stuff is exactly the same as in the
>>>> HBaseWriteExample.
>>>>
>>>> The problem I see is that the job never completes (been running for
>>>> more than an hour now) and it is only 13 key/value pairs that is to be
>>>> written to HBase :-)
>>>> I have tested the map/reduce stuff works if I replace the HBase
>>>> connection stuff with just a write to a text file - works OK. I have also
>>>> tested that I can insert data in HBase from a similar Hadoop MapReduce job.
>>>>
>>>> Here is the part of the code where I guess the problem is:
>>>>
>>>>       @Override
>>>>       public Tuple2<Text, Mutation> map(Tuple2<String, Integer>
t)
>>>> throws Exception {
>>>>         LOG.info("Tuple2 map() called");
>>>>         reuse.f0 = new Text(t.f0);
>>>>         Put put = new Put(t.f0.getBytes());
>>>>         put.add(MasterConstants.CF_SOME, MasterConstants.COUNT,
>>>> Bytes.toBytes(t.f1));
>>>>         reuse.f1 = put;
>>>>         return reuse;
>>>>       }
>>>>     }).output(new HadoopOutputFormat<Text, Mutation>(new
>>>> TableOutputFormat<Text>(), job));
>>>>
>>>>     env.execute("Flink HBase Event Count Hello World Test");
>>>>
>>>> This code matches the code in the HBaseWriteExample.java I should think.
>>>>
>>>> The "Tuple2" log line I see exactly the 13 times I expect, and the last
>>>> log line I see is this:
>>>> 2016-05-10 21:48:42,715 INFO
>>>> org.apache.hadoop.hbase.mapreduce.TableOutputFormat           - Created
>>>> table instance for event_type_count
>>>>
>>>> Any suggestions to what the problem could be?
>>>>
>>>> Thanks,
>>>> Palle
>>>
>>>
>>>
>>>
>>
>>
>
>
>

Mime
View raw message