flink-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lydia Ickler <ickle...@googlemail.com>
Subject Re: HBase issue
Date Thu, 24 Sep 2015 07:52:53 GMT
Hi I tried that but unfortunately it still gets stuck at the second split.

Can it be that I have set something in my configurations wrong? In Hadoop? Or Flink?

The strange thing is that the HBaseWriteExample works great!

Best regards,
Lydia


> Am 23.09.2015 um 17:40 schrieb Maximilian Michels <mxm@apache.org>:
> 
> In the issue, it states that it should be sufficient to append the hbase-protocol.jar
file to the Hadoop classpath. Flink respects the Hadoop classpath and will append it to its
own classpath upon launching a cluster.
> 
> To do that, you need to modify the classpath with one of the commands below. Note that
this has to be performed on all cluster nodes.
> 
> export HADOOP_CLASSPATH="${HADOOP_CLASSPATH}:/path/to/hbase-protocol.jar"
> export HADOOP_CLASSPATH="${HADOOP_CLASSPATH}:$(hbase mapredcp)"
> export HADOOP_CLASSPATH="${HADOOP_CLASSPATH}:$(hbase classpath)"
> 
> Alternatively, you can build a fat jar from your project with the missing dependency.
Flink will then automatically distribute the jar file upon job submission. Just add this Maven
dependency to your fat-jar pom:
> 
> <dependency>
>     <groupId>org.apache.hbase</groupId>
>     <artifactId>hbase-protocol</artifactId>
>     <version>1.1.2</version>
> </dependency>
> 
> Let me know if any of the two approaches work for you. After all, this is a workaround
because of an HBase optimzation..
> 
> Cheers,
> Max
> 
> 
> 
> On Wed, Sep 23, 2015 at 11:16 AM, Aljoscha Krettek <aljoscha@apache.org <mailto:aljoscha@apache.org>>
wrote:
> It might me that this is causing the problem: https://issues.apache.org/jira/browse/HBASE-10304
<https://issues.apache.org/jira/browse/HBASE-10304>
> 
> In your log I see the same exception. Anyone has any idea what we could do about this?
> 
> 
> On Tue, 22 Sep 2015 at 22:40 Lydia Ickler <icklerly@googlemail.com <mailto:icklerly@googlemail.com>>
wrote:
> Hi, 
> 
> I am trying to get the HBaseReadExample to run. I have filled a table with the HBaseWriteExample
and purposely split it over 3 regions.
> Now when I try to read from it the first split seems to be scanned (170 rows) fine and
after that the Connections of Zookeeper and RCP are suddenly closed down.
> 
> Does anyone has an idea why this is happening?
> 
> Best regards,
> Lydia 
> 
> 
> 22:28:10,178 DEBUG org.apache.flink.runtime.operators.DataSourceTask             - Opening
input split Locatable Split (2) at [grips5:60020]:  DataSource (at createInput(ExecutionEnvironment.java:502)
(org.apache.flink.HBaseReadExample$1)) (1/1)
> 22:28:10,178 INFO  org.apache.flink.addons.hbase.TableInputFormat                - opening
split [2|[grips5:60020]|aaaaaaaa|-]
> 22:28:10,189 DEBUG org.apache.zookeeper.ClientCnxn                               - Reading
reply sessionid:0x24ff6a96ecd000a, packet:: clientPath:null serverPath:null finished:false
header:: 3,4  replyHeader:: 3,51539607639,0  request:: '/hbase/meta-region-server,F  response::
#ffffffff0001a726567696f6e7365727665723a363030$
> 22:28:10,202 DEBUG org.apache.zookeeper.ClientCnxn                               - Reading
reply sessionid:0x24ff6a96ecd000a, packet:: clientPath:null serverPath:null finished:false
header:: 4,4  replyHeader:: 4,51539607639,0  request:: '/hbase/meta-region-server,F  response::
#ffffffff0001a726567696f6e7365727665723a363030$
> 22:28:10,211 DEBUG LocalActorRefProvider(akka://flink <>)                     
     - resolve of path sequence [/temp/$b] failed
> 22:28:10,233 DEBUG org.apache.hadoop.hbase.util.ByteStringer                     - Failed
to classload HBaseZeroCopyByteString: java.lang.IllegalAccessError: class com.google.protobuf.HBaseZeroCopyByteString
cannot access its superclass com.google.protobuf.LiteralByteString
> 22:28:10,358 DEBUG org.apache.hadoop.ipc.RpcClient                               - Use
SIMPLE authentication for service ClientService, sasl=false
> 22:28:10,370 DEBUG org.apache.hadoop.ipc.RpcClient                               - Connecting
to grips1/130.73.20.14:60020 <http://130.73.20.14:60020/>
> 22:28:10,380 DEBUG org.apache.hadoop.ipc.RpcClient                               - IPC
Client (2145423150 <tel:%282145423150>) connection to grips1/130.73.20.14:60020 <http://130.73.20.14:60020/>
from hduser: starting, connections 1
> 22:28:10,394 DEBUG org.apache.hadoop.ipc.RpcClient                               - IPC
Client (2145423150 <tel:%282145423150>) connection to grips1/130.73.20.14:60020 <http://130.73.20.14:60020/>
from hduser: got response header call_id: 0, totalSize: 469 bytes
> 22:28:10,397 DEBUG org.apache.hadoop.ipc.RpcClient                               - IPC
Client (2145423150 <tel:%282145423150>) connection to grips1/130.73.20.14:60020 <http://130.73.20.14:60020/>
from hduser: wrote request header call_id: 0 method_name: "Get" request_param: true
> 22:28:10,413 DEBUG org.apache.zookeeper.ClientCnxn                               - Reading
reply sessionid:0x24ff6a96ecd000a, packet:: clientPath:null serverPath:null finished:false
header:: 5,4  replyHeader:: 5,51539607639,0  request:: '/hbase/meta-region-server,F  response::
#ffffffff0001a726567696f6e7365727665723a363030$
> 22:28:10,424 DEBUG org.apache.hadoop.ipc.RpcClient                               - IPC
Client (2145423150 <tel:%282145423150>) connection to grips1/130.73.20.14:60020 <http://130.73.20.14:60020/>
from hduser: wrote request header call_id: 1 method_name: "Scan" request_param: true priority:
100
> 22:28:10,426 DEBUG org.apache.hadoop.ipc.RpcClient                               - IPC
Client (2145423150 <tel:%282145423150>) connection to grips1/130.73.20.14:60020 <http://130.73.20.14:60020/>
from hduser: got response header call_id: 1 cell_block_meta { length: 480 }, totalSize: 497
bytes
> 22:28:10,432 DEBUG org.apache.hadoop.hbase.client.ClientSmallScanner             - Finished
with small scan at {ENCODED => 1588230740, NAME => 'hbase:meta,,1', STARTKEY => '',
ENDKEY => ''}
> 22:28:10,434 DEBUG org.apache.hadoop.ipc.RpcClient                               - Use
SIMPLE authentication for service ClientService, sasl=false
> 22:28:10,434 DEBUG org.apache.hadoop.ipc.RpcClient                               - Connecting
to grips5/130.73.20.16:60020 <http://130.73.20.16:60020/>
> 22:28:10,435 DEBUG org.apache.hadoop.ipc.RpcClient                               - IPC
Client (2145423150 <tel:%282145423150>) connection to grips5/130.73.20.16:60020 <http://130.73.20.16:60020/>
from hduser: wrote request header call_id: 2 method_name: "Scan" request_param: true
> 22:28:10,436 DEBUG org.apache.hadoop.ipc.RpcClient                               - IPC
Client (2145423150 <tel:%282145423150>) connection to grips5/130.73.20.16:60020 <http://130.73.20.16:60020/>
from hduser: starting, connections 2
> 22:28:10,437 DEBUG org.apache.hadoop.ipc.RpcClient                               - IPC
Client (2145423150 <tel:%282145423150>) connection to grips5/130.73.20.16:60020 <http://130.73.20.16:60020/>
from hduser: got response header call_id: 2, totalSize: 12 bytes
> 22:28:10,438 DEBUG org.apache.flink.runtime.operators.DataSourceTask             - Starting
to read input from split Locatable Split (2) at [grips5:60020]:  DataSource (at createInput(ExecutionEnvironment.java:502)
(org.apache.flink.HBaseReadExample$1)) (1/1)
> 22:28:10,438 DEBUG org.apache.hadoop.ipc.RpcClient                               - IPC
Client (2145423150 <tel:%282145423150>) connection to grips5/130.73.20.16:60020 <http://130.73.20.16:60020/>
from hduser: wrote request header call_id: 3 method_name: "Scan" request_param: true
> 22:28:10,457 DEBUG org.apache.hadoop.ipc.RpcClient                               - IPC
Client (2145423150 <tel:%282145423150>) connection to grips5/130.73.20.16:60020 <http://130.73.20.16:60020/>
from hduser: got response header call_id: 3 cell_block_meta { length: 4679 }, totalSize: 4899
bytes
> 22:28:10,476 DEBUG org.apache.hadoop.ipc.RpcClient                               - IPC
Client (2145423150 <tel:%282145423150>) connection to grips5/130.73.20.16:60020 <http://130.73.20.16:60020/>
from hduser: wrote request header call_id: 4 method_name: "Scan" request_param: true
> 22:28:10,480 DEBUG org.apache.hadoop.ipc.RpcClient                               - IPC
Client (2145423150 <tel:%282145423150>) connection to grips5/130.73.20.16:60020 <http://130.73.20.16:60020/>
from hduser: got response header call_id: 4 cell_block_meta { length: 3306 }, totalSize: 3466
bytes
> 22:28:10,482 DEBUG org.apache.hadoop.ipc.RpcClient                               - IPC
Client (2145423150 <tel:%282145423150>) connection to grips5/130.73.20.16:60020 <http://130.73.20.16:60020/>
from hduser: got response header call_id: 5, totalSize: 8 bytes
> 22:28:10,482 DEBUG org.apache.hadoop.ipc.RpcClient                               - IPC
Client (2145423150 <tel:%282145423150>) connection to grips5/130.73.20.16:60020 <http://130.73.20.16:60020/>
from hduser: wrote request header call_id: 5 method_name: "Scan" request_param: true
> 22:28:10,487 DEBUG org.apache.flink.runtime.operators.DataSourceTask             - Closing
input split Locatable Split (2) at [grips5:60020]:  DataSource (at createInput(ExecutionEnvironment.java:502)
(org.apache.flink.HBaseReadExample$1)) (1/1)
> 22:28:10,489 INFO  org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation
 - Closing zookeeper sessionid=0x24ff6a96ecd000a
> 22:28:10,489 DEBUG org.apache.zookeeper.ZooKeeper                                - Closing
session: 0x24ff6a96ecd000a
> 22:28:10,489 DEBUG org.apache.zookeeper.ClientCnxn                               - Closing
client for session: 0x24ff6a96ecd000a
> 22:28:10,499 DEBUG org.apache.zookeeper.ClientCnxn                               - Reading
reply sessionid:0x24ff6a96ecd000a, packet:: clientPath:null serverPath:null finished:false
header:: 6,-11  replyHeader:: 6,51539607640,0  request:: null response:: null
> 22:28:10,499 DEBUG org.apache.zookeeper.ClientCnxn                               - Disconnecting
client for session: 0x24ff6a96ecd000a
> 22:28:10,499 INFO  org.apache.zookeeper.ClientCnxn                               - EventThread
shut down
> 22:28:10,499 INFO  org.apache.zookeeper.ZooKeeper                                - Session:
0x24ff6a96ecd000a closed
> 22:28:10,499 DEBUG org.apache.hadoop.ipc.RpcClient                               - Stopping
rpc client
> 22:28:10,501 DEBUG org.apache.hadoop.ipc.RpcClient                               - IPC
Client (2145423150 <tel:%282145423150>) connection to grips1/130.73.20.14:60020 <http://130.73.20.14:60020/>
from hduser: closed
> 22:28:10,502 DEBUG org.apache.hadoop.ipc.RpcClient                               - IPC
Client (2145423150 <tel:%282145423150>) connection to grips1/130.73.20.14:60020 <http://130.73.20.14:60020/>
from hduser: stopped, connections 0
> 22:28:10,502 DEBUG org.apache.hadoop.ipc.RpcClient                               - IPC
Client (2145423150 <tel:%282145423150>) connection to grips5/130.73.20.16:60020 <http://130.73.20.16:60020/>
from hduser: closed
> 22:28:10,502 DEBUG org.apache.hadoop.ipc.RpcClient                               - IPC
Client (2145423150 <tel:%282145423150>) connection to grips5/130.73.20.16:60020 <http://130.73.20.16:60020/>
from hduser: stopped, connections 0
> 22:28:10,502 INFO  org.apache.flink.addons.hbase.TableInputFormat                - Closing
split (scanned 170 rows)
> 22:28:10,508 DEBUG org.apache.flink.runtime.operators.DataSourceTask             - Opening
input split Locatable Split (1) at [grips4:60020]:  DataSource (at createInput(ExecutionEnvironment.java:502)
(org.apache.flink.HBaseReadExample$1)) (1/1)
> 22:28:10,509 INFO  org.apache.flink.addons.hbase.TableInputFormat                - opening
split [1|[grips4:60020]|55555555|aaaaaaaa]
> 22:28:11,380 DEBUG org.apache.flink.runtime.taskmanager.TaskManager              - Received
message SendHeartbeat at akka://flink/user/taskmanager <> from Actor[akka://flink/deadLetters
<>].
> 22:28:11,380 DEBUG org.apache.flink.runtime.taskmanager.TaskManager              - Sending
heartbeat to JobManager
> 
> 


Mime
View raw message