hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Eugene Koifman <ekoif...@hortonworks.com>
Subject Re: Storm hive bolt
Date Mon, 27 Mar 2017 15:05:24 GMT
https://community.hortonworks.com/questions/59681/puthivestreaming-nifi-processor-various-errors.html
has 
2016-10-03 23:40:24,322 ERROR [pool-5-thread-114]: metastore.RetryingHMSHandler (RetryingHMSHandler.java:invokeInternal(195))
- java.lang.IllegalStateException: Unexpected DataOperationType: UNSET agentInfo=Unknown txnid:98201

I don’t see this in the stack trace below but if you are seeing this, I think  you need
to recompile Storm bolt since it is an uber jar that includes some Hive classes.
Based on the error above it is using old classes (from before HDP 2.5).

Eugene



On 3/26/17, 1:20 PM, "Marcin Kasiński" <marcin.kasinski@gmail.com> wrote:

    Hello.
    
    I have problem with Storm hive bolt.
    When I try to save data to hiev I get error Unable to acquire lock
    (storm logs below)
    
    
    I have very simple application (save simple data to hive ).
    
    It works with HDP 2.4 (Apache Hive 1.2.1 and Apache Storm 0.10.0)
    
    I switched to  HDP 2.5 (Apache Hive 1.2.1 and Apache Storm 1.0.1)
    
    It stopped working.
    
    I saw simillar error here:
    
    https://community.hortonworks.com/questions/59681/puthivestreaming-nifi-processor-various-errors.html
    
    Ther are saying that there is a issue for Hive Streaming between HDF
    2.0 and HDP 2.5
    
    I Ilike HDP 2.5 .
    
    My question is:
    
    Do you know how I can solve this problem ?
    ... or the only way is to switch back to HDP 2.4 ?
    
    Storm logs below:
    
    
    org.apache.storm.hive.common.HiveWriter$ConnectFailure: Failed
    connecting to EndPoint {metaStoreUri='thrift://ambari.local:9083',
    database='default', table='stock_prices', partitionVals=[Marcin] }
    at org.apache.storm.hive.common.HiveWriter.<init>(HiveWriter.java:80)
    ~[stormjar.jar:?]
    at org.apache.storm.hive.common.HiveUtils.makeHiveWriter(HiveUtils.java:50)
    ~[stormjar.jar:?]
    at org.apache.storm.hive.bolt.HiveBolt.getOrCreateWriter(HiveBolt.java:259)
    ~[stormjar.jar:?]
    at org.apache.storm.hive.bolt.HiveBolt.execute(HiveBolt.java:112)
    [stormjar.jar:?]
    at org.apache.storm.daemon.executor$fn__9362$tuple_action_fn__9364.invoke(executor.clj:734)
    [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
    at org.apache.storm.daemon.executor$mk_task_receiver$fn__9283.invoke(executor.clj:466)
    [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
    at org.apache.storm.disruptor$clojure_handler$reify__8796.onEvent(disruptor.clj:40)
    [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
    at org.apache.storm.utils.DisruptorQueue.consumeBatchToCursor(DisruptorQueue.java:451)
    [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
    at org.apache.storm.utils.DisruptorQueue.consumeBatchWhenAvailable(DisruptorQueue.java:430)
    [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
    at org.apache.storm.disruptor$consume_batch_when_available.invoke(disruptor.clj:73)
    [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
    at org.apache.storm.daemon.executor$fn__9362$fn__9375$fn__9428.invoke(executor.clj:853)
    [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
    at org.apache.storm.util$async_loop$fn__656.invoke(util.clj:484)
    [storm-core-1.0.1.2.5.0.0-1245.jar:1.0.1.2.5.0.0-1245]
    at clojure.lang.AFn.run(AFn.java:22) [clojure-1.7.0.jar:?]
    at java.lang.Thread.run(Thread.java:745) [?:1.8.0_77]
    Caused by: org.apache.storm.hive.common.HiveWriter$TxnBatchFailure:
    Failed acquiring Transaction Batch from EndPoint:
    {metaStoreUri='thrift://ambari.local:9083', database='default',
    table='stock_prices', partitionVals=[Marcin] }
    at org.apache.storm.hive.common.HiveWriter.nextTxnBatch(HiveWriter.java:264)
    ~[stormjar.jar:?]
    at org.apache.storm.hive.common.HiveWriter.<init>(HiveWriter.java:72)
    ~[stormjar.jar:?]
    ... 13 more
    Caused by: org.apache.hive.hcatalog.streaming.TransactionError: Unable
    to acquire lock on {metaStoreUri='thrift://ambari.local:9083',
    database='default', table='stock_prices', partitionVals=[Marcin] }
    at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:575)
    ~[stormjar.jar:?]
    at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java:544)
    ~[stormjar.jar:?]
    at org.apache.storm.hive.common.HiveWriter.nextTxnBatch(HiveWriter.java:259)
    ~[stormjar.jar:?]
    at org.apache.storm.hive.common.HiveWriter.<init>(HiveWriter.java:72)
    ~[stormjar.jar:?]
    ... 13 more
    Caused by: org.apache.thrift.transport.TTransportException
    at org.apache.thrift.transport.TIOStreamTransport.read(TIOStreamTransport.java:132)
    ~[stormjar.jar:?]
    at org.apache.thrift.transport.TTransport.readAll(TTransport.java:84)
    ~[stormjar.jar:?]
    at org.apache.thrift.protocol.TBinaryProtocol.readAll(TBinaryProtocol.java:378)
    ~[stormjar.jar:?]
    at org.apache.thrift.protocol.TBinaryProtocol.readI32(TBinaryProtocol.java:297)
    ~[stormjar.jar:?]
    at org.apache.thrift.protocol.TBinaryProtocol.readMessageBegin(TBinaryProtocol.java:204)
    ~[stormjar.jar:?]
    at org.apache.thrift.TServiceClient.receiveBase(TServiceClient.java:69)
    ~[stormjar.jar:?]
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.recv_lock(ThriftHiveMetastore.java:3781)
    ~[stormjar.jar:?]
    at org.apache.hadoop.hive.metastore.api.ThriftHiveMetastore$Client.lock(ThriftHiveMetastore.java:3768)
    ~[stormjar.jar:?]
    at org.apache.hadoop.hive.metastore.HiveMetaStoreClient.lock(HiveMetaStoreClient.java:1736)
    ~[stormjar.jar:?]
    at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransactionImpl(HiveEndPoint.java:570)
    ~[stormjar.jar:?]
    at org.apache.hive.hcatalog.streaming.HiveEndPoint$TransactionBatchImpl.beginNextTransaction(HiveEndPoint.java:544)
    ~[stormjar.jar:?]
    at org.apache.storm.hive.common.HiveWriter.nextTxnBatch(HiveWriter.java:259)
    ~[stormjar.jar:?]
    at org.apache.storm.hive.common.HiveWriter.<init>(HiveWriter.java:72)
    ~[stormjar.jar:?]
    ... 13 more
    2017-03-26 21:43:05.112 h.metastore [INFO] Trying to connect to
    metastore with URI thrift://ambari.local:9083
    
    
    
    pozdrawiam
    Marcin Kasiński
    http://itzone.pl
    
    

Mime
View raw message