gora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Alparslan Avcı (JIRA) <j...@apache.org>
Subject [jira] [Commented] (GORA-375) Upgrade HBase to 0.98
Date Wed, 24 Dec 2014 08:22:14 GMT

    [ https://issues.apache.org/jira/browse/GORA-375?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14258097#comment-14258097
] 

Alparslan Avcı commented on GORA-375:
-------------------------------------

[~talat], I've tested your patch but when running hadoop jobs, I could not see the job progress
of map jobs. You can see the log below:

Normal run:
{code}
14/12/23 17:04:49 INFO mapred.JobClient:  map 0% reduce 0%
14/12/23 17:04:53 INFO mapred.MapTask: Starting flush of map output
14/12/23 17:04:53 INFO mapred.MapTask: Finished spill 0
14/12/23 17:04:53 INFO mapred.Task: Task:attempt_local_0001_m_000000_0 is done. And is in
the process of commiting
14/12/23 17:04:54 INFO mapred.LocalJobRunner: 
14/12/23 17:04:54 INFO mapred.LocalJobRunner: 
14/12/23 17:04:54 INFO mapred.Task: Task 'attempt_local_0001_m_000000_0' done.
14/12/23 17:04:54 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@30fd4204
14/12/23 17:04:54 INFO mapred.LocalJobRunner: 
14/12/23 17:04:54 INFO mapred.Merger: Merging 1 sorted segments
14/12/23 17:04:54 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of
total size: 532529 bytes
14/12/23 17:04:54 INFO mapred.LocalJobRunner: 
14/12/23 17:04:54 ERROR store.HBaseStore: KeyClass in gora-hbase-mapping is not the same as
the one in the databean.
14/12/23 17:04:54 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181
sessionTimeout=180000 watcher=catalogtracker-on-org.apache.hadoop.hbase.client.HConnectionManager$HConnectionImplementation@436daf20
14/12/23 17:04:54 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181.
Will not attempt to authenticate using SASL (unknown error)
14/12/23 17:04:54 INFO zookeeper.RecoverableZooKeeper: The identifier of this process is 15793@alparslan
14/12/23 17:04:54 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181,
initiating session
14/12/23 17:04:54 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181,
sessionid = 0x14a77a9d8b90017, negotiated timeout = 40000
14/12/23 17:04:54 INFO zookeeper.ZooKeeper: Session: 0x14a77a9d8b90017 closed
14/12/23 17:04:54 INFO zookeeper.ClientCnxn: EventThread shut down
14/12/23 17:04:54 INFO mapreduce.GoraRecordWriter: gora.buffer.write.limit = 10000
14/12/23 17:04:54 INFO mapred.Task: Task:attempt_local_0001_r_000000_0 is done. And is in
the process of commiting
14/12/23 17:04:55 INFO mapred.JobClient:  map 100% reduce 0%
14/12/23 17:04:57 INFO mapred.LocalJobRunner: reduce > reduce
14/12/23 17:04:57 INFO mapred.Task: Task 'attempt_local_0001_r_000000_0' done.
14/12/23 17:04:57 WARN mapred.FileOutputCommitter: Output path is null in cleanup
14/12/23 17:04:58 INFO mapred.JobClient:  map 100% reduce 100%
14/12/23 17:04:58 INFO mapred.JobClient: Job complete: job_local_0001
{code}

When your patch applied:
{code}
14/12/23 17:11:23 INFO mapred.JobClient:  map 0% reduce 0%
14/12/23 17:11:24 INFO zookeeper.ZooKeeper: Session: 0x14a77b10921001e closed
14/12/23 17:11:24 INFO zookeeper.ClientCnxn: EventThread shut down
14/12/23 17:11:24 INFO zookeeper.RecoverableZooKeeper: Process identifier=catalogtracker-on-hconnection-0x50344c81
connecting to ZooKeeper ensemble=localhost:2181
14/12/23 17:11:24 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181
sessionTimeout=90000 watcher=catalogtracker-on-hconnection-0x50344c81, quorum=localhost:2181,
baseZNode=/hbase
14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181.
Will not attempt to authenticate using SASL (unknown error)
14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181,
initiating session
14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181,
sessionid = 0x14a77b10921001f, negotiated timeout = 40000
14/12/23 17:11:24 INFO zookeeper.ZooKeeper: Session: 0x14a77b10921001f closed
14/12/23 17:11:24 INFO zookeeper.ClientCnxn: EventThread shut down
14/12/23 17:11:24 INFO zookeeper.RecoverableZooKeeper: Process identifier=catalogtracker-on-hconnection-0x50344c81
connecting to ZooKeeper ensemble=localhost:2181
14/12/23 17:11:24 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181
sessionTimeout=90000 watcher=catalogtracker-on-hconnection-0x50344c81, quorum=localhost:2181,
baseZNode=/hbase
14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181.
Will not attempt to authenticate using SASL (unknown error)
14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181,
initiating session
14/12/23 17:11:24 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181,
sessionid = 0x14a77b109210020, negotiated timeout = 40000
14/12/23 17:11:24 INFO zookeeper.ZooKeeper: Session: 0x14a77b109210020 closed
14/12/23 17:11:24 INFO zookeeper.ClientCnxn: EventThread shut down
14/12/23 17:11:24 INFO mapred.MapTask: Processing split: org.apache.gora.mapreduce.GoraInputSplit@40b86a70
14/12/23 17:11:24 INFO mapreduce.GoraRecordReader: gora.buffer.read.limit = 10000
14/12/23 17:11:24 INFO mapred.MapTask: io.sort.mb = 100
14/12/23 17:11:24 INFO mapred.MapTask: data buffer = 79691776/99614720
14/12/23 17:11:24 INFO mapred.MapTask: record buffer = 262144/327680
14/12/23 17:11:25 INFO mapred.MapTask: Starting flush of map output
14/12/23 17:11:25 INFO mapred.MapTask: Finished spill 0
14/12/23 17:11:25 INFO mapred.Task: Task:attempt_local139853648_0001_m_000000_0 is done. And
is in the process of commiting
14/12/23 17:11:25 INFO mapred.LocalJobRunner: 
14/12/23 17:11:25 INFO mapred.Task: Task 'attempt_local139853648_0001_m_000000_0' done.
14/12/23 17:11:25 INFO mapred.LocalJobRunner: Finishing task: attempt_local139853648_0001_m_000000_0
14/12/23 17:11:25 INFO mapred.LocalJobRunner: Map task executor complete.
14/12/23 17:11:25 INFO mapred.Task:  Using ResourceCalculatorPlugin : org.apache.hadoop.util.LinuxResourceCalculatorPlugin@3589085
14/12/23 17:11:25 INFO mapred.LocalJobRunner: 
14/12/23 17:11:25 INFO mapred.Merger: Merging 1 sorted segments
14/12/23 17:11:25 INFO mapred.Merger: Down to the last merge-pass, with 1 segments left of
total size: 532529 bytes
14/12/23 17:11:25 INFO mapred.LocalJobRunner: 
14/12/23 17:11:25 ERROR store.HBaseStore: KeyClass in gora-hbase-mapping is not the same as
the one in the databean.
14/12/23 17:11:25 INFO zookeeper.RecoverableZooKeeper: Process identifier=catalogtracker-on-hconnection-0x50344c81
connecting to ZooKeeper ensemble=localhost:2181
14/12/23 17:11:25 INFO zookeeper.ZooKeeper: Initiating client connection, connectString=localhost:2181
sessionTimeout=90000 watcher=catalogtracker-on-hconnection-0x50344c81, quorum=localhost:2181,
baseZNode=/hbase
14/12/23 17:11:25 INFO zookeeper.ClientCnxn: Opening socket connection to server localhost/127.0.0.1:2181.
Will not attempt to authenticate using SASL (unknown error)
14/12/23 17:11:25 INFO zookeeper.ClientCnxn: Socket connection established to localhost/127.0.0.1:2181,
initiating session
14/12/23 17:11:25 INFO zookeeper.ClientCnxn: Session establishment complete on server localhost/127.0.0.1:2181,
sessionid = 0x14a77b109210021, negotiated timeout = 40000
14/12/23 17:11:25 INFO zookeeper.ZooKeeper: Session: 0x14a77b109210021 closed
14/12/23 17:11:25 INFO zookeeper.ClientCnxn: EventThread shut down
14/12/23 17:11:25 INFO mapreduce.GoraRecordWriter: gora.buffer.write.limit = 10000
14/12/23 17:11:26 INFO mapred.Task: Task:attempt_local139853648_0001_r_000000_0 is done. And
is in the process of commiting
14/12/23 17:11:26 INFO mapred.LocalJobRunner: reduce > reduce
14/12/23 17:11:26 INFO mapred.Task: Task 'attempt_local139853648_0001_r_000000_0' done.
14/12/23 17:11:26 WARN mapred.FileOutputCommitter: Output path is null in cleanup
14/12/23 17:11:26 INFO mapred.JobClient:  map 0% reduce 100%
14/12/23 17:11:26 INFO mapred.JobClient: Job complete: job_local139853648_0001
{code}

Is this normal on hadoop-1.2.1? Or is there a problem caused by us?


> Upgrade HBase to 0.98
> ---------------------
>
>                 Key: GORA-375
>                 URL: https://issues.apache.org/jira/browse/GORA-375
>             Project: Apache Gora
>          Issue Type: Improvement
>          Components: gora-hbase
>    Affects Versions: 0.5
>            Reporter: Ted Yu
>            Assignee: Talat UYARER
>              Labels: hbase
>             Fix For: 0.6
>
>         Attachments: GORA-375.patch, org.apache.gora.avro.mapreduce.TestDataFileAvroStoreMapReduce.txt
>
>
> HBase 0.98 release is the current stable release.
> Gora should be built based on HBase 0.98.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message