hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Clum, Josh" <josh.c...@merck.com>
Subject Re: BytesWritable
Date Mon, 29 Jul 2013 16:06:14 GMT
Thank you for the reply. I fixed it by using the stored as clause in my hive query. Everything
is good.

From: Roberto Congiu <roberto.congiu@openx.com<mailto:roberto.congiu@openx.com>>
Reply-To: "user@hive.apache.org<mailto:user@hive.apache.org>" <user@hive.apache.org<mailto:user@hive.apache.org>>
Date: Monday, July 29, 2013 11:33 AM
To: "user@hive.apache.org<mailto:user@hive.apache.org>" <user@hive.apache.org<mailto:user@hive.apache.org>>
Subject: Re: BytesWritable

Hi Josh,
it doesn't have to. However, your table does not specify any input format, which makes Hive
use the default HiveIgnoreKeyTextOutputFormat (see http://grepcode.com/file/repository.cloudera.com/content/repositories/releases/org.apache.hadoop.hive/hive-exec/0.7.1-cdh3u1/org/apache/hadoop/hive/ql/io/HiveIgnoreKeyTextOutputFormat.java).

What's the bigger picture, meaning, what is the actual format you want to write down to disk
?

If you're storing it in text format, your SerDe's serialize method has to return either a
Text or a BytesWritable, as you can see in the code linked earlier. It looks like it's returning
a DocumentWritable object, which HiveIgnoreKeyTextOutputFormat does not know how to handle.

R.


On Mon, Jul 29, 2013 at 5:42 AM, Clum, Josh <josh.clum@merck.com<mailto:josh.clum@merck.com>>
wrote:
Hi,

I wrote a custom serde that stores takes data and stores it in objects called DocumentWritable
that inherits from Writable. When I try to run code to insert into the table that uses my
serde, I get an error saying that DocumentWritable cannot be cast to BytesWritable. Does my
DocumentWritable need to inherit from org.apache.hadoop.io.BytesWritable?

I'm creating a table using a custom serde that I wrote:

CREATE TABLE avro_test (
    NOTEBOOK_ID STRING,
   …
    DLC_XID STRING
  )
ROW FORMAT SERDE 'com.merck.huron.hive.LuceneSerde';

When I try to insert data using a select statement I am storing it into a class I have called
DocumentWritable:

INSERT OVERWRITE TABLE avro_test SELECT * FROM avro

DocumentWritable inherits from Writable. When I do the select and my serde code gets executed
I get the following error:

Error: java.lang.RuntimeException: org.apache.hadoop.hive.ql.metadata.HiveException: Hive
Runtime Error while processing row {"notebook_id":"1","stuff_id":"1","person_id":"0","person_role_id":"1","reference":"NO
DOC","lastname":"OMITTED","firstinitial":"TBD","source_id":null,"source_name":"MRL SYNTHESIS","notebook":null,"pg":null,"firstname":null,"middlename":null,"site":null,"mail":null,"dlc_xid":null}
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)
at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:54)
at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:429)
at org.apache.hadoop.mapred.MapTask.run(MapTask.java:341)
at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:158)
at java.security.AccessController.doPrivileged(Native Method)
at javax.security.auth.Subject.doAs(Subject.java:415)
at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1441)
at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:153)
Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error while processing
row {"notebook_id":"1","stuff_id":"1","person_id":"0","person_role_id":"1","reference":"NO
DOC","lastname":"OMITTED","firstinitial":"TBD","source_id":null,"source_name":"MRL SYNTHESIS","notebook":null,"pg":null,"firstname":null,"middlename":null,"site":null,"mail":null,"dlc_xid":null}
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:565)
at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)
... 8 more
Caused by: java.lang.ClassCastException: com.comp.proj.indexing.DocumentWritable cannot be
cast to org.apache.hadoop.io.BytesWritable
at org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat$1.write(HiveIgnoreKeyTextOutputFormat.java:90)
at org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:606)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83)
at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)
at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)
at org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:546)
... 9 more


Notice:  This e-mail message, together with any attachments, contains
information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station,
New Jersey, USA 08889), and/or its affiliates Direct contact information
for affiliates is available at
http://www.merck.com/contact/contacts.html) that may be confidential,
proprietary copyrighted and/or legally privileged. It is intended solely
for the use of the individual or entity named on this message. If you are
not the intended recipient, and have received this message in error,
please notify us immediately by reply e-mail and then delete it from
your system.



--
----------------------------------------------------------
Good judgement comes with experience.
Experience comes with bad judgement.
----------------------------------------------------------
Roberto Congiu - Data Engineer - OpenX
tel: +1 626 466 1141
Notice:  This e-mail message, together with any attachments, contains
information of Merck & Co., Inc. (One Merck Drive, Whitehouse Station,
New Jersey, USA 08889), and/or its affiliates Direct contact information
for affiliates is available at 
http://www.merck.com/contact/contacts.html) that may be confidential,
proprietary copyrighted and/or legally privileged. It is intended solely
for the use of the individual or entity named on this message. If you are
not the intended recipient, and have received this message in error,
please notify us immediately by reply e-mail and then delete it from 
your system.

Mime
View raw message