hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Stephen Sprague <sprag...@gmail.com>
Subject Re: Textfile compression using Gzip codec
Date Thu, 06 Jun 2013 18:39:37 GMT
aha!   All's well that ends well then! :)


On Thu, Jun 6, 2013 at 9:49 AM, Sachin Sudarshana
<sachin.hadoop@gmail.com>wrote:

> Hi Stephen,
>
> Thank you for your reply.
>
> But, its the silliest error from my side. Its a typo!
>
> The codec is : org.apache.hadoop.io.compress.*GzipCodec* and not
> org.apache.hadoop.io.compress.*GZipCodec.*
> *
> *
> I regret making that mistake.
>
> Thank you,
> Sachin
>
>
> On Thu, Jun 6, 2013 at 10:07 PM, Stephen Sprague <spragues@gmail.com>wrote:
>
>> Hi Sachin,
>> LIke you say looks like something to do with the GZipCodec all right. And
>> that would make sense given your original problem.
>>
>> Yeah, one would think it'd be in there by default but for whatever reason
>> its not finding it but at least the problem is now identified.
>>
>> Now _my guess_ is that maybe your hadoop core-site.xml file might need to
>> list the codecs available under the property name:
>> "io.compression.codecs".  Can you chase that up as a possibility and let us
>> know what you find out?
>>
>>
>>
>>
>> On Thu, Jun 6, 2013 at 4:02 AM, Sachin Sudarshana <
>> sachin.hadoop@gmail.com> wrote:
>>
>>> Hi Stephen,
>>>
>>> *hive> show create table facts520_normal_text;*
>>> *OK*
>>> *CREATE  TABLE facts520_normal_text(*
>>> *  fact_key bigint,*
>>>  *  products_key int,*
>>> *  retailers_key int,*
>>> *  suppliers_key int,*
>>> *  time_key int,*
>>> *  units int)*
>>> *ROW FORMAT DELIMITED*
>>> *  FIELDS TERMINATED BY ','*
>>> *  LINES TERMINATED BY '\n'*
>>> *STORED AS INPUTFORMAT*
>>> *  'org.apache.hadoop.mapred.TextInputFormat'*
>>> *OUTPUTFORMAT*
>>> *  'org.apache.hadoop.hive.ql.io.HiveIgnoreKeyTextOutputFormat'*
>>> *LOCATION*
>>> *  'hdfs://
>>> aana1.ird.com/user/hive/warehouse/facts_520.db/facts520_normal_text'*
>>> *TBLPROPERTIES (*
>>> *  'numPartitions'='0',*
>>> *  'numFiles'='1',*
>>> *  'transient_lastDdlTime'='1369395430',*
>>> *  'numRows'='0',*
>>> *  'totalSize'='545216508',*
>>> *  'rawDataSize'='0')*
>>> *Time taken: 0.353 seconds*
>>>
>>>
>>> The syserror log shows this:
>>>
>>> *java.lang.IllegalArgumentException: Compression codec
>>> org.apache.hadoop.io.compress.GZipCodec was not found.*
>>> * at
>>> org.apache.hadoop.mapred.FileOutputFormat.getOutputCompressorClass(FileOutputFormat.java:85)
>>> *
>>> * at
>>> org.apache.hadoop.hive.ql.exec.Utilities.getFileExtension(Utilities.java:934)
>>> *
>>> * at
>>> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:469)
>>> *
>>> * at
>>> org.apache.hadoop.hive.ql.exec.FileSinkOperator.processOp(FileSinkOperator.java:543)
>>> *
>>> * at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)*
>>> * at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)*
>>> * at
>>> org.apache.hadoop.hive.ql.exec.SelectOperator.processOp(SelectOperator.java:84)
>>> *
>>> * at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)*
>>> * at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)*
>>> * at
>>> org.apache.hadoop.hive.ql.exec.TableScanOperator.processOp(TableScanOperator.java:83)
>>> *
>>> * at org.apache.hadoop.hive.ql.exec.Operator.process(Operator.java:474)*
>>> * at org.apache.hadoop.hive.ql.exec.Operator.forward(Operator.java:800)*
>>> * at
>>> org.apache.hadoop.hive.ql.exec.MapOperator.process(MapOperator.java:546)
>>> *
>>> * at org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:143)*
>>> * at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)*
>>> * at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)*
>>> * at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)*
>>> * at org.apache.hadoop.mapred.Child$4.run(Child.java:268)*
>>> * at java.security.AccessController.doPrivileged(Native Method)*
>>> * at javax.security.auth.Subject.doAs(Subject.java:415)*
>>> * at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>>> *
>>> * at org.apache.hadoop.mapred.Child.main(Child.java:262)*
>>> *Caused by: java.lang.ClassNotFoundException: Class
>>> org.apache.hadoop.io.compress.GZipCodec not found*
>>> * at
>>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1493)
>>> *
>>> * at
>>> org.apache.hadoop.mapred.FileOutputFormat.getOutputCompressorClass(FileOutputFormat.java:82)
>>> *
>>> * ... 21 more*
>>> *java.lang.IllegalArgumentException: Compression codec
>>> org.apache.hadoop.io.compress.GZipCodec was not found.*
>>> * at
>>> org.apache.hadoop.mapred.FileOutputFormat.getOutputCompressorClass(FileOutputFormat.java:85)
>>> *
>>> * at
>>> org.apache.hadoop.hive.ql.exec.Utilities.getFileExtension(Utilities.java:934)
>>> *
>>> * at
>>> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:469)
>>> *
>>> * at
>>> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:739)
>>> *
>>> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:558)*
>>> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)*
>>> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)*
>>> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)*
>>> * at
>>> org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:193)*
>>> * at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)*
>>> * at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)*
>>> * at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)*
>>> * at org.apache.hadoop.mapred.Child$4.run(Child.java:268)*
>>> * at java.security.AccessController.doPrivileged(Native Method)*
>>> * at javax.security.auth.Subject.doAs(Subject.java:415)*
>>> * at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>>> *
>>> * at org.apache.hadoop.mapred.Child.main(Child.java:262)*
>>> *Caused by: java.lang.ClassNotFoundException: Class
>>> org.apache.hadoop.io.compress.GZipCodec not found*
>>> * at
>>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1493)
>>> *
>>> * at
>>> org.apache.hadoop.mapred.FileOutputFormat.getOutputCompressorClass(FileOutputFormat.java:82)
>>> *
>>> * ... 16 more*
>>> *org.apache.hadoop.hive.ql.metadata.HiveException:
>>> java.lang.IllegalArgumentException: Compression codec
>>> org.apache.hadoop.io.compress.GZipCodec was not found.*
>>> * at
>>> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:479)
>>> *
>>> * at
>>> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:739)
>>> *
>>> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:558)*
>>> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)*
>>> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)*
>>> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)*
>>> * at
>>> org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:193)*
>>> * at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)*
>>> * at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)*
>>> * at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)*
>>> * at org.apache.hadoop.mapred.Child$4.run(Child.java:268)*
>>> * at java.security.AccessController.doPrivileged(Native Method)*
>>> * at javax.security.auth.Subject.doAs(Subject.java:415)*
>>> * at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>>> *
>>> * at org.apache.hadoop.mapred.Child.main(Child.java:262)*
>>> *Caused by: java.lang.IllegalArgumentException: Compression codec
>>> org.apache.hadoop.io.compress.GZipCodec was not found.*
>>> * at
>>> org.apache.hadoop.mapred.FileOutputFormat.getOutputCompressorClass(FileOutputFormat.java:85)
>>> *
>>> * at
>>> org.apache.hadoop.hive.ql.exec.Utilities.getFileExtension(Utilities.java:934)
>>> *
>>> * at
>>> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:469)
>>> *
>>> * ... 14 more*
>>> *Caused by: java.lang.ClassNotFoundException: Class
>>> org.apache.hadoop.io.compress.GZipCodec not found*
>>> * at
>>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1493)
>>> *
>>> * at
>>> org.apache.hadoop.mapred.FileOutputFormat.getOutputCompressorClass(FileOutputFormat.java:82)
>>> *
>>> * ... 16 more*
>>> *org.apache.hadoop.hive.ql.metadata.HiveException:
>>> java.lang.IllegalArgumentException: Compression codec
>>> org.apache.hadoop.io.compress.GZipCodec was not found.*
>>> * at
>>> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:479)
>>> *
>>> * at
>>> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:739)
>>> *
>>> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:558)*
>>> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)*
>>> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)*
>>> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)*
>>> * at
>>> org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:193)*
>>> * at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)*
>>> * at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)*
>>> * at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)*
>>> * at org.apache.hadoop.mapred.Child$4.run(Child.java:268)*
>>> * at java.security.AccessController.doPrivileged(Native Method)*
>>> * at javax.security.auth.Subject.doAs(Subject.java:415)*
>>> * at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>>> *
>>> * at org.apache.hadoop.mapred.Child.main(Child.java:262)*
>>> *Caused by: java.lang.IllegalArgumentException: Compression codec
>>> org.apache.hadoop.io.compress.GZipCodec was not found.*
>>> * at
>>> org.apache.hadoop.mapred.FileOutputFormat.getOutputCompressorClass(FileOutputFormat.java:85)
>>> *
>>> * at
>>> org.apache.hadoop.hive.ql.exec.Utilities.getFileExtension(Utilities.java:934)
>>> *
>>> * at
>>> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:469)
>>> *
>>> * ... 14 more*
>>> *Caused by: java.lang.ClassNotFoundException: Class
>>> org.apache.hadoop.io.compress.GZipCodec not found*
>>> * at
>>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1493)
>>> *
>>> * at
>>> org.apache.hadoop.mapred.FileOutputFormat.getOutputCompressorClass(FileOutputFormat.java:82)
>>> *
>>> * ... 16 more*
>>> *org.apache.hadoop.hive.ql.metadata.HiveException:
>>> java.lang.IllegalArgumentException: Compression codec
>>> org.apache.hadoop.io.compress.GZipCodec was not found.*
>>> * at
>>> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:479)
>>> *
>>> * at
>>> org.apache.hadoop.hive.ql.exec.FileSinkOperator.closeOp(FileSinkOperator.java:739)
>>> *
>>> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:558)*
>>> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)*
>>> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)*
>>> * at org.apache.hadoop.hive.ql.exec.Operator.close(Operator.java:567)*
>>> * at
>>> org.apache.hadoop.hive.ql.exec.ExecMapper.close(ExecMapper.java:193)*
>>> * at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:57)*
>>> * at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)*
>>> * at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)*
>>> * at org.apache.hadoop.mapred.Child$4.run(Child.java:268)*
>>> * at java.security.AccessController.doPrivileged(Native Method)*
>>> * at javax.security.auth.Subject.doAs(Subject.java:415)*
>>> * at
>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>>> *
>>> * at org.apache.hadoop.mapred.Child.main(Child.java:262)*
>>> *Caused by: java.lang.IllegalArgumentException: Compression codec
>>> org.apache.hadoop.io.compress.GZipCodec was not found.*
>>> * at
>>> org.apache.hadoop.mapred.FileOutputFormat.getOutputCompressorClass(FileOutputFormat.java:85)
>>> *
>>> * at
>>> org.apache.hadoop.hive.ql.exec.Utilities.getFileExtension(Utilities.java:934)
>>> *
>>> * at
>>> org.apache.hadoop.hive.ql.exec.FileSinkOperator.createBucketFiles(FileSinkOperator.java:469)
>>> *
>>> * ... 14 more*
>>> *Caused by: java.lang.ClassNotFoundException: Class
>>> org.apache.hadoop.io.compress.GZipCodec not found*
>>> * at
>>> org.apache.hadoop.conf.Configuration.getClassByName(Configuration.java:1493)
>>> *
>>> * at
>>> org.apache.hadoop.mapred.FileOutputFormat.getOutputCompressorClass(FileOutputFormat.java:82)
>>> *
>>> * ... 16 more*
>>>
>>> It says that  GZipCodec is not found.
>>> Isn't Snappy,GZip and BZip codecs available on Hadoop by default?
>>>
>>> Thank you,
>>> Sachin
>>>
>>>
>>>
>>>
>>>
>>> On Wed, Jun 5, 2013 at 11:58 PM, Stephen Sprague <spragues@gmail.com>wrote:
>>>
>>>> well...   the hiveException has the word "metadata" in it.  maybe
>>>> that's a hint or a red-herrring. :)    Let's try the following:
>>>>
>>>> 1.  show create table * facts520_normal_text;
>>>>
>>>> *
>>>> *2.  anything useful at this URL? **
>>>> http://aana1.ird.com:50030/taskdetails.jsp?jobid=job_201306051948_0010&tipid=task_201306051948_0010_m_000002or
is it just the same stack dump?
>>>>
>>>>
>>>> *
>>>>
>>>>
>>>> On Wed, Jun 5, 2013 at 3:17 AM, Sachin Sudarshana <
>>>> sachin.hadoop@gmail.com> wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I have hive 0.10 + (CDH 4.2.1 patches) installed on my cluster.
>>>>>
>>>>> I have a table facts520_normal_text stored as a textfile. I'm trying
>>>>> to create a compressed table from this table using GZip codec.
>>>>>
>>>>> *hive> SET hive.exec.compress.output=true;*
>>>>> *hive> SET
>>>>> mapred.output.compression.codec=org.apache.hadoop.io.compress.GZipCodec;
>>>>> *
>>>>> *hive> SET mapred.output.compression.type=BLOCK;*
>>>>> *
>>>>> *
>>>>> *hive>*
>>>>> *    > Create table facts520_gzip_text*
>>>>> *    > (fact_key BIGINT,*
>>>>> *    > products_key INT,*
>>>>> *    > retailers_key INT,*
>>>>> *    > suppliers_key INT,*
>>>>> *    > time_key INT,*
>>>>> *    > units INT)*
>>>>> *    > ROW FORMAT DELIMITED FIELDS TERMINATED BY ','*
>>>>> *    > LINES TERMINATED BY '\n'*
>>>>> *    > STORED AS TEXTFILE;*
>>>>> *
>>>>> *
>>>>> *hive> INSERT OVERWRITE TABLE facts520_gzip_text SELECT * from
>>>>> facts520_normal_text;*
>>>>>
>>>>>
>>>>> When I run the above queries, the MR job fails.
>>>>>
>>>>> The error that the Hive CLI itself shows is the following:
>>>>>
>>>>> *Total MapReduce jobs = 3*
>>>>> *Launching Job 1 out of 3*
>>>>> *Number of reduce tasks is set to 0 since there's no reduce operator*
>>>>> *Starting Job = job_201306051948_0010, Tracking URL =
>>>>> http://aana1.ird.com:50030/jobdetails.jsp?jobid=job_201306051948_0010*
>>>>> *Kill Command = /usr/lib/hadoop/bin/hadoop job  -kill
>>>>> job_201306051948_0010*
>>>>> *Hadoop job information for Stage-1: number of mappers: 3; number of
>>>>> reducers: 0*
>>>>> *2013-06-05 21:09:42,281 Stage-1 map = 0%,  reduce = 0%*
>>>>> *2013-06-05 21:10:11,446 Stage-1 map = 100%,  reduce = 100%*
>>>>> *Ended Job = job_201306051948_0010 with errors*
>>>>> *Error during job, obtaining debugging information...*
>>>>> *Job Tracking URL:
>>>>> http://aana1.ird.com:50030/jobdetails.jsp?jobid=job_201306051948_0010*
>>>>> *Examining task ID: task_201306051948_0010_m_000004 (and more) from
>>>>> job job_201306051948_0010*
>>>>> *Examining task ID: task_201306051948_0010_m_000001 (and more) from
>>>>> job job_201306051948_0010*
>>>>> *
>>>>> *
>>>>> *Task with the most failures(4):*
>>>>> *-----*
>>>>> *Task ID:*
>>>>> *  task_201306051948_0010_m_000002*
>>>>> *
>>>>> *
>>>>> *URL:*
>>>>> *
>>>>> http://aana1.ird.com:50030/taskdetails.jsp?jobid=job_201306051948_0010&tipid=task_201306051948_0010_m_000002
>>>>> *
>>>>> *-----*
>>>>> *Diagnostic Messages for this Task:*
>>>>> *java.lang.RuntimeException:
>>>>> org.apache.hadoop.hive.ql.metadata.HiveException: Hive Runtime Error
while
>>>>> processing row
>>>>> {"fact_key":7549094,"products_key":205,"retailers_key":304,"suppliers_key":402,"time_key":103,"units":23}
>>>>> *
>>>>> *        at
>>>>> org.apache.hadoop.hive.ql.exec.ExecMapper.map(ExecMapper.java:161)*
>>>>> *        at org.apache.hadoop.mapred.MapRunner.run(MapRunner.java:50)*
>>>>> *        at
>>>>> org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:418)*
>>>>> *        at org.apache.hadoop.mapred.MapTask.run(MapTask.java:333)*
>>>>> *        at org.apache.hadoop.mapred.Child$4.run(Child.java:268)*
>>>>> *        at java.security.AccessController.doPrivileged(Native Method)
>>>>> *
>>>>> *        at javax.security.auth.Subject.doAs(Subject.java:415)*
>>>>> *        at
>>>>> org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1408)
>>>>> *
>>>>> *        at org.apache.hadoop.mapred.Child.main(Child.java:262)*
>>>>> *Caused by: org.apache.hadoop.hive.ql.metadata.HiveException: Hive
>>>>> Runtime Error while processing row
>>>>> {"fact_key":7549094,"products_key":205,"retailers_key":304,"suppliers_key":402,"time_key":103,"units":23}
>>>>> *
>>>>> *        at org.apach*
>>>>> *
>>>>> *
>>>>> *FAILED: Execution Error, return code 2 from
>>>>> org.apache.hadoop.hive.ql.exec.MapRedTask*
>>>>> *MapReduce Jobs Launched:*
>>>>> *Job 0: Map: 3   HDFS Read: 0 HDFS Write: 0 FAIL*
>>>>> *Total MapReduce CPU Time Spent: 0 msec*
>>>>>
>>>>>
>>>>> I'm unable to figure out why this is happening. It looks like the data
>>>>> is not being able to be copied properly.
>>>>> Or is it that GZip codec is not supported on textfiles?
>>>>>
>>>>> Any help in this issue is greatly appreciated!
>>>>>
>>>>> Thank you,
>>>>> Sachin
>>>>>
>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message