Return-Path: X-Original-To: apmail-hive-user-archive@www.apache.org Delivered-To: apmail-hive-user-archive@www.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id E482C10BA9 for ; Tue, 23 Jul 2013 02:11:20 +0000 (UTC) Received: (qmail 91862 invoked by uid 500); 23 Jul 2013 02:11:19 -0000 Delivered-To: apmail-hive-user-archive@hive.apache.org Received: (qmail 91812 invoked by uid 500); 23 Jul 2013 02:11:19 -0000 Mailing-List: contact user-help@hive.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: user@hive.apache.org Delivered-To: mailing list user@hive.apache.org Received: (qmail 91802 invoked by uid 99); 23 Jul 2013 02:11:19 -0000 Received: from nike.apache.org (HELO nike.apache.org) (192.87.106.230) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Jul 2013 02:11:19 +0000 X-ASF-Spam-Status: No, hits=1.5 required=5.0 tests=HTML_MESSAGE,RCVD_IN_DNSWL_LOW,SPF_PASS,WEIRD_PORT X-Spam-Check-By: apache.org Received-SPF: pass (nike.apache.org: domain of Sanjay.Subramanian@wizecommerce.com designates 216.32.181.185 as permitted sender) Received: from [216.32.181.185] (HELO ch1outboundpool.messaging.microsoft.com) (216.32.181.185) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 23 Jul 2013 02:11:10 +0000 Received: from mail142-ch1-R.bigfish.com (10.43.68.237) by CH1EHSOBE010.bigfish.com (10.43.70.60) with Microsoft SMTP Server id 14.1.225.22; Tue, 23 Jul 2013 02:10:47 +0000 Received: from mail142-ch1 (localhost [127.0.0.1]) by mail142-ch1-R.bigfish.com (Postfix) with ESMTP id 6EE494C0357; Tue, 23 Jul 2013 02:10:47 +0000 (UTC) X-Forefront-Antispam-Report: CIP:157.56.232.197;KIP:(null);UIP:(null);IPV:NLI;H:BLUPRD0411HT001.namprd04.prod.outlook.com;RD:none;EFVD:NLI X-SpamScore: 1 X-BigFish: PS1(zzbb2dI98dI9371I936eIc85fh181fIdd85k9a6kzz1f42h208ch1ee6h1de0h1fdah2073h1202h1e76h1d1ah1d2ah1fc6hzz1de098h17326ah18c673h1de097h1de096h18de19h8275bh8275dhz2fh2a8h668h839hbe3he5bhf0ah1288h12a5h12bdh137ah1441h1504h1537h153bh162dh1631h1758h18e1h1946h19b5h1ad9h1b0ah1bceh1d0ch1d2eh1d3fh1dc1h1dfeh1dffh1e1dh1155h) Received-SPF: pass (mail142-ch1: domain of wizecommerce.com designates 157.56.232.197 as permitted sender) client-ip=157.56.232.197; envelope-from=Sanjay.Subramanian@wizecommerce.com; helo=BLUPRD0411HT001.namprd04.prod.outlook.com ;.outlook.com ; Received: from mail142-ch1 (localhost.localdomain [127.0.0.1]) by mail142-ch1 (MessageSwitch) id 1374545444416788_15279; Tue, 23 Jul 2013 02:10:44 +0000 (UTC) Received: from CH1EHSMHS033.bigfish.com (snatpool1.int.messaging.microsoft.com [10.43.68.241]) by mail142-ch1.bigfish.com (Postfix) with ESMTP id 5FB7360049; Tue, 23 Jul 2013 02:10:44 +0000 (UTC) Received: from BLUPRD0411HT001.namprd04.prod.outlook.com (157.56.232.197) by CH1EHSMHS033.bigfish.com (10.43.70.33) with Microsoft SMTP Server (TLS) id 14.16.227.3; Tue, 23 Jul 2013 02:10:44 +0000 Received: from BLUPRD0411MB426.namprd04.prod.outlook.com ([169.254.10.182]) by BLUPRD0411HT001.namprd04.prod.outlook.com ([10.255.127.36]) with mapi id 14.16.0329.000; Tue, 23 Jul 2013 02:10:43 +0000 From: Sanjay Subramanian To: "user@hive.apache.org" , "bejoy_ks@yahoo.com" Subject: Re: how to let hive support lzo Thread-Topic: how to let hive support lzo Thread-Index: AQHOhp2i5szwsE72akyb1S0Wx2XvUplwLt2AgABsdwCAAHXBgA== Date: Tue, 23 Jul 2013 02:10:42 +0000 Message-ID: In-Reply-To: <1697495375-1374494952-cardhu_decombobulator_blackberry.rim.net-75424370-@b16.c16.bise7.blackberry> Accept-Language: en-US Content-Language: en-US X-MS-Has-Attach: X-MS-TNEF-Correlator: x-originating-ip: [10.255.127.4] Content-Type: multipart/alternative; boundary="_000_CE1332D89651sanjaysubramanianwizecommercecom_" MIME-Version: 1.0 X-OriginatorOrg: wizecommerce.com X-FOPE-CONNECTOR: Id%0$Dn%*$RO%0$TLS%0$FQDN%$TlsDn% X-Virus-Checked: Checked by ClamAV on apache.org --_000_CE1332D89651sanjaysubramanianwizecommercecom_ Content-Type: text/plain; charset="us-ascii" Content-Transfer-Encoding: quoted-printable This works for us SET hive.exec.compress.intermediate=3Dtrue SET hive.exec.compress.output=3Dtrue SET mapreduce.output.fileoutputformat.compress.codec=3Dcom.hadoop.compressi= on.lzo.LzopCodec SET mapreduce.map.output.compress=3Dtrue SET mapreduce.map.output.compress.codec=3Dorg.apache.hadoop.io.compress.Sna= ppyCodec SET mapreduce.output.fileoutputformat.compress=3Dtrue From: "bejoy_ks@yahoo.com" > Reply-To: "user@hive.apache.org" >, "bejoy_ks@yahoo.com" > Date: Monday, July 22, 2013 5:09 AM To: "user@hive.apache.org" > Subject: Re: how to let hive support lzo Hi, Along with the mapred.compress* properties try to set hive.exec.compress.output to true. Regards Bejoy KS Sent from remote device, Please excuse typos ________________________________ From: ch huang > Date: Mon, 22 Jul 2013 13:41:01 +0800 To: > ReplyTo: user@hive.apache.org Subject: Re: how to let hive support lzo # hbase org.apache.hadoop.hbase.util.CompressionTest hdfs://CH22:9000/alex/= my.txt lzo 13/07/22 13:27:58 WARN conf.Configuration: hadoop.native.lib is deprecated.= Instead, use io.native.lib.available 13/07/22 13:27:59 INFO util.ChecksumType: Checksum using org.apache.hadoop.= util.PureJavaCrc32 13/07/22 13:27:59 INFO util.ChecksumType: Checksum can use org.apache.hadoo= p.util.PureJavaCrc32C 13/07/22 13:27:59 ERROR metrics.SchemaMetrics: Inconsistent configuration. = Previous configuration for using table name in metrics: true, new configura= tion: false 13/07/22 13:27:59 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library 13/07/22 13:27:59 INFO lzo.LzoCodec: Successfully loaded & initialized nati= ve-lzo library [hadoop-lzo rev 6bb1b7f8b9044d8df9b4d2b6641db7658aab3cf8] 13/07/22 13:27:59 INFO compress.CodecPool: Got brand-new compressor [.lzo_d= eflate] 13/07/22 13:28:00 INFO compress.CodecPool: Got brand-new decompressor [.lzo= _deflate] SUCCESS # hadoop jar /usr/lib/hadoop/lib/hadoop-lzo-0.4.15.jar com.hadoop.compressi= on.lzo.LzoIndexer /alex 13/07/22 09:39:04 INFO lzo.GPLNativeCodeLoader: Loaded native gpl library 13/07/22 09:39:04 INFO lzo.LzoCodec: Successfully loaded & initialized nati= ve-lzo library [hadoop-lzo rev 6bb1b7f8b9044d8df9b4d2b6641db7658aab3cf8] 13/07/22 09:39:04 INFO lzo.LzoIndexer: LZO Indexing directory /alex... 13/07/22 09:39:04 INFO lzo.LzoIndexer: LZO Indexing directory hdfs://CH22= :9000/alex/alex_t... 13/07/22 09:39:04 INFO lzo.LzoIndexer: [INDEX] LZO Indexing file hdfs://C= H22:9000/alex/sqoop-1.99.2-bin-hadoop200.tar.gz.lzo, size 0.02 GB... 13/07/22 09:39:05 WARN conf.Configuration: hadoop.native.lib is deprecated.= Instead, use io.native.lib.available 13/07/22 09:39:06 INFO lzo.LzoIndexer: Completed LZO Indexing in 1.16 sec= onds (13.99 MB/s). Index size is 0.52 KB. 13/07/22 09:39:06 INFO lzo.LzoIndexer: [INDEX] LZO Indexing file hdfs://C= H22:9000/alex/test1.lzo, size 0.00 GB... 13/07/22 09:39:06 INFO lzo.LzoIndexer: Completed LZO Indexing in 0.08 sec= onds (0.00 MB/s). Index size is 0.01 KB. On Mon, Jul 22, 2013 at 1:37 PM, ch huang > wrote: hi ,all: i already install and testing lzo in hadoop and hbase,all success,but = when i try it on hive ,it failed ,how can i do let hive can recognize lzo? hive> set mapred.map.output.compression.codec; mapred.map.output.compression.codec=3Dorg.apache.hadoop.io.compress.SnappyC= odec hive> set mapred.map.output.compression.codec=3Dcom.hadoop.compression.lzo.= LzoCodec hive> select count(*) from test; Total MapReduce jobs =3D 1 Launching Job 1 out of 1 Number of reduce tasks determined at compile time: 1 In order to change the average load for a reducer (in bytes): set hive.exec.reducers.bytes.per.reducer=3D In order to limit the maximum number of reducers: set hive.exec.reducers.max=3D In order to set a constant number of reducers: set mapred.reduce.tasks=3D Starting Job =3D job_1374463239553_0003, Tracking URL =3D http://CH22:8088/= proxy/application_1374463239553_0003/ Kill Command =3D /usr/lib/hadoop/bin/hadoop job -kill job_1374463239553_00= 03 Hadoop job information for Stage-1: number of mappers: 1; number of reducer= s: 1 2013-07-22 13:33:27,243 Stage-1 map =3D 0%, reduce =3D 0% 2013-07-22 13:33:45,403 Stage-1 map =3D 100%, reduce =3D 0% Ended Job =3D job_1374463239553_0003 with errors Error during job, obtaining debugging information... Job Tracking URL: http://CH22:8088/proxy/application_1374463239553_0003/ Examining task ID: task_1374463239553_0003_m_000000 (and more) from job job= _1374463239553_0003 Task with the most failures(4): ----- Task ID: task_1374463239553_0003_m_000000 URL: http://CH22:8088/taskdetails.jsp?jobid=3Djob_1374463239553_0003&tipid=3Dt= ask_1374463239553_0003_m_000000 ----- Diagnostic Messages for this Task: Error: java.lang.RuntimeException: native-lzo library not available at com.hadoop.compression.lzo.LzoCodec.getCompressorType(LzoCodec.j= ava:155) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.= java:104) at org.apache.hadoop.io.compress.CodecPool.getCompressor(CodecPool.= java:118) at org.apache.hadoop.mapred.IFile$Writer.(IFile.java:115) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.sortAndSpill(Ma= pTask.java:1580) at org.apache.hadoop.mapred.MapTask$MapOutputBuffer.flush(MapTask.j= ava:1457) at org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:435) at org.apache.hadoop.mapred.MapTask.run(MapTask.java:340) at org.apache.hadoop.mapred.YarnChild$2.run(YarnChild.java:157) at java.security.AccessController.doPrivileged(Native Method) at javax.security.auth.Subject.doAs(Subject.java:396) at org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupIn= formation.java:1408) at org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:152) CONFIDENTIALITY NOTICE =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D This email message and any attachments are for the exclusive use of the int= ended recipient(s) and may contain confidential and privileged information.= Any unauthorized review, use, disclosure or distribution is prohibited. If= you are not the intended recipient, please contact the sender by reply ema= il and destroy all copies of the original message along with any attachment= s, from your computer system. If you are the intended recipient, please be = advised that the content of this message is subject to access, review and d= isclosure by the sender's Email System Administrator. --_000_CE1332D89651sanjaysubramanianwizecommercecom_ Content-Type: text/html; charset="us-ascii" Content-ID: Content-Transfer-Encoding: quoted-printable
This works for us

SET hive.exec.compress.intermediate=3Dtrue
SET hive.exec.compress.output=3Dtrue
SET mapreduce.output.fileoutputformat.compress.codec=3Dcom.hadoop.comp= ression.lzo.LzopCodec
SET mapreduce.map.output.compress=3Dtrue
SET mapreduce.map.output.compress.codec=3Dorg.apache.hadoop.io.compres= s.SnappyCodec
SET mapreduce.output.fileoutputformat.compress=3Dtrue


From: "bejoy_ks@yahoo.com" <bejoy_ks@yahoo.com>
Reply-To: "user@hive.apache.org" <user@hive.apache.org>, "bejoy_ks@yahoo.com" <bejoy_ks@yahoo.com>
Date: Monday, July 22, 2013 5:09 AM=
To: "user@hive.apache.org" <user@hive.apache.org>
Subject: Re: how to let hive suppor= t lzo


Hi,

Along with the mapred.compress* properties try to set
hive.exec.compress.output to true.
Regards
Bejoy KS

Sent from remote device, Please excuse typos
From: ch huang <justl= ooks@gmail.com>
Date: Mon, 22 Jul 2013 13:41:01 +0800
Subject: Re: how to let hive support lzo

 
# hbase org.apache.hadoop.hbas= e.util.CompressionTest hdfs://CH22:9000/alex/my.txt lzo
13/07/22 13:27:58 WARN conf.Configu= ration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
13/07/22 13:27:59 INFO util.Checksu= mType: Checksum using org.apache.hadoop.util.PureJavaCrc32
13/07/22 13:27:59 INFO util.Checksu= mType: Checksum can use org.apache.hadoop.util.PureJavaCrc32C
13/07/22 13:27:59 ERROR metrics.Sch= emaMetrics: Inconsistent configuration. Previous configuration for using table name in metrics: true, new configur= ation: false
13/07/22 13:27:59 INFO lzo.GPLNativ= eCodeLoader: Loaded native gpl library
13/07/22 13:27:59 INFO lzo.LzoCodec= : Successfully loaded & initialized native-lzo library [hadoop-lzo rev 6bb1b7f8b9044d8df9b4d2b6641db7658aab3cf= 8]
13/07/22 13:27:59 INFO compress.Cod= ecPool: Got brand-new compressor [.lzo_deflate]
13/07/22 13:28:00 INFO compress.Cod= ecPool: Got brand-new decompressor [.lzo_deflate]
SUCCESS

 

 
 
# hadoop jar /usr/lib/hadoop/l= ib/hadoop-lzo-0.4.15.jar com.hadoop.compression.lzo.LzoIndexer /alex
13/07/22 09:39:04 INFO lzo.GPLNativ= eCodeLoader: Loaded native gpl library
13/07/22 09:39:04 INFO lzo.LzoCodec= : Successfully loaded & initialized native-lzo library [hadoop-lzo rev 6bb1b7f8b9044d8df9b4d2b6641db7658aab3cf= 8]
13/07/22 09:39:04 INFO lzo.LzoIndex= er: LZO Indexing directory /alex...
13/07/22 09:39:04 INFO lzo.LzoIndex= er:   LZO Indexing directory hdfs://CH22:9000/alex/alex_t...
13/07/22 09:39:04 INFO lzo.LzoIndex= er:   [INDEX] LZO Indexing file hdfs://CH22:9000/alex/sqoop-1.99.= 2-bin-hadoop200.tar.gz.lzo, size 0.02 GB...
13/07/22 09:39:05 WARN conf.Configu= ration: hadoop.native.lib is deprecated. Instead, use io.native.lib.available
13/07/22 09:39:06 INFO lzo.LzoIndex= er:   Completed LZO Indexing in 1.16 seconds (13.99 MB/s).  Index size is 0.52 KB.

13/07/22 09:39:06 INFO lzo.LzoIndex= er:   [INDEX] LZO Indexing file hdfs://CH22:9000/alex/test1.lzo, size 0.00 GB...
13/07/22 09:39:06 INFO lzo.LzoIndex= er:   Completed LZO Indexing in 0.08 seconds (0.00 MB/s).  Index size is 0.01 KB.


On Mon, Jul 22, 2013 at 1:37 PM, ch huang <justlooks@gmai= l.com> wrote:
hi ,all:
     i already install and testing lzo in hadoop a= nd hbase,all success,but when i try it on hive ,it failed ,how can i do let= hive can recognize lzo?
 
 
hive> set mapred.map.output.compression.codec;
mapred.map.output.compression.codec=3Dorg.apache.hadoop.io.compress.SnappyC= odec
hive> set mapred.map.output.compression.codec=3Dcom.hadoop.compression.l= zo.LzoCodec
hive> select count(*) from test;
Total MapReduce jobs =3D 1
Launching Job 1 out of 1
Number of reduce tasks determined at compile time: 1
In order to change the average load for a reducer (in bytes):
  set hive.exec.reducers.bytes.per.reducer=3D<number>
In order to limit the maximum number of reducers:
  set hive.exec.reducers.max=3D<number>
In order to set a constant number of reducers:
  set mapred.reduce.tasks=3D<number>
Starting Job =3D job_1374463239553_0003, Tracking URL =3D http://CH22:8088/proxy/application_1374463239553_0003/
Kill Command =3D /usr/lib/hadoop/bin/hadoop job  -kill job_13744632395= 53_0003
Hadoop job information for Stage-1: number of mappers: 1; number of reducer= s: 1
2013-07-22 13:33:27,243 Stage-1 map =3D 0%,  reduce =3D 0%
2013-07-22 13:33:45,403 Stage-1 map =3D 100%,  reduce =3D 0%
Ended Job =3D job_1374463239553_0003 with errors
Error during job, obtaining debugging information...
Job Tracking URL: http://CH22:8088/proxy/application_1374463239553_0003/
Examining task ID: task_1374463239553_0003_m_000000 (and more) from job job= _1374463239553_0003
Task with the most failures(4):
-----
Task ID:
  task_1374463239553_0003_m_000000
URL:
  http://CH22:8088/taskdetails.jsp?jobid=3Djob_1374463239553_0003&tipid= =3Dtask_1374463239553_0003_m_000000
-----
Diagnostic Messages for this Task:
Error: java.lang.RuntimeException: native-lzo library not available
        at com.hadoop.compression.lzo.Lz= oCodec.getCompressorType(LzoCodec.java:155)
        at org.apache.hadoop.io.compress= .CodecPool.getCompressor(CodecPool.java:104)
        at org.apache.hadoop.io.compress= .CodecPool.getCompressor(CodecPool.java:118)
        at org.apache.hadoop.mapred.IFil= e$Writer.<init>(IFile.java:115)
        at org.apache.hadoop.mapred.MapT= ask$MapOutputBuffer.sortAndSpill(MapTask.java:1580)
        at org.apache.hadoop.mapred.MapT= ask$MapOutputBuffer.flush(MapTask.java:1457)
        at org.apache.hadoop.mapred.MapT= ask.runOldMapper(MapTask.java:435)
        at org.apache.hadoop.mapred.MapT= ask.run(MapTask.java:340)
        at org.apache.hadoop.mapred.Yarn= Child$2.run(YarnChild.java:157)
        at java.security.AccessControlle= r.doPrivileged(Native Method)
        at javax.security.auth.Subject.d= oAs(Subject.java:396)
        at org.apache.hadoop.security.Us= erGroupInformation.doAs(UserGroupInformation.java:1408)
        at org.apache.hadoop.mapred.Yarn= Child.main(YarnChild.java:152)
 


CONFIDENTIALITY NOTICE
=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
This email message and any attachments are for the exclusive use of the int= ended recipient(s) and may contain confidential and privileged information.= Any unauthorized review, use, disclosure or distribution is prohibited. If= you are not the intended recipient, please contact the sender by reply email and destroy all copies of the ori= ginal message along with any attachments, from your computer system. If you= are the intended recipient, please be advised that the content of this mes= sage is subject to access, review and disclosure by the sender's Email System Administrator.
--_000_CE1332D89651sanjaysubramanianwizecommercecom_--