hadoop-hdfs-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From pradeep T <itpradeep...@gmail.com>
Subject Hadoop / Hive Error when trying to load data to table that had Lzo compression in it.
Date Mon, 15 Jul 2013 10:55:06 GMT
Hi All,

SET hive.exec.compress.output=true;
SET io.seqfile.compression.type=BLOCK;
SET mapred.output.compression.codec=com.hadoop.compression.lzo.LzopCodec;
set mapred.job.priority=VERY_HIGH;
set mapred.job.name=loading data from YYY to XXXXXX;

insert overwrite table XXXXXXXXX partition (pixeldate, pixelhr, pixelmin,
reportts, clustertype, logStatus, reportgroup, requesttype, clicktype,
plugintype) select key, ipaddress, timezone,  url, responsecode,
responsesize, referrer, useragent, decodedurl, domain , specificdomain,
sourceurl, destinationurl, destinationdomain, cid, cpc, rpc,
geocountry,geoDefault,userid, sessionid, requestid, pagetypeid, articleid,
keywords, responsetime, widgetid, pagetype, sourceIP, geocity, isnewuser,
errormessage, logValue, contractstatus, pagetypederived, domainid, noofads,
noofrelatedposts, adopt, addomain, prid, topposition, leftposition,
requestTimestamp, occurenceFreq, suspectStatus, isbillable, platform,
to_date(logdate) pixeldate, hour(logdate) as pixelhr, (hour(logdate) * 60)
+ (floor(minute(logdate) / 15) * 15) as pixelmin, reportts, clustertype,
logStatus,  reportgroup, requesttype, clicktype, plugintype from YYYYYYY
where reportts=20130619042507 and clustertype = 'DEFAULT'  and
reportdate='2013-06-19' and logstatus in('VALID','Invalid','Error')

When I run this query. I am seeing this error in the tasktracker log.

In task tracker System Logs we are seeing the following errors.


We see "*Error:

*syslog logs*

2013-07-15 05:39:41,239 INFO org.apache.hadoop.util.NativeCodeLoader:
Loaded the native-hadoop library
2013-07-15 05:39:41,241 ERROR org.apache.hadoop.io.nativeio.NativeIO:
*Unable to initialize NativeIO libraries*
java.lang.NoSuchFieldError: workaroundNonThreadSafePasswdCalls
	at org.apache.hadoop.io.nativeio.NativeIO.initNative(Native Method)
	at org.apache.hadoop.io.nativeio.NativeIO.<clinit>(NativeIO.java:89)
	at org.apache.hadoop.fs.FileUtil.setPermission(FileUtil.java:653)
	at org.apache.hadoop.fs.RawLocalFileSystem.setPermission(RawLocalFileSystem.java:509)

Can someone please throw light in to it.

Some added Info :

I have upgraded Hadoop from .20.2 to 1.1.2

and hive from .7.0 to .11.

After Upgrade I faced lot of Jar file issues and cleared each with lot
of search in the internet.

But this haunting us for few days. Any help would be grateful.



View raw message