hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Bryan Jeffrey <bryan.jeff...@gmail.com>
Subject Hive & HDFS Network Encryption
Date Fri, 14 Feb 2014 18:35:19 GMT
Hello.

I'm running Hadoop 2.2.0 and Hive 0.12.0.

Hadoop supports encryption of client writes to the data node, as described
under 'Data Transfer Protocol' here:
http://hortonworks.com/blog/wire-encryption-hadoop/.  The encryption
appears to work for simple hadoop commands (copyFromLocal, load data
infile, etc.)  However, when I attempt to execute a Hive job I receive the
following errors:

java.lang.RuntimeException: Cannot serialize object
        at
org.apache.hadoop.hive.ql.exec.Utilities$1.exceptionThrown(Utilities.java:652)
        at java.beans.XMLEncoder.writeln(XMLEncoder.java:489)
        at java.beans.XMLEncoder.outputValue(XMLEncoder.java:543)
        at java.beans.XMLEncoder.outputStatement(XMLEncoder.java:682)
        at java.beans.XMLEncoder.outputStatement(XMLEncoder.java:687)
        at java.beans.XMLEncoder.outputValue(XMLEncoder.java:552)
        at java.beans.XMLEncoder.outputStatement(XMLEncoder.java:682)
        at java.beans.XMLEncoder.outputStatement(XMLEncoder.java:687)
        at java.beans.XMLEncoder.outputValue(XMLEncoder.java:552)
        at java.beans.XMLEncoder.outputStatement(XMLEncoder.java:682)
        at java.beans.XMLEncoder.outputStatement(XMLEncoder.java:687)
        at java.beans.XMLEncoder.outputStatement(XMLEncoder.java:687)
        at java.beans.XMLEncoder.outputValue(XMLEncoder.java:552)
        at java.beans.XMLEncoder.flush(XMLEncoder.java:398)
        at java.beans.XMLEncoder.close(XMLEncoder.java:429)
        at
org.apache.hadoop.hive.ql.exec.Utilities.serializeObject(Utilities.java:667)
        at
org.apache.hadoop.hive.ql.exec.Utilities.serializePlan(Utilities.java:610)
        at
org.apache.hadoop.hive.ql.exec.Utilities.setBaseWork(Utilities.java:495)
        at
org.apache.hadoop.hive.ql.exec.Utilities.setMapWork(Utilities.java:479)
        at
org.apache.hadoop.hive.ql.exec.Utilities.setMapRedWork(Utilities.java:472)
        at
org.apache.hadoop.hive.ql.exec.mr.ExecDriver.execute(ExecDriver.java:383)
        at
org.apache.hadoop.hive.ql.exec.mr.MapRedTask.execute(MapRedTask.java:144)
        at org.apache.hadoop.hive.ql.exec.Task.executeTask(Task.java:151)
        at
org.apache.hadoop.hive.ql.exec.TaskRunner.runSequential(TaskRunner.java:65)
        at org.apache.hadoop.hive.ql.Driver.launchTask(Driver.java:1414)
        at org.apache.hadoop.hive.ql.Driver.execute(Driver.java:1192)
        at org.apache.hadoop.hive.ql.Driver.runInternal(Driver.java:1020)
        at org.apache.hadoop.hive.ql.Driver.run(Driver.java:888)
        at
org.apache.hadoop.hive.cli.CliDriver.processLocalCmd(CliDriver.java:259)
        at
org.apache.hadoop.hive.cli.CliDriver.processCmd(CliDriver.java:216)
        at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:413)
        at
org.apache.hadoop.hive.cli.CliDriver.processLine(CliDriver.java:348)
        at
org.apache.hadoop.hive.cli.CliDriver.executeDriver(CliDriver.java:731)
        at org.apache.hadoop.hive.cli.CliDriver.run(CliDriver.java:675)
        at org.apache.hadoop.hive.cli.CliDriver.main(CliDriver.java:614)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at
sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:39)
        at
sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:25)
        at java.lang.reflect.Method.invoke(Method.java:597)
        at org.apache.hadoop.util.RunJar.main(RunJar.java:212)
Caused by: java.nio.channels.ClosedChannelException
        at
org.apache.hadoop.hdfs.DFSOutputStream.checkClosed(DFSOutputStream.java:1317)
        at org.apache.hadoop.fs.FSOutputSummer.write(FSOutputSummer.java:98)
        at
org.apache.hadoop.fs.FSDataOutputStream$PositionCache.write(FSDataOutputStream.java:59)
        at java.io.DataOutputStream.write(DataOutputStream.java:90)
        at java.io.FilterOutputStream.write(FilterOutputStream.java:80)
        at java.beans.XMLEncoder.writeln(XMLEncoder.java:485)
        ... 38 more


This seems to imply that Hive does not have encryption enabled and thus
objects cannot be serialized to the encrypted communication channel.  Does
Hive support DN transfer protocol encryption?  If so, is there
documentation somewhere on usage?

I did find several pages on Hive encryption, but these merely refer to Hive
JDBC connection encryption.

Regards,

Bryan Jeffrey

Mime
View raw message