Mailing-List: contact dev-help@hive.apache.org; run by ezmlm
Precedence: bulk
Reply-To: dev@hive.apache.org
Date: Tue, 25 Mar 2014 22:39:16 +0000 (UTC)
From: "Ashutosh Chauhan (JIRA)" <jira@apache.org>
To: hive-dev@hadoop.apache.org
Message-ID: <JIRA.12701530.1394818621743.4967.1395787156071@arcas>
In-Reply-To: <JIRA.12701530.1394818621743@arcas>
References: <JIRA.12701530.1394818621743@arcas>
Subject: [jira] [Commented] (HIVE-6670) ClassNotFound with Serde
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: quoted-printable


    [ https://issues.apache.org/jira/browse/HIVE-6670?page=3Dcom.atlassian.=
jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=3D13947=
287#comment-13947287 ]=20

Ashutosh Chauhan commented on HIVE-6670:
----------------------------------------

Query you posted in description is a good testcase. Just add it in as .q fi=
le in ql/src/test/queries/clientpositive/ where all other test queries are.=
 More info at [wiki site | https://cwiki.apache.org/confluence/display/Hive=
/HiveDeveloperFAQ#HiveDeveloperFAQ] You can create review request on [revie=
w board | https://reviews.apache.org/r/new/]=20

> ClassNotFound with Serde
> ------------------------
>
>                 Key: HIVE-6670
>                 URL: https://issues.apache.org/jira/browse/HIVE-6670
>             Project: Hive
>          Issue Type: Bug
>    Affects Versions: 0.12.0
>            Reporter: Abin Shahab
>            Assignee: Abin Shahab
>         Attachments: HIVE-6670-branch-0.12.patch, HIVE-6670.patch
>
>
> We are finding a ClassNotFound exception when we use CSVSerde(https://git=
hub.com/ogrodnek/csv-serde) to create a table.
> This is happening because MapredLocalTask does not pass the local added j=
ars to ExecDriver when that is launched.
> ExecDriver's classpath does not include the added jars. Therefore, when t=
he plan is deserialized, it throws a ClassNotFoundException in the deserial=
ization code, and results in a TableDesc object with a Null DeserializerCla=
ss.
> This results in an NPE during Fetch.
> Steps to reproduce:
> wget https://drone.io/github.com/ogrodnek/csv-serde/files/target/csv-serd=
e-1.1.2-0.11.0-all.jar into somewhere local eg. /home/soam/HiveSerdeIssue/c=
sv-serde-1.1.2-0.11.0-all.jar.
> Place some sample SCV files in HDFS as follows:
> hdfs dfs -mkdir /user/soam/HiveSerdeIssue/sampleCSV/
> hdfs dfs -put /home/soam/sampleCSV.csv /user/soam/HiveSerdeIssue/sampleCS=
V/
> hdfs dfs -mkdir /user/soam/HiveSerdeIssue/sampleJoinTarget/
> hdfs dfs -put /home/soam/sampleJoinTarget.csv /user/soam/HiveSerdeIssue/s=
ampleJoinTarget/
> =3D=3D=3D=3D
> create the tables in hive:
> ADD JAR /home/soam/HiveSerdeIssue/csv-serde-1.1.2-0.11.0-all.jar;
> create external table sampleCSV (md5hash string, filepath string)
> row format serde 'com.bizo.hive.serde.csv.CSVSerde'
> stored as textfile
> location '/user/soam/HiveSerdeIssue/sampleCSV/'
> ;
> create external table sampleJoinTarget (md5hash string, filepath string, =
datestamp string, nblines string, nberrors string)
> ROW FORMAT DELIMITED=20
> FIELDS TERMINATED BY ','=20
> LINES TERMINATED BY '\n'
> STORED AS TEXTFILE
> LOCATION '/user/soam/HiveSerdeIssue/sampleJoinTarget/'
> ;
> =3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D=3D
> Now, try the following JOIN:
> ADD JAR /home/soam/HiveSerdeIssue/csv-serde-1.1.2-0.11.0-all.jar;
> SELECT=20
> sampleCSV.md5hash,=20
> sampleCSV.filepath=20
> FROM sampleCSV
> JOIN sampleJoinTarget
> ON (sampleCSV.md5hash =3D sampleJoinTarget.md5hash)=20
> ;
> =E2=80=94
> This will fail with the error:
> Execution log at: /tmp/soam/.log
> java.lang.ClassNotFoundException: com/bizo/hive/serde/csv/CSVSerde
> Continuing ...
> 2014-03-11 10:35:03 Starting to launch local task to process map join; ma=
ximum memory =3D 238551040
> Execution failed with exit status: 2
> Obtaining error information
> Task failed!
> Task ID:
> Stage-4
> Logs:
> /var/log/hive/soam/hive.log
> FAILED: Execution Error, return code 2 from org.apache.hadoop.hive.ql.exe=
c.mr.MapredLocalTask
> Try the following LEFT JOIN. This will work:
> SELECT=20
> sampleCSV.md5hash,=20
> sampleCSV.filepath=20
> FROM sampleCSV
> LEFT JOIN sampleJoinTarget
> ON (sampleCSV.md5hash =3D sampleJoinTarget.md5hash)=20
> ;
> =3D=3D


--
This message was sent by Atlassian JIRA
(v6.2#6252)