hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Takahiko Saito <tysa...@gmail.com>
Subject Re: Re: HiveServer2 load data inpath fails
Date Tue, 27 Oct 2015 16:48:01 GMT
Hi Vineet,

Were you able to find anything in HS2 log?

I was just able to run 'load data inpath' with hive ver. 0.14 without any
issue.
My env is hive.execution.engine=tez though.
Also Based on your error message, it may be worth checking what value is
set for datanucleus.connectionPoolingType. I have mine set
to datanucleus.connectionPoolingType=BONECP.

Cheers,

On Tue, Oct 20, 2015 at 8:45 PM, Vineet Mishra <clearmidoubt@gmail.com>
wrote:

> Hi
>
> I tried truncating the table(ensuring no data files are present in the
> warehouse directory) and then used load data inpath with/without overwrite
> option but nothing worked.
>
> Hive directory is owned by hive user but my existing user which is a
> application user owns the complete access on hive warehouse directory as I
> can easily delete the files from hive warehouse dir.
>
> Will check the HS2 logs for further debugging.
> On Oct 21, 2015 7:38 AM, "Matthew Bigelow" <matthew.h.bigelow@gmail.com>
> wrote:
>
>> This is a shot in the dark but...  have you tried load data inpath into a
>> fresh table.  I have not had issues with this commmand and HS2.  I havent
>> done it with Overwrite though.  Just load data inpath 'hdfs-file' onto
>> table.
>>
>> I dont see it in the exception but permission issues can cause it to fail
>> the movetask portion of this command.  The user running HS2 need full
>> access to the hive warehouse directory. Is the table managed by hive?
>> On Oct 20, 2015 6:54 PM, "Vineet Mishra" <clearmidoubt@gmail.com> wrote:
>>
>>> Hi,
>>>
>>> I already stated my requirement since the files are generating on hdfs
>>> and i want to load the file from hdfs to hive instead of local file system.
>>>
>>> Moreover in production like environment one just have port access to HS2
>>> node and no direct access to machine, in that particular case how do we
>>> expect load local to be the only solution.
>>>
>>> I have been doing the load inpath using HS1 but due to concurrency and
>>> gc issue I am moving to HS2 but if its just dependent on load local then
>>> its really strange.
>>>
>>> Thanks!
>>> On Oct 21, 2015 6:22 AM, "r7raul1984@163.com" <r7raul1984@163.com>
>>> wrote:
>>>
>>>>   Place  the file on the hiveserver2 host local filesystem . Then try
>>>>  load data local inpath
>>>>
>>>> ------------------------------
>>>> r7raul1984@163.com
>>>>
>>>>
>>>> *From:* Vineet Mishra <clearmidoubt@gmail.com>
>>>> *Date:* 2015-10-21 03:05
>>>> *To:* user <user@hive.apache.org>
>>>> *CC:* cdh-user <cdh-user@cloudera.org>
>>>> *Subject:* Re: HiveServer2 load data inpath fails
>>>> Any update around this?
>>>>
>>>> 0: jdbc:hive2://hadoop-hs2:10000> load data inpath '/data/my_table/'
>>>> overwrite into table mydb.mytable;
>>>> Error: Error while processing statement: FAILED: Execution Error,
>>>> return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
>>>> (state=08S01,code=1)
>>>>
>>>> I couldn't get any possible solution for this. Does it mean HS2 is
>>>> incapable of supporting load data or there's no JIRA open for it?
>>>>
>>>> Version : 0.13.1-cdh5.3.2
>>>>
>>>> On Tue, Oct 20, 2015 at 10:54 PM, Vineet Mishra <clearmidoubt@gmail.com
>>>> > wrote:
>>>>
>>>>> My data files resides on hdfs and for the given command I guess it
>>>>> will point to local filesystem instead of hdfs.
>>>>> On Oct 20, 2015 10:25 PM, "Vikas Srivastava" <hadoophive@gmail.com>
>>>>> wrote:
>>>>>
>>>>>> You Need to use “load data local inpath”
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> *From:* Vineet Mishra [mailto:clearmidoubt@gmail.com]
>>>>>> *Sent:* Tuesday, October 20, 2015 6:08 PM
>>>>>> *To:* user@hive.apache.org; cdh-user@cloudera.or to
>>>>>> *Subject:* HiveServer2 load data inpath fails
>>>>>>
>>>>>>
>>>>>>
>>>>>> Hi All,
>>>>>>
>>>>>> I am trying to run load data inpath to update/refresh my table data
>>>>>> in hive. I am using HiveServer2 to make jdbc connection and run the
>>>>>> following command
>>>>>>
>>>>>> load data inpath 'path_to_file_on_hdfs' overwrite into table
>>>>>> tablename;
>>>>>>
>>>>>> Eventually this query works fine with HiveServer1 through thrift
>>>>>> connection but fails through jdbc connection on HS2, stack trace
mentioned
>>>>>> below.
>>>>>>
>>>>>> java.sql.SQLException: Error while processing statement: FAILED:
>>>>>> Execution Error, return code 1 from org.apache.hadoop.hive.ql.exec.MoveTask
>>>>>>
>>>>>> at org.apache.hive.jdbc.HiveStatement.execute(HiveStatement.java:277)
>>>>>>
>>>>>> at
>>>>>> com.sd.dwh.sc.tungsten.misc.CommonFunction.executeQuery(CommonFunction.java:813)
>>>>>>
>>>>>> at
>>>>>> com.sd.dwh.sc.tungsten.misc.CommonFunction.dataLoadFromFile(CommonFunction.java:836)
>>>>>>
>>>>>> at
>>>>>> com.sd.dwh.sc.tungsten.custom.JDBCMaterialize.merge(JDBCMaterialize.java:105)
>>>>>>
>>>>>> at
>>>>>> com.sd.dwh.sc.tungsten.custom.JDBCMaterialize.run(JDBCMaterialize.java:54)
>>>>>>
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1145)
>>>>>>
>>>>>> at
>>>>>> java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:615)
>>>>>>
>>>>>> at java.lang.Thread.run(Thread.java:745)
>>>>>>
>>>>>> Any suggestions would be highly appreciated.
>>>>>>
>>>>>> URGENT CALL.
>>>>>>
>>>>>> Thanks!
>>>>>>
>>>>>
>>>> --
>>>
>>> ---
>>> You received this message because you are subscribed to the Google
>>> Groups "CDH Users" group.
>>> To unsubscribe from this group and stop receiving emails from it, send
>>> an email to cdh-user+unsubscribe@cloudera.org.
>>> For more options, visit
>>> https://groups.google.com/a/cloudera.org/d/optout.
>>>
>> --
>>
>> ---
>> You received this message because you are subscribed to the Google Groups
>> "CDH Users" group.
>> To unsubscribe from this group and stop receiving emails from it, send an
>> email to cdh-user+unsubscribe@cloudera.org.
>> For more options, visit https://groups.google.com/a/cloudera.org/d/optout
>> .
>>
>


-- 
Takahiko Saito

Mime
View raw message