hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dean Wampler <dean.wamp...@thinkbiganalytics.com>
Subject Re: Creating Indexes
Date Sat, 03 Nov 2012 00:02:06 GMT
Wow. Lots of quirks. I'm glad the ADD JAR

On Fri, Nov 2, 2012 at 6:59 PM, Peter Marron <
Peter.Marron@trilliumsoftware.com> wrote:

>  Hi Dean,****
>
> ** **
>
> At this stage I’m really not worried about this being a hack.****
>
> I just want to get it to work, and I’m grateful for all your help.****
>
> I did as you suggested and now, as far as I can see, the Map/Reduce****
>
> has succeeded. When I look in the log for the last reduce I no longer****
>
> find an error. However this is the output from the hive command****
>
> session:****
>
> ** **
>
> MapReduce Total cumulative CPU time: 0 days 1 hours 14 minutes 51 seconds
> 360 msec****
>
> Ended Job = job_201211021743_0001****
>
> Loading data to table default.default__score_bigindex__****
>
> Deleted hdfs://localhost/data/warehouse/default__score_bigindex__****
>
> Invalid alter operation: Unable to alter index.****
>
> Table default.default__score_bigindex__ stats: [num_partitions: 0,
> num_files: 138, num_rows: 0, total_size: 446609024, raw_data_size: 0]****
>
> FAILED: Execution Error, return code 1 from
> org.apache.hadoop.hive.ql.exec.DDLTask****
>
> MapReduce Jobs Launched: ****
>
> Job 0: Map: 511  Reduce: 138   Accumulative CPU: 4491.36 sec   HDFS Read:
> 137123460712 HDFS Write: 446609024 SUCESS****
>
> Total MapReduce CPU Time Spent: 0 days 1 hours 14 minutes 51 seconds 360
> msec****
>
> hive>     ****
>
> ** **
>
> I find this very confusing. We have the bit where it says “Job 0:….
> SUCCESS”****
>
> and this seems to fit with the fact that I can’t find errors in the
> Map/Reduce.****
>
> On the other hand we have the bit where it says: “Invalid alter operation:
> Unable to alter index.”****
>
> So has it successfully created the index  or not? And if not, then what do
> I do next?****
>
> Is there somewhere else where it records Hive errors as opposed to
> Map/Reduce errors?****
>
> ** **
>
> Regards,****
>
> ** **
>
> Peter Marron****
>
>                                    ****
>
> ** **
>
> *From:* Dean Wampler [mailto:dean.wampler@thinkbiganalytics.com]
> *Sent:* 02 November 2012 14:03
>
> *To:* user@hive.apache.org
> *Subject:* Re: Creating Indexes****
>
>  ** **
>
> Oh, I saw this line in your Hive output and just assumed you were running
> in a cluster:****
>
> ** **
>
> Hadoop job information for Stage-1: number of mappers: 511; number of
> reducers: 138****
>
> ** **
>
> I haven't tried running a job that big in pseudodistributed mode either,
> but that's beside the point.****
>
> ** **
>
> So it seems to be an issue with indexing, but it still begs the question
> why derby isn't on the classpath for the task. I would try using the ADD
> JAR command, which copies the jar around the "cluster" and puts it on the
> classpath. It's what you would use with UDFs, for example:****
>
> ** **
>
> ADD JAR /path/to/derby.jar****
>
> ALTER INDEX ...;****
>
> ** **
>
> It's a huge hack, but it just might work.****
>
> dean****
>
> ** **
>
> On Fri, Nov 2, 2012 at 3:44 AM, Peter Marron <
> Peter.Marron@trilliumsoftware.com> wrote:****
>
> Hi Dean,****
>
>  ****
>
> I’m running everything on a single physical machine in pseudo-distributed
> mode.****
>
>  ****
>
> Well it certainly looks like the reducer is looking for a derby.jar,
> although I must****
>
> confess I don’t really understand why it would be doing that.****
>
> In an effort to fix that I copied the derby.jar (derby-10.4.2.0.jar) into
> the****
>
> Hadoop directory, where I assume that the reducer would be able to find it.
> ****
>
> However I get exactly the same problem as before.****
>
> Is there some particular place that I should put the derby.jar to make this
> ****
>
> problem go away? Is there anything else that I can try?****
>
>  ****
>
> Peter Marron****
>
>  ****
>
> *From:* Dean Wampler [mailto:dean.wampler@thinkbiganalytics.com]
> *Sent:* 01 November 2012 13:02****
>
>
> *To:* user@hive.apache.org
> *Subject:* Re: Creating Indexes****
>
>  ****
>
> It looks like you're using Derby with a real cluster, not just a single
> machine in local or pseudo-distributed mode. I haven't tried this myself,
> but the derby jar is probably not on the machine that ran the reducer task
> that failed.****
>
>  ****
>
> dean****
>
> On Thu, Nov 1, 2012 at 4:31 AM, Peter Marron <
> Peter.Marron@trilliumsoftware.com> wrote:****
>
> Hi Shreepadma,****
>
>  ****
>
> I agree that the error looks odd. However I can’t believe that I would have
> ****
>
> got this far with Hive if there was no derby jar. Nevertheless I checked.*
> ***
>
> Here is a directory listing of the Hive install:****
>
>  [snip]****
>
>  ****
>
>
>
> ****
>
> ** **
>
> --
> *Dean Wampler, Ph.D.*****
>
> thinkbiganalytics.com****
>
> +1-312-339-1330****
>
> ** **
>
> ** **
>



-- 
*Dean Wampler, Ph.D.*
thinkbiganalytics.com
+1-312-339-1330

Mime
View raw message