hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vijay <tec...@gmail.com>
Subject Re: First import into new partition disappears
Date Wed, 30 Dec 2009 19:45:09 GMT
You can try "svn info" in the directory to get detailed information.

On Tue, Dec 29, 2009 at 11:02 PM, Saurabh Nanda <saurabhnanda@gmail.com>wrote:

> I reverted back to the old build and the same query is working fine now.
> How do I find out the SVN revision of the old build?
>
> Saurabh.
>
>
> On Wed, Dec 30, 2009 at 12:24 PM, Saurabh Nanda <saurabhnanda@gmail.com>wrote:
>
>> The rate at which "Map input bytes" and "Map input records" is growing is
>> extremely slow. Is something wrong with the HDFS configuration? But, it was
>> working perfectly fine with the previous Hive version.
>>
>> Saurabh.
>>
>>
>> On Wed, Dec 30, 2009 at 12:20 PM, Saurabh Nanda <saurabhnanda@gmail.com>wrote:
>>
>>> Attached are the task logs of one of the tasks.
>>>
>>> Saurabh.
>>>
>>>
>>> On Wed, Dec 30, 2009 at 12:16 PM, Zheng Shao <zshao9@gmail.com> wrote:
>>>
>>>> This should be compiled into a single map-only job.
>>>> Can you take a look at the progress and the task logs of the job?
>>>>
>>>> We are not aware of any changes that might cause this problem.
>>>>
>>>> Zheng
>>>>
>>>> On Tue, Dec 29, 2009 at 10:35 PM, Saurabh Nanda <saurabhnanda@gmail.com>
>>>> wrote:
>>>> > Picking up data from the 'raw' table, filtering the unwanted lines and
>>>> > inserting into 'raw_compressed' table which is stored as sequencefile:
>>>> >
>>>> > insert overwrite table raw_compressed partition(dt='2009-04-01')
>>>> select line
>>>> > from raw where dt='2009-04-01' and lower(line) rlike '.*get
>>>> > .*/confirmation.*http.*' and not lower(line) rlike
>>>> >
>>>> '(/images.*?|/styles.*?|/javascripts.*?|/adserver.*?|.*?favicon.*?|/includes/thwarte-logo.html.*)';
>>>> >
>>>> > Saurabh.
>>>> >
>>>> > On Wed, Dec 30, 2009 at 11:59 AM, Zheng Shao <zshao9@gmail.com>
>>>> wrote:
>>>> >>
>>>> >> What is the import query? Do you mean "load data"?
>>>> >> Can you give an example?
>>>> >>
>>>> >> Zheng
>>>> >>
>>>> >> On Tue, Dec 29, 2009 at 10:22 PM, Saurabh Nanda <
>>>> saurabhnanda@gmail.com>
>>>> >> wrote:
>>>> >> > Also has something changed drastically in Hive over the last
2-3
>>>> months?
>>>> >> > A
>>>> >> > simply import query seems to be taking forever now!
>>>> >> >
>>>> >> > Saurabh.
>>>> >> >
>>>> >> > On Wed, Dec 30, 2009 at 11:48 AM, Saurabh Nanda <
>>>> saurabhnanda@gmail.com>
>>>> >> > wrote:
>>>> >> >>
>>>> >> >> I'm taking a look at the HDFS directories through the web
>>>> interface and
>>>> >> >> I
>>>> >> >> can see only 5 files there, not 6. I tried creating the
partition
>>>> using
>>>> >> >> the
>>>> >> >> ADD PARTITION command. After that all 6 files get imported
>>>> >> >> successfully.
>>>> >> >>
>>>> >> >> Saurabh.
>>>> >> >>
>>>> >> >> On Wed, Dec 30, 2009 at 11:39 AM, Zheng Shao <zshao9@gmail.com>
>>>> wrote:
>>>> >> >>>
>>>> >> >>> Can you list the HDFS directories? Are the files in
the
>>>> corresponding
>>>> >> >>> directories yet?
>>>> >> >>>
>>>> >> >>>
>>>> >> >>> Zheng
>>>> >> >>>
>>>> >> >>> On Tue, Dec 29, 2009 at 9:57 PM, Saurabh Nanda
>>>> >> >>> <saurabhnanda@gmail.com>
>>>> >> >>> wrote:
>>>> >> >>> > Hi,
>>>> >> >>> >
>>>> >> >>> > I'm revisiting Hive after a long hiatus, so I
may not be aware
>>>> of
>>>> >> >>> > any
>>>> >> >>> > new
>>>> >> >>> > developments. I had written a script some time
back to import
>>>> >> >>> > webserver
>>>> >> >>> > logs
>>>> >> >>> > for a day into a new partition. The same script
now running on
>>>> the
>>>> >> >>> > latest
>>>> >> >>> > version of Hive (r894548 compiled off trunk) seems
to be
>>>> >> >>> > misbehaving.
>>>> >> >>> >
>>>> >> >>> > I'm importing about 6 files into each partition.
However, after
>>>> the
>>>> >> >>> > script
>>>> >> >>> > ends, only 5 files show up in each partition.
Do I need to
>>>> >> >>> > explicitly
>>>> >> >>> > issue
>>>> >> >>> > the ADD PARTITION command before loading data?
Isn't the
>>>> partition
>>>> >> >>> > implicitly created?
>>>> >> >>> >
>>>> >> >>> > Saurabh.
>>>> >> >>> > --
>>>> >> >>> > http://nandz.blogspot.com
>>>> >> >>> > http://foodieforlife.blogspot.com
>>>> >> >>> >
>>>> >> >>>
>>>> >> >>>
>>>> >> >>>
>>>> >> >>> --
>>>> >> >>> Yours,
>>>> >> >>> Zheng
>>>> >> >>
>>>> >> >>
>>>> >> >>
>>>> >> >> --
>>>> >> >> http://nandz.blogspot.com
>>>> >> >> http://foodieforlife.blogspot.com
>>>> >> >
>>>> >> >
>>>> >> >
>>>> >> > --
>>>> >> > http://nandz.blogspot.com
>>>> >> > http://foodieforlife.blogspot.com
>>>> >> >
>>>> >>
>>>> >>
>>>> >>
>>>> >> --
>>>> >> Yours,
>>>> >> Zheng
>>>> >
>>>> >
>>>> >
>>>> > --
>>>> > http://nandz.blogspot.com
>>>> > http://foodieforlife.blogspot.com
>>>> >
>>>>
>>>>
>>>>
>>>> --
>>>> Yours,
>>>> Zheng
>>>>
>>>
>>>
>>>
>>> --
>>> http://nandz.blogspot.com
>>> http://foodieforlife.blogspot.com
>>>
>>
>>
>>
>> --
>> http://nandz.blogspot.com
>> http://foodieforlife.blogspot.com
>>
>
>
>
> --
> http://nandz.blogspot.com
> http://foodieforlife.blogspot.com
>

Mime
View raw message