hive-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hamza Asad <hamza.asa...@gmail.com>
Subject Re: How to delete Specific date data using hive QL?
Date Tue, 04 Jun 2013 07:52:49 GMT
Thank u soooo much nitin for your help.. :)


On Tue, Jun 4, 2013 at 12:18 PM, Nitin Pawar <nitinpawar432@gmail.com>wrote:

> 1- Does partitioning improve performance?
> --Only if you make use of partitions in your queries (mostly in where
> clause to limit data to your query for a specific value of partitioned
> column)
>
> 2- Do i have to create partition table new or i can create partition on
> existing table by renaming that date column and add partition column
> event_date (the actual column name) ?
> you can not create partitions on already existing data unless the data is
> in partitioned directories on hdfs.
> I would recommend create a new table with partitioned columns.
> load data from old table into partitioned table
> dump old table
>
> 3- can i import data directly into partition table using sqoop command?
> you can import data directly into a partition.
>
> for exported data, you don't have to worry. it remains as it is
>
>
> On Tue, Jun 4, 2013 at 12:41 PM, Hamza Asad <hamza.asad13@gmail.com>wrote:
>
>> No i don't want to change my queries. I want that my queries work on same
>> table and partition does not change its schema.
>> and from schema i means schema on mysql (exported data).
>>
>> Few more things
>> 1- Does partitioning improve performance?
>> 2- Do i have to create partition table new or i can create partition on
>> existing table by renaming that date column and add partition column
>> event_date (the actual column name) ?
>> 3- can i import data directly into partition table using sqoop command?
>>
>>
>>
>>
>> On Tue, Jun 4, 2013 at 11:40 AM, Nitin Pawar <nitinpawar432@gmail.com>wrote:
>>
>>> partitioning of data in hive is more for the reasons on how you layout
>>> data in a well defined manner so that when you access your data , you
>>> request only for specific data by specifying the partition columns in where
>>> clause.
>>>
>>> to answer your question,
>>> do you have to change your queries? out of the box the queries should
>>> work as it is unless and until you are changing the table schema by
>>> removing/adding new columns.
>>> does the format change when you export data? if your select statement is
>>> not changing it will not change
>>> will table schema change? do you mean schema on hive or mysql ?
>>>
>>>
>>> On Tue, Jun 4, 2013 at 11:37 AM, Hamza Asad <hamza.asad13@gmail.com>wrote:
>>>
>>>> thats far more better :) ..
>>>> Please tell me few more things. Do i have to change my query if i
>>>> create table with partition on date? rest of the columns would be same as
>>>> it is? Also if i export that partitioned table to mysql, does schema of
>>>> that table would same as it was before partition?
>>>>
>>>>
>>>> On Tue, Jun 4, 2013 at 12:09 AM, Stephen Sprague <spragues@gmail.com>wrote:
>>>>
>>>>> there is no delete semantic.
>>>>>
>>>>> you either partition on the data you want to drop and use drop
>>>>> partition (or drop table for the whole shebang) or you can do as Nitin
>>>>> suggests by selecting the inverse of the data you want to delete and
store
>>>>> it back into the table itself.  Not ideal but maybe it could work for
your
>>>>> situation.
>>>>>
>>>>> Now here's another idea.  This was just _recently_ discussed on this
>>>>> group as coincidence would have it.  if you were to have scanned just
a
>>>>> little of the groups messages you would have seen that and could then
have
>>>>> added to the discussion! :)
>>>>>
>>>>>
>>>>> On Mon, Jun 3, 2013 at 2:19 AM, Hamza Asad <hamza.asad13@gmail.com>wrote:
>>>>>
>>>>>> Thanx for your response nitin. Anybody else have any better solution?
>>>>>>
>>>>>>
>>>>>> On Mon, Jun 3, 2013 at 1:27 PM, Nitin Pawar <nitinpawar432@gmail.com>wrote:
>>>>>>
>>>>>>> hive does not give you a record level deletion as of now.
>>>>>>>
>>>>>>> so unless you have partitioned, other option is you overwrite
the
>>>>>>> table with data which you want
>>>>>>> please wait for others to suggest you more options. this one
is just
>>>>>>> mine and can be costly too
>>>>>>>
>>>>>>>
>>>>>>> On Mon, Jun 3, 2013 at 12:36 PM, Hamza Asad <hamza.asad13@gmail.com>wrote:
>>>>>>>
>>>>>>>> no, its not partitioned by date.
>>>>>>>>
>>>>>>>>
>>>>>>>> On Mon, Jun 3, 2013 at 11:19 AM, Nitin Pawar <
>>>>>>>> nitinpawar432@gmail.com> wrote:
>>>>>>>>
>>>>>>>>> how is the data laid out?
>>>>>>>>> is it partitioned data by the date?
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> On Mon, Jun 3, 2013 at 11:20 AM, Hamza Asad <
>>>>>>>>> hamza.asad13@gmail.com> wrote:
>>>>>>>>>
>>>>>>>>>> Dear all,
>>>>>>>>>>             How can i remove data of specific dates
from HDFS
>>>>>>>>>> using hive query language?
>>>>>>>>>>
>>>>>>>>>> --
>>>>>>>>>> *Muhammad Hamza Asad*
>>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>>
>>>>>>>>> --
>>>>>>>>> Nitin Pawar
>>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>>
>>>>>>>> --
>>>>>>>> *Muhammad Hamza Asad*
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> --
>>>>>>> Nitin Pawar
>>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> --
>>>>>> *Muhammad Hamza Asad*
>>>>>>
>>>>>
>>>>>
>>>>
>>>>
>>>> --
>>>> *Muhammad Hamza Asad*
>>>>
>>>
>>>
>>>
>>> --
>>> Nitin Pawar
>>>
>>
>>
>>
>> --
>> *Muhammad Hamza Asad*
>>
>
>
>
> --
> Nitin Pawar
>



-- 
*Muhammad Hamza Asad*

Mime
View raw message