Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: domain of harsh@cloudera.com designates
 209.85.223.181 as permitted sender)
MIME-Version: 1.0
In-Reply-To: <5FD6A36D-4F5D-4ECB-83A2-CFBFFB321FA7@gmail.com>
References: <3F80E783-79EE-4D10-8E84-C5881B9D56EF@gmail.com>
 <CAOcnVr22dUyR+KGW1XzNFMBm+3VomdPQf6X=w2oyGFHSdE41Ng@mail.gmail.com>
 <09674432-2891-4729-979A-3E41D0A7F361@gmail.com>
 <5FD6A36D-4F5D-4ECB-83A2-CFBFFB321FA7@gmail.com>
From: Harsh J <harsh@cloudera.com>
Date: Tue, 9 Apr 2013 22:55:37 +0530
Message-ID: 
 <CAOcnVr0LdCd59kzC_-GmUY81Qq-C5x8=a=2gpzyFM4BuHJf3jg@mail.gmail.com>
Subject: Re: Best format to use
To: "<user@hadoop.apache.org>" <user@hadoop.apache.org>
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: quoted-printable

Pig and Hive both have support for compressed sequence files.

Regarding best format - if its just text log data (i.e. no
types/structure) then the best format to keep it in is in
text+compress. SequenceFiles help make it splittable but add a small
overhead in space and efficiency and none of the good codecs out there
are splittable on their own for compression (LZO is good, but needs
pre-indexing to be viewed splittable).

On Tue, Apr 9, 2013 at 10:21 PM, Mark <static.void.dev@gmail.com> wrote:
> Actually, compressed sequence files may not work with Pig or Hive then ri=
ght?
>
> On Apr 9, 2013, at 9:50 AM, Mark <static.void.dev@gmail.com> wrote:
>
>> Forgetting Impala, what format would be best to use with daily logs?
>>
>> Block-compressed sequence files?
>>
>> On Apr 8, 2013, at 8:12 PM, Harsh J <harsh@cloudera.com> wrote:
>>
>>> Hey Mark,
>>>
>>> Gzip codec creates extension .gzip, not .deflate (which is
>>> DeflateCodec). You may want to re-check your settings.
>>>
>>> Impala questions are best resolved at its current user and developer
>>> community at https://groups.google.com/a/cloudera.org/forum/#!forum/imp=
ala-user.
>>> Impala does currently support LZO (and also Indexed LZO) compressed
>>> text files however, so you may want to try that as its splittable
>>> (compared to Gzip ones).
>>>
>>> On Tue, Apr 9, 2013 at 5:18 AM, Mark <static.void.dev@gmail.com> wrote:
>>>> Trying to determine what the best format to use for storing daily logs=
. We recently switch from snappy (.snappy) to gzip (.deflate) but I'm wonde=
ring if there is something better? Our main clients for these daily logs ar=
e pig and hive using an external table. We were thinking about testing out =
impala but we see that it doesn't work with compressed text files. Any sugg=
estions?
>>>>
>>>> Thanks
>>>
>>>
>>>
>>> --
>>> Harsh J
>>
>


--=20
Harsh J