Mailing-List: contact user-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@hadoop.apache.org
Received-SPF: pass (nike.apache.org: local policy)
Message-ID: <54203E79.7040202@vesseltracker.com>
Date: Mon, 22 Sep 2014 17:21:29 +0200
From: Georgi Ivanov <ivanov@vesseltracker.com>
User-Agent: Mozilla/5.0 (X11; Linux x86_64;
 rv:31.0) Gecko/20100101 Thunderbird/31.1.1
MIME-Version: 1.0
To: user@hadoop.apache.org
Subject: Re: Bzip2 files as an input to MR job
References: <542034F6.2020903@vesseltracker.com>
 <CADoiZqoMYXMawp=mayL7hi6ZxF2F=-2PkVrqyJ5ZaWOzpyoK3Q@mail.gmail.com>
In-Reply-To: 
 <CADoiZqoMYXMawp=mayL7hi6ZxF2F=-2PkVrqyJ5ZaWOzpyoK3Q@mail.gmail.com>
Content-Type: multipart/alternative;
 boundary="------------010004040501090608040103"

This is a multi-part message in MIME format.
--------------010004040501090608040103
Content-Type: text/plain; charset=utf-8; format=flowed
Content-Transfer-Encoding: 7bit

Hi Niels,
Thanks for the reply.
Changing the avro files is not really an option for me as it will 
require a lot of time( i have a lot ).
The Avro files themself are compressed a bit.
But still bzip2 gives 50% compression on one avro file.

So what i want is , to use Bzip2 compressed file as an input to my MR jobs.
Bzip2 is splittable.
Should be possible somehow , but i don't seem to find it atm.

On 22.09.2014 17:13, Niels Basjes wrote:
> Hi,
>
> You can use the GZip inside the AVRO files and still have splittable 
> AVRO files.
> This has the to with the fact that there is a block structure inside 
> the AVRO and these blocks are gzipped.
>
> I suggest you simply try it.
>
> Niels
>
>
> On Mon, Sep 22, 2014 at 4:40 PM, Georgi Ivanov 
> <ivanov@vesseltracker.com <mailto:ivanov@vesseltracker.com>> wrote:
>
>     Hi guys,
>     I would like to compress the files on HDFS to save some storage.
>
>     As far as i see bzip2 is the only format which is splitable (and
>     slow).
>
>     The actual files are Avro.
>
>     So in my driver class i have :
>
>     job.setInputFormatClass(AvroKeyInputFormat.class);
>
>     I have number of jobs running processing Avro files so i would
>     like to keep the code change to a minimum.
>
>     Is it possible to comrpess these avro files with bzip2 and keep
>     the code of MR jobs the same (or with little change)
>     If it is , please give me some hints as so far i don't seem to
>     find any good resources on the Internet.
>
>
>     Georgi
>
>
>
>
> -- 
> Best regards / Met vriendelijke groeten,
>
> Niels Basjes


--------------010004040501090608040103
Content-Type: text/html; charset=utf-8
Content-Transfer-Encoding: 7bit

<html>
  <head>
    <meta content="text/html; charset=utf-8" http-equiv="Content-Type">
  </head>
  <body bgcolor="#FFFFFF" text="#000000">
    <div class="moz-cite-prefix">Hi Niels,<br>
      Thanks for the reply.<br>
      Changing the avro files is not really an option for me as it will
      require a lot of time( i have a lot ).<br>
      The Avro files themself are compressed a bit.<br>
      But still bzip2 gives 50% compression on one avro file.<br>
      <br>
      So what i want is , to use Bzip2 compressed file as an input to my
      MR jobs.<br>
      Bzip2 is splittable.<br>
      Should be possible somehow , but i don't seem to find it atm.<br>
      <br>
      On 22.09.2014 17:13, Niels Basjes wrote:<br>
    </div>
    <blockquote
cite="mid:CADoiZqoMYXMawp=mayL7hi6ZxF2F=-2PkVrqyJ5ZaWOzpyoK3Q@mail.gmail.com"
      type="cite">
      <div dir="ltr">
        <div>Hi,</div>
        <div><br>
        </div>
        You can use the GZip inside the AVRO files and still have
        splittable AVRO files.
        <div>This has the to with the fact that there is a block
          structure inside the AVRO and these blocks are gzipped.<br>
        </div>
        <div class="gmail_extra"><br>
        </div>
        <div class="gmail_extra">I suggest you simply try it.</div>
        <div class="gmail_extra"><br>
        </div>
        <div class="gmail_extra">Niels</div>
        <div class="gmail_extra"><br>
        </div>
        <div class="gmail_extra"><br>
          <div class="gmail_quote">On Mon, Sep 22, 2014 at 4:40 PM,
            Georgi Ivanov <span dir="ltr">&lt;<a moz-do-not-send="true"
                href="mailto:ivanov@vesseltracker.com" target="_blank">ivanov@vesseltracker.com</a>&gt;</span>
            wrote:<br>
            <blockquote class="gmail_quote" style="margin:0 0 0
              .8ex;border-left:1px #ccc solid;padding-left:1ex">Hi guys,<br>
              I would like to compress the files on HDFS to save some
              storage.<br>
              <br>
              As far as i see bzip2 is the only format which is
              splitable (and slow).<br>
              <br>
              The actual files are Avro.<br>
              <br>
              So in my driver class i have :<br>
              <br>
              job.setInputFormatClass(AvroKeyInputFormat.class);<br>
              <br>
              I have number of jobs running processing Avro files so i
              would like to keep the code change to a minimum.<br>
              <br>
              Is it possible to comrpess these avro files with bzip2 and
              keep the code of MR jobs the same (or with little change)<br>
              If it is , please give me some hints as so far i don't
              seem to find any good resources on the Internet.<span
                class="HOEnZb"><font color="#888888"><br>
                  <br>
                  <br>
                  Georgi<br>
                </font></span></blockquote>
          </div>
          <br>
          <br clear="all">
          <div><br>
          </div>
          -- <br>
          Best regards / Met vriendelijke groeten,<br>
          <br>
          Niels Basjes
        </div>
      </div>
    </blockquote>
    <br>
  </body>
</html>

--------------010004040501090608040103--