Mailing-List: contact core-dev-help@hadoop.apache.org; run by ezmlm
Precedence: bulk
Reply-To: core-dev@hadoop.apache.org
Received-SPF: pass (athena.apache.org: domain of jay.kreps@gmail.com
 designates 72.14.220.156 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=message-id:date:from:to:subject:mime-version:content-type
         :content-transfer-encoding:content-disposition;
        b=UA5oC3U18ajg/JF2TGF3VcVCFM7gXb4a19hsVh9LJBK2eJ49p1nWIbKitKVdmoqWRU
         esUBuVi/Z7sIaMaNxuqOcBc2h7KL8y79Lrn61XusgYuDmR33Lc4LZMAcagCiBpdhIPFz
         1np9ThNctdXIhkv1bWJWlesY6xdXOXS9eJeus=
Message-ID: <83290b460809040735m1bb1f315l5b3ae61551382406@mail.gmail.com>
Date: Thu, 4 Sep 2008 07:35:39 -0700
From: "Jay Kreps" <jay.kreps@gmail.com>
To: ted.dunning@gmail.com, core-dev@hadoop.apache.org
Subject: Re: Serialization with additional schema info
MIME-Version: 1.0
Content-Type: text/plain; charset=ISO-8859-1
Content-Transfer-Encoding: 7bit
Content-Disposition: inline

Yes, I mean this is just the trade-off between structured and
unstructured data.  In my case 99% of my data sources are structured.
So if I am expecting List<String> and get List<Integer> then something
is broken and I want to catch the bug before someone writes the bad
data. I agree that in principle a compression algorithm should be able
to give me comparable compactness with some CPU trade-off.

-Jay

---------- Forwarded message ----------
From: "Ted Dunning" <ted.dunning@gmail.com>
To: core-dev@hadoop.apache.org
Date: Wed, 3 Sep 2008 21:24:00 -0700
Subject: Re: Serialization with additional schema info
I talked to the IBM guys about this problem with JSON-like formats.

Their answer was that if you care enough, then any compression algorithm
around will compress away the type information.

So if you have a splittable compressed format (bz2 works with hadoop), you
are set except for the compression cost.  Decompression cost is usually
compensated for by the I/O advantage.

On Wed, Sep 3, 2008 at 3:52 PM, Jay Kreps <jay.kreps@gmail.com> wrote:

> ...
>
> Thanks for the pointer to jaql, that seems very cool, but I believe
> jaql would have the same problem if they tried to implement any kind
> of compact structured storage.  Jaql would return a JArray or JRecord
> which might have a variety of fields and you would want to store the
> data about what kinds of fields separately.
>