Mailing-List: contact user-help@pig.apache.org; run by ezmlm
Precedence: bulk
Reply-To: user@pig.apache.org
Received-SPF: pass (nike.apache.org: domain of jeremy.hanna1234@gmail.com
 designates 209.85.160.177 as permitted sender)
DomainKey-Signature: a=rsa-sha1; c=nofws;
        d=gmail.com; s=gamma;
        h=content-type:mime-version:subject:from:in-reply-to:date
         :content-transfer-encoding:message-id:references:to:x-mailer;
        b=wZbpos6mJIpTa2mJ7jUFTKoOHmSghxZrONsq4bT0No75n3p+XFod8B6bTO/WUGvl3P
         nB3em/jhlsoFuMNDQIhYvmqA2FDPBGuG4cswB2K1noKIIKj6PCknJgEPv4ZgAG0FTPYx
         C5tUwgI5CO5mslgMob8jfQRPlfHzv4xBWIb0g=
Content-Type: text/plain; charset=us-ascii
Mime-Version: 1.0 (Apple Message framework v1084)
Subject: Re: prep for cassandra storage from pig
From: Jeremy Hanna <jeremy.hanna1234@gmail.com>
In-Reply-To: <BANLkTim01V37vHO1dnO_uVuOzo4g_PGjeg@mail.gmail.com>
Date: Wed, 15 Jun 2011 14:04:02 -0500
Content-Transfer-Encoding: quoted-printable
Message-Id: <7F9E700F-C655-4639-A830-0DF405EC2CDB@gmail.com>
References: <BANLkTim01V37vHO1dnO_uVuOzo4g_PGjeg@mail.gmail.com>
To: user@cassandra.apache.org,
 user@pig.apache.org

Hi Will,

That's partly why I like to use FromCassandraBag and ToCassandraBag from =
pygmalion - it does the work for you to get it back into a form that =
cassandra understands.

Others may know better how to massage the data into that form using just =
pig, but if all else fails, you could write a udf to do that.

Jeremy

On Jun 15, 2011, at 1:17 PM, William Oberman wrote:

> I think I'm stuck on typing issues trying to store data in cassandra.  =
To verify, cassandra wants (key, {tuples})
>=20
> My pig script is fairly brief:
> raw =3D LOAD 'cassandra://test_in/test_cf' USING CassandraStorage() AS =
(key:chararray, columns:bag {column:tuple (name, value)});
> --colums =3D=3D timeUUID -> JSON
> rows =3D FOREACH raw GENERATE key, FLATTEN(columns);
> alias_target_day =3D FOREACH rows {
>     --I wrote a specialized parser that does exactly what I need
>     observation_map =3D com.civicscience.pig.ParseObservation($2);
>     GENERATE $0 as alias, observation_map#'_fqt' as target, =
observation_map#'_day' as day;
> };
> grouping =3D GROUP alias_target_day BY =
((chararray)target,(chararray)day);
> X =3D FOREACH grouping GENERATE group.$0 as target, TOTUPLE(group.$1, =
COUNT($1)) as day_count;
>=20
> This gets me:
> (targetA, (day1, count))
> (targetA, (day2, count))
> (targetB, (day1, count))
> ....
>=20
> But, cassandra wants the 2nd item to be a bag.  So, I tried:
> X =3D FOREACH grouping GENERATE group.$0 as target, =
TOBAG(TOTUPLE(group.$1, COUNT($1))) as day_count;
>=20
> But this results in:
> (targetA, {((day1, count))})
> (targetA, {((day2, count))})
> (targetB, {((day1, count))})
> It's hard to see, but the 2nd item now has a nested tuple as the first =
value, which is still bad.
>=20
> How to I get (key, {tuple})???  I wasn't sure where to post this (pig =
or cassandra), so I'm posting to the pig list too.
>=20
> will