incubator-cassandra-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Miguel Angel Martin junquera <mianmarjun.mailingl...@gmail.com>
Subject Re: CqlStorage creates wrong schema for Pig
Date Wed, 04 Sep 2013 10:06:22 GMT
Oppps,   sorry by  my oversight

I was checking the code and  I was surprised it did not work with that pig
script ...

now , It works fine ..

Many thanks,Chad

Have a nice day


Miguel Angel Martín Junquera
Analyst Engineer.
miguelangel.martin@brainsins.com



2013/9/3 Chad Johnston <cjohnston@megatome.com>

> You're trying to use FromCqlColumn on a tuple that has been flattened. The
> schema still thinks it's {title: chararray}, but the flattened tuple is now
> two values. I don't know how to retrieve the data values in this case.
>
> Your code will work correctly if you do this:
> *values3 = FOREACH rows GENERATE FromCqlColumn(title) AS title;*
> *dump values3;*
> *describe values3;*
>
> (Use FromCqlColumn on the original data, not the flattened data.)
>
> Chad
>
>
> On Mon, Sep 2, 2013 at 8:45 AM, Miguel Angel Martin junquera <
> mianmarjun.mailinglist@gmail.com> wrote:
>
>> Hi
>>
>>
>> 1.-
>>
>> May be?
>>
>> -- Register the UDF
>> REGISTER /path/to/cqlstorageudf-1.0-SNAPSHOT
>>
>> -- FromCqlColumn will convert chararray, int, long, float, double
>> DEFINE FromCqlColumn com.megatome.pig.piggybank.tuple.FromCqlColumn();
>>
>> -- Load data as normal
>> data_raw = LOAD 'cql://bookcrossing/books' USING CqlStorage();
>>
>> -- Use the UDF
>> data = FOREACH data_raw GENERATE
>>     *FromCqlColumn*(isbn) AS ISBN,
>>     *FromCqlColumn*(bookauthor) AS BookAuthor,
>>
>>
>>     *FromCqlColumn*(booktitle) AS BookTitle,
>>     *FromCqlColumn*(publisher) AS Publisher,
>>
>>
>>     *FromCqlColumn*(yearofpublication) AS YearOfPublication;
>>
>>
>>
>>
>>
>> and  2.:
>>
>> with  the data in cql cassandra 1.2.8, pig 0.11.11 and cql3:
>>
>> *CREATE KEYSPACE keyspace1*
>>
>> *  WITH REPLICATION = { 'class' : 'SimpleStrategy', 'replication_factor'
>> : 1 }*
>>
>> *  AND durable_writes = true;*
>>
>> *
>> *
>>
>> *use keyspace2;*
>>
>>  *
>> *
>>
>> *  CREATE TABLE test (*
>>
>> *    id text PRIMARY KEY,*
>>
>> *    title text,*
>>
>> *    age int*
>>
>> *  )  WITH COMPACT STORAGE;*
>>
>> *
>> *
>>
>> *
>> *
>>
>> *  insert into test (id, title, age) values('1', 'child', 21);*
>>
>> *  insert into test (id, title, age) values('2', 'support', 21);*
>>
>> *  insert into test (id, title, age) values('3', 'manager', 31);*
>>
>> *  insert into test (id, title, age) values('4', 'QA', 41);*
>>
>> *  insert into test (id, title, age) values('5', 'QA', 30);*
>>
>> *  insert into test (id, title, age) values('6', 'QA', 30);*
>>
>>
>>
>>
>>
>> and script:
>>
>> *
>> *
>> *register './libs/cqlstorageudf-1.0-SNAPSHOT.jar';*
>> *DEFINE FromCqlColumn com.megatome.pig.piggybank.tuple.FromCqlColumn();*
>> *rows = LOAD
>> 'cql://keyspace1/test?page_size=1&split_size=4&where_clause=age%3D30' USING
>> CqlStorage();*
>> *dump rows;*
>> *ILLUSTRATE rows;*
>> *describe rows;*
>> *A = FOREACH rows GENERATE FLATTEN(title);*
>> *dump A;*
>> *values3 = FOREACH A GENERATE FromCqlColumn(title) AS title;*
>> *dump values3;*
>> *describe values3;*
>>
>>
>> --
>>
>>
>>
>> I have this error:
>>
>>
>>
>>
>> ....
>>
>> -------------------------------------------------------------
>> | rows     | id:chararray   | age:int   | title:chararray   |
>> -------------------------------------------------------------
>> |          | (id, 5)        | (age, 30) | (title, QA)       |
>> -------------------------------------------------------------
>>
>> rows: {id: chararray,age: int,title: chararray}
>>
>>
>> ...
>>
>> (title,QA)
>> (title,QA)
>> ..
>> 2013-09-02 16:40:52,454 [Thread-11] WARN
>>  org.apache.hadoop.mapred.LocalJobRunner - job_local_0003
>> *java.lang.ClassCastException: java.lang.String cannot be cast to
>> org.apache.pig.data.Tuple*
>> at com.megatome.pig.piggybank.tuple.ColumnBase.exec(ColumnBase.java:32)
>>  at
>> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:337)
>>  at
>> org.apache.pig.backend.hadoop.executionengine.physicalLayer.expressionOperators.POUserFunc.getNext(POUserFunc.java:434)
>>  at
>> org.apache.pig.backend.hadoop.executionengine.physicalLayer.PhysicalOperator.getNext(PhysicalOperator.java:340)
>>  at
>> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.processPlan(POForEach.java:372)
>>  at
>> org.apache.pig.backend.hadoop.executionengine.physicalLayer.relationalOperators.POForEach.getNext(POForEach.java:297)
>> at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.runPipeline(PigGenericMapBase.java:283)
>>  at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:278)
>>  at
>> org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.PigGenericMapBase.map(PigGenericMapBase.java:64)
>> at org.apache.hadoop.mapreduce.Mapper.run(Mapper.java:144)
>>  at org.apache.hadoop.mapred.MapTask.runNewMapper(MapTask.java:764)
>> at org.apache.hadoop.mapred.MapTask.run(MapTask.java:370)
>>  at
>> org.apache.hadoop.mapred.LocalJobRunner$Job.run(LocalJobRunner.java:212)
>> 2013-09-02 16:40:52,832 [main] INFO
>>  org.apache.pig.backend.hadoop.executionengine.mapReduceLayer.MapReduceLauncher
>> - HadoopJobId: job_local_0003
>>
>>
>>
>> 8-|
>>
>> Regards
>>
>> ...
>>
>>
>> Miguel Angel Martín Junquera
>> Analyst Engineer.
>> miguelangel.martin@brainsins.com
>>
>>
>>
>> 2013/9/2 Miguel Angel Martin junquera <mianmarjun.mailinglist@gmail.com>
>>
>>> hi all:
>>>
>>> More info :
>>>
>>> https://issues.apache.org/jira/browse/CASSANDRA-5941
>>>
>>>
>>>
>>> I tried this (and gen. cassandra 1.2.9)  but do not work for me,
>>>
>>> git clone http://git-wip-us.apache.org/repos/asf/cassandra.git
>>> cd cassandra
>>> git checkout cassandra-1.2
>>> patch -p1 < 5867-bug-fix-filter-push-down-1.2-branch.txt
>>> ant
>>>
>>>
>>>
>>> Miguel Angel Martín Junquera
>>> Analyst Engineer.
>>> miguelangel.martin@brainsins.com
>>>
>>>
>>>
>>> 2013/9/2 Miguel Angel Martin junquera <mianmarjun.mailinglist@gmail.com>
>>>
>>>> *good/nice job !!!*
>>>> *
>>>> *
>>>> *
>>>> *
>>>> *I'd testing with an udf only with  string schema type  this is better
>>>> and elaborate work..*
>>>> *
>>>> *
>>>> *Regads*
>>>>
>>>>
>>>> Miguel Angel Martín Junquera
>>>> Analyst Engineer.
>>>> miguelangel.martin@brainsins.com
>>>>
>>>>
>>>>
>>>> 2013/8/31 Chad Johnston <cjohnston@megatome.com>
>>>>
>>>>> I threw together a quick UDF to work around this issue. It just
>>>>> extracts the value portion of the tuple while taking advantage of the
>>>>> CqlStorage generated schema to keep the type correct.
>>>>>
>>>>> You can get it here: https://github.com/iamthechad/cqlstorage-udf
>>>>>
>>>>> I'll see if I can find more useful information and open a defect,
>>>>> since that's what this seems to be.
>>>>>
>>>>> Chad
>>>>>
>>>>>
>>>>> On Fri, Aug 30, 2013 at 2:02 AM, Miguel Angel Martin junquera <
>>>>> mianmarjun.mailinglist@gmail.com> wrote:
>>>>>
>>>>>> I try this:
>>>>>>
>>>>>>  *rows = LOAD
>>>>>> 'cql://keyspace1/test?page_size=1&split_size=4&where_clause=age%3D30'
USING
>>>>>> CqlStorage();*
>>>>>>
>>>>>> *dump rows;*
>>>>>>
>>>>>> *ILLUSTRATE rows;*
>>>>>>
>>>>>> *describe rows;*
>>>>>>
>>>>>> *
>>>>>> *
>>>>>>
>>>>>> *values2= FOREACH rows GENERATE  TOTUPLE (id) as
>>>>>> (mycolumn:tuple(name,value));*
>>>>>>
>>>>>> *dump values2;*
>>>>>>
>>>>>> *describe values2;*
>>>>>> *
>>>>>> *
>>>>>>
>>>>>> But I get this results:
>>>>>>
>>>>>>
>>>>>>
>>>>>> -------------------------------------------------------------
>>>>>> | rows     | id:chararray   | age:int   | title:chararray   |
>>>>>> -------------------------------------------------------------
>>>>>> |          | (id, 6)        | (age, 30) | (title, QA)       |
>>>>>> -------------------------------------------------------------
>>>>>>
>>>>>> rows: {id: chararray,age: int,title: chararray}
>>>>>> 2013-08-30 09:54:37,831 [main] ERROR org.apache.pig.tools.grunt.Grunt
>>>>>> - ERROR 1031: Incompatable field schema: left is
>>>>>> "tuple_0:tuple(mycolumn:tuple(name:bytearray,value:bytearray))",
right is
>>>>>> "org.apache.pig.builtin.totuple_id_1:tuple(id:chararray)"
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> or
>>>>>>
>>>>>>
>>>>>>
>>>>>> ....
>>>>>>
>>>>>> *values2= FOREACH rows GENERATE  TOTUPLE (id) ;*
>>>>>> *dump values2;*
>>>>>> *describe values2;*
>>>>>>
>>>>>>
>>>>>>
>>>>>>
>>>>>> and  the results are:
>>>>>>
>>>>>>
>>>>>> ...
>>>>>> (((id,6)))
>>>>>> (((id,5)))
>>>>>> values2: {org.apache.pig.builtin.totuple_id_8: (id: chararray)}
>>>>>>
>>>>>>
>>>>>>
>>>>>> Aggg!!!!!
>>>>>>
>>>>>>
>>>>>> *
>>>>>> *
>>>>>>
>>>>>>
>>>>>>
>>>>>> Miguel Angel Martín Junquera
>>>>>> Analyst Engineer.
>>>>>> miguelangel.martin@brainsins.com
>>>>>>
>>>>>>
>>>>>>
>>>>>> 2013/8/26 Miguel Angel Martin junquera <
>>>>>> mianmarjun.mailinglist@gmail.com>
>>>>>>
>>>>>>> hi Chad .
>>>>>>>
>>>>>>> I have this issue
>>>>>>>
>>>>>>> I send a mail to user-pig-list and  I still i can resolve this,
and
>>>>>>> I can not  access to column values.
>>>>>>> In this mail  I write some things that I try without results...
and
>>>>>>> information about this issue.
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> http://mail-archives.apache.org/mod_mbox/pig-user/201308.mbox/%3CCAJeG_hQ9S2Po3_XytZX5Xki4J1maO8q26jYdG2Wndy_KYiv9CQ@mail.gmail.com%3E
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> I hope  someOne reply  one comment, idea or  solution about 
this
>>>>>>> issue or bug.
>>>>>>>
>>>>>>>
>>>>>>> I have reviewed the CqlStorage class in code cassandra 1.2.8
 but i
>>>>>>> do not have configure the environmetn to debug  and trace this
issue.
>>>>>>>
>>>>>>> Only  I find some comments like, but I do not understand at all.
>>>>>>>
>>>>>>>
>>>>>>> /**
>>>>>>>
>>>>>>>  * A LoadStoreFunc for retrieving data from and storing data
to
>>>>>>> Cassandra
>>>>>>>
>>>>>>>  *
>>>>>>>
>>>>>>>  * A row from a standard CF will be returned as nested tuples:
>>>>>>>
>>>>>>>  * (((key1, value1), (key2, value2)), ((name1, val1), (name2,
>>>>>>> val2))).
>>>>>>>  */
>>>>>>>
>>>>>>>
>>>>>>> I you found some idea or solution, please post it
>>>>>>>
>>>>>>> thanks
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>> 2013/8/23 Chad Johnston <cjohnston@megatome.com>
>>>>>>>
>>>>>>>> (I'm using Cassandra 1.2.8 and Pig 0.11.1)
>>>>>>>>
>>>>>>>> I'm loading some simple data from Cassandra into Pig using
>>>>>>>> CqlStorage. The CqlStorage loader defines a Pig schema based
on the
>>>>>>>> Cassandra schema, but it seems to be wrong.
>>>>>>>>
>>>>>>>> If I do:
>>>>>>>>
>>>>>>>> data = LOAD 'cql://bookdata/books' USING CqlStorage();
>>>>>>>> DESCRIBE data;
>>>>>>>>
>>>>>>>> I get this:
>>>>>>>>
>>>>>>>> data: {isbn: chararray,bookauthor: chararray,booktitle:
>>>>>>>> chararray,publisher: chararray,yearofpublication: int}
>>>>>>>>
>>>>>>>> However, if I DUMP data, I get results like these:
>>>>>>>>
>>>>>>>> ((isbn,0425093387),(bookauthor,Georgette Heyer),(booktitle,Death
in
>>>>>>>> the Stocks),(publisher,Berkley Pub Group),(yearofpublication,1986))
>>>>>>>>
>>>>>>>> Clearly the results from Cassandra are key/value pairs, as
would be
>>>>>>>> expected. I don't know why the schema generated by CqlStorage()
would be so
>>>>>>>> different.
>>>>>>>>
>>>>>>>> This is really causing me problems trying to access the column
>>>>>>>> values. I tried a naive approach of FLATTENing each tuple,
then trying to
>>>>>>>> access the values that way:
>>>>>>>>
>>>>>>>> flattened = FOREACH data GENERATE
>>>>>>>>   FLATTEN(isbn),
>>>>>>>>   FLATTEN(booktitle),
>>>>>>>>   ...
>>>>>>>> values = FOREACH flattened GENERATE
>>>>>>>>   $1 AS ISBN,
>>>>>>>>   $3 AS BookTitle,
>>>>>>>>   ...
>>>>>>>>
>>>>>>>> As soon as I try to access field $5, Pig complains about
the index
>>>>>>>> being out of bounds.
>>>>>>>>
>>>>>>>> Is there a way to solve the schema/reality mismatch? Am I
doing
>>>>>>>> something wrong, or have I stumbled across a defect?
>>>>>>>>
>>>>>>>> Thanks,
>>>>>>>> Chad
>>>>>>>>
>>>>>>>
>>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
View raw message