cassandra-commits mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Alex Liu (JIRA)" <j...@apache.org>
Subject [jira] [Comment Edited] (CASSANDRA-5867) The Pig CqlStorage/AbstractCassandraStorage classes don't handle collection types
Date Fri, 16 Aug 2013 23:00:48 GMT

    [ https://issues.apache.org/jira/browse/CASSANDRA-5867?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13742689#comment-13742689
] 

Alex Liu edited comment on CASSANDRA-5867 at 8/16/13 11:00 PM:
---------------------------------------------------------------

To store data to CQL3 table, we supports the following prepared statements

{code}
Set
   e.g.
   UPDATE users SET top_places = ?
   UPDATE users SET top_places = [ 'rivendell', 'rohan' ] WHERE user_id = 'frodo';

   UPDATE users SET top_places = ? + top_places
   UPDATE users SET top_places = [ 'the shire' ] + top_places WHERE user_id = 'frodo';

   UPDATE users SET top_places[2] = ?
   UPDATE users SET top_places[2] = 'riddermark' WHERE user_id = 'frodo';

   DELETE top_places[3] FROM users;
   DELETE top_places[3] FROM users WHERE user_id = 'frodo';

   UPDATE users SET top_places = top_places - ?;
   UPDATE users SET top_places = top_places - ['riddermark'] WHERE user_id = 'frodo';

List statements are similar to Set

Map
  UPDATE users SET todo[?] = ?
  UPDATE users SET todo['2012-10-2 12:10'] = 'die' WHERE user_id = 'frodo';

  UPDATE users SET todo = ?
  UPDATE users
       SET todo = { '2012-9-24' : 'enter mordor',
                    '2012-10-2 12:00' : 'throw ring into mount doom' }
       WHERE user_id = 'frodo';

{code}

The output schema for collections is as following

{code}
 (((name, value), (name, value)), (value ... value), (value...value)
 If a value of tuple (value...value) is a tuple of (inner_value ...inner_value) 
 and the first inner_value is the collection type. 
 it is either "set", "list" or "map".

 e.g. (value ... value) as (value1, value2, (set, riddermark, tom))
 map  (value1, value2, (map, (sfo, 12), (ny, 34))
{code}

                
      was (Author: alexliu68):
    To store data to CQL3 table, we supports the following prepared statements

{code}
Set
   e.g.
   UPDATE users SET top_places = ?
   UPDATE users SET top_places = [ 'rivendell', 'rohan' ] WHERE user_id = 'frodo';

   UPDATE users SET top_places = ? + top_places
   UPDATE users SET top_places = [ 'the shire' ] + top_places WHERE user_id = 'frodo';

   UPDATE users SET top_places[2] = ?
   UPDATE users SET top_places[2] = 'riddermark' WHERE user_id = 'frodo';

   DELETE top_places[3] FROM users;
   DELETE top_places[3] FROM users WHERE user_id = 'frodo';

   UPDATE users SET top_places = top_places - ?;
   UPDATE users SET top_places = top_places - ['riddermark'] WHERE user_id = 'frodo';

List statements are similar to Set

Map
  UPDATE users SET todo[?] = ?
  UPDATE users SET todo['2012-10-2 12:10'] = 'die' WHERE user_id = 'frodo';

  UPDATE users SET todo = ?
  UPDATE users
       SET todo = { '2012-9-24' : 'enter mordor',
                    '2012-10-2 12:00' : 'throw ring into mount doom' }
       WHERE user_id = 'frodo';

{code}

The output schema for collections is as following

{code}
 (((name, value), (name, value)), (value ... value), (value...value)
 If a value of tuple (value...value) is a tuple of (inner_value ...inner_value) 
 and the first inner_value is the collection type. 
 it is either "set", "list" or "map".

 e.g. (value ... value) as (value1, value2, ('set', 'riddermark', 'tom'))
 map  (value1, value2, ('map', ('sfo', '12'), ('ny', '34'))
{code}

                  
> The Pig CqlStorage/AbstractCassandraStorage classes don't handle collection types
> ---------------------------------------------------------------------------------
>
>                 Key: CASSANDRA-5867
>                 URL: https://issues.apache.org/jira/browse/CASSANDRA-5867
>             Project: Cassandra
>          Issue Type: Bug
>          Components: Hadoop
>            Reporter: Jeremy Hanna
>            Assignee: Alex Liu
>              Labels: pig
>         Attachments: 5867-1.2-branch.txt, 5867-2-1.2-branch.txt
>
>
> The CqlStorage class gets the Pig data type for values from the AbstractCassandraStorage
class, in the getPigType method.  If it isn't a known data type, it makes the value into a
ByteArray.  Currently there aren't any cases there for lists, maps, and sets.
> https://github.com/apache/cassandra/blob/cassandra-1.2.8/src/java/org/apache/cassandra/hadoop/pig/AbstractCassandraStorage.java#L336
> See this describe output from the grunt shell:
> {code}
> grunt> describe listdata ;                                        
> listdata: {id: (name: chararray,value: int),alist: (name: chararray,value: bytearray),amap:
(name: chararray,value: bytearray),aset: (name: chararray,value: bytearray)}
> {code}
> where the cql data structures had this schema:
> {code}
> CREATE TABLE alltypes (
>   id int PRIMARY KEY,
>   alist list<text>,
>   amap map<text, text>,
>   aset set<text>
> {code}
> It turns out that if you cast the map in grunt to a pig map, then it sort of works, but
I don't think we should probably use a pig map.  Lists don't appear to work at all, as there
is no Pig analogue.  I *think* you could probably just do a UDF to cast these things, but
we already have all of the type information, so we just need to change them to tuples or bags
or whatever.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators
For more information on JIRA, see: http://www.atlassian.com/software/jira

Mime
View raw message