HBaseStorage

+ HBaseStorage +

Loads and stores data from an HBase table.

+ +

+ Syntax + + + + +

HBaseStorage('columns', ['options'])

+ +

+ Terms + + + + + + + + + +

columns

A list of qualified HBase columns to read data from or store data to. + The column family name and column qualifier are seperated by a colon (:). + Only the columns used in the Pig script need to be specified. Columns are specified + in one of three different ways as described below.

Explicitly specify a column family and column qualifier (e.g., user_info:id). This + will produce a scalar in the resultant tuple.
Specify a column family and a portion of column qualifier name as a prefix followed + by an asterisk (i.e., user_info:address_*). This approach is used to read one or + more columns from the same column family with a matching descriptor prefix. + The datatype for this field will be a map of column descriptor name to field value. + Note that combining this style of prefix with a long list of fully qualified + column descriptor names could cause perfomance degredation on the HBase scan. + This will produce a Pig map in the resultant tuple with column descriptors as keys.
Specify all the columns of a column family using the column family name followed + by an asterisk (i.e., user_info:*). This will produce a Pig map in the resultant + tuple with column descriptors as keys.

'options'

A string that contains space-separated options (‘-optionA=valueA -optionB=valueB -optionC=valueC’)

Currently supported options are:

-loadKey=(true|false) Load the row key as the first value in every tuple + returned from HBase (default=false)
-gt=minKeyVal Return rows with a rowKey greater than minKeyVal
-lt=maxKeyVal Return rows with a rowKey less than maxKeyVal
-gte=minKeyVal Return rows with a rowKey greater than or equal to minKeyVal
-lte=maxKeyVal Return rows with a rowKey less than or equal to maxKeyVal
-limit=numRowsPerRegion Max number of rows to retrieve per region
-caching=numRows Number of rows to cache (faster scans, more memory)
-delim=delimiter Column delimiter in columns list (default is whitespace)
-ignoreWhitespace=(true|false) When delim is set to something other than + whitespace, ignore spaces when parsing column list (default=true)
-caster=(HBaseBinaryConverter|Utf8StorageConverter) Class name of Caster to use + to convert values (default=Utf8StorageConverter). The default caster can be + overridden with the pig.hbase.caster config param. Casters must implement LoadStoreCaster.
-noWAL=(true|false) During storage, sets the write ahead to false for faster + loading into HBase (default=false). To be used with extreme caution since this + could result in data loss (see http://hbase.apache.org/book.html#perf.hbase.client.putwal).
-minTimestamp=timestamp Return cell values that have a creation timestamp + greater or equal to this value
-maxTimestamp=timestamp Return cell values that have a creation timestamp + less than this value
-timestamp=timestamp Return cell values that have a creation timestamp equal to + this value

+ +

+ Usage +

HBaseStorage stores and loads data from HBase. The function takes two arguments. The first + argument is a space seperated list of columns. The second optional argument is a + space seperated list of options. Column syntax and available options are listed above.

+ +

+ Load Example +

In this example HBaseStorage is used with the LOAD function with an explicit schema.

+ +raw = LOAD 'hbase://SomeTableName' + USING org.apache.pig.backend.hadoop.hbase.HBaseStorage( + 'info:first_name info:last_name tags:work_* info:*', '-loadKey=true -limit=5') AS + (id:bytearray, first_name:chararray, last_name:chararray, tags_map:map[], info_map:map[]); + +

The datatypes of the columns are declared with the "AS" clause. The first_name and last_name + columns are specified as fully qualified column names with a chararray datatype. The third + specification of tags:work_* requests a set of columns in the tags column family that begin + with "work_". There can be zero, one or more columns of that type in the HBase table. The + type is specified as tags_map:map[]. This indicates that the set of column values returned + will be accessed as a map, where the key is the column name and the value is the cell value + of the column. The fourth column specification is also a map of column descriptors to cell + values.

When the type of the column is specified as a map in the "AS" clause, the map keys are the + column descriptor names and the data type is chararray. The datatype of the columns values can + be declared explicitly as shown in the examples below:

tags_map[chararray] - In this case, the column values are all declared to be of type chararray
tags_map[int] - In this case, the column values are all declared to be of type int.

+ +

+ Store Example +

In this example HBaseStorage is used to store a relation into HBase.

+ +A = LOAD 'hdfs_users' AS (id:bytearray, first_name:chararray, last_name:chararray); +STORE A INTO 'hbase://users_table' USING org.apache.pig.backend.hadoop.hbase.HBaseStorage( + 'info:first_name info:last_name'); + +

In the example above relation A is loaded from HDFS and stored in HBase. Note that the schema + of relation A is a tuple of size 3, but only two column descriptor names are passed to the + HBaseStorage constructor. This is because the first entry in the tuple is used as the HBase + rowKey.