apex-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jaspal Singh <jaspal.singh1...@gmail.com>
Subject Re: Datatorrent operator for Hbase
Date Thu, 20 Oct 2016 19:23:42 GMT
Thomas,

When we try excluding the transitive dependencies using 'exclusion *',
getting the below error while running maven build:

[ERROR] COMPILATION ERROR :
[INFO] -------------------------------------------------------------
[ERROR]
/C:/Work/Datatorrent-projects/bdp-datatorrent-mapr-streams/src/main/java/com/example/datatorrent/HbaseTableUpdate.java:[51,18]
cannot access org.apache.hadoop.hbase.CellScannable
  class file for org.apache.hadoop.hbase.CellScannable not found
[ERROR]
/C:/Work/Datatorrent-projects/bdp-datatorrent-mapr-streams/src/main/java/com/example/datatorrent/HbaseTableUpdate.java:[105,10]
cannot access org.apache.hadoop.hbase.io.HeapSize
  class file for org.apache.hadoop.hbase.io.HeapSize not found


Thanks!!

On Thu, Oct 20, 2016 at 1:44 PM, Thomas Weise <thw@apache.org> wrote:

> This may also help:
>
> http://docs.datatorrent.com/troubleshooting/#hadoop-dependencies-conflicts
>
>
> On Thu, Oct 20, 2016 at 11:39 AM, Thomas Weise <thw@apache.org> wrote:
>
>> Please see the HBase dependency and its exclusions here:
>>
>> https://github.com/apache/apex-malhar/blob/master/contrib/pom.xml#L342
>>
>> Thanks,
>> Thomas
>>
>> On Thu, Oct 20, 2016 at 9:07 AM, Jaspal Singh <jaspal.singh1404@gmail.com
>> > wrote:
>>
>>> Team,
>>>
>>> While using the Hbase operator with Datatorrent application, we have
>>> added hbase client dependency in pom.xml. Do we need to exclude transitive
>>> hadoop dependencies using 'exclusion *' ?
>>>
>>> If we do that then HbaseConfiguration and Bytes methods are also getting
>>> excluded and giving error in the application. Is there a way to fix it ??
>>>
>>>
>>> Thanks!!
>>>
>>> On Thu, Oct 20, 2016 at 1:38 AM, Tushar Gosavi <tushar@datatorrent.com>
>>> wrote:
>>>
>>>> Hi Jaspal,
>>>>
>>>> You can pass the store name through property file, like
>>>>
>>>> <property>
>>>>   <name>dt.operator.HbaseOperatorName.store.tableName</name>
>>>>   <value>{name of the table}</value>
>>>> </property>
>>>>
>>>> In the code,  you can set the table name and other properties in
>>>> constructor.
>>>> {code}
>>>> public static class Status2Hbase extends AbstractHBasePutOutputOperator
>>>> <Status>
>>>> {
>>>>
>>>> public Status2Hbase()
>>>> {
>>>>   super();
>>>>   // store is initialized to HBaseStore
>>>>   store.setTableName("nameofTable");
>>>> }
>>>>
>>>> @Override
>>>> public Put operationPut(Status t)
>>>> {
>>>> Put put = new Put(ByteBuffer.allocate(8).put
>>>> Long(t.getCreatedAt().getTime()).array());
>>>> put.add("cf".getBytes(), "text".getBytes(), t.getText().getBytes());
>>>> put.add("cf".getBytes(), "userid".getBytes(), t.getText().getBytes());
>>>> return put;
>>>> }
>>>> }
>>>> {code}
>>>>
>>>> - Tushar.
>>>>
>>>>
>>>> On Thu, Oct 20, 2016 at 11:59 AM, Jaspal Singh
>>>> <jaspal.singh1404@gmail.com> wrote:
>>>> > Hi Thomas, Thanks for sharing this example code.
>>>> >  Still I couldn't see where the hbase tablename is configured, it
>>>> says in
>>>> > description that it can be configured.
>>>> >
>>>> > Can you please highlight where it is specified ?
>>>> >
>>>> > Thanks!!
>>>> >
>>>> >
>>>> > On Wednesday, October 19, 2016, Thomas Weise <thw@apache.org>
wrote:
>>>> >>
>>>> >> Here is an example that uses HBase that may be helpful:
>>>> >>
>>>> >>
>>>> >> https://github.com/apache/apex-malhar/blob/master/demos/twit
>>>> ter/src/main/java/com/datatorrent/demos/twitter/TwitterDumpH
>>>> BaseApplication.java
>>>> >>
>>>> >> Thomas
>>>> >>
>>>> >> On Wed, Oct 19, 2016 at 6:36 PM, Jaspal Singh <
>>>> jaspal.singh1404@gmail.com>
>>>> >> wrote:
>>>> >>>
>>>> >>> Where I need to set the table name. In property file or the
>>>> application
>>>> >>> code ?
>>>> >>>
>>>> >>>
>>>> >>> Thanks!!
>>>> >>>
>>>> >>>
>>>> >>> On Wednesday, October 19, 2016, Sanjay Pujare <
>>>> sanjay@datatorrent.com>
>>>> >>> wrote:
>>>> >>>>
>>>> >>>> Take a look at
>>>> >>>> https://github.com/apache/apex-malhar/tree/master/contrib/sr
>>>> c/main/java/com/datatorrent/contrib/hbase
>>>> >>>> . There are multiple output operators there.
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> You specify the table name using HBaseStore.setTableName
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> From: "Bandaru, Srinivas" <srinivas.bandaru@optum.com>
>>>> >>>> Reply-To: <users@apex.apache.org>
>>>> >>>> Date: Wednesday, October 19, 2016 at 3:09 PM
>>>> >>>> To: "users@apex.apache.org" <users@apex.apache.org>
>>>> >>>> Subject: Datatorrent operator for Hbase
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> Hi,
>>>> >>>>
>>>> >>>> I want to write the data from an operator to a hbase table.
 Which
>>>> >>>> operator I can use to write to  Hbase table?
>>>> >>>>
>>>> >>>> Also how to specify the Hbase table name?
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> Thanks,
>>>> >>>>
>>>> >>>> Srinivas Bandaru
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>>
>>>> >>>> This e-mail, including attachments, may include confidential
and/or
>>>> >>>> proprietary information, and may be used only by the person
or
>>>> entity
>>>> >>>> to which it is addressed. If the reader of this e-mail is
not the
>>>> >>>> intended
>>>> >>>> recipient or his or her authorized agent, the reader is
hereby
>>>> notified
>>>> >>>> that any dissemination, distribution or copying of this
e-mail is
>>>> >>>> prohibited. If you have received this e-mail in error, please
>>>> notify the
>>>> >>>> sender by replying to this message and delete this e-mail
>>>> immediately.
>>>> >>
>>>> >>
>>>> >
>>>>
>>>
>>>
>>
>

Mime
View raw message