flume-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Hari Shreedharan <hshreedha...@cloudera.com>
Subject Re: Real time reads
Date Sun, 05 Aug 2012 01:25:28 GMT
Let me correct what I said. You could implement a source that implements PollableSource interface
to actually pull data. We do have a source that does pull data, the ExecSource - though it
pulls data from the output stream of a local process. You could write a source that connects
to HBase and pulls data out. The implementation can be on the lines of the ExecSource, except
that you should replace the polling logic. 

Hope this helps.


Thanks,
Hari

-- 
Hari Shreedharan


On Saturday, August 4, 2012 at 3:37 PM, Mohit Anchlia wrote:

> 
> 
> On Sat, Aug 4, 2012 at 3:28 PM, Patrick Wendell <pwendell@gmail.com (mailto:pwendell@gmail.com)>
wrote:
> > Mohit,
> > 
> > This sounds like something where you would want your web tier to
> > directly access HBase through its existing client API.
> > 
> > Do you need functionality not offered there?
>  
> Yes that's how I am currently testing but I just thought it would be nice to use one
framework for both read and write.
> > 
> > - Patrick
> > 
> > On Sat, Aug 4, 2012 at 2:49 PM, Hari Shreedharan
> > <hshreedharan@cloudera.com (mailto:hshreedharan@cloudera.com)> wrote:
> > > Mohit,
> > >
> > > Flume NG does not really have an HBase source. More importantly our sources
> > > do not actually pull data. It waits for data to be pushed to it, at least
> > > that is what all the current sources do. I don't know if it is a good idea
> > > to have a source proactively pull stuff from a source. But Flume is designed
> > > to make practically everything pluggable, so you can write your own source
> > > that does this.
> > >
> > >
> > > Hari
> > >
> > > --
> > > Hari Shreedharan
> > >
> > > On Saturday, August 4, 2012 at 2:34 PM, Mohit Anchlia wrote:
> > >
> > > I am prototyping flume ng for real time sink to hbase and hdfs for
> > > clickstream. I was wondering if Flume provides something that can read from
> > > HBase too. That is API request comes to our web server and after we receive
> > > the request we use flume to retrieve the result from HBase. Is flume ng
> > > meant for this type of scenario?
> > >
> > >
> 


Mime
View raw message