accumulo-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Joe Stein <joe.st...@stealth.ly>
Subject Re: Kafka to Accumulo
Date Wed, 10 Dec 2014 16:08:35 GMT
I just merged in some sample code we had for Kafka + Accumulo
https://github.com/stealthly/hdp-accumulo

We are working on a new JVM Consumer Client that won't have some of the
intrinsic issue(s) the high level Kafka consumer has.  Basically you won't
need Storm to have at least once processing guarantee and still properly
consume from Kafka and pass that through (based on how you define the work)
the processing. We just released that "fan out work" consumer for Go
https://github.com/stealthly/go_kafka_client if you are interested. The JVM
version should be out in January 2015.

We also have a meet-up in NYC exactly about this topic (no pun intended)
http://www.meetup.com/Apache-Kafka-NYC/events/219067373/ on January 21st,
2015 where we will also talk about Accumulo on Mesos in addition to Kafka +
Accumulo. There will be other speakers too at this event (2 confirmed so
far).

/*******************************************
 Joe Stein
 Founder, Principal Consultant
 Big Data Open Source Security LLC
 http://www.stealth.ly
 Twitter: @allthingshadoop <http://www.twitter.com/allthingshadoop>
********************************************/

On Wed, Dec 10, 2014 at 10:31 AM, Webb, Ryan L. <Ryan.Webb@jhuapl.edu>
wrote:

>  Storm is slowly making its Kafka spout a first class citizen.
>
> It wouldn’t be too hard to have a simple topology that uses the
> storm-kafka spout and a custom bolt that rights out to Accumulo.
>
>
>
> Ryan Webb
>
>
>
> *From:* Ariel Valentin [mailto:ariel@arielvalentin.com]
> *Sent:* Wednesday, December 10, 2014 8:06 AM
> *To:* user@accumulo.apache.org
> *Subject:* Re: Kafka to Accumulo
>
>
>
> I have mostly seen storm in the wild but that doesn't mean it's the right
> one. I think the answer depends on your context. Do you have
> flume/morphline experts? Do you run storm topologies now? I have never used
> spark and there isn't any official integration support AFAIK.
>
>
> Thanks,
>
> Ariel
>
> ---
>
> Sent from my mobile device. Please excuse any errors.
>
>
> On Dec 10, 2014, at 7:21 AM, kishore kumar <akishore769@gmail.com> wrote:
>
>  flume/spark/storm ?
>
>
>
> On Wed, Dec 10, 2014 at 5:29 PM, Ariel Valentin <ariel@arielvalentin.com>
> wrote:
>
> I am not aware of any tools that do this right now. Most of the apps I
> have seen use custom programs that pull the messages off the queue, parse
> them, then write mutations to the tables. Does that seem reasonable? Is
> there another data store that has the kind of Kafka integration you are
> hoping for in accumulo?
>
>
> Thanks,
>
> Ariel
>
> ---
>
> Sent from my mobile device. Please excuse any errors.
>
>
> On Dec 10, 2014, at 6:51 AM, kishore kumar <akishore769@gmail.com> wrote:
>
>  yes ariel.
>
>
>
> On Wed, Dec 10, 2014 at 5:06 PM, Ariel Valentin <ariel@arielvalentin.com>
> wrote:
>
> Kishore,
>
> I am afraid I don't understand your question. Are you trying to consume
> data from a queue and write it into a table?
>
> Thanks,
> Ariel
> ---
> Sent from my mobile device. Please excuse any errors.
>
>
> > On Dec 10, 2014, at 3:18 AM, kishore kumar <akishore769@gmail.com>
> wrote:
> >
> > Hi Experts,
> >
> > Is there any link to integrate the data from kafka to accumulo, if yes
> please share with me.
> >
> > --
> > Thanks,
> > Kishore.
>
>
>
>
>
> --
>
> Thanks,
>
> Kishore.
>
>
>
>
>
> --
>
> Thanks,
>
> Kishore.
>
>

Mime
View raw message