Return-Path: X-Original-To: apmail-apex-users-archive@minotaur.apache.org Delivered-To: apmail-apex-users-archive@minotaur.apache.org Received: from mail.apache.org (hermes.apache.org [140.211.11.3]) by minotaur.apache.org (Postfix) with SMTP id DE11C18F02 for ; Tue, 12 Jan 2016 15:07:34 +0000 (UTC) Received: (qmail 93655 invoked by uid 500); 12 Jan 2016 15:07:34 -0000 Delivered-To: apmail-apex-users-archive@apex.apache.org Received: (qmail 93595 invoked by uid 500); 12 Jan 2016 15:07:34 -0000 Mailing-List: contact users-help@apex.incubator.apache.org; run by ezmlm Precedence: bulk List-Help: List-Unsubscribe: List-Post: List-Id: Reply-To: users@apex.incubator.apache.org Delivered-To: mailing list users@apex.incubator.apache.org Received: (qmail 93585 invoked by uid 99); 12 Jan 2016 15:07:34 -0000 Received: from Unknown (HELO spamd1-us-west.apache.org) (209.188.14.142) by apache.org (qpsmtpd/0.29) with ESMTP; Tue, 12 Jan 2016 15:07:34 +0000 Received: from localhost (localhost [127.0.0.1]) by spamd1-us-west.apache.org (ASF Mail Server at spamd1-us-west.apache.org) with ESMTP id 42290C3B6A for ; Tue, 12 Jan 2016 15:07:34 +0000 (UTC) X-Virus-Scanned: Debian amavisd-new at spamd1-us-west.apache.org X-Spam-Flag: NO X-Spam-Score: 2.999 X-Spam-Level: ** X-Spam-Status: No, score=2.999 tagged_above=-999 required=6.31 tests=[DKIM_SIGNED=0.1, DKIM_VALID=-0.1, HTML_MESSAGE=3, RCVD_IN_MSPIKE_H2=-0.001, SPF_PASS=-0.001, URIBL_BLOCKED=0.001] autolearn=disabled Authentication-Results: spamd1-us-west.apache.org (amavisd-new); dkim=pass (2048-bit key) header.d=datatorrent-com.20150623.gappssmtp.com Received: from mx1-eu-west.apache.org ([10.40.0.8]) by localhost (spamd1-us-west.apache.org [10.40.0.7]) (amavisd-new, port 10024) with ESMTP id 7CmN1T53qnMD for ; Tue, 12 Jan 2016 15:07:27 +0000 (UTC) Received: from mail-pf0-f174.google.com (mail-pf0-f174.google.com [209.85.192.174]) by mx1-eu-west.apache.org (ASF Mail Server at mx1-eu-west.apache.org) with ESMTPS id 4D0DA31AD6 for ; Tue, 12 Jan 2016 15:07:26 +0000 (UTC) Received: by mail-pf0-f174.google.com with SMTP id n128so64168602pfn.3 for ; Tue, 12 Jan 2016 07:07:26 -0800 (PST) DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=datatorrent-com.20150623.gappssmtp.com; s=20150623; h=mime-version:in-reply-to:references:date:message-id:subject:from:to :content-type; bh=IqmheWmp3q0hSIbcIzMH8U6Ov8c2VSt2ZTjiAakqa+A=; b=zDE1hiYzwLMmDp1uUCARIqbwV+yHI+zMFOwapyGeuyVuUl9qWSwhhYlg7+xPn0YnLT J2M/mpKnW+bQpEuXXBFWVcmCBjQUbtCPFYqv5THJ28xLTCdl0pV9HIgd+cmteAO7UTSH FEmWNpGqWRXYDBbYhKDjAJpmd3DWr1FzNkcVGoB9UE4L/8C6MaJO9JHkgjj5yTTt0KHS q2zLhxSHnW180hOfNbtP9YbYp261T/G1Y6Z9trAM1Z0fRqnuN9I3+5LXxdtomAuNCvxz XaCoX2/Gafjlag99qEZezNC7cvDMpDJH5QN+Y8rTT8cei8Ag/5yqfTHZkuGkgECzL6+j 6L5A== X-Google-DKIM-Signature: v=1; a=rsa-sha256; c=relaxed/relaxed; d=1e100.net; s=20130820; h=x-gm-message-state:mime-version:in-reply-to:references:date :message-id:subject:from:to:content-type; bh=IqmheWmp3q0hSIbcIzMH8U6Ov8c2VSt2ZTjiAakqa+A=; b=kA/2GW8Eqs9yhrJU99k2M7OHjOwlKYU3qI+PLIZMNzk3MgelWPtKCZYgDCh+tL0gYg nt1NrImt6GhlqaI0CbTk7AyRaooMW55abU4S8D5AKCwbIU6qOnnKTNaVC1MSi96dvQHG RmCwuD59XTxHQ5GwNirCYP4381gVldZCRKhPVXKUSNXz/yQqlEDERWwpEGaEeLsJde+n Na8oFO3zYK3sTD4mQ1ApYSvYOmbymt54IRMmNm3m4dCSX8+2tOMCbJO5vio9zwTYXDXc 8LusHEyr2m62+Tn9wyAxbP5wFXBuI834tXCMwAH9zDQ5xgmEmQb5jVRqd8K0SP6F35Ih ZQfw== X-Gm-Message-State: ALoCoQmTcJt6pEtMx4UJv/OMtwaKy2bAtHdSQjKE/Dmsu70TrjHftzxVCQF9LH86nJ1NQNMZ/IO49WCZpM5ul4metM2QOi9oj7onTcPpcXiu01fdLXmsjcM= MIME-Version: 1.0 X-Received: by 10.98.13.197 with SMTP id 66mr34546539pfn.125.1452611238813; Tue, 12 Jan 2016 07:07:18 -0800 (PST) Received: by 10.66.147.102 with HTTP; Tue, 12 Jan 2016 07:07:18 -0800 (PST) In-Reply-To: References: Date: Tue, 12 Jan 2016 07:07:18 -0800 Message-ID: Subject: Re: Concrete classes in Malhar From: Munagala Ramanath To: users@apex.incubator.apache.org Content-Type: multipart/alternative; boundary=001a114767a6e2f848052924682e --001a114767a6e2f848052924682e Content-Type: text/plain; charset=UTF-8 There are some concrete classes in the demos subdirectory of Malhar, see for example: *demos/frauddetect/src/main/java/com/datatorrent/demos/frauddetect/operator/HdfsStringOutputOperator.java* *demos/wordcount/src/main/java/com/datatorrent/demos/wordcount/WordCountWriter.java* Ram On Tue, Jan 12, 2016 at 3:30 AM, Akshay Gore wrote: > Thanks Yogi. I am able to perform file write operation on HDFS. > > One suggestion here, Malhar library should have some concrete classes like > hdfs file input/output operations so that it would be easy for a new Apex > user to quickly start with. > > -Akshay > > On Tue, Jan 12, 2016 at 3:26 PM, Yogi Devendra > wrote: > >> Here is the Javadoc: >> >> >> https://www.datatorrent.com/docs/apidocs/com/datatorrent/lib/io/fs/AbstractFileOutputOperator.html >> >> Here is the user guide: >> >> http://docs.datatorrent.com/operators/file_output/ >> >> You will need to provide implementation for couple of methods: >> >> 1. protected abstract String getFileName(INPUT tuple) >> You could simply return "output.txt" >> >> 2. protected abstract byte[] getBytesForTuple(INPUT tuple) >> >> You need to provide some way to serialize input tuple. >> It could be as simple as return tuple.toString().getBytes() >> >> It would be great if you use some IDE like eclipse or intellij; it should >> give you suggestions, auto-complete template for methods to be overridden. >> >> ~ Yogi >> >> On Tue, Jan 12, 2016 at 2:58 PM, Akshay Gore >> wrote: >> >>> Hi, >>> >>> I am developing the sample Apex application to explore Malhar operators. >>> In my case, I need to write the processed data to HDFS. I did not find >>> concrete class for file writing, instead I got AbstractFileOutput operator >>> class which will have to implement it. Can I have references/ examples to >>> how to implement the abstract classes in Malhar ? If not, it would be nice >>> to have such classes in Malhar. >>> >>> -- >>> Thanks, >>> Akshay >>> >> >> > > > -- > Thanks, > Akshay > --001a114767a6e2f848052924682e Content-Type: text/html; charset=UTF-8 Content-Transfer-Encoding: quoted-printable
There are some concrete classes in the demos subdirectory = of Malhar, see for example:

demos/frauddetect/src/main/java/com/datatorrent/demos/fraud= detect/operator/HdfsStringOutputOperator.java
demos/wordcount/src/main/java= /com/datatorrent/demos/wordcount/WordCountWriter.java
<= div>
=
Ram

On Tue, Jan 12,= 2016 at 3:30 AM, Akshay Gore <goreakshay@gmail.com> wrot= e:
Thanks Yogi. I a= m able to perform file write operation on HDFS.

One suggestion= here, Malhar library should have some concrete classes like hdfs file inpu= t/output operations so that it would be easy for a new Apex user to quickly= start with.

-Akshay

On Tue, Jan 12, 2016 at 3:26 PM,= Yogi Devendra <devendra@datatorrent.com> wrote:
<= blockquote class=3D"gmail_quote" style=3D"margin:0 0 0 .8ex;border-left:1px= #ccc solid;padding-left:1ex">
Here is the Javadoc:


Here is the user= guide:


You w= ill need to provide implementation for couple of methods:

1. =C2=A0protected ab= stract String=C2=A0getFileName(= INPUT tuple)
=C2=A0You could simply return = "output.txt"

2. protected abstract byte[] getBytesForTuple(INPUT tuple)

You need to provide some way to = serialize input tuple.
It could be as simpl= e as return tuple.toString().getBytes()
It would be great if you use some IDE lik= e eclipse or intellij; it should give you suggestions, auto-complete templa= te for methods to be overridden.
~ Yogi
<= div>
On Tue, Jan 12, 2016 at 2:58 PM, Akshay Gore= <goreakshay@gmail.com> wrote:
Hi,

I am developing the sample Apex appli= cation to explore Malhar operators. In my case, I need to write the process= ed data to HDFS. I did not find concrete class for file writing, instead I = got AbstractFileOutput operator class which will have to implement it. Can = I have references/ examples to how to implement the abstract classes in Mal= har ?=C2=A0 If not, it would be nice to have such classes in Malhar.

--
Thank= s,
Akshay




--
Thanks,
Akshay

--001a114767a6e2f848052924682e--