chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Ariel Rabkin <asrab...@gmail.com>
Subject Re: Using SocketTeeWriter without SeqFileWriter
Date Mon, 02 Nov 2009 05:52:22 GMT
I had originally intended the NullWriter for this purpose.  But your
approach is cleaner.  If you open a JIRA, I'm happy to commit it.

--Ari

On Sat, Oct 31, 2009 at 4:31 PM, Thushara Wijeratna <thushw@gmail.com> wrote:
> actually there was one more change:
>
> [~/hadoop-src/chukwa/trunk] svn diff
> Index: src/java/org/apache/hadoop/chukwa/datacollection/writer/SocketTeeWriter.java
> ===================================================================
> --- src/java/org/apache/hadoop/chukwa/datacollection/writer/SocketTeeWriter.java    
   (revision
> 831608)
> +++ src/java/org/apache/hadoop/chukwa/datacollection/writer/SocketTeeWriter.java    
   (working
> copy)
> @@ -225,7 +225,9 @@
>
>   @Override
>   public CommitStatus add(List<Chunk> chunks) throws WriterException {
> -    CommitStatus rv = next.add(chunks); //pass data through
> +    CommitStatus rv = ChukwaWriter.COMMIT_OK;
> +    if (next != null)
> +       rv = next.add(chunks); //pass data through
>     synchronized(tees) {
>       Iterator<Tee> loop = tees.iterator();
>       while(loop.hasNext()) {
> @@ -240,7 +242,8 @@
>
>   @Override
>   public void close() throws WriterException {
> -    next.close();
> +    if (next != null)
> +       next.close();
>     running = false;
>     listenThread.shutdown();
>   }
> [~/hadoop-src/chukwa/trunk]
>
>
> On Sat, Oct 31, 2009 at 5:11 PM, Thushara Wijeratna <thushw@gmail.com> wrote:
>> I wanted to use SocketTeeWriter without going through the steps of
>> writing to HDFS.
>> It seems that PipelineStageWriter is designed to have any number of
>> PipeLineable writers, optionally followed by a SeqFileWriter.
>> So I changed my collector config :
>>
>>  <property>
>>    <name>chukwaCollector.writerClass</name>
>>    <value>org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter</value>
>>  </property>
>>
>>  <property>
>>    <name>chukwaCollector.pipeline</name>
>>    <value>org.apache.hadoop.chukwa.datacollection.writer.SocketTeeWriter</value>
>>  </property>
>>
>> After doing one minor change to SocketTeeWriter, I could get this to
>> work. The advantage is that now I do not need to set up HDFS.
>>
>> Please let me know if this is something we should patch, I will submit
>> the patch.
>>
>> SocketTeeWriter changes:
>>
>> [~/hadoop-src/chukwa/trunk] svn diff
>> Index: src/java/org/apache/hadoop/chukwa/datacollection/writer/SocketTeeWriter.java
>> ===================================================================
>> --- src/java/org/apache/hadoop/chukwa/datacollection/writer/SocketTeeWriter.jav(revision
>> 831608)
>> +++ src/java/org/apache/hadoop/chukwa/datacollection/writer/SocketTeeWriter.jav(working
>> copy)
>> @@ -225,7 +225,9 @@
>>
>>   @Override
>>   public CommitStatus add(List<Chunk> chunks) throws WriterException {
>> -    CommitStatus rv = next.add(chunks); //pass data through
>> +    CommitStatus rv = ChukwaWriter.COMMIT_OK;
>> +    if (next != null)
>> +       rv = next.add(chunks); //pass data through
>>     synchronized(tees) {
>>       Iterator<Tee> loop = tees.iterator();
>>       while(loop.hasNext()) {
>> [~/hadoop-src/chukwa/trunk]
>>
>



-- 
Ari Rabkin asrabkin@gmail.com
UC Berkeley Computer Science Department

Mime
View raw message