chukwa-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thushara Wijeratna <thu...@gmail.com>
Subject Re: Using SocketTeeWriter without SeqFileWriter
Date Sun, 01 Nov 2009 00:31:39 GMT
actually there was one more change:

[~/hadoop-src/chukwa/trunk] svn diff
Index: src/java/org/apache/hadoop/chukwa/datacollection/writer/SocketTeeWriter.java
===================================================================
--- src/java/org/apache/hadoop/chukwa/datacollection/writer/SocketTeeWriter.java	(revision
831608)
+++ src/java/org/apache/hadoop/chukwa/datacollection/writer/SocketTeeWriter.java	(working
copy)
@@ -225,7 +225,9 @@

   @Override
   public CommitStatus add(List<Chunk> chunks) throws WriterException {
-    CommitStatus rv = next.add(chunks); //pass data through
+    CommitStatus rv = ChukwaWriter.COMMIT_OK;
+    if (next != null)
+	rv = next.add(chunks); //pass data through
     synchronized(tees) {
       Iterator<Tee> loop = tees.iterator();
       while(loop.hasNext()) {
@@ -240,7 +242,8 @@

   @Override
   public void close() throws WriterException {
-    next.close();
+    if (next != null)
+	next.close();
     running = false;
     listenThread.shutdown();
   }
[~/hadoop-src/chukwa/trunk]


On Sat, Oct 31, 2009 at 5:11 PM, Thushara Wijeratna <thushw@gmail.com> wrote:
> I wanted to use SocketTeeWriter without going through the steps of
> writing to HDFS.
> It seems that PipelineStageWriter is designed to have any number of
> PipeLineable writers, optionally followed by a SeqFileWriter.
> So I changed my collector config :
>
>  <property>
>    <name>chukwaCollector.writerClass</name>
>    <value>org.apache.hadoop.chukwa.datacollection.writer.PipelineStageWriter</value>
>  </property>
>
>  <property>
>    <name>chukwaCollector.pipeline</name>
>    <value>org.apache.hadoop.chukwa.datacollection.writer.SocketTeeWriter</value>
>  </property>
>
> After doing one minor change to SocketTeeWriter, I could get this to
> work. The advantage is that now I do not need to set up HDFS.
>
> Please let me know if this is something we should patch, I will submit
> the patch.
>
> SocketTeeWriter changes:
>
> [~/hadoop-src/chukwa/trunk] svn diff
> Index: src/java/org/apache/hadoop/chukwa/datacollection/writer/SocketTeeWriter.java
> ===================================================================
> --- src/java/org/apache/hadoop/chukwa/datacollection/writer/SocketTeeWriter.jav(revision
> 831608)
> +++ src/java/org/apache/hadoop/chukwa/datacollection/writer/SocketTeeWriter.jav(working
> copy)
> @@ -225,7 +225,9 @@
>
>   @Override
>   public CommitStatus add(List<Chunk> chunks) throws WriterException {
> -    CommitStatus rv = next.add(chunks); //pass data through
> +    CommitStatus rv = ChukwaWriter.COMMIT_OK;
> +    if (next != null)
> +       rv = next.add(chunks); //pass data through
>     synchronized(tees) {
>       Iterator<Tee> loop = tees.iterator();
>       while(loop.hasNext()) {
> [~/hadoop-src/chukwa/trunk]
>

Mime
View raw message