incubator-hama-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Edward J. Yoon" <edwardy...@apache.org>
Subject Re: [jira] [Assigned] (ACCUMULO-532) Add BSP input/output formats to client package
Date Tue, 24 Apr 2012 01:33:34 GMT
According to CHANGE log, I've added input/output system to BSP
framework. But, I don't know exactly why we need to check
(conf.get("bsp.input (or output).dir") != null) conditions when
initialize record reader/writer objects?

If there's no objection or opinion, I'd like to change like this:

Index: core/src/main/java/org/apache/hama/bsp/BSPPeerImpl.java
===================================================================
--- core/src/main/java/org/apache/hama/bsp/BSPPeerImpl.java	(revision 1329523)
+++ core/src/main/java/org/apache/hama/bsp/BSPPeerImpl.java	(working copy)
@@ -185,8 +185,7 @@

     initInput();

-    // just output something when the user configured it
-    if (conf.get("bsp.output.dir") != null) {
+    if (!bspJob.getInputFormat().getClass().equals(NullOutputFormat.class)) {
       Path outdir = new Path(conf.get("bsp.output.dir"),
           Task.getOutputName(partition));
       outWriter = bspJob.getOutputFormat().getRecordWriter(fs, bspJob,
@@ -204,8 +203,7 @@

   @SuppressWarnings("unchecked")
   public final void initInput() throws IOException {
-    // just read input if the user defined one
-    if (conf.get("bsp.input.dir") != null) {
+    if (!bspJob.getInputFormat().getClass().equals(NullInputFormat.class)) {
       InputSplit inputSplit = null;
       // reinstantiate the split
       try {


On Tue, Apr 24, 2012 at 6:39 AM, Edward J. Yoon <edwardyoon@apache.org> wrote:
> FYI,
>
> "Let me know if you see any issues with this.  It could probably use some more testing.
 I was able to get the unit tests working (even the part commented out in the patch) but
I had to set fake input and output paths.  It seems that BSP doesn't initialize the RecordReader
and RecordWriter unless the configuration options "bsp.input.dir" and "bsp.output.dir" are
set."
>
> Sent from my iPad
>
> Begin forwarded message:
>
>> From: "Billie Rinaldi (JIRA)" <jira@apache.org>
>> Date: April 24, 2012 2:54:35 AM GMT+09:00
>> To: dev@accumulo.apache.org
>> Subject: [jira] [Assigned] (ACCUMULO-532) Add BSP input/output formats to client
package
>> Reply-To: dev@accumulo.apache.org
>>
>> Let me know if you see any issues with this.  It could probably use some more testing.
 I was able to get the unit tests working (even the part commented out in the patch) but
I had to set fake input and output paths.  It seems that BSP doesn't initialize the RecordReader
and RecordWriter unless the configuration options "bsp.input.dir" and "bsp.output.dir" are
set.



-- 
Best Regards, Edward J. Yoon
@eddieyoon

Mime
View raw message