avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Steven Willis (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AVRO-570) python implementation of mapreduce connector
Date Fri, 03 Oct 2014 01:24:34 GMT

    [ https://issues.apache.org/jira/browse/AVRO-570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14157557#comment-14157557
] 

Steven Willis commented on AVRO-570:
------------------------------------

Hmm, just looking through the patch again, I noticed this:

{noformat}
Index: lang/java/tools/pom.xml
===================================================================
--- lang/java/tools/pom.xml	(revision 1143583)
+++ lang/java/tools/pom.xml	(working copy)
@@ -17,7 +17,7 @@
 -->
 <project
   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"
-  xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
+  xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instancesavro">
   <modelVersion>4.0.0</modelVersion>
   <parent>
     <artifactId>avro-parent</artifactId>
{noformat}

It was in the original patch, but I'm guessing that's not right.

There were also some odd whitespace changes in some of the {{pom.xml}}'s that I didn't change.

I also noticed this change in {{TetheredProcess.java}}:

{noformat}
       }
       if (hasexited) {
-        LOG.error("Could not start subprocess");
-        throw new RuntimeException("Could not start subprocess");
+        //What's the best way to log this
+        System.out.println("Error: Could not start subprocess");
+        throw new RuntimeException("Error: Could not start subprocess");
+      }
{noformat}

Seems odd, like it was getting logged via {{LOG}} just fine, and then replaced with a comment
asking how to log it.

I also noticed that it seems to create a new jar for {{avro-tools}} with {{maven-assembly-plugin}}
that's supposed to be useable as a hadoop job jar. However, I found that I actually had to
use the {{-withdeps}} jar that's created to get it to work. So maybe the code that creates
the {{-job}} jar isn't necessary? It also looks like the {{-withdeps}} jar was just the originally
created jar but now has a classifier.

I wouldn't want to drastically change the artifacts being published with this change, so that
might need more scrutiny. I also personally tested with {{-Dhadoop.version=2}}, I had some
issues compiling and testing without that, and I didn't try {{-Dhadoop.version=1}}.

> python implementation of mapreduce connector
> --------------------------------------------
>
>                 Key: AVRO-570
>                 URL: https://issues.apache.org/jira/browse/AVRO-570
>             Project: Avro
>          Issue Type: New Feature
>          Components: python
>    Affects Versions: 1.7.0
>            Reporter: Doug Cutting
>            Assignee: Jeremy Lewi
>            Priority: Critical
>              Labels: hadoop
>             Fix For: 1.8.0
>
>         Attachments: AVRO-570.patch, AVRO-570.patch, AVRO-570.patch, AVRO-570.patch,
AVRO-570.patch, AVRO-570.patch, AVRO-570.patch, AVRO-570.patch
>
>
> AVRO-512 defines protocols for implementing mapreduce tasks.  It would be good to have
a Python implementation of this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message