Return-Path: 
 <dev-return-14487-apmail-avro-dev-archive=avro.apache.org@avro.apache.org>
X-Original-To: apmail-avro-dev-archive@www.apache.org
Delivered-To: apmail-avro-dev-archive@www.apache.org
Received: from mail.apache.org (hermes.apache.org [140.211.11.3])
	by minotaur.apache.org (Postfix) with SMTP id A8C0E17D48
	for <apmail-avro-dev-archive@www.apache.org>;
 Fri,  3 Oct 2014 01:24:34 +0000 (UTC)
Received: (qmail 30637 invoked by uid 500); 3 Oct 2014 01:24:34 -0000
Delivered-To: apmail-avro-dev-archive@avro.apache.org
Received: (qmail 30553 invoked by uid 500); 3 Oct 2014 01:24:34 -0000
Mailing-List: contact dev-help@avro.apache.org; run by ezmlm
Precedence: bulk
List-Help: <mailto:dev-help@avro.apache.org>
List-Unsubscribe: <mailto:dev-unsubscribe@avro.apache.org>
List-Post: <mailto:dev@avro.apache.org>
List-Id: <dev.avro.apache.org>
Reply-To: dev@avro.apache.org
Delivered-To: mailing list dev@avro.apache.org
Received: (qmail 30541 invoked by uid 99); 3 Oct 2014 01:24:34 -0000
Received: from arcas.apache.org (HELO arcas.apache.org) (140.211.11.28)
    by apache.org (qpsmtpd/0.29) with ESMTP; Fri, 03 Oct 2014 01:24:34 +0000
Date: Fri, 3 Oct 2014 01:24:34 +0000 (UTC)
From: "Steven Willis (JIRA)" <jira@apache.org>
To: dev@avro.apache.org
Message-ID: <JIRA.12466471.1276017362000.181432.1412299474291@Atlassian.JIRA>
In-Reply-To: <JIRA.12466471.1276017362000@Atlassian.JIRA>
References: <JIRA.12466471.1276017362000@Atlassian.JIRA>
 <JIRA.12466471.1276017362121@arcas>
Subject: [jira] [Commented] (AVRO-570) python implementation of mapreduce
 connector
MIME-Version: 1.0
Content-Type: text/plain; charset=utf-8
Content-Transfer-Encoding: 7bit
X-JIRA-FingerPrint: 30527f35849b9dde25b450d4833f0394


    [ https://issues.apache.org/jira/browse/AVRO-570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14157557#comment-14157557 ] 

Steven Willis commented on AVRO-570:
------------------------------------

Hmm, just looking through the patch again, I noticed this:

{noformat}
Index: lang/java/tools/pom.xml
===================================================================
--- lang/java/tools/pom.xml	(revision 1143583)
+++ lang/java/tools/pom.xml	(working copy)
@@ -17,7 +17,7 @@
 -->
 <project
   xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/xsd/maven-4.0.0.xsd"
-  xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance">
+  xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instancesavro">
   <modelVersion>4.0.0</modelVersion>
   <parent>
     <artifactId>avro-parent</artifactId>
{noformat}

It was in the original patch, but I'm guessing that's not right.

There were also some odd whitespace changes in some of the {{pom.xml}}'s that I didn't change.

I also noticed this change in {{TetheredProcess.java}}:

{noformat}
       }
       if (hasexited) {
-        LOG.error("Could not start subprocess");
-        throw new RuntimeException("Could not start subprocess");
+        //What's the best way to log this
+        System.out.println("Error: Could not start subprocess");
+        throw new RuntimeException("Error: Could not start subprocess");
+      }
{noformat}

Seems odd, like it was getting logged via {{LOG}} just fine, and then replaced with a comment asking how to log it.

I also noticed that it seems to create a new jar for {{avro-tools}} with {{maven-assembly-plugin}} that's supposed to be useable as a hadoop job jar. However, I found that I actually had to use the {{-withdeps}} jar that's created to get it to work. So maybe the code that creates the {{-job}} jar isn't necessary? It also looks like the {{-withdeps}} jar was just the originally created jar but now has a classifier.

I wouldn't want to drastically change the artifacts being published with this change, so that might need more scrutiny. I also personally tested with {{-Dhadoop.version=2}}, I had some issues compiling and testing without that, and I didn't try {{-Dhadoop.version=1}}.

> python implementation of mapreduce connector
> --------------------------------------------
>
>                 Key: AVRO-570
>                 URL: https://issues.apache.org/jira/browse/AVRO-570
>             Project: Avro
>          Issue Type: New Feature
>          Components: python
>    Affects Versions: 1.7.0
>            Reporter: Doug Cutting
>            Assignee: Jeremy Lewi
>            Priority: Critical
>              Labels: hadoop
>             Fix For: 1.8.0
>
>         Attachments: AVRO-570.patch, AVRO-570.patch, AVRO-570.patch, AVRO-570.patch, AVRO-570.patch, AVRO-570.patch, AVRO-570.patch, AVRO-570.patch
>
>
> AVRO-512 defines protocols for implementing mapreduce tasks.  It would be good to have a Python implementation of this.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)