avro-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jeremy Lewi (Commented) (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (AVRO-570) python implementation of mapreduce connector
Date Wed, 21 Dec 2011 05:59:30 GMT

    [ https://issues.apache.org/jira/browse/AVRO-570?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=13173878#comment-13173878
] 

Jeremy Lewi commented on AVRO-570:
----------------------------------

Doug, I'm a little confused by the stack trace that's being printed out. The stack trace shows
that "test_tether_word_count.py" is throwing the exception. However, the text (e.g)
{noformat}
{"name": "ReferencedRecord", "type": "record", 
{noformat}
looks like schemas that I think are being used in a different test, "test_protocol.py" which
shouldn't have anything to do with the mapreduce API. (Is it possible ant runs the tests in
different threads and jumbles the stdout from the different tests?).

My suggestion would be to run the individual tests manually as follows. You'll need to grab
my latest patch as I fixed a couple issues that would prevent this.
Build the python files and java jars
{noformat}
cd SVN_AVRO/lang/java
mvn package -DskipTests
cd SVN_AVRO/lang/py/
ant dist
{noformat}

Set your python path
{noformat}
export PYTHONPATH=SVN_AVRO/lang/py/build/src:SVN_AVRO/lang/py/build/test
{noformat}

Run the individual tests 
{noformat}
cd SVN_AVRO/lang/py
python build/test/test_tether_task.py 
python build/test/test_tether_task_runner.py 
python build/test/test_tether_word_count.py 
{noformat}

Please send me the complete output (not just stack traces) of the tests that fail as that
will help me troubleshoot the problem.

Thanks for your patience and help.
                
> python implementation of mapreduce connector
> --------------------------------------------
>
>                 Key: AVRO-570
>                 URL: https://issues.apache.org/jira/browse/AVRO-570
>             Project: Avro
>          Issue Type: New Feature
>          Components: python
>    Affects Versions: 1.6.0
>            Reporter: Doug Cutting
>            Assignee: Jeremy Lewi
>            Priority: Critical
>              Labels: hadoop
>             Fix For: 1.7.0
>
>         Attachments: AVRO-570.patch, AVRO-570.patch, AVRO-570.patch, AVRO-570.patch,
AVRO-570.patch, AVRO-570.patch, AVRO-570.patch
>
>
> AVRO-512 defines protocols for implementing mapreduce tasks.  It would be good to have
a Python implementation of this.

--
This message is automatically generated by JIRA.
If you think it was sent incorrectly, please contact your JIRA administrators: https://issues.apache.org/jira/secure/ContactAdministrators!default.jspa
For more information on JIRA, see: http://www.atlassian.com/software/jira

        

Mime
View raw message