ignite-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Vladimir Ozerov (JIRA)" <j...@apache.org>
Subject [jira] [Created] (IGNITE-3167) Hadoop: restore external execution.
Date Wed, 18 May 2016 12:42:12 GMT
Vladimir Ozerov created IGNITE-3167:

             Summary: Hadoop: restore external execution.
                 Key: IGNITE-3167
                 URL: https://issues.apache.org/jira/browse/IGNITE-3167
             Project: Ignite
          Issue Type: Task
          Components: hadoop
    Affects Versions: 1.5.0.final
            Reporter: Vladimir Ozerov
            Priority: Critical
             Fix For: 1.7

Some time ago we decided to get rid external execution mode. It appears to be a wrong decision.

Hadoop users rely on its process-per-job nature in lot's places. One of such case could be
observed in HiBench Bayes benchmark:
1) Job creates something in the local file system through Hadoop FileSystem API.
2) Then it tries to get this data using regular java.io.FileReader and relative paths. 

This doesn't work in embedded mode because our LocalFileSystem wrapper assigns different work
dirs for jobs, but process-wide working directory is always the same. As a result, aforementioned
benchmark doesn't work in Ignite, but work with standard Hadoop job tracker.

It seems that we must return external execution back.

This message was sent by Atlassian JIRA

View raw message