gora-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From GitBox <...@apache.org>
Subject [GitHub] [gora] djkevincr commented on issue #161: GORA-565: Enable Spark in Unit Tests
Date Tue, 30 Apr 2019 07:04:13 GMT
djkevincr commented on issue #161: GORA-565: Enable Spark in Unit Tests
URL: https://github.com/apache/gora/pull/161#issuecomment-487840780
 
 
   ```
   public <K, V extends Persistent> Configuration generateOutputConf(DataStore<K,
V> dataStore)
          throws IOException {
   
         Configuration hadoopConf = ((Configurable) dataStore).getConf(); // Configuration
hadoopConf = new Configuration(); -  previously it was
         GoraMapReduceUtils.setIOSerializations(hadoopConf, true);
         Job job = Job.getInstance(hadoopConf);
   
         return generateOutputConf(job, dataStore.getClass(), dataStore.getKeyClass(),
              dataStore.getPersistentClass());
       }
   ```
   To properly fix the issue, I think GoraSparkEngine class method generateOutputConf should
be changed to the above. ( Check java single line comment ) Basically we should reuse the
Hadoop conf created at GoraMongodbTestDriver through out entire tests. Initial test startup
phase, we set these mongo server details to Hadoop conf created.
   I am not sure whether there exist a case which returns dataStore conf as null. ( Eg:- de
serilizations ) so we can do something similar to below. This code is extreacted from the
GoraSparkEngine class method initialize. Can you please address the changes and update the
PR?
   ```
   if ((dataStore instanceof Configurable)
           && ((Configurable) dataStore).getConf() != null) {
         hadoopConf = ((Configurable) dataStore).getConf();
       } else {
         hadoopConf = new Configuration();
       }
   ```
   

----------------------------------------------------------------
This is an automated message from the Apache Git Service.
To respond to the message, please log on to GitHub and use the
URL above to go to the specific comment.
 
For queries about this service, please contact Infrastructure at:
users@infra.apache.org


With regards,
Apache Git Services

Mime
View raw message