ignite-user mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Vladimir Ozerov <voze...@gridgain.com>
Subject Re: about mr accelerator question.
Date Tue, 15 Mar 2016 07:53:15 GMT

1) If you have 200G of data, and all this data is used, 6 nodes with 24G
each will not be able to hold it. Possible solutions:
- Allocate more memory. Note that having 24G on-heap and 24G off-heap,
doesn't mean that you have 48G for IGFS. IGFS store data in either onheap or
offheap depending on data cache configuration, but not in both. 
- If you cannot allocate more memory, you can configure
IgfsPerBlockLruEvictionPolicy which will evict some blocks from memory if
memory consumption is too high and pull them from secondary file system
again when needed. It might affect performance, but will prevent
- Also please note that Hadoop Accelerator require bigger amount of
permgen/metaspace than normal application.

We can dig further if you provide the following information:
- XML configuration you use to start node.
- Exact reason of out-of-memory error.

2) About HDFS startup - when running HDFS as a secondary file system, it is
better not to change default file system in the main code-site.xml because
Hadoop expects it to be of "hdfs" type. Instead, you can specify only IGFS
classes in this file, and access IGFS using fully-qualified paths. E.g.
"igfs:///path/to/file" instead of "/path/to/file". 
Alternatively, you can create separate configuration file with default file
system set to IGFS and then specify it when starting Hadoop. E.g.:
hadoop --config [folder_with_your_config] ...

View this message in context: http://apache-ignite-users.70518.x6.nabble.com/about-mr-accelerator-question-tp3502p3509.html
Sent from the Apache Ignite Users mailing list archive at Nabble.com.

View raw message