apex-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Thomas Weise <tho...@datatorrent.com>
Subject Re: Vagrant folders under Malhar/lib
Date Thu, 03 Sep 2015 15:42:54 GMT
It makes sense to use the synchronous checkpointing for the local mode. LM
is meant to simplify dependencies and setup. The default for execution on
YARN remains async.

Thomas


On Thu, Sep 3, 2015 at 8:34 AM, Chandni Singh <chandni@datatorrent.com>
wrote:

> APPLICATION_PATH isn't related to local base dir of Async as far as I
> know. StramLocalCluster sets the APP_PATH to "target/...".
> StramLocalCluster should use FSStorageAgent.
>
> - Chandni
>
> On Thu, Sep 3, 2015 at 8:20 AM, Gaurav Gupta <gaurav@datatorrent.com>
> wrote:
>
>> As Thomas mentioned as default remains to be async. You can either change
>> the storage agent or set the APPLICATION_PATH.
>>
>> When container runs in cluster, "." specifies the containers local path
>> on the node where container specific jars and other resources resides. It
>> creates a folder under that which is live as long as container lives. So
>> there are no vagrant folders anywhere
>>
>> Thanks
>> -Gaurav
>>
>> On Wed, Sep 2, 2015 at 11:33 PM, Chandni Singh <chandni@datatorrent.com>
>> wrote:
>>
>>> I think there is a problem in the default Async as well. It also uses
>>> the working directory as its local base path.
>>>
>>> In the Async -> copyToHdfs()  method, we delete the window files but the
>>> folder with the operator name never gets deleted.
>>> So on the cluster there  will be such vagrant folders in the working
>>> directory?
>>>
>>> On Wed, Sep 2, 2015 at 11:17 PM, Thomas Weise <thomas@datatorrent.com>
>>> wrote:
>>>
>>>> Chandni,
>>>>
>>>> Agreed. See whether the tests work with the synchronous storage agent.
>>>> If yes, change them. The default needs to remain async.
>>>>
>>>> Thomas
>>>>
>>>>
>>>> On Wed, Sep 2, 2015 at 11:05 PM, Chandni Singh <chandni@datatorrent.com
>>>> > wrote:
>>>>
>>>>> Hi,
>>>>>
>>>>> I would like to know what was the reason to use AsyncFSStorageAgent
>>>>> with StramLocalCluster?
>>>>> StramLocalCluster is mainly for testing in a non-distributed mode and
>>>>> I am unclear how AsyncFSStorageAgent is helpful in this mode.
>>>>>
>>>>> Thanks,
>>>>> Chandni
>>>>>
>>>>> On Wed, Sep 2, 2015 at 10:45 PM, Chandni Singh <
>>>>> chandni@datatorrent.com> wrote:
>>>>>
>>>>>> This is because of recent changes to StramLocalCluster where
>>>>>> AsyncFSStorageAgent is used for checkpointing
>>>>>>
>>>>>> dag.setAttribute(OperatorContext.STORAGE_AGENT, new AsyncFSStorageAgent(new
Path(pathUri, LogicalPlan.SUBDIR_CHECKPOINTS).toString(), null));
>>>>>>
>>>>>> The AsyncFSStorageAgent(String path, Configuration conf) uses "."
as localBasePath and therefore creates sub-directories per operator in the current working
directory.
>>>>>>
>>>>>> I am going to create a ticket to address this and will fix it.
>>>>>>
>>>>>> -Chandni
>>>>>>
>>>>>>
>>>>>> On Wed, Sep 2, 2015 at 7:13 PM, Chandni Singh <
>>>>>> chandni@datatorrent.com> wrote:
>>>>>>
>>>>>>> Hi,
>>>>>>>
>>>>>>> I can see empty folders getting created under Malhar/lib called
'1'
>>>>>>> and '2'.
>>>>>>> I think this is because of using LocalMode to run a test application.
>>>>>>>
>>>>>>>
>>>>>>> If anyone has checked in such cases please do check and let us
know.
>>>>>>>
>>>>>>> Thanks,
>>>>>>> Chandni
>>>>>>>
>>>>>>
>>>>>>
>>>>>
>>>>
>>>
>>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message