oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Lewis John Mcgibbney <lewis.mcgibb...@gmail.com>
Subject Re: CAS pge-configuration - stage file on batch stub without shared filesystem
Date Thu, 08 Oct 2015 21:09:03 GMT
Hi Andrea,
Just picked this up from my inbox.
Did you find a solution?
If not then we can try to help.
Thanks

On Mon, Oct 5, 2015 at 9:28 AM, Andrea Fonti <andrea.fonti@altecspace.it>
wrote:

> Hi,
>
>
>
> I would like to implement the following scenario (all components are
> virtually separated host, implemented as a linked docker containers
> topology):
>
>
>
> 1.       [PUSHPULL] - The push pull component download files to the staging
> area
>
> 2.       [CRAWLER] - The crawler component extracts metadata and triggers
> filemgr ingestion
>
> 3.       [FILEMGR] - Filemgr ingest metadata in solr and archives the
> ingested file
>
> 4.       [CRAWLER] - Sends a ProductType-Ingest event to WMGR
>
> 5.       [WMGR] - Receives the event that trigger a data processing
> workflow: the ingested file should be processed and a new data product
> should be generated and ingested
>
> 6.       [RMGR] - Handles the request to schedule the processing task and
> dispatches it to the batch stub
>
> 7.       [BATCHSTUB] - a configured batch stub performs the processing task
> over the ingested file and produces the new data product
>
>
>
> I already implemented 1,2,3,4,5,6 but  I'm stuck with a problem:
>
>
>
> How can I instruct the batchstub to query the filemgr, initiate a data
> transfer to stage the source data product in a directory on the batch stub
> host, perform the processing and then stage back the new product? I would
> like to not rely on a shared filesystem.
>
>
>
> Any help would be really appreciated.
>
>
>
> Best regards
>
> Andrea Fonti
>
> ---- ALTEC S.p.A. ----
>
> COMUNICAZIONE IMPORTANTE
> La presente email è destinata solo al/i destinatario/i sopra indicato/i.
> Può riguardare un messaggio confidenziale e contenere informazioni e dati
> di
> natura RISERVATA O CLASSIFICATA. Non è consentita la sua lettura indebita,
> la sua duplicazione o la sua divulgazione. Le opinioni dell'autore di
> questa
> email non sono  necessariamente le opinioni di ALTEC S.p.A.
> Nulla di quanto contenuto in questa email può sostituire un contratto
> ovvero una obbligazione vincolante per ALTEC S.p.A.
> Nel caso doveste ricevere la presente email per errore, siete
> cortesemente pregati di distruggere la Vostra copia e di contattarci
> telefonicamente o a mezzo email o facsimile. Grazie.
>
> IMPORTANT NOTICE
> This email is intended only for the use of the above mentioned
> addressee(s).
> It is confidential and may contain legally PRIVILEGED OR CLASSIFIED
> information and data. Unauthorized reading, copying, disclosure or use is
> strictly prohibited. The views of the author hereof may not necessarily
> constitute the views of ALTEC S.p.A.
> Nothing in this email shall bind ALTEC S.p.A. in any contract or
> obligation.
> If you received this email/facsimile through an error, please destroy your
> copy and telephone or email/fax us immediately. Thank you.
>



-- 
*Lewis*

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message