oodt-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Andrea Fonti <andrea.fo...@altecspace.it>
Subject CAS pge-configuration - stage file on batch stub without shared filesystem
Date Mon, 05 Oct 2015 16:28:10 GMT
Hi,

 

I would like to implement the following scenario (all components are
virtually separated host, implemented as a linked docker containers
topology):

 

1.       [PUSHPULL] - The push pull component download files to the staging
area

2.       [CRAWLER] - The crawler component extracts metadata and triggers
filemgr ingestion

3.       [FILEMGR] - Filemgr ingest metadata in solr and archives the
ingested file

4.       [CRAWLER] - Sends a ProductType-Ingest event to WMGR

5.       [WMGR] - Receives the event that trigger a data processing
workflow: the ingested file should be processed and a new data product
should be generated and ingested

6.       [RMGR] - Handles the request to schedule the processing task and
dispatches it to the batch stub

7.       [BATCHSTUB] - a configured batch stub performs the processing task
over the ingested file and produces the new data product

 

I already implemented 1,2,3,4,5,6 but  I'm stuck with a problem:

 

How can I instruct the batchstub to query the filemgr, initiate a data
transfer to stage the source data product in a directory on the batch stub
host, perform the processing and then stage back the new product? I would
like to not rely on a shared filesystem.

 

Any help would be really appreciated.

 

Best regards

Andrea Fonti

---- ALTEC S.p.A. ----

COMUNICAZIONE IMPORTANTE
La presente email č destinata solo al/i destinatario/i sopra indicato/i.
Puņ riguardare un messaggio confidenziale e contenere informazioni e dati di
natura RISERVATA O CLASSIFICATA. Non č consentita la sua lettura indebita,
la sua duplicazione o la sua divulgazione. Le opinioni dell'autore di questa
email non sono  necessariamente le opinioni di ALTEC S.p.A.
Nulla di quanto contenuto in questa email puņ sostituire un contratto
ovvero una obbligazione vincolante per ALTEC S.p.A.
Nel caso doveste ricevere la presente email per errore, siete
cortesemente pregati di distruggere la Vostra copia e di contattarci
telefonicamente o a mezzo email o facsimile. Grazie.

IMPORTANT NOTICE
This email is intended only for the use of the above mentioned addressee(s).
It is confidential and may contain legally PRIVILEGED OR CLASSIFIED
information and data. Unauthorized reading, copying, disclosure or use is
strictly prohibited. The views of the author hereof may not necessarily
constitute the views of ALTEC S.p.A.
Nothing in this email shall bind ALTEC S.p.A. in any contract or obligation.
If you received this email/facsimile through an error, please destroy your
copy and telephone or email/fax us immediately. Thank you.

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message