streampipes-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Philipp Zehnder <zehn...@fzi.de>
Subject Re: AW: Run and develop StreamPipes without configuring the IP address
Date Mon, 18 Nov 2019 14:06:36 GMT
Hi,

thanks for your tips.

I tried to harmonize the configurations for StreamPipes during development. Please try it
out with your local IDE in case I missed something.

There are three different set up’s how you can run StreamPipes:

1. Run all in docker:
  * This works now without changing anything

2. Develop PE’s / adapters in an IDE and run the rest in docker
  * Users must set the environment variables according to the 'development/env' file
  * The rest should work as expected.

3. Develop backend in IDE 
  * To develop the backend the file ‘/etc/hosts’ must be edited
    * Add the following line: '127.0.0.1 localhost host.docker.internal’

3.1 Start backend, ui, PE’s, connect-master, connect-workers in IDE
   * Works without any further changes

3.2 Run backend + ui in IDE, PE’s and adapters in docker
  * This setup currently does not work out of the box. Users have to set several configurations
like the couchdb, or hosts of PE’s / adapters manually
  * For this setup it depends what the user starts in the IDE and what is started in docker.
Some services use the same configuration variables and they must be configured accordingly.

3.3. Run backend in IDE, UI in docker
  * Also requires additional configuration. The nginx server configurations must be edited.


Locations of config files:
  * backend (streampipes-backend/development/env)
  * connect-Master (streampipes-connect-container/development/env)
  * connect-worker-main (streampipes-connect-adapters/development/env)
  * pipeline-elements: (in each module in the /development directory)

Additionally I deleted the deployment directories and renamed the .env files to env (some
IDE’s hide files with a leading dot).

Cheers, 
Philipp

On 2019/11/16 09:31:31, "Johannes Tex" <johannes.tex@web.de> wrote: 
> Hi,
> 
> 
> 
> I think we don't need any more these configurations for the backend :
> 
> -        SP_DATALAKE_HOST
> 
> -        SP_DATALAKE_PORT
> 
> 
> 
> It was used for the first version of the data-lake with Elasticsearch , we
> changed the data-lake database to Influx and removed the old data-lake API
> with Elasticsearch last week.
> 
> 
> 
> Furthermore I think it would be useful to add the port configuration for
> Influx:
> 
> SP_INFLUX_HOST=8086
> 
> 
> 
> 
> 
> Regards
> 
> Johannes
> 
> 
> 
> 
> 
> 
> 
> Von: Philipp Zehnder [mailto:zehnder@fzi.de]
> Gesendet: Samstag, 16. November 2019 09:57
> An: dev@streampipes.apache.org
> Betreff: Run and develop StreamPipes without configuring the IP address
> 
> 
> 
> Hi all,
> 
> 
> 
> we removed the requirement to define the servers IP address when StreamPipes
> is run completely in Docker.
> 
> However, we still have to change several configs to ensure the new setting
> also works during development.
> 
> 
> 
> This change effects the following projects:
> 
> 
> 
> * streampipes
> 
>   * Do we still need all of those configurations for the backend?
> 
> SP_COUCHDB_HOST=localhost
> uri=http://localhost:8031/rdf4j-server
> SP_KAFKA_HOST=host.docker.internal
> SP_ZOOKEEPER_HOST=host.docker.internal
> SP_JMS_HOST=host.docker.internal
> SP_KAFKA_REST_HOST=host.docker.internal
> SP_KAFKA_REST_PORT=8073
> SP_BACKEND_HOST=host.docker.internal
> SP_ELASTICSEARCH_HOST=host.docker.internal
> SP_DATALAKE_HOST=host.docker.internal
> SP_DATALAKE_PORT=9200
> SP_INFLUX_HOST=host.docker.internal
> 
> 
> 
> * streampipes-pipeline-elements
> 
>   * Suggestion of an example configuration for processing containers:
> 
> 
> 
> SP_PORT=6060
> SP_HOST=host.docker.internal
> SP_DEBUG=true
> 
> 
> 
> * streampipes-connect-adapters
> 
>   * Suggestion of an example configuration for connect adapters
> 
> 
> 
> SP_KAFKA_HOST=localhost
> 
> SP_CONNECT_CONTAINER_HOST=localhost
> 
> SP_CONNECT_CONTAINER_MASTER_HOST=localhost
> 
> SP_CONNECT_CONTAINER_WORKER_HOST=localhost
> 
> SP_DATA_LOCATION=./test_data/
> 
> SP_DEBUG=true
> 
> 
> 
> 
> 
> Did I forget any configurations we have to change?
> 
> 
> 
> I am still not completely certain when to use localhost and when to use
> host.docker.internal during development. Currently I added
> host.docker.internal to my /etc/hosts file.
> 
> Is there a way to remove this entry?
> 
> 
> 
> Additionally I suggest to remove all deployment folders (containing a
> docker-compose.yml file) in the projects and manage them centrally in the
> cli project.
> 
> 
> 
> Cheers,
> 
> Philipp
> 
>  

.........................................................
M. Sc. Philipp Zehnder
Wissenschaftlicher Mitarbeiter | Research Scientist
Information Process Engineering (IPE)
 
FZI Forschungszentrum Informatik
Haid-und-Neu-Str. 10–14 
76131 Karlsruhe, Germany
Tel.: +49 721 9654-805
Fax: +49 721 9654-806
 
zehnder@fzi.de <mailto:zehnder@fzi.de>
https://www.fzi.de/wir-ueber-uns/organisation/mitarbeiter/address/zehnder/ <https://www.fzi.de/wir-ueber-uns/organisation/mitarbeiter/address/zehnder/>
 
.........................................................
FZI Forschungszentrum Informatik
Stiftung des bürgerlichen Rechts
Stiftung Az: 14-0563.1 Regierungspräsidium Karlsruhe
Vorstand: Prof. Dr. Andreas Oberweis, Jan Wiesenberger, Prof. Dr.-Ing. J. Marius Zöllner
Vorsitzender des Kuratoriums: Ministerialdirigent Günther Leßnerkraus
.........................................................


Mime
View raw message