streampipes-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Dominik Riemer <rie...@fzi.de>
Subject Re: AW: Run and develop StreamPipes without configuring the IP address
Date Sat, 16 Nov 2019 12:25:52 GMT
Hi,

cool, the config for pipeline elements looks good and is *much* easier to understand.

For connect-adapters, can't we also remove SP_KAFKA_HOST (can be automatically set in debug
mode similar to pipeline elements)? What about SP_DATA_LOCATION?

For the backend config, we need to remove Kafka-Rest anyways (Confluent Community License)
and AFAIK could also remove uri (RDF4J is now managed internally) and Zookeeper_HOST (not
longer needed).
Let's do some tests whether we can also just set a DEBUG flag when developing on the backend
that automatically sets KAFKA_HOST, COUCHDB_HOST and JMS_HOST to localhost.

Dominik


On 2019/11/16 09:31:31, "Johannes Tex" <johannes.tex@web.de> wrote:
> Hi,
>
>
>
> I think we don't need any more these configurations for the backend :
>
> -        SP_DATALAKE_HOST
>
> -        SP_DATALAKE_PORT
>
>
>
> It was used for the first version of the data-lake with Elasticsearch , we
> changed the data-lake database to Influx and removed the old data-lake API
> with Elasticsearch last week.
>
>
>
> Furthermore I think it would be useful to add the port configuration for
> Influx:
>
> SP_INFLUX_HOST=8086
>
>
>
>
>
> Regards
>
> Johannes
>
>
>
>
>
>
>
> Von: Philipp Zehnder [mailto:zehnder@fzi.de]
> Gesendet: Samstag, 16. November 2019 09:57
> An: dev@streampipes.apache.org
> Betreff: Run and develop StreamPipes without configuring the IP address
>
>
>
> Hi all,
>
>
>
> we removed the requirement to define the servers IP address when StreamPipes
> is run completely in Docker.
>
> However, we still have to change several configs to ensure the new setting
> also works during development.
>
>
>
> This change effects the following projects:
>
>
>
> * streampipes
>
>   * Do we still need all of those configurations for the backend?
>
> SP_COUCHDB_HOST=localhost
> uri=http://localhost:8031/rdf4j-server
> SP_KAFKA_HOST=host.docker.internal
> SP_ZOOKEEPER_HOST=host.docker.internal
> SP_JMS_HOST=host.docker.internal
> SP_KAFKA_REST_HOST=host.docker.internal
> SP_KAFKA_REST_PORT=8073
> SP_BACKEND_HOST=host.docker.internal
> SP_ELASTICSEARCH_HOST=host.docker.internal
> SP_DATALAKE_HOST=host.docker.internal
> SP_DATALAKE_PORT=9200
> SP_INFLUX_HOST=host.docker.internal
>
>
>
> * streampipes-pipeline-elements
>
>   * Suggestion of an example configuration for processing containers:
>
>
>
> SP_PORT=6060
> SP_HOST=host.docker.internal
> SP_DEBUG=true
>
>
>
> * streampipes-connect-adapters
>
>   * Suggestion of an example configuration for connect adapters
>
>
>
> SP_KAFKA_HOST=localhost
>
> SP_CONNECT_CONTAINER_HOST=localhost
>
> SP_CONNECT_CONTAINER_MASTER_HOST=localhost
>
> SP_CONNECT_CONTAINER_WORKER_HOST=localhost
>
> SP_DATA_LOCATION=./test_data/
>
> SP_DEBUG=true
>
>
>
>
>
> Did I forget any configurations we have to change?
>
>
>
> I am still not completely certain when to use localhost and when to use
> host.docker.internal during development. Currently I added
> host.docker.internal to my /etc/hosts file.
>
> Is there a way to remove this entry?
>
>
>
> Additionally I suggest to remove all deployment folders (containing a
> docker-compose.yml file) in the projects and manage them centrally in the
> cli project.
>
>
>
> Cheers,
>
> Philipp
>
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message