streampipes-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Johannes Tex" <johannes....@web.de>
Subject AW: AW: Run and develop StreamPipes without configuring the IP address
Date Sat, 16 Nov 2019 14:27:31 GMT
Hi,

in regarding to the SP_DATA_LOCATION configuration of the connect-adapters:
It also can be removed and the path can be set automatically if the adapter
is running in debug mode. 

The only disadvantage is that the users cannot simply change the file
locations of the uploaded files. Are there scenarios where this would be
necessary?

Johannes

-----Urspr√ľngliche Nachricht-----
Von: Dominik Riemer [mailto:riemer@fzi.de] 
Gesendet: Samstag, 16. November 2019 13:26
An: dev@streampipes.apache.org
Betreff: Re: AW: Run and develop StreamPipes without configuring the IP
address

Hi,

cool, the config for pipeline elements looks good and is *much* easier to
understand.

For connect-adapters, can't we also remove SP_KAFKA_HOST (can be
automatically set in debug mode similar to pipeline elements)? What about
SP_DATA_LOCATION?

For the backend config, we need to remove Kafka-Rest anyways (Confluent
Community License) and AFAIK could also remove uri (RDF4J is now managed
internally) and Zookeeper_HOST (not longer needed).
Let's do some tests whether we can also just set a DEBUG flag when
developing on the backend that automatically sets KAFKA_HOST, COUCHDB_HOST
and JMS_HOST to localhost.

Dominik


On 2019/11/16 09:31:31, "Johannes Tex" <johannes.tex@web.de> wrote:
> Hi,
>
>
>
> I think we don't need any more these configurations for the backend :
>
> -        SP_DATALAKE_HOST
>
> -        SP_DATALAKE_PORT
>
>
>
> It was used for the first version of the data-lake with Elasticsearch , we
> changed the data-lake database to Influx and removed the old data-lake API
> with Elasticsearch last week.
>
>
>
> Furthermore I think it would be useful to add the port configuration for
> Influx:
>
> SP_INFLUX_HOST=8086
>
>
>
>
>
> Regards
>
> Johannes
>
>
>
>
>
>
>
> Von: Philipp Zehnder [mailto:zehnder@fzi.de]
> Gesendet: Samstag, 16. November 2019 09:57
> An: dev@streampipes.apache.org
> Betreff: Run and develop StreamPipes without configuring the IP address
>
>
>
> Hi all,
>
>
>
> we removed the requirement to define the servers IP address when
StreamPipes
> is run completely in Docker.
>
> However, we still have to change several configs to ensure the new setting
> also works during development.
>
>
>
> This change effects the following projects:
>
>
>
> * streampipes
>
>   * Do we still need all of those configurations for the backend?
>
> SP_COUCHDB_HOST=localhost
> uri=http://localhost:8031/rdf4j-server
> SP_KAFKA_HOST=host.docker.internal
> SP_ZOOKEEPER_HOST=host.docker.internal
> SP_JMS_HOST=host.docker.internal
> SP_KAFKA_REST_HOST=host.docker.internal
> SP_KAFKA_REST_PORT=8073
> SP_BACKEND_HOST=host.docker.internal
> SP_ELASTICSEARCH_HOST=host.docker.internal
> SP_DATALAKE_HOST=host.docker.internal
> SP_DATALAKE_PORT=9200
> SP_INFLUX_HOST=host.docker.internal
>
>
>
> * streampipes-pipeline-elements
>
>   * Suggestion of an example configuration for processing containers:
>
>
>
> SP_PORT=6060
> SP_HOST=host.docker.internal
> SP_DEBUG=true
>
>
>
> * streampipes-connect-adapters
>
>   * Suggestion of an example configuration for connect adapters
>
>
>
> SP_KAFKA_HOST=localhost
>
> SP_CONNECT_CONTAINER_HOST=localhost
>
> SP_CONNECT_CONTAINER_MASTER_HOST=localhost
>
> SP_CONNECT_CONTAINER_WORKER_HOST=localhost
>
> SP_DATA_LOCATION=./test_data/
>
> SP_DEBUG=true
>
>
>
>
>
> Did I forget any configurations we have to change?
>
>
>
> I am still not completely certain when to use localhost and when to use
> host.docker.internal during development. Currently I added
> host.docker.internal to my /etc/hosts file.
>
> Is there a way to remove this entry?
>
>
>
> Additionally I suggest to remove all deployment folders (containing a
> docker-compose.yml file) in the projects and manage them centrally in the
> cli project.
>
>
>
> Cheers,
>
> Philipp
>
>


Mime
View raw message