nifi-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Jorge Machado <jom...@me.com>
Subject Re: Stackoverflow question: Moving data from one RDB to another through NiFi
Date Thu, 22 Mar 2018 07:27:00 GMT
Hi Bryan Sure ? Database to Database ? Or  with a step in between ? Can you past the command
that you use ? That would be new to me and I would be interested.

Jorge Machado





> On 22 Mar 2018, at 08:22, Brett Ryan <brett.ryan@gmail.com> wrote:
> 
> Sure it does, I’m using it for postgres and MariaDB (which is essentially MySQL).
> 
>> On 22 Mar 2018, at 18:18, Jorge Machado <jomach@me.com> wrote:
>> 
>> Sqoop does not import into a mySql database. Just into Hive if you tell him to do
so. 
>> You could use Nifi but if you have a lot of data may be you should try Spark.  which
reads and writes in Parallel. 
>> Using Nifi would work to but you have the overhead of pumping the data over “insert”
unless you copy the files into the server and on the server then use some import bulk…..
>> 
>> Jorge Machado
>> 
>> 
>> 
>> 
>> 
>>> On 22 Mar 2018, at 08:13, Brett Ryan <brett.ryan@gmail.com> wrote:
>>> 
>>> Could Sqoop [1] be an option?
>>> 
>>> [1]: http://sqoop.apache.org/
>>> 
>>>> On 22 Mar 2018, at 16:33, Sivaprasanna <sivaprasanna246@gmail.com>
wrote:
>>>> 
>>>> I had a chance to attempt a question raised on stackoverflow regarding
>>>> moving data from SQL Server to MySQL using NiFi. The user is using
>>>> GenerateTableFetch to read data from SQL Server and then try to use LOAD
>>>> DATA command in ExecuteSQL but this involves writing the read SQL Server
>>>> data to filesystem and then load it, which is a performance hit, I
>>>> suggested the user to try PutDatabaseRecord but I have never tried the
>>>> approach myself and going by the docs, I think it won't show any
>>>> performance benefit than LOAD DATA because the former reads from file and
>>>> inserts at a high speed while the latter reads content and parses it
>>>> according to the configured Record Reader and insert the rows as a single
>>>> batch. Confused, I wanted to get the community's opinion/thoughts on this.
>>>> Please attempt the questions, if you have better suggestions.
>>>> 
>>>> Links:
>>>> 
>>>> -
>>>> https://stackoverflow.com/questions/49400447/bulk-load-sql-server-data-into-mysql-apache-nifi?noredirect=1#comment85843021_49400447
>>>> -
>>>> https://stackoverflow.com/questions/49380307/flowfile-absolute-path-nifi/49398500?noredirect=1#comment85805848_49398500
>>>> 
>>>> Thanks,
>>>> 
>>>> Sivaprasanna
>> 


Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message