camel-users mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From rwaheng <rwah...@gmail.com>
Subject SFTP leader election using ZooKeeper RoutePolicy:- Error on moving file
Date Fri, 13 Jan 2017 07:20:38 GMT
from(" 
sftp://X.X.X.X:2222/ts/sdf?username=sftpuser&password=*****&delay=100&initialDelay=100&streamDownload=false&recursive=false&readLock=changed&disconnect=true&KnownHostsFile=~/.ssh/known_hosts&localWorkDirectory=/tmp&moveFailed=/ts/error&include=.*\.xml&move=inprogress")
                .routePolicy(new ZooKeeperRoutePolicy("{{zookeeper." +
feedName + ".forwarder.endpoint}}", 1))
               
.to("sftp://Y.Y.Y.T:2222/sftp/sdf?username=sftpuser&password=*****&fileExist=move&moveExisting=20170112221528-***.xml&KnownHostsFile=~/.ssh/known_hosts");

I narrow done the issue to above code snippet, its FTPing the file from 1
FTP server to another with move=inprogress folder using the 
ZooKeeperRoutePolicy.
- it works as expected on the single instance.
- on having multiple instance(of camel route) , FTPing is successful but
move file to inprogress is failing due to below error. However this error
happen only on the first file(205.xml). As you can see on the log, the 
subsequence file(206.xml) are transfer sucessfully as expected

according to error message it is "Caused by: com.jcraft.jsch.SftpException:
No such file."
But if i remove the "move=inprogress" from the route, i can see the file in
/sftp/sd folder.

is there any other way to FTP the file and move the file to folder(using
zookeeper for leader election)? or how do i eliminate this error?


2017-01-12T00:11:31.768517747Z 2017-01-12 00:11:31  INFO [vzw/sdf]
[o.a.c.c.z.policy.ZooKeeperElection :138]  Initializing ZookeeperElection
with uri '{{zookeeper.sdf.forwarder.endpoint}}'
2017-01-12T00:11:31.788697843Z 2017-01-12 00:11:31  INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper     :100]  Client
environment:zookeeper.version=3.4.8--1, built on 02/06/2016 03:18 GMT
2017-01-12T00:11:31.788763764Z 2017-01-12 00:11:31  INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper     :100]  Client
environment:host.name=a810a44c767c
2017-01-12T00:11:31.788774252Z 2017-01-12 00:11:31  INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper     :100]  Client
environment:java.version=1.8.0_60
2017-01-12T00:11:31.788783141Z 2017-01-12 00:11:31  INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper     :100]  Client
environment:java.vendor=Oracle Corporation
2017-01-12T00:11:31.788791331Z 2017-01-12 00:11:31  INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper     :100]  Client
environment:java.home=/opt/jre1.8.0_60
2017-01-12T00:11:31.789350564Z 2017-01-12 00:11:31  INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper     :100]  Client
environment:java.class.path=/app.jar
2017-01-12T00:11:31.789388119Z 2017-01-12 00:11:31  INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper     :100]  Client
environment:java.library.path=/usr/java/packages/lib/amd64:/usr/lib64:/lib64:/lib:/usr/lib
2017-01-12T00:11:31.790108758Z 2017-01-12 00:11:31  INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper     :100]  Client
environment:java.io.tmpdir=/tmp
2017-01-12T00:11:31.791108364Z 2017-01-12 00:11:31  INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper     :100]  Client
environment:java.compiler=<NA>
2017-01-12T00:11:31.791583396Z 2017-01-12 00:11:31  INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper     :100]  Client environment:os.name=Linux
2017-01-12T00:11:31.792170295Z 2017-01-12 00:11:31  INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper     :100]  Client environment:os.arch=amd64
2017-01-12T00:11:31.792194966Z 2017-01-12 00:11:31  INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper     :100]  Client
environment:os.version=4.4.39-moby
2017-01-12T00:11:31.792379046Z 2017-01-12 00:11:31  INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper     :100]  Client environment:user.name=root
2017-01-12T00:11:31.793107176Z 2017-01-12 00:11:31  INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper     :100]  Client
environment:user.home=/root
2017-01-12T00:11:31.793138639Z 2017-01-12 00:11:31  INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper     :100]  Client
environment:user.dir=/etc/service/dataf_gateway
2017-01-12T00:11:31.794761800Z 2017-01-12 00:11:31  INFO [vzw/sdf]
[org.apache.zookeeper.ZooKeeper     :438]  Initiating client connection,
connectString=68.128.182.149:2181 sessionTimeout=5000
watcher=org.apache.camel.component.zookeeper.ConnectionHolder@4ec951ba
2017-01-12T00:11:31.813834319Z 2017-01-12 00:11:31  INFO [9:2181)]
[org.apache.zookeeper.ClientCnxn    :032]  Opening socket connection to
server 68.128.182.149/68.128.182.149:2181. Will not attempt to authenticate
using SASL (unknown error)
2017-01-12T00:11:31.904467961Z 2017-01-12 00:11:31  INFO [9:2181)]
[org.apache.zookeeper.ClientCnxn    :876]  Socket connection established to
68.128.182.149/68.128.182.149:2181, initiating session
2017-01-12T00:11:32.006541339Z 2017-01-12 00:11:32  INFO [9:2181)]
[org.apache.zookeeper.ClientCnxn    :299]  Session establishment complete on
server 68.128.182.149/68.128.182.149:2181, sessionid = 0x159855fc87400c2,
negotiated timeout = 5000
2017-01-12T00:11:32.109418356Z 2017-01-12 00:11:32  WARN [vzw/sdf]
[o.a.c.c.zookeeper.ZookeeperProducer:237]  Node
'/zk/file_forwarder_node/a810a44c767c-8a3d4433-5042-4e24-bf45-694afc61c66a'
did not exist, creating it.
2017-01-12T00:11:32.398124509Z 2017-01-12 00:11:32  INFO [vzw/sdf]
[o.a.c.c.z.policy.ZooKeeperElection :151]  Candidate node
'/zk/file_forwarder_node/a810a44c767c-8a3d4433-5042-4e24-bf45-694afc61c66a'
has been created
2017-01-12T00:11:32.431922439Z 2017-01-12 00:11:32  INFO [vzw/sdf]
[o.a.camel.impl.DefaultCamelContext :570]  Route:
election-route-a810a44c767c-8a3d4433-5042-4e24-bf45-694afc61c66a started and
consuming from:
Endpoint[zookeeper://68.128.182.149:2181/zk/file_forwarder_node?createMode=EPHEMERAL]
2017-01-12T00:11:32.620071603Z 2017-01-12 00:11:32  INFO [vzw/sdf]
[Datafeed.Sdf.dc3Forwarder          :159]  Moving Starts fileName=205.xml
from /ts/vzw/sdf
2017-01-12T00:11:39.674491004Z 2017-01-12 00:11:39  INFO [vzw/sdf]
[Datafeed.Sdf.dc3Forwarder          :159]  Moving Completed fileName=205.xml
transfer to sftp://68.128.182.149:2222
2017-01-12T00:11:40.692277702Z 2017-01-12 00:11:40  WARN [vzw/sdf] 


+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++
+++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++

[o.a.c.c.f.GenericFileOnCompletion  :213]  Error during commit.
Exchange[ID-a810a44c767c-44403-1484179857573-0-2]. Caused by:
[org.apache.camel.component.file.GenericFileOperationFailedException -
Cannot rename file from: ts/sdf/205.xml to: ts/sdf/inprogress/205.xml]
2017-01-12T00:11:40.692309664Z
org.apache.camel.component.file.GenericFileOperationFailedException: Cannot
rename file from: ts//sdf/205.xml to: ts/sdf/inprogress/205.xml
2017-01-12T00:11:40.692319752Z 	at
org.apache.camel.component.file.remote.SftpOperations.renameFile(SftpOperations.java:431)
2017-01-12T00:11:40.692331238Z 	at
org.apache.camel.component.file.strategy.GenericFileProcessStrategySupport.renameFile(GenericFileProcessStrategySupport.java:113)
2017-01-12T00:11:40.692336731Z 	at
org.apache.camel.component.file.strategy.GenericFileRenameProcessStrategy.commit(GenericFileRenameProcessStrategy.java:88)
2017-01-12T00:11:40.692341626Z 	at
org.apache.camel.component.file.GenericFileOnCompletion.processStrategyCommit(GenericFileOnCompletion.java:127)
2017-01-12T00:11:40.692349716Z 	at
org.apache.camel.component.file.GenericFileOnCompletion.onCompletion(GenericFileOnCompletion.java:83)
2017-01-12T00:11:40.692354510Z 	at
org.apache.camel.component.file.GenericFileOnCompletion.onComplete(GenericFileOnCompletion.java:57)
2017-01-12T00:11:40.692359205Z 	at
org.apache.camel.util.UnitOfWorkHelper.doneSynchronizations(UnitOfWorkHelper.java:104)
2017-01-12T00:11:40.692364099Z 	at
org.apache.camel.impl.DefaultUnitOfWork.done(DefaultUnitOfWork.java:229)
2017-01-12T00:11:40.692375485Z 	at
org.apache.camel.util.UnitOfWorkHelper.doneUow(UnitOfWorkHelper.java:65)
2017-01-12T00:11:40.692381378Z 	at
org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:674)
2017-01-12T00:11:40.692386372Z 	at
org.apache.camel.processor.CamelInternalProcessor$UnitOfWorkProcessorAdvice.after(CamelInternalProcessor.java:629)
2017-01-12T00:11:40.692390967Z 	at
org.apache.camel.processor.CamelInternalProcessor$InternalCallback.done(CamelInternalProcessor.java:246)
2017-01-12T00:11:40.692395761Z 	at
org.apache.camel.processor.CamelInternalProcessor.process(CamelInternalProcessor.java:179)
2017-01-12T00:11:40.692409245Z 	at
org.apache.camel.component.file.GenericFileConsumer.processExchange(GenericFileConsumer.java:454)
2017-01-12T00:11:40.692414039Z 	at
org.apache.camel.component.file.remote.RemoteFileConsumer.processExchange(RemoteFileConsumer.java:137)
2017-01-12T00:11:40.692418534Z 	at
org.apache.camel.component.file.GenericFileConsumer.processBatch(GenericFileConsumer.java:226)
2017-01-12T00:11:40.692422929Z 	at
org.apache.camel.component.file.GenericFileConsumer.poll(GenericFileConsumer.java:190)
2017-01-12T00:11:40.692427323Z 	at
org.apache.camel.impl.ScheduledPollConsumer.doRun(ScheduledPollConsumer.java:175)
2017-01-12T00:11:40.692432817Z 	at
org.apache.camel.impl.ScheduledPollConsumer.run(ScheduledPollConsumer.java:102)
2017-01-12T00:11:40.692437211Z 	at
java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:511)
2017-01-12T00:11:40.692441606Z 	at
java.util.concurrent.FutureTask.runAndReset(FutureTask.java:308)
2017-01-12T00:11:40.692449497Z 	at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.access$301(ScheduledThreadPoolExecutor.java:180)
2017-01-12T00:11:40.692457887Z 	at
java.util.concurrent.ScheduledThreadPoolExecutor$ScheduledFutureTask.run(ScheduledThreadPoolExecutor.java:294)
2017-01-12T00:11:40.692464779Z 	at
java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1142)
2017-01-12T00:11:40.692469273Z 	at
java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:617)
2017-01-12T00:11:40.692474567Z 	at java.lang.Thread.run(Thread.java:745)
2017-01-12T00:11:40.692479761Z Caused by: com.jcraft.jsch.SftpException: No
such file
2017-01-12T00:11:40.692484055Z 	at
com.jcraft.jsch.ChannelSftp.throwStatusError(ChannelSftp.java:2846)
2017-01-12T00:11:40.692488550Z 	at
com.jcraft.jsch.ChannelSftp.rename(ChannelSftp.java:1923)
2017-01-12T00:11:40.692493245Z 	at
org.apache.camel.component.file.remote.SftpOperations.renameFile(SftpOperations.java:428)
2017-01-12T00:11:40.692503133Z 	... 25 common frames omitted

2017-01-12T00:12:08.133521922Z 2017-01-12 00:12:08  INFO [/sdf]
[Dataf.Sdf.dc3Forwarder          :159]  Moving Starts fileName=206.xml from
/ts/sdf
2017-01-12T00:12:11.826120872Z 2017-01-12 00:12:11  INFO [/sdf]
[Dataf.Sdf.dc3Forwarder          :159]  Moving Completed fileName=206.xml
transfer to sftp://X.X.X.X:2222



--
View this message in context: http://camel.465427.n5.nabble.com/SFTP-leader-election-using-ZooKeeper-RoutePolicy-Error-on-moving-file-tp5792480.html
Sent from the Camel - Users mailing list archive at Nabble.com.

Mime
View raw message