falcon-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Rai, Jainendra" <J...@beckman.com>
Subject RE: Falcon Incremental Import support not available
Date Wed, 01 Jun 2016 13:30:30 GMT
Hi Team,

It seems the code could have some errors..

This is the method that throws the error.. The method says that incremental and Mergetype
append is supported but then according to this Incremental extract will always throw the exception.

Am I looking into it correct? Please advise

[cid:image003.jpg@01D1BC37.DA3B0820]

Kind Regards
Jainendra
[unnamed] Go Green

From: Jainendra Rai [mailto:jainendra72@gmail.com]
Sent: Wednesday, June 01, 2016 6:56 PM
To: Rai, Jainendra
Subject: Fwd: Falcon Incremental Import support not available


Best Regards,
+91-9741823970
jainendra72@gmail.com<mailto:jainendra72@gmail.com>

---------- Forwarded message ----------
From: Jainendra Rai <jainendra72@gmail.com<mailto:jainendra72@gmail.com>>
Date: Wed, Jun 1, 2016 at 1:41 PM
Subject: Falcon Incremental Import support not available
To: dev@falcon.apache.org<mailto:dev@falcon.apache.org>

Hi Team,

We were trying to run an Import job with Apache Falcon. The document says that
ยท        [incremental, append] - data is extracted incrementally using the key specified
in the deltacolumn is supported


<import>

                <source name="mysql-db" tableName="simple">

                    <extract type="incremental">

                        <deltacolumn>modified_time</deltacolumn>

                        <mergepolicy>append</mergepolicy>

                    </extract>

                    <fields>

                        <includes>

                            <field>id</field>

                            <field>name</field>

                        </includes>

                    </fields>

                </source>

                <arguments>

                    <argument name="--split-by" value="id"/>

                    <argument name="--num-mappers" value="2"/>

                </arguments>
</import>


Whille [incremental, snapshot] -  this is not supported.

We were trying to work with the former one and suprisingly when we submit the feed we get
an Error

ERROR: Bad Request;default/Feed feed-employee2 is using unsupported extraction mechanism incremental

Log:

 2016-06-01 08:04:24,761 INFO  - [718278514@qtp-2060207812-55 - c3a7419c-f8f6-4b93-9bf8-1349ec166407:hadoop:POST//entities/submit/feed]
~ All locks released on feed-employee2 (AbstractEntityManager:371)
2016-06-01 08:04:24,762 ERROR - [718278514@qtp-2060207812-55 - c3a7419c-f8f6-4b93-9bf8-1349ec166407:hadoop:POST//entities/submit/feed]
~ Unable to persist entity object (AbstractEntityManager:194)
org.apache.falcon.entity.parser.ValidationException: Feed feed-employee2 is using unsupported
extraction mechanism incremental
        at org.apache.falcon.entity.parser.FeedEntityParser.validateFeedExtractionType(FeedEntityParser.java:595)
        at org.apache.falcon.entity.parser.FeedEntityParser.validate(FeedEntityParser.java:105)
        at org.apache.falcon.entity.parser.FeedEntityParser.validate(FeedEntityParser.java:71)
        at org.apache.falcon.resource.AbstractEntityManager.validate(AbstractEntityManager.java:519)
        at org.apache.falcon.resource.AbstractEntityManager.submitInternal(AbstractEntityManager.java:443)
        at org.apache.falcon.resource.AbstractEntityManager.submit(AbstractEntityManager.java:191)
        at org.apache.falcon.resource.ConfigSyncService.submit(ConfigSyncService.java:52)
        at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:57)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at org.apache.falcon.resource.channel.IPCChannel.invoke(IPCChannel.java:49)
        at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$2.doExecute(SchedulableEntityManagerProxy.java:177)
        at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$EntityProxy.execute(SchedulableEntityManagerProxy.java:773)
        at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.submit_aroundBody2(SchedulableEntityManagerProxy.java:179)
        at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy$AjcClosure3.run(SchedulableEntityManagerProxy.java:1)
        at org.aspectj.runtime.reflect.JoinPointImpl.proceed(JoinPointImpl.java:149)
        at org.apache.falcon.aspect.AbstractFalconAspect.logAroundMonitored(AbstractFalconAspect.java:51)
        at org.apache.falcon.resource.proxy.SchedulableEntityManagerProxy.submit(SchedulableEntityManagerProxy.java:157)
        at sun.reflect.GeneratedMethodAccessor55.invoke(Unknown Source)
        at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.lang.reflect.Method.invoke(Method.java:606)
        at com.sun.jersey.spi.container.JavaMethodInvokerFactory$1.invoke(JavaMethodInvokerFactory.java:60)
        at com.sun.jersey.server.impl.model.method.dispatch.AbstractResourceMethodDispatchProvider$TypeOutInvoker._dispatch(AbstractResourceMethodDispatchProvider.java:185)
        at com.sun.jersey.server.impl.model.method.dispatch.ResourceJavaMethodDispatcher.dispatch(ResourceJavaMethodDispatcher.java:75)
        at com.sun.jersey.server.impl.uri.rules.HttpMethodRule.accept(HttpMethodRule.java:288)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.ResourceClassRule.accept(ResourceClassRule.java:108)
        at com.sun.jersey.server.impl.uri.rules.RightHandPathRule.accept(RightHandPathRule.java:147)
        at com.sun.jersey.server.impl.uri.rules.RootResourceClassesRule.accept(RootResourceClassesRule.java:84)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1469)
        at com.sun.jersey.server.impl.application.WebApplicationImpl._handleRequest(WebApplicationImpl.java:1400)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1349)
        at com.sun.jersey.server.impl.application.WebApplicationImpl.handleRequest(WebApplicationImpl.java:1339)
        at com.sun.jersey.spi.container.servlet.WebComponent.service(WebComponent.java:416)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:537)
        at com.sun.jersey.spi.container.servlet.ServletContainer.service(ServletContainer.java:699)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:820)
        at org.mortbay.jetty.servlet.ServletHolder.handle(ServletHolder.java:511)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1221)
        at org.apache.falcon.security.FalconAuthorizationFilter.doFilter(FalconAuthorizationFilter.java:108)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.apache.falcon.security.FalconAuthenticationFilter$2.doFilter(FalconAuthenticationFilter.java:188)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:595)
        at org.apache.hadoop.security.authentication.server.AuthenticationFilter.doFilter(AuthenticationFilter.java:554)
        at org.apache.falcon.security.FalconAuthenticationFilter.doFilter(FalconAuthenticationFilter.java:197)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.apache.falcon.security.FalconAuditFilter.doFilter(FalconAuditFilter.java:64)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.apache.falcon.security.HostnameFilter.doFilter(HostnameFilter.java:82)
        at org.mortbay.jetty.servlet.ServletHandler$CachedChain.doFilter(ServletHandler.java:1212)
        at org.mortbay.jetty.servlet.ServletHandler.handle(ServletHandler.java:399)
        at org.mortbay.jetty.security.SecurityHandler.handle(SecurityHandler.java:216)
        at org.mortbay.jetty.servlet.SessionHandler.handle(SessionHandler.java:182)
        at org.mortbay.jetty.handler.ContextHandler.handle(ContextHandler.java:766)
        at org.mortbay.jetty.webapp.WebAppContext.handle(WebAppContext.java:450)
        at org.mortbay.jetty.handler.HandlerWrapper.handle(HandlerWrapper.java:152)
        at org.mortbay.jetty.Server.handle(Server.java:326)
        at org.mortbay.jetty.HttpConnection.handleRequest(HttpConnection.java:542)
        at org.mortbay.jetty.HttpConnection$RequestHandler.content(HttpConnection.java:945)
        at org.mortbay.jetty.HttpParser.parseNext(HttpParser.java:756)
        at org.mortbay.jetty.HttpParser.parseAvailable(HttpParser.java:212)
        at org.mortbay.jetty.HttpConnection.handle(HttpConnection.java:404)
        at org.mortbay.jetty.bio.SocketConnector$Connection.run(SocketConnector.java:228)
        at org.mortbay.jetty.security.SslSocketConnector$SslConnection.run(SslSocketConnector.java:713)
        at org.mortbay.thread.QueuedThreadPool$PoolThread.run(QueuedThreadPool.java:582)
2016-06-01 08:04:24,763 INFO  - [718278514@qtp-2060207812-55 - c3a7419c-f8f6-4b93-9bf8-1349ec166407:hadoop:POST//entities/submit/feed]
~ {Action:submit, Dimensions:{entityType=feed, colo=NULL}, Status: FAILED, Time-taken:11824449
ns} (METRIC:38)
2016-06-01 08:04:24,763 ERROR - [718278514@qtp-2060207812-55 - c3a7419c-f8f6-4b93-9bf8-1349ec166407:hadoop:POST//entities/submit/feed]
~ Exception in sending Notification from EmailNotificationPlugin:java.lang.NullPointerException
(EmailNotificationPlugin:71)
2016-06-01 08:04:24,764 DEBUG - [718278514@qtp-2060207812-55 - c3a7419c-f8f6-4b93-9bf8-1349ec166407:]
~ Audit: hadoop/127.0.0.1<https://urldefense.proofpoint.com/v2/url?u=http-3A__127.0.0.1&d=CwMFaQ&c=9mghv0deYPYDGP-W745IEdQLV1kHpn4XJRvR6xMRXtA&r=4MEyUWrAGv2l6VkTuU5AUQ&m=6O59RRLhSGucTB0o7DaVzqLkQEDbTFeWrLQMrtbULMk&s=AuZ2Uqn6QXWYp_ESwW_1AMsG9fhQCT1olHCBR6HPoRg&e=>
performed request https://localhost:15443/api/entities/submit/feed<https://urldefense.proofpoint.com/v2/url?u=https-3A__localhost-3A15443_api_entities_submit_feed&d=CwMFaQ&c=9mghv0deYPYDGP-W745IEdQLV1kHpn4XJRvR6xMRXtA&r=4MEyUWrAGv2l6VkTuU5AUQ&m=6O59RRLhSGucTB0o7DaVzqLkQEDbTFeWrLQMrtbULMk&s=jKPtKEZZX76LCp06hGh_Vlv4AbotHIotULACNZmpbUE&e=>
(127.0.0.1) at time 2016-06-01T08:04Z (FalconAuditFilter:86)
2016-06-01 08:04:24,764 INFO  - [718278514@qtp-2060207812-55 - c3a7419c-f8f6-4b93-9bf8-1349ec166407:]
~ Audit: hadoop@127.0.0.1<mailto:hadoop@127.0.0.1> performed https://localhost:15443/api/entities/submit/feed<https://urldefense.proofpoint.com/v2/url?u=https-3A__localhost-3A15443_api_entities_submit_feed&d=CwMFaQ&c=9mghv0deYPYDGP-W745IEdQLV1kHpn4XJRvR6xMRXtA&r=4MEyUWrAGv2l6VkTuU5AUQ&m=6O59RRLhSGucTB0o7DaVzqLkQEDbTFeWrLQMrtbULMk&s=jKPtKEZZX76LCp06hGh_Vlv4AbotHIotULACNZmpbUE&e=>
(127.0.0.1) at 2016-06-01T08:04Z (AUDIT:48



Should this error come up.

PLease have a look at the submitted feed too.

<?xml version="1.0" encoding="UTF-8"?>
      <!--
       A feed representing Hourly customer email data retained for 90 days
       -->
      <feed description="Raw customer email feed" name="feed1" xmlns="uri:falcon:feed:0.1">
          <tags>externalSystem=USWestEmailServers,classification=secure</tags>

          <groups>DataImportPipeline</groups>
          <frequency>minutes(6)</frequency>
          <late-arrival cut-off="hours(4)"/>
          <clusters>
              <cluster name="primaryCluster" type="source">
                  <validity start="2016-05-23T00:00Z" end="2017-05-31T00:00Z"/>
                  <retention limit="days(90)" action="delete"/>
                  <import>
                      <source name="mysql-db" tableName="employee">

 <extract type="incremental">
                        <deltacolumn>modified_item</deltacolumn>
                        <mergepolicy>append</mergepolicy>
                    </extract>



                      </source>
 <arguments>
                          <argument name="--split-by" value="id"/>
                          <argument name="--num-mappers" value="2"/>
                      </arguments>

                  </import>


              </cluster>
          </clusters>


          <locations>
              <location type="data" path="/falcon/${YEAR}-${MONTH}-${DAY}-${HOUR}-${MINUTE}"/>
              <location type="stats" path="/none"/>
              <location type="meta" path="/none"/>
          </locations>
<notification type="email" to="xyz.com<https://urldefense.proofpoint.com/v2/url?u=http-3A__xyz.com&d=CwMFaQ&c=9mghv0deYPYDGP-W745IEdQLV1kHpn4XJRvR6xMRXtA&r=4MEyUWrAGv2l6VkTuU5AUQ&m=6O59RRLhSGucTB0o7DaVzqLkQEDbTFeWrLQMrtbULMk&s=0bHHCBjZnnogp83kfVwhfyxKkEWQizicw7CrGfPklGI&e=>"/>
          <ACL owner="hadoop" group="hadoop" permission="0755"/>
          <schema location="/none" provider="none"/>

      </feed>


Please be advised that this email may contain confidential information. If you are not the
intended recipient, please notify us by email by replying to the sender and delete this message.
The sender disclaims that the content of this email constitutes an offer to enter into, or
the acceptance of, any agreement; provided that the foregoing does not invalidate the binding
effect of any digital or other electronic reproduction of a manual signature that is included
in any attachment.
Mime
  • Unnamed multipart/related (inline, None, 0 bytes)
View raw message