drill-issues mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Jacques Nadeau (JIRA)" <j...@apache.org>
Subject [jira] [Commented] (DRILL-3742) Improve classpath scanning to reduce the time it takes
Date Sun, 06 Sep 2015 00:56:45 GMT

    [ https://issues.apache.org/jira/browse/DRILL-3742?page=com.atlassian.jira.plugin.system.issuetabpanels:comment-tabpanel&focusedCommentId=14732148#comment-14732148
] 

Jacques Nadeau commented on DRILL-3742:
---------------------------------------

I've added an initial patch for lazy function initialization on top of this patch. You can
see my work here: 

https://github.com/jacques-n/drill/commits/classpath_scanning

One thing I found was that your existing function scanning logic duplicates entries after
the first run. This causes problems when trying to register functions. Currently for this
patch, one needs to need to delete the cache file after each run or else the next run will
fail to load the function registry.

With this setup on my machine, classpath scanning for functions and function registry initialization
average between 800ms and 1s.

> Improve classpath scanning to reduce the time it takes
> ------------------------------------------------------
>
>                 Key: DRILL-3742
>                 URL: https://issues.apache.org/jira/browse/DRILL-3742
>             Project: Apache Drill
>          Issue Type: Improvement
>            Reporter: Julien Le Dem
>            Assignee: Julien Le Dem
>
> classpath scanning and function registry take a long time (seconds every time).
> We'd want to avoid loading the classes (use bytecode inspection instead) and have a build
time cache to avoid doing the scanning at startup.



--
This message was sent by Atlassian JIRA
(v6.3.4#6332)

Mime
View raw message