asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From Yingyi Bu <buyin...@gmail.com>
Subject Re: Do we have a method to append local files to existed dataset?
Date Fri, 04 Mar 2016 19:53:15 GMT
Xikui,

If the number of localfs files is too large,  a solution could be to put
your files on HDFS and then load it.  Loading from HDFS always has a fixed
degree of parallelism regardless of the number of files.

>> I am wondering is there a way to append adm file to existed dataset?
You can create an external dataset and then write an insert statement where
the body is a scan query. AsterixDB doesn't load any data into its own
storage for an external dataset but just keeps file paths.
Here is a manual for external datasets:
https://ci.apache.org/projects/asterixdb/aql/externaldata.html

Best,
Yingyi


On Fri, Mar 4, 2016 at 11:47 AM, Xikui Wang <xikuiw@uci.edu> wrote:

> Hi,
>
> I want to import data from multiple adm files into a same dataset. Merging
> them together and then loading from localfs can be a viable solution, but
> this may become a problem when the number become too large. I am wondering
> is there a way to append adm file to existed dataset?
>
> Thank you.
>
> Best,
> Xikui
>

Mime
  • Unnamed multipart/alternative (inline, None, 0 bytes)
View raw message