asterixdb-dev mailing list archives

Site index · List index
Message view « Date » · « Thread »
Top « Date » · « Thread »
From "Yingyi Bu (Code Review)" <do-not-re...@asterix-gerrit.ics.uci.edu>
Subject Change in asterixdb[master]: Allow lazy loading for persistent local resources
Date Sun, 30 Aug 2015 00:02:42 GMT
Yingyi Bu has posted comments on this change.

Change subject: Allow lazy loading for persistent local resources
......................................................................


Patch Set 5:

(1 comment)

https://asterix-gerrit.ics.uci.edu/#/c/344/5/asterix-transactions/src/main/java/edu/uci/ics/asterix/transaction/management/resource/PersistentLocalResourceRepository.java
File asterix-transactions/src/main/java/edu/uci/ics/asterix/transaction/management/resource/PersistentLocalResourceRepository.java:

Line 205:     public HashMap<Long, LocalResource> loadAndGetAllResources() throws HyracksDataException
{
> Yes, we would have to persist a new file at the root of every IO Device and
IMO, "slower" is better than "crashing when too many resources"...
During recovery,  the buffer cache and in-memory components of indexes also consume memory.

Since this CL is monotonically better than the current master, I'm OK for merging it back.
I propose two minor changes:

1.  Add a TODO comment to say that during recovery, the memory usage currently is proportional
to the number of created indexes (this includes deleted indexes, I guess, because their directories
are not deleted and resource ids are not released.) and should be fixed.  (Or create an issue
for that.) 

2.  In ResourceIdFactoryProvider,  call getMaxResourceId() instead of getAllResources(). 
 In initialize(...),  call getMaxResourceId() instead of loadAndGetAllResources().   In getMaxResourceId(),
it just iterates over resources without loading them into a map.


-- 
To view, visit https://asterix-gerrit.ics.uci.edu/344
To unsubscribe, visit https://asterix-gerrit.ics.uci.edu/settings

Gerrit-MessageType: comment
Gerrit-Change-Id: I48b9260a3280750145f6ddb3783673a299055910
Gerrit-PatchSet: 5
Gerrit-Project: asterixdb
Gerrit-Branch: master
Gerrit-Owner: Murtadha Hubail <hubailmor@gmail.com>
Gerrit-Reviewer: Ian Maxon <imaxon@apache.org>
Gerrit-Reviewer: Jenkins <jenkins@fulliautomatix.ics.uci.edu>
Gerrit-Reviewer: Murtadha Hubail <hubailmor@gmail.com>
Gerrit-Reviewer: Yingyi Bu <buyingyi@gmail.com>
Gerrit-Reviewer: Yingyi Bu <yingyib@google.com>
Gerrit-Reviewer: Young-Seok Kim <kisskys@gmail.com>
Gerrit-Reviewer: abdullah alamoudi <bamousaa@gmail.com>
Gerrit-HasComments: Yes

Mime
View raw message