Hi,
I'm looking for an opinion on whether this method of swapping memory between hard disk and RAM seems sensible.
I'm planning on storing a lot of data (for what exactly isn't really important). There will probably be far too much to hold in memory so I'm planning on swapping it in and out of RAM as it's accessed.
My basic plan is to implement a queue, say with 500 spaces. When a data item is queried, the queue would first be searched. If it's not in the queue, the data item would be loaded and placed at the head of the queue. If the data item is found in the queue, it would be moved to the queue's head and then returned.
As the queue grows it will eventually reach its max limit (500 in this example). When this has happened, if more items are queried, the items at the very bottom of the queue would be popped off and written to disk.
This is the first time I've tried to design something like this, so I'm not entirely sure if it's the best solution. The main advantage that I can see is that the most frequently accessed data items will be stored near the head of the queue, and so will be returned the fastest.
Does this sound sane?
ID:151956
 Aug 12 2008, 5:16 am
|
|
Jtgibson wrote:
Quite sane. It's a standard hybrid between caching and freeing memory. Most likely the list will hold pointers to where the other data is. It may or may not be in RAM depending on what's needed. I haven't worked out the exact details on that micro-level, but it's good to know I'm on the right sort of lines. | |
If you're using pointers, all of your data needs to be loaded into memory at once and you're probably going to have quite a bit of problems. That's not what you're going to want to do.
| |
Sounds sane, it's effectively an implementation of the Least Recently Used caching strategy. Based on your pointer comment, would it be fair to suggest this isn't going to be in DM?
| |
CaptFalcon33035 wrote:
If you're using pointers, all of your data needs to be loaded into memory at once and you're probably going to have quite a bit of problems. That's not what you're going to want to do. Huh? If I use pointers then I would only need to load the pointers. The data can be loaded when something queries the pointer. Maybe I'm explaining myself wrong, but my implementation would work something like: type pointer : { data : object; file : string; } When a process queries some data from the pointer, it will first check if data is null. If it is, it will attempt to load data using the filename held in file. Data could be reset to null (taking it out of memory) if a condition is met (e.g. the pointer's data hasn't been queried in a long time or x number of other pointers have been loaded). I'm not sure what you mean when you say using pointers means all the data has to be loaded at once. Pointers are just data items that tell a process where to find the object when it's needed, whether it's loaded in RAM or held on disk. | |
Stephen001 wrote:
Sounds sane, it's effectively an implementation of the Least Recently Used caching strategy. Based on your pointer comment, would it be fair to suggest this isn't going to be in DM? Actually it is for a DM project. I mean pointers in an abstract sense, so it should be doable in any OO language really. | |
DM's native pointers are transparent and will reference VM address space. How did you plan on handling access to object members that may or may not be on disk?
Edit: Nevermind, I've just seen your reply to Capt. | |
Wouldn't it be better to perhaps implement a procedure on your pointer like:
pointer As such, the fact the data is in RAM or not is not a matter users of the pointer have to worry about, seen as it strikes me as the responsibility of the pointer to provide the data it points to. | |
Stephen001 wrote:
Wouldn't it be better to perhaps implement a procedure on your pointer like: > pointer As such, the fact the data is in RAM or not is not a matter users of the pointer have to worry about, seen as it strikes me as the responsibility of the pointer to provide the data it points to. Yeah, that's how I'll implement it when it comes to translating the data structure to DM. But when I'm designing data structures and algorithms in pseudocode I like to keep the data seperate from the algorithms. I'm not quite at the implementation stage for this project yet ^.^ Still drawing up the game design document and musing about how I could implement it all. The game should pretty much rock ass when it's realised, so watch this space :P | |
Stephen001 wrote:
I'd be happy to help with the 'cache' mechanism, if you feel you need help. I should be able to get it done without too much of a problem once I brush up on my DM, but thanks. Any walls I run into will probably be described on these forums, juxtaposed with 'wtf's and '*beats head off desk*'s :) | |
However, it depends on what exactly you're caching -- if it's everything in one 500-element list, that might not be enough space. If you're caching only a certain type (or group of types) of objects, a 500-element object table would be plenty.