The method I used was simple. Loop through all turfs in a certain area then save a single variable (which was a number between 1 and 9) they had. It was significantly slower than simply saving text to a file.
Loading is equally as slow (though removes the need to split up strings).
Try it if you don't believe me.
Just tested it again.
My method saves a 25x25 map as raw text to a 1.2kb file. Using BYONDs save files, the exact same data is saved to a file about 17kb in size. It's also slower. My exact code for saving things is this
SaveArea(nam,xx,yy,zz,size)vs this
SaveSlice(nam,xx,yy,zz,size) The bottom code is faster to save and load and results in smaller save files. | |
Both of your approaches need significant improvement:
SaveArea(nam,xx,yy,zz,size)
SaveSlice(nam,xx,yy,zz,size)Mainly, you aren't using savefile buffers in an efficient format. I strongly recommend you read my savefile tutorial, because you are generating a lot of excess data under the assumption that a savefile buffer is only good for a single value. In addition, you are adding a lot of overhead to the savefile generation owing directly to your string processing. Also, you should note that you don't need to pump all that data to the text file at once. You can append to the file bit by bit, thus saving a lot of overhead that would otherwise be spent putting all that information in strings. Even so, I would not recommend saving the icon_states. I'd really recommend a parameterized tile id list and a global palette lookup table, which will associate tile ids with tile types. You'll get a lot more mileage out of that approach, and I assure you it'll be a lot faster than the equivalent version of the text strings. In addition, each number in the buffer takes 5 bytes. If your icon_state strings reach greater than 3 characters in length, they will take the same or more disk space. The issue that your savefiles have, is that they are including redundant locational data (The position in the buffer should be enough to generate the locational data from), and you are also hanging on to the wrong type of information (in my opinion). Again, though, when it comes to this kind of setup, there is an upper threshold to how much data you can move at once, and you need to tune your savefile sizes to be sized according to the limit. Your earlier analogy relating to 3D games shows a serious misunderstanding of how 3D games are constructed, and how they relate to 2D games. 2D tile-based games are actually much less space-efficient than 3D heightmap based games in a lot of respects, so you are working with some of the hardest-to-pack information in the gaming world. Again, why is this criticism relevant? Because in 3D games you have to know about how to split information up into a manner that will allow the CPU and GPU to communicate efficiently. In your case, you have to worry about the HDD to CPU serialization. You need to find the optimal chunk size. Generally, with how I process and store information, keeping buffers of 32x32 to 64x64 is usually right at about where the savefiles are optimized in both speed and size. Go much above that upper bound, and you are going to see a lot of problems. | |
And without saving locational data, how do you suppose I load the correct turfs to the correct position?
| |
And without saving locational data, how do you suppose I load the correct turfs to the correct position? Their position in the savefile is generated from their position in the world. Therefore, their position in the savefile should match their given position in the world. As such, processing the same sequence of data again will result in the same layout. Stop and think about it. If they go in in order from top-right to bottom-left, when you read them back, you start at the top-right corner, and end at the bottom-left corner. Literally all you have to do is reverse the operation:
loadArea(nam,xx,yy,zz,size)Same goes for the slice loading. Reverse the operation.
loadSlice(nam,xx,yy,zz,size) | |
Maybe I'm missing something here. But how are you reading each individual line of a save file you're loading?
for(var/turf/t in turfs) F >> t.icon_state As far as I am aware, just doing this is causing the file to be read to a variable. You're not specifying which part of the file you're reading from, nor are you progressing through it in any sort of manner. You're looping through turfs, then sending the entire file to their icon_state. EDIT: I just tried it. It works, but there is still a brief delay when loading the world. It's effectively no different than the method I've been using. So now what? | |
The Mangic Man wrote:
As far as I am aware, just doing this is causing the file to be read to a variable. You're not specifying which part of the file you're reading from, nor are you progressing through it in any sort of manner. You're looping through turfs, then sending the entire file to their icon_state. This is why I recommended you read my savefile tutorial. You have misunderstandings about what savefile buffers are and how to use them. Let's pretend we have a savefile: var/savefile/F = new/savefile("derp.sav") Okay, so let's read it back: var/savefile/F = new/savefile("derp.sav") In order to understand what our output is, we need to understand how the savefile format actually works under the hood. A buffer is just a series of data that has been stored in a linear area in the file. Each buffer has a set length, and a set position in the file. By telling the savefile that we are saving to F["buffer2"], we are actually telling it to add a variable to the end of the buffer. By attempting to load from the savefile from the current buffer, we are actually grabbing an item from the beginning of the buffer and moving the position of the read marker to the next value. If we change the buffer during the read operation, we go back to the beginning of the buffer when we switch back to it. Changing the cd variable of the savefile effectively changes the top-level buffer we are reading from. the .eof variable is set to 1 when we reach the end of a particular buffer. Okay, so all of that means that our output when we run the above read operation should look like this: "Read default: 1" Now, to understand what is saved, and how much space it takes, we need to pay attention to how BYOND stores stuff within savefile buffers. All values are stored: [byte type] [serialized data] Numbers are stored as: [number byte] [byte1] [byte2] [byte3] [byte4] Strings are stored as: [string byte] [char1] ... [0x00] datums are stored as: [datum byte] [string refid] Note on datums: Datums will store each variable that isn't temp, const, global, etc, which is also not equal to its initial value. An internal buffer will be created within the savefile ".[refid]", and all variables of the datum that meet these criteria will be saved in child buffers named [variable] and with a value equal to the serialized value of that variable. lists are saved a lot like datums, but instead of variables, it saves each position of the list in a separate buffer. When we find a token in a buffer, the default action is to read the individual token from the buffer, construct it from prototype data if necessary, and then return it. Thus, since we are using F >> t.icon_state, t.icon_state will be set to equal from the next token in the default buffer of the savefile, not the entire savefile. | |
I just tried it. It works, but there is still a brief delay when loading the world. It's effectively no different than the method I've been using. So now what? Profiling information, please. Also, relevant information: 1) Are there any clients within view of tiles being loaded when the appearance updates are being performed? If so, it will trigger viewport resending, which will appear to be a lag spike. 2) How big are the chunks you are fobbing off to your savefile? 3) Are you breaking anywhere for forced permissive processing of queued actions? | |
Also, I've ran into another issue. At times I need to only load a part of a map (such as the entire left side).
Since I'm not saving locations, this means loading the entire file, looping through it all and then keeping check of the location and if it's within the range I need to load, loading it. The problem is, when it's not inside of the range I need to load, then what? At the moment I am loading it to a dummy object that doesn't get used in anyway. Is there any way of skipping through the save file like this without having to load each and every part of it? Anyway. 1. No. The process goes like this. 1. If checks to see if the map is already loaded, if it is the player is simply moved to it. 2. If not, it finds an empty z level (or creates one if needed). 3. It loads the map to the z level. 4. It moves the player to the z level. 2. The chunks I am using are 25x25. The delay when loading is very brief (we're talking 1/10th of a second, if not less), but it ruins the entire illusion of the world being seamless. 3. No. | |
I took the liberty of profiling our two approaches.
The methodology I used were three successive tests. One on a 1024x1024 map, one on a 256x256 map, and one on a 32x32 map. This should demonstrate the key differences in the algorithms, and show clear justification to which is superior, and why each and every one of your objections to my initial approach were inaccurate. It also demonstrates clearly why what you are encountering is not a problem with BYOND, but a problem with your approach. Test machine details: OS: Windows 7 x64 Ultimate RAM: 8GB DDR3 1600 HDD: 8TB (2x2TB internal raid, 2x1TB internal raid 2, 2x1TB external, one networked device, one USB backup.) CPU: Intel core i7 920 stock 3ghz, OC'd to 3.6ghz GPU: 2x GTX 660 in SLI mode (doesn't really matter) Format: The results for each test will be presented in tabular form, with Ter13's method at the top, and TMM's method at the bottom. Order of the tests has no relevance to their strengths or weaknesses. They are arbitrarily ordered. The first series of results for each test will show the self CPU time only of the function in question, as well as the final savefile size generated by the function. The second series of results for each test will show comparative CPU and filesize percentages. A value of <100% means that the test was faster/smaller than the compared method. A value of >100% means the test was slower/larger than the compared method. Screenshots: Each test will be followed by a series of screenshots demonstrating the profiling information for the sake of confirmation, as well as the file sizes of the two files generated by the tests. This will serve as a good-faith confirmation that the results of this test are indeed accurate, and the source code for the comparative test will be offered so that independent confirmation of these results is possible. The outcomes were as follows: Test 1: 1024x1024 map note: A slight alteration had to be made to the optimized approach. Exporting the text to file after building the file in strings simply was too slow to test. After 20 minutes, I gave up, modified your approach, and re-ran the test. Results:  Screenshots:   You will notice that load2 took so long that GMT rolled over past midnight, causing the profiler to report a negative self CPU time. We had to rely on Real time in order to validate the loading test, because I'm not sitting through another 55 minute test. Test 2: 256x256 map Results:  Screenshots:   note: Ran load2 twice for confirmation. Values were unexpectedly nonlinear, averaged results. Initial result was 13.112, after averaging, dropped to 13.111 Test 3: 32x32 map Results:  Screenshots:    The bottom screenshot shows the actual file size of the test1 savefile. Source code: Last but not least, here is the source code for the test: world Feel free to contradict my results, but be prepared to bring numbers. We're not doing this "Your method is just as slow" business, because even at the best case scenario, my method outperforms yours at a minimum of 31,000% on save, and a minimum of 300% on load. Not only that, but yours falls victim to a crippling lack of ability to scale up, while mine does not. Last but not least, you will notice that your file sizes all have an asterix next to them. This is because your file sizes are variable based on the average length of the tile ids being saved. Assuming You only use numbers, effectively this means that if you have 10 tile types, you will be multiplying your file size by 133%. Each factor of ten will increase this size a further 133%. Assuming you have 999 tile types or less, your files will always be smaller than mine. However, the moment you decide to implement variable linear property saving, your files will balloon out to be quite a bit larger than mine, and significantly slower than they already are. My savefile format supports up to 16,777,216 unique tile types without changing the file size. | |
This is interesting stuff.
| |
The Magic Man wrote:
Since I'm not saving locations, this means loading the entire file, looping through it all and then keeping check of the location and if it's within the range I need to load, loading it. No. There isn't a skip() command for the savefiles. Even if you load and then throw away the majority of the data, it will still be hundreds of times faster to not do string manipulation during loading. Thus, getting rid of the locational data eliminates your concern. A significant chunk of how long it takes to load maps in my approach, is the palette lookup and the instantiation of the turf itself. Thus, if you just throw the unneeded data away, you will be speeding up the overall approach. Were you to go back to saving your locational data, you'd end up bloating your loading CPU use by hundreds of times. | |
Clearly you're doing something different. And from the looks of it, you're trying to save the entire world as one huge chunk.
http://files.byondhome.com/TheMagicMan/saving.jpg I used the code I have, and then tried yours. This is a 1024x1024 world (mine saves it in slices of 32x32) Clearly yours is faster at saving while mine appears faster at loading the entire world. Which is the problem I'm having, loading is simply too slow. The entire world mine saves is 2.06MB divided into 1024 separate files. But how about testing yours under actual game circumstances? http://files.byondhome.com/TheMagicMan/world.jpg This image represents the world a player sees. The black square is the location they're stood at, the grey squares are surrounding maps (assume the world is divided into 32x32 squares). All of these squares need to be loaded. The red areas are areas from other maps that need to be loaded when the player loads a new area (to give the appearance of a seamless world), obviously it's not to scale, but you get the idea. These red areas also need to be loaded, and affixed to the edges of the map the player is currently on. What I have so far is basically doing this when a map is loaded, it requires loading 9 different maps at once (each 25x25 in size) of which 8 are also used as edges, it does it in about 0.06 seconds (I think, it's actually 0.04, 0.01 and 0.01, http://files.byondhome.com/TheMagicMan/loaded.jpg ) according to profiling. That might not seem like much, but it's more than enough to cause a noticeable delay when moving from one map to another. Yours probably will work better, but until I see it in action I can't say for sure, and you never answered my question about how I'd only load a small section of a map using your method (without loading it all and only using the small section I needed). | |
The Magic Man wrote:
But how about testing yours under actual game circumstances? You've just seen my algorithm operating at-load. The chunk sizes are entirely arbitrary, thus I was demonstrating that your algorithm simply does not scale, and suffers from significant speed loss due to your over-reliance on redundant data. The Magic Man wrote: Clearly yours is faster at saving while mine appears faster at loading the entire world. I have hard statistical data to back up that this is not the case. In every single scenario, mine outperformed yours. The Magic Man wrote: All of these squares need to be loaded. Sure. Multiply my data for 32x32 areas' loading operations by 9. You'll actually notice that if you do that, My results are still three times faster than your test. My algorithm would still outperform yours. As a matter of fact, my algorithm's performance is linear, while yours features an exponential decay, so the larger the scale of the application, the worse yours will get, while mine will simply continue performing similarly. The Magic Man wrote: 0.06 seconds... That might not seem like much, but it's more than enough to cause a noticeable delay when moving from one map to another. This is not enough to produce a significant hiccup unless your operations are blocking operations. You need to perform them in an asynchronous manner by forcibly giving way to other instructions on the scheduler stack. The Magic Man wrote: But how about testing yours under actual game circumstances? If you want to disagree with my results, feel free to contradict them, or my methodology, but either way, simply disagreeing isn't going to prove anything. You need to show me hard data, a clear methodology, and do so in a manner that is repeatable and able to be analyzed. I was kind enough to put several hours into waiting around for your method to finish loading. All the data and methodology is there for you to work with. Please, I invite you to prove me wrong, and to do so in a manner that demonstrates exactly where, and why I am wrong. | |
Quick comment on where this phantom non-profiled "lag" is coming from.
I'm willing to bet that the turf type in world.turf doesn't have an overridden New() function. Meaning quite simply, that you aren't seeing where the majority of this phantom stutter is coming from. When you are increasing world.maxz, you are going to be initializing world.maxx*world.maxy turfs in sequence. This is going to take some time, and will probably result in some visual stutter. This means 1) My loading approach isn't causing this stutter, and 2) Your loading approach isn't causing the stutter despite consuming between 3 and a few thousand times what it should be. The best way to fix this, is to pre-initialize your map with a set maximum number of z-layers ready to go before the players ever connect, but do it at runtime so you aren't generating a 1GB DMB. I'm pretty sure DS/DD will only send data regarding the map to the player based on what they can actually see, so the extra, blank layers won't affect your communication at all. It will just eat up some cycles during initialization, and increase your memory footprint. | |
Ter, I just wanted to say that you show some real dedication to helping people. Members like you are what makes BYOND's development community so great. Thanks for being awesome. lol
Sorry for going off topic, Magic Man! | |
You say that savefiles are thousands of times larger/slower than tokenized strings? I assure you they are NOT.
Again, a saving/loading system I set up for maps was able to bang through a million tiles in around 9 seconds. I've shown you the methodology, but I'd like to see your example so that I can tell you why it's slow, and how to improve it.
Otherwise, if you just want to tell me all about how your opinion is gospel, there's nothing further I can do to show you that you are mistaken.