SaveFile Size

Page: 1 2
ID:1185382
 
Mar 2 2013, 1:37 pm (Edited on Mar 2 2013, 1:45 pm)
Keywords: savefiles
(See the best response by Ter13.)
Problem description:
Hi,

I have noticed with some experimenting that my savefile size fluctuates greatly depending on how i handle the actual functions of my "Builder" proc.

mob/human
    proc
        BuildEquipment()
            var/olays[0]

            if(type_hair != "bald")
                var/hair = types_hair[type_hair || "bald"] + colour_hair;   olays += hair

            for(var/A in equipped)

                var/cr1
                var/obj/cr2
                var/icon/cr3
                var/item/equipment/B = equipped[A]

                cr1         = B.eqwear
                cr2         = new cr1()
                cr3         = cr2.icon + B.colour
                cr2.icon    = cr3
                olays += cr2
            src.overlays = olays

This above code is the code i was using in a previous environment and it generates a 6KB savefile with 2 player slots one of which has 40 items, 3 of them in equipped list.
..

The same code in my new enviroment but instead of using eqwear (which is a link to a "holder object" with the same icon just no changed icon state) i use just B.type and it outputs a 28kb save with 14 items, 2 in equipped.. bit large IMO.

                var/item/equipment/B = equipped[A]
                var/icon/eqicon

                eqicon      = B.icon + B.colour
                B.icon      = eqicon
                B.icon_state = "" //setting the iconstate back to null so it works with PM and stuff (usually on a map state)
                B.layer = B.olayer

                olays += B

Changing all the "Cr" parts to this snippet outputs a 6KB save. which is alright, it was previously outputting 16kb which i found odd (maybe cause i changed FILE_DIR settings?) (now gives 12kb with 76 items unequipped items, 2 equipped)


In any case im curious as to why the savefile sizes would change so much between the 2 different codes and enviroments, considering my old one still has more info stored yet less space taken.

Anyone got any thoughts?
Mar 2 2013, 5:05 pm
Looked at my old environment and noticed in the actual "save" proc that i was completely wiping the icon before saving using src.icon/overlays = null. then saving

Then i did abit of further code to clean it up by removing the directories (not that it did much other then remove icon = null lines)

I noticed that in old environment not wiping the icon/overlays then just F.dir.Remove(xxx) had a substantial increase in save sizes (42kb over 6kb.. WOA) whereas new enviroment was only 12kb over 9kb (when nulling icon/olay before save).


Why is this the case when you are wiping the data anyway through f.dir.remove or F["icons"] << null?
Mar 3 2013, 3:30 am
Have you tried deleting the savefile between saves?

Seems to me that you are probably storing a lot more data each time you change the format of the savefile because it's still got the old data format in the file as well as the new.
Mar 3 2013, 1:09 pm
Yes i am deleting the savefile each time.
Mar 3 2013, 1:24 pm
Wait, are you actually saving the overlays list?

Show me where you are doing your saving. When I get home tonight, I'll teach you how to write a really efficient saving and loading system that is also incredibly easy to use.
Mar 3 2013, 1:33 pm
I'm interested to see the results of this topic as well. It got me re-examining my save files, and wanting to improve them wherever possible.
Mar 3 2013, 1:33 pm
The current save proc.
mob/human
    proc
        PlayerSave()
        //  if(!istype(src,/mob/player)) return //Currently have not made players become well players
            var/savefile/save = new(savedirsave)
    //      var/savefile/data = new(savedirdata)

            save << src

            BuildBase() //Remaking icons after the wipe inside the Write.
            BuildEquipment()

            var/txtfile = file(savedirtxt)
            fdel(txtfile)
            save.ExportText("/",txtfile)

    Write(var/savefile/F)
        F["savefile_version"] << SVER
        src.overlays = null  //"Wiping the icons"
        src.icon = null
        src.underlays = null
        ..()

        F.dir.Remove("icon","overlays","underlays","controls","watching","camera")
        F["SavedX"] =   x
        F["SavedY"] =   y
        F["SavedZ"] =   z

    Read(var/savefile/F)
        loc         =   locate(F["SavedX"],F["SavedY"],F["SavedZ"])
        var/version
        F["savefile_version"] >> version
        if(isnull(version)) version = 0
        ..()
Mar 3 2013, 2:23 pm
Best response
Okay, I'm starting to understand where you are going wrong. You are actually saving the modified icons for that equipment in the savefile, which is why they've gotten so large.

Since you changed over to objects as equipment with icons that are generated with rgb overlays, it's caching the icon in the savefile. When you ave atoms, their entire contents list gets saved in the file too, so you are going to want to edit the read/write functions for equipment. My advice is to manually determine what variables are going to be saved.
Mar 3 2013, 2:27 pm
I wipe the items aswell (well i dir.Remove them anyway)
item
    Write(var/savefile/F)

        ..()

        F.dir.Remove("overlays","mouse_drag_pointer","icon","layer","icon_state","step_x","step_y","px","py","pwidth","pheight")
Mar 3 2013, 2:58 pm
I'll point out what's going on when I get home.

I can't type code too well on my android phone, so it'll be about four hours until I get off work.
Mar 3 2013, 3:00 pm
Thats cool, i know how annoying it is on the phone aswell :p ive tried
Mar 4 2013, 1:01 am
First of all, a couple of things:

1) When you were doing F["icons"] << null, you were actually just wiping out the first item in the icon buffer being saved. Most of the time, that's all that's there, though.

In order to delete a buffer, you need to set F.cd to "icons", then set eof to -1. That's the only way to delete an entire buffer.

2) I'd like you to download foomer's savefile editor real fast: http://www.byond.com/developer/Foomer/SavefileEditor#

Savefiles store objects in hidden directories counting up from .0. You won't be able to see them using savefile.dir

That will show you everything in the file you are saving, and what byte for byte, is actually being saved.

If you have any questions about how to eliminate the fat in your files, feel free to ask. In the meantime, check out the tutorial I'm writing.

BYOND generates files that are roughly optimal... If you also want to save icon files themselves in the savefiles. If you don't, well, you are going to have to tell BYOND to stop saving icon files.

I'll be completing an efficient, versioned savefile system that is somewhat... more space-efficient (but slower) than BYOND's default system.
Mar 4 2013, 1:31 am
Well i am wiping all icons, well it was what i was aiming for anyway (and i assumed i was achieving that) so that i have as small as possible savefiles, as i played a game quite awhile back and eventually found out that their savefile sizes were up in the 2mb range due to icons/overlays/underlays all being saved which should be deemed by anyone as unacceptable for a online game IMO.

If i remove any wiping of the icons i do get large files, which as i pointed out above is expected.

The main things i was curious about is as i pointed out in my previous posts i get 2 different sizes for the same amount of stuff saved.

all cases are saved with absolutely no change other then small code adjustments.

Case 1. Using just f.dir.Remove("icon","underlays","overlays") i receive a 10kb savefile which is acceptable for now

Case 2. Wiping the Icons,Underlays and Overlays in the Write() proc before the ..() by using src.icons/etc = null results in a 7kb savefile (then use f.dir.remove to remove those from the listing no point having them there)


(this one is just to point out the different sizes no other relevance to what i really what to figure out)
Case 3. Removed all forms of "dir.remove" and received a 21kb save


So the main thing i was really wondering is why does making the actual vars null before saving, as indicated in case 2. have a smaller result then case 1. they are BOTH essentially doing the exact same thing which is removing the data.

I also read your post and everything in that i was already familiar with the reason i went with save << src is because i do use quite afew tmp vars (else my saves would be larger) and figured it would be easier in the long run if done properly to just use that method instead of manually making a large list of things wanting to be saved. i simply assumed F.dir.remove would indeed REMOVE the data saved but i guess thats not truly the case.


Also on the note of foomers savefile editor, im currently outputting everything to a .txt aswell so i can see everything that is saved (exactly what is shown by the dem)
Mar 4 2013, 1:39 am
Well, setting icons to null isn't actually going to remove them from the save completely.

Every variable will be saved if they change from their compile-time values.

You are way better off specifying which variables need to be saved. You'll wind up with way smaller file sizes and less headaches in the long run. BYOND's default saving system is pretty tight, but honestly, I think I can do better, and make it way easier in the long run. Part two is in the works.
Mar 4 2013, 2:03 am
I guess ill wait and see what part 2 showcases.

But when im making the icons and whatnot null i do understand that are still technically saved as icon = null in the savefile thats why i go one more step and then actually f.dir.Remove("icon") aswell to get rid of the residual.

and a main reason im reluctant to go with a full specify system is because i do use afew datums for defining things such as stats, they would i assume save fine just using f["str"] << str (which is a datum that has about 5-10 procs that govern the inner workings of the stat)

just seems odd that making the icons null then saving them then removing the dir gets a smaller size then just straight up removing the dir, in the end that directory is non existent and should not have any size difference
Mar 4 2013, 4:53 am
http://www.byond.com/forum/?post=1187381

There you go. It's done. A compact little saving system. If you copy/paste my saving modifications into your code, I'm quite sure

Midgetbuster said:
just seems odd that making the icons null then saving them then removing the dir gets a smaller size then just straight up removing the dir, in the end that directory is non existent and should not have any size difference

Actually, it isn't, considering how savefiles work.

Savefiles store every object in order that the object is saved.

Let's say you save a mob whose icon has been changed, but otherwise, everything else is the same.

The file would look something like this:

.0
  type = /mob/player
  icon = icon(.1)
  .1
    //the icon itself. It can be pretty big too!


Like I said earlier. Just removing icon variable doesn't remove the saved object. you will still have your .1 directory hanging around. You can't see the directories because they start with ".", which makes them hidden from simple snooping.

Check out part two of my tutorial. You shouldn't have to do much to my code to get it working with your project. In fact, I think you'll like how easy it is to use.
Mar 4 2013, 5:48 am
Savefiles at present do not clear old data should it shrink in any form. Instead, it is updated with new data with old data that lingers around (but no longer in use).

The only way at present to shrink savefiles is to delete the old copy and make a new copy. That way, there won't be old data lingering around.
Mar 4 2013, 8:35 am
The only way at present to shrink savefiles is to delete the old copy and make a new copy. That way, there won't be old data lingering around.

That's not true. You can set savefile.cd to the directory you want to change, then set savefile.eof to -1.

You can also set savefile.cd to the directory you want to change, then delete it from savefile.dir using Remove().

So really, that's three ways total.
Mar 4 2013, 8:43 am
That's not true. I've had experiences where the savefile corrupted itself and those methods were inept of getting rid of the scrambled data. Most notably the best way and most ensured way is what Bandock described.

Then it's a BYOND bug.

I've got that working just fine in my tutorial. Maybe you should test the cases, and report them as bugs.
Mar 4 2013, 9:35 am
Performed a test with using Remove or -= and that definitely fixes it. Good find Ter13.

Eof doesn't do anything though for clearing out old entries.
Page: 1 2