Generating 1 million objects

ID:138844
 
Oct 21 2011, 8:56 am (Edited on Oct 21 2011, 9:34 am)
Code:
    genplanundersterrain()
        for(var/celestial/planet/P in world)
            genplanunderterrain(P)
    genplanunderterrain(var/celestial/planet/O)
        if(O.name!="volcanic" && O.name!="temperate") return
        var/max=O.pMaxZ-1
        var/min=O.pMinZ
        for(var/iz=max,iz>=min,iz--)
            for(var/ix=1,ix<=world.maxx,ix++)
                for(var/iy=1,iy<=world.maxy,iy++)
                    var/a=text2path("/turf/land/dirt")
                    new a(locate(ix,iy,iz))
            sleep(world.tick_lag)
    genplanundersores()
        for(var/celestial/planet/P in world)
            genplanunderores(P)
    genplanunderores(var/celestial/planet/O)
        if(O.name!="temperate") return
        var/max=O.pMaxZ-1
        var/min=O.pMinZ
        for(var/iz=min,iz<=max,iz++)
            for(var/ix=1,ix<=world.maxx,ix++)
                for(var/iy=1,iy<=world.maxy,iy++)
                    var/b=text2path("/obj/harvestable/Dirt")
                    new b(locate(ix,iy,iz))
            sleep(1)


Problem description:

Okay so let me explain what's going on here. In my project, every planet has 5 z levels. The first z level is for the surface, the other 4 z levels is for below the surface, which is particularly used for digging, mining and/or underground construction.

Currently only temperate/volcanic planets spawn a surface and underground terrain. Only temperate planets spawn objects for dirt (and eventually ores). This is to make it easier for me to see the effects immediately.

So I have a 500x500 map, and that means that one million 'things' need to be spawned for each underground z level set per planet. Currently, I run two passes. One pass for turfs, the second pass for objects. My problem is that objects take 3-4x the amount of time to generate (probably more) VS turfs. Due to this, a generation proc wont cause a loop check problem if it's with turfs, but as soon as I use objects I start seeing them. I don't have a full understanding of why objects are so much more intensive, but I've been assuming there's much more information attached to them.

Should I rig turfs to use them for what I want and gain a generation speed increase (they don't need to move o.O)? Or is there something else I can be doing to make this overall better?

I cannot use pre-made maps. The entire universe (space, suns, planets, planet surface, planet underground) is generated at run-time.

P.S. I'm using a loading screen so I don't expect this process to go all that fast. It's updated periodically throughout the process to show where we are in the generation stages. Currently universe generation takes up to 2 minutes to complete (per planet that has underground objects spawned).

P.S.P.S. I'm using text2path because those entries will be dynamic to each planet later.
Oct 21 2011, 9:11 am
"Holy crap" is all I'm going to say.
Oct 21 2011, 10:38 am
Some ideas:

1. Make the maps smaller.
2. Only generate sections of the map as they're needed.
3. Find a way to compress it to one z level per planet. For example, store information in vars of the turf instead of using objs.
Oct 21 2011, 10:42 am
You don't need to use text2path to have a dynamic path.

var/b = pick(/turf/land/dirt, /turf/land/grass)
new b(locate(ix, iy, iz))
Oct 21 2011, 12:25 pm
In response to Immibis
It has more to do with a temperate planet having a different terrain type than a volcanic planet. Currently I retrieve what type it is via a var attached to the planet. So where you see 'dirt' will become '[O.land]' as an example.
Oct 21 2011, 12:27 pm
In response to Forum_account
Forum_account wrote:
Some ideas:

1. Make the maps smaller.
2. Only generate sections of the map as they're needed.
3. Find a way to compress it to one z level per planet. For example, store information in vars of the turf instead of using objs.

1. That'd cripple the space level which is already cramped as it is.

2. I was thinking of doing something along the lines of chunks, like how Minecraft handles it.

3. This seems to be the option I'm going to have to do. While I require multiple layers, only spawning turfs would lessen the load.
Oct 21 2011, 1:33 pm
In response to Stevenw9
Turning the objects into turfs has lessen the load big time. Also, all the inf loop errors are gone. Now the only potential problem is if I go generating more than just a few planets, lol. I only expect to do things so efficiently though, it's not like BYOND is using more than one of my CPU cores x.X
Oct 22 2011, 12:04 am
It seems like objs and turfs have very similar number of vars and procs, then why objects take longer to generate?
Turfs-vars and procs
Objs-vars and procs
Oct 22 2011, 12:07 am
In response to Martys1103
I'd surmise because objs are also atom/movables and can therefore be mobile.
Oct 22 2011, 6:51 am
In response to LordAndrew
That's what I was assuming. However, I wonder if it would be faster to generate these objects as turfs, or would it be faster to load randomly from preset maps (I.E. 50 different space maps and choosing one to use rather than generating the map the hard way. This would result in about 17 map loads per universe generation VS millions of turf generation calls, looping through each x,y,z position.).
Oct 22 2011, 10:31 am
In response to Stevenw9
Turfs also get folded in the map format, so you can basically re-use the object so long as it's not deviating from the object default values. So a run of grass for example would only require one instance in memory for a 50x50 grid of just 'grass'.
Oct 22 2011, 1:47 pm
In response to Stephen001
Stephen001 wrote:
Turfs also get folded in the map format, so you can basically re-use the object so long as it's not deviating from the object default values. So a run of grass for example would only require one instance in memory for a 50x50 grid of just 'grass'.

No, that's only in the map file. There's only one copy of this blank grass turf in the map file, but the game will make 50x50 of them.
Oct 22 2011, 1:59 pm
In response to Immibis
I'm assuming the VM has a slightly better spatial representation in memory than that. That one is pretty naive.
Oct 22 2011, 2:08 pm
In response to Stephen001
If all of them were the same turf, then they'd all have the same properties.
So they'd all have the same x, y, and z values. And you can't have multiple turfs on one tile (because the turfs *are* the tiles).
Oct 22 2011, 2:30 pm
In response to Immibis
Sure, but those are the only differing properties. So clearly the VM could implement a strategy to avoid that otherwise unfortunate in memory duplication. This is what's so handy about the fly-weight pattern.
Oct 22 2011, 2:39 pm
In response to Stephen001
Stephen001 wrote:
Turfs also get folded in the map format, so you can basically re-use the object so long as it's not deviating from the object default values. So a run of grass for example would only require one instance in memory for a 50x50 grid of just 'grass'.

I wouldn't say that there's just one instance in memory. Each instance is there. Only storing the variables whose values differ from default is also applied to objs, so I don't think this is where the savings come from.

It seems like turfs aren't managed the same way as objs. When you create a new turf, the old one isn't getting deleted, it's just getting modified. Instead of allocating a new object and deleting the old one, you're just updating the existing turf's vars to make it act as the new turf you're creating.

If you have a reference to a turf it's not broken when you create a new turf over top of the referenced turf:

var/turf/t = locate(1,1,1)
new /turf/new_type(locate(1,1,1))
world << t


t will be the new turf.