New Map Saving Format

Page: 1 2
ID:1896428
 
Jul 14 2015, 10:39 pm
Applies to:Dream Maker
Status: Open
Issue hasn't been assigned a status value.
The current map saving format is hell for anyone using git.
A temporary fix is here: https://github.com/tgstation/-tg-station/pull/10623
But I would like to see native support for this.

Also instead of a map saved like:

aaaaaaaaaaaabaaaa
abaaacadaaeafaaaa
etc.

It could just be coordinates with everything saved on that tile.

So like:

(x,y,z) = ()

(1,1,1) = (
/obj/example{
height = 22;
width = 18
},
/turf/example)

(1,2,1) = (
/turf/example
/area/example)

(2,1,1) = (
/mob/example{
hostile = 1;
})

(2,2,1) = (
/area/example)

etc.
Jul 14 2015, 10:46 pm
Using your system would be massively inefficient and cause horrendous loading times.

.dmm is a format for use by a computer, not a format designed to be easily readable.
Jul 14 2015, 10:54 pm
Actually Dream maker is able to open the format and compile with it in pretty much the same time frame.

The tool I wrote can also translate normal .dmms into this format in less than a second or two (Tested with our most popular SS13 map which contains a lot of object instances and map-level variable edits)

The goal was not human-readability, it just ended up being a nice easy addition to the system, the goal was to have each object and each variable be on a separate line, so that git could actually make a decent attempt at merging two or more edits to the file since for SS13 Map conflicts are by far one of the worst things we have to deal with, to the point where any two mapping prs are almost certainly going to conflict.
Jul 14 2015, 10:55 pm
.dmm is a format for use by a computer, not a format designed to be easily readable.

DMM format is designed to be a human readable format. I agree though that the above induces an unnecessary amount of extra characters that will bloat the file size.

I'm not aware of really any tilemap format out there that's going to play nice with versioning software. Most of the time, map formats are binary. DMM is one of those weird cases where it's really just its own animal.

The really troublesome part, I think is in engineering a more appropriate map format. I've been trying to think of a better format for the map editor and frankly I just can't really think of a good one that would fit all cases. Honestly, DM's concept of tiles each indivudally being object instances is completely bizarre as far as 2D game development goes.

The fact that DM even compiles this format is mind-blowing, though.

EDIT: I really think the best-case solution is making a multi-user DMM editor and pushing changes to the DMM through that rather than dealing with SVN/Git rules, which are mostly designed around human readable text.
Jul 14 2015, 10:56 pm
In response to CrimsonVision
Actually Dream maker is able to open the format and compile with it in pretty much the same time frame.

What evidence do you have to show for this? Your program being able to write the new format has nothing to do with compiling.
Jul 14 2015, 11:00 pm
What evidence do you have to show for this? Your program being able to write the new format has nothing to do with compiling.

Oh boy... And the tone of the thread just went nuclear. Strap in, grab yer popcorn and let's watch this unfold... GG Zec, you have no idea what you just did.
Jul 14 2015, 11:00 pm
In response to Ter13
Ter13 wrote:
.dmm is a format for use by a computer, not a format designed to be easily readable.

DMM format is designed to be a human readable format. I agree though that the above induces an unnecessary amount of extra characters that will bloat the file size.

I'm not aware of really any tilemap format out there that's going to play nice with versioning software. Most of the time, map formats are binary. DMM is one of those weird cases where it's really just its own animal.

The really troublesome part, I think is in engineering a more appropriate map format. I've been trying to think of a better format for the map editor and frankly I just can't really think of a good one that would fit all cases. Honestly, DM's concept of tiles each indivudally being object instances is completely bizarre as far as 2D game development goes.

The fact that DM even compiles this format is mind-blowing, though.

EDIT: I really think the best-case solution is making a multi-user DMM editor and pushing changes to the DMM through that rather than dealing with SVN/Git rules, which are mostly designed around human readable text.

I'm Amazed myself to be quite honest.
My guess is byond isn't going line by line but instead taking note of open and closed quotes and brackets to determine what is and isn't an object or a variable.

Jul 14 2015, 11:01 pm
In response to Zecronious
Zecronious wrote:
Actually Dream maker is able to open the format and compile with it in pretty much the same time frame.

What evidence do you have to show for this? Your program being able to write the new format has nothing to do with compiling.

My evidence is that if I put the converted map in Dream Maker, I can both make additions, removals, and then save the map.

even if I don't edit the map, thus keeping it in our "new" format, Byond will HAPPILY build the object tree for it and report no errors, because a .dmm is just a .dm file with special syntax.
Jul 14 2015, 11:10 pm
My guess is byond isn't going line by line but instead taking note of open and closed quotes and brackets to determine what is and isn't an object or a variable.

Well, each one of those entries is a pop. What I'm interested in, is whether you guys are generating a unique pop for each space in the map due to your format, or whether DM's compiler is able to recognize duplicate pops and merge them with their redundant entries, or whether it's burning extra entries because of the format you guys are using.
Jul 14 2015, 11:10 pm
YES! This does not require byond change how it parses maps, only how it saves them.

The format shown below both compiles, loads, and is openable in the map editer:

"aab" = (
/obj/docking_port/stationary{
        dheight = 9;
         dir = 2;
         dwidth = 5;
         height = 22;
         id = "syndicate_n";
         name = "north of station";
         turf_type = /turf/space;
         width = 18
        },
/turf/space,
/area/space)


The following format for the latter part below SHOULD also be compatible (but i haven't fully tested)

(1,1,1) = "aaa"
(1,2,1) = "aaa"
(1,3,1) = "aab"


We aren't asking byond to change how it parses maps, it can currently support this without changing that, we only just want it to have more newlines in the map files (and maybe be less aggressive about re-using no longer needed cell keys) so that it is more git merge friendly for when mutiple people edit the same map file
Jul 14 2015, 11:19 pm (Edited on Jul 14 2015, 11:25 pm)
@StonedOne:

I think this would be a great opportunity to actually bring in some of you guys' best and brightest into this thread and start brainstorming on a better map format in general. The one we have is entirely top to bottom problematic.

For starters, the 65K pop limit is a low ceiling these days and storing objects by type path is probably inadvisable. I know the compiler internally assigns numeric values (one-up serialization) to types in some cases...

I dunno, I don't have a lot of great ideas on the subject despite trying to think up a better, cleaner, more powerful format that doesn't cripple existing games for a year or so now.
Jul 14 2015, 11:25 pm
In response to Ter13
Ter13 wrote:
Well, each one of those entries is a pop. What I'm interested in, is whether you guys are generating a unique pop for each space in the map due to your format, or whether DM's compiler is able to recognize duplicate pops and merge them with their redundant entries, or whether it's burning extra entries because of the format you guys are using.

^This
Jul 14 2015, 11:27 pm
A good format might be to define areas, map out the bounding rects that make the area (or just have a special null tile for tiles in a bounding rect that aren't part of that area.) and do each tile by the coords

At that point, making it human readable/editable is only a matter of tabbing.

but the more I think about it, the more I realize that honestly, a map file is just formated short cut for:

new /type/path{vars}(locate(x,y,z))

What harm would it be to just remove that formated shortcut and just keep it like it is?

Jul 15 2015, 2:35 am
Pretty sure Tom said once upon a time that he wanted to open source the map format and just make an api to hook ibto so we could write our own nap formats?
Jul 15 2015, 7:48 am
The map format is kind of already open-source, it's plain-text.
Jul 15 2015, 4:20 pm
The only bit that is missing would be a plugin to the byond compilation step that lets us take custom map format -> reformat to byond supported format -> compile

Jul 15 2015, 10:49 pm
I know some map formats serialize tile data and entities separately, going about the process of building the maps and populating them afterwards. I'm not sure if this would be any more efficient, though, having entity pop data stored separately from tile data, but it's worth taking a look at, I would guess.
Jul 24 2015, 9:46 am
I dont know why I haven't thought of this before, because I use this quite often. But you could look at either something similar, or full integration with Tiled. It is a great 2d map editor and it saves like XML files.

Tiled: http://www.mapeditor.org/
Demo of how it saves: https://github.com/HaxeFlixel/flixel-demos/blob/master/ Editors/TiledEditor/assets/tiled/level.tmx
Jul 24 2015, 10:01 am
Someone else mentioned Tiled recently. Its format is not compatible so at a minimum, BYOND would have to be altered to read and write that specific XML format.

I have to agree however that any map format is unlikely to be very friendly to a version management system. Too many lines are likely to change between files. It's probably way easier to simply build a sort into the .dmm writer so the symbols at least fall in a more consistent order, but even then any inserted symbol would mess with subsequent symbols and cause major line changes.

The only map change I can think of that'd be truly "nice" in terms of version-control software is one where matching symbols could be reserved and kept between iterations of the map. That'd be hard to setup on a lot of levels, but it'd probably be the best option in terms of minimizing diffs.
Jul 24 2015, 10:09 am
We already have something that does exactly what you just said.
It is out map merge tool, and it works great, the only problem is the area at the bottom of the map where it goes AAABAC, etc.
MapMerge tool: https://github.com/tgstation/-tg-station/tree/master/tools/ mapmerge
Page: 1 2