Discussion over long compile times for 64x64 icons

ID:2127016
 
Aug 5 2016, 2:42 am (Edited on Aug 5 2016, 2:54 am)
Resolved
BYOND Version:511.1350/510
Operating System:Windows 7 Ultimate 64-bit
Web Browser:Chrome 52.0.2743.82
Applies to:Dream Maker
Status: Resolved
This issue has been resolved.
So Clusterfack of /vg/station had a convo with lummox jr over an issue, in pager, and because it happened there, then was paste bined and sent around, nobody else can comment or offer other info or ask follow up questions, and thats frustrating, so this thread is for that to hopefully find a resolution. at /tg/station we are also trying to move to 64x64, so getting this resolved in some way would be nice, as after we can pull it off, other ss13 codebases are likely to follow.


Clusterfack:
Hi lummox quick question about an issue we've been having with compiling.

We recently changed our codebase to have world.icon_size to 64x64 and upscaled all our sprites to that size as well. Since we used nearest neighbor interpolation and optiping to optimize the dmis, it only increased file size by about 10-20%.

When we compile now though, compiles will either take 3 minutes like they used to but also occasionally they will take a full 20 minutes. During which time the mem usage of dream maker cycles up and down by about 100mb repeatedly.

Is there some issue with compiling at that icon size, or with bigger rscs are we hitting some unknown limit, or is there some unknown issue with using optiping on .dmis which strips out information that massively slows down compile speed (randomly and not consistently). Do you know if there is any way to avoid this problem?

If you want to try compiling yourself to see it, this 64x64 version is now the bleeding edge version of our code here
https://github.com/d3athrow/vgstation13

Lummox JR:
The compilation process itself shouldn't, AFAIK, be unpacking icons at all, so I doubt that the icon size has any bearing on compilation proper. But the object tree for Dream Maker might be another story. That's probably where the slowdown is happening; it does unpack icons, and internally it wants them to be 32x32 because it uses an imagelist structure.

Compiling in dm.exe will of course skip the expensive parts of the object tree generation.

Clusterfack:
Alright, I will spread that around to our coders and we'll see if that eliminates our issue. Thank you!


Clusterfack:
Is there any option to set so the project wont generate the object tree in DM? I only ask because a lot of newer people start with dream maker originally simply because they already have it installed.


Lummox JR:
Not really; DM generates the object tree automatically so they can do mapping. But I think there's something to be said for maybe making deferral of the tree a feature request.


So I bring this up because things lummox mentioned are being contradicted by other observations, the key of which is that this bug doesn't happen while the dm status bar says "generating object tree"

It seems to happen right after it reads the first map file, but so far i haven't gotten it to happen in dm.exe


(note: among ss13 devs, dm = dream maker, dm.exe = dm.exe)
Aug 5 2016, 3:00 am
So when compiling, the output looks something like this:
loading tgstation.dme
loading interface\skin.dmf
loading map_files\TgStation\tgstation.2.1.3.dmm
loading map_files\generic\z2.dmm
loading map_files\generic\z3.dmm
loading map_files\generic\z4.dmm
loading map_files\generic\lavaland.dmm
loading map_files\generic\z6.dmm
loading map_files\generic\z7.dmm
loading map_files\generic\z8.dmm
loading map_files\generic\z9.dmm
loading map_files\generic\z10.dmm
loading map_files\generic\z11.dmm
saving tgstation.dmb (DEBUG mode)
tgstation.dmb - 0 errors, 0 warnings (8/5/16 1:57 am)


Between loading the first map file, and the second map file, the time is 5 to 15 second. under 64x64, that jumped to over 10 minutes. (I force closed it after about 10 minutes to do some more tests in dm.exe)

Mind you, the codebase is on a samsung evo ssd, my computer is a 8 core overclocked machine with 16gb of overclocked ddr3 ram, no page file (so no slow ram reads from paged ram). Its really beefy
Aug 5 2016, 9:29 am
I've noticed in the past that sometimes I'd see a longer compile time right after loading the first map file as well; this may be entirely independent of the 64x64 size change, or possibly that change may be exacerbating the problem.
Aug 5 2016, 10:19 am
I wonder if the slowdown is with the map dump to a single file, or compiling into the dmb?

Try dm.exe -map_dump and see how long that takes by itself.
Aug 5 2016, 4:42 pm (Edited on Aug 5 2016, 5:10 pm)
So, running a profile on the compile does show the highest self time being in CVImageList::PrepareToAdd(), and the highest total time being in readIcon()

Memcopy and Deflate were also high up there.

but a more longer profile showed DungBuilder::SetHaltDB() and InternalDebugPrinter has the top figure heads with CVImageList::PrepareToAdd and readIcon() further down.

Im not actually sure what to believe, profiling native code like this isn't an exact science, CodeXL basically just takes a stack trace every millisecond and maths it all out at the end of the profile. Another issue is sometimes non-exported functions are seen as part of the exported function above it when doing this without debugging symbols. in either case hopefully this is a hint.

I haven't gotten the long profile compile time to reproduce in dm.exe, and -map_dump doesn't work, it just displays the help.
Aug 5 2016, 5:17 pm
Ah, it's -dump_map, not -map_dump - it combines all maps into one dmm file.
Aug 5 2016, 5:32 pm
alright, so i noticed that the upscaling script we got from PJB at /vg/ is doing some wierd shit to the dmis.

It has an off by one error that causes it to add a blank 64pixel row at the bottom of the png with no attached icon states, (making it no longer 1:1 aspect ratio), and it stores the states in an odd way, it seems it's trying to make the icon's fit evenly in the png with no left over blank areas by changing the rows and columns.

lummox, could something like that trip up byond and cause some of the issues like this one and the other 64x bugs that were reported?
Aug 5 2016, 5:43 pm
Have yall tried disabling auto file dir when ur not needing it?
Aug 5 2016, 5:44 pm
It is disabled, in fact, any code change request on github that attempts to re-enable it will trigger a big automated red X as the request will fail automated testing.
Aug 6 2016, 4:07 am
This is the script in question:

https://gist.github.com/PJB3005/ 1f8911552b251672ad582f4f3a92af80#file-nearest_dmi-py

And this is the Python module I used:

https://gitlab.com/N3X15/ByondTools/blob/master/byond/DMI/ __init__.py

There's definitely some bugs with that. For one thing it caused all transparency to be pitch black when editing a sprite in DM.

Oh and if you have some of your own icons to try it on I made a web thing to upscale them: http://84.195.252.227/dmi
Aug 6 2016, 5:23 pm
There's literally no reason for object tree generation to have such a big impact.

Hell, my java code can generate an object tree pretty fast:
http://puu.sh/qsfTD/a89c8e1dd5.mp4
Aug 6 2016, 6:16 pm
awe, but you see, you do some things at runtime while viewing the map, where as byond does it all at once before showing the map.

on slower computers, browsing thru the map on your setup would be much slower/shuttery then byond's setup.

design choices my man.

Anywho, thats not all there is to it, I wonder if byond is doing something odd like re-decoding the png for each state or each atom that uses that icon/state, (maybe even for child atoms that don't override their parent's icon/state) I'd assume not, but no idea how old that code is, from what i hear, Dan was a memory saving nut, and I could see him doing something odd like that if he touched it.
Aug 7 2016, 12:26 pm
The important part is that it generates the object tree *very fast*.
Aug 27 2016, 6:42 pm
Even if the effort to go to 64x64 has been put on hold, I would still like to know what causes the background in the icon editor to become pitch black because it's a complete pain in the ass.

Using automated scripts can be a real time saver in some cases, but with this bug it becomes impossible to use automated Python scripts.
Oct 27 2016, 7:03 pm (Edited on Oct 27 2016, 7:14 pm)
I can confirm that this was fixed by 511.1363, that or I can no longer reproduce it under this revision. I can reliably compile a map from a fresh start of dream maker in just a few minutes at most again, even on 64x64.
Oct 27 2016, 7:43 pm
Nadrew resolved issue