ID:2863262
 
BYOND Version:514
Operating System:Linux
Web Browser:Firefox 111.0
Applies to:Dream Maker
Status: Open

Issue hasn't been assigned a status value.
Descriptive Problem Summary:

The main bad effect is that compilation is slowed down.

It appears that for every .dm file in a codebase, even during a command-line compile, DM readdir()s /, stat()s every file, readdir()s /home, stat()s every file(), readdir()s /home/$USER, stat()s every file, and so on. This is a ton of wasted work.

I noticed the issue because I had a network drive mounted at /z that was down, so DreamMaker appeared to completely hang, waiting through what should have been a short timeout period, multiplied by ~3,500.

Occurs on 514.1588 and on 515.1603.

Code Snippet (if applicable) to Reproduce Problem:
strace DreamMaker big_environment.dme 2>&1 | grep openat | grep home

Expected Results:
DreamMaker CLI is not this slow.

Actual Results:
DreamMaker CLI is slow.

Does the problem occur:
Every time? Or how often?
Every time.
In other games?
Yes, in a sandbox game, although it is less noticeable in a smaller codebase.
In other user accounts?
Not applicable.
On other computers?
Not tested.

When does the problem NOT occur?
Windows not tested.

Did the problem NOT occur in any earlier versions? If so, what was the last version that worked?
Unknown.

Workarounds:
None known.
Can confirm the issue is present on Windows as well: https://i.imgur.com/Zt52zsJ.png

EDIT: That said, the difference is so little it may as well be within margin of error: https://i.imgur.com/teUut7F.png
On Linux I'm able to turn a 60-second build into a 54-second one (that's ten percent savings!) by building out of /tg instead of /home/me/work/ss13/tgstation.

Total number of openat system calls goes from 2,491,450 to 1,222,720 which frankly is still enormous.

The difference may be less pronounced for smaller repositories or on smaller systems. Of course this still includes tons of stat()ing every file and directory in /.
Bumping because this happened to me again recently and it took me a while to remember that I'd seen it before.

Since it seems that the aggressive directory walking serves the genuine purpose of implementing case-insensitivity, I suggest some combination of:
  1. Not stat()ing all the files, since you don't need that to learn their names;
  2. Caching the readdir() + stat() results for each directory, instead of hitting disk 3,500 times each;
  3. Caching the correct case for each directory so you only have to walk it once. This could be seeded with the path to the .dme file given on the command line, since that currently isn't case-corrected, and from there you only really need to correct #include and single-quoted paths in the DM source.