Actually, sorry for the double post here, and this is just a wild freaking guess.
But looking at your profiling records, I'm seeing things that are way above 1000 to 9000 in the realtime tab.
I'm guessing these are loops but, just for experimentation's sake, could you change them to loops that don't do that much sleeping? I know for a fact my loops don't look like that, so it's possible.
Reasoning behind this request is that I think it may be bogging stuff down, even though it is sleeping. Never know until you've tried different.
But looking at your profiling records, I'm seeing things that are way above 1000 to 9000 in the realtime tab. That's not a problem. Realtime is a non-issue because of the way the scheduler works. I use a blocking loop structure for much of my code, and depite realtime being high, my world CPU always stays low. You can look at that profiler and see that softcode doesn't seem to be the issue, as that profile result is indicating that spread over 20 minutes, only about 0.04% of the CPU is being used by softcode itself. Something internal is causing the CPU usage to spike dramatically, since it's not showing in the profiler. | |
Ter13 wrote:
But looking at your profiling records, I'm seeing things that are way above 1000 to 9000 in the realtime tab. Well then, I guess we can tick that off the list. Hm... Beside trying another OS, you could always try to log the amount of deletions and creations the game makes, see if that's anything. I have it in my code at the moment, but that is mainly to figure out why atom/movable/Del() is so high up in our list. var; var/global/debug_file_cret = file("creations.txt") Could give that a shot, too. | |
The reason that del is so high is because forcibly deleting objects has to look through every variable of every object in the world and null out references to ref. The more datums and variables in your world, the longer that search will take.
There are almost no cases where using the del keyword is correct. | |
Ter13 wrote:
The reason that del is so high is because forcibly deleting objects has to look through every variable of every object in the world and null out references to ref. If possible, I replace Del() with loc = null. Having it garbage collect itself is a lot better than throwing it in front of all your ongoing processes. And of course because SS13's code is a shitty mess, which I'm starting to repair slow enough. But the reason I'm asking you to keep track of this is because at some points in time I enable the in-game build mode, and start constructing some walls. If I do this too fast, or too much, the game will slow down, lag like shit, and it never stops doing so, even though the objects have already been spawned in and initialized some time ago. I have yet to figure out why it does that. But could this be happening to you? That may be the way to find out.. Or not, I'm really just throwing out whatever seems sensible to try and see if you can either get more info from it, or a fix. | |
This is why you should never trust the profiler. Stupid thing is useless.
| |
MisterPerson wrote:
This is why you should never trust the profiler. Stupid thing is useless. ...What? No. The profiler is great for what it s capable of. | |
So, how do we figure out what's causing the internal CPU spike? Perhaps remove suspects from the game's source and throw up a test server to see if there's any difference after code removal?
Would really like to see this resolved soon if the schedule permits it, since there's crippling CPU usage during prime time unless we reboot every 1-2 hours. | |
Writing A New One wrote:
So, how do we figure out what's causing the internal CPU spike? Perhaps remove suspects from the game's source and throw up a test server to see if there's any difference after code removal? Well, I'd love to try and help you out with that, but all I'm capable of is suggesting things. But for starters, I'd start logging your creations and deletions and see how many that is. Other than that.. I'm thinking.. You could try forcing everyone to use the latest BYOND version, both server and client on 506, see how that goes. | |
Yes, the profiler's fine for what it reports, but there's a LOT it doesn't report. Mostly things BYOND handles behind the scenes like GC checks and redetecting verb visibility. I don't have an exhaustive list of things not handled by the profiler obviously so anything to help narrow this down would be great.
A large number of objects will cause problems. Are we absolutely sure there's not objects that are supposed to be deleted but aren't, and instead they just sit around in nullspace? I would suggest implementing a system to make sure GC'd objects are in fact actually deleted, assuming you don't already have such a system. And if you're just leaving objects lying around constantly, that could cause problems. You can always test for things like logging as an issue by disabling it for a day to see if the problem goes away. | |
Are we absolutely sure there's not objects that are supposed to be deleted but aren't and are instead just sitting around in nullspace? A large number of objects with verbs attached causes stuttering. 300,000 objects with 3 verbs attached caused a lot of nonprofiled stuttering for me. I would suggest implementing a system to make sure GC'd objects are in fact actually deleted, assuming you don't already have such a system. It's a good idea anyways just for the stat gathering (how much grass is getting deleted, anyways? etc). And if you're just leaving objects lying around and they have object verbs attached, there's your problem. Dream Daemon's profiler allows you to check this. There's a "Memory Stats" option under the World menu: server mem usage: The number on the left is how many bytes of memory the objects potentially use, I think. The number on the right is how many objects there are. | |
Can that be accessed if I'm using a VPS to host?
| |
I'm not sure. Paging Dr. Lummox.
| |
Can only do so if you're hosting it on windows.
No.. Actually.. If you call SIGUSR2 on your process on linux, it should output the same info in your .log file. And if you need some comparison, this is my empty SS13 server's memory stats: Prototypes: Soo, that's probably not your bottleneck here.. LOL. | |
The memory stats I showed had nothing to do with Eternia. That was a personal project I hosted just for example. I'm pretty sure Eternia's list, string, and object counts will be equally as high as SS13's. Anyway, you'll obviously need more than one of these stat readouts to know what's going on.
As the CPU use increases, it'll probably be best to look at what pools are increasing alongside it to find the source of the issue. | |
Oh, hm.. Well, then we'd still need to see one from Eterna, if possible.
Beside that, this was probably already answered.. But is RAM increasing with the CPU usage? | |
We actually logged SIGUSR2 calls for approximately 18 hours of the game running and even though the numbers were rather high (it's a big game with a big population) Tom and Lummox didn't seem to find anything abnormal to account for the CPU usage. If it were a problem of creation/deletion stacking up we'd be seeing memory usage getting out of hand at the same time, but memory usage actually remains fairly consistent the entire time we're hosting, even during these CPU issues.
| |
Okay, that's good to know. Hopefully it's due to the log system overloading the string table.
| |
I honestly have not a damn clue how I've managed to get 700mb worth of disk writes.. I haven't changed the game's files. This is just how SS13 rolls.
I would love to figure out what it's writing to, though.
But beside that point, you guys are having an issue, and it probably needs to be looked at..
I could propose a few things to you, but since it surely isn't harddrive stuff that is bogging you down, we can probably tick that off the list. (But a new hard drive always does miracles.)
So, as I'm not sure what OS you're running on, is there any way you could try hosting on Windows, and see if that brings other results than what you're currently getting?
My general idea would be to try a different OS and have a look at the results. I've once moved from CentOS to Ubuntu and found it to be a whole new world opening up for me in terms of performance.. But I probably broke everything at the time.
But yeah, you should try that.