Apt-get install htop.
Htop is a bit more informative and, at least in my opinion, a lot more friendly than top itself, while providing the same function and features.
When you have that, just hit F9 on any process running, switch to Sigusr2, and hit enter. It should output it in the game's logfile.
I thought the -logself tag stopped all output to stdout and threw it in a log file, Sigusr2 would then only show the usual
World started ... Hub says port is open ... Right? | |
Using -logself will put everything into a .log file in the game's main directory, yes. Sigusr2 would output as normal, and write itself into the logfile, too.
| |
Just throwing out ideas, a friend of mine was having troubles with CPU problems and then he disabled threads, and the problems went away.
| |
Threads are disabled with Eternia, so we can probably rule that out as far as our instance goes.
| |
Laser50 wrote:
server mem usage: I think it would help to see this output at intervals, along with the RAM usage so we can see if there is a correlation (this goes for WANO and anyone else having the problem as well as Laser50). My suspicion is that we have a leak in something not output here-- such as images or dynamic icons, where the resources could build up rather quickly if not freed. But seeing this output would be a start before we can do more in-depth profiling next week. If there is a leak (and it sounds like there is) and/or we can recycle unused resources over time, this could be a big improvement. So it's a priority. | |
I'm not sure I follow.. If these outputs remain the same across all these checks, or at least, nearly the same, how would that help you in your tests?
Any way, I'll see what I can do. But I can't do a *lot* because there's players I don't want to bother too much with my testing. Expect my results in a few days. | |
I just want to see if those numbers are increasing over time (or if one particular one is) and how that correlates with the RAM usage. That will let us know if there is a leak in any of those structures or if it is internal.
| |
Regarding our memory issues, My server crashed this night.
The weird part is that it states out of memory, while I have 16 GB's of it. So something went terribly wrong.. Any way, this may give you the info you need to solve the whole problem. Problem is that the core file is a whooping 4 GB big. I'll compress it and send it over, any way, I don't have any idea why it is so huge, though.. https://drive.google.com/file/d/ 0B1UXNp_ofzs_VEt3UG0yUk5LTjA/edit?usp=sharing Enjoy, hope it helps. EDIT: I will be making memory logs every 3 hours, so you can still get that bit of info you need. | |
A core file is the dump of your program's working memory.
Since BYOND can only address around 4GB, it will run out of memory and crash if it reaches that level. | |
I've taken logs every 3 hours today, starting since that post. Looking at it... Stuff looks normal.
Here it is: http://pastebin.com/c4QMGMgJ | |
Thanks. It does look like it's something internal. We'll get some debugging info in and make this a priority this week. I wonder how long this has been around for.. could make a big difference.
| |
Awesome, guys, I'll await with patching my own code until this is in, so I can combine it with threading :D
| |
If you want any extra feedback, I started doing backtraces off of a freshly started server.
http://xirre.dyndns.org/share/byond/debug/Backtraces.txt I marked the time with @@Date Time -- I'll be updating it more. I'm only including the backtraces. If you need the full logs, I don't mind handing them to you privately. Updated 02/25/14 at 9:00 PM EST | Be sure to click "refresh" after you return to the page again. Just to make sure it updates to the most recent file. | |
I just wanted to add a note that I'm still looking into this. There is a definite leak going on but I haven't figured out what. It's very pronounced in SS13, less so in NEStalgia and other games. Significant amounts of memory aren't being freed when the server closes in Windows DD.
One of the factors complicating this was that the beta had a bug regarding memory growth in threaded mode in Linux. I posted a bug report for that on Laser50's behalf to note the fix in 506.1236. | |
Laser50, I have a followup question. Your logging of the memory usage over time was very helpful. I found another leak in the threading code, but it turned out to be exclusively in map-threads. I just wanted to confirm you turned off threading--also that it wasn't still in on the command line or something like that. If you only turned off map-threads but left regular threads on, it would explain the results you had. If not, then it's something else.
I did discover that SS13 seems relatively memory-stable after the round has started if nothing is happening. If there is a leak outside of threaded mode, it's happening at startup. I'm still looking into it, but it'd be nice to confirm the conditions of your log. | |
Threading is off, I can guarantee you this. Stable build has them off by default, any way, which I haven't touched.
I'll try to get another bit of log, more of a "Start to end" one, which I hope might provide any more info.. If the regular logging would be of more use, however, I could continue that, too. | |
Please test this with the latest beta to let us know if it is fixed. Also, we'd like to know if there are still issues in threaded mode (and if those are with threads in general, or just map-threads). The goal is, of course, to bring that into the stable build and assess performance.
| |
I've already promised my community that I'd enable multi-threading. So lets see how this goes!
| |
Shutting down and then starting it back up isn't really a good thing because not all games have it so that way if kill is passed it auto saves. Thus, savefiles are lost. But, I'll try what you said again. I really can't get SIGUSR2 to work on any of the games.
Edit: