Server crashes

ID:95158
 
May 6 2010, 9:56 am
BYOND Version:467
Operating System:Linux
Web Browser:Firefox 3.6.3
Applies to:Dream Daemon
Status: Unverified
Thus far we've been unable to verify or reproduce this bug. Or, it has been observed but it cannot be triggered with a reliable test case. You can help us out by editing your report or adding a comment with more information.
Duplicates:id:97859
Descriptive Problem Summary:
Server crashes from what looks like an invalid save file. The log file told me to post this here, so have at it.

Numbered Steps to Reproduce Problem:
N/a.

Log Snippet:
File offset: 73486
Real position: 73491/73543
BYOND(467.1069) Error: corrupt or invalid savefile '/var/www//.byond/cache/F8CD44'
Attempting auto-recovery of '/var/www//.byond/cache/F8CD44'.
BYOND(467.1069) BUG: crc mismatch in ReadCacheItem
Failed to backup savefile; aborting.
Aborted restoration.
BUG: Crashing due to an illegal operation!
proc name: safe load (/proc/safe_load)
  source file: Savefile.dm,238
  usr: null
  src: null
  call stack:
safe load( (/savefile))
101ZERO101 (/client): Load()
101ZERO101 (/client): New()

Backtrace for BYOND 467.1069 on Linux:
Generated at Thu May  6 17:39:43 2010

DreamDaemon [0x8048000, 0x0], [0x8048000, 0x804a8f8]
libbyond.so 0x1d2020, 0x1d2032
libc.so.6 [0xb7932000, 0x0], 0x2ae10
libbyond.so 0x1d2020, 0x1d2032
libbyond.so [0xb7bf5000, 0x0], 0x1d9c9b
libbyond.so 0x1db5ec, 0x1db65a
libbyond.so [0xb7bf5000, 0x0], 0x22fcbd
libbyond.so 0x231cd0, 0x231e13
libbyond.so [0xb7bf5000, 0x0], 0x23278a
libbyond.so [0xb7bf5000, 0x0], 0x23b411
libbyond.so 0x250214, 0x250300
libbyond.so [0xb7bf5000, 0x0], 0x23760c
libbyond.so [0xb7bf5000, 0x0], 0x234fe5
libbyond.so [0xb7bf5000, 0x0], 0x24ef4a
libbyond.so 0x2b8be8, 0x2b8c2f
libbyond.so 0x290f88, 0x2911e0
DreamDaemon [0x8048000, 0x0], [0x8048000, 0x8049e77]
libc.so.6 0x16348, 0x1642d (__libc_start_main)
DreamDaemon [0x8048000, 0x0], [0x8048000, 0x8049b81]

To help the BYOND developers debug this, please send the above trace as part
of a very detailed bug report: http://www.byond.com/members/?command=view_tracker&tracker=1


Does the problem occur:
Every time? Or how often? Whenever someone logs in with an invalid save, it seems.
In other games? Only host one, unfortunately.
In other user accounts? N/a.
On other computers? N/a; hosted from a VPS.

When does the problem NOT occur?
When no one logs on with an invalid save file.

Did the problem NOT occur in any earlier versions? If so, what was the last version that worked? (Visit http://www.byond.com/download/build to download old versions for testing.)
Has been occuring in all versions to date.

Workarounds:
None.
May 6 2010, 1:52 pm
This is proving rather difficult to trace (at least in any way that gives me insight about the problem), but I found out that ExportText() is in that stack trace and it seems that the crash is happening somewhere in there.

Some of the information I'm coming across hints that nested savefiles could be involved. It would help to see the source code to know for sure what's going on here, and in particular it would help to know which savefiles cause the bug. If you have source I can check out, you can send it to me in a .zip file at [email protected].

If I'm right about nested savefiles causing the corruption, then what's probably happening is that you have a var belonging to a mob or client or such that can hold a savefile reference but is not a /tmp var. If this is the case, adding the /tmp keyword to that var could fix the problem for future savefiles, though it won't help savefiles that are already corrupted.

This could also be a case of a savefile being open in more than one place. You might want to check your code for anything like that as well. If this is the case then there might not be a particular savefile setting it off so much as a certain sequence of events.
May 6 2010, 9:26 pm (Edited on May 6 2010, 9:59 pm)
This seems like a malicious attack by 101zero101. I found that on at least 4 separate occasions he had crashed my server using the same method.

Also Lummox, this was on Duel Monsters Genesis, so you would have to get into contact with EternalDuelistSoul if you wish to obtain the source to try and remedy this problem.

Also, each time it happens, it dumps a _bad_00(#).txt into my home folder.

Here is a link to the bad files incase you can gather some information from them: http://www.megaupload.com/?d=42YZ0XYL
May 7 2010, 5:29 am
I wouldn't rule out that his savefile is corrupt; if you remove the savefiles assigned to his key (you could always copy them over to another directory where they can't be used), does the problem go away?
May 7 2010, 1:39 pm
Lummox JR wrote:
I wouldn't rule out that his savefile is corrupt; if you remove the savefiles assigned to his key (you could always copy them over to another directory where they can't be used), does the problem go away?

The .sav files are client side, if they were server side, this wouldn't be much of a problem because they could be controlled.
May 15 2010, 7:09 am
I can provide you with the save.dm Lummox if you think this will be of use to solving.

The game uses the Safe_Save() lib, and other than that it's pretty bog standard save code. There is a hash to stop tampering (a long and multi part one) and a bit of skin integration. Nothing that would really result in such a critical issue as this... the most any of that would do is make the game reject the save file or make the skin bug out.

The Save_Save() does use ExportText(). So perhaps if you take a look at that first;

http://www.byond.com/developer/Garthor/SafeSave
May 15 2010, 7:25 pm
There is no way to look into this issue without the broken savefile. Since you do think you know who's causing this, what I suggest is that you add code to interdict this user's savefile before calling Garthor's lib, and save it to your server so you have a copy. Then just deny the login or start him from scratch. Once you have a copy of the bad file you can send it to me to take a look at.