Dream Seeker crash issue

ID:2537461
 
Jan 14 2020, 10:41 am
Resolved
Animations could cause crashes in some situations.
BYOND Version:513.1506
Operating System:Ubuntu 18.04 (Wine)
Web Browser:Chrome 79.0.3945.79
Applies to:Dream Seeker
Status: Resolved (513.1507)
This issue has been resolved.
Descriptive Problem Summary:
Dream Seeker crashes frequently in Wine:
"Unhandled exception: page fault on write access to 0x0000715a in 32-bit code (0x7bc4f02e)."

I have put two backtraces on pastebin here, since they're fairly large: https://pastebin.com/raw/MWxWvzNU

Unhandled exception: page fault on write access to 0x00015dc3 in 32-bit code (0x7bc4f02e).
Register dump:
 CS:0023 SS:002b DS:002b ES:002b FS:006b GS:0063
 EIP:7bc4f02e ESP:006bcf00 EBP:006bcf48 EFLAGS:00010206(  R- --  I   - -P- )
 EAX:00015dbf EBX:7bcdccb4 ECX:087f0000 EDX:001101c8
 ESI:00110000 EDI:709cac00
Stack dump:
0x006bcf00:  00000039 006bcfb4 3bc63ff4 00000000
0x006bcf10:  000000b0 00110000 00000002 000000ac
0x006bcf20:  00000014 7bc7d736 6e4c0000 cae22e00
0x006bcf30:  006bcf50 7bc4ef86 006bcf58 00000000
0x006bcf40:  000000ac 0000cc53 006bcf70 0024ed63
0x006bcf50:  00000000 0024ed63 00110000 00000000
Backtrace:
=>0 0x7bc4f02e (0x006bcf48)
  1 0x0024ed63 in msvcr120 (+0xed62) (0x006bcf70)
  2 0x0024ee1f in msvcr120 (+0xee1e) (0x006bcf8c)
  3 0x006ffaa1 in byondcore (+0x3faa0) (0x006bcfb0)
  4 0x006f4321 in byondcore (+0x34320) (0x006bd060)
  5 0x006ed822 in byondcore (+0x2d821) (0x006bd088)
  6 0x006ea7a9 in byondcore (+0x2a7a8) (0x006bd09c)
  7 0x008c3ee5 in byondcore (+0x203ee4) (0x006bd0c4)
  8 0x008c2925 in byondcore (+0x202924) (0x006bd0d4)
  9 0x008d52d1 in byondcore (+0x2152d0) (0x006bd0fc)
  10 0x004444a2 in dreamseeker (+0x444a1) (0x006bd12c)
  11 0x00e3ac41 in mfc120u (+0x23ac40) (0x006bd1fc)
  12 0x00e3a901 in mfc120u (+0x23a900) (0x006bd21c)
  13 0x00e38f33 in mfc120u (+0x238f32) (0x006bd28c)
  14 0x00e39155 in mfc120u (+0x239154) (0x006bd2ac)
  15 0x00d37e8e in mfc120u (+0x137e8d) (0x006bd2e8)
  .
  .
  .


Numbered Steps to Reproduce Problem:
1. Update to 513.1506
2. Run game for a while
3. Experience crash. (Sorry, I can't find something that reliably causes it beyond "play for a minute or two".)


Expected Results:
Dream Seeker does not crash.

Actual Results:
Dream Seeker crashes within a minute or two.

Does the problem occur:
Every time? Or how often? - I was unable to run DS for more than a few minutes at a time.
In other games? - Unchecked (I was only testing Goonstation)
In other user accounts? - Unknown
On other computers? - No crashes in native Windows 10



Did the problem NOT occur in any earlier versions? If so, what was the last version that worked? (Visit http://www.byond.com/download/build to download old versions for testing.)
The problem did not occur in 513.1505. It has only begun happening since upgrading to 513.1506, and it stopped happening upon downgrade to 513.1505.

Workarounds:
Downgrading to 513.1505 (or earlier).


I'm aware that this is an unsupported configuration.
Jan 14 2020, 1:28 pm (Edited on Jan 14 2020, 1:42 pm)
Both of these traces point to something happening when creating an appearance on the client. With the error happening in new() and apparently being related to some heap corruption, I can't really tell what's going on here. I'll review my code changes in 1506 however and see what I can find that could possibly have impacted the client.

[edit]
The only thing I can see here is a change to animations that was actually a bug fix to a specific issue, so if anything this should be crashing less. I do see there's an opportunity for a sanity check against a situation that should never come up, so I can try adding that, but otherwise I see no client changes that could possibly have impacted this.
Jan 14 2020, 5:00 pm
I'm able to reliably get it to crash, but I'm not sure what exactly is causing it. I'm not sure how to move forward with getting a potential distillation of the issue.

All I can do is confirm that it happens in 513.1506 but not in 513.1505.
Jan 14 2020, 6:08 pm
I believe this also happens on native Windows -- I've heard reports of players crashing on 1506. None of them are willing to go through the effort to provide a crash dump, but I have a feeling it's similar to this. Likely related to DX, drivers, or GPU: some people seem to have it constantly, some none at all. The filter-related crashes that 1506 was supposed to fix seems to cause this more consistently for affected clients.
Jan 14 2020, 6:27 pm
The disconnects Windows users are experiencing are a separate issue and it's server-side.
Jan 14 2020, 6:28 pm
In response to Lummox JR
Lummox JR wrote:
The disconnects Windows users are experiencing are a separate issue and it's server-side.

No, I mean, some people are having that, some people are having a straight up "not responding + exit" crash.
Jan 14 2020, 7:08 pm
With an actual crash it'll help if they can get crash details.
Jan 15 2020, 3:11 am
In response to Lummox JR
Lummox JR wrote:
With an actual crash it'll help if they can get crash details.

I've been trying...
Jan 15 2020, 12:47 pm
I have a crash dump now, from a user. If it's a different crash, I'll open a new bug report.

https://cdn.discordapp.com/attachments/467793069043875842/ 667092261456052224/dreamseeker.exe.10804.dmp
Jan 15 2020, 2:00 pm (Edited on Jan 15 2020, 2:15 pm)
In response to Steamp0rt
I'm not positive now that these are entirely separate bugs, but the cause eludes me.

The crash dump from Windows indicates a problem reading what should be a valid appearance in the appearances list, which should never ever happen. Somehow an appearance is getting destroyed prematurely. It's possible this could have the same cause as the issue in WINE. I'll keep investigating.

The really weird thing is, the only client-side change that touched anything related to appearances should, if anything, have made them more resilient.
Jan 16 2020, 4:31 am
I personally suspect the problem is related to DX/GPU driver configuration (or something that varies between computers), as I've been completely unable to reproduce crashes, even after spamming the effect that caused some to crash on a local server.
Jan 16 2020, 9:21 am
I have a weird hunch that it's possibe I did something wrong in the animate fix's logic and that's the problem, but wothout reproducing the issue myself I can only speculate as to the effectiveness of my prospective fix.
Jan 16 2020, 12:05 pm
One possible solution might be to provide test builds that dump more output about what is happening, similar to -trace. Again, I can consistently get it to crash, so I'm open to trying things if it will help.

Both of these traces point to something happening when creating an appearance on the client. With the error happening in new() and apparently being related to some heap corruption, I can't really tell what's going on here. I'll review my code changes in 1506 however and see what I can find that could possibly have impacted the client.


If there's any particular DM code that would trigger the code you changed, I could test it to see if it's causing the problem.
Jan 16 2020, 2:37 pm
I believe there's a fix for this in 513.1507 but I haven't documented it. Please everyone retest in 1507 and let me know if you see any difference.
Jan 16 2020, 5:08 pm
12 minutes of runtime without a crash in 1507 so far.
Jan 19 2020, 3:41 pm
I've also received no more reports of crashes from players.
Jan 19 2020, 7:03 pm
Lummox JR resolved issue with message:
Animations could cause crashes in some situations.
Jan 24 2020, 12:58 pm (Edited on Jan 24 2020, 1:18 pm)
Problem's back with 513.1508:

11:01:12 ~/Programs/byond$ wine: Unhandled page fault on write access to 0x00016944 at address 0x7bc4f02e (thread 009e), starting debugger...

11:11:08 ~/Programs/byond$ wine: Unhandled page fault on write access to 0x00009c54 at address 0x7bc4f02e (thread 0141), starting debugger...

Looks like it's the same thing that happened before.

Unhandled exception: page fault on write access to 0x0000bd4c in 32-bit code (0x7bc4f02e).
Register dump:
 CS:0023 SS:002b DS:002b ES:002b FS:006b GS:0063
 EIP:7bc4f02e ESP:006bd0d0 EBP:006bd118 EFLAGS:00010206(  R- --  I   - -P- )
 EAX:0000bd48 EBX:7bcdccb4 ECX:014d0000 EDX:00159d28
 ESI:00110000 EDI:114644c0
Stack dump:
0x006bd0d0:  000000a4 008d5a9a 000000a4 00000000
0x006bd0e0:  00000100 00110000 00000002 00000100
0x006bd0f0:  006bd578 00000000 101f0000 b7f9d800
0x006bd100:  004444a2 7bc4ef86 006bd130 00165558
0x006bd110:  00000000 00000000 006bd448 7f9dd80f
0x006bd120:  00c7ce0f 00eb0698 7f98c519 7f9dd80f
Backtrace:
=>0 0x7bc4f02e (0x006bd118)
  1 0x7f9dd80f in user32 (+0x5d80e) (0x006bd448)
  2 0x7f9e458f in user32 (+0x6458e) (0x006bd4b8)
  3 0x00e3d83a in mfc120u (+0x23d839) (0x006bd4fc)
  4 0x00de0773 in mfc120u (+0x1e0772) (0x006bd514)
  5 0x00de09f6 in mfc120u (+0x1e09f5) (0x006bd568)
  6 0x0042476e in dreamseeker (+0x2476d) (0x006bfe60)
  7 0x00e4bdc4 in mfc120u (+0x24bdc3) (0x006bfe74)
  8 0x0045760d in dreamseeker (+0x5760c) (0x006bfec0)
  9 0x7b46277c in kernel32 (+0x4277b) (0x006bfed8)
  10 0x7b4641ae in kernel32 (+0x441ad) (0x006bffd8)
  11 0x7b46278a in kernel32 (+0x42789) (0x006bffec)
0x7bc4f02e: movl        %edx,0x4(%eax)





Wine-dbg>cont
Unhandled exception: page fault on write access to 0x000072e0 in 32-bit code (0x7bc4f02e).
Register dump:
 CS:0023 SS:002b DS:002b ES:002b FS:006b GS:0063
 EIP:7bc4f02e ESP:006bcf00 EBP:006bcf48 EFLAGS:00010202(  R- --  I   - - - )
 EAX:000072dc EBX:7bcdccb4 ECX:04400000 EDX:00159da8
 ESI:00110000 EDI:0bcdf098
Stack dump:
0x006bcf00:  00000036 006bcfb4 321fbc12 00000000
0x006bcf10:  000000b0 00110000 00000002 000000ac
0x006bcf20:  0000006a 7bc7d736 0acc0000 b9af1000
0x006bcf30:  006bcf50 7bc4ef86 006bcf58 00000000
0x006bcf40:  000000ac 00009d7e 006bcf70 0024ed63
0x006bcf50:  00000000 0024ed63 00110000 00000000
Backtrace:
=>0 0x7bc4f02e (0x006bcf48)
  1 0x0024ed63 in msvcr120 (+0xed62) (0x006bcf70)
  2 0x0024ee1f in msvcr120 (+0xee1e) (0x006bcf8c)
  3 0x006ffc01 in byondcore (+0x3fc00) (0x006bcfb0)
  4 0x006f4321 in byondcore (+0x34320) (0x006bd060)
  5 0x006ed812 in byondcore (+0x2d811) (0x006bd088)
  6 0x006ea779 in byondcore (+0x2a778) (0x006bd09c)
  7 0x008c4675 in byondcore (+0x204674) (0x006bd0c4)
  8 0x008c30c5 in byondcore (+0x2030c4) (0x006bd0d4)
  9 0x008d5a71 in byondcore (+0x215a70) (0x006bd0fc)
  10 0x004444a2 in dreamseeker (+0x444a1) (0x006bd12c)
  11 0x00e3ac41 in mfc120u (+0x23ac40) (0x006bd1fc)
  12 0x00e3a901 in mfc120u (+0x23a900) (0x006bd21c)
  13 0x00e38f33 in mfc120u (+0x238f32) (0x006bd28c)
  14 0x00e39155 in mfc120u (+0x239154) (0x006bd2ac)
  15 0x00d37e8e in mfc120u (+0x137e8d) (0x006bd2e8)
  16 0x7fa174fa in user32 (+0x974f9) (0x006bd318)
  17 0x7fa17b47 in user32 (+0x97b46) (0x006bd358)
  18 0x7fa19fea in user32 (+0x99fe9) (0x006bd3a8)
  19 0x7f9dc94c in user32 (+0x5c94b) (0x006bd4b8)
  20 0x00e286f2 in mfc120u (+0x2286f1) (0x006bd4fc)
  21 0x00de0773 in mfc120u (+0x1e0772) (0x006bd514)
  22 0x00de09f6 in mfc120u (+0x1e09f5) (0x006bd568)
  23 0x0042476e in dreamseeker (+0x2476d) (0x006bfe60)
  24 0x00e4bdc4 in mfc120u (+0x24bdc3) (0x006bfe74)
  25 0x0045760d in dreamseeker (+0x5760c) (0x006bfec0)
  26 0x7b46277c in kernel32 (+0x4277b) (0x006bfed8)
  27 0x7b4641ae in kernel32 (+0x441ad) (0x006bffd8)
  28 0x7b46278a in kernel32 (+0x42789) (0x006bffec)
0x7bc4f02e: movl        %edx,0x4(%eax)
Jan 24 2020, 2:01 pm
In response to Xkeeper
A new issue needs a new bug report; this one has been closed.

The fix I put in 1507 was causing some other, different crashes so the new fix in 1508 should have taken care of everything. This is very strange.