json_encode spits out non-JSON-valid encodings of unicode

ID:2215619
 
Feb 19 2017, 8:36 pm
Resolved
json_encode() produced code that was valid for JavaScript but not for JSON in the case of certain special characters.
BYOND Version:511.1374
Operating System:Windows 7 Home Premium 64-bit
Web Browser:Firefox 51.0
Applies to:Dream Daemon
Status: Resolved (511.1375)
This issue has been resolved.
Descriptive Problem Summary:
Text containing unicode characters are encoded as \x(hex) with json_encode(). This is invalid JSON and won't be parsed by JS.

Code Snippet (if applicable) to Reproduce Problem:
/client/proc/sendUni()
    var/list/test = list("foo" = "Toupé")
    src << output(json_encode(test), "someWindow:someFunction")


And in the browser...
function someFunction(test) {
    try {
        test = JSON.parse(test);
    } catch (e) {
        alert(e);
    }
}


Expected Results:
Success parsing the JSON, clearly

Actual Results:
JS throws a JSON parse exception of "Error: JSON parse error for: (json)".
The JSON received by the JS function in the example above is: "{"foo": "Toup\xe9"}".

Workarounds:
Sanitize the JSON before parsing:
    test = test.replace(/\\x[\dA-F]{2}/ig, function(match) {
        return String.fromCharCode(parseInt(match.replace(/\\x/g, ''), 16));
    });


Feb 19 2017, 9:31 pm
Hrm. I wasn't aware that \x would be invalid for JSON, considering it's valid in JavaScript. I can make a change for that.
Feb 20 2017, 2:32 pm
Both Chromium (56.0.2924.87, ie the same version listed in your report) and NodeJS (6.9.2) parse the following JSON object as expected:
{"foo": "Toup\xe9"}


Neither, however, parse the following as expected, though in a different way (both give the string as "Toupà ©" minus the space; space added because the forums tries to be smart and make it an é):
{"foo": "Toup\xc3\xa9"}


I suspect this is a character encoding problem, not a JSON format problem; "\xe9" is invalid UTF-8. Perhaps your system is encoding as Windows-1252/ISO 8859-1 then decoding as UTF-8 (and throwing errors since "\xe9" is invalid UTF-8), and mine is encoding as UTF-8 and decoding as 1252/8859-1?
Feb 20 2017, 2:35 pm
Are you aware that byond uses IE. Like, at all. I guess I should have removed the automatic browser version from my post but dang dude.
Feb 20 2017, 2:42 pm
So it's not parsed by Internet Explorer's JS implementation.

That's not the way the issue read to me; yes, BYOND uses IE, but that's not the only place you can send json_encode()'s results.

Does it parse the following as expected?
{"foo": "Toup\u00e9"}
Feb 20 2017, 2:45 pm
Presumably yes, because that's valid according to the spec. That certainly isn't what json_encode spits out though.

A note: when in doubt, shove your crap in http://jsonlint.com/
Feb 20 2017, 4:05 pm
Just gonna leave this here:



Notably, no \x
Feb 20 2017, 4:39 pm
In response to GinjaNinja32
It "smartly converts it to an é" because the forums use utf8 encoding.
It works in browsers(and ditto to v8) because its a compliant javascript string, not a compliant JSON string.
Feb 22 2017, 12:58 pm
Lummox JR resolved issue with message:
json_encode() produced code that was valid for JavaScript but not for JSON in the case of certain special characters.