ID:2777898
 
Applies to:DM Language
Status: Open

Issue hasn't been assigned a status value.
BYOND has supported Unicode for two stable versions. This makes Private Use Areas available for strings. Private Use Areas are reserved Unicode blocks not associated with meaningful symbols - they can be given control or signalling roles, defined by a font to fill a custom role, etc.

https://en.wikipedia.org/wiki/Private_Use_Areas

Text macros currently don't play nice with a variety of things not native to BYOND. For example they are not valid in JSON, which does not permit control characters or lone backslashes in strings.

https://www.json.org/json-en.html

Because of this, they are simply stripped when written out.

https://i.imgur.com/IaLTFvH.png

ohno.dm: https://pastebin.com/bSJW1wxN

This can (and does) cause key and value mismatches that must be worked around. It is currently possible to do so either by implementing an expensive compatibility strategy in user code, or by avoiding text macros altogether -- and needing to avoid using a feature because it is incompatible with common-use technologies that are also included in the software itself is obviously less than ideal.

Instead of the current low ASCII control codes, using PUA codes for text macros would be beneficial because they are well supported, widely permitted, and the number of reserved points for this role is huge. They would allow BYOND to non-destructively communicate text macros, as well as permitting many more to be defined.

Workload permitting, a new language feature for text could be built with them by allowing projects to associate a code with a proc to carry out user defined text transforms.

They also work right now -- they are simply not used in this role. This image demonstrates reading both PUA symbols and \uXXXX escapes from JSON, creating PUA symbols with ascii2text, and writing and re-reading them from compiled DM.

https://i.imgur.com/KGhLQrR.png

readme.js: https://pastebin.com/H0aR05bh
test.dm: https://pastebin.com/Tb89CRcY

Technically, this could increase text size by moving from 1 byte per macro to 2, or 3 (although staying within the BMP, at 2, is probably preferable), but I do not believe this represents a credible concern given the rarity of macros compared to other symbols.

There is the obvious issue of compatibility; I believe this would be a breaking change done right, and so would be best associated with the next major version (whichever that is at time).
Bump!

webview2 is about to be on beta channel with 516 and revising core tech is seeming more feasible.

this would be great to stop the endless jank