ID:1190782
 
Hey there,

With all the apparent hype over cross-server saving, I've decided to have my go at it and whipped up a nifty little piece of work (not fully complete) in the past two hours or so that currently handles inserting and updating user fields to an external SQL database using PHP as a middleman.

database
{

var
{
processor = "http://localhost/byond_db/processor.php"; // the location of the database connect page
Parser/dbParser = null;

client/user = null;
}

New() { . = ..(); }

proc
{

initialize( var/client/USER )
{
user = USER;
dbParser = new /Parser();
}

check_db()
{
var http[] = world.Export( processor );
if( !http ) return FALSE;
var/F = http["CONTENT"];
var/content;
if(F) content = html_encode(file2text(F));
if( dbParser.ParseBetween( content, "<dbcon>", "</dbcon>" ) == "Connected to database." ) return TRUE;
else return FALSE;
}

}

}

MySQL
{

var
{
database/db = null;
}

New( database/DB )
{
db = DB;
}

proc
{
// updates a current entry
// if the entry is not in the database, then it will be created.
update( var/table, var/list/arg_list )
{
if( !db.check_db() ) return;
var url = db.processor;
url = "[url]?action=update&table=[table]";
for( var/DATA_TYPE/argument in arg_list ) url = "[url]&[argument.field]=[argument.value]";

var http[] = world.Export( url );
if( !http ) return FALSE;
var/F = http["CONTENT"];
var/content;
if(F) content = html_encode(file2text(F));
if( db.dbParser.ParseBetween( content, "<updateStatus>", "</updateStatus>" ) == "Update successful." ) return TRUE;
else return FALSE;
}

read()
{
// write this function! and then saving is done!
}
}

}

DATA_TYPE
{
var
{
field;
value;
}

New( var/FIELD, var/VALUE )
{
. = ..();
field = FIELD;
value = VALUE;
}
}


You don't want to see the PHP code right now - it's far too messy, but if you were interested, it essentially parses each GET parameter and inserts into a table created to accept such parameters and values. Effectively, if I pass an argument_list holding DATA_TYPE with field="name" and value="cauti0n", it will place it in my SQL database, in the table "characters", column "name" with value "cauti0n".

It seems that a decent amount of people disapprove of this method of handling the saving, namely because of server reliability outside of BYOND. Unfortunately, there is more of a reason why I am working on this outside of just BYOND - I'm working on something that sort of ties BYOND and websites together, effectively requiring PHP as the middleground. However, I'm sure I'm not the first person to create something like this and as such, I was wondering if people had any nice performance tests that showed the reliability of this sort of system. I imagine that if the SQL database is hosted on a pretty decent server, there should be very minimal issues with this sort of process (unless someday I happen to have thousands of players playing my BYOND game).

Any information from others who have attempted something like this?

Note: Feel free to take this/modify this as you see fit. My career isn't bent on what I do in BYOND, so my thoughts on credit don't matter too much. As far as I see it, anything I do here can be open-source and I would never mind :). The above is fairly modular, and plug-in-play if you have a responding PHP file. The only other exception is the Parser, but that's a very minimalistic parser - you can create yours yourself if you wish. It's all an independent file, though :)
BYOND already supports native MySQL access using the Dantom.DB library, there's no need to use PHP as a middleman.
Generally, I'd expect the MySQL server to be able to complete ~1000 transactions per second (assuming they aren't huge, there's no massive table contention due to pessimistic locks etc) on a reasonable dual-core machine. It won't be your bottleneck, that much I can assure you, even under "high load" from the BYOND server.

If we take the assumption that you have many BYOND worlds, and each player causes a transaction to complete every ... I dunno, 10 seconds (which is probably too frequently for a save really), then you'd expect the MySQL server to start to struggle at a player count of 10,000. At which point you'd switch to InnoDB table formats, get yourself an 8 core server with an okay RAID system, and suddenly it scales to ~3000 transactions per second, with no code changes on your part, and you're good for 30,000 players. MySQL tends to cap out at ~9000 transactions per second on a single machine (a 36 core beast, usually), so 90,000 players.

Long before that though, you'll get other contention, most notably on the BYOND end. Assuming a well optimised game, you're looking at ... 20 servers to support your 10,000 players, at a guess. The PHP end also, can be quite problematic. Disturbingly, a complex web-app like wordpress can only really manage to serve up ~100 requests a second. Your own scripts presumably are quite straight to point and bare, so perhaps could manage ~500+ requests per second on the PHP side, meaning you'd need to load balance between a few (say ... 2) PHP servers connecting to the same database server, to actually saturate the MySQL end. But still, that's support for say 5,000 players off one PHP server. Need more performance? Load balance the PHP servers first, then scale up the DB server.

Your biggest scalability concern is as stated, BYOND itself, needing more servers for that etc.

Now, latency, however, is a more interesting story. The best means of reducing latency (so, making an individual save complete faster) is reducing the number of hops, particularly network-bound ones. If your BYOND servers, PHP server and DB server are all co-located in the same data-centre, you're looking at a network I/O response time of say ... 0.25 ms between each machine. So 0.25 ms BYOND -> PHP, again to MySQL, again to PHP, and finally back to BYOND. So ... 1 ms, meaning you'd get say ... 100 completions in a 0.1 second BYOND tick, assuming (unrealistically) no CPU time on the request itself, impossibly small data size over the wire etc.

If you're just buying off random VPS providers or your VPS provider spreads across data centres, you start to get into trouble. Introduce a 20 ms flight time for data between servers, and suddenly you're looking at an 80 ms response and can only really process one request per tick, 10 per second, 100 players on the server in terms of saving. Suddenly the latency is your bottleneck. Your BYOND world can do 500 players, but would only perform well with 100, more and suddenly there's lag off the saving, due to the distance between servers. Bummer. If they're all colocated, then your BYOND world can manage 1000 saves a second, so theoretically 10,000 players, much more than we know the actual game logic would scale to, or other non-saving I/O etc.

If we took PHP out of the mix there, we half the latency, and remove PHP's through-put limitations. It's not essential to do though, as described, the BYOND server will be the bottleneck in good scenarios.

So your priority for good through-put AND latency is essentially:
  1. Co-locate servers in the same data centre
  2. Optimise your game logic
  3. Remove extra hops / processes from the request
  4. Optimise / scale your SQL server and queries
With your implementation you may run into trouble with PHP/the webserver's GET size limits.
In response to Murrawhip
Murrawhip wrote:
With your implementation you may run into trouble with PHP/the webserver's GET size limits.

This was an issue I was concerned about as well.
EDIT: However, the clever programmer will easily be able to get around this - IE: update() multiple times using different fields.

My SQL statement goes along the lines of
INSERT INTO characters ( name, key_name ) VALUES( "cauti0n", "CauTi0N" ) ON DUPLICATE KEY UPDATE name="cauti0n", key_name="CauTi0N"

with name, key_name, and their respective values being auto-generated by the PHP script. This can be remedied by sending multiple values and just sending the key_name (or any UNIQUE index) each time.
Your limit in standard apache config is 8kb of GET data.
In response to Nadrew
Nadrew wrote:
BYOND already supports native MySQL access using the Dantom.DB library, there's no need to use PHP as a middleman.

Reading your post on the subject I think this may very well be a more feasible solution - though, I must admit mine is not exclusively for MySQL connection, but also for some tests on a website. This definitely looks helpful, however, and I will probably migrate into this library.
In response to Stephen001
And PHP with suhosin has a 512 character limit per value. It's not too rare of a situation where someone might store a lengthy body of text, I think.
Just making sure you know ;)
In response to Stephen001
Stephen001 wrote:
Your limit in standard apache config is 8kb of GET data.

Do you happen to know the limit for POST data? Mine is easily modifiable to send POST requests through AJAX.

Also, taking a look at Nadrew/Dantom's DB library - where/how would I find instructions on including a MySQL installation, and would hosts need to download MySQL as well, or does it get compiled as well?
You just need to distribute libmysql.dll/libmysql.so with your project, look at the license for doing so beforehand though.
POST requests (in theory) have no limit, it's what you'd use to upload files etc also. As a matter of curiosity, how is the server doing an AJAX POST?
In response to Stephen001
Stephen001 wrote:
POST requests (in theory) have no limit, it's what you'd use to upload files etc also. As a matter of curiosity, how is the server doing an AJAX POST?

Just like any normal way you would do it. Build the JS in the browser referencing a processing page and send the arguments through the client's browser. I feel like you are asking more here, but I'm not sure what exactly.
Then the save is actually being sent from the client, not the server? I just found it a little odd, as it means for example that you cannot save world state (global info about economy, weather etc) via that, as it requires you to pick a client to send via.

Also, as the AJAX request originates from the client, you cannot avoid latency (which say for me, in the UK, is 150 ms) in the request to PHP, by co-locating.
In response to Stephen001
Stephen001 wrote:
Then the save is actually being sent from the client, not the server? I just found it a little odd, as it means for example that you cannot save world state (global info about economy, weather etc) via that, as it requires you to pick a client to send via.

That's correct - all information saved would only be client information. Any world information would have to be saved through GET posts (or the host's browser).

Stephen001 wrote:
Also, as the AJAX request originates from the client, you cannot avoid latency (which say for me, in the UK, is 150 ms) in the request to PHP, by co-locating.

This part is true, and not a consideration I took account of.

Do you have any suggestions to utilize POST data without requiring a browser to run on a client end? Theoretically if Dreamseeker could just run a .js file without client specifications, that would work fine - except it's dynamically generated...
Well, you'd want some DLL or external process to allow for POST requests to be sent on the server, which is kind of messy. Or ... go direct to MySQL via Dantom.DB.
In response to Stephen001
Yeah - Dantom.DB is what this all seems to be pointing at.