DLL/SO Numerical Return Type

ID:816589
 
Jun 15 2012, 3:09 am
Keywords: call, dll, library, so
Not Feasible
Applies to:DM Language
Status: Not Feasible
Implementing this feature is not possible now or in the foreseeable future
Lummox JR wrote:
I do agree that numerical calls with floating point numbers could be useful to avoid having to convert to and from a string, though I'm not sure how feasible that would be to setup. Presumably we'd need some kind of way to differentiate between what kinds of data are expected for which functions.

Being able to use DLL functions that return floating point numbers would be very useful, because we would avoid the conversion to and from a string.

The current DLL function signature we are allowed to use is the following:
extern "C" char *func(int argc, char *argv[]) 
// argc = #arguments, argv[] = array of arguments

The current format for using the call proc to access the functions of a DLL is as follows:
Format: 
    call(LibName,FuncName)(Arguments) 
Args: 
    LibName: name of external library ("test.DLL") 
    FuncName: name of function in external library ("func")


My suggestion is the following:
Format: 
    call(LibName,FuncName,FuncSig=LIBFUNCSIG_STRING)(Arguments) 
Args: 
    LibName: name of external library ("test.DLL") 
    FuncName: name of function in external library ("func") 
    FuncSig: LIBFUNC_STRING or LIBFUNC_NUMBER

This adds a third argument to the call proc that tells it to assume a certain function signature.
The other signature I propose is:
extern "C" float func(int argc, float argv[]) 
// argc = #arguments, argv[] = array of arguments

A new signature that takes an array of floats as input and returns a float as output.

This won't break existing, working DM code with shared libraries, and can easily be extended to include other signatures if need be. (If desired, it could even include a fourth argument for the call proc, and use the two new arguments to determine both input and output types.)
Jun 15 2012, 3:21 am
This doesn't really offer a very future-proofed implementation, or permit mixing and matching of parameter types, which is the more probable use-case for parameters passed to DLLs/SOs.

To make a reference back to a topic that discussed this earlier (and I see you've made a forward reference to here from ...):

http://www.byond.com/ forum/?post=675792&hl=dll#comment1863839

Access to more primitive types (basically just numeric ones) would be pretty neat and genuinely helpful, I think. the call() syntax on the C/C++ side could offer us a void ** instead of a char **, and we could state that you'd expect numeric arguments to always be doubles or some such, and the implementer can then cast the pointers appropriately for their call. This saves this overhead I keep hearing people mention.

This does carry binary compatibility to older DLLs/SOs (although in my experience, many of the DLL/SO developers are still around, and can update), but probably makes the dynamic linker throw a fit at older DLLs/SOs. The interface is not one BYOND should be afraid of breaking however, in the interest of "getting it right", as the current value of our library set that uses DLLs/SOs, to the general developer-base is quite small, mostly I think for deployment and dev testing reasons (no linker inclusion of library directories / copying of the DLL into your project etc).

What I propose instead, is you and I have a chat together about DLL/SO based libraries in general, and put together a more all encompassing feature request, that provides a minimal effort plan to making DLL/SO based libraries useful, and accessible to BYOND developers as a whole. Discussions about improvements to provided / return types is nice, but it's only really part of the issue.
Jun 15 2012, 6:24 am
In response to Stephen001
Stephen001 wrote:
What I propose instead, is you and I have a chat together about DLL/SO based libraries in general, and put together a more all encompassing feature request, that provides a minimal effort plan to making DLL/SO based libraries useful, and accessible to BYOND developers as a whole. Discussions about improvements to provided / return types is nice, but it's only really part of the issue.

You're right! This really got me thinking, and I've come up with something that should be the real feature request!

What if the function signature we had to use was something like this?:
extern "C" LPDMOBJECT func(LPDMOBJECT object)
/* object = datum, list, savefile, sound, icon or primitive directly from DM structure */

The implementation for the DMOBJECT struct (provided by Tom and Lummox JR) would be something like this:
/* DM_Object.h */
#include <stdio.h>

struct DMOBJECT_;

    /* A reference to an object or a primitive */
typedef struct DMREF_
{
        /* Evertyhing in DM is an instance of one of these things or null */
    union val_
    {
        struct DMOBJECT_ * vars; /* Pointer to an object     */
        char *              str; /* A null-terminated string */
        float               num; /* A number                 */
        FILE *             file; /* A file                   */
        /* ICON */
        /* SOUND */
        /* SAVEFILE */
    } val;
} DMREF, *LPDMREF;

    /* An abstract instance of anything in DM */
typedef struct DMOBJECT_
{
    char *    type;        /* string containing type, i.e. "/mob", "/list", "text", "num"    */
    char *    parent_type; /* string containing parent_type, i.e. "/atom", "/datum", NULL    */
    DMREF *   data;        /* For datums: array of var names. For lists: array of objects.    /
                                                 /  For text: pointer to a string. For numbers: pointer to a float */
    struct DMOBJECT_ ** vars; /* For datums: vars list. For lists: list associations         */
    unsigned  len;         /* The length of the above 2 arrays                               */
} DMOBJECT, *LPDMOBJECT;

/* I haven't included how icons, savefiles or sounds will work, but they too can be included  /
/  and will have their own structures and functions.                                         */

When you pass an object to a DLL via call, it would automatically be converted into a struct for use in C code.
The usefulness of this:
  1. You have automatic data serialization.
  2. You don't need to worry about converting a string to a number and back and then back to a number.
  3. We never have to worry about having different function signatures. If we need more parameters, we can simply pass a list.
  4. An icon sent this way would contain a raw pixel buffer and all of the icon_state and frame data for you to manipulate at will.
  5. A sound would allow you to access the raw data of the sound file so there would be absolutely no limit to what you can do with sounds.
  6. Many features that are commonly requested could now be accomplished by the developer themself and would have no need to be implemented natively.
  7. This saves everyone time, and gives Tom and Lummox JR time to work on what matters. (Like the Flash client?)
  8. The possibilities are endless!


What do you think? This would increase the usefulness of shared libraries several times over.
Do you have anything to add, and/or do you think this is feasible?
Did this address all of your concerns with DLLs/SOs?
Jun 15 2012, 7:49 am
If you're using a DLL or SO it's most likely because there's something else lacking with BYOND that'd be far easier to add than this. This sounds like a practical solution, you're just dealing with an impractical problem. Is there a particular reason why you'd like to have better DLL support?
Jun 15 2012, 8:37 am
In response to Forum_account
Are you replying to the original post or the newer post?
The newer post already has 8 reasons why I would use it.

As for why... Many times I come across a problem, and I have to think of how to accomplish it efficiently and still stay within BYOND's limits. For example, trying to provide lighting with an independent normal for every pixel on the map is a very slow process that is quick and easy in a regular 3D engine with a normal map. Using DrawBox to set each pixel is a very slow process. (I'm guessing it has to compress and uncompress the PNG with every DrawBox call.)
That's one example I would use it for. That could be its own feature request, but there's always something else I want to optimize myself. Rather than requesting features again and again for specific things that only I will really use, I would rather just program it myself.
Another one is a pseudo-random number generator. I've taken to making my own random number generator, but it relies on the property of integer multiplication, and the fact that when it overflows (due to multiplication), there are still bits left where the old number was. In BYOND, if you multiply, it will use a floating-point model and lose precision instead of overflowing. Nonetheless, I couldn't access the bits myself.
I didn't want to rely on a DLL and passing in a string, because that conversion is going to slow down an algorithm that I want to be as fast as possible. So, I ended up making a proc that multiplies them as though they are integers using the mathematical rules an electrically engineered circuit would use. (Luckily I'm a smart computer scientist, and I'm able to come up with these things.)
In any case, that problem would be solved if I could just pass the float directly to the DLL (conversion from float to integer and back isn't all that bad, not to mention integer operations are faster that floating point ones.).

Other examples I've seen is people wanting to have dynamic sound modification, such as the feature request to query the sound's length and other such details. If you had access to the raw data, you could do whatever you want with it. (Another thing is like having a bullet fire sound be slightly different every time, or having levels that are generated based on the music, i.e. Audiosurf.)

In the end, people could start making libraries for different things that were never possible before. Rather than putting all of the stress on two developers to make everything (they never will), we might as well help them out by doing some work ourselves.

This would also basically add pseudo-support for multi-threading. If you can send literally entire objects to a DLL, you could spawn that DLL call off and let it handle something (of course, polling it every-so-often) while the DM code runs the rest of the game.

And, as you can see, there is a fairly simple and intuitive approach to making this possible, as I showed in my C code example. The only question is how feasible it is for them to do. Obviously, their structs may be a lot different than the ones I showed there, because I don't know exactly how their internals are stored. The basic idea is there, though, and hopefully the conversion will be as simple as converting the type and parent_type to strings and then assigning pointers. They have to choose the interface intelligently to make the conversion as simple as possible.
Jun 15 2012, 9:02 am
I don't see how this feature makes any of the things you mentioned more easily doable. I guess It'd help with the PRNG, but parsing a single number isn't that hard.

lighting with an independent normal for every pixel
dynamic sound modification

With the amount of resources the server would have to send to clients, these things just aren't practical. If you want these kinds of features, BYOND is probably not the platform for you. I'm all for adding new features if they help people make games, but I can't imagine how someone would need either of those features to make a fun game - especially when you consider how poorly they'd perform over a network.
Jun 15 2012, 9:05 am
In response to Forum_account
I never said anything about doing it over a network. I was doing static lighting, and I just wanted the initial generation of normals and textures to be faster. Right now I'm investigating ways to do it.
Also, I don't think dynamic sound modification is particularly slow. You just can't do it in DM code at all.
Jun 15 2012, 9:44 am
The sound modification may not be slow, but having to send updated sounds to clients would be much slower than having them already downloaded as part of the .rsc. If you're not doing it over a network you're going to have security problems when people try to run you're game, which also makes it not practical.

Again, these things aren't practical. There's nothing wrong with being curious about how these features work and wanting to goof around with them, but BYOND's just not the tool for that.
Jun 15 2012, 9:50 am
In response to Forum_account
Anyway, I haven't tried that yet. I was just trying to give another example of why it might be useful.

You haven't argued against any other points. How does just one point bring the entire discussion suddenly meaningless?

I don't get that at all.
Jun 15 2012, 9:57 am
Better DLL support is irrelevant because using DLLs is not practical. The only things that are worth using DLLs for, which there aren't many such features (database connectivity is the only one I can think of), should simply be features of BYOND so you can avoid the security issues and cumbersome nature of using DLLs.
Jun 15 2012, 10:00 am
In response to Forum_account
Well, that's your opinion.
I happen to not agree.
If I'm the only one. So be it.
Then this will never be implemented.
However, I see no reason to discuss our opinions.
Dec 8 2015, 12:07 am
Potential solution: Accept a void* when calling DLL/SO functions, which may be a pointer to a char* array or an integer/float/double (whichever works for you) which is then converted to a number in BYOND, or a nullptr which is converted to null in BYOND.
Oct 9 2018, 11:09 am (Edited on Oct 9 2018, 11:32 am)
 
struct value_t {
  type = V_NUM|V_STRING|V_NULL
  union {
    float num;
    char * c;
  }
};

extern "C" char *func(int argc, value_t * argv)


Don't make it too complicated. While i'm sure you all want to be able to pass in objects, thats more work then you realize and focus should be put on making this easy for lummox to bang out as otherwise he won't.