So, yes, I haven't been posting on this blog for some time. I've been focusing on my own personal projects, for now (which does include fotoDen). Part 2 of that post series, making a website from scratch will be coming up soon, of course, but I felt like making this quick post about something I did recently.
If you know me, then you know that I like playing Space Station 13, a FOSS game on a closed engine that focuses on attempting to survive in a hellish space environment. By hellish, I don't mean horror, I mean chaotic. With each round being defined by both random events, and the people that play within the round, it tends to become... a mess. A fun mess, but alas, a mess.
This isn't about Space Station 13, however. It's about the engine behind Space Station 13, BYOND, and my attempts to communicate to it from beyond (ha, ha.) its current set of accessible endpoints. BYOND is a bit of an ancient engine, and its entire schtick is being an 'easy engine to learn' - however, it's very much stuck in the early 2000s, with its single threaded Dream Daemon, and its Internet Explorer/WinForms based user interface. Now, WinForms isn't bad (I haven't used it, but the interface seems fine enough) - but, it's still based on Internet Explorer (or rather, Trident), the web engine that Microsoft abandoned several years ago. We can leave further criticism for another time, however. DM is apparently a language that... has several smells to it, and is why I leer away from actively contributing to BYOND projects, as even though it's a good thing to contribute to the thing you enjoy doing, I would much rather be contributing in... other ways.
This is what I'll be describing, here.
Simply enough, this has already been reverse engineered long ago by people who were active enough on BYOND - PHP applets and the like power the method to actually get information about BYOND server states. How does this work?
This is achieved by something known as a Topic() call. Topic calls are
essentially ways for Dream Daemon (the server that runs BYOND executable
files) servers to communicate to one another, sending either plain text, or a
32-bit floating point number (why? Who knows!). Topic calls are the equivalent
to a remote procedure call for a BYOND server, as calling a Topic on a BYOND
server will make it execute some code, before returning its result.
BYOND topic calls between servers are now encrypted - I don't know enough about cryptography at the moment to actually reverse it, so instead, I looked up an existing description of how unencrypted calls work (which still work for non-BYOND clients to BYOND servers).
Simply put, a request would look like this:
[ 00 83 (data size as pair) 00 00 00 00 00 (data) 00 ]
In order, we have:
The data itself looks like a URL search query string - so, essentially, it would be in the form of:
?TopicCallHere
I'll explain how the pair works in the next section, here.
A response would look like this:
[ 00 83 00 01 06 43 00 ]
This is different from the request:
First off, of course, we have our magic number, which defines this as a BYOND
Topic communication (even the encrypted Topics start with this same magic
number). Up next, however, we have two bytes from T[2], T[3].
This pair of bytes is actually, in fact, the length of the topic as a pair of two 8-bit bytes. So, it's actually a 16-bit integer split apart into two bytes. Say you had something that had a length of about... 255 characters (yes, that's too easy). If you had it as a 16 bit integer, it would look like this in binary:
0000000011111111
1 8 16
Now, in order to get this as its pair, you would have to split it across the middle, and then store each one separate in order to get it as a pair of bytes.
00000000
11111111
Now, you have your byte pair - 0x00 and 0x0F.
The fifth byte, at T[4], indicates what type of response we expect to recieve.
0x06 indicates that this is an ASCII string - and, since we can easily count
our topic length in our above example (hint, 1 in hexadecimal is still 1), we
can immediately assume that our 43 is just an ASCII-encoded string.
So, what is 0x43 in ASCII? That's just C.
This covers our Topic() calls - however, Topic calls have their own issues. They're all URL encoded - so, some characters will appear as their percentage-escaped forms, which is not ideal for some forms of data. Yes, you can just interpret it that way - but sometimes, you need more than just that.
Now, Topic calls are for external, remote calls, of course - however, what if you want to interface with something internal? Say, perform a system call, or interface with some other kind of protocol. BYOND Topic can't do that for you.
This is where another part of BYOND can come in - the call() function. This
function can actually run external libraries for you - as long as you have your
Dream Daemon security set low enough so it doesn't nag you - so that you can
actually communicate to other programs, or perform other library functions
(e.g., if you want to do something that would be much faster than it would be
in BYOND). The call function itself has some issues (it blocks - thanks, single
threaded applications), but for the most part, it will run what you need it
to.
It requires some specifics, however, but it's easy enough to get over. The functions that you call from BYOND must have this following C signature:
char func(int argc, char *argv[])
As according to the DM reference. This is simple enough to implement, and it really just depends on what you want to implement, and how efficient it is. That's the programmer's problem.
Now we come to our final part - how does this relate to Go?
Luckily, Go has a big enough library that can help facilitate a lot of this within the language. We can start backwards, with our interfacing.
So, I wanted to avoid doing this in C - yes, it's a very respectable language,
and yes, I do need to eventually try my hand at C so I understand its
complexities, but I also wanted to solve the problem I had of communicating to a
Dream Daemon server.
Go has facilities to, in fact, interface with C programs while also potentially keeping its pre-existing memory safety. (You can view that documentation here.) So, what would the above look like in Go?
It's actually a little complicated to look at - but is simple enough in execution. It just requires a couple of tricks.
BYOND requires that the function signature be exactly the same as it were - otherwise it will have issues running the external library itself. So, the first instinct, to just do:
func MyFunction(n int, s string) string
Seems like that it would work, right? This is actually completely incorrect.
Why is this incorrect, however? A peek into the .h file that gets generated
can have some
more insight...
...
typedef GoInt32 GoInt
...
typedef _GoString_ GoString
...
A-ha! Those two types above aren't the C types int and string (the string
type doesn't even exist in C that explicitly) - it's actually these two types,
GoInt and GoString. This does not align with our needed function signature
at all, as it'll end up looking like:
GoString MyFunction(GoInt n, GoString s)
Which is incorrect. So, how the hell do you actually get our signature?
A look into the cgo documents doesn't exactly reveal this - but you can, in fact, set the types in our function string to C types.
So, if we modify our function signature:
func MyFunction(n C.int, args *C.char) *C.char
we now get a completely different result in our header file:
char* MyFunction(int n, char* args)
Which is exactly what we want. We come to one more issue, however: how the hell do we actually parse that array? If you look above, you'll be able to see that BYOND actually passes its arguments to the library as an array of C strings, as well as the actual length of the array itself.
This comes down to a trick that I found in the official Go wiki (or elsewhere - but the Go wiki has the solution to this). If you want to convert a pointer to some array into an actual Go array, you would obviously require the number of args, and the pointer to the C array itself. Our solution for a set of string arguments would look like this:
a := (*[(1 << 29) - 1]*C.char)(unsafe.Pointer(args))[:n:n]
(edit: I looked at the Go implementation details, and found out some more interesting information that actually allowed me to at least figure out how this works.)
This line of code:
(1 << 29 - 1) sized *C.char arraysn, and capacity size n. (the
'low' portion of it is left out, because it's automatically assumed to be
0.)Of course, there is an alternative, which gives us a Go byte slice, but in this case, we need the strings more than anything.
This allows us to address our arguments as a native Go slice - which makes things much more easier to work with.
You can find this solution on the Go wiki on GitHub.
So, we now have a valid array of C strings. We can allocate a new, actual Go
array for our C strings by just iterating over it, and performing C.GoString
on each of its elements, but again - that's a programmer's decision.
Finally, we need to actually export our function so that BYOND can actually see it. We can combine all our steps to get this final set that we can define as the general body of our function:
//export MyFunction
func MyFunction(n C.int, args *C.char) *C.char {
a := (*[(1 << 29) - 1]*C.char)(unsafe.Pointer(args))[:n:n}
return C.GoString("result")
}
There are some issues still with some standard library functions (an example that I'll give later had to be written in a more specific way so that BYOND wouldn't just cause errors when the library ran), but the implementation details of this are, once again, up to the programmer - feel free to experiment, and figure out what does and what does not cause crashes!
Up next, of course, is our Topic call - which is a bit more easier than what
I've just described.
In essence, we just have to ensure that the topics we write are completely valid, so, all we have to do is figure out how to actually write a topic, send it, and interpret the response, right?
There's a little more to it. BYOND doesn't necessarily close the connection for you once it's completed its transmission, meaning that you'll have to figure out how to close it yourself. How, you could ask - the method I used was actually reading out the response, ensuring its validity, and then closing the connection client-side in order to complete the transmission.
Writing a topic is simple enough, so I'll focus on actually recieving the result and parsing it.
This is taken from pre-existing code that I wrote, which I will post a link to later:
func readTopic(r io.Reader) (string, error) {
head := make([]byte, 5)
_, err := r.Read(head)
if err != nil {
return "", err
}
// if it doesn't have the correct magic number,
// or isn't an ASCII string,
// return InvalidTopic
if bytes.Compare(BYONDMagicNumber, head[:2]) != 0 || head[4] != 0x06 {
return "", InvalidTopic
}
var l uint16
if head[2] == 0x00 {
// it'll just be the fourth byte as a uint16
l = uint16(head[3])
} else {
// use the BigEndian conversion
l = binary.BigEndian.Uint16(head[2:4])
}
m := make([]byte, l)
_, err = r.Read(m)
if err != nil {
return "", err
}
return string(m), nil
}
(edit: some of this was wrong, and it's been fixed)
As seen, it corresponds to the explanation of the topic response from earlier. The first five bytes of the response are read into an array, and then compared against for any issues (is the magic number wrong? Did we get a 32 bit floating point instead of an ASCII string?), as well as checking its size by checking if the pair of bytes is both populated, or if the second of the two bytes is the only populated byte. At the end, we eventually create a new byte array with our parsed length, and then read out the rest of the result into that array, and then return our string.
This, compared to the kneejerk usage of using io.ReadAll(), allows for the
client to close its connection once it's recieved the Topic result.
As for writing a topic? I can let this code explain itself for you:
func (t *Topic) Close() error {
if len(t.raw) > (1 << 16) {
return fmt.Errorf("topic is too big")
}
t.buf.Write(BYONDMagicNumber)
// make our uint16 buffer
b := make([]byte, 2)
// put our topic's length into our buffer (big-endian 8-bit byte pair)
binary.BigEndian.PutUint16(b, uint16(len(t.raw)+6))
// write it
t.buf.Write(b)
// write the spacing
t.buf.Write([]byte{0x00, 0x00, 0x00, 0x00, 0x00})
// write the actual topic
t.buf.Write(t.raw)
// write the end of the topic string (null termination)
t.buf.Write([]byte{0x00})
// declare it to be closed
t.c = true
return nil
}
So, of course, where is all of this actually used? I've implemented both of these into a couple projects I have, which I'll list here:
Both of these implement the details I found above in order to communicate and facilitate data from Dream Daemon servers, and make it a little bit easier to talk to BYOND servers - whether internal, or external - from Go programs.
Hopefully, this gives you some insight as to how to communicate to a BYOND server from a language such as Go!
