Bayle Shanks's website: proj-oot-ootInteropNotes2

"JNA- and ctypes-like FFI is convenient, there is no doubt about it. But making it the default way to interface with native APIs is a major safety issue. C and C++ have headers for a reason and redeclaring everything by hand is not only time-consuming, but also error-prone. A little mistake in your ccall and you just happily segfaulted, and that’s an optimistic scenario. And now try to correctly wrap strerror_r or similar...So this “feature” instead of eliminating boilerplate eliminates type checking. In fact, there is more boilerplate per function in ccall than in, say, pybind11. This is another area where Python and even Java with their C API win. There is a way to abuse FFI there too, but at least it’s not actively encouraged." [1]

---

"In my opinion, in Clojure, the Java layer pokes the Clojure layer way more than it should. Another example is that Clojure require to access the Java standard library in many cases. The problem with this is that Java functions and Clojure functions are not interchangable. For instance you can do "(apply clojureFunction [0 0])". But you can not do "(apply .javaFunction [0 0])". You need to wrap the Java function into a Clojure function explicitly in this way "(apply #(.javaFunction %1 %2) [0 0])". I don’t find this particularly elegant." [2]

---

valarauca1 5 days ago [-]

This is more complex then just a target. Rust has to communicate to the Go Runtime Environment.

Two main points:

-Go uses a very small stack for GoRoutines? to make them dirt cheap. When you exceed this stack, Go maps in more stack for you transparently. Rust generated ASM is running on the Go Stack, but it is expecting when it exceeds its stack to explode, like a C program. As that should be the OS Stack. Larger problem then you think, Rust likes to put a ton of stuff on the stack. This is one of the nice things about Rust is putting _a ton_ of data on the stack is cheap, and makes ownership simpler.

-Go's system calls, and concurrency primitives are cooperative with its runtime. When make they communicate the routine can yield to the runtime. Targeted Rust code would _also_ have to make these calls, as well as 3rd party crates.

Again none of this is impossible, linker directives and FFI magic could import these functions/symbols. But this would also require Go have a stabilized runtime environment for other languages to link against. Currently just stating Go has a runtime is controversial, so I expect this won't happen soon.

masklinn 5 days ago [-]

> This point confuses me; if Rust expects to run on a limited stack, why would it expect to put a ton of data on the stack?

Rust runs on a C stack, while it's not infinite[0] it's a whole other ballpark than a Go stack since it's non-growable (Rust used to use growable stacks before 1.0): the default C stack size is in the megabyte range (8MB virtual on most unices[1]), in Go the initial stack is 2kB (since 1.4, 8k before).

[0] you can set the size to "unlimited", systems will vary in their behaviour there, on my OSX it sets the stack size to 64MB, Linux apparently allows actual unlimited stack sizes but I've no idea how it actually works to provide that

[1] I think libpthread defines its own stack size circa 2MB so 8MB would be the stack of your main thread and 2MB for sub-threads, but I'm not actually sure

vvanders 5 days ago [-]

Things may have changed from when I last looked at it but there's another issue which is pinning.

If you ever want to do more than the most trivial FFI you'll eventually want to be able to pass types back and forth(usually opaque to either side). AFAIK Go doesn't offer any pinning if it's GC'd types so you can have the collector move them from under you.

C# has this beautiful thing where you can pass a delegate as a raw C fn pointer. It makes building interop a wonderful thing but you have to make sure to pin/GCHandle it appropriately.

FiloSottile? 5 days ago [-]

Folks, stop, you are reinventing cgo now.

Defining a Go target for Rust actually makes sense in the context of replacing assembly (which has no runtime, GC or concurrency connotations), I was just too lazy to do it that way :)

valarauca1 5 days ago [-]

> Folks, stop, you are reinventing cgo now.

This is a good thing because `cgo` is really bad. No ALSR, always forking. These are completely _insane_ defaults, it manages to be slower then the JNI [1] which is an FFI from a completely managed stack based VM universe! Not a _compiled_ language.

Somehow a compile language calling a static binary manages to be slower then a dynamic language's runtime calling a static binary...

`cgo` isn't doing _anything right_. It is doing a lot of things wrong.

[1] https://lwn.net/Articles/446701/

---

Why not cgo

Go has a Foreign Function Interface, cgo. cgo allows Go programs to call C functions in the most natural way possible—which is unfortunately not very natural at all. (I know more than I'd like to about cgo, and I can tell you it's not fun.)

By using the C ABI as lingua franca of FFIs, we can call anything from anything: Rust can compile into a library exposing the C ABI, and cgo can use that. It's awkward, but it works.

We can even use reverse-cgo to build Go into a C library and call it from random languages, like I did with Python as a stunt. (It was a stunt folks, stop taking me seriously.)

But cgo does a lot of things to enable that bit of Go naturalness it provides: it will setup a whole stack for C to live in, it makes defer calls to prepare for a panic in a Go callback... this could be will be a whole post of its own.

As a result, the performance cost of each cgo call is way too high for the use case we are thinking about—small hot functions. "

---

probably irrelevant but just putting here in case i need it later: https://blog.filippo.io/rustgo/

---

mbrubeck 14 hours ago [-]

Hi, I'm a Servo developer who worked on some of the Rust code that's in Firefox. Calls between C++ and Rust code in Firefox all go through "extern C" functions. Some of the code involves reference-counted smart pointers. We use RAII wrapper types in both Rust and C++ to ensure that refcounts are incremented and decremented correctly on either side of the FFI boundary.

P.S. This old blog post is not about Rust-in-Firefox, but it does cover a related topic: How the Servo browser engine (written in Rust) interacts with the Spidermonkey JavaScript? engine (written in C++ and embedded in both Gecko and Servo), including garbage-collected JavaScript? objects:

https://research.mozilla.org/2014/08/26/javascript-servos-on...

---

kibwen 131 days ago [-]

Looking ahead to 1.19 (currently in beta, which you can try out easily via rustup), the RFC for unsafe (C-style) unions recently stabilized, which closes IMO the biggest remaining hole in Rust's FFI story. By removing the need for various manual hacks to interoperate with unions from C libraries,...

---

" int_19h 129 days ago [-]

That's exactly the problem. R has a very specific API for its extensions - it's not just vanilla C, it's stuff like SEXP.

Although now that I think more about it, it's not quite as bad as Python, because the data structures are mostly opaque pointers (at least until someone uses USE_RINTERNALS - which people do sometimes, even though they're not supposed to), so all operations have to be done via functions, where you can map them accordingly.

You'd also need to emulate R's object litetime management scheme with Rf_protect etc; but that shouldn't be too difficult, either.

Some more reading on all this:

https://cran.r-project.org/doc/manuals/r-release/R-exts.html...

http://adv-r.had.co.nz/C-interface.html

peatmoss 127 days ago [-]

Oh, yeah, now that you mention it I have seen the SEXP and protect / unprotect stuff before. ... "

---

random (good) Python<->Rust interop example:

[3]

---

C# 'span' for encapsulating memory:

https://github.com/dotnet/corefxlab/blob/master/docs/specs/span.md

---

Rust 'bindgen' apparently helps with interop, macros too:

" bindgen & macros are amazing

I wrote blog posts about these already but I want to talk about these again!

I used bindgen to generate Rust struct definitions for every Ruby struct I need to reference (across 35 different Ruby version). It was kind of… magical? Like I just pointed at some internal Ruby header files (from my local clone of the Ruby source) that I wanted to extract struct definitions from, told it the 8 structs I was interested in, and it just worked.

I think the fact that bindgen lets you interoperate so well with code written in C is really incredible.

Then I used macros (see: my first rust macro) and wrote a bunch of code that referenced those 35 different struct versions made sure that my code works properly with all of them.

And when I introduced a new Ruby version (like 2.5.0) which had internal API changes, the compiler said “hey, your old code working with the structs from Ruby 2.4 doesn’t compile now, you have to deal with that”. "

-- [4]

---

[5]

" ... you can’t just pass a string into a WebAssembly? function. Instead, you have to go through a bunch of steps:

    On the JS side, encode the string into numbers (using something like the TextEncoder API)
    Encoder ring encoding Hello into number equivalent
    Put those numbers into WebAssembly’s memory, which is basically an array of numbers
    JS putting numbers into WebAssembly's memory
    Pass the array index for the first letter of the string to the WebAssembly function
    On the WebAssembly side, use that integer as a pointer to pull out the numbers

And that’s only what’s required for strings. If you have more complex types, then you’re going to have a more convoluted process to get the data back and forth.

If you’re using a lot of WebAssembly? code, you’ll probably abstract this kind of glue code out into a library. Wouldn’t it be nice if you didn’t have to write all that glue code, though? If you could just pass complex values across the language boundary and have them magically work?

That’s what wasm-bindgen does. If you add a few annotations to your Rust code, it will automatically create the code that’s needed (on both sides) to make more complex types work. ... Under the hood, wasm-bindgen is designed to be language-independent. This means that as the tool stabilizes it should be possible to expand support for constructs in other languages, like C/C++. ...

Q. How do you package it all up for npm?

A. wasm-pack

---

huh, even these guys couldn't keep up with their interop:

[6]

---

 ioddly 3 days ago [-]

I'd think one of the big advantages of rewriting Lua in a GC'd language would be that you could ditch the stack API, which is necessary to keep track of pointers but not so intuitive to use. Seems like they didn't take that route though.

wahern 3 days ago [-]

What's the alternative to a stack API? If you look at Python and Perl they use a combination of (1) an explicit stack similar to Lua, (2) instantiation of heavy-weight list or array objects for passing arguments, (3) explicit reference counting, and (4) code generation.

Lua's stack protocol can sometimes be verbose, but I find it infinitely more simple and elegant than the existing alternatives.

I think you tend to see long, unwieldy code blocks that invoke Lua functions multiple times because you can, despite how tricky it can be to track the top of your stack. You rarely see this in Perl or Python because it becomes impossible to manage long before that point. So the solution for Lua is the same as for other interfaces--use shorter, simpler functions and compose them. This is usually easier and more performant to do in Lua, anyhow, because there's less fixed boiler plate and indirection necessary; not to mention Lua gives you many more facilities for stashing and managing your C state across calls (like closure upvalues, userdata values, continuation context cookies, etc) without resorting to global variables either inside or outside the VM.

ioddly 3 days ago [-]

Yeah, implementing a precise GC is a pain, but in Go the hard work of that is already done for you. If I were to write my own Lua-in-Go, my inclination would be to try to make an API something like this

    table := lua.MakeTable()
    table.Set(lua.MakeString("five"), lua.MakeNumber(5))

Rather than manipulating values on the stack. But it's possible there are reasons for not doing that that I'm unaware of. Was just curious.

wahern 3 days ago [-]

Why do you prefer that over

  lua_pushstring(L, "five");
  lua_pushnumber(L, 5);

The above is shorter and also significantly more efficient because you're not creating the intermediate table object. Even in Go, given Lua's coroutine and tailcall semantics (assuming they were implemented) the Go compiler would likely be forced to always heap allocate the argument list table. There's a reason PUC Lua is blazingly fast, and its partly the same reason why LuaJIT?