proj-oot-ootInteropNotes2

"JNA- and ctypes-like FFI is convenient, there is no doubt about it. But making it the default way to interface with native APIs is a major safety issue. C and C++ have headers for a reason and redeclaring everything by hand is not only time-consuming, but also error-prone. A little mistake in your ccall and you just happily segfaulted, and that’s an optimistic scenario. And now try to correctly wrap strerror_r or similar...So this “feature” instead of eliminating boilerplate eliminates type checking. In fact, there is more boilerplate per function in ccall than in, say, pybind11. This is another area where Python and even Java with their C API win. There is a way to abuse FFI there too, but at least it’s not actively encouraged." [1]

---

"In my opinion, in Clojure, the Java layer pokes the Clojure layer way more than it should. Another example is that Clojure require to access the Java standard library in many cases. The problem with this is that Java functions and Clojure functions are not interchangable. For instance you can do "(apply clojureFunction [0 0])". But you can not do "(apply .javaFunction [0 0])". You need to wrap the Java function into a Clojure function explicitly in this way "(apply #(.javaFunction %1 %2) [0 0])". I don’t find this particularly elegant." [2]

---

valarauca1 5 days ago [-]

This is more complex then just a target. Rust has to communicate to the Go Runtime Environment.

Two main points:

-Go uses a very small stack for GoRoutines? to make them dirt cheap. When you exceed this stack, Go maps in more stack for you transparently. Rust generated ASM is running on the Go Stack, but it is expecting when it exceeds its stack to explode, like a C program. As that should be the OS Stack. Larger problem then you think, Rust likes to put a ton of stuff on the stack. This is one of the nice things about Rust is putting _a ton_ of data on the stack is cheap, and makes ownership simpler.

-Go's system calls, and concurrency primitives are cooperative with its runtime. When make they communicate the routine can yield to the runtime. Targeted Rust code would _also_ have to make these calls, as well as 3rd party crates.

Again none of this is impossible, linker directives and FFI magic could import these functions/symbols. But this would also require Go have a stabilized runtime environment for other languages to link against. Currently just stating Go has a runtime is controversial, so I expect this won't happen soon.

reply

masklinn 5 days ago [-]

> This point confuses me; if Rust expects to run on a limited stack, why would it expect to put a ton of data on the stack?

Rust runs on a C stack, while it's not infinite[0] it's a whole other ballpark than a Go stack since it's non-growable (Rust used to use growable stacks before 1.0): the default C stack size is in the megabyte range (8MB virtual on most unices[1]), in Go the initial stack is 2kB (since 1.4, 8k before).

[0] you can set the size to "unlimited", systems will vary in their behaviour there, on my OSX it sets the stack size to 64MB, Linux apparently allows actual unlimited stack sizes but I've no idea how it actually works to provide that

[1] I think libpthread defines its own stack size circa 2MB so 8MB would be the stack of your main thread and 2MB for sub-threads, but I'm not actually sure

reply

vvanders 5 days ago [-]

Things may have changed from when I last looked at it but there's another issue which is pinning.

If you ever want to do more than the most trivial FFI you'll eventually want to be able to pass types back and forth(usually opaque to either side). AFAIK Go doesn't offer any pinning if it's GC'd types so you can have the collector move them from under you.

C# has this beautiful thing where you can pass a delegate as a raw C fn pointer. It makes building interop a wonderful thing but you have to make sure to pin/GCHandle it appropriately.

reply

FiloSottile? 5 days ago [-]

Folks, stop, you are reinventing cgo now.

Defining a Go target for Rust actually makes sense in the context of replacing assembly (which has no runtime, GC or concurrency connotations), I was just too lazy to do it that way :)

reply

valarauca1 5 days ago [-]

> Folks, stop, you are reinventing cgo now.

This is a good thing because `cgo` is really bad. No ALSR, always forking. These are completely _insane_ defaults, it manages to be slower then the JNI [1] which is an FFI from a completely managed stack based VM universe! Not a _compiled_ language.

Somehow a compile language calling a static binary manages to be slower then a dynamic language's runtime calling a static binary...

`cgo` isn't doing _anything right_. It is doing a lot of things wrong.

[1] https://lwn.net/Articles/446701/

reply

---

"

Why not cgo

Go has a Foreign Function Interface, cgo. cgo allows Go programs to call C functions in the most natural way possible—which is unfortunately not very natural at all. (I know more than I'd like to about cgo, and I can tell you it's not fun.)

By using the C ABI as lingua franca of FFIs, we can call anything from anything: Rust can compile into a library exposing the C ABI, and cgo can use that. It's awkward, but it works.

We can even use reverse-cgo to build Go into a C library and call it from random languages, like I did with Python as a stunt. (It was a stunt folks, stop taking me seriously.)

But cgo does a lot of things to enable that bit of Go naturalness it provides: it will setup a whole stack for C to live in, it makes defer calls to prepare for a panic in a Go callback... this could be will be a whole post of its own.

As a result, the performance cost of each cgo call is way too high for the use case we are thinking about—small hot functions. "

---

probably irrelevant but just putting here in case i need it later: https://blog.filippo.io/rustgo/

---

mbrubeck 14 hours ago [-]

Hi, I'm a Servo developer who worked on some of the Rust code that's in Firefox. Calls between C++ and Rust code in Firefox all go through "extern C" functions. Some of the code involves reference-counted smart pointers. We use RAII wrapper types in both Rust and C++ to ensure that refcounts are incremented and decremented correctly on either side of the FFI boundary.

P.S. This old blog post is not about Rust-in-Firefox, but it does cover a related topic: How the Servo browser engine (written in Rust) interacts with the Spidermonkey JavaScript? engine (written in C++ and embedded in both Gecko and Servo), including garbage-collected JavaScript? objects:

https://research.mozilla.org/2014/08/26/javascript-servos-on...

reply

---

kibwen 131 days ago [-]

Looking ahead to 1.19 (currently in beta, which you can try out easily via rustup), the RFC for unsafe (C-style) unions recently stabilized, which closes IMO the biggest remaining hole in Rust's FFI story. By removing the need for various manual hacks to interoperate with unions from C libraries,...

---

" int_19h 129 days ago [-]

That's exactly the problem. R has a very specific API for its extensions - it's not just vanilla C, it's stuff like SEXP.

Although now that I think more about it, it's not quite as bad as Python, because the data structures are mostly opaque pointers (at least until someone uses USE_RINTERNALS - which people do sometimes, even though they're not supposed to), so all operations have to be done via functions, where you can map them accordingly.

You'd also need to emulate R's object litetime management scheme with Rf_protect etc; but that shouldn't be too difficult, either.

Some more reading on all this:

https://cran.r-project.org/doc/manuals/r-release/R-exts.html...

http://adv-r.had.co.nz/C-interface.html

peatmoss 127 days ago [-]

Oh, yeah, now that you mention it I have seen the SEXP and protect / unprotect stuff before. ... "

---

random (good) Python<->Rust interop example:

[3]

---

C# 'span' for encapsulating memory:

https://github.com/dotnet/corefxlab/blob/master/docs/specs/span.md

---

Rust 'bindgen' apparently helps with interop, macros too:

" bindgen & macros are amazing

I wrote blog posts about these already but I want to talk about these again!

I used bindgen to generate Rust struct definitions for every Ruby struct I need to reference (across 35 different Ruby version). It was kind of… magical? Like I just pointed at some internal Ruby header files (from my local clone of the Ruby source) that I wanted to extract struct definitions from, told it the 8 structs I was interested in, and it just worked.

I think the fact that bindgen lets you interoperate so well with code written in C is really incredible.

Then I used macros (see: my first rust macro) and wrote a bunch of code that referenced those 35 different struct versions made sure that my code works properly with all of them.

And when I introduced a new Ruby version (like 2.5.0) which had internal API changes, the compiler said “hey, your old code working with the structs from Ruby 2.4 doesn’t compile now, you have to deal with that”. "

-- [4]

---

[5]

" ... you can’t just pass a string into a WebAssembly? function. Instead, you have to go through a bunch of steps:

    On the JS side, encode the string into numbers (using something like the TextEncoder API)
    Encoder ring encoding Hello into number equivalent
    Put those numbers into WebAssembly’s memory, which is basically an array of numbers
    JS putting numbers into WebAssembly's memory
    Pass the array index for the first letter of the string to the WebAssembly function
    On the WebAssembly side, use that integer as a pointer to pull out the numbers

And that’s only what’s required for strings. If you have more complex types, then you’re going to have a more convoluted process to get the data back and forth.

If you’re using a lot of WebAssembly? code, you’ll probably abstract this kind of glue code out into a library. Wouldn’t it be nice if you didn’t have to write all that glue code, though? If you could just pass complex values across the language boundary and have them magically work?

That’s what wasm-bindgen does. If you add a few annotations to your Rust code, it will automatically create the code that’s needed (on both sides) to make more complex types work. ... Under the hood, wasm-bindgen is designed to be language-independent. This means that as the tool stabilizes it should be possible to expand support for constructs in other languages, like C/C++. ...

Q. How do you package it all up for npm?

A. wasm-pack

"

---

huh, even these guys couldn't keep up with their interop:

[6]

---

 ioddly 3 days ago [-]

I'd think one of the big advantages of rewriting Lua in a GC'd language would be that you could ditch the stack API, which is necessary to keep track of pointers but not so intuitive to use. Seems like they didn't take that route though.

reply

wahern 3 days ago [-]

What's the alternative to a stack API? If you look at Python and Perl they use a combination of (1) an explicit stack similar to Lua, (2) instantiation of heavy-weight list or array objects for passing arguments, (3) explicit reference counting, and (4) code generation.

Lua's stack protocol can sometimes be verbose, but I find it infinitely more simple and elegant than the existing alternatives.

I think you tend to see long, unwieldy code blocks that invoke Lua functions multiple times because you can, despite how tricky it can be to track the top of your stack. You rarely see this in Perl or Python because it becomes impossible to manage long before that point. So the solution for Lua is the same as for other interfaces--use shorter, simpler functions and compose them. This is usually easier and more performant to do in Lua, anyhow, because there's less fixed boiler plate and indirection necessary; not to mention Lua gives you many more facilities for stashing and managing your C state across calls (like closure upvalues, userdata values, continuation context cookies, etc) without resorting to global variables either inside or outside the VM.

reply

ioddly 3 days ago [-]

Yeah, implementing a precise GC is a pain, but in Go the hard work of that is already done for you. If I were to write my own Lua-in-Go, my inclination would be to try to make an API something like this

    table := lua.MakeTable()
    table.Set(lua.MakeString("five"), lua.MakeNumber(5))

Rather than manipulating values on the stack. But it's possible there are reasons for not doing that that I'm unaware of. Was just curious.

reply

wahern 3 days ago [-]

Why do you prefer that over

  lua_pushstring(L, "five");
  lua_pushnumber(L, 5);

The above is shorter and also significantly more efficient because you're not creating the intermediate table object. Even in Go, given Lua's coroutine and tailcall semantics (assuming they were implemented) the Go compiler would likely be forced to always heap allocate the argument list table. There's a reason PUC Lua is blazingly fast, and its partly the same reason why LuaJIT? is so fast--the Lua stack protocol is a thin abstraction that easily admits a simple, efficient implementation yet still managing to provide an effective interface boundary that doesn't leak (beyond the fact of the protocol itself).

I don't think there's any good middle ground here. The best options are (1) Lua's explicit stack protocol or (2) some sort of type inference that permitted a direct call, like lua_tcall(L, "five", 5), into the VM. You can actually implement the latter in C somewhat easily (at least if you stick to simple data types) by using _Generic and __VA_ARG__ macros to generate the stack pushing code.

But I rarely see this done because it's a lot of magic for little gain. And where I have seen it done it's always been annoying because it's too leaky--invariably you have to fallback to the stack pushing code because there's a limit to the types of automagic type coercions you can accomplish between two drastically different languages. So you have to learn their magical wrapper routines in addition to the regular Lua API.

Many years ago I abused __VA_ARG__, __builtin_types_compatible_p, and libffi so I could invoke ad hoc functions as libevent callbacks without having to proxy the call through a function that unwrapped a pointer-to-void cookie. See http://25thandclement.com/~william/projects/delegate.c.html I still think it's kinda clever but I can't remember the last time I actually used it. Even in code bases already using delegate.c I ended up returning to the slightly more repetitive but transparent and idiomatic patterns for installing libevent events.

reply

omaranto 3 days ago [-]

I'm guessing the code was intended to be equivalent to:

    lua_newtable(L);
    lua_pushstring(L, "five");
    lua_pushnumber(L, 5);
    lua_settable(L, -3);

reply

wahern 3 days ago [-]

Ah, I see. So it's more a preference for a richer standard library, like,

  lua_newtable(L);
  luaL_setfieldi(L, -1, "five", 5);

rather than having to roll your own. Part of the function of the stack protocol is to minimize the number of unique functions needed to load and store values across language boundaries. So there's a single [low-level] function for communicating a number value in a type-safe manner, rather than a multitude of functions--one routine for setting a number value for a table string key, another for a number value for a table number key, another for a string value for a table number key, etc. The permutations explode, and unless the host language has some sort of generics capability then it adds significant complexity both to the interface and to implementation maintenance. (See Perl XS.) Alternatively you can reduce the number of permuted interfaces by using printf-style formatting strings and va_lists, but that's not type safe. (See Py_BuildValue?.)

Notably, Lua 5.3 removed lua_pushunsigned, lua_tounsigned, luaL_checkint, luaL_checklong, and many similar routines as there was never an end to the number of precomposed type coercing, constraint checking auxiliary routines people demanded. They decided that the only way to win that game was to not play at all. But with C11 _Generic and a dash of compiler extension magic you can actually implement a richer set of type agnostic (but type safe) numeric load/store interfaces in just a handful of auxiliary routines. For better or worse, though, Lua doesn't target C11 nor wish to depend on compiler extensions. "Batteries not included" is a whole 'nother debate in the Lua universe.

reply

as-j 2 days ago [-]

> Lua's stack protocol can sometimes be verbose, but I find it infinitely more simple and elegant than the existing alternatives.

This. TL;DNR, used a C/Lua binding to explain the Lua stack to a fellow engineer. Took a few post-it notes to keep track of 4 levels of stack, 1 pcall back to Lua to sort an array, and forward the results on. It just worked, was elegant, and blew our minds.

Today on Friday at 5pm I was talking about a ‘problem’ we had with a Lua daemons compared to their C brothers. Due to the use of tables, each time you queried them, the output order was always random. Computers don’t care, but a big user is also engineers and finding fields wandering all over the place is a pita. So what if we ordered the keys alphabetically instead? The code is Lua -> C -> IPC -> Output. The order is set in the Lua->C binding, so what if we sorted it in the binding.

But the tables are arbitrary size and you’d have to allocate memory, put them in a structure of some kind, etc, etc. Then we realized we could just put all the keys in a Lua array, and sort them with table.sort(). We could then use the array to get key/values in order, all from within the C binding.

Being late Friday, and having talked it through let’s just do it. Plus it was a handy teaching moment to explain to a fellow engineer how Lua binding worked, including a call back into Lua. By 6pm we had the binding recompiled, tested and it’s just worked.

Lessons learned:

1. The stack can seem confusing, but once you learn how to work with it, it’s pretty amazing.

2. The Lua reference docs are amazing. They clearly list what they pop and what they push onto the stack.

3. Calling back into Lua is.

4. It looks like write only code. Reading the code needs a post it note to keep track of the stack, along with the Lua reference manual. -- comments...

reply

---

"Thanks to not having a runtime and being quite easy to create a shared library with a C-ABI, pretty much any software written in any language could be extended with code written in Rust. "

---

" We’ve defined a set of levels to encourage implementors to create tighter integrations with the JavaScript? runtime:

    Level 1: Just string output, so it’s useful as a basic console REPL (read-eval-print-loop).
    Level 2: Converts basic data types (numbers, strings, arrays and objects) to and from JavaScript.
    Level 3: Sharing of class instances (objects with methods) between the guest language and JavaScript.  This allows for Web API access.
    Level 4: Sharing of data science related types  (n-dimensional arrays and data frames) between the guest language and JavaScript." -- [7]

---

" LSP host and Jupyter kernel. Between the two you can support every development environment and awesome tooling in O(1) effort. "

note: most of the infrastructure of Jupyter can be gained for free by using Python to call out to your language via pyexpect or via Python bindings (both of these methods are suggested in the official Jupyter docs)

---

like Lua, have an interop stack

---

mhh__ 4 hours ago [-]

D's C++ interop looks better at a glance.

With a simple extern(C++) you can use templates, classes and vtable's are matched up to single inheritance. There is also some experimental work on catching C++ exceptions but I've never tried to use it.

reply

p0nce 2 hours ago [-]

Also: COM support. This can be incredibly useful for interop.

reply

---

https://github.com/pybind/pybind11

---

https://github.com/metacall/core

METACALL is a library that allows calling functions, methods or procedures between programming languages. With METACALL you can transparently execute code from / to any programming language, for example, call Python code from JavaScript? code.

sum.py

def sum(a, b): return a + b

main.js

metacall_load_from_file('py', [ 'sum.py' ]);

metacall('sum', 3, 4); 7

    Currently supported languages and run-times:

Language Runtime Version Tag Python Python C API >= 3.2 <= 3.7 py NodeJS? N API >= 8.11.1 <= 10.15.3 node JavaScript? V8 5.1.117 js C# NetCore? 1.1.10 cs Ruby Ruby C API >= 2.1 <= 2.3 rb Mock ∅ 0.1.0 mock

    Languages and run-times under construction:

Language Runtime Tag Java JNI java C/C++ Clang - LLVM - libffi c File ∅ file Go Go Runtime go Haskell Haskell FFI hs JavaScript? SpiderMonkey? jsm WebAssembly? WebAssembly? Virtual Machine wasm

---

" METACALL maintains most of the types of the languages but not all are supported. If new types are added they have to be implemented in the reflect module and also in the loaders and serials to fully support it. Type Value Boolean true or false Char -128 to 127 Short -32,768 to 32,767 Int -2,147,483,648 to 2,147,483,647 Long –9,223,372,036,854,775,808 to 9,223,372,036,854,775,807 Float 1.2E-38 to 3.4E+38 Double 2.3E-308 to 1.7E+308 String NULL terminated list of characters Buffer Blob of memory representing a binary data Array Arrangement of values of any type Map List of elements formed by a key (String) value (Any) pair (Array) Pointer Low level representation of a memory reference Null Representation of NULL value type "

" 5.1.2 Modules

    adt provides a base for Abstract Data Types and algorithms used in METACALL. Implementation must be done in an efficient and generic way. Some of the data structures implemented are vector, set, hash, comparable or trie.
    detour provides an interface to hook into functions. Detours are used by the fork model to intercept fork calls.
    detours implement the detour interface by using a plugin architecture. The current list of available detour plugins is the following one.
        funchook_detour implemented by means of FuncHook library.
    distributable defines the compilation of METACALL that generates an unique library with all core libraries bundled into it. As the METACALL architecture is divided by modules, in order to distribute METACALL is needed to build all of them into a single library. This module implements this compilation by means of CMake.
    dynlink implements a cross-platform method to dynamically load libraries. It is used to dynamically load plugins into METACALL.
    environment implements an standard way to deal with environment variables. METACALL uses environment variables to define custom paths for plugins and scripts.
    examples ...
    filesystem provides an abstraction for operative system file system.
    format provides an standard way for printing to standard input output for old C versions that does not support newest constructions.
    loader ...
    loaders
    log
    memory
    metacall
    ports
    preprocessor
    reflect
    scripts
    serial
    serials
    tests
    version

"

" 5.7 Fork Model

METACALL implements a fork safe model. This means if METACALL is running in any program instance, the process where is running can be forked safely at any moment of the execution. This fact has many implications at design, implementation and use level. But the whole METACALL architecture tries to remove all responsibility from the developer and make this transparent.

... "

" Because of these restrictions, METACALL cannot preserve the status of the run-times. In the future this model will be improved to maintain consistency and preserve the execution state of the run-times making METACALL more robust.

Although the state is not preserved, fork safety is. The mechanism METACALL uses to allow fork safety is described in the following enumeration.

    Intercept fork call done by the program where METACALL is running.
    Shutdown all run-times by means of unloading all loaders.
    Execute the real fork function.
    Restore all run-times by means of reloading all loaders.
    Execute user defined fork callback if any.

To achieve this, METACALL hooks fork primitives depending on the platform.

    fork on POSIX systems.
    RtlCloneUserProcess on Windows systems.

If you use clone instead of fork to spawn a new process in a POSIX system, METACALL won't catch it.

Whenever you call a to a cloning primitive METACALL intercepts it by means of detour. Detours is a way to intercept functions at low level by editing the memory and introducing a jump over your own function preserving the address of the old one. METACALL uses this method instead of POSIX pthread_atfork for three main reasons.

    The first one is that pthread_atfork is only supported by POSIX systems. So it is not a good solution because of the philosophy of METACALL is to be as cross-platform as possible.

...

the developer can register a callback by means of metacall_fork to know when a fork is executed to do the actions needed after the fork, for example, re-loading all previous code and restore the state of the run-times. This gives a partial solution to the problem of losing the state when doing a fork. "

rwmj 2 hours ago [-]

How does it differ from libffi or Swig?

reply

HerrMonnezza? 52 minutes ago [-]

From a cursory reading of the README, I would say that Metacall allows any of the supported languages to call any other one, whereas Swig only allows calling C or C++ code. (Swig takes a `.h` file as input and then generates the bindings for the desired target language.)

reply

---

in ocaml, "However when the garbage collector looks at data stored in generic structures it needs to tell pointers from integers, so integers are tagged using a 1 bit in a place where valid aligned pointers never have a 1 bit, leaving only 31 or 63 bits of range.". how to deal with this in Boot?

---

"To understand the METACALL fork model, first of all we have to understand the implications of the forking model in operative systems and the difference between fork-one and fork-all models. The main difference between fork-one and fork-all is that in fork-one only the thread which called the fork is preserved after the fork (i.e. gets cloned). In fork-all model, all threads are preserved after cloning. POSIX uses fork-one model, meanwhile Oracle Solaris use the fork-all model. Because of fork-one model, forking a running run-time like NodeJS? (which has a thread pool) implies that in the child process the thread pool will be almost dead except the thread which did the fork call. So NodeJS? run-time cannot continue the execution anymore and the event-loop enters into a deadlock state. When a fork is done, the status of the execution is lost by the moment. METACALL is not able to preserve the state when a fork is done. Some run-times do not allow to preserve the internal state. For example, the bad design of NodeJS? does not allow to manage the thread pool from outside, so it cannot be preserved after a fork. "

---

bb88 2 hours ago [-]

Interesting nugget about golang from their FAQ:

https://github.com/canonical/dqlite/blob/master/doc/faq.md

Why C?

The first prototype implementation of dqlite was in Go, leveraging the hashicorp/raft implementation of the Raft algorithm. The project was later rewritten entirely in C because of performance problems due to the way Go interoperates with C: Go considers a function call into C that lasts more than ~20 microseconds as a blocking system call, in that case it will put the goroutine running that C call in waiting queue and resuming it will effectively cause a context switch, degrading performance (since there were a lot of them happening). See also this issue in the Go bug tracker.

The added benefit of the rewrite in C is that it's now easy to embed dqlite into project written in effectively any language, since all major languages have provisions to create C bindings.

reply

---

[8]

---

https://blog.rust-lang.org/inside-rust/2020/06/08/new-inline-asm.html https://news.ycombinator.com/item?id=23466795

---

andrewmcwatters 11 hours ago [–]

If you use LuaJIT?, you can generate bindings to C libraries, versus handwriting bindings for PUC Lua. The productivity difference is staggering. Additions to Lua since 5.1.5 have not helped me write more or better software, as much as I love Lua.

reply

---

"

XPCOM is a technology that lets you write code in two languages and have each other call the other. The code of Firefox is full of C++ calling JavaScript?, JavaScript? calling C++ and a long time ago, we had projects that added Python and .Net in the mix. This piece of machinery is extremely complicated because languages do not share the same definitions (what’s a 64-bit integer in JavaScript?? what’s a JavaScript? exception in C++?) or the same memory model (how do you handle a JavaScript? object holding a reference to a C++ object that C++ might wish to delete from memory?) or the same concurrency model (JavaScript? workers share nothing while C++ threads share everything).

Gecko itself was originally designed as thousands of XPCOM components that could each be implemented in C++ or in JavaScript?, tested individually, plugged, unplugged or replaced dynamically and it worked. In addition, the XPCOM architecture made for much cleaner C++ programming than was available at the time, worked on dozens of platforms, and let us combine the convenience of writing code in JavaScript? and the raw speed permitted by C++. "

---

IRIS interop:

" Rich native-language interface metadata and clean decoupling of underlying primitive functions should enable partial/full transpilation of native iris code to optimized Swift code, e.g. eliminating redundant native➞primitive➞native bridging coercions so that Swift functions can pass Swift values directly.

e.g. Consider the expression:

“HELLO, ” & uppercase my_name

Handler definitions:

to uppercase {text as string} returning string requires { }

to ‘&’ {left as string, right as string} returning string requires { can_error: true swift_function: joinValues operator: {form: #infix, precedence: 340} }

Swift functions:

func uppercase(text: String) -> String { return text.uppercased() }

func joinValues(left: String, right: String) throws -> String { return left + right }

Generated Swift code, obviating the unnecessary String➞Text➞String? bridging coercions between the uppercase and & (concatenation) commands:

let native_value = try env.get("my_name") let swift_value = try asString.coerce(native_value, in: env) let swift_result = try joinValues(left: "HELLO, ", right: uppercase(text: swift_value)) let native_result = asString.wrap(swift_result, in: env)

" -- https://github.com/hhas/iris-script

---

" matklad May 15

One super rough idea I am thinking about recently is that maybe we don‘t need „Rust“ ABI, but „System ABI“? Today, the interop language of software components is C ABI, and it is wholly inadequate: it doesn’t even have slices, strings are slow and error-prone, etc.

It seems like developing language-independent ABI which significantly improves over C, without being Rust specific, is possible. Slices, tagged unions, utf8 strings, borrowed closures are features which immediately come to mind and have obvious-ish implementations.

Distinction between borrowed and owned (callee must invoke destructor) data and simple lifetimes are somewhat more Rust/C++ specific, but don’t seem too controversial either.

Support for dynamic dispatch (where fat vs thin pointer is a big tradeoff) and templates seems pretty hard, but also desirable and plausible.

It seems like if we had this „C ABI, but you also can have nice things“ , than interoperability between various software components would be easier and far less error prone.

...

kornel 1 May 15

Generics are not a problem as long as all types are known in the interface exposed through the ABI. So it wouldn't be a problem to return Result<i32, String> or even Result<(), Box<dyn Error>> (assuming trait objects have defined ABI).

They're only an issue for functions like pub fn generic<T, E>() -> Result<T, E> that can't be monomorphized on the library side. For these functions the options are:

    Just forbid them. Library interfaces would have to use dyn Trait or concrete types instead. IMHO this is quite sensible limitation, especially for an MVP.
    Require defining ahead of time which parameters can be used, and compile monomorphic versions just for these types (similar to template instantiations in C++)
    Do what Swift does and compile a universal version of the generic function that uses run time type information to support arbitrary types. It's a very clever approach from ABI perspective, but it's equivalent of changing everything into dyn Trait, so it may be a poor fit for Rust.

...

CAD97 Regular May 15

    It seems like developing language-independent ABI which significantly improves over C, without being Rust specific, is possible.

The Swift ABI isn't specific to Swift, and is an impressive feat of engineering that could fill this slot.

Witness table indirection obviously adds some overhead over static linking or "just repr(Rust)" linking, but it successfully maintains the ability to evolve private implementation details, and it's a necessary cost to do so. (Also, it supports freezing the ABI to remove (most) witness table overhead.)

The one thing I don't recall off the top of my head is whether it requires any shared heap object to be managed by Swift atomic reference counting. I think it just uses opaque "clone pointers", though.

"Add repr(Swift) to Rust" is an interesting research topic independent of "Add user-defined repr to Rust".

But, yes, an initial MVP would handle concretized generics fine as "just another type" and punt on unconcrete generics, as you can "just" expose a dyn API yourself. Tom-Phinney May 15

Since the Swift ABI would be included in any reasonable set of modular Rust ABIs, it probably makes sense to subset the modular ABI task to first just develop the Swift ABI, noting during the development process those places where additional effort would be needed to generalize to additional ABIs beyond the obvious three: Rust's native unstable/unspecified ABI, the stable# C ABI, and the new stable# Swift ABI.

  1. Of course those externally-specified ABIs are subject to change through their own language specification maintenance processes. zicklag May 15
    Since the Swift ABI would be included in any reasonable set of modular Rust ABIs, it probably makes sense to subset the modular ABI task to first just develop the Swift ABI,

That saves us the development of the proposal, which would be large in-and-of-itself, for a „System ABI“. I think that makes a lot of sense.

So the goal for this proposal would be to create a modular ABI system that could support a Swift ABI crate for Rust.

:+1: CAD97 Regular 1 Tom-Phinney May 15

The problem with that is that the Swift ABI is much more complicated than the Rust or C ABIs, as it has e.g. different behavior inside the defining compilation static linking and things using it via dynamic linking.

And another thing is that IIRC, the Swift ABI requires alloca, which Rust doesn't have yet (and is decently low priority IIUC).

But if a modular ABI is powerful enough to support the Swift ABI, it's probably good enough to support basically any feasible ABI. And it would be nice if the definition of repr(Rust) and/or repr(C) could be separate from rustc, I suppose.

Personally, I'd target the MVP at being able to specify repr(C), stage 2 at (most of?) repr(Rust) (~niche filling), and stage 3 at the (most of?) nongeneric subset of repr(Swift) (witness tables outside the main static link). isaac May 15

    IIRC, the Swift ABI requires alloca,

IIRC, in some situations where alloca isn't appropriate, another technique is used.

...

vomlehn May 18

There are lots of good ideas here, with the possibilities of multiple ABIs, one per language. But, as someone who was deeply involved in the C/C++ ABI for the MIPS processor, one thing that people seem to be missing is that there are NxMxTBD? ABIs. One per processor (and possibly more if you consider little-endian vs. big-endian, small vs. medium vs. large memory models etc), one per language, and maybe more (the TBD). So, tossing around the term "ABI" hides a world of work. And you need to consider ABIs you're not working on to preclude conflicts with the ones you are.

People touched upon the various things Swift supports; the general question arises about support for things which don't have a Rust equivalent. FFIs work from language A to language B and the reverse. My initial reaction is, if Rust doesn't support it, the ABI doesn't support it. This is just a matter of trying to manage the scope of such a project. This doesn't preclude adding in support later, it just makes things more achievable in the near term.

Having said that, it's an awesome idea. The benefits are large. It allows third-party Rust compilers to emerge which may be highly optimized, and hence useful, for specific architectures, much as using an Intel C compiler may give you better performance in generated code than LLVM or GCC. This means slightly less support for the open source language but significantly more support for Rust.

...

zicklag 1 May 18

    In my experience, it is solved by having a general-purpose communication channel that is flexible enough to not require changes going forward. On top of that, the actual interaction is negotiated using capabilities on one or both sides.

This sounds a little like Cap'nProto 6 to me, which could actually be an option for implementing the plugin protocol. I'm not sure if building the plugins into the compiler through macros might be a better idea, though. That would probably make distributing the ABI plugins as crates easier. ( This doesn't preclude us from using a capabilities based ABI plugin implementation model )

Also, WASM is another idea. Something that has already shown promise as a way to compile and use procedural macros, with a successful POC I think ( there's a post around here somewhere about it ). CAD97 Regular May 18

    WASM [...] procedural macros [...] POC

watt 30 https://lib.rs/crates/watt

HerrJemine? May 19

I think this is a great idea and I hope it succeeds. But I think it is important to clarify the goals of this modular ABI project. I reread this article 6 https://gankra.github.io/blah/rust-layouts-and-abis/ and I think it provides a great overview of the Rust ABI problem space. This 4 https://gankra.github.io/blah/swift-abi/ is also very relevant.

The questions I have:

    Does the modular ABI deal only with type layout or also with calling conventions?
    Is it handled as annotation per struct similar to #[repr(C)] or is it a global compiler/cargo option or a singleton similar to global_allocator? Making repr(C) just another ABI module would be very elegant, but it limits the usability if every struct has to be annotated, especially in dependencies.
    If the ABI is applied globally, does it apply to all types in the crate or only the public ones? What about re-exported types from dependencies?
    How are different targets handled? Is there one ABI module per target?" -- https://internals.rust-lang.org/t/a-stable-modular-abi-for-rust/12347/13

i just realized i'm only on comment 28/122 in that thread! i don't have time to go thru the rest now. You can pick up where i left off.

https://internals.rust-lang.org/t/a-stable-modular-abi-for-rust/12347/24

---

" Scala is completely interoperable with Java (and with some qualifications also to C#). A Scala component can: • access all methods and fields of a Java component, • create instances of Java classes, • inherit from Java classes and implement Java interfaces, • be itself instantiated and called from a Java component. "

---

"

Second, while we generally find Windows platform bindings to be straightforward, thanks to the excellent winapi bindings (and hopefully a stable com-rs shortly), the situation on macOS is not as good. There are real problems interoperating with Objective-C code, including type-safety in wrappers that also allow subclassing, and continual friction around autorelease pools. There are also different, incompatible conventions around wrapping foreign types (foreign_object vs TCFType). None of this is blocking, but it makes developing macOS platform integration less fun. (See this HN thread for a bit more background)

https://github.com/microsoft/com-rs https://news.ycombinator.com/item?id=24309565 "

---

samatman 1 day ago [–]

This is coming from an outside perspective, and I'm quite prepared to be badly wrong.

I think the Rust language needs to learn a few tricks from Objective C and Swift to really make this possible. In particular the work being done on extensible protocols.

What you can do, is build a 'Rust-flavored' GUI library. But this isn't what users want: they want a native app, while developers want to write once-run anywhere.

This requires a kind of double decoupling, where UI intent is expressed as an abstraction, and is realized as an OS-native control — but one which itself will dynamically update, insofar as possible, as the OS evolves.

This calls for a kind of API wizardry which Objective C always got right, and C++ never really did. I see that dichotomy extending into Swift and Rust, but it isn't inevitable, Rust hasn't painted itself into that particular corner.

Anyway, it's encouraging to see this work happening, I wish you good fortune in 2021 and beyond.

reply

---

"What made it really easy for me to port from Objective-C to Swift in the past is that I could replace one single Objective-C method at a time with a Swift version, compile and see that everything still worked."

---

" What ruins a lot of C-like languages is that they don’t play nice with C. By that I mean it should be easy to call C functions from the language and it should be easy to call function on the language from C.

In addition the general way you program should be quite C-compatible so you don’t have to create a large C-abstraction level. E.g. C++ isn’t very C friendly, because a typical C++ library cannot be used from C without extensive wrapping.

Zig however if very C friendly because there are no oddball stuff exposed that C doesn’t get. There are no vtables (table to virtual functions in C++) in structs. No constructors or destructors that C has no clue about how to call. Nor are there any exceptions that C also would struggle with catching. "

---

https://kotlinlang.org/docs/reference/mpp-connect-to-apis.html

---

python<->rust https://github.com/PyO3/pyo3

---

a feature that Swift added to allow it to interoperate with dynamic language such as Python, Javascript, Ruby:

https://github.com/apple/swift-evolution/blob/master/proposals/0195-dynamic-member-lookup.md

---

a class that Swift added to interoperate with Python:

https://www.tensorflow.org/swift/api_docs/Structs/PythonObject

---

Transpile

https://engineering.mongodb.com/post/transpiling-between-any-programming-languages-part-1

https://news.ycombinator.com/item?id=22325169

---

https://news.ycombinator.com/item?id=27634456

---

should be able to be used to write plugins for things like Obsidian, which are in Javascript (typescript, with typescript spec: https://github.com/obsidianmd/obsidian-api/blob/master/obsidian.d.ts ). Should compile to WASM.

like this does for Rust: https://github.com/trashhalo/obsidian-rust-plugin

"App Store allows external code if it strictly runs under WebKit? and JavaScript? Core. There's a few more general restrictions (like the additional functionality must be free or use IAP)." [9]

i don't really understand if this means external WASM modules will work well? If so, we should test under that environment.

---

i don't understand this comment, in https://lobste.rs/s/a9ghhz/maintain_it_with_zig:

jackdk 8 days ago

link

I believe that “wrap the compiler” is the wrong approach, because what happens if you need to wrap the compiler twice? This can happen if you want to do FFI between two higher-level languages.

My gut instinct has been to recommend that tools provide information in a way that existing tooling can consume, such as .pc files to include FFI headers/RTS libs, and maybe a dependency tool that emits Makefile-format information. Is this wrong? I have only been on the user side of these discussions, never the toolchain developer.

---

old Fantom post on targeting LLVM, interop:

[10]

---

another random language's interop:

https://github.com/rochus-keller/Oberon/releases/tag/FFI_MVP

---

" My roommate had an assembler program for the computer, along with a manual. I pored over them and because BASIC has such a paucity of features, it was not too difficult to convert the lexer to assembler, though it took me a few days.

But I had to call this routine from BASIC. That’s an odd combination of CLEAR (reserve memory), DATA (define your program), POKE (put your program in memory), DEFUSR... (define the user routine used to call your program) and USR... (call the user routine) commands. Here’s a small program to print out the ASCII value of whatever character the user typed (from the user manual ) (warning: pdf): " [11]

---

https://zig.news/ranciere/zoltan-a-minimalist-lua-binding-4mp4

---

Type matching

The type set of Lua is kept to a minimum: bool, integer, number, string, table, function, userdata. The following table contains the matching between Lua and Zig: Lua Zig string [] const u8 integer i16, i32, i64 number f32, f64 boolean bool table Lua.Table function Lua.Function userdata Lua.Ref

The instances of registered user types become Lua userdata with appropriate metatable.

---

https://ahgamut.github.io/2022/07/27/ape-rust-example/ https://news.ycombinator.com/item?id=32260619

---

"... I was working on a profiling agent lately, and one of the issues was running it on multiple platforms (just the four big ones linux/mac-x86/arm) on FFI (because it'll be run directly from python/ruby/etc...) and preferably having the thing just work without having to install or configure any dependencies." -- [12]

---

" To solve this, Vale has Fearless FFI which decouples and isolates unsafe C data from safe Vale data.

    Separate the safe memory from the unsafe memory (such as the memory managed by C). This includes:
        Not allowing safe objects to contain unsafe objects.
        Not allowing unsafe objects to contain safe objects.
        Using a different stack for the unsafe code.
    Allowing references between the two:
        A safe object can contain a reference to an unsafe object.
        An unsafe object can contain a reference to a safe object, and it's automatically scrambled.
    Enable passing memory between the two by copying, also known as message passing.

See Fearless FFI for more on this! " -- [13]

---

~ david_chisnall 13 hours ago (unread)

link flag

Pointer size is fixed in LLVM IR, so once you’ve lowered to LLVM you cannot move between systems with different pointer sizes. The size of C types such as long are also fixed in LLVM to known bit widths, so anything that depends on the platform ABI (which is almost always defined in terms of C types) is non-portable. The different back ends have different conventions for lowering calls. For example, on FreeBSD? or Darwin on i386 a union of a pointer and an integer that is returned from a function will be lowered to an i32 return, whereas on Linux it will be a void function that takes an i32* as an sret parameter. Anything involving structure layouts will be lowered to a fixed memory layout embedding the target ABI’s rules.

---

2 animatronic 2 months ago

link

Take a look at https://hpyproject.org/

The Python C API is trash and is being massively cleaned up to be more Lua like.

Discerning hackers use the cffi.

---

wg0 8 hours ago

prev next [–]

While the community is here, anyone has embedded pypy as scriptable language for some larger program? Like Inkscape or scripting as part of a rule engine. Or for that, CPython is more suitable?

reply

mattip 8 hours ago

parent next [–]

It is much easier to embed CPython, PyPy? can only be embedded via CFFI [0].

[0] https://cffi.readthedocs.io/en/latest/embedding.html

reply

---

the CPython "limited" API:

https://peps.python.org/pep-0384/ https://docs.python.org/3/c-api/stable.html#stable

other notes on issues with the CPython C API:

https://pythoncapi.readthedocs.io/bad_api.html

more details: https://pythoncapi.readthedocs.io/ https://pythoncapi.readthedocs.io/status.html https://peps.python.org/pep-0620/ https://github.com/hpyproject/hpy/wiki/c-api-next-level-manifesto https://mail.python.org/archives/list/python-dev@python.org/thread/DN6JAK62ZXZUXQK4MTGYOFEC67XFQYI5/ https://docs.google.com/document/d/1lrvx-ujHOCuiuqH71L1-nBQFHreI8jsXC966AFu9Mqc/edit?pli=1 https://pythoncapi-compat.readthedocs.io/en/latest/api.html https://vstinner.github.io/c-api-abstract-pyobject.html https://vstinner.github.io/c-api-opaque-structures.html https://mail.python.org/archives/list/python-dev@python.org/thread/HKM774XKU7DPJNLUTYHUB5U6VR6EQMJF/#TKHNENOXP6H34E73XGFOL2KKXSM4Z6T2 https://pythoncapi.readthedocs.io/cffi.html https://pythoncapi.readthedocs.io/type_object.html

---

PyPy?-CPython C extension API interop:

[14]

---

"Python's Buffer Protocol: The #1 Reason Python Is The Fastest Growing Programming Language Today

The buffer protocol was (and still is) an extremely low-level API for direct manipulation of memory buffers by other libraries. These are buffers created and used by the interpreter to store certain types of data (initially, primarily "array-like" structures where the type and size of data was known ahead of time) in contiguous memory.

The primary motivation for providing such an API is to eliminate the need to copy data when only reading, clarify ownership semantics of the buffer, and to store the data in contiguous memory (even in the case of multi-dimensional data structures), where read access is extremely fast. Those "other libraries" that would make use of the API would almost certainly be written in C and highly performance sensitive. The new protocol meant that if I create a NumPy? array of ints, other libraries can directly access the underlying memory buffer rather than requiring indirection or, worse, copying of that data before it can be used.

And now to bring this extended trip down memory lane full-circle, a question: what type of programmer would greatly benefit from fast, zero-copy memory access to large amounts of data?

Why, a Data Scientist of course. How We Got Here From There

So now we see the whole picture:

    Oliphant and Banks propose a revision of Python's buffer protocol to simplify the direct access of the underlying memory of certain data structures, driven by work on the fledgling NumPy project.
    PEP 3118 is submitted, accepted, and implemented.
    By virtue of PEP 3118's implementation, Python has quietly become an incredibly compelling language on which to build numerical computing libraries given that C extensions can share and manipulate data with very little overhead"

---

"Rust’s async is fine...Ability to use futures with a custom executor makes it possible to use them, efficiently, on top of other languages’ runtimes," -- https://lobste.rs/s/cryfiu/async_rust_is_bad_language#c_quwnov

BayleShanks RecentChanges Bayle Shanks's website
Comments on proj-oot-ootInteropNotes2 This page is read-only View other revisions Administration
Last edited 2023-12-11 03:15 UTC by BayleShanks (diff)