Note: anytime you see 🙋♂️ RFC, that’s a “Request For Comments” about a topic I didn’t understand or take the time to look into. Please feel free to add what you know!
If this post tickles your fancy, check out the follow-up post: Writing a Rust gem from scratch
In December 2021, Ian Ker-Seymer (@ianks) submitted a pull request to enable native extensions in Rust!
I was so excited, I had to try it out, even though it hadn’t been merged yet. A lot of maintainers are showing interest and pitching in, so I have high hopes for it being merged into main. So here are my notes on writing a Rust gem extension.
Requirements
Requirements / dependencies / utilities I used and their versions on macOS Monterey v12.1 (21C52) as of 2022-01-29:
- Bundler version 2.4.0.dev
- gem version 3.4.0.dev
- cargo 1.58.0 (f01b232bc 2022-01-19)
- rustc 1.58.1 (db9d1b20b 2022-01-20)
Using a rubygems fork
Warning! The following is based on ianks’s development branch of Rubygems. The feature may have changed – or not exist at all – by the time you read this. I’ll modify this warning if the feature ends up being merged.
Find somewhere cozy to clone @ianks’s cargo-builder branch of rubygems and run the following:
Aliasing your default rubygems
We need to be able to use the gem and bundle commands from the cargo-builder branch of rubygems. As per the directions in the CONTRIBUTING.md:
To run commands like
gem installfrom the repo:ruby -Ilib bin/gem installTo run commands like
bundle installfrom the repo:ruby bundler/spec/support/bundle.rb install
But this is a hassle, so we’ll use aliases instead of typing all of this up every time. cd into your rubygems directory and alias these commands with the following:
We’re going to use the RUBYGEMS_PATH variable later on, so keep that handy! Now if you check the version numbers of your default gems, they should be as follows:
Note that these aliases won’t be present in a new terminal shell!
Compiling an example gem
We’re ready to test the functionality of a Rust-based gem. For starters, let’s use the rust_ruby_example gem that I’ve extracted from @ianks’s pull request:
Let’s confirm that it does, indeed, allow us to run Rust code from Ruby.
First, we need to build the gem. We do this by pointing the gem command at a .gemspec file. Luckily, the repo has one of those:
We also explicitly name the output file, otherwise we get something like rust_ruby_example-0.1.0.gem, which is just a tad bit more awkward.
And we’re done!
…well, not exactly. As it turns out, extensions aren’t compiled until you install the gem. It makes sense that building the gem and installing the gem are two separate steps. So next we need to install it:
Firing up cargo took a minute or so on my machine.
Potential errors
Speaking of cargo, if you don’t have it installed, you may see a message that looks like this:
This means that you don’t have cargo installed, or rubygems couldn’t find cargo in your $PATH. Make sure to install Rust and come back when you’re done!
You may also see an error like this:
The key here being the message "No builder for extension 'Cargo.toml'." If that’s the case, double check your bundler --version and gem --version to make sure they match the versions above. Your current version of the gem utility is missing @ianks’s CargoBuilder addition.
Inspecting rust_ruby_example code
rust_ruby_example includes some sample code in src/lib.rs:
If you’ve never seen Ruby internal code before, a few of these methods look like exactly what you’d call in C code, courtesy of a library called rb-sys. The key here is in the name of the method – pub_reverse reverses strings. Here’s where the reversal actually happens:
There’s also an initialization function, Init_rust_ruby_example, to actually define the Ruby modules and methods. Let’s piece together what it’s doing. Here are the relevant lines for declaring a Ruby module:
…and the rest is all adding the reverse method to the module:
Note that it needs to translate everything into Matz’s Ruby compatible data structures. That includes the module, the module function, and even the string name for the function.
(🙋♂️ RFC: what is the purpose of std::mem::transmute here?)
💡 Click to read more about the purpose of std::mem::transmute
From a comment I wrote on the follow up post:
After staring at the C header file where rb_define_module_function is defined – I don’t know C 😰 – I think it’s necessary because Rust won’t let you pass a function pointer with arbitrary arity, but the C code just assumes that you can. Note that the last argument in rb_define_module_function is an arity indicator. So the transmutation is just ceremony to get a function pointer – any function pointer – past Rust’s type system. That’s my guess, anyway.
EDIT – Seems right! https://twitter.com/_ianks/status/1489419634168184834
Trying out rust_ruby_example
Alright, if you’ve seen this message:
…you’re ready to go! Fire up IRB and require rust_ruby_example to take it for a test drive:
It reverses the string, as promised. It works!
…or does it? Let’s see if it’s really doing our bidding by modifying the code.
Adding a lowercase method
Let’s add a RustRubyExample#lowercase method. It will be exactly the same as RustRubyExample#reverse, except it converts case-convertible text to lower case.
It should work like this:
And we can confirm that it currently does not work:
So let’s add it. Once again we need #[no_mangle] to tell the compiler not to alter the name of the function once it’s been compiled. Mangling essentially namespaces function names so there are no name collisions in the final binary. However, in our case, we want to be able to refer to it by the name we give it in C Ruby, so we don’t want our function name to be mangled
Add this block of code between the pub_reverse and Init_rust_ruby_example functions in src/lib.rs:
We’re also going to copy the function signature:
(🙋♂️ RFC: why does this need the _klass argument?)
💡 Click to read more about _klass argument
After figuring out the purpose of the call to std::mem::transmute, the _klass argument isn’t too confusing. It has a leading underscore because it’s unused, but the type of functions with that arity on the C side requires a receiver object for the method, even if it goes unused.
Next we take the Ruby VALUE input and cast it to a Rust string, then lowercase it using standard Rust String methods:
…and the rest is all glue code to convert it to a C string and then to a Ruby string:
We also need to add the method to the Ruby module. We can do that by duplicating the relevant code in Init_rust_ruby_example:
Now to build and reinstall it:
Finally, let’s test our new functionality:
Awesome!
Conclusion
This is all possible due to Ian’s long, hard slog to get this into rubygems proper: https://github.com/rubygems/rubygems/pull/5175. This could be a hugely impactful addition to rubygems that’s been stewing since 2019 and it’s mostly been Ian’s efforts to get it there. Thanks, Ian!
If you got this far, check out the follow-up post: Writing a Rust gem from scratch
Leave a Reply