3
$\begingroup$

I think there are languages where most of the standard library is written in themselves, however with most things marked as native or external and imported from a low-level library or module. These things would usually be functions, not variables; though maybe variable declarations can help on describing memory layout, but still maybe dangerous, I suppose.

Suppose you have a class TextMatch that can be inherited, in Java-like terms (where a single class is inherited and multiple protocols (interfaces) implemented). I think that for this to be possible, a super() call would have to take a pre-allocated object and all of the subtype's fields would come after the size of the base memory of TextMatch. For example, given this Java:

class C1 {
    long x;
}
class C2 extends C1 {
    long y;
}

The base structures for these two types (not the garbage-collected counterpart) could be, in Rust:

#[repr(C)]
struct C1Base {
    x: i64;
}
#[repr(C)]
struct C2Base {
    c1: C1Base;
    y: i64;
}

I guessed of using some decorator in each native non-final class to express its base memory size, but I find it a bit dangerous:

(After that example follows an explanation of memorySize.)

package;
[FFI(memorySize = n)]
public class TextMatch {
    // fooProperty is defined from Rust
    public native function get fooProperty(): FooType;
    /* etc ... */
}

memorySize indicates the byte payload for TextMatch fields as implemented in Rust. It's not necessarily what will be allocated for the instance object: that depends on the subtype being directly constructed. thus, calling new TextMatch() would be similiar to malloc(memorySize) and a call to the constructor code, however malloc() would be a garbage-collected object.

Calling a new SubtypeTextMatch() would construct an object with more memory, where its fields follow the byte payload specified by memorySize.

That will require that, whenever I add or remove new fields to C1Base from Rust, then I've to update the FFI decorator too for that supertype C1.

I've thought of avoiding this by separating all fields of that supertype into another heap object, exactly like Box<C1Base> from Rust:

Box<C1Base>

One thing I thought of too is that I can separate the base into another heap object (a pointer), thus allowing the base to be a constant memory size.

So it'd look like:

#[repr(C)]
struct C2Base {
    c1: usize;
    y: i64;
}

It's worthy remembering that pointers are platform-dependent, so it's best to use u64 and cast it to usize as needed, I suppose.

That said, the payload wouldn't change if I take this approach for every supertype. So it's a possible solution in my opinion, but I still am not sure if I'm taking the good practices here, including storing pointer as u64.

$\endgroup$
14
  • $\begingroup$ What dangers do you mean? Is it more dangerous for a language to provide a native memory representation of a data structure, than it is to compile user-defined data structures to a memory representation which is also decided by the compiler? $\endgroup$ Commented Jun 30, 2023 at 18:53
  • $\begingroup$ @kaya3-supportthestrike I wanted to mean that the memory size for a native super type can change as the sources for the runtime are changed too. So if you add, change the type of or remove any field from the base type, you've to update the FFI decorator I showed above. It's fine for the compiler to automatically decide the layout as long as there's no native super class inherited. $\endgroup$ Commented Jun 30, 2023 at 18:56
  • 1
    $\begingroup$ When a library is implemented in a lower-level language, it typically uses macros and language runtime functions to structure the visible parts of the data in the same way as "normal" data. So if it has an integer field, it may need to box it like the language's normal integers. Look at the implementation of Python's list methods, for example. $\endgroup$ Commented Jun 30, 2023 at 21:45
  • $\begingroup$ I’m afraid I also don’t understand the question. What is the semantics of your hypothetical FFI decorator? It’s not clear to me what it is intended to allow you to do. Is the idea that you declare TextMatch essentially like you’d declare an interface, and the actual implementation is defined elsewhere? Your question discusses inheritance, but it’s not clear to me why inheriting from a class would be useful if it doesn’t have fields and methods accessible from within the language, and those must also conform to some agreed-upon protocol, so I don’t see how memorySize could be enough. $\endgroup$ Commented Jun 30, 2023 at 21:47
  • 1
    $\begingroup$ Okay, so in that case, it seems like the same effect could be obtained via a fixed-size byte array private field, correct? Both things specify that the objects in question include a payload of arbitrary data of a given size, so I don’t see how they could possibly differ. And if that is correct, I think the question would be clearer if you expressed it that way rather than using a decorator, since it would make it much more obvious what the number means. $\endgroup$ Commented Jun 30, 2023 at 22:52

1 Answer 1

Reset to default
3
$\begingroup$

Preface

After some discussion in the comments of your question, it is clear that when you write

[FFI(memorySize = n)]
public class TextMatch {
   ...
}

the semantics you intend the FFI annotation to have is entirely interchangeable with defining a class with a single private member of fixed size and unspecified layout:

public class TextMatch {
  private byte[n] payload;
  ...
}

I find this much easier to reason about, so I am going to dispense with your hypothetical FFI annotation and use the version with a private field, instead.

Answer

I think what your question is asking about essentially amounts to a matter of ABI.

Whenever code written in different languages must interoperate, it must agree to some protocol. In some cases, this protocol might be quite structured, using something like Protobuf or even COM, but what you have in mind involves significantly less ceremony. You are envisioning a garbage collected, object-oriented programming language implementation with a compiler and runtime that assumes a particular heap layout for objects, and you are supposing that you wish to use some systems programming language, like C or Rust, to create values in memory that happen to have that layout.

In your question, you seem to be very focused on one problem, namely mismatches in the size of an object’s payload:

I wanted to mean that it's dangerous for native classes to use that FFI decorator. Why? Because if you've written the implementation for TextMatch in Rust or any systems language and everytime you change the fields or memory size of the base type, you've to update that FFI decorator for these native non-final classes.

I would like to make the case that this is somewhat tunnel-visioned, and ensuring a consistent specification of the sizes of your values is just one of numerous obligations you accept when trying to make languages interoperate in this way. Consider:

  • Since your language is garbage collected, it assumes that objects are all allocated from memory originating in its managed heap. Your systems language will therefore need to coordinate with your language’s allocator to ensure the memory comes from the right place.

  • If your garbage collector is moving, you’ll need to take care to ensure that it never moves the memory while your systems language is using it. Rust’s safety mechanisms assume that it owns the memory it operates on, but if it doesn’t, you’re already in unsafe territory.

  • A reference to a heap object can’t be represented by a pointer to the start of some opaque payload, it needs some sort of header to identify it. Therefore, your systems language will need to know the size of that header, and where the payload is placed relative to it, in order to know how to access the fields in its custom payload.

  • If your object’s payload contains any pointers to GC-managed memory, the GC needs to know about them, so the payload can’t truly be opaque. Therefore, you’ll need some way to specify which fields are pointers and which are nonpointers, and you’ll need to ensure that your language’s runtime and your systems language agree on both the order in which the fields are laid out and how they are aligned.

    • If your runtime utilizes a pointer tagging scheme, then any GC pointers in your object’s payload will need to be properly tagged before they are written and untagged before they are read.

  • If your object’s payload contains any resources that are not GC-managed, you’ll need some protocol to arrange for them to be properly disposed of when the object is garbage collected.

When any of these aspects of your language’s ABI changes, the code written in your systems language will have to be changed to reflect it. This is just the nature of working at such a low level: anything that lives outside of the boundaries of your language is also outside the scope of its mechanized integrity checks.

There are of course numerous strategies (of variable effort) for mitigating these risks. One option is to automatically generate type definitions for your systems language from declarations in your managed language (or vice versa), which reduces the surface area for bugs to the code that performs the translation. You could also have some sort of static analyzer that checks that type definitions written in each language match up, with similar results. You could also utilize fuzz testing to search for ways in which the implementations don’t match up, or you could run a handwritten test suite in a special mode that enables lots of internal checks to ensure everything is working properly.

But no matter what you do, some of these requirements are always going to be difficult to guarantee in general. Ultimately, the chain of trust has to start somewhere.

$\endgroup$

You must log in to answer this question.

Start asking to get answers

Find the answer to your question by asking.

Ask question

Explore related questions

See similar questions with these tags.