Easy Binary Compatible Interfaces Across Compilers in C++ - Part 0 of n: Introduction and a Sneak Preview

The problem of using a C++ library compiled with Compiler A, from a program compiled with Compiler B has been a problem for a while. This is especially true on Windows where Visual C++ generally breaks binary compatibility from release to release. Shipping a library for Windows involves shipping several versions for Visual C++ as well now often for mingw gcc.
Some of the problems C++ has in regards to binary compatibility across different compilers are:name mangling,object layout, exception support.
There are several ways to get around this.

• Use extern “C” and have a C-style interface.The 2nd answer of this StackOverflow article gives an example of this technique.
• Use COM if you are on Windows
• Use Matthew Wilson’s technique from Imperfect C++ chapters 7-8.

There are whole books written on COM, so I won’t try to go into too many details. A brief overview in regards to the binary interface is here.
The basic idea is that you define an interface like this

Interface Definition

1. struct Interface;
2. struct InterfaceVtable{
3.     int (*Function1)(struct Interface*);
4.     int (*Function2)(struct Interface*, int);
5. };
6.  
7. struct Interface{
8.     struct InterfaceVtable* pTable;
9.  
10. };

It can be used like this

Using an Interface

1. struct Interface* pInterface = GetInterfaceSomehow();
2. int a = pInterface->pTable->Function1(pInterface);

Implementing an interface like this is painful and will be left as an exercise to the reader .
Fortunately, (and by design), Microsoft Visual C++ and most Windows C++ compilers will generate something compatible to the above with an abstract base class using pure virtual functions.

Inteface using C++ (MSVC)

1. struct InterfaceCpp{
2.     virtual int Function1() = 0;
3.     virtual int Function2(int) = 0;
4. };

You can implement and use like this

Code Snippet

1. struct InterfaceImplementation:public InterfaceCpp{
2.     virtual int Function1(){return 5;}
3.     virtual int Function2(int i){return 5 + i;}
4. };
5.  
6. InterfaceImplementation imp;
7. InterfaceCpp* pInterfaceCpp = &imp;
8. std::cout << pInterfaceCpp->Function2(5) << std::endl;

The reason for this, is that the version with function pointers was doing a vtable and a vptr by hand and this version is letting the compiler do it. For more information about vtable and vptr see the excellent article by Dan Saks in Dr. Dobbs.
While the above solution works on Windows (generally), this is not guaranteed to always work A more general cross-platform solution is presented in Matthew Wilson’s Imperfect C++ in chapters 7 and 8. He basically provides a way and macros that allow you to define the above structure manually (ie define your own vtables).
By using either COM style interfaces with compilers that have a compatible vtable layout or rolling your own, you can have cross-compiler binary compatible interfaces.However, you do not have

• Exceptions
• Due to not having exceptions, you often have to use error codes and thus do not have real return values.
• Standard C++ types such as vector and string (use arrays and const char*)

In fact, in an article explaining why Microsoft created C++/CX Jim Springfield stated one of the problems with COM even with libraries such ATL was
“There is no way to automatically map interfaces from low-level to a higher level (modern) form that throws exceptions and has real return values.”
During this series of posts, I will discuss the development of a C++11 library that has the following benefits

• Able to use std::string and std::vector as function parameters and return values
• Use exceptions for error handling
• Compatible across compilers – able to use MSVC to create.exe and g++ to create .dll on Windows, and g++ for executable and clang++ to create .so on Linux
• Works on Linux and Windows
• Written in Standard C++11
• No Macro magic
• Header only library

As we progress we will talk about some of the disadvantages and areas for improvements and possible alternatives
Here is how we would define an interface DemoInterface. Note jrb_interface is the namespace of the library.

Code Snippet

1. using namespace jrb_interface;
2.  
3. template<bool b>
4. struct DemoInterface
5. :public define_interface<b,4>
6. {
7.     cross_function<DemoInterface,0,int(int)> plus_5;
8.  
9.     cross_function<DemoInterface,1,int(std::string)> count_characters;
10.  
11.     cross_function<DemoInterface,2,std::string(std::string)> say_hello;
12.  
13.     cross_function<DemoInterface,3,std::vector<std::string>(std::string)>
14.         split_into_words;
15.  
16.     template<class T>
17.     DemoInterface(T t):DemoInterface<b>::base_t(t),
18.         plus_5(t), count_characters(t),say_hello(t),split_into_words(t){}
19. };

.
In this library, all interfaces are actually templates that take a bool parameter. The reason for this will become clear as we discuss the implementation in later posts.
All interfaces inherit from define_interface which takes a bool parameter (just use the bool passed in to the template) and an int parameter specifying how many functions are in the interface.  If you pass in a too small number, you will get a static_assert telling you that the number is too small.
To define a function in the interface, use the cross_function template
The first parameter is the interface in this case DemoInterface. The second parameter is the 0 based position of the function. The first function is 0, the second is 1, the third 2, etc. The third and final parameter of cross_function is the signature of the function is the name style as std::function.
Finally all interfaces need a templated constructor that takes a value t and passes it on to the base class as well as each function. For convenience the define_interface template defines a typedef base_t that you can use in your constructor initializer.
To implement an interface you would do this

Code Snippet

1. struct DemoInterfaceImplemention:
2.     public implement_interface<DemoInterface>{
3.  
4.         DemoInterfaceImplemention(){
5.  
6.             plus_5 = [](int i){
7.                 return i+5;
8.             };
9.  
10.             say_hello = [](std::string name)->std::string{
11.                 return "Hello " + name;
12.             };
13.  
14.             count_characters = [](std::string s)->int{
15.                 return s.length();
16.             };
17.  
18.             split_into_words =
19.                 [](std::string s)->std::vector<std::string>{
20.                     std::vector<std::string> ret;
21.                     auto wbegin = s.begin();
22.                     auto wend = wbegin;
23.                     for(;wbegin!= s.end();wend = std::find(wend,s.end(),' ')){
24.                         if(wbegin==wend)continue;
25.                         ret.push_back(std::string(wbegin,wend));
26.                         wbegin = std::find_if(wend,s.end(),
27.                             [](char c){return c != ' ';});
28.                         wend = wbegin;
29.                     }
30.                     return ret;
31.             };
32.  
33.         }
34. };

To implement an interface, you derive from implement_interface specifying your Interface as the template parameter. Then in your constructor you assign a lambda with the same signature you specified in the definition of the interface to each of the cross_function variables.
To use an interface, you construct use_interface providing the Interface as the template parameter.

Code Snippet

1. // Assume iDemo is defined as follows
2. // use_interface<DemoInterface> iDemo = ...
3. int i = iDemo.plus_5(5);
4.  
5. int count = iDemo.count_characters("Hello World");
6.  
7. std::string s =  iDemo.say_hello("John");
8.  
9. std::vector<std::string> words = iDemo.split_into_words("This is a test");

You then call the functions just as you would with any class object. Note the use of . instead of –>
Thank you taking the time to read this post. I hope this has piqued your interest. In future posts we will explore how we create this library, and how we can extend this library to do more. I hope you will join me.
You can find compilable code at
https://github.com/jbandela/cross_compiler_call
The code has been tested on

• Windows with compiling the executable with MSVC 2012 Milan (Nov CTP) and the DLL with mingw g++ 4.7.2
• Ubuntu 12.10 with compiling the executable with g++ 4.7.2 and the .so file with clang++ 3.1

Instructions on how to compile are included in the README.txt file.
Please let me know what you think in the comments section
- John Bandela
CodeProject

Another alternative to lambda move capture

Today the post on isocpp.org called Learn How to Capture By Move caught my attention. I found the post informative and thought provoking, and you should go and read it before reading the rest of this post.

The problem is how do we lambda capture a large object we want to avoid copying.

The motivating example is below

 function<void()> CreateLambda()  
 {  
   vector<HugeObject> hugeObj;  
   // ...preparation of hugeObj...  
   auto toReturn = [hugeObj] { ...operate on hugeObj... };  
   return toReturn;  
 }  

The solution proposed is a template class move_on_copy that is used like this

 auto moved = make_move_on_copy(move(hugeObj));  
 auto toExec = [moved] { ...operate on moved.value... };  

However, there are problems with this approach, mainly in safety. The move_on_copy acts as auto_ptr and silently performs moves instead of copies (by design).

I present here a different take on the problem which accomplishes much of the above safely, however, with a little more verbosity in exchange for more clarity and safety.

First let me tell you how you would use the final product

 HugeObject hugeObj;  
 // ...preparation of hugeObj...  
 auto f = create_move_lambda(std::move(hugeObj),[](moved_value<HugeObject> hugeObj){  
   // manipulate huge object  
   // In this example just output it  
   std::cout << hugeObj.value() << std::endl;  
 });  

The point of interest is the create_move_lambda
The first argument is r-value reference of the object we want to move generated with std::move
The second argument is the lambda
Instead of having the moved object in the capture list, we take an extra argument of moved_value which looks like this

 template<class T>
 using moved_value = std::reference_wrapper<T>;

You can access the moved object using moved_value.get()

Currently, you can have any number or parameters or any return type for your lambda, but only 1 move capture. That restriction, I am sure could eventually be removed.

So how does this work? Instead of attempting to change the capture type or wrap the capture type, we instead create a function object which wraps the lambda, stores the moved object, and when called with a set of arguments, forwards to the lambda with the moved_value as the first parameter.  Below is the implementation of move_lambda and create_move_lambda

template<class T,class F>
struct move_lambda{
private:
 T val;
 F f_;
public:
 move_lambda(T&& v, F f):val(std::move(v)),f_(f){};
 move_lambda(move_lambda&& other) = default;
 move_lambda& operator=(move_lambda&& other) = default;

 template<class... Args>
 auto operator()(Args&& ...args) -> decltype(this->f_(moved_value<T>(this->val),std::forward<Args>(args)...))
 {
    moved_value<T> mv(val);
    return f_(mv,std::forward<Args>(args)...);
 }

 move_lambda() = delete;
 move_lambda(const move_lambda&) = delete;
 move_lambda& operator=(const move_lambda&) = delete;


};

 template<class T,class F>
 move_lambda<T,F>create_move_lambda(T&& t, F f){
    return move_lambda<T,F>(std::move(t),f);
 }

So now we have move_lambda returned from create_move_lambda that can be used just like a lambda with move capture. In addition, copy construction and assignment are disabled so you cannot inadvertently copy the lambda. However, move construction and move assignment are enabled so you can move the lambda. Further examples are below

 // A movable only type, not copyable
 TestMove m;
 m.k = 5;

 // A movable only type, not copyable
 TestMove m2;
 m2.k = 6;

 // Create a lambda that takes 2 parameters and returns int
 auto lambda = create_move_lambda(std::move(m),[](moved_value<TestMove> m,int i,int)->int{
    std::cout << m.get().k << " " << i << std::endl;return 7;
 });

 // Create a lambda that takes 0 parameters and returns void
 auto lambda2 = create_move_lambda(std::move(m2),[](moved_value<TestMove> m){
    std::cout << m.get().k << std::endl;
 });

 std::cout <<  lambda(1,2) << std::endl;

 lambda2();

 // Compiler error if you try to copy
 //auto lambda4 = lambda;

 // Able to move
 auto lambda3 = std::move(lambda2);
 lambda3();

However, there is still one more problem left in using move_lambda. You cannot store move_lambda in a std::function because move_lambda does not have a copy constructor. So how do we write the original function we wanted. Well we write a movable_function which is presented below

// Unfortunately, std::function does not seem to support move-only callables
// See § 20.8.11.2.1 point 7 where it requires F be CopyConstructible 
// From draft at http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2011/n3242.pdf
// Here is our movable replacement for std::function
template< class ReturnType, class... ParamTypes>
struct movable_function_base{
 virtual ReturnType callFunc(ParamTypes&&... p) = 0;

};


template<class F, class ReturnType, class... ParamTypes>
struct movable_function_imp:public movable_function_base<ReturnType,ParamTypes...>{
 F f_;
 virtual ReturnType callFunc(ParamTypes&&... p){
    return f_(std::forward<ParamTypes>(p)...);
 }
 explicit movable_function_imp(F&& f):f_(std::move(f)){};

 movable_function_imp() = delete;
 movable_function_imp(const movable_function_imp&) = delete;
 movable_function_imp& operator=(const movable_function_imp&) = delete;
};


template<class FuncType>
struct movable_function{};

template<class ReturnType, class... ParamTypes>
struct movable_function<ReturnType(ParamTypes...)>{
 std::unique_ptr<movable_function_base<ReturnType,ParamTypes...>> ptr_;

 template<class F>
 explicit movable_function(F&& f):ptr_(new movable_function_imp<F,ReturnType,ParamTypes...>(std::move(f))){}
 movable_function(movable_function&& other) = default;
 movable_function& operator=(movable_function&& other) = default;

 template<class... Args>
 auto operator()(Args&& ...args) -> ReturnType
 {
    return ptr_->callFunc(std::forward<Args>(args)...);

 }
 movable_function() = delete;
 movable_function(const movable_function&) = delete;
 movable_function& operator=(const movable_function&) = delete;

};

Based on the above we can write our CreateLambda() as

movable_function<void()> CreateLambda()
{
 // Pretend our TestMove is a HugeObject that we do not want to copy
 typedef TestMove HugeObject;

 // Manipulate our "HugeObject"
 HugeObject hugeObj;
 hugeObj.k = 9;

 auto f = create_move_lambda(std::move(hugeObj),[](moved_value<HugeObject> hugeObj){// manipulate huge object
    std::cout << hugeObj.get().k << std::endl;
 });

 movable_function<void()> toReturn(std::move(f));

 return toReturn;
}

And use it like this

 // Moved out of function  
 auto lambda4 = CreateLambda();  
 lambda4();  

Alternatives and Extensions
A simple alternative would be instead of using moved_value as the first parameter, take a reference. This would make the lambda look like this.

 // You can take a reference instead of a moved_value if you want 
 auto lambda5 = create_move_lambda(std::move(m3),[](TestMove& m){ 
    std::cout << m.k << std::endl; 
 }); 

This actually works due to moved_value being a template alias for reference_type. This is a little bit shorter than the previous code, but you cannot tell what are your real lambda parameters and what are the parameters used to simulate move capture.

An extension to this code would be allowing for multiple captured variables. Currently, the code only allows for 1 move captured variable.

Thanks for taking the time to read this.

You can find a compilable example at https://gist.github.com/4208898
The code above requires a fairly compliant C++11 compiler and has been tested with GCC 4.7.2 (Windows nuwen.net distribution).


An older version of the code that compiles with ideone and with VC++ 2012 November CTP is at http://ideone.com/OXYVyp

Please leave comments and let me know what you think.

- John Bandela
Modified 12/4/2012
Many thanks to Eric Niebler for his comments on the previous version of the code for this article.