Mobile cross platform C++ code part 2: Consuming OpenCV

In this post, I’l go through our findings to compile and run cross platform C++ code for iOS and Android that consumes OpenCV.

The first thing to note here is that I’m not going to expose the OpenCV C++ code directly to ObjectiveC or Java. Instead, I’m using Dropbox’s Djinni to abstract it behind a cross platform C++ bridge. This is why most of the tutorials out there weren’t very helpfull for me. The code will be consumed by the React Native side of a ReactXP application.

Basically, the same OpenCV consuming code base, will be compiled by Xcode and Android build tools into static or dynamic libraries.

iOS

We had to use the latest OpenCV2 Cocoa Pod to be aligned with the version 3.4.2 of the Android code. Interestingly, I had to fist install OpenCV2 then update it.

Local Storage

We’re using an experimental feature of OpenCV which is the Deep Neural Network module. We’re loading assets and I still have to figure out how to deal with downloading them to the local storage.

In the meantime, I used iTunes to copy them on the phone. And if you wan’t your app to appear in the file transfer tool, you need to explicitly tell the OS that you want it.

The way I get to the app’s local storage root path is like the following:

std::string appData(getenv(“HOME”));

GET requests to http endpoints

For security reasons, an iOS app should only access secured https endpoints. And guess what, I don’t have one during development. I have to grab a snapshot from a camera that does not offer a secure http endpoint.

At this point, I’m not able to make the OpenCV code run on an iPhone. In the previous episode, I could not open the Android phone camera. This time, opening VideoCapture providing an unsecure http endpoint returns false.

cv::VideoCapture cap;
// This call returns false on iOS
cap.open(cameraUrl);

And I do not know why 😦 because the OpenCV logs do not tell me what is wrong. Frustrating. Not sure that a secure one would work though.

Android

For Android, OpenCV 3.4.2 source code was downloaded and built as static libraries for various ABIs including armeabi-v7a because this is what’s powering my Moto E4 Plus.

Make the consuming code link with OpenCV static libraries

I also spent a lot of time figuring out the correct link order of the static librairies. In my opinion there should be an option to tell the linker to figure it out and to spit it in the output. Is it that complicated ?

Make it run

The trick here was to compile with the following options for the code to run properly:

-fpack-struct=8 -mllvm -arm-promote-constant=0

Believe me it took us a lot of time an energy to get there. And it made me angry as I explained earlier. But I’m happy that this has been acknowledged as a bug.

Local Storage

I do not know if there is a dedicated local storage for an Android app yet. So we copied the assets to the phone through adb and put them in a dedicated directory, which is then accessed explicitly from the C++ code. You must grant your application the right to access local storage obviously.

Takeways

• As you can see, it is not “it just works” yet. I really think efforts should be made to think that, yes, it could not work as expected and that it should be reported even in release builds. Because you know, you’ll never be able to mimic each production environment of the consumers of your code.
• As a consumer of OSS, always kindly report issues. Sometimes, it may be a usage mistake on your side. But it could also be that the implementation is missing on a specific platform. Or it could be a bug. Now you have to choose the right channel to do that.

For OpenCV, there is a Q/A forum. You should start from here, then  later switch to the Github issues.

The entire community will be beneficial at some point.

What’s next

In the next iteration I will tell you how we solved the remaining problems.

In the meantime, I’m heading to React Native Europe in Wroclaw, Poland, next week. I’m sure I will be back with a ton of new informations about the native part of React Native.

Stay tuned.