Artificial Intelligence & Mobile Software & OpenCV & TensorFlow
I have previously shared with you ML & DL & AI, vehicle lane tracking and license plate recognition software and articles explaining how to write them.
I worked on these for more than 10 days, wondering how the object recognition and vehicle lane tracking software would give a result, especially in terms of performance on mobile devices. You know, some people take the device and say that this device does this and that, or I wonder if I should find the code somewhere with a fully working example and run it without doing any work and make an article or video about it.
The lane tracking and license plate recognition software that I wrote before and shared my experiences with you were the kind of software we call desktop and were considered quite satisfactory depending on the power of the computer. By the way, I also talked about the approaches that we call Haar cascade LBP and CNN, which include DL and AI algorithmically.
Now let’s see how the performance is on mobile devices. I have tried it on an android based device. I had 3 different models of the same device, but I used 2 of them.
The most important issues I have identified are:
1-) If you write only android java (or even kotlin etc) your job is very difficult. Even iOS includes Swift or Objective C. It is sufficient for a single subject, that is, only object recognition, but if the job is more complex, for example, lane recognition tracking is also involved, you are burned out.
2-) You should definitely use C++. You should write the layer that will see the accounts that will do the job, write it in C++ and compile it into a Shared Object file, which we call SO, and use it in android projects. You can also use static .a if you want. I prefer SO.
3-) Most people (beginners or stubborn) write CPP, that is, C++ code in the project they do in android studio, put them in a directory and compile with cmake and/or ndk-build. I don’t think you should do it at all, it will take longer and you will deal with meaningless error messages. I will explain the practice below.
4-) I think the project that will be examined in object recognition is Google’s tensorflow demo software. Download and see. There’s a reason it works fast. First of all, it uses the onpreviewframe event method of the camera, converts the incoming byte array frame into a picture, searches for an object, marks it, puts it back in the camera buffer and does not show it to us. (So you also understand how some apps are made). So what does it do:
OnImageAvailableListener uses the Camera.PreviewCallback interfaces and disables the onpreviewframe event of the camera, takes it directly from the camera, yes directly with the Imagereader, and takes it in android.media.image type (it’s abstract, actually) and does the following. It crops the image to a certain size and puts it into image recognition. It evaluates the incoming results and draws a rectangle with canvas on the OverlayView defined on the android screen. In fact, we can call it a deceiver. Imagine that you opened a transparent window on the camera image and it was Overlay, it examined the incoming picture and perceived the objects, then you create a canvas on it, draw a canvas on it, render it overlay and see that the objects are drawn with a rectangle on the image you see on the screen! how easy is it? Examine the codes, you will finally understand.
Therefore, it works very fast because there is no manipulation with the camera image, that image always works without any barriers.
You wanted to write a second solution. In other words, you will mount the lane tracking system on it. Then pay attention to what I write and do not go astray. Otherwise, this integrated software works very slowly on your mobile device. You can use the same technique to integrate your second solution into it.
If you do it with classical methods, your device will be very slow.
5-) If you do lane recognition, that is, your second software with Haar, LBP, etc., you will be on the wrong path. What you need to do is not even write java v. You will sit down and write in C++. Even if the desktop is python , C# is now whatever you write with. But if you are going to read the license plate and speed of more than 20 vehicles from the common information of the camera and radar device that comes with municipal software on the desktop, it is recommended to write it in C++. Write this application in C++ on the Android side. If I said android side, it is not necessarily android studio. Write it down somewhere. If you have installed NDK under android sdk on your PC after writing, there is a compiler in the NDK-BUILD directory. Compile with it.
As I wrote above, do not do it in android studio, it is expert advice.
In the meantime, let’s talk about CV or Computer Vision right here. It’s missing. Computer Vision is NOT an image processing. Recognition of objects etc. If OPENCV; It is a library that helps us in Computer Vision, that is, AI, including image processing. We will use that here too.
At that time I had downloaded version 2.4.9 of OPENCV or version 3.2.0. They have an SDK which is downloaded as ZIP, and they have full Source codes which are downloaded as EXE. If you examine the SDK and source codes and start working a bit, you will see that something is missing. There are some SO files you will need. There are some that start with libopencv_. (Libopencv_java.so is valid in 2.x and libopencv_java3.so is valid in 3.x.) For example, you will use a method in objdetect object, but 2.4.9 does not have it or it does not exist, but it is slow even if you use it, etc.
In this case, there are C++ source codes in the modules directory in the source codes. You will compile them accordingly and create the SO file.
I OVERLOADED some of the methods I needed from this source library by searching for hours in C++. I put it inside my own C++ software. I think 3-5 methods worked for me. Then I compiled 6-7 C++ CPP and HPP codes I wrote from the command line with ndk-build. Here’s what I did:
First of all, select File-New-import Module, on the screen that comes up, there is a Java directory under the folder in the directory where you installed OpenCV. Choose it. Android Studio will do the necessary work. You will now see this in your project. This is not enough, you need to do one more job.
In your Android Project, if you come to the project Structure from File and select your app on the left, there is the dependency tab above, tap the plus + on the right, select module dependency, and select OpenCV320 etc. on the screen that comes up.
Open a directory on drive D or C. Open another directory with the extension Jni in that directory. Put your C++ source files there. Then type Android.mk and Application.mk. These are needed because they are read by the compiler and compile accordingly. You can find examples on the Internet. Maybe I will write to you and explain the contents line by line. Application.mk is already at most 4 lines. Android.mk needs some attention. Yes these are ok. Then type ndk-build.cmd from the directory you created (not the jni one below) and press enter. (If it says ndk-build is not recognized, then either ndk is not installed or you did not specify it in the path, do this)
You just took a look, it created directories named libs and obj, and according to what you specified in Application.mk and abi_filter, it also created arm directories and brought both your so file and other necessary so files there. Isn’t it super?
Now copy them to arm (armeabi-v7a, arme64-v8a etc) directories in JniLibs under your android project. Load it with the System.Loadlibrary command in the project, good-bye, do you say? You need to do a few more things. You have to specify your method that you will call from there, that is, that you will call it from the file that is C++ and from the SO file, as a reference with public native — that is, with a single line. All right, you ran it and you saw the error in the LogCat window (I mean Runtime) it says UnsatisfiedLink (others fasa fiso) then look what you did wrong. Which has happened to you before. That’s why you couldn’t work with a SO file you stole from somewhere, right?
Like this: Your C++ codes are compiled and run with Android Studio via JNI interface I. In other words, JAVA and C++ work with each other if there is a JNI supported structure. Then the JNI definitions you specified at the beginning of your C++ codes ATTENTION HERE (remember from somewhere) your project name in your android project is var, you know, package name, it must be compatible and the same. Only in android studio, for example, if org.selcukcelik.lanedetection is in your C++ codes, for example:
JNIEXPORT void JNICALL java_org_selcukcelik_lanedetection_LaneDetector_mainDelegate
MUST HAVE. So be careful on the underlined C++ side. If this is compatible, the UnsatisfiedLink error will NOT come.
You ran it and if there is an error on the C++ side, make changes on that side, compile it again and copy-paste the SO file back into the project.
You will see the performance clearly when you run it.
Apart from that, usually when an Android Studio project is made, it opens a simple native-lib.cpp and you get a SO file when you compile the code there is a simple piece that has already been written. What I am describing is a more advanced and practical approach.
There is another thing called SymLink. The libraries in Jni in OPENCV are in directories like armeabi-v7a, so you don’t have to copy them to your automatic project if necessary. Go to the app/src/main directory of your project from the command line and create a virtual directory with the mklink command from the command line. Learn how it’s done. You look at the project, jniLibs came there automatically.
In short, if you are going to dive into these works, it will be useful to know not only how to code on IOS or Android, but also many subjects very well. You need to think about speed and performance.
You may encounter a lot of error messages, some of them very strange and may mislead you.
Now it’s time to drive out in the fresh air and do a full test. The next step is to notify us of the obstacles in front of us while driving (slowing down) and keeping the distance with the vehicle in front, etc.
The fun continues. Stay well.
Selcuk Celik
