Object Recognition with OpenCV on Android
This article is for a person who has some knowledge on Android and OpenCV. We will look at how to use the OpenCV library to recognize objects on Android using feature extraction.
- Download and setup Android Studio
I am using Android Studio and you can follow this link to download and install Android studio and SDK tools but if you are a die hard eclipse fan you also can follow this tutorial( no hard feelings ;) )
2. Setting up OpenCV library inside Android Studio
You have to download and import OpenCV library to android studio and there is a stackoverflow answer which you can follow to setup everything. If you are using Eclipse use this link.
Now you are ready to mingle with me ;). The algorithm we are going to use is ORB(Oriented FAST and Rotated BRIEF). As an OpenCV enthusiast, the most important thing about the ORB is that it came from “OpenCV Labs”. This algorithm was brought up by Ethan Rublee, Vincent Rabaud, Kurt Konolige and Gary R. Bradski in their paper ORB: An efficient alternative to SIFT or SURF in 2011. It is a good alternative to SIFT and SURF in computation cost, matching performance and mainly the patents. Yes, SIFT and SURF are patented and you are supposed to pay them for its use. But ORB is not . You still can use SIFT and SURF but you have to compile them separately as they are not contained in the latest version of the OpenCV library and if you are going to make a commercial application you have to pay and get them.
Here is a link for the other existing algorithms in OpenCV for object detection
Now you know about ORB and it’s history, lets start coding.
3. Create a new project on Android Studio
This step is trivial therefore I am giving this link so you could follow that and create a new project. Please do read what is on that page it is very informative.
4.Creating the XML for the UI
If you are not new to Android you know that very well that you need to create a XML file for the UI. These files are created inside the layout folder.
<FrameLayout xmlns:android="http://schemas.android.com/apk/res/android"
xmlns:tools="http://schemas.android.com/tools"
xmlns:opencv="http://schemas.android.com/apk/res-auto"
android:layout_width="match_parent"
android:layout_height="match_parent" >
<org.opencv.android.JavaCameraView
android:layout_width="fill_parent"
android:layout_height="fill_parent"
android:visibility="gone"
android:screenOrientation="portrait"
android:id="@+id/tutorial1_activity_java_surface_view"
opencv:show_fps="true"
opencv:camera_id="any"
android:layout_below="@+id/text1"/> <TextView
android:layout_width="wrap_content"
android:layout_height="wrap_content"
android:gravity="center"
android:id="@+id/text1"
android:text="test"
android:textStyle="bold"
android:textColor="#ff00ff"
android:background="#00ff00"
android:textColorHighlight="#000000"
android:textIsSelectable="true"
android:textSize="32sp"
/>
</FrameLayout>
You can create a XML file using the above code. First segment is the Camera View, second segment (TextView) is quite not necessary but that is there for testing purposes ( just in case if you want to get some data on the screen rather than the log)
5. Write about the Main Activity
package com.example.akshika.opencvandroidtut;
import android.app.Activity;
import android.content.res.AssetManager;
import android.graphics.Bitmap;
import android.graphics.BitmapFactory;
import android.os.Bundle;
import android.util.Log;
import android.view.SurfaceView;
import android.view.WindowManager;
import android.widget.TextView;
import org.opencv.android.BaseLoaderCallback;
import org.opencv.android.CameraBridgeViewBase;
import org.opencv.android.CameraBridgeViewBase.CvCameraViewFrame;
import org.opencv.android.LoaderCallbackInterface;
import org.opencv.android.OpenCVLoader;
import org.opencv.android.Utils;
import org.opencv.core.DMatch;
import org.opencv.core.Mat;
import org.opencv.core.MatOfByte;
import org.opencv.core.MatOfDMatch;
import org.opencv.core.MatOfKeyPoint;
import org.opencv.core.Scalar;
import org.opencv.features2d.DescriptorExtractor;
import org.opencv.features2d.DescriptorMatcher;
import org.opencv.features2d.FeatureDetector;
import org.opencv.features2d.Features2d;
import org.opencv.imgproc.Imgproc;
import java.io.IOException;
import java.io.InputStream;
import java.util.LinkedList;
import java.util.List;
public class MainActivity extends Activity implements CameraBridgeViewBase.CvCameraViewListener2 {
private static final String TAG = "OCVSample::Activity";
private int w, h;
private CameraBridgeViewBase mOpenCvCameraView;
TextView tvName;
Scalar RED = new Scalar(255, 0, 0);
Scalar GREEN = new Scalar(0, 255, 0);
FeatureDetector detector;
DescriptorExtractor descriptor;
DescriptorMatcher matcher;
Mat descriptors2,descriptors1;
Mat img1;
MatOfKeyPoint keypoints1,keypoints2;
static {
if (!OpenCVLoader.initDebug())
Log.d("ERROR", "Unable to load OpenCV");
else
Log.d("SUCCESS", "OpenCV loaded");
}
private BaseLoaderCallback mLoaderCallback = new BaseLoaderCallback(this) {
@Override
public void onManagerConnected(int status) {
switch (status) {
case LoaderCallbackInterface.SUCCESS: {
Log.i(TAG, "OpenCV loaded successfully");
mOpenCvCameraView.enableView();
try {
initializeOpenCVDependencies();
} catch (IOException e) {
e.printStackTrace();
}
}
break;
default: {
super.onManagerConnected(status);
}
break;
}
}
};
private void initializeOpenCVDependencies() throws IOException {
mOpenCvCameraView.enableView();
detector = FeatureDetector.create(FeatureDetector.ORB);
descriptor = DescriptorExtractor.create(DescriptorExtractor.ORB);
matcher = DescriptorMatcher.create(DescriptorMatcher.BRUTEFORCE_HAMMING);
img1 = new Mat();
AssetManager assetManager = getAssets();
InputStream istr = assetManager.open("a.jpeg");
Bitmap bitmap = BitmapFactory.decodeStream(istr);
Utils.bitmapToMat(bitmap, img1);
Imgproc.cvtColor(img1, img1, Imgproc.COLOR_RGB2GRAY);
img1.convertTo(img1, 0); //converting the image to match with the type of the cameras image
descriptors1 = new Mat();
keypoints1 = new MatOfKeyPoint();
detector.detect(img1, keypoints1);
descriptor.compute(img1, keypoints1, descriptors1);
}
public MainActivity() {
Log.i(TAG, "Instantiated new " + this.getClass());
}
/**
* Called when the activity is first created.
*/
@Override
public void onCreate(Bundle savedInstanceState) {
Log.i(TAG, "called onCreate");
super.onCreate(savedInstanceState);
getWindow().addFlags(WindowManager.LayoutParams.FLAG_KEEP_SCREEN_ON);
setContentView(R.layout.layout);
mOpenCvCameraView = (CameraBridgeViewBase) findViewById(R.id.tutorial1_activity_java_surface_view);
mOpenCvCameraView.setVisibility(SurfaceView.VISIBLE);
mOpenCvCameraView.setCvCameraViewListener(this);
tvName = (TextView) findViewById(R.id.text1);
}
@Override
public void onPause() {
super.onPause();
if (mOpenCvCameraView != null)
mOpenCvCameraView.disableView();
}
@Override
public void onResume() {
super.onResume();
if (!OpenCVLoader.initDebug()) {
Log.d(TAG, "Internal OpenCV library not found. Using OpenCV Manager for initialization");
OpenCVLoader.initAsync(OpenCVLoader.OPENCV_VERSION_3_0_0, this, mLoaderCallback);
} else {
Log.d(TAG, "OpenCV library found inside package. Using it!");
mLoaderCallback.onManagerConnected(LoaderCallbackInterface.SUCCESS);
}
}
public void onDestroy() {
super.onDestroy();
if (mOpenCvCameraView != null)
mOpenCvCameraView.disableView();
}
public void onCameraViewStarted(int width, int height) {
w = width;
h = height;
}
public void onCameraViewStopped() {
}
public Mat recognize(Mat aInputFrame) {
Imgproc.cvtColor(aInputFrame, aInputFrame, Imgproc.COLOR_RGB2GRAY);
descriptors2 = new Mat();
keypoints2 = new MatOfKeyPoint();
detector.detect(aInputFrame, keypoints2);
descriptor.compute(aInputFrame, keypoints2, descriptors2);
// Matching
MatOfDMatch matches = new MatOfDMatch();
if (img1.type() == aInputFrame.type()) {
matcher.match(descriptors1, descriptors2, matches);
} else {
return aInputFrame;
}
List<DMatch> matchesList = matches.toList();
Double max_dist = 0.0;
Double min_dist = 100.0;
for (int i = 0; i < matchesList.size(); i++) {
Double dist = (double) matchesList.get(i).distance;
if (dist < min_dist)
min_dist = dist;
if (dist > max_dist)
max_dist = dist;
}
LinkedList<DMatch> good_matches = new LinkedList<DMatch>();
for (int i = 0; i < matchesList.size(); i++) {
if (matchesList.get(i).distance <= (1.5 * min_dist))
good_matches.addLast(matchesList.get(i));
}
MatOfDMatch goodMatches = new MatOfDMatch();
goodMatches.fromList(good_matches);
Mat outputImg = new Mat();
MatOfByte drawnMatches = new MatOfByte();
if (aInputFrame.empty() || aInputFrame.cols() < 1 || aInputFrame.rows() < 1) {
return aInputFrame;
}
Features2d.drawMatches(img1, keypoints1, aInputFrame, keypoints2, goodMatches, outputImg, GREEN, RED, drawnMatches, Features2d.NOT_DRAW_SINGLE_POINTS);
Imgproc.resize(outputImg, outputImg, aInputFrame.size());
return outputImg;
}
public Mat onCameraFrame(CvCameraViewFrame inputFrame) {
return recognize(inputFrame.rgba());
}
}You can create the Main Activity copying the above code. But I would like to point out few important factors in the above code. After all we are software engineers not code monkeys ;)
First you would notice is that MainActivity is extending CameraBridgeViewBase.CvCameraViewListener2 . This interface would enforce us to implement few methods which are related to the camera. The method which is important to us is the onCameraFrame(CvCameraViewFrame inputFrame). This would receive the video as frames and you can do all the image processing inside this method and return a Mat with that image :)
There are three main variables
FeatureDetector detector;
DescriptorExtractor descriptor;
DescriptorMatcher matcher;detector is used to detector features and the descriptor will compute the descriptors and the matcher would match the descriptors.
Later in the code you can find there are distance values given which are to set the level of threshold that you want the feature matching has to happen, after that in the code best matches have been sorted out and drawn using Features2d.drawMatches() method. That image has been returned to the camera view.
6. Image preprocessing
It is important to have the input image and the image that you receive from the camera has the same dimensions. You definitely can change those values in the code but for now I am changing that manually before I insert the image to the assets folder in the project. if there is a mismatch between your image’s dimensions, image’s type you will get few errors. I got them and I solved them using the following two links.
Then you can create an assets folder on android and then you can copy paste the image to that folder. You can follow this link to create an assets folder inside your project.
7. AndroidManifest.XML file
This is the file which manages permissions and controls which layout loaded first etc.You don’t have to copy paste this, but make sure to add camera permission and also the it’s features in this file so the application can use the camera on the phone.
<?xml version="1.0" encoding="utf-8"?>
<manifest xmlns:android="http://schemas.android.com/apk/res/android"
package="com.example.akshika.opencvtest">
<application
android:allowBackup="true"
android:icon="@mipmap/ic_launcher"
android:label="@string/app_name"
android:supportsRtl="true"
android:theme="@style/AppTheme">
<activity android:name="com.example.akshika.opencvandroidtut.MainActivity">
<intent-filter>
<action android:name="android.intent.action.MAIN" />
<category android:name="android.intent.category.LAUNCHER" />
</intent-filter>
</activity><!-- ATTENTION: This was auto-generated to add Google Play services to your project for
App Indexing. See https://g.co/AppIndexing/AndroidStudio for more information. -->
<meta-data
android:name="com.google.android.gms.version"
android:value="@integer/google_play_services_version" />
</application>
<supports-screens
android:anyDensity="true"
android:largeScreens="true"
android:normalScreens="true"
android:resizeable="true"
android:smallScreens="true" />
<uses-sdk android:minSdkVersion="8" />
<uses-permission android:name="android.permission.CAMERA" />
<uses-feature
android:name="android.hardware.camera"
android:required="false" />
<uses-feature
android:name="android.hardware.camera.autofocus"
android:required="false" />
<uses-feature
android:name="android.hardware.camera.front"
android:required="false" />
<uses-feature
android:name="android.hardware.camera.front.autofocus"
android:required="false" />
</manifest>7. Sample
Sorry about the background music, I will upload a better video later :)
8. Git Hub Link : https://github.com/akshika47/OpencvAndroid
Please do leave your comments and feedback :)
