On Device Text Recognition using Firebase ML Kit in Android {Scratch Series}

This Google IO 2018, Google announced Firebase ML Kit, which is basically a magic package for mobile developers who are not ML experts to add ML magic into their mobile applications.

Yeah, you heard that right! Our lives just got easier.

Right now, it’s in beta, and can serve the following use-cases :

• Text recognition
• Face detection
• Barcode scanning
• Image labeling
• Landmark recognition

This tutorial extends this Google Codelab to add an image capturing feature using Wonderkiln Camerkit in our Android application that recognizes text in the captured image. For your convenience, we will go through the entire process from scratch starting from creating a new project.

Setting up a basic project

Create a new project in Android Studio. I’ve named mine TextRecognizer. Select the preferred minimum SDK, and Empty Activity template and continue with the rest of the default options and proceed to Finish.

We will use Butterknife by Jake Wharton which is a view injection library that injects views into android activity / fragments using annotations.
In your build.gradle(Module:app) file, under dependencies, add the following 2 lines —

implementation ‘com.jakewharton:butterknife:8.8.1’
annotationProcessor ‘com.jakewharton:butterknife-compiler:8.8.1’

Setting up CameraKit

CameraKit for Android is an extraordinarily easy to use utility to work with the infamous Android Camera and Camera2 APIs.

• In your build.gradle(Module:app) file, under dependencies, add this line — compile ‘com.wonderkiln:camerakit:0.13.1
Sync now.
• In your activity_main.xml add the following code that adds a CameraKit CameraView to the layout along with a button.

• Now that we have our UI, let’s add some logic. In your MainActivity.java , bind the CameraView and Button views using butterknife.

@BindView(R.id.camView) CameraView mCameraView;
@BindView(R.id.cameraBtn) Button mCameraButton;
@Override
protected void onCreate(Bundle savedInstanceState) {
    super.onCreate(savedInstanceState);
    setContentView(R.layout.activity_main);
    ButterKnife.bind(this);
}

• The CameraView has to be started in onResume() and stopped in onPause().

@Override
public void onResume() {
 super.onResume();
 mCameraView.start();
}
@Override
public void onPause() {
 mCameraView.stop();
 super.onPause();
}

• Now we add the CameraKitEventListener in onCreate() which will allow us to listen to the CameraView events

mCameraView.addCameraKitListener(new CameraKitEventListener() {
    @Override
    public void onEvent(CameraKitEvent cameraKitEvent) {

    }

    @Override
    public void onError(CameraKitError cameraKitError) {

    }

    @Override
    public void onImage(CameraKitImage cameraKitImage) {

        Bitmap bitmap = cameraKitImage.getBitmap();
        bitmap = Bitmap.createScaledBitmap(bitmap, mCameraView.getWidth(), mCameraView.getHeight(), false);
    }

    @Override
    public void onVideo(CameraKitVideo cameraKitVideo) {

    }
});

Under the onImage method, we are storing the captured image in a bitmap.

• Now we are adding the Button click listener in onCreate() and on button click, the cameraView will capture the image.

mCameraButton.setOnClickListener(new View.OnClickListener() {
    @Override
    public void onClick(View v) {
    mCameraView.captureImage();
    }
});

So we are almost done with the camera setup. Now onto…

Setting up Firebase ML Kit

• Lets add the app level dependencies first

implementation ‘com.google.firebase:firebase-core:15.0.0’
implementation ‘com.google.firebase:firebase-ml-vision:15.0.0’

• apply plugin: ‘com.google.gms.google-services’ Add this right after the dependencies{...} section at the bottom of the file.
• In your build.gradle(Project: TextRecognizer) file, add this under dependencies classpath ‘com.google.gms:google-services:3.2.0’

Hooking up with Firebase Console

• Go to the Firebase Console
• Select Create New Project, and name your project.
• From the overview screen of your new project, click Add Firebase to your Android app
• Enter your project’s package name and register app. You will find yours in manifest file

package="com.poojab26.textrecognizer">

• Download the google-services.json file and add it under your app folder. Now the project view under app folder will look like this. Sync now.

• Uninstall and reinstall your application in your phone/emulator to run the app with Firebase capabilities.

Setup the Graphic Overlay utils

This part is just for your app to draw graphics over your captured image to display the recognized text in the following way.

Right now we wont go details over the logic behind Graphic Overlay. Let’s keep that for another tutorial or {Scratch Series}

• Now, just create a new package called GraphicUtils in the root folder and create two new Java empty files GraphicOverlay and TextGraphic. The project hierarchy will look like this —

Get the code for GraphicOverlay.java and TextGraphic.java

Note: After adding it to the project, check the TODO in Android Studio.

P.S. For Android noobs, make sure to paste the entire thing under package declaration in your Java files.

Add the Text Recognition logic

In this step, we will add functionality to your app to recognize text in images.

In your activity_main.xml add this code under FrameLayout —

<com.poojab26.textrecognizer.GraphicUtils.GraphicOverlay
    android:id="@+id/graphic_overlay"
    android:layout_width="match_parent"
    android:layout_height="match_parent"
    android:layout_alignParentStart="true"
    android:layout_alignParentTop="true"
    android:layout_alignParentBottom="true"
    android:layout_alignParentLeft="true" />

The package address of your GraphicOverlay will be different for you. Replace com.poojab26.textrecognizer with your package name.

• Bind the GraphicOverlay view into your activity.

@BindView(R.id.graphic_overlay) GraphicOverlay mGraphicOverlay;

• Add the following to the runTextRecognition method into yourMainActivity class:

private void runTextRecognition(Bitmap bitmap) {
    FirebaseVisionImage image = FirebaseVisionImage.fromBitmap(bitmap);
    FirebaseVisionTextDetector detector = FirebaseVision.getInstance()
            .getVisionTextDetector();
    detector.detectInImage(image)
            .addOnSuccessListener(
                    new OnSuccessListener<FirebaseVisionText>() {
                        @Override
                        public void onSuccess(FirebaseVisionText texts) {
                            processTextRecognitionResult(texts);
                        }
                    })
            .addOnFailureListener(
                    new OnFailureListener() {
                        @Override
                        public void onFailure(@NonNull Exception e) {
                            // Task failed with an exception
                            e.printStackTrace();
                        }
                    });
}

The code above configures the text recognition detector and calls the function processTextRecognitionResultwith the response.

• Add the following code to processTextRecognitionResult in the MainActivity class to parse the results and display them in your app.

private void processTextRecognitionResult(FirebaseVisionText texts) {
    List<FirebaseVisionText.Block> blocks = texts.getBlocks();
    if (blocks.size() == 0) {
        Log.d("TAG", "No text found");
        return;
    }
    mGraphicOverlay.clear();
    for (int i = 0; i < blocks.size(); i++) {
        List<FirebaseVisionText.Line> lines = blocks.get(i).getLines();
        for (int j = 0; j < lines.size(); j++) {
            List<FirebaseVisionText.Element> elements = lines.get(j).getElements();
            for (int k = 0; k < elements.size(); k++) {
                GraphicOverlay.Graphic textGraphic = new TextGraphic(mGraphicOverlay, elements.get(k));
                mGraphicOverlay.add(textGraphic);

            }
        }
    }
}

Call the method runTextRecognition under onImage method of camerakit after stopping the cameraview.

@Override
public void onImage(CameraKitImage cameraKitImage) {

    Bitmap bitmap = cameraKitImage.getBitmap();
    bitmap = Bitmap.createScaledBitmap(bitmap, mCameraView.getWidth(), mCameraView.getHeight(), false);
    mCameraView.stop();
    runTextRecognition(bitmap);

}

Now clear the GraphicOverlay and start the cameraView again on button click just before we capture the image. This will start the camera for another image capture.

mCameraButton.setOnClickListener(new View.OnClickListener() {
    @Override
    public void onClick(View v) {
        mGraphicOverlay.clear();
        mCameraView.start();
        mCameraView.captureImage();

    }
});

The entire code of MainActivity.java can be viewed here.

Your app is ready

Yes, that was all. You are ready to test the app. How long did that take you? 30mins? 1 hour? Let me know in the comments.

Anyway, download the entire project from GitHub if you still need some help with the code.

{Scratch Series} is a series I’m developing that will focus on building short weekend projects from scratch.

Clap your hands 👏 as many times as you can to show your support!

Hi, I am Pooja Bhaumik. A creative developer but also a logical designer. You can find me on Linkedin or stalk me on GitHub. Or just drop a mail to pbhaumik26@gmail.com if you wish to talk tech with me.