Android

Note

If you haven’t set up the SDK yet, make sure to go through those directions first. You’ll need to add the Core library to the app before using the specific feature API or custom model. Follow iOS setup or Android setup directions.

1. Add the dependencies via Gradle

Add our repository in order to download the Vision API:

repositories {
    maven { url "https://fritz.mycloudrepo.io/public/repositories/android" }
}

Add renderscript support and include the vision dependency in app/build.gradle. Renderscript is used in order to improve image processing performance. You’ll also need to specify aaptOptions in order to prevent compressing TensorFlow Lite models.

android {
    defaultConfig {
        renderscriptTargetApi 21
        renderscriptSupportModeEnabled true
    }

    // Don't compress included TensorFlow Lite models on build.
    aaptOptions {
        noCompress "tflite"
    }
}

dependencies {
    implementation 'ai.fritz:vision:+'
}

(Optional include model in your app) To include Object Detection model with your build, then you’ll need to add the dependency as shown below. Note: This includes the model with your app when you publish it to the play store and will increase your app size.

Note

Behind the scenes, Object Detection uses a TensorFlow Lite model. In order to include this with your app, you’ll need to make sure that the model is not compressed in the APK by setting aaptOptions.

dependencies {
  implementation 'ai.fritz:vision-object-detection-model-fast:3.0.0'
}

Now you’re ready to detect objects with the Object Detection API.

2. Create an ObjectDetectionOnDeviceModel

You will first need to define a model and then use a predictor to run it.

If you are using on of the Fritz SDK pre-trained model and have followed the optional step above and included the Object Detection model, you can get a predictor to use immediately:

ObjectDetectionOnDeviceModel onDeviceModel = FritzVisionModels.getObjectDetectionOnDeviceModel();

If you have trained a custom object detection model with Fritz Model Training, you’ll need to download the tflite model file from your model’s detail page in Fritz webapp and add it to the assets folder of your app. You can then define an on-device model as follows:

List<String> labels = Arrays.asList("Background", "List", "Of", "Labels");
ObjectDetectionOnDeviceModel onDeviceModel = new ObjectDetectionOnDeviceModel(
    "file:///android_asset/your_model.tflite",
    "YOUR MODEL ID",
    model_version,
    labels
);

Note that the first item in the list of labels is a “None” or “Background” label for when no object in predicted. The length of the labels list should be N + 1 where N is the number of objects your model predicts.

3. Create an Object Detection Predictor

Now that you have a model, you can create a FritzVisionObjectPredictor to make predictions. Predictors handle necessary pre- and post-processing.

FritzVisionObjectPredictor predictor = FritzVision.ObjectDetection.getPredictor(onDeviceModel);

If you did not include the on-device model in your APK bundle, you’ll have to load the model before you can get a predictor. To do that, you’ll use ObjectDetectionManagedModel object and call FritzVision.ObjectDetection.loadPredictor to start the model download.

FritzVisionObjectPredictor predictor;

ObjectDetectionManagedModel managedModel = FritzVisionModels.getObjectDetectionManagedModel();
FritzVision.ObjectDetection.loadPredictor(managedModel, new PredictorStatusListener<FritzVisionObjectPredictor>() {
    @Override
    public void onPredictorReady(FritzVisionObjectPredictor objectDetectionPredictor) {
        Log.d(TAG, "Object Detection predictor is ready");
        predictor = objectDetectionPredictor;
    }
});

Note

No objects found

If you follow the steps above to implement your custom object detection model and you aren’t seeing any results, consider lowering the confidence threshold for displaying predictions. By default, the Fritz SDK filters out predictions that models are not confident about. If your model is not very accurate, results may have low confidence scores. To decrease the score threshold, create a predictor with the following options:

// Create predictor options
FritzVisionObjectPredictorOptions options = new FritzVisionObjectPredictorOptions()
options.confidenceThreshold = 0.1f;  // A low threshold
FritzVisionObjectPredictor predictor = FritzVision.ObjectDetection.getPredictor(onDeviceModel, options);

Additional predictor options can be found below.

4. Create a FritzVisionImage from an image or a video stream

To create a FritzVisionImage from a Bitmap:

FritzVisionImage visionImage = FritzVisionImage.fromBitmap(bitmap);
var visionImage = FritzVisionImage.fromBitmap(bitmap)

To create a FritzVisionImage from a media.Image object when capturing the result from a camera, first determine the orientation of the image. This will rotate the image to account for device rotation and the orientation of the camera sensor.

// Get the system service for the camera manager
final CameraManager manager = (CameraManager) getSystemService(Context.CAMERA_SERVICE);

// Gets the first camera id
String cameraId = manager.getCameraIdList().get(0);

// Determine the rotation on the FritzVisionImage from the camera orientaion and the device rotation.
// "this" refers to the calling Context (Application, Activity, etc)
ImageRotation imageRotationFromCamera = FritzVisionOrientation.getImageRotationFromCamera(this, cameraId);
// Get the system service for the camera manager
val manager = getSystemService(Context.CAMERA_SERVICE) as CameraManager

// Gets the first camera id
var cameraId = manager.getCameraIdList().get(0)

// Determine the rotation on the FritzVisionImage from the camera orientaion and the device rotation.
// "this" refers to the calling Context (Application, Activity, etc)
var imageRotationFromCamera = FritzVisionOrientation.getImageRotationFromCamera(this, cameraId)

Finally, create the FritzVisionImage object with the rotation

FritzVisionImage visionImage = FritzVisionImage.fromMediaImage(image, imageRotationFromCamera);
val visionImage = FritzVisionImage.fromMediaImage(image, imageRotationFromCamera);

5. Run prediction - Detect different objects in the image

Next, convert a bitmap into a FritzVisionImage and pass the image into the predictor in order to evaluate the objects in the image:

FritzVisionObjectResult objectResult = objectPredictor.predict(visionImage);

The predict method returns back a FritzVisionObjectResult object that contains the following methods:

FritzVisionObjectResult methods
Type Method and Description
List<FritzVisionObject>
getObjects()
Gets a list of all objects detected.
List<FritzVisionObject>
getObjectsAboveThreshold(float confidenceThreshold)
Gets all objects detected above a certain threshold.
List<FritzVisionObject>
getVisionObjectsByClass(String labelName)
Gets all detected objects by a class name (e.g person, cat, etc)
List<FritzVisionObject>
getVisionObjectsByClass(String labelName, float labelConfidence)
Gets all detected objects by a class name (e.g person, cat, etc) and confidence score
List<FritzVisionObject>
getVisionObjectsByClasses(List<String> labelNames)
Gets all objects matching any of the provided label names
List<FritzVisionObject>
getVisionObjectsByClasses(List<String> labelNames, float labelConfidence)
Gets all objects matching any of the provided label names and confidence scores

6. Displaying the result

Create a bitmap with the bounding boxes on the original image:

// Draw the original image that was passed into the predictor
FritzVisionObject visionObject = ...;
Bitmap boundingBoxOnImage = visionImage.overlayBoundingBox(visionObject);

Draw the bounding box on a canvas:

Canvas canvas = ...;
FritzVisionObject visionObject = ...;

// Draws the bounding box on the canvas.
visionObject.draw(canvas);

Configuring the Predictor

You can configure the predictor with FritzVisionObjectPredictorOptions to return specific results that match the options given:

FritzVisionObjectPredictorOptions methods
Option Default Description
confidenceThreshold .6 Return objects detected above the confidence threshold
labels MobileNet labels a list of labels for the model.
iouThreshold .2 Intersection over union (IOU) used to detect overlapping instances and deduping detections. A higher value will requires a larger overlapping area to filter out the detection.

In order to change model performance for different devices, you may also expose the underlying TensorFlow Lite Interpreter options.

FritzVisionPredictorOptions methods
Option Default Description
useGPU false Return labels above the confidence threshold. Please note, this is an experimental option and should not be used in production apps.
useNNAPI false Uses the NNAPI for running model inference. Please note, this is an experimental option and should not be used in production apps.
numThreads <number of available threads> For CPU Only, run model inference using the specified number of threads

For more details, please visit the official TensorFlow Lite website.

  • Example:
// Create predictor options
FritzVisionObjectPredictorOptions options = new FritzVisionObjectPredictorOptions()
options.confidenceThreshold = 0.7f;
options.numThreads = 2;

// Pass in the options when initializing the predictor
FritzVisionObjectPredictor predictor = FritzVision.ObjectDetection.getPredictor(onDeviceModel, options);

7. Use the record method on the predictor to collect data

The FritzVisionObjectPredictor used to make predictions has a record method allowing you to send an image, a model-predicted annotation, and a user-generated annotation back to your Fritz AI account.

FritzVisionObjectResult predictedResults = visionPredictor.predict(visionImage);

// Implement your own custom UX for users to annotate an image and store
// that as a FritzVisionObjectResult.
visionPredictor.record(visionImage, predictedResults.toAnnotations(), modifiedResults.toAnnotations())