Project import generated by Copybara.

GitOrigin-RevId: d0039a576e2db9c0fcefffd26a527df74cbe145b
2020-04-21 22:43:01 -04:00
parent 024f7bf0f1
commit 7bad8fce62
45 changed files with 1566 additions and 227 deletions
@@ -121,6 +121,14 @@ and model details are described in the

 *   [Android](./hair_segmentation_mobile_gpu.md)

+### Template Matching using KNIFT with CPU
+
+[Template Matching using KNIFT on Mobile](./template_matching_mobile_cpu.md)
+shows how to use MediaPipe with TFLite model for template matching using Knift
+on mobile using CPU.
+
+*   [Android](./template_matching_mobile_cpu.md)
+
 ## Desktop

 ### Hello World for C++
@@ -171,7 +179,6 @@ on desktop with webcam input.
 *   [Desktop GPU](./face_mesh_desktop.md)
 *   [Desktop CPU](./face_mesh_desktop.md)

-
 ### Hand Tracking on Desktop with Webcam

 [Hand Tracking on Desktop with Webcam](./hand_tracking_desktop.md) shows how to
@@ -198,7 +205,7 @@ GPU with live video from a webcam.

 *   [Desktop GPU](./hair_segmentation_desktop.md)

-## Google Coral (machine learning acceleration with Google EdgeTPU)
+## Google Coral (ML acceleration with Google EdgeTPU)

 Below are code samples on how to run MediaPipe on Google Coral Dev Board.

@@ -0,0 +1,31 @@
+# Template Matching using KNIFT on Desktop
+
+This doc focuses on the
+[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/template_matching/template_matching_desktop.pbtxt)
+that performs template matching with KNIFT (Keypoint Neural Invariant Feature
+Transform) on desktop CPU.
+
+If you are interested in more detail about KNIFT or running the example on
+mobile, please see
+[Template Matching using KNIFT on Mobile (CPU)](template_matching_mobile_cpu.md).
+
+To build the desktop app, run:
+
+```bash
+$ bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 \
+    mediapipe/examples/desktop/template_matching:template_matching_tflite
+```
+
+To run the desktop app, please specify a template index file
+([example](https://github.com/google/mediapipe/tree/master/mediapipe/models/knift_index.pb)) and a
+video to be matched. For how to build your own index file, please see
+[here](template_matching_mobile_cpu.md#build-index-file).
+
+```bash
+$ GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/template_matching/template_matching_tflite \
+    --calculator_graph_config_file=mediapipe/graphs/template_matching/template_matching_desktop.pbtxt --input_side_packets="input_video_path=<input video path>,output_video_path=<output video path>"
+```
+
+## Graph
+
+[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/template_matching/template_matching_desktop.pbtxt)
@@ -0,0 +1,94 @@
+# Template Matching using KNIFT on Mobile (CPU)
+
+This doc focuses on the
+[example graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/template_matching/template_matching_mobile_cpu.pbtxt)
+that performs template matching with KNIFT (Keypoint Neural Invariant Feature
+Transform) on mobile CPU.
+
+![template_matching_mobile_cpu.gif](images/mobile/template_matching_android_cpu.gif)
+
+In the visualization above, the green dots represent detected keypoints on each
+frame and the red box represents the targets matched by templates using KNIFT
+features (see also [model card](https://mediapipe.page.link/knift-mc)). For more
+information, please see
+[Google Developers Blog](https://mediapipe.page.link/knift-blog).
+
+## Build Index Files
+
+In MediaPipe, we've already provided a file in
+[knift_index.pb](https://github.com/google/mediapipe/tree/master/mediapipe/models/knift_index.pb),
+pre-computed from the 3 template images (of USD bills) shown below. If you'd
+like to use your own template images, please follow the steps below, or
+otherwise you can jump directly to [Android](#android).
+
+![template_matching_mobile_template.jpg](images/mobile/template_matching_mobile_template.jpg)
+
+### Step 1:
+
+Put all template images in a single directory.
+
+### Step 2:
+
+To build the index file for all templates in the directory, run:
+
+```bash
+$ bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 \
+    mediapipe/examples/desktop/template_matching:template_matching_tflite
+$ bazel-bin/mediapipe/examples/desktop/template_matching/template_matching_tflite \
+    --calculator_graph_config_file=mediapipe/graphs/template_matching/index_building.pbtxt \
+    --input_side_packets="file_directory=<template image directory>,file_suffix='png',output_index_filename=<output index filename>"
+```
+
+The output index file includes the extracted KNIFT features.
+
+### Step 3:
+
+Replace
+[mediapipe/models/knift_index.pb](https://github.com/google/mediapipe/tree/master/mediapipe/models/knift_index.pb)
+with the index file you generated, and update
+[mediapipe/models/knift_labelmap.txt](https://github.com/google/mediapipe/tree/master/mediapipe/models/knift_labelmap.txt)
+with your own template names.
+
+## Android
+
+[Source](https://github.com/google/mediapipe/tree/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/templatematchingcpu)
+
+A prebuilt arm64 APK can be
+[downloaded here](https://drive.google.com/open?id=1tSWRfes9rAM4NrzmJBplguNQQvaeBZSa).
+
+To build and install the app yourself, run:
+
+Note: MediaPipe uses OpenCV 3 by default. However, because of
+[issues](https://github.com/opencv/opencv/issues/11488) between NDK 17+ and
+OpenCV 3 when using
+[knnMatch](https://docs.opencv.org/3.4/db/d39/classcv_1_1DescriptorMatcher.html#a378f35c9b1a5dfa4022839a45cdf0e89),
+please use the following commands to temporarily switch to OpenCV 4 for the
+template matching exmaple on Android, and switch back to OpenCV 3 afterwards.
+
+```bash
+# Switch to OpenCV 4
+sed -i -e 's:3.4.3/opencv-3.4.3:4.0.1/opencv-4.0.1:g' WORKSPACE
+sed -i -e 's:libopencv_java3:libopencv_java4:g' third_party/opencv_android.BUILD
+
+# Build and install app
+bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/templatematchingcpu:templatematchingcpu
+adb install -r bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/templatematchingcpu/templatematchingcpu.apk
+
+# Switch back to OpenCV 3
+sed -i -e 's:4.0.1/opencv-4.0.1:3.4.3/opencv-3.4.3:g' WORKSPACE
+sed -i -e 's:libopencv_java4:libopencv_java3:g' third_party/opencv_android.BUILD
+```
+
+## Use XNNPACK Delegate
+
+The example uses XNNPACK delegate by default. Users can change the
+[option in TfLiteInferenceCalculator](https://github.com/google/mediapipe/tree/master/mediapipe/calculators/tflite/tflite_inference_calculator.proto)
+to use default TF Lite inference.
+
+## Graph
+
+### Main Graph
+
+![template_matching_mobile_graph](images/mobile/template_matching_mobile_graph.png)
+
+[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/template_matching/template_matching_mobile_cpu.pbtxt)