Project import generated by Copybara.

GitOrigin-RevId: 0517756260533d374df93679965ca662d0ec6943
2020-01-10 13:13:24 -08:00
parent 38ee2603a7
commit ae6be10afe
192 changed files with 16410 additions and 592 deletions
@@ -0,0 +1,321 @@
+# Saliency-Aware Video Cropping using AutoFlip
+
+## Introduction
+
+AutoFlip is an automatic video cropping pipeline built on top of MediaPipe. This
+example focuses on demonstrating how to use AutoFlip to convert an input video
+to arbitrary aspect ratios.
+
+For overall context on AutoFlip, please read this
+[Google AI Blog](https://mediapipe.page.link/autoflip).
+
+![graph is_required](images/autoflip_edited_example.gif)
+
+## Building
+
+Run the following command to build the AutoFlip pipeline:
+
+```bash
+bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/autoflip:run_autoflip
+```
+
+## Running
+
+```bash
+GLOG_logtostderr=1 bazel-bin/mediapipe/examples/desktop/autoflip/run_autoflip \
+  --calculator_graph_config_file=mediapipe/examples/desktop/autoflip/autoflip_graph.pbtxt \
+  --input_side_packets=input_video_path=/absolute/path/to/the/local/video/file,\
+    output_video_path=/absolute/path/to/save/the/output/video/file,\
+    aspect_ratio=9:16
+```
+
+Use the `aspect_ratio` flag to provide the output aspect ratio. The format
+should be `weight:height`, where the `weight` and `height` are two positive
+integers. AutoFlip supports both landscape-to-portrait and portrait-to-landscape
+conversions. The pipeline internally compares the target aspect ratio against
+the original one, and determines the correct conversion automatically.
+
+We have put a couple test videos under this
+[Google Drive folder](https://drive.google.com/corp/drive/u/0/folders/1KK9LV--Ey0UEVpxssVLhVl7dypgJSQgk).
+You could download the videos into your local file system, then modify the
+command above accordingly to run AutoFlip against the videos.
+
+## MediaPipe Graph
+
+![graph visualization](images/autoflip_graph.png)
+
+To visualize the graph as shown above, copy the text specification of the graph
+below and paste it into [MediaPipe Visualizer](https://viz.mediapipe.dev).
+
+```bash
+# Autoflip graph that only renders the final cropped video. For use with
+# end user applications.
+max_queue_size: -1
+
+# VIDEO_PREP: Decodes an input video file into images and a video header.
+node {
+  calculator: "OpenCvVideoDecoderCalculator"
+  input_side_packet: "INPUT_FILE_PATH:input_video_path"
+  output_stream: "VIDEO:video_raw"
+  output_stream: "VIDEO_PRESTREAM:video_header"
+  output_side_packet: "SAVED_AUDIO_PATH:audio_path"
+}
+
+# VIDEO_PREP: Scale the input video before feature extraction.
+node {
+  calculator: "ScaleImageCalculator"
+  input_stream: "FRAMES:video_raw"
+  input_stream: "VIDEO_HEADER:video_header"
+  output_stream: "FRAMES:video_frames_scaled"
+  node_options: {
+    [type.googleapis.com/mediapipe.ScaleImageCalculatorOptions]: {
+      preserve_aspect_ratio: true
+      output_format: SRGB
+      target_width: 480
+      algorithm: DEFAULT_WITHOUT_UPSCALE
+    }
+  }
+}
+
+# VIDEO_PREP: Create a low frame rate stream for feature extraction.
+node {
+  calculator: "PacketThinnerCalculator"
+  input_stream: "video_frames_scaled"
+  output_stream: "video_frames_scaled_downsampled"
+  node_options: {
+    [type.googleapis.com/mediapipe.PacketThinnerCalculatorOptions]: {
+      thinner_type: ASYNC
+      period: 500000
+    }
+  }
+}
+
+# DETECTION: find borders around the video and major background color.
+node {
+  calculator: "BorderDetectionCalculator"
+  input_stream: "VIDEO:video_raw"
+  output_stream: "DETECTED_BORDERS:borders"
+}
+
+# DETECTION: find shot/scene boundaries on the full frame rate stream.
+node {
+  calculator: "ShotBoundaryCalculator"
+  input_stream: "VIDEO:video_frames_scaled"
+  output_stream: "IS_SHOT_CHANGE:shot_change"
+  options {
+    [type.googleapis.com/mediapipe.autoflip.ShotBoundaryCalculatorOptions] {
+      min_shot_span: 0.2
+      min_motion: 0.3
+      window_size: 15
+      min_shot_measure: 10
+      min_motion_with_shot_measure: 0.05
+    }
+  }
+}
+
+# DETECTION: find faces on the down sampled stream
+node {
+  calculator: "AutoFlipFaceDetectionSubgraph"
+  input_stream: "VIDEO:video_frames_scaled_downsampled"
+  output_stream: "DETECTIONS:face_detections"
+}
+node {
+  calculator: "FaceToRegionCalculator"
+  input_stream: "VIDEO:video_frames_scaled_downsampled"
+  input_stream: "FACES:face_detections"
+  output_stream: "REGIONS:face_regions"
+}
+
+# DETECTION: find objects on the down sampled stream
+node {
+  calculator: "AutoFlipObjectDetectionSubgraph"
+  input_stream: "VIDEO:video_frames_scaled_downsampled"
+  output_stream: "DETECTIONS:object_detections"
+}
+node {
+  calculator: "LocalizationToRegionCalculator"
+  input_stream: "DETECTIONS:object_detections"
+  output_stream: "REGIONS:object_regions"
+  options {
+    [type.googleapis.com/mediapipe.autoflip.LocalizationToRegionCalculatorOptions] {
+      output_all_signals: true
+    }
+  }
+}
+
+# SIGNAL FUSION: Combine detections (with weights) on each frame
+node {
+  calculator: "SignalFusingCalculator"
+  input_stream: "shot_change"
+  input_stream: "face_regions"
+  input_stream: "object_regions"
+  output_stream: "salient_regions"
+  options {
+    [type.googleapis.com/mediapipe.autoflip.SignalFusingCalculatorOptions] {
+      signal_settings {
+        type { standard: FACE_CORE_LANDMARKS }
+        min_score: 0.85
+        max_score: 0.9
+        is_required: false
+      }
+      signal_settings {
+        type { standard: FACE_ALL_LANDMARKS }
+        min_score: 0.8
+        max_score: 0.85
+        is_required: false
+      }
+      signal_settings {
+        type { standard: FACE_FULL }
+        min_score: 0.8
+        max_score: 0.85
+        is_required: false
+      }
+      signal_settings {
+        type: { standard: HUMAN }
+        min_score: 0.75
+        max_score: 0.8
+        is_required: false
+      }
+      signal_settings {
+        type: { standard: PET }
+        min_score: 0.7
+        max_score: 0.75
+        is_required: false
+      }
+      signal_settings {
+        type: { standard: CAR }
+        min_score: 0.7
+        max_score: 0.75
+        is_required: false
+      }
+      signal_settings {
+        type: { standard: OBJECT }
+        min_score: 0.1
+        max_score: 0.2
+        is_required: false
+      }
+    }
+  }
+}
+
+# CROPPING: make decisions about how to crop each frame.
+node {
+  calculator: "SceneCroppingCalculator"
+  input_side_packet: "EXTERNAL_ASPECT_RATIO:aspect_ratio"
+  input_stream: "VIDEO_FRAMES:video_raw"
+  input_stream: "KEY_FRAMES:video_frames_scaled_downsampled"
+  input_stream: "DETECTION_FEATURES:salient_regions"
+  input_stream: "STATIC_FEATURES:borders"
+  input_stream: "SHOT_BOUNDARIES:shot_change"
+  output_stream: "CROPPED_FRAMES:cropped_frames"
+  node_options: {
+    [type.googleapis.com/mediapipe.autoflip.SceneCroppingCalculatorOptions]: {
+      max_scene_size: 600
+      key_frame_crop_options: {
+        score_aggregation_type: CONSTANT
+      }
+      scene_camera_motion_analyzer_options: {
+        motion_stabilization_threshold_percent: 0.3
+        salient_point_bound: 0.499
+      }
+      padding_parameters: {
+        blur_cv_size: 200
+        overlay_opacity: 0.6
+      }
+      target_size_type: MAXIMIZE_TARGET_DIMENSION
+    }
+  }
+}
+
+# ENCODING(required): encode the video stream for the final cropped output.
+node {
+  calculator: "VideoPreStreamCalculator"
+  # Fetch frame format and dimension from input frames.
+  input_stream: "FRAME:cropped_frames"
+  # Copying frame rate and duration from original video.
+  input_stream: "VIDEO_PRESTREAM:video_header"
+  output_stream: "output_frames_video_header"
+}
+
+node {
+  calculator: "OpenCvVideoEncoderCalculator"
+  input_stream: "VIDEO:cropped_frames"
+  input_stream: "VIDEO_PRESTREAM:output_frames_video_header"
+  input_side_packet: "OUTPUT_FILE_PATH:output_video_path"
+  input_side_packet: "AUDIO_FILE_PATH:audio_path"
+  node_options: {
+    [type.googleapis.com/mediapipe.OpenCvVideoEncoderCalculatorOptions]: {
+      codec: "avc1"
+      video_format: "mp4"
+    }
+  }
+}
+```
+
+## Advanced Parameters
+
+### Required vs. Best-Effort Saliency Features
+
+AutoFlip allows users to implement and specify custom features to be used in the
+camera trajectory computation. If the user would like to detect and preserve
+scenes of lions in a wildlife protection video, for example, they could
+implement and add a feature detection calculator for lions into the pipeline.
+Refer to `AutoFlipFaceDetectionSubgraph` and `FaceToRegionCalculator`, or
+`AutoFlipObjectDetectionSubgraph` and `LocalizationToRegionCalculator` for
+examples of how to create new feature detection calculators.
+
+After adding different feature signals into the graph, use the
+`SignalFusingCalculator` node to specify types and weights for different feature
+signals. For example, in the graph above, we specified a `face_region` and an
+`object_region` input streams, to represent face signals and agnostic object
+signals, respectively.
+
+The larger the weight, the more important the features will be considered when
+AutoFlip computes the camera trajectory. Use the `is_required` flag to mark a
+feature as a hard constraint, in which case the computed camera trajectory will
+try best to cover these feature types in the cropped videos. If for some reason
+the required features cannot be all covered (for example, when they are too
+spread out in the video), AutoFlip will apply a padding effect to cover as much
+salient content as possible. See an illustration below.
+
+![graph is_required](images/autoflip_is_required.gif)
+
+### Visualization to Facilitate Debugging
+
+`SceneCroppingCalculator` provides two extra output streams
+`KEY_FRAME_CROP_REGION_VIZ_FRAMES` and `SALIENT_POINT_FRAME_VIZ_FRAMES` to
+visualize the cropping window as well as salient points detected on each frame.
+You could modify the `SceneCroppingCalculator` node like below to enable these
+two output streams.
+
+```bash
+node {
+  calculator: "SceneCroppingCalculator"
+  input_side_packet: "EXTERNAL_ASPECT_RATIO:aspect_ratio"
+  input_stream: "VIDEO_FRAMES:video_raw"
+  input_stream: "KEY_FRAMES:video_frames_scaled_downsampled"
+  input_stream: "DETECTION_FEATURES:salient_regions"
+  input_stream: "STATIC_FEATURES:borders"
+  input_stream: "SHOT_BOUNDARIES:shot_change"
+  output_stream: "CROPPED_FRAMES:cropped_frames"
+  output_stream: "KEY_FRAME_CROP_REGION_VIZ_FRAMES:key_frame_crop_viz_frames"
+  output_stream: "SALIENT_POINT_FRAME_VIZ_FRAMES:salient_point_viz_frames"
+  node_options: {
+    [type.googleapis.com/mediapipe.autoflip.SceneCroppingCalculatorOptions]: {
+      max_scene_size: 600
+      key_frame_crop_options: {
+        score_aggregation_type: CONSTANT
+      }
+      scene_camera_motion_analyzer_options: {
+        motion_stabilization_threshold_percent: 0.3
+        salient_point_bound: 0.499
+      }
+      padding_parameters: {
+        blur_cv_size: 200
+        overlay_opacity: 0.6
+      }
+      target_size_type: MAXIMIZE_TARGET_DIMENSION
+    }
+  }
+}
+```
@@ -151,7 +151,6 @@ node {
   input_stream: "room_video_tick_signal"
   output_stream: "cloned_room_mic_signal"
   output_stream: "cloned_lighting_sensor"
-   output_stream: "cloned_video_tick_signal"
 }
 ```

@@ -108,9 +108,9 @@ and model details are described in the
 [Hello World for C++](./hello_world_desktop.md) shows how to run a simple graph
 using the MediaPipe C++ APIs.

-### Feature Extration for YouTube-8M Challenge
+### Feature Extraction and Model Inference for YouTube-8M Challenge

-[Feature Extration and Model Inference for YouTube-8M Challenge](./youtube_8m.md)
+[Feature Extraction and Model Inference for YouTube-8M Challenge](./youtube_8m.md)
 shows how to use MediaPipe to prepare training data for the YouTube-8M Challenge
 and do the model inference with the baseline model.

@@ -120,6 +120,11 @@ and do the model inference with the baseline model.
 MediaPipe for media processing to prepare video data sets for training a
 TensorFlow model.

+### Automatic video cropping
+
+[AutoFlip](./autoflip.md) shows how to use MediaPipe to build an automatic video
+cropping pipeline that can convert an input video to arbitrary aspect ratios.
+
 ### Object Detection on Desktop

 [Object Detection on Desktop](./object_detection_desktop.md) shows how to run
@@ -2,9 +2,10 @@

 MediaPipe is able to run cross platform across device types like desktop, mobile
 and edge devices. Here is an example of running MediaPipe
-[face detection pipeline](./face_detection_desktop.md) on edge device like
-[Google Coral dev board](https://coral.withgoogle.com/products/dev-board) with
-[Edge TPU](https://cloud.google.com/edge-tpu/). This MediaPipe Coral face
+[face detection pipeline](./face_detection_desktop.md) on edge device like the
+[Coral Dev Board](https://coral.ai/products/dev-board).
+
+This MediaPipe Coral face
 detection pipeline is running [coral specific quantized version](https://github.com/google/mediapipe/blob/master/mediapipe/examples/coral/models/face-detector-quantized_edgetpu.tflite)
 of the [MediaPipe face detection TFLite model](https://github.com/google/mediapipe/blob/master/mediapipe/models/face_detection_front.tflite)
 accelerated on Edge TPU.
@@ -13,8 +14,10 @@ accelerated on Edge TPU.

 We recommend building the MediaPipe binaries not on the edge device due to
 limited compute resulting in long build times. Instead, we will build MediaPipe
-binaries using Docker containers on a more powerful host machine. For step by
-step details of cross compiling and running MediaPipe binaries on Coral dev
-board, please refer to [README.md in MediaPipe Coral example folder](https://github.com/google/mediapipe/blob/master/mediapipe/examples/coral/README.md).
+binaries using Docker containers on a more powerful host machine.

-![Face Detection running on Coral](images/face_detection_demo_coral.jpg)
+For step by
+step details of cross compiling and running MediaPipe binaries on the Coral Dev
+Board, please refer to [README.md in MediaPipe Coral example folder](https://github.com/google/mediapipe/tree/master/mediapipe/examples/coral).
+
+![Face Detection running on Coral](images/face_detection_demo_coral.gif)
@@ -24,9 +24,9 @@ Mediapipe. At a minimum, a new calculator must implement the below four methods
 * `GetContract()`
  * Calculator authors can specify the expected types of inputs and outputs of a calculator in GetContract(). When a graph is initialized, the framework calls a static method to verify if the packet types of the connected inputs and outputs match the information in this specification.
 * `Open()`
-  * After a graph starts, the framework calls `Open()`. The input side packets are available to the calculator at this point. `Open()` interprets the node configuration (see Section \ref{graph_config}) operations and prepares the calculator's per-graph-run state. This function may also write packets to calculator outputs. An error during `Open()` can terminate the graph run.
+  * After a graph starts, the framework calls `Open()`. The input side packets are available to the calculator at this point. `Open()` interprets the node configuration operations (see Section [GraphConfig](#graphconfig)) and prepares the calculator's per-graph-run state. This function may also write packets to calculator outputs. An error during `Open()` can terminate the graph run.
 * `Process()`
-  * For a calculator with inputs, the framework calls `Process()` repeatedly whenever at least one input stream has a packet available. The framework by default guarantees that all inputs have the same timestamp (see Section \ref{scheduling} for more information). Multiple `Process()` calls can be invoked simultaneously when parallel execution is enabled. If an error occurs during `Process()`, the framework calls `Close()` and the graph run terminates.
+  * For a calculator with inputs, the framework calls `Process()` repeatedly whenever at least one input stream has a packet available. The framework by default guarantees that all inputs have the same timestamp (see [Framework Architecture](scheduling_sync.md) for more information). Multiple `Process()` calls can be invoked simultaneously when parallel execution is enabled. If an error occurs during `Process()`, the framework calls `Close()` and the graph run terminates.
 * `Close()`
  * After all calls to `Process()` finish or when all input streams close, the framework calls `Close()`. This function is always called if `Open()` was called and succeeded and even if the graph run terminated because of an error. No inputs are available via any input streams during `Close()`, but it still has access to input side packets and therefore may write outputs. After `Close()` returns, the calculator should be considered a dead node. The calculator object is destroyed as soon as the graph finishes running.

@@ -39,19 +39,12 @@ To build and run iOS apps:
    $ cd mediapipe
    ```

-2.  Install Bazel (0.24.1 and above required).
+2.  Install Bazel (version between 0.24.1 and 1.2.1).

-    Option 1. Use package manager tool to install the latest version of Bazel.
-
-    ```bash
-    $ sudo apt-get install bazel
-
-    # Run 'bazel version' to check version of bazel installed
-    ```
-
-    Option 2. Follow the official
+    Follow the official
    [Bazel documentation](https://docs.bazel.build/versions/master/install-ubuntu.html)
-    to install any version of Bazel manually.
+    to install Bazel manually. Note that MediaPipe doesn't support Bazel 2.0.0+
+    yet.

 3.  Install OpenCV and FFmpeg.

@@ -158,11 +151,12 @@ To build and run iOS apps:
    $ cd mediapipe
    ```

-2.  Install Bazel (0.24.1 and above required).
+2.  Install Bazel (version between 0.24.1 and 1.2.1).

    Follow the official
    [Bazel documentation](https://docs.bazel.build/versions/master/install-redhat.html)
-    to install Bazel manually.
+    to install Bazel manually. Note that MediaPipe doesn't support Bazel 2.0.0+
+    yet.

 3.  Install OpenCV.

@@ -361,6 +355,9 @@ To build and run iOS apps:

 ### Installing on Windows Subsystem for Linux (WSL)

+Note: WSL has historically not provided access to USB cameras. Mediapipe can use
+a video file as input.
+
 1.  Follow the
    [instruction](https://docs.microsoft.com/en-us/windows/wsl/install-win10) to
    install Windows Sysystem for Linux (Ubuntu).
@@ -383,7 +380,7 @@ To build and run iOS apps:
    username@DESKTOP-TMVLBJ1:~$ sudo apt-get update && sudo apt-get install -y build-essential git python zip adb openjdk-8-jdk
    ```

-5.  Install Bazel (0.24.1 and above required).
+5.  Install Bazel (version between 0.24.1 and 1.2.1).

    ```bash
    username@DESKTOP-TMVLBJ1:~$ curl -sLO --retry 5 --retry-max-time 10 \
@@ -3,8 +3,9 @@
 MediaPipe is able to run cross platform across device types like desktop, mobile
 and edge devices. Here is an example of running MediaPipe
 [object detection pipeline](./object_detection_desktop.md) on edge device like
-[Google Coral dev board](https://coral.withgoogle.com/products/dev-board) with
-[Edge TPU](https://cloud.google.com/edge-tpu/). This MediaPipe Coral object
+the [Coral Dev Board](https://coral.ai/products/dev-board).
+
+This MediaPipe Coral object
 detection pipeline is running [coral specific quantized version](https://github.com/google/mediapipe/blob/master/mediapipe/examples/coral/models/object-detector-quantized_edgetpu.tflite)
 of the [MediaPipe object detection TFLite model](https://github.com/google/mediapipe/blob/master/mediapipe/models/object_detection_front.tflite)
 accelerated on Edge TPU.
@@ -13,8 +14,10 @@ accelerated on Edge TPU.

 We recommend building the MediaPipe binaries not on the edge device due to
 limited compute resulting in long build times. Instead, we will build MediaPipe
-binaries using Docker containers on a more powerful host machine. For step by
-step details of cross compiling and running MediaPipe binaries on Coral dev
-board, please refer to [README.md in MediaPipe Coral example folder](https://github.com/google/mediapipe/blob/master/mediapipe/examples/coral/README.md).
+binaries using Docker containers on a more powerful host machine.
+
+For step by
+step details of cross compiling and running MediaPipe binaries on the Coral Dev
+Board, please refer to [README.md in MediaPipe Coral example folder](https://github.com/google/mediapipe/tree/master/mediapipe/examples/coral).

 ![Object Detection running on Coral](images/object_detection_demo_coral.jpg)
@@ -6,7 +6,7 @@ that performs object detection and tracking.
 Note that object detection is using TensorFlow Lite on GPU while tracking is using CPU.

 For overall context on object detection and tracking, please read this
-[Google Developer Blog](https://mediapipe.page.link/objecttrackingblog).
+[Google Developers Blog](https://mediapipe.page.link/objecttrackingblog).

 ![object_tracking_android_gpu_gif](images/mobile/object_tracking_android_gpu.gif)

@@ -238,7 +238,7 @@ tracking library that can be used for other use cases.

 ![object_tracking_subgraph](images/mobile/object_tracking_subgraph.png)

-[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/tracking/subgraphs/object_tracking.pbtxt)
+[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/tracking/subgraphs/object_tracking_gpu.pbtxt)

 ```bash
 # MediaPipe object tracking subgraph.
@@ -304,7 +304,7 @@ node: {

 ![box_tracking_subgraph](images/mobile/box_tracking_subgraph.png)

-[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/tracking/subgraphs/box_tracking.pbtxt)
+[Source pbtxt file](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/tracking/subgraphs/box_tracking_gpu.pbtxt)

 ```bash
 # MediaPipe box tracking subgraph.
@@ -473,3 +473,19 @@ node {
 }

 ```
+
+## Desktop
+
+[Source](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/object_tracking)
+
+Note that object detection is using TensorFlow Lite on CPU and tracking is using
+CPU.
+
+To build and run the app:
+
+```bash
+bazel build -c opt mediapipe/examples/desktop/object_tracking:object_tracking_cpu
+
+bazel-bin/mediapipe/examples/desktop/object_tracking/object_tracking_cpu \
+  --calculator_graph_config_file=mediapipe/graphs/tracking/object_detection_tracking_desktop_live.pbtxt
+```
@@ -1,4 +1,4 @@
-# Feature Extration and Model Inference for YouTube-8M Challenge
+# Feature Extraction and Model Inference for YouTube-8M Challenge

 MediaPipe is a useful and general framework for media processing that can assist
 with research, development, and deployment of ML models. This example focuses on
@@ -31,7 +31,9 @@ videos.

 ### Steps to run the YouTube-8M feature extraction graph

-1.  Checkout the mediapipe repository.
+1.  Checkout the repository and follow
+    [the installation instructions](https://github.com/google/mediapipe/blob/master/mediapipe/docs/install.md)
+    to set up MediaPipe.

    ```bash
    git clone https://github.com/google/mediapipe.git