Compare commits
| Author | SHA1 | Date | |
|---|---|---|---|
 MediaPipe Teamandjqtang
|
374f5e2e7e | ||
|
MediaPipe Teamandchuoling
|
139237092f | ||
|
MediaPipe Teamandchuoling
|
b544a314b3 |
@@ -1,3 +1,9 @@
|
||||
---
|
||||
name: "Build/Installation Issue"
|
||||
about: Use this template for build/installation issues
|
||||
labels: type:build/install
|
||||
|
||||
---
|
||||
<em>Please make sure that this is a build/installation issue and also refer to the [troubleshooting](https://google.github.io/mediapipe/getting_started/troubleshooting.html) documentation before raising any issues.</em>
|
||||
|
||||
**System information** (Please provide as much relevant information as possible)
|
||||
|
||||
@@ -1,3 +1,9 @@
|
||||
---
|
||||
name: "Solution Issue"
|
||||
about: Use this template for assistance with a specific mediapipe solution, such as "Pose" or "Iris", including inference model usage/training, solution-specific calculators, etc.
|
||||
labels: type:support
|
||||
|
||||
---
|
||||
<em>Please make sure that this is a [solution](https://google.github.io/mediapipe/solutions/solutions.html) issue.<em>
|
||||
|
||||
**System information** (Please provide as much relevant information as possible)
|
||||
|
||||
@@ -1,3 +1,9 @@
|
||||
---
|
||||
name: "Documentation Issue"
|
||||
about: Use this template for documentation related issues
|
||||
labels: type:docs
|
||||
|
||||
---
|
||||
Thank you for submitting a MediaPipe documentation issue.
|
||||
The MediaPipe docs are open source! To get involved, read the documentation Contributor Guide
|
||||
## URL(s) with the issue:
|
||||
|
||||
+6
@@ -1,3 +1,9 @@
|
||||
---
|
||||
name: "Bug Issue"
|
||||
about: Use this template for reporting a bug
|
||||
labels: type:bug
|
||||
|
||||
---
|
||||
<em>Please make sure that this is a bug and also refer to the [troubleshooting](https://google.github.io/mediapipe/getting_started/troubleshooting.html), FAQ documentation before raising any issues.</em>
|
||||
|
||||
**System information** (Please provide as much relevant information as possible)
|
||||
+6
@@ -1,3 +1,9 @@
|
||||
---
|
||||
name: "Feature Request"
|
||||
about: Use this template for raising a feature request
|
||||
labels: type:feature
|
||||
|
||||
---
|
||||
<em>Please make sure that this is a feature request.</em>
|
||||
|
||||
**System information** (Please provide as much relevant information as possible)
|
||||
+6
@@ -1,3 +1,9 @@
|
||||
---
|
||||
name: "Other Issue"
|
||||
about: Use this template for any other non-support related issues.
|
||||
labels: type:others
|
||||
|
||||
---
|
||||
This template is for miscellaneous issues not covered by the other issue categories
|
||||
|
||||
For questions on how to work with MediaPipe, or support for problems that are not verified bugs in MediaPipe, please go to [StackOverflow](https://stackoverflow.com/questions/tagged/mediapipe) and [Slack](https://mediapipe.page.link/joinslack) communities.
|
||||
@@ -8,6 +8,7 @@ include README.md
|
||||
include requirements.txt
|
||||
|
||||
recursive-include mediapipe/modules *.tflite *.txt *.binarypb
|
||||
exclude mediapipe/modules/face_detection/face_detection_full_range.tflite
|
||||
exclude mediapipe/modules/objectron/object_detection_3d_chair_1stage.tflite
|
||||
exclude mediapipe/modules/objectron/object_detection_3d_sneakers_1stage.tflite
|
||||
exclude mediapipe/modules/objectron/object_detection_3d_sneakers.tflite
|
||||
|
||||
@@ -55,46 +55,22 @@ See also
|
||||
[MediaPipe Models and Model Cards](https://google.github.io/mediapipe/solutions/models)
|
||||
for ML models released in MediaPipe.
|
||||
|
||||
## MediaPipe in Python
|
||||
|
||||
MediaPipe offers customizable Python solutions as a prebuilt Python package on
|
||||
[PyPI](https://pypi.org/project/mediapipe/), which can be installed simply with
|
||||
`pip install mediapipe`. It also provides tools for users to build their own
|
||||
solutions. Please see
|
||||
[MediaPipe in Python](https://google.github.io/mediapipe/getting_started/python)
|
||||
for more info.
|
||||
|
||||
## MediaPipe on the Web
|
||||
|
||||
MediaPipe on the Web is an effort to run the same ML solutions built for mobile
|
||||
and desktop also in web browsers. The official API is under construction, but
|
||||
the core technology has been proven effective. Please see
|
||||
[MediaPipe on the Web](https://developers.googleblog.com/2020/01/mediapipe-on-web.html)
|
||||
in Google Developers Blog for details.
|
||||
|
||||
You can use the following links to load a demo in the MediaPipe Visualizer, and
|
||||
over there click the "Runner" icon in the top bar like shown below. The demos
|
||||
use your webcam video as input, which is processed all locally in real-time and
|
||||
never leaves your device.
|
||||
|
||||
|
||||
|
||||
* [MediaPipe Face Detection](https://viz.mediapipe.dev/demo/face_detection)
|
||||
* [MediaPipe Iris](https://viz.mediapipe.dev/demo/iris_tracking)
|
||||
* [MediaPipe Iris: Depth-from-Iris](https://viz.mediapipe.dev/demo/iris_depth)
|
||||
* [MediaPipe Hands](https://viz.mediapipe.dev/demo/hand_tracking)
|
||||
* [MediaPipe Hands (palm/hand detection only)](https://viz.mediapipe.dev/demo/hand_detection)
|
||||
* [MediaPipe Pose](https://viz.mediapipe.dev/demo/pose_tracking)
|
||||
* [MediaPipe Hair Segmentation](https://viz.mediapipe.dev/demo/hair_segmentation)
|
||||
|
||||
## Getting started
|
||||
|
||||
Learn how to [install](https://google.github.io/mediapipe/getting_started/install)
|
||||
MediaPipe and
|
||||
[build example applications](https://google.github.io/mediapipe/getting_started/building_examples),
|
||||
and start exploring our ready-to-use
|
||||
[solutions](https://google.github.io/mediapipe/solutions/solutions) that you can
|
||||
further extend and customize.
|
||||
To start using MediaPipe
|
||||
[solutions](https://google.github.io/mediapipe/solutions/solutions) with only a few
|
||||
lines code, see example code and demos in
|
||||
[MediaPipe in Python](https://google.github.io/mediapipe/getting_started/python) and
|
||||
[MediaPipe in JavaScript](https://google.github.io/mediapipe/getting_started/javascript).
|
||||
|
||||
To use MediaPipe in C++, Android and iOS, which allow further customization of
|
||||
the [solutions](https://google.github.io/mediapipe/solutions/solutions) as well as
|
||||
building your own, learn how to
|
||||
[install](https://google.github.io/mediapipe/getting_started/install) MediaPipe and
|
||||
start building example applications in
|
||||
[C++](https://google.github.io/mediapipe/getting_started/cpp),
|
||||
[Android](https://google.github.io/mediapipe/getting_started/android) and
|
||||
[iOS](https://google.github.io/mediapipe/getting_started/ios).
|
||||
|
||||
The source code is hosted in the
|
||||
[MediaPipe Github repository](https://github.com/google/mediapipe), and you can
|
||||
|
||||
@@ -35,8 +35,8 @@ http_archive(
|
||||
|
||||
http_archive(
|
||||
name = "rules_cc",
|
||||
strip_prefix = "rules_cc-master",
|
||||
urls = ["https://github.com/bazelbuild/rules_cc/archive/master.zip"],
|
||||
strip_prefix = "rules_cc-main",
|
||||
urls = ["https://github.com/bazelbuild/rules_cc/archive/main.zip"],
|
||||
)
|
||||
|
||||
http_archive(
|
||||
@@ -242,6 +242,20 @@ http_archive(
|
||||
url = "https://github.com/opencv/opencv/releases/download/3.2.0/opencv-3.2.0-ios-framework.zip",
|
||||
)
|
||||
|
||||
http_archive(
|
||||
name = "stblib",
|
||||
strip_prefix = "stb-b42009b3b9d4ca35bc703f5310eedc74f584be58",
|
||||
sha256 = "13a99ad430e930907f5611325ec384168a958bf7610e63e60e2fd8e7b7379610",
|
||||
urls = ["https://github.com/nothings/stb/archive/b42009b3b9d4ca35bc703f5310eedc74f584be58.tar.gz"],
|
||||
build_file = "@//third_party:stblib.BUILD",
|
||||
patches = [
|
||||
"@//third_party:stb_image_impl.diff"
|
||||
],
|
||||
patch_args = [
|
||||
"-p1",
|
||||
],
|
||||
)
|
||||
|
||||
# You may run setup_android.sh to install Android SDK and NDK.
|
||||
android_ndk_repository(
|
||||
name = "androidndk",
|
||||
@@ -337,8 +351,8 @@ maven_install(
|
||||
"androidx.test.espresso:espresso-core:3.1.1",
|
||||
"com.github.bumptech.glide:glide:4.11.0",
|
||||
"com.google.android.material:material:aar:1.0.0-rc01",
|
||||
"com.google.auto.value:auto-value:1.6.4",
|
||||
"com.google.auto.value:auto-value-annotations:1.6.4",
|
||||
"com.google.auto.value:auto-value:1.8.1",
|
||||
"com.google.auto.value:auto-value-annotations:1.8.1",
|
||||
"com.google.code.findbugs:jsr305:3.0.2",
|
||||
"com.google.flogger:flogger-system-backend:0.3.1",
|
||||
"com.google.flogger:flogger:0.3.1",
|
||||
@@ -369,9 +383,9 @@ http_archive(
|
||||
)
|
||||
|
||||
# Tensorflow repo should always go after the other external dependencies.
|
||||
# 2021-05-27
|
||||
_TENSORFLOW_GIT_COMMIT = "d6bfcdb0926173dbb7aa02ceba5aae6250b8aaa6"
|
||||
_TENSORFLOW_SHA256 = "ec40e1462239d8783d02f76a43412c8f80bac71ea20e41e1b7729b990aad6923"
|
||||
# 2021-06-07
|
||||
_TENSORFLOW_GIT_COMMIT = "700533808e6016dc458bb2eeecfca4babfc482ec"
|
||||
_TENSORFLOW_SHA256 = "b6edd7f4039bfc19f3e77594ecff558ba620091d0dc48181484b3d9085026126"
|
||||
http_archive(
|
||||
name = "org_tensorflow",
|
||||
urls = [
|
||||
|
||||
@@ -262,7 +262,7 @@ specified, appear as literal values in the `node_options` field of the
|
||||
output_stream: "TENSORS:main_model_output"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
|
||||
model_path: "mediapipe/models/active_speaker_detection/audio_visual_model.tflite"
|
||||
model_path: "mediapipe/models/detection_model.tflite"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -272,14 +272,13 @@ The `node_options` field accepts the proto3 syntax. Alternatively, calculator
|
||||
options can be specified in the `options` field using proto2 syntax.
|
||||
|
||||
```
|
||||
node: {
|
||||
calculator: "IntervalFilterCalculator"
|
||||
node {
|
||||
calculator: "TfLiteInferenceCalculator"
|
||||
input_stream: "TENSORS:main_model_input"
|
||||
output_stream: "TENSORS:main_model_output"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.IntervalFilterCalculatorOptions] {
|
||||
intervals {
|
||||
start_us: 20000
|
||||
end_us: 40000
|
||||
}
|
||||
[type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
|
||||
model_path: "mediapipe/models/detection_model.tflite"
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -287,13 +286,26 @@ options can be specified in the `options` field using proto2 syntax.
|
||||
|
||||
Not all calculators accept calcuator options. In order to accept options, a
|
||||
calculator will normally define a new protobuf message type to represent its
|
||||
options, such as `IntervalFilterCalculatorOptions`. The calculator will then
|
||||
options, such as `PacketClonerCalculatorOptions`. The calculator will then
|
||||
read that protobuf message in its `CalculatorBase::Open` method, and possibly
|
||||
also in the `CalculatorBase::GetContract` function or its
|
||||
also in its `CalculatorBase::GetContract` function or its
|
||||
`CalculatorBase::Process` method. Normally, the new protobuf message type will
|
||||
be defined as a protobuf schema using a ".proto" file and a
|
||||
`mediapipe_proto_library()` build rule.
|
||||
|
||||
```
|
||||
mediapipe_proto_library(
|
||||
name = "packet_cloner_calculator_proto",
|
||||
srcs = ["packet_cloner_calculator.proto"],
|
||||
visibility = ["//visibility:public"],
|
||||
deps = [
|
||||
"//mediapipe/framework:calculator_options_proto",
|
||||
"//mediapipe/framework:calculator_proto",
|
||||
],
|
||||
)
|
||||
```
|
||||
|
||||
|
||||
## Example calculator
|
||||
|
||||
This section discusses the implementation of `PacketClonerCalculator`, which
|
||||
|
||||
@@ -92,12 +92,12 @@ each project.
|
||||
and copy
|
||||
[the binary graph](https://github.com/google/mediapipe/blob/master/mediapipe/examples/android/src/java/com/google/mediapipe/apps/facedetectiongpu/BUILD#L41)
|
||||
and
|
||||
[the face detection tflite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_front.tflite).
|
||||
[the face detection tflite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_short_range.tflite).
|
||||
|
||||
```bash
|
||||
bazel build -c opt mediapipe/graphs/face_detection:face_detection_mobile_gpu_binary_graph
|
||||
cp bazel-bin/mediapipe/graphs/face_detection/face_detection_mobile_gpu.binarypb /path/to/your/app/src/main/assets/
|
||||
cp mediapipe/modules/face_detection/face_detection_front.tflite /path/to/your/app/src/main/assets/
|
||||
cp mediapipe/modules/face_detection/face_detection_short_range.tflite /path/to/your/app/src/main/assets/
|
||||
```
|
||||
|
||||
|
||||
@@ -117,7 +117,6 @@ each project.
|
||||
implementation 'com.google.flogger:flogger-system-backend:0.3.1'
|
||||
implementation 'com.google.code.findbugs:jsr305:3.0.2'
|
||||
implementation 'com.google.guava:guava:27.0.1-android'
|
||||
implementation 'com.google.guava:guava:27.0.1-android'
|
||||
implementation 'com.google.protobuf:protobuf-java:3.11.4'
|
||||
// CameraX core library
|
||||
def camerax_version = "1.0.0-beta10"
|
||||
@@ -125,7 +124,7 @@ each project.
|
||||
implementation "androidx.camera:camera-camera2:$camerax_version"
|
||||
implementation "androidx.camera:camera-lifecycle:$camerax_version"
|
||||
// AutoValue
|
||||
def auto_value_version = "1.6.4"
|
||||
def auto_value_version = "1.8.1"
|
||||
implementation "com.google.auto.value:auto-value-annotations:$auto_value_version"
|
||||
annotationProcessor "com.google.auto.value:auto-value:$auto_value_version"
|
||||
}
|
||||
|
||||
Binary file not shown.
+14
-38
@@ -55,46 +55,22 @@ See also
|
||||
[MediaPipe Models and Model Cards](https://google.github.io/mediapipe/solutions/models)
|
||||
for ML models released in MediaPipe.
|
||||
|
||||
## MediaPipe in Python
|
||||
|
||||
MediaPipe offers customizable Python solutions as a prebuilt Python package on
|
||||
[PyPI](https://pypi.org/project/mediapipe/), which can be installed simply with
|
||||
`pip install mediapipe`. It also provides tools for users to build their own
|
||||
solutions. Please see
|
||||
[MediaPipe in Python](https://google.github.io/mediapipe/getting_started/python)
|
||||
for more info.
|
||||
|
||||
## MediaPipe on the Web
|
||||
|
||||
MediaPipe on the Web is an effort to run the same ML solutions built for mobile
|
||||
and desktop also in web browsers. The official API is under construction, but
|
||||
the core technology has been proven effective. Please see
|
||||
[MediaPipe on the Web](https://developers.googleblog.com/2020/01/mediapipe-on-web.html)
|
||||
in Google Developers Blog for details.
|
||||
|
||||
You can use the following links to load a demo in the MediaPipe Visualizer, and
|
||||
over there click the "Runner" icon in the top bar like shown below. The demos
|
||||
use your webcam video as input, which is processed all locally in real-time and
|
||||
never leaves your device.
|
||||
|
||||
|
||||
|
||||
* [MediaPipe Face Detection](https://viz.mediapipe.dev/demo/face_detection)
|
||||
* [MediaPipe Iris](https://viz.mediapipe.dev/demo/iris_tracking)
|
||||
* [MediaPipe Iris: Depth-from-Iris](https://viz.mediapipe.dev/demo/iris_depth)
|
||||
* [MediaPipe Hands](https://viz.mediapipe.dev/demo/hand_tracking)
|
||||
* [MediaPipe Hands (palm/hand detection only)](https://viz.mediapipe.dev/demo/hand_detection)
|
||||
* [MediaPipe Pose](https://viz.mediapipe.dev/demo/pose_tracking)
|
||||
* [MediaPipe Hair Segmentation](https://viz.mediapipe.dev/demo/hair_segmentation)
|
||||
|
||||
## Getting started
|
||||
|
||||
Learn how to [install](https://google.github.io/mediapipe/getting_started/install)
|
||||
MediaPipe and
|
||||
[build example applications](https://google.github.io/mediapipe/getting_started/building_examples),
|
||||
and start exploring our ready-to-use
|
||||
[solutions](https://google.github.io/mediapipe/solutions/solutions) that you can
|
||||
further extend and customize.
|
||||
To start using MediaPipe
|
||||
[solutions](https://google.github.io/mediapipe/solutions/solutions) with only a few
|
||||
lines code, see example code and demos in
|
||||
[MediaPipe in Python](https://google.github.io/mediapipe/getting_started/python) and
|
||||
[MediaPipe in JavaScript](https://google.github.io/mediapipe/getting_started/javascript).
|
||||
|
||||
To use MediaPipe in C++, Android and iOS, which allow further customization of
|
||||
the [solutions](https://google.github.io/mediapipe/solutions/solutions) as well as
|
||||
building your own, learn how to
|
||||
[install](https://google.github.io/mediapipe/getting_started/install) MediaPipe and
|
||||
start building example applications in
|
||||
[C++](https://google.github.io/mediapipe/getting_started/cpp),
|
||||
[Android](https://google.github.io/mediapipe/getting_started/android) and
|
||||
[iOS](https://google.github.io/mediapipe/getting_started/ios).
|
||||
|
||||
The source code is hosted in the
|
||||
[MediaPipe Github repository](https://github.com/google/mediapipe), and you can
|
||||
|
||||
@@ -45,6 +45,15 @@ section.
|
||||
|
||||
Naming style and availability may differ slightly across platforms/languages.
|
||||
|
||||
#### model_selection
|
||||
|
||||
An integer index `0` or `1`. Use `0` to select a short-range model that works
|
||||
best for faces within 2 meters from the camera, and `1` for a full-range model
|
||||
best for faces within 5 meters. For the full-range option, a sparse model is
|
||||
used for its improved inference speed. Please refer to the
|
||||
[model cards](./models.md#face_detection) for details. Default to `0` if not
|
||||
specified.
|
||||
|
||||
#### min_detection_confidence
|
||||
|
||||
Minimum confidence value (`[0.0, 1.0]`) from the face detection model for the
|
||||
@@ -72,6 +81,7 @@ install MediaPipe Python package, then learn more in the companion
|
||||
|
||||
Supported configuration options:
|
||||
|
||||
* [model_selection](#model_selection)
|
||||
* [min_detection_confidence](#min_detection_confidence)
|
||||
|
||||
```python
|
||||
@@ -83,7 +93,7 @@ mp_drawing = mp.solutions.drawing_utils
|
||||
# For static images:
|
||||
IMAGE_FILES = []
|
||||
with mp_face_detection.FaceDetection(
|
||||
min_detection_confidence=0.5) as face_detection:
|
||||
model_selection=1, min_detection_confidence=0.5) as face_detection:
|
||||
for idx, file in enumerate(IMAGE_FILES):
|
||||
image = cv2.imread(file)
|
||||
# Convert the BGR image to RGB and process it with MediaPipe Face Detection.
|
||||
@@ -103,7 +113,7 @@ with mp_face_detection.FaceDetection(
|
||||
# For webcam input:
|
||||
cap = cv2.VideoCapture(0)
|
||||
with mp_face_detection.FaceDetection(
|
||||
min_detection_confidence=0.5) as face_detection:
|
||||
model_selection=0, min_detection_confidence=0.5) as face_detection:
|
||||
while cap.isOpened():
|
||||
success, image = cap.read()
|
||||
if not success:
|
||||
@@ -139,6 +149,7 @@ and the following usage example.
|
||||
|
||||
Supported configuration options:
|
||||
|
||||
* [modelSelection](#model_selection)
|
||||
* [minDetectionConfidence](#min_detection_confidence)
|
||||
|
||||
```html
|
||||
@@ -189,6 +200,7 @@ const faceDetection = new FaceDetection({locateFile: (file) => {
|
||||
return `https://cdn.jsdelivr.net/npm/@mediapipe/[email protected]/${file}`;
|
||||
}});
|
||||
faceDetection.setOptions({
|
||||
modelSelection: 0
|
||||
minDetectionConfidence: 0.5
|
||||
});
|
||||
faceDetection.onResults(onResults);
|
||||
@@ -255,10 +267,6 @@ same configuration as the GPU pipeline, runs entirely on CPU.
|
||||
* Target:
|
||||
[`mediapipe/examples/desktop/face_detection:face_detection_gpu`](https://github.com/google/mediapipe/tree/master/mediapipe/examples/desktop/face_detection/BUILD)
|
||||
|
||||
### Web
|
||||
|
||||
Please refer to [these instructions](../index.md#mediapipe-on-the-web).
|
||||
|
||||
### Coral
|
||||
|
||||
Please refer to
|
||||
|
||||
@@ -69,7 +69,7 @@ and renders using a dedicated
|
||||
The
|
||||
[face landmark subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark/face_landmark_front_gpu.pbtxt)
|
||||
internally uses a
|
||||
[face_detection_subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_front_gpu.pbtxt)
|
||||
[face_detection_subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_short_range_gpu.pbtxt)
|
||||
from the
|
||||
[face detection module](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection).
|
||||
|
||||
|
||||
@@ -51,7 +51,14 @@ to visualize its associated subgraphs, please see
|
||||
|
||||
### Web
|
||||
|
||||
Please refer to [these instructions](../index.md#mediapipe-on-the-web).
|
||||
Use [this link](https://viz.mediapipe.dev/demo/hair_segmentation) to load a demo
|
||||
in the MediaPipe Visualizer, and over there click the "Runner" icon in the top
|
||||
bar like shown below. The demos use your webcam video as input, which is
|
||||
processed all locally in real-time and never leaves your device. Please see
|
||||
[MediaPipe on the Web](https://developers.googleblog.com/2020/01/mediapipe-on-web.html)
|
||||
in Google Developers Blog for details.
|
||||
|
||||
|
||||
|
||||
## Resources
|
||||
|
||||
|
||||
@@ -176,6 +176,16 @@ A list of pose landmarks. Each landmark consists of the following:
|
||||
* `visibility`: A value in `[0.0, 1.0]` indicating the likelihood of the
|
||||
landmark being visible (present and not occluded) in the image.
|
||||
|
||||
#### pose_world_landmarks
|
||||
|
||||
Another list of pose landmarks in world coordinates. Each landmark consists of
|
||||
the following:
|
||||
|
||||
* `x`, `y` and `z`: Real-world 3D coordinates in meters with the origin at the
|
||||
center between hips.
|
||||
* `visibility`: Identical to that defined in the corresponding
|
||||
[pose_landmarks](#pose_landmarks).
|
||||
|
||||
#### face_landmarks
|
||||
|
||||
A list of 468 face landmarks. Each landmark consists of `x`, `y` and `z`. `x`
|
||||
@@ -245,6 +255,9 @@ with mp_holistic.Holistic(
|
||||
mp_drawing.draw_landmarks(
|
||||
annotated_image, results.pose_landmarks, mp_holistic.POSE_CONNECTIONS)
|
||||
cv2.imwrite('/tmp/annotated_image' + str(idx) + '.png', annotated_image)
|
||||
# Plot pose world landmarks.
|
||||
mp_drawing.plot_landmarks(
|
||||
results.pose_world_landmarks, mp_holistic.POSE_CONNECTIONS)
|
||||
|
||||
# For webcam input:
|
||||
cap = cv2.VideoCapture(0)
|
||||
|
||||
+12
-2
@@ -69,7 +69,7 @@ and renders using a dedicated
|
||||
The
|
||||
[face landmark subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_landmark/face_landmark_front_gpu.pbtxt)
|
||||
internally uses a
|
||||
[face detection subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_front_gpu.pbtxt)
|
||||
[face detection subgraph](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_short_range_gpu.pbtxt)
|
||||
from the
|
||||
[face detection module](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection).
|
||||
|
||||
@@ -193,7 +193,17 @@ on how to build MediaPipe examples.
|
||||
|
||||
### Web
|
||||
|
||||
Please refer to [these instructions](../index.md#mediapipe-on-the-web).
|
||||
You can use the following links to load a demo in the MediaPipe Visualizer, and
|
||||
over there click the "Runner" icon in the top bar like shown below. The demos
|
||||
use your webcam video as input, which is processed all locally in real-time and
|
||||
never leaves your device. Please see
|
||||
[MediaPipe on the Web](https://developers.googleblog.com/2020/01/mediapipe-on-web.html)
|
||||
in Google Developers Blog for details.
|
||||
|
||||
|
||||
|
||||
* [MediaPipe Iris](https://viz.mediapipe.dev/demo/iris_tracking)
|
||||
* [MediaPipe Iris: Depth-from-Iris](https://viz.mediapipe.dev/demo/iris_depth)
|
||||
|
||||
## Resources
|
||||
|
||||
|
||||
@@ -14,17 +14,27 @@ nav_order: 30
|
||||
|
||||
### [Face Detection](https://google.github.io/mediapipe/solutions/face_detection)
|
||||
|
||||
* Face detection model for front-facing/selfie camera:
|
||||
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_front.tflite),
|
||||
* Short-range model (best for faces within 2 meters from the camera):
|
||||
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_short_range.tflite),
|
||||
[TFLite model quantized for EdgeTPU/Coral](https://github.com/google/mediapipe/tree/master/mediapipe/examples/coral/models/face-detector-quantized_edgetpu.tflite),
|
||||
[Model card](https://mediapipe.page.link/blazeface-mc)
|
||||
* Face detection model for back-facing camera:
|
||||
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_back.tflite),
|
||||
* Full-range model (dense, best for faces within 5 meters from the camera):
|
||||
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_full_range.tflite),
|
||||
[Model card](https://mediapipe.page.link/blazeface-back-mc)
|
||||
* Face detection model for back-facing camera (sparse):
|
||||
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_back_sparse.tflite),
|
||||
* Full-range model (sparse, best for faces within 5 meters from the camera):
|
||||
[TFLite model](https://github.com/google/mediapipe/tree/master/mediapipe/modules/face_detection/face_detection_full_range_sparse.tflite),
|
||||
[Model card](https://mediapipe.page.link/blazeface-back-sparse-mc)
|
||||
|
||||
Full-range dense and sparse models have the same quality in terms of
|
||||
[F-score](https://en.wikipedia.org/wiki/F-score) however differ in underlying
|
||||
metrics. The dense model is slightly better in
|
||||
[Recall](https://en.wikipedia.org/wiki/Precision_and_recall) whereas the sparse
|
||||
model outperforms the dense one in
|
||||
[Precision](https://en.wikipedia.org/wiki/Precision_and_recall). Speed-wise
|
||||
sparse model is ~30% faster when executing on CPU via
|
||||
[XNNPACK](https://github.com/google/XNNPACK) whereas on GPU the models
|
||||
demonstrate comparable latencies. Depending on your application, you may prefer
|
||||
one over the other.
|
||||
|
||||
### [Face Mesh](https://google.github.io/mediapipe/solutions/face_mesh)
|
||||
|
||||
|
||||
+27
-1
@@ -194,10 +194,23 @@ A list of pose landmarks. Each landmark consists of the following:
|
||||
* `z`: Represents the landmark depth with the depth at the midpoint of hips
|
||||
being the origin, and the smaller the value the closer the landmark is to
|
||||
the camera. The magnitude of `z` uses roughly the same scale as `x`.
|
||||
|
||||
* `visibility`: A value in `[0.0, 1.0]` indicating the likelihood of the
|
||||
landmark being visible (present and not occluded) in the image.
|
||||
|
||||
#### pose_world_landmarks
|
||||
|
||||
*Fig 5. Example of MediaPipe Pose real-world 3D coordinates.* |
|
||||
:-----------------------------------------------------------: |
|
||||
<video autoplay muted loop preload style="height: auto; width: 480px"><source src="../images/mobile/pose_world_landmarks.mp4" type="video/mp4"></video> |
|
||||
|
||||
Another list of pose landmarks in world coordinates. Each landmark consists of
|
||||
the following:
|
||||
|
||||
* `x`, `y` and `z`: Real-world 3D coordinates in meters with the origin at the
|
||||
center between hips.
|
||||
* `visibility`: Identical to that defined in the corresponding
|
||||
[pose_landmarks](#pose_landmarks).
|
||||
|
||||
### Python Solution API
|
||||
|
||||
Please first follow general [instructions](../getting_started/python.md) to
|
||||
@@ -242,6 +255,9 @@ with mp_pose.Pose(
|
||||
mp_drawing.draw_landmarks(
|
||||
annotated_image, results.pose_landmarks, mp_pose.POSE_CONNECTIONS)
|
||||
cv2.imwrite('/tmp/annotated_image' + str(idx) + '.png', annotated_image)
|
||||
# Plot pose world landmarks.
|
||||
mp_drawing.plot_landmarks(
|
||||
results.pose_world_landmarks, mp_pose.POSE_CONNECTIONS)
|
||||
|
||||
# For webcam input:
|
||||
cap = cv2.VideoCapture(0)
|
||||
@@ -294,6 +310,7 @@ Supported configuration options:
|
||||
<meta charset="utf-8">
|
||||
<script src="https://cdn.jsdelivr.net/npm/@mediapipe/camera_utils/camera_utils.js" crossorigin="anonymous"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/@mediapipe/control_utils/control_utils.js" crossorigin="anonymous"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/@mediapipe/drawing_utils/control_utils_3d.js" crossorigin="anonymous"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/@mediapipe/drawing_utils/drawing_utils.js" crossorigin="anonymous"></script>
|
||||
<script src="https://cdn.jsdelivr.net/npm/@mediapipe/pose/pose.js" crossorigin="anonymous"></script>
|
||||
</head>
|
||||
@@ -312,8 +329,15 @@ Supported configuration options:
|
||||
const videoElement = document.getElementsByClassName('input_video')[0];
|
||||
const canvasElement = document.getElementsByClassName('output_canvas')[0];
|
||||
const canvasCtx = canvasElement.getContext('2d');
|
||||
const landmarkContainer = document.getElementsByClassName('landmark-grid-container')[0];
|
||||
const grid = new LandmarkGrid(landmarkContainer);
|
||||
|
||||
function onResults(results) {
|
||||
if (!results.poseLandmarks) {
|
||||
grid.updateLandmarks([]);
|
||||
return;
|
||||
}
|
||||
|
||||
canvasCtx.save();
|
||||
canvasCtx.clearRect(0, 0, canvasElement.width, canvasElement.height);
|
||||
canvasCtx.drawImage(
|
||||
@@ -323,6 +347,8 @@ function onResults(results) {
|
||||
drawLandmarks(canvasCtx, results.poseLandmarks,
|
||||
{color: '#FF0000', lineWidth: 2});
|
||||
canvasCtx.restore();
|
||||
|
||||
grid.updateLandmarks(results.poseWorldLandmarks);
|
||||
}
|
||||
|
||||
const pose = new Pose({locateFile: (file) => {
|
||||
|
||||
@@ -284,6 +284,6 @@ on how to build MediaPipe examples.
|
||||
* Google AI Blog:
|
||||
[Background Features in Google Meet, Powered by Web ML](https://ai.googleblog.com/2020/10/background-features-in-google-meet.html)
|
||||
* [ML Kit Selfie Segmentation API](https://developers.google.com/ml-kit/vision/selfie-segmentation)
|
||||
* [Models and model cards](./models.md#selfie_segmentation)
|
||||
* [Models and model cards](./models.md#selfie-segmentation)
|
||||
* [Web demo](https://code.mediapipe.dev/codepen/selfie_segmentation)
|
||||
* [Python Colab](https://mediapipe.page.link/selfie_segmentation_py_colab)
|
||||
|
||||
@@ -419,6 +419,23 @@ cc_library(
|
||||
alwayslink = 1,
|
||||
)
|
||||
|
||||
cc_test(
|
||||
name = "make_pair_calculator_test",
|
||||
size = "small",
|
||||
srcs = ["make_pair_calculator_test.cc"],
|
||||
deps = [
|
||||
":make_pair_calculator",
|
||||
"//mediapipe/framework:calculator_framework",
|
||||
"//mediapipe/framework:calculator_runner",
|
||||
"//mediapipe/framework:timestamp",
|
||||
"//mediapipe/framework/port:gtest_main",
|
||||
"//mediapipe/framework/port:status",
|
||||
"//mediapipe/framework/tool:validate_type",
|
||||
"//mediapipe/util:packet_test_util",
|
||||
"//mediapipe/util:time_series_test_util",
|
||||
],
|
||||
)
|
||||
|
||||
cc_library(
|
||||
name = "matrix_multiply_calculator",
|
||||
srcs = ["matrix_multiply_calculator.cc"],
|
||||
@@ -933,8 +950,8 @@ cc_test(
|
||||
)
|
||||
|
||||
cc_library(
|
||||
name = "split_normalized_landmark_list_calculator",
|
||||
srcs = ["split_normalized_landmark_list_calculator.cc"],
|
||||
name = "split_landmarks_calculator",
|
||||
srcs = ["split_landmarks_calculator.cc"],
|
||||
visibility = ["//visibility:public"],
|
||||
deps = [
|
||||
":split_vector_calculator_cc_proto",
|
||||
@@ -948,10 +965,10 @@ cc_library(
|
||||
)
|
||||
|
||||
cc_test(
|
||||
name = "split_normalized_landmark_list_calculator_test",
|
||||
srcs = ["split_normalized_landmark_list_calculator_test.cc"],
|
||||
name = "split_landmarks_calculator_test",
|
||||
srcs = ["split_landmarks_calculator_test.cc"],
|
||||
deps = [
|
||||
":split_normalized_landmark_list_calculator",
|
||||
":split_landmarks_calculator",
|
||||
":split_vector_calculator_cc_proto",
|
||||
"//mediapipe/framework:calculator_framework",
|
||||
"//mediapipe/framework:calculator_runner",
|
||||
|
||||
@@ -28,6 +28,10 @@ typedef EndLoopCalculator<std::vector<::mediapipe::NormalizedRect>>
|
||||
EndLoopNormalizedRectCalculator;
|
||||
REGISTER_CALCULATOR(EndLoopNormalizedRectCalculator);
|
||||
|
||||
typedef EndLoopCalculator<std::vector<::mediapipe::LandmarkList>>
|
||||
EndLoopLandmarkListVectorCalculator;
|
||||
REGISTER_CALCULATOR(EndLoopLandmarkListVectorCalculator);
|
||||
|
||||
typedef EndLoopCalculator<std::vector<::mediapipe::NormalizedLandmarkList>>
|
||||
EndLoopNormalizedLandmarkListVectorCalculator;
|
||||
REGISTER_CALCULATOR(EndLoopNormalizedLandmarkListVectorCalculator);
|
||||
|
||||
@@ -0,0 +1,70 @@
|
||||
// Copyright 2021 The MediaPipe Authors.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
#include "mediapipe/framework/calculator_framework.h"
|
||||
#include "mediapipe/framework/calculator_runner.h"
|
||||
#include "mediapipe/framework/port/canonical_errors.h"
|
||||
#include "mediapipe/framework/port/gmock.h"
|
||||
#include "mediapipe/framework/port/gtest.h"
|
||||
#include "mediapipe/framework/port/status.h"
|
||||
#include "mediapipe/framework/port/status_matchers.h"
|
||||
#include "mediapipe/framework/timestamp.h"
|
||||
#include "mediapipe/framework/tool/validate_type.h"
|
||||
#include "mediapipe/util/packet_test_util.h"
|
||||
#include "mediapipe/util/time_series_test_util.h"
|
||||
|
||||
namespace mediapipe {
|
||||
|
||||
class MakePairCalculatorTest
|
||||
: public mediapipe::TimeSeriesCalculatorTest<mediapipe::NoOptions> {
|
||||
protected:
|
||||
void SetUp() override {
|
||||
calculator_name_ = "MakePairCalculator";
|
||||
num_input_streams_ = 2;
|
||||
}
|
||||
};
|
||||
|
||||
TEST_F(MakePairCalculatorTest, ProducesExpectedPairs) {
|
||||
InitializeGraph();
|
||||
AppendInputPacket(new std::string("first packet"), Timestamp(1),
|
||||
/* input_index= */ 0);
|
||||
AppendInputPacket(new std::string("second packet"), Timestamp(5),
|
||||
/* input_index= */ 0);
|
||||
AppendInputPacket(new int(10), Timestamp(1), /* input_index= */ 1);
|
||||
AppendInputPacket(new int(20), Timestamp(5), /* input_index= */ 1);
|
||||
|
||||
MP_ASSERT_OK(RunGraph());
|
||||
|
||||
EXPECT_THAT(
|
||||
output().packets,
|
||||
::testing::ElementsAre(
|
||||
mediapipe::PacketContainsTimestampAndPayload<
|
||||
std::pair<Packet, Packet>>(
|
||||
Timestamp(1),
|
||||
::testing::Pair(
|
||||
mediapipe::PacketContainsTimestampAndPayload<std::string>(
|
||||
Timestamp(1), std::string("first packet")),
|
||||
mediapipe::PacketContainsTimestampAndPayload<int>(
|
||||
Timestamp(1), 10))),
|
||||
mediapipe::PacketContainsTimestampAndPayload<
|
||||
std::pair<Packet, Packet>>(
|
||||
Timestamp(5),
|
||||
::testing::Pair(
|
||||
mediapipe::PacketContainsTimestampAndPayload<std::string>(
|
||||
Timestamp(5), std::string("second packet")),
|
||||
mediapipe::PacketContainsTimestampAndPayload<int>(
|
||||
Timestamp(5), 20)))));
|
||||
}
|
||||
|
||||
} // namespace mediapipe
|
||||
+29
-23
@@ -12,8 +12,8 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
#ifndef MEDIAPIPE_CALCULATORS_CORE_SPLIT_NORMALIZED_LANDMARK_LIST_CALCULATOR_H_ // NOLINT
|
||||
#define MEDIAPIPE_CALCULATORS_CORE_SPLIT_NORMALIZED_LANDMARK_LIST_CALCULATOR_H_ // NOLINT
|
||||
#ifndef MEDIAPIPE_CALCULATORS_CORE_SPLIT_LANDMARKS_CALCULATOR_H_ // NOLINT
|
||||
#define MEDIAPIPE_CALCULATORS_CORE_SPLIT_LANDMARKS_CALCULATOR_H_ // NOLINT
|
||||
|
||||
#include "mediapipe/calculators/core/split_vector_calculator.pb.h"
|
||||
#include "mediapipe/framework/calculator_framework.h"
|
||||
@@ -24,29 +24,30 @@
|
||||
|
||||
namespace mediapipe {
|
||||
|
||||
// Splits an input packet with NormalizedLandmarkList into
|
||||
// multiple NormalizedLandmarkList output packets using the [begin, end) ranges
|
||||
// Splits an input packet with LandmarkListType into
|
||||
// multiple LandmarkListType output packets using the [begin, end) ranges
|
||||
// specified in SplitVectorCalculatorOptions. If the option "element_only" is
|
||||
// set to true, all ranges should be of size 1 and all outputs will be elements
|
||||
// of type NormalizedLandmark. If "element_only" is false, ranges can be
|
||||
// non-zero in size and all outputs will be of type NormalizedLandmarkList.
|
||||
// of type LandmarkType. If "element_only" is false, ranges can be
|
||||
// non-zero in size and all outputs will be of type LandmarkListType.
|
||||
// If the option "combine_outputs" is set to true, only one output stream can be
|
||||
// specified and all ranges of elements will be combined into one
|
||||
// NormalizedLandmarkList.
|
||||
class SplitNormalizedLandmarkListCalculator : public CalculatorBase {
|
||||
// LandmarkListType.
|
||||
template <typename LandmarkType, typename LandmarkListType>
|
||||
class SplitLandmarksCalculator : public CalculatorBase {
|
||||
public:
|
||||
static absl::Status GetContract(CalculatorContract* cc) {
|
||||
RET_CHECK(cc->Inputs().NumEntries() == 1);
|
||||
RET_CHECK(cc->Outputs().NumEntries() != 0);
|
||||
|
||||
cc->Inputs().Index(0).Set<NormalizedLandmarkList>();
|
||||
cc->Inputs().Index(0).Set<LandmarkListType>();
|
||||
|
||||
const auto& options =
|
||||
cc->Options<::mediapipe::SplitVectorCalculatorOptions>();
|
||||
|
||||
if (options.combine_outputs()) {
|
||||
RET_CHECK_EQ(cc->Outputs().NumEntries(), 1);
|
||||
cc->Outputs().Index(0).Set<NormalizedLandmarkList>();
|
||||
cc->Outputs().Index(0).Set<LandmarkListType>();
|
||||
for (int i = 0; i < options.ranges_size() - 1; ++i) {
|
||||
for (int j = i + 1; j < options.ranges_size(); ++j) {
|
||||
const auto& range_0 = options.ranges(i);
|
||||
@@ -81,9 +82,9 @@ class SplitNormalizedLandmarkListCalculator : public CalculatorBase {
|
||||
return absl::InvalidArgumentError(
|
||||
"Since element_only is true, all ranges should be of size 1.");
|
||||
}
|
||||
cc->Outputs().Index(i).Set<NormalizedLandmark>();
|
||||
cc->Outputs().Index(i).Set<LandmarkType>();
|
||||
} else {
|
||||
cc->Outputs().Index(i).Set<NormalizedLandmarkList>();
|
||||
cc->Outputs().Index(i).Set<LandmarkListType>();
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -110,40 +111,39 @@ class SplitNormalizedLandmarkListCalculator : public CalculatorBase {
|
||||
}
|
||||
|
||||
absl::Status Process(CalculatorContext* cc) override {
|
||||
const NormalizedLandmarkList& input =
|
||||
cc->Inputs().Index(0).Get<NormalizedLandmarkList>();
|
||||
const LandmarkListType& input =
|
||||
cc->Inputs().Index(0).Get<LandmarkListType>();
|
||||
RET_CHECK_GE(input.landmark_size(), max_range_end_)
|
||||
<< "Max range end " << max_range_end_ << " exceeds landmarks size "
|
||||
<< input.landmark_size();
|
||||
|
||||
if (combine_outputs_) {
|
||||
NormalizedLandmarkList output;
|
||||
LandmarkListType output;
|
||||
for (int i = 0; i < ranges_.size(); ++i) {
|
||||
for (int j = ranges_[i].first; j < ranges_[i].second; ++j) {
|
||||
const NormalizedLandmark& input_landmark = input.landmark(j);
|
||||
const LandmarkType& input_landmark = input.landmark(j);
|
||||
*output.add_landmark() = input_landmark;
|
||||
}
|
||||
}
|
||||
RET_CHECK_EQ(output.landmark_size(), total_elements_);
|
||||
cc->Outputs().Index(0).AddPacket(
|
||||
MakePacket<NormalizedLandmarkList>(output).At(cc->InputTimestamp()));
|
||||
MakePacket<LandmarkListType>(output).At(cc->InputTimestamp()));
|
||||
} else {
|
||||
if (element_only_) {
|
||||
for (int i = 0; i < ranges_.size(); ++i) {
|
||||
cc->Outputs().Index(i).AddPacket(
|
||||
MakePacket<NormalizedLandmark>(input.landmark(ranges_[i].first))
|
||||
MakePacket<LandmarkType>(input.landmark(ranges_[i].first))
|
||||
.At(cc->InputTimestamp()));
|
||||
}
|
||||
} else {
|
||||
for (int i = 0; i < ranges_.size(); ++i) {
|
||||
NormalizedLandmarkList output;
|
||||
LandmarkListType output;
|
||||
for (int j = ranges_[i].first; j < ranges_[i].second; ++j) {
|
||||
const NormalizedLandmark& input_landmark = input.landmark(j);
|
||||
const LandmarkType& input_landmark = input.landmark(j);
|
||||
*output.add_landmark() = input_landmark;
|
||||
}
|
||||
cc->Outputs().Index(i).AddPacket(
|
||||
MakePacket<NormalizedLandmarkList>(output).At(
|
||||
cc->InputTimestamp()));
|
||||
MakePacket<LandmarkListType>(output).At(cc->InputTimestamp()));
|
||||
}
|
||||
}
|
||||
}
|
||||
@@ -159,9 +159,15 @@ class SplitNormalizedLandmarkListCalculator : public CalculatorBase {
|
||||
bool combine_outputs_ = false;
|
||||
};
|
||||
|
||||
typedef SplitLandmarksCalculator<NormalizedLandmark, NormalizedLandmarkList>
|
||||
SplitNormalizedLandmarkListCalculator;
|
||||
REGISTER_CALCULATOR(SplitNormalizedLandmarkListCalculator);
|
||||
|
||||
typedef SplitLandmarksCalculator<Landmark, LandmarkList>
|
||||
SplitLandmarkListCalculator;
|
||||
REGISTER_CALCULATOR(SplitLandmarkListCalculator);
|
||||
|
||||
} // namespace mediapipe
|
||||
|
||||
// NOLINTNEXTLINE
|
||||
#endif // MEDIAPIPE_CALCULATORS_CORE_SPLIT_NORMALIZED_LANDMARK_LIST_CALCULATOR_H_
|
||||
#endif // MEDIAPIPE_CALCULATORS_CORE_SPLIT_LANDMARKS_CALCULATOR_H_
|
||||
@@ -80,6 +80,16 @@ mediapipe_proto_library(
|
||||
],
|
||||
)
|
||||
|
||||
mediapipe_proto_library(
|
||||
name = "segmentation_smoothing_calculator_proto",
|
||||
srcs = ["segmentation_smoothing_calculator.proto"],
|
||||
visibility = ["//visibility:public"],
|
||||
deps = [
|
||||
"//mediapipe/framework:calculator_options_proto",
|
||||
"//mediapipe/framework:calculator_proto",
|
||||
],
|
||||
)
|
||||
|
||||
cc_library(
|
||||
name = "color_convert_calculator",
|
||||
srcs = ["color_convert_calculator.cc"],
|
||||
@@ -602,3 +612,52 @@ cc_test(
|
||||
"//mediapipe/framework/port:parse_text_proto",
|
||||
],
|
||||
)
|
||||
|
||||
cc_library(
|
||||
name = "segmentation_smoothing_calculator",
|
||||
srcs = ["segmentation_smoothing_calculator.cc"],
|
||||
visibility = ["//visibility:public"],
|
||||
deps = [
|
||||
":segmentation_smoothing_calculator_cc_proto",
|
||||
"//mediapipe/framework:calculator_options_cc_proto",
|
||||
"//mediapipe/framework/formats:image_format_cc_proto",
|
||||
"//mediapipe/framework:calculator_framework",
|
||||
"//mediapipe/framework/formats:image_frame",
|
||||
"//mediapipe/framework/formats:image_frame_opencv",
|
||||
"//mediapipe/framework/formats:image",
|
||||
"//mediapipe/framework/formats:image_opencv",
|
||||
"//mediapipe/framework/port:logging",
|
||||
"//mediapipe/framework/port:opencv_core",
|
||||
"//mediapipe/framework/port:status",
|
||||
"//mediapipe/framework/port:vector",
|
||||
] + select({
|
||||
"//mediapipe/gpu:disable_gpu": [],
|
||||
"//conditions:default": [
|
||||
"//mediapipe/gpu:gl_calculator_helper",
|
||||
"//mediapipe/gpu:gl_simple_shaders",
|
||||
"//mediapipe/gpu:gl_quad_renderer",
|
||||
"//mediapipe/gpu:shader_util",
|
||||
],
|
||||
}),
|
||||
alwayslink = 1,
|
||||
)
|
||||
|
||||
cc_test(
|
||||
name = "segmentation_smoothing_calculator_test",
|
||||
srcs = ["segmentation_smoothing_calculator_test.cc"],
|
||||
deps = [
|
||||
":image_clone_calculator",
|
||||
":image_clone_calculator_cc_proto",
|
||||
":segmentation_smoothing_calculator",
|
||||
":segmentation_smoothing_calculator_cc_proto",
|
||||
"//mediapipe/framework:calculator_framework",
|
||||
"//mediapipe/framework:calculator_runner",
|
||||
"//mediapipe/framework/deps:file_path",
|
||||
"//mediapipe/framework/formats:image_frame",
|
||||
"//mediapipe/framework/formats:image_opencv",
|
||||
"//mediapipe/framework/port:gtest_main",
|
||||
"//mediapipe/framework/port:opencv_imgcodecs",
|
||||
"//mediapipe/framework/port:opencv_imgproc",
|
||||
"//mediapipe/framework/port:parse_text_proto",
|
||||
],
|
||||
)
|
||||
|
||||
@@ -0,0 +1,429 @@
|
||||
// Copyright 2021 The MediaPipe Authors.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
#include <algorithm>
|
||||
#include <memory>
|
||||
|
||||
#include "mediapipe/calculators/image/segmentation_smoothing_calculator.pb.h"
|
||||
#include "mediapipe/framework/calculator_framework.h"
|
||||
#include "mediapipe/framework/calculator_options.pb.h"
|
||||
#include "mediapipe/framework/formats/image.h"
|
||||
#include "mediapipe/framework/formats/image_format.pb.h"
|
||||
#include "mediapipe/framework/formats/image_frame.h"
|
||||
#include "mediapipe/framework/formats/image_frame_opencv.h"
|
||||
#include "mediapipe/framework/formats/image_opencv.h"
|
||||
#include "mediapipe/framework/port/logging.h"
|
||||
#include "mediapipe/framework/port/opencv_core_inc.h"
|
||||
#include "mediapipe/framework/port/status.h"
|
||||
#include "mediapipe/framework/port/vector.h"
|
||||
|
||||
#if !MEDIAPIPE_DISABLE_GPU
|
||||
#include "mediapipe/gpu/gl_calculator_helper.h"
|
||||
#include "mediapipe/gpu/gl_simple_shaders.h"
|
||||
#include "mediapipe/gpu/shader_util.h"
|
||||
#endif // !MEDIAPIPE_DISABLE_GPU
|
||||
|
||||
namespace mediapipe {
|
||||
|
||||
namespace {
|
||||
constexpr char kCurrentMaskTag[] = "MASK";
|
||||
constexpr char kPreviousMaskTag[] = "MASK_PREVIOUS";
|
||||
constexpr char kOutputMaskTag[] = "MASK_SMOOTHED";
|
||||
|
||||
enum { ATTRIB_VERTEX, ATTRIB_TEXTURE_POSITION, NUM_ATTRIBUTES };
|
||||
} // namespace
|
||||
|
||||
// A calculator for mixing two segmentation masks together,
|
||||
// based on an uncertantity probability estimate.
|
||||
//
|
||||
// Inputs:
|
||||
// MASK - Image containing the new/current mask.
|
||||
// [ImageFormat::VEC32F1, or
|
||||
// GpuBufferFormat::kBGRA32/kRGB24/kGrayHalf16/kGrayFloat32]
|
||||
// MASK_PREVIOUS - Image containing previous mask.
|
||||
// [Same format as MASK_CURRENT]
|
||||
// * If input channels is >1, only the first channel (R) is used as the mask.
|
||||
//
|
||||
// Output:
|
||||
// MASK_SMOOTHED - Blended mask.
|
||||
// [Same format as MASK_CURRENT]
|
||||
// * The resulting filtered mask will be stored in R channel,
|
||||
// and duplicated in A if 4 channels.
|
||||
//
|
||||
// Options:
|
||||
// combine_with_previous_ratio - Amount of previous to blend with current.
|
||||
//
|
||||
// Example:
|
||||
// node {
|
||||
// calculator: "SegmentationSmoothingCalculator"
|
||||
// input_stream: "MASK:mask"
|
||||
// input_stream: "MASK_PREVIOUS:mask_previous"
|
||||
// output_stream: "MASK_SMOOTHED:mask_smoothed"
|
||||
// options: {
|
||||
// [mediapipe.SegmentationSmoothingCalculatorOptions.ext] {
|
||||
// combine_with_previous_ratio: 0.9
|
||||
// }
|
||||
// }
|
||||
// }
|
||||
//
|
||||
class SegmentationSmoothingCalculator : public CalculatorBase {
|
||||
public:
|
||||
SegmentationSmoothingCalculator() = default;
|
||||
|
||||
static absl::Status GetContract(CalculatorContract* cc);
|
||||
|
||||
// From Calculator.
|
||||
absl::Status Open(CalculatorContext* cc) override;
|
||||
absl::Status Process(CalculatorContext* cc) override;
|
||||
absl::Status Close(CalculatorContext* cc) override;
|
||||
|
||||
private:
|
||||
absl::Status RenderGpu(CalculatorContext* cc);
|
||||
absl::Status RenderCpu(CalculatorContext* cc);
|
||||
|
||||
absl::Status GlSetup(CalculatorContext* cc);
|
||||
void GlRender(CalculatorContext* cc);
|
||||
|
||||
float combine_with_previous_ratio_;
|
||||
|
||||
bool gpu_initialized_ = false;
|
||||
#if !MEDIAPIPE_DISABLE_GPU
|
||||
mediapipe::GlCalculatorHelper gpu_helper_;
|
||||
GLuint program_ = 0;
|
||||
#endif // !MEDIAPIPE_DISABLE_GPU
|
||||
};
|
||||
REGISTER_CALCULATOR(SegmentationSmoothingCalculator);
|
||||
|
||||
absl::Status SegmentationSmoothingCalculator::GetContract(
|
||||
CalculatorContract* cc) {
|
||||
CHECK_GE(cc->Inputs().NumEntries(), 1);
|
||||
|
||||
cc->Inputs().Tag(kCurrentMaskTag).Set<Image>();
|
||||
cc->Inputs().Tag(kPreviousMaskTag).Set<Image>();
|
||||
cc->Outputs().Tag(kOutputMaskTag).Set<Image>();
|
||||
|
||||
#if !MEDIAPIPE_DISABLE_GPU
|
||||
MP_RETURN_IF_ERROR(mediapipe::GlCalculatorHelper::UpdateContract(cc));
|
||||
#endif // !MEDIAPIPE_DISABLE_GPU
|
||||
|
||||
return absl::OkStatus();
|
||||
}
|
||||
|
||||
absl::Status SegmentationSmoothingCalculator::Open(CalculatorContext* cc) {
|
||||
cc->SetOffset(TimestampDiff(0));
|
||||
|
||||
auto options =
|
||||
cc->Options<mediapipe::SegmentationSmoothingCalculatorOptions>();
|
||||
combine_with_previous_ratio_ = options.combine_with_previous_ratio();
|
||||
|
||||
#if !MEDIAPIPE_DISABLE_GPU
|
||||
MP_RETURN_IF_ERROR(gpu_helper_.Open(cc));
|
||||
#endif // !MEDIAPIPE_DISABLE_GPU
|
||||
|
||||
return absl::OkStatus();
|
||||
}
|
||||
|
||||
absl::Status SegmentationSmoothingCalculator::Process(CalculatorContext* cc) {
|
||||
if (cc->Inputs().Tag(kCurrentMaskTag).IsEmpty()) {
|
||||
return absl::OkStatus();
|
||||
}
|
||||
if (cc->Inputs().Tag(kPreviousMaskTag).IsEmpty()) {
|
||||
// Pass through current image if previous is not available.
|
||||
cc->Outputs()
|
||||
.Tag(kOutputMaskTag)
|
||||
.AddPacket(cc->Inputs().Tag(kCurrentMaskTag).Value());
|
||||
return absl::OkStatus();
|
||||
}
|
||||
|
||||
// Run on GPU if incoming data is on GPU.
|
||||
const bool use_gpu = cc->Inputs().Tag(kCurrentMaskTag).Get<Image>().UsesGpu();
|
||||
|
||||
if (use_gpu) {
|
||||
#if !MEDIAPIPE_DISABLE_GPU
|
||||
MP_RETURN_IF_ERROR(gpu_helper_.RunInGlContext([this, cc]() -> absl::Status {
|
||||
if (!gpu_initialized_) {
|
||||
MP_RETURN_IF_ERROR(GlSetup(cc));
|
||||
gpu_initialized_ = true;
|
||||
}
|
||||
MP_RETURN_IF_ERROR(RenderGpu(cc));
|
||||
return absl::OkStatus();
|
||||
}));
|
||||
#else
|
||||
return absl::InternalError("GPU processing is disabled.");
|
||||
#endif // !MEDIAPIPE_DISABLE_GPU
|
||||
} else {
|
||||
MP_RETURN_IF_ERROR(RenderCpu(cc));
|
||||
}
|
||||
|
||||
return absl::OkStatus();
|
||||
}
|
||||
|
||||
absl::Status SegmentationSmoothingCalculator::Close(CalculatorContext* cc) {
|
||||
#if !MEDIAPIPE_DISABLE_GPU
|
||||
gpu_helper_.RunInGlContext([this] {
|
||||
if (program_) glDeleteProgram(program_);
|
||||
program_ = 0;
|
||||
});
|
||||
#endif // !MEDIAPIPE_DISABLE_GPU
|
||||
|
||||
return absl::OkStatus();
|
||||
}
|
||||
|
||||
absl::Status SegmentationSmoothingCalculator::RenderCpu(CalculatorContext* cc) {
|
||||
// Setup source images.
|
||||
const auto& current_frame = cc->Inputs().Tag(kCurrentMaskTag).Get<Image>();
|
||||
const cv::Mat current_mat = mediapipe::formats::MatView(¤t_frame);
|
||||
RET_CHECK_EQ(current_mat.type(), CV_32FC1)
|
||||
<< "Only 1-channel float input image is supported.";
|
||||
|
||||
const auto& previous_frame = cc->Inputs().Tag(kPreviousMaskTag).Get<Image>();
|
||||
const cv::Mat previous_mat = mediapipe::formats::MatView(&previous_frame);
|
||||
RET_CHECK_EQ(previous_mat.type(), current_mat.type())
|
||||
<< "Warning: mixing input format types: " << previous_mat.type()
|
||||
<< " != " << previous_mat.type();
|
||||
|
||||
RET_CHECK_EQ(current_mat.rows, previous_mat.rows);
|
||||
RET_CHECK_EQ(current_mat.cols, previous_mat.cols);
|
||||
|
||||
// Setup destination image.
|
||||
auto output_frame = std::make_shared<ImageFrame>(
|
||||
current_frame.image_format(), current_mat.cols, current_mat.rows);
|
||||
cv::Mat output_mat = mediapipe::formats::MatView(output_frame.get());
|
||||
output_mat.setTo(cv::Scalar(0));
|
||||
|
||||
// Blending function.
|
||||
const auto blending_fn = [&](const float prev_mask_value,
|
||||
const float new_mask_value) {
|
||||
/*
|
||||
* Assume p := new_mask_value
|
||||
* H(p) := 1 + (p * log(p) + (1-p) * log(1-p)) / log(2)
|
||||
* uncertainty alpha(p) =
|
||||
* Clamp(1 - (1 - H(p)) * (1 - H(p)), 0, 1) [squaring the uncertainty]
|
||||
*
|
||||
* The following polynomial approximates uncertainty alpha as a function
|
||||
* of (p + 0.5):
|
||||
*/
|
||||
const float c1 = 5.68842;
|
||||
const float c2 = -0.748699;
|
||||
const float c3 = -57.8051;
|
||||
const float c4 = 291.309;
|
||||
const float c5 = -624.717;
|
||||
const float t = new_mask_value - 0.5f;
|
||||
const float x = t * t;
|
||||
|
||||
const float uncertainty =
|
||||
1.0f -
|
||||
std::min(1.0f, x * (c1 + x * (c2 + x * (c3 + x * (c4 + x * c5)))));
|
||||
|
||||
return new_mask_value + (prev_mask_value - new_mask_value) *
|
||||
(uncertainty * combine_with_previous_ratio_);
|
||||
};
|
||||
|
||||
// Write directly to the first channel of output.
|
||||
for (int i = 0; i < output_mat.rows; ++i) {
|
||||
float* out_ptr = output_mat.ptr<float>(i);
|
||||
const float* curr_ptr = current_mat.ptr<float>(i);
|
||||
const float* prev_ptr = previous_mat.ptr<float>(i);
|
||||
for (int j = 0; j < output_mat.cols; ++j) {
|
||||
const float new_mask_value = curr_ptr[j];
|
||||
const float prev_mask_value = prev_ptr[j];
|
||||
out_ptr[j] = blending_fn(prev_mask_value, new_mask_value);
|
||||
}
|
||||
}
|
||||
|
||||
cc->Outputs()
|
||||
.Tag(kOutputMaskTag)
|
||||
.AddPacket(MakePacket<Image>(output_frame).At(cc->InputTimestamp()));
|
||||
|
||||
return absl::OkStatus();
|
||||
}
|
||||
|
||||
absl::Status SegmentationSmoothingCalculator::RenderGpu(CalculatorContext* cc) {
|
||||
#if !MEDIAPIPE_DISABLE_GPU
|
||||
// Setup source textures.
|
||||
const auto& current_frame = cc->Inputs().Tag(kCurrentMaskTag).Get<Image>();
|
||||
RET_CHECK(
|
||||
(current_frame.format() == mediapipe::GpuBufferFormat::kBGRA32 ||
|
||||
current_frame.format() == mediapipe::GpuBufferFormat::kGrayHalf16 ||
|
||||
current_frame.format() == mediapipe::GpuBufferFormat::kGrayFloat32 ||
|
||||
current_frame.format() == mediapipe::GpuBufferFormat::kRGB24))
|
||||
<< "Only RGBA, RGB, or 1-channel Float input image supported.";
|
||||
|
||||
auto current_texture = gpu_helper_.CreateSourceTexture(current_frame);
|
||||
|
||||
const auto& previous_frame = cc->Inputs().Tag(kPreviousMaskTag).Get<Image>();
|
||||
if (previous_frame.format() != current_frame.format()) {
|
||||
LOG(ERROR) << "Warning: mixing input format types. ";
|
||||
}
|
||||
auto previous_texture = gpu_helper_.CreateSourceTexture(previous_frame);
|
||||
|
||||
// Setup destination texture.
|
||||
const int width = current_frame.width(), height = current_frame.height();
|
||||
auto output_texture = gpu_helper_.CreateDestinationTexture(
|
||||
width, height, current_frame.format());
|
||||
|
||||
// Process shader.
|
||||
{
|
||||
gpu_helper_.BindFramebuffer(output_texture);
|
||||
glActiveTexture(GL_TEXTURE1);
|
||||
glBindTexture(GL_TEXTURE_2D, current_texture.name());
|
||||
glActiveTexture(GL_TEXTURE2);
|
||||
glBindTexture(GL_TEXTURE_2D, previous_texture.name());
|
||||
GlRender(cc);
|
||||
glActiveTexture(GL_TEXTURE2);
|
||||
glBindTexture(GL_TEXTURE_2D, 0);
|
||||
glActiveTexture(GL_TEXTURE1);
|
||||
glBindTexture(GL_TEXTURE_2D, 0);
|
||||
}
|
||||
glFlush();
|
||||
|
||||
// Send out image as GPU packet.
|
||||
auto output_frame = output_texture.GetFrame<Image>();
|
||||
cc->Outputs()
|
||||
.Tag(kOutputMaskTag)
|
||||
.Add(output_frame.release(), cc->InputTimestamp());
|
||||
#endif // !MEDIAPIPE_DISABLE_GPU
|
||||
|
||||
return absl::OkStatus();
|
||||
}
|
||||
|
||||
void SegmentationSmoothingCalculator::GlRender(CalculatorContext* cc) {
|
||||
#if !MEDIAPIPE_DISABLE_GPU
|
||||
static const GLfloat square_vertices[] = {
|
||||
-1.0f, -1.0f, // bottom left
|
||||
1.0f, -1.0f, // bottom right
|
||||
-1.0f, 1.0f, // top left
|
||||
1.0f, 1.0f, // top right
|
||||
};
|
||||
static const GLfloat texture_vertices[] = {
|
||||
0.0f, 0.0f, // bottom left
|
||||
1.0f, 0.0f, // bottom right
|
||||
0.0f, 1.0f, // top left
|
||||
1.0f, 1.0f, // top right
|
||||
};
|
||||
|
||||
// program
|
||||
glUseProgram(program_);
|
||||
|
||||
// vertex storage
|
||||
GLuint vbo[2];
|
||||
glGenBuffers(2, vbo);
|
||||
GLuint vao;
|
||||
glGenVertexArrays(1, &vao);
|
||||
glBindVertexArray(vao);
|
||||
|
||||
// vbo 0
|
||||
glBindBuffer(GL_ARRAY_BUFFER, vbo[0]);
|
||||
glBufferData(GL_ARRAY_BUFFER, 4 * 2 * sizeof(GLfloat), square_vertices,
|
||||
GL_STATIC_DRAW);
|
||||
glEnableVertexAttribArray(ATTRIB_VERTEX);
|
||||
glVertexAttribPointer(ATTRIB_VERTEX, 2, GL_FLOAT, 0, 0, nullptr);
|
||||
|
||||
// vbo 1
|
||||
glBindBuffer(GL_ARRAY_BUFFER, vbo[1]);
|
||||
glBufferData(GL_ARRAY_BUFFER, 4 * 2 * sizeof(GLfloat), texture_vertices,
|
||||
GL_STATIC_DRAW);
|
||||
glEnableVertexAttribArray(ATTRIB_TEXTURE_POSITION);
|
||||
glVertexAttribPointer(ATTRIB_TEXTURE_POSITION, 2, GL_FLOAT, 0, 0, nullptr);
|
||||
|
||||
// draw
|
||||
glDrawArrays(GL_TRIANGLE_STRIP, 0, 4);
|
||||
|
||||
// cleanup
|
||||
glDisableVertexAttribArray(ATTRIB_VERTEX);
|
||||
glDisableVertexAttribArray(ATTRIB_TEXTURE_POSITION);
|
||||
glBindBuffer(GL_ARRAY_BUFFER, 0);
|
||||
glBindVertexArray(0);
|
||||
glDeleteVertexArrays(1, &vao);
|
||||
glDeleteBuffers(2, vbo);
|
||||
|
||||
#endif // !MEDIAPIPE_DISABLE_GPU
|
||||
}
|
||||
|
||||
absl::Status SegmentationSmoothingCalculator::GlSetup(CalculatorContext* cc) {
|
||||
#if !MEDIAPIPE_DISABLE_GPU
|
||||
const GLint attr_location[NUM_ATTRIBUTES] = {
|
||||
ATTRIB_VERTEX,
|
||||
ATTRIB_TEXTURE_POSITION,
|
||||
};
|
||||
const GLchar* attr_name[NUM_ATTRIBUTES] = {
|
||||
"position",
|
||||
"texture_coordinate",
|
||||
};
|
||||
|
||||
// Shader to blend in previous mask based on computed uncertainty probability.
|
||||
const std::string frag_src =
|
||||
absl::StrCat(std::string(mediapipe::kMediaPipeFragmentShaderPreamble),
|
||||
R"(
|
||||
DEFAULT_PRECISION(mediump, float)
|
||||
|
||||
#ifdef GL_ES
|
||||
#define fragColor gl_FragColor
|
||||
#else
|
||||
out vec4 fragColor;
|
||||
#endif // defined(GL_ES);
|
||||
|
||||
in vec2 sample_coordinate;
|
||||
uniform sampler2D current_mask;
|
||||
uniform sampler2D previous_mask;
|
||||
uniform float combine_with_previous_ratio;
|
||||
|
||||
void main() {
|
||||
vec4 current_pix = texture2D(current_mask, sample_coordinate);
|
||||
vec4 previous_pix = texture2D(previous_mask, sample_coordinate);
|
||||
float new_mask_value = current_pix.r;
|
||||
float prev_mask_value = previous_pix.r;
|
||||
|
||||
// Assume p := new_mask_value
|
||||
// H(p) := 1 + (p * log(p) + (1-p) * log(1-p)) / log(2)
|
||||
// uncertainty alpha(p) =
|
||||
// Clamp(1 - (1 - H(p)) * (1 - H(p)), 0, 1) [squaring the uncertainty]
|
||||
//
|
||||
// The following polynomial approximates uncertainty alpha as a function
|
||||
// of (p + 0.5):
|
||||
const float c1 = 5.68842;
|
||||
const float c2 = -0.748699;
|
||||
const float c3 = -57.8051;
|
||||
const float c4 = 291.309;
|
||||
const float c5 = -624.717;
|
||||
float t = new_mask_value - 0.5;
|
||||
float x = t * t;
|
||||
|
||||
float uncertainty =
|
||||
1.0 - min(1.0, x * (c1 + x * (c2 + x * (c3 + x * (c4 + x * c5)))));
|
||||
|
||||
new_mask_value +=
|
||||
(prev_mask_value - new_mask_value) * (uncertainty * combine_with_previous_ratio);
|
||||
|
||||
fragColor = vec4(new_mask_value, 0.0, 0.0, new_mask_value);
|
||||
}
|
||||
)");
|
||||
|
||||
// Create shader program and set parameters.
|
||||
mediapipe::GlhCreateProgram(mediapipe::kBasicVertexShader, frag_src.c_str(),
|
||||
NUM_ATTRIBUTES, (const GLchar**)&attr_name[0],
|
||||
attr_location, &program_);
|
||||
RET_CHECK(program_) << "Problem initializing the program.";
|
||||
glUseProgram(program_);
|
||||
glUniform1i(glGetUniformLocation(program_, "current_mask"), 1);
|
||||
glUniform1i(glGetUniformLocation(program_, "previous_mask"), 2);
|
||||
glUniform1f(glGetUniformLocation(program_, "combine_with_previous_ratio"),
|
||||
combine_with_previous_ratio_);
|
||||
|
||||
#endif // !MEDIAPIPE_DISABLE_GPU
|
||||
|
||||
return absl::OkStatus();
|
||||
}
|
||||
|
||||
} // namespace mediapipe
|
||||
@@ -0,0 +1,35 @@
|
||||
// Copyright 2021 The MediaPipe Authors.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
syntax = "proto2";
|
||||
|
||||
package mediapipe;
|
||||
|
||||
import "mediapipe/framework/calculator.proto";
|
||||
|
||||
message SegmentationSmoothingCalculatorOptions {
|
||||
extend CalculatorOptions {
|
||||
optional SegmentationSmoothingCalculatorOptions ext = 377425128;
|
||||
}
|
||||
|
||||
// How much to blend in previous mask, based on a probability estimate.
|
||||
// Range: [0-1]
|
||||
// 0 = Use only current frame (no blending).
|
||||
// 1 = Blend in the previous mask based on uncertainty estimate.
|
||||
// With ratio at 1, the uncertainty estimate is trusted completely.
|
||||
// When uncertainty is high, the previous mask is given higher weight.
|
||||
// Therefore, if both ratio and uncertainty are 1, only old mask is used.
|
||||
// A pixel is 'uncertain' if its value is close to the middle (0.5 or 127).
|
||||
optional float combine_with_previous_ratio = 1 [default = 0.0];
|
||||
}
|
||||
@@ -0,0 +1,206 @@
|
||||
// Copyright 2018 The MediaPipe Authors.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
#include <memory>
|
||||
|
||||
#include "mediapipe/calculators/image/segmentation_smoothing_calculator.pb.h"
|
||||
#include "mediapipe/framework/calculator_framework.h"
|
||||
#include "mediapipe/framework/calculator_runner.h"
|
||||
#include "mediapipe/framework/deps/file_path.h"
|
||||
#include "mediapipe/framework/formats/image_frame.h"
|
||||
#include "mediapipe/framework/formats/image_opencv.h"
|
||||
#include "mediapipe/framework/port/gmock.h"
|
||||
#include "mediapipe/framework/port/gtest.h"
|
||||
#include "mediapipe/framework/port/opencv_imgcodecs_inc.h"
|
||||
#include "mediapipe/framework/port/opencv_imgproc_inc.h"
|
||||
#include "mediapipe/framework/port/parse_text_proto.h"
|
||||
#include "mediapipe/framework/port/status_matchers.h"
|
||||
|
||||
namespace mediapipe {
|
||||
|
||||
namespace {
|
||||
|
||||
// 4x4 VEC32F1, center 2x2 block set at ~250
|
||||
const float mask_data[] = {
|
||||
0.00, 0.00, 0.00, 0.00, //
|
||||
0.00, 0.98, 0.98, 0.00, //
|
||||
0.00, 0.98, 0.98, 0.00, //
|
||||
0.00, 0.00, 0.00, 0.00, //
|
||||
};
|
||||
|
||||
void RunGraph(Packet curr_packet, Packet prev_packet, bool use_gpu, float ratio,
|
||||
cv::Mat* result) {
|
||||
CalculatorGraphConfig graph_config;
|
||||
if (use_gpu) {
|
||||
graph_config = ParseTextProtoOrDie<CalculatorGraphConfig>(absl::Substitute(
|
||||
R"pb(
|
||||
input_stream: "curr_mask"
|
||||
input_stream: "prev_mask"
|
||||
output_stream: "new_mask"
|
||||
node {
|
||||
calculator: "ImageCloneCalculator"
|
||||
input_stream: "curr_mask"
|
||||
output_stream: "curr_mask_gpu"
|
||||
options: {
|
||||
[mediapipe.ImageCloneCalculatorOptions.ext] {
|
||||
output_on_gpu: true
|
||||
}
|
||||
}
|
||||
}
|
||||
node {
|
||||
calculator: "ImageCloneCalculator"
|
||||
input_stream: "prev_mask"
|
||||
output_stream: "prev_mask_gpu"
|
||||
options: {
|
||||
[mediapipe.ImageCloneCalculatorOptions.ext] {
|
||||
output_on_gpu: true
|
||||
}
|
||||
}
|
||||
}
|
||||
node {
|
||||
calculator: "SegmentationSmoothingCalculator"
|
||||
input_stream: "MASK:curr_mask_gpu"
|
||||
input_stream: "MASK_PREVIOUS:prev_mask_gpu"
|
||||
output_stream: "MASK_SMOOTHED:new_mask"
|
||||
node_options {
|
||||
[type.googleapis.com/
|
||||
mediapipe.SegmentationSmoothingCalculatorOptions]: {
|
||||
combine_with_previous_ratio: $0
|
||||
}
|
||||
}
|
||||
}
|
||||
)pb",
|
||||
ratio));
|
||||
} else {
|
||||
graph_config = ParseTextProtoOrDie<CalculatorGraphConfig>(absl::Substitute(
|
||||
R"pb(
|
||||
input_stream: "curr_mask"
|
||||
input_stream: "prev_mask"
|
||||
output_stream: "new_mask"
|
||||
node {
|
||||
calculator: "SegmentationSmoothingCalculator"
|
||||
input_stream: "MASK:curr_mask"
|
||||
input_stream: "MASK_PREVIOUS:prev_mask"
|
||||
output_stream: "MASK_SMOOTHED:new_mask"
|
||||
node_options {
|
||||
[type.googleapis.com/
|
||||
mediapipe.SegmentationSmoothingCalculatorOptions]: {
|
||||
combine_with_previous_ratio: $0
|
||||
}
|
||||
}
|
||||
}
|
||||
)pb",
|
||||
ratio));
|
||||
}
|
||||
std::vector<Packet> output_packets;
|
||||
tool::AddVectorSink("new_mask", &graph_config, &output_packets);
|
||||
CalculatorGraph graph(graph_config);
|
||||
MP_ASSERT_OK(graph.StartRun({}));
|
||||
|
||||
MP_ASSERT_OK(
|
||||
graph.AddPacketToInputStream("curr_mask", curr_packet.At(Timestamp(0))));
|
||||
MP_ASSERT_OK(
|
||||
graph.AddPacketToInputStream("prev_mask", prev_packet.At(Timestamp(0))));
|
||||
MP_ASSERT_OK(graph.WaitUntilIdle());
|
||||
ASSERT_EQ(1, output_packets.size());
|
||||
|
||||
Image result_image = output_packets[0].Get<Image>();
|
||||
cv::Mat result_mat = formats::MatView(&result_image);
|
||||
result_mat.copyTo(*result);
|
||||
|
||||
// Fully close graph at end, otherwise calculator+Images are destroyed
|
||||
// after calling WaitUntilDone().
|
||||
MP_ASSERT_OK(graph.CloseInputStream("curr_mask"));
|
||||
MP_ASSERT_OK(graph.CloseInputStream("prev_mask"));
|
||||
MP_ASSERT_OK(graph.WaitUntilDone());
|
||||
}
|
||||
|
||||
void RunTest(bool use_gpu, float mix_ratio, cv::Mat& test_result) {
|
||||
cv::Mat mask_mat(cv::Size(4, 4), CV_32FC1, const_cast<float*>(mask_data));
|
||||
cv::Mat curr_mat = mask_mat;
|
||||
// 3x3 blur of 250 block produces all pixels '111'.
|
||||
cv::Mat prev_mat;
|
||||
cv::blur(mask_mat, prev_mat, cv::Size(3, 3));
|
||||
|
||||
Packet curr_packet = MakePacket<Image>(std::make_unique<ImageFrame>(
|
||||
ImageFormat::VEC32F1, curr_mat.size().width, curr_mat.size().height));
|
||||
curr_mat.copyTo(formats::MatView(&(curr_packet.Get<Image>())));
|
||||
Packet prev_packet = MakePacket<Image>(std::make_unique<ImageFrame>(
|
||||
ImageFormat::VEC32F1, prev_mat.size().width, prev_mat.size().height));
|
||||
prev_mat.copyTo(formats::MatView(&(prev_packet.Get<Image>())));
|
||||
|
||||
cv::Mat result;
|
||||
RunGraph(curr_packet, prev_packet, use_gpu, mix_ratio, &result);
|
||||
|
||||
ASSERT_EQ(curr_mat.rows, result.rows);
|
||||
ASSERT_EQ(curr_mat.cols, result.cols);
|
||||
ASSERT_EQ(curr_mat.type(), result.type());
|
||||
result.copyTo(test_result);
|
||||
|
||||
if (mix_ratio == 1.0) {
|
||||
for (int i = 0; i < 4; ++i) {
|
||||
for (int j = 0; j < 4; ++j) {
|
||||
float in = curr_mat.at<float>(i, j);
|
||||
float out = result.at<float>(i, j);
|
||||
// Since the input has high value (250), it has low uncertainty.
|
||||
// So the output should have changed lower (towards prev),
|
||||
// but not too much.
|
||||
if (in > 0) EXPECT_NE(in, out);
|
||||
EXPECT_NEAR(in, out, 3.0 / 255.0);
|
||||
}
|
||||
}
|
||||
} else if (mix_ratio == 0.0) {
|
||||
for (int i = 0; i < 4; ++i) {
|
||||
for (int j = 0; j < 4; ++j) {
|
||||
float in = curr_mat.at<float>(i, j);
|
||||
float out = result.at<float>(i, j);
|
||||
EXPECT_EQ(in, out); // Output should match current.
|
||||
}
|
||||
}
|
||||
} else {
|
||||
LOG(ERROR) << "invalid ratio";
|
||||
}
|
||||
}
|
||||
|
||||
TEST(SegmentationSmoothingCalculatorTest, TestSmoothing) {
|
||||
bool use_gpu;
|
||||
float mix_ratio;
|
||||
|
||||
use_gpu = false;
|
||||
mix_ratio = 0.0;
|
||||
cv::Mat cpu_0;
|
||||
RunTest(use_gpu, mix_ratio, cpu_0);
|
||||
|
||||
use_gpu = false;
|
||||
mix_ratio = 1.0;
|
||||
cv::Mat cpu_1;
|
||||
RunTest(use_gpu, mix_ratio, cpu_1);
|
||||
|
||||
use_gpu = true;
|
||||
mix_ratio = 1.0;
|
||||
cv::Mat gpu_1;
|
||||
RunTest(use_gpu, mix_ratio, gpu_1);
|
||||
|
||||
// CPU & GPU should match.
|
||||
for (int i = 0; i < 4; ++i) {
|
||||
for (int j = 0; j < 4; ++j) {
|
||||
float gpu = gpu_1.at<float>(i, j);
|
||||
float cpu = cpu_1.at<float>(i, j);
|
||||
EXPECT_EQ(cpu, gpu);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
} // namespace
|
||||
} // namespace mediapipe
|
||||
@@ -109,6 +109,8 @@ cc_library(
|
||||
"//mediapipe/gpu:MPPMetalUtil",
|
||||
"//mediapipe/gpu:gpu_buffer",
|
||||
"//mediapipe/objc:mediapipe_framework_ios",
|
||||
"//mediapipe/util/tflite:config",
|
||||
"@com_google_absl//absl/memory",
|
||||
"@org_tensorflow//tensorflow/lite/delegates/gpu:metal_delegate",
|
||||
"@org_tensorflow//tensorflow/lite/delegates/gpu:metal_delegate_internal",
|
||||
"@org_tensorflow//tensorflow/lite/delegates/gpu/common:shape",
|
||||
@@ -478,7 +480,6 @@ cc_library(
|
||||
deps = [
|
||||
":image_to_tensor_calculator_cc_proto",
|
||||
":image_to_tensor_converter",
|
||||
":image_to_tensor_converter_opencv",
|
||||
":image_to_tensor_utils",
|
||||
"//mediapipe/framework/api2:node",
|
||||
"//mediapipe/framework/formats:image",
|
||||
@@ -494,6 +495,9 @@ cc_library(
|
||||
] + select({
|
||||
"//mediapipe/gpu:disable_gpu": [],
|
||||
"//conditions:default": [":image_to_tensor_calculator_gpu_deps"],
|
||||
}) + select({
|
||||
"//mediapipe/framework/port:disable_opencv": [],
|
||||
"//conditions:default": [":image_to_tensor_converter_opencv"],
|
||||
}),
|
||||
alwayslink = 1,
|
||||
)
|
||||
|
||||
@@ -18,7 +18,6 @@
|
||||
|
||||
#include "mediapipe/calculators/tensor/image_to_tensor_calculator.pb.h"
|
||||
#include "mediapipe/calculators/tensor/image_to_tensor_converter.h"
|
||||
#include "mediapipe/calculators/tensor/image_to_tensor_converter_opencv.h"
|
||||
#include "mediapipe/calculators/tensor/image_to_tensor_utils.h"
|
||||
#include "mediapipe/framework/api2/node.h"
|
||||
#include "mediapipe/framework/calculator_framework.h"
|
||||
@@ -33,6 +32,10 @@
|
||||
#include "mediapipe/framework/port/statusor.h"
|
||||
#include "mediapipe/gpu/gpu_origin.pb.h"
|
||||
|
||||
#if !MEDIAPIPE_DISABLE_OPENCV
|
||||
#include "mediapipe/calculators/tensor/image_to_tensor_converter_opencv.h"
|
||||
#endif
|
||||
|
||||
#if !MEDIAPIPE_DISABLE_GPU
|
||||
#include "mediapipe/gpu/gpu_buffer.h"
|
||||
|
||||
@@ -301,8 +304,13 @@ class ImageToTensorCalculator : public Node {
|
||||
}
|
||||
} else {
|
||||
if (!cpu_converter_) {
|
||||
#if !MEDIAPIPE_DISABLE_OPENCV
|
||||
ASSIGN_OR_RETURN(cpu_converter_,
|
||||
CreateOpenCvConverter(cc, GetBorderMode()));
|
||||
#else
|
||||
LOG(FATAL) << "Cannot create image to tensor opencv converter since "
|
||||
"MEDIAPIPE_DISABLE_OPENCV is defined.";
|
||||
#endif // !MEDIAPIPE_DISABLE_OPENCV
|
||||
}
|
||||
}
|
||||
return absl::OkStatus();
|
||||
|
||||
@@ -312,7 +312,7 @@ class GlProcessor : public ImageToTensorConverter {
|
||||
return absl::OkStatus();
|
||||
}));
|
||||
|
||||
return std::move(tensor);
|
||||
return tensor;
|
||||
}
|
||||
|
||||
~GlProcessor() override {
|
||||
@@ -338,8 +338,7 @@ CreateImageToGlBufferTensorConverter(CalculatorContext* cc,
|
||||
auto result = absl::make_unique<GlProcessor>();
|
||||
MP_RETURN_IF_ERROR(result->Init(cc, input_starts_at_bottom, border_mode));
|
||||
|
||||
// Simply "return std::move(result)" failed to build on macOS with bazel.
|
||||
return std::unique_ptr<ImageToTensorConverter>(std::move(result));
|
||||
return result;
|
||||
}
|
||||
|
||||
} // namespace mediapipe
|
||||
|
||||
@@ -334,9 +334,7 @@ CreateImageToGlTextureTensorConverter(CalculatorContext* cc,
|
||||
BorderMode border_mode) {
|
||||
auto result = absl::make_unique<GlProcessor>();
|
||||
MP_RETURN_IF_ERROR(result->Init(cc, input_starts_at_bottom, border_mode));
|
||||
|
||||
// Simply "return std::move(result)" failed to build on macOS with bazel.
|
||||
return std::unique_ptr<ImageToTensorConverter>(std::move(result));
|
||||
return result;
|
||||
}
|
||||
|
||||
} // namespace mediapipe
|
||||
|
||||
@@ -383,7 +383,7 @@ class MetalProcessor : public ImageToTensorConverter {
|
||||
tflite::gpu::HW(output_dims.height, output_dims.width),
|
||||
command_buffer, buffer_view.buffer()));
|
||||
[command_buffer commit];
|
||||
return std::move(tensor);
|
||||
return tensor;
|
||||
}
|
||||
}
|
||||
|
||||
@@ -399,8 +399,7 @@ absl::StatusOr<std::unique_ptr<ImageToTensorConverter>> CreateMetalConverter(
|
||||
auto result = absl::make_unique<MetalProcessor>();
|
||||
MP_RETURN_IF_ERROR(result->Init(cc, border_mode));
|
||||
|
||||
// Simply "return std::move(result)" failed to build on macOS with bazel.
|
||||
return std::unique_ptr<ImageToTensorConverter>(std::move(result));
|
||||
return result;
|
||||
}
|
||||
|
||||
} // namespace mediapipe
|
||||
|
||||
@@ -103,7 +103,7 @@ class OpenCvProcessor : public ImageToTensorConverter {
|
||||
GetValueRangeTransformation(kInputImageRangeMin, kInputImageRangeMax,
|
||||
range_min, range_max));
|
||||
transformed.convertTo(dst, CV_32FC3, transform.scale, transform.offset);
|
||||
return std::move(tensor);
|
||||
return tensor;
|
||||
}
|
||||
|
||||
private:
|
||||
@@ -114,10 +114,7 @@ class OpenCvProcessor : public ImageToTensorConverter {
|
||||
|
||||
absl::StatusOr<std::unique_ptr<ImageToTensorConverter>> CreateOpenCvConverter(
|
||||
CalculatorContext* cc, BorderMode border_mode) {
|
||||
// Simply "return absl::make_unique<OpenCvProcessor>()" failed to build on
|
||||
// macOS with bazel.
|
||||
return std::unique_ptr<ImageToTensorConverter>(
|
||||
absl::make_unique<OpenCvProcessor>(border_mode));
|
||||
return absl::make_unique<OpenCvProcessor>(border_mode);
|
||||
}
|
||||
|
||||
} // namespace mediapipe
|
||||
|
||||
@@ -35,20 +35,28 @@ namespace api2 {
|
||||
|
||||
namespace {
|
||||
|
||||
int GetXnnpackDefaultNumThreads() {
|
||||
#if defined(MEDIAPIPE_ANDROID) || defined(MEDIAPIPE_IOS) || \
|
||||
defined(__EMSCRIPTEN_PTHREADS__)
|
||||
constexpr int kMinNumThreadsByDefault = 1;
|
||||
constexpr int kMaxNumThreadsByDefault = 4;
|
||||
return std::clamp(NumCPUCores() / 2, kMinNumThreadsByDefault,
|
||||
kMaxNumThreadsByDefault);
|
||||
#else
|
||||
return 1;
|
||||
#endif // MEDIAPIPE_ANDROID || MEDIAPIPE_IOS || __EMSCRIPTEN_PTHREADS__
|
||||
}
|
||||
|
||||
// Returns number of threads to configure XNNPACK delegate with.
|
||||
// (Equal to user provided value if specified. Otherwise, it returns number of
|
||||
// high cores (hard-coded to 1 for Emscripten without Threads extension))
|
||||
// Returns user provided value if specified. Otherwise, tries to choose optimal
|
||||
// number of threads depending on the device.
|
||||
int GetXnnpackNumThreads(const mediapipe::InferenceCalculatorOptions& opts) {
|
||||
static constexpr int kDefaultNumThreads = -1;
|
||||
if (opts.has_delegate() && opts.delegate().has_xnnpack() &&
|
||||
opts.delegate().xnnpack().num_threads() != kDefaultNumThreads) {
|
||||
return opts.delegate().xnnpack().num_threads();
|
||||
}
|
||||
#if !defined(__EMSCRIPTEN__) || defined(__EMSCRIPTEN_PTHREADS__)
|
||||
return InferHigherCoreIds().size();
|
||||
#else
|
||||
return 1;
|
||||
#endif // !__EMSCRIPTEN__ || __EMSCRIPTEN_PTHREADS__
|
||||
return GetXnnpackDefaultNumThreads();
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
@@ -269,8 +269,8 @@ absl::Status InferenceCalculatorGlImpl::InitTFLiteGPURunner(
|
||||
break;
|
||||
}
|
||||
}
|
||||
MP_RETURN_IF_ERROR(
|
||||
tflite_gpu_runner_->InitializeWithModel(model, op_resolver));
|
||||
MP_RETURN_IF_ERROR(tflite_gpu_runner_->InitializeWithModel(
|
||||
model, op_resolver, /*allow_quant_ops=*/true));
|
||||
|
||||
// Create and bind OpenGL buffers for outputs.
|
||||
// The buffers are created once and their ids are passed to calculator outputs
|
||||
|
||||
@@ -226,6 +226,10 @@ absl::Status InferenceCalculatorMetalImpl::LoadDelegate(CalculatorContext* cc) {
|
||||
|
||||
// Configure and create the delegate.
|
||||
TFLGpuDelegateOptions options;
|
||||
// `enable_quantization` enables the run of sparse models i.e. the models with
|
||||
// DENSIFY op preceding DEQUINTIZE op. Both ops get removed from the execution
|
||||
// graph after the tensor of the weights is read.
|
||||
options.enable_quantization = true;
|
||||
options.allow_precision_loss = allow_precision_loss_;
|
||||
options.wait_type = TFLGpuDelegateWaitType::TFLGpuDelegateWaitTypeDoNotWait;
|
||||
delegate_ =
|
||||
|
||||
@@ -763,9 +763,13 @@ out vec4 fragColor;
|
||||
#endif // defined(GL_ES);
|
||||
|
||||
void main() {
|
||||
|
||||
vec4 input_value = texture2D(input_texture, sample_coordinate);
|
||||
vec2 gid = sample_coordinate;
|
||||
#ifdef FLIP_Y_COORD
|
||||
float y_coord = 1.0 - sample_coordinate.y;
|
||||
#else
|
||||
float y_coord = sample_coordinate.y;
|
||||
#endif // defined(FLIP_Y_COORD)
|
||||
vec2 adjusted_coordinate = vec2(sample_coordinate.x, y_coord);
|
||||
vec4 input_value = texture2D(input_texture, adjusted_coordinate);
|
||||
|
||||
// Run activation function.
|
||||
// One and only one of FN_SOFTMAX,FN_SIGMOID,FN_NONE will be defined.
|
||||
@@ -787,13 +791,6 @@ void main() {
|
||||
float new_mask_value = input_value.r;
|
||||
#endif // FN_NONE
|
||||
|
||||
#ifdef FLIP_Y_COORD
|
||||
float y_coord = 1.0 - gid.y;
|
||||
#else
|
||||
float y_coord = gid.y;
|
||||
#endif // defined(FLIP_Y_COORD)
|
||||
vec2 output_coordinate = vec2(gid.x, y_coord);
|
||||
|
||||
vec4 out_value = vec4(new_mask_value, 0.0, 0.0, new_mask_value);
|
||||
fragColor = out_value;
|
||||
})";
|
||||
|
||||
@@ -4,7 +4,7 @@ output_stream: "detections"
|
||||
|
||||
# Subgraph that detects faces.
|
||||
node {
|
||||
calculator: "FaceDetectionFrontCpu"
|
||||
calculator: "FaceDetectionShortRangeCpu"
|
||||
input_stream: "IMAGE:image"
|
||||
output_stream: "DETECTIONS:detections"
|
||||
}
|
||||
|
||||
@@ -490,7 +490,7 @@ class TensorFlowInferenceCalculator : public CalculatorBase {
|
||||
<< keyed_tensors.first;
|
||||
}
|
||||
} else {
|
||||
// Pad by replicating the first tens or, then ignore the values.
|
||||
// Pad by replicating the first tensor, then ignore the values.
|
||||
keyed_tensors.second.resize(options_.batch_size());
|
||||
std::fill(keyed_tensors.second.begin() +
|
||||
inference_state->batch_timestamps_.size(),
|
||||
|
||||
@@ -128,9 +128,23 @@ struct GPUData {
|
||||
} // namespace
|
||||
#endif // MEDIAPIPE_TFLITE_GPU_SUPPORTED
|
||||
|
||||
namespace {
|
||||
|
||||
int GetXnnpackDefaultNumThreads() {
|
||||
#if defined(MEDIAPIPE_ANDROID) || defined(MEDIAPIPE_IOS) || \
|
||||
defined(__EMSCRIPTEN_PTHREADS__)
|
||||
constexpr int kMinNumThreadsByDefault = 1;
|
||||
constexpr int kMaxNumThreadsByDefault = 4;
|
||||
return std::clamp(NumCPUCores() / 2, kMinNumThreadsByDefault,
|
||||
kMaxNumThreadsByDefault);
|
||||
#else
|
||||
return 1;
|
||||
#endif // MEDIAPIPE_ANDROID || MEDIAPIPE_IOS || __EMSCRIPTEN_PTHREADS__
|
||||
}
|
||||
|
||||
// Returns number of threads to configure XNNPACK delegate with.
|
||||
// (Equal to user provided value if specified. Otherwise, it returns number of
|
||||
// high cores (hard-coded to 1 for Emscripten without Threads extension))
|
||||
// Returns user provided value if specified. Otherwise, tries to choose optimal
|
||||
// number of threads depending on the device.
|
||||
int GetXnnpackNumThreads(
|
||||
const mediapipe::TfLiteInferenceCalculatorOptions& opts) {
|
||||
static constexpr int kDefaultNumThreads = -1;
|
||||
@@ -138,13 +152,11 @@ int GetXnnpackNumThreads(
|
||||
opts.delegate().xnnpack().num_threads() != kDefaultNumThreads) {
|
||||
return opts.delegate().xnnpack().num_threads();
|
||||
}
|
||||
#if !defined(__EMSCRIPTEN__) || defined(__EMSCRIPTEN_PTHREADS__)
|
||||
return InferHigherCoreIds().size();
|
||||
#else
|
||||
return 1;
|
||||
#endif // !__EMSCRIPTEN__ || __EMSCRIPTEN_PTHREADS__
|
||||
return GetXnnpackDefaultNumThreads();
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
// Calculator Header Section
|
||||
|
||||
// Runs inference on the provided input TFLite tensors and TFLite model.
|
||||
@@ -737,8 +749,8 @@ absl::Status TfLiteInferenceCalculator::InitTFLiteGPURunner(
|
||||
break;
|
||||
}
|
||||
}
|
||||
MP_RETURN_IF_ERROR(
|
||||
tflite_gpu_runner_->InitializeWithModel(model, *op_resolver_ptr));
|
||||
MP_RETURN_IF_ERROR(tflite_gpu_runner_->InitializeWithModel(
|
||||
model, *op_resolver_ptr, /*allow_quant_ops=*/true));
|
||||
|
||||
// Allocate interpreter memory for cpu output.
|
||||
if (!gpu_output_) {
|
||||
@@ -969,6 +981,10 @@ absl::Status TfLiteInferenceCalculator::LoadDelegate(CalculatorContext* cc) {
|
||||
const int kHalfSize = 2; // sizeof(half)
|
||||
// Configure and create the delegate.
|
||||
TFLGpuDelegateOptions options;
|
||||
// `enable_quantization` enables the run of sparse models i.e. the models with
|
||||
// DENSIFY op preceding DEQUINTIZE op. Both ops get removed from the execution
|
||||
// graph after the tensor of the weights is read.
|
||||
options.enable_quantization = true;
|
||||
options.allow_precision_loss = allow_precision_loss_;
|
||||
options.wait_type = TFLGpuDelegateWaitType::TFLGpuDelegateWaitTypeActive;
|
||||
if (!delegate_)
|
||||
|
||||
@@ -840,6 +840,20 @@ cc_test(
|
||||
],
|
||||
)
|
||||
|
||||
cc_library(
|
||||
name = "world_landmark_projection_calculator",
|
||||
srcs = ["world_landmark_projection_calculator.cc"],
|
||||
visibility = ["//visibility:public"],
|
||||
deps = [
|
||||
"//mediapipe/framework:calculator_framework",
|
||||
"//mediapipe/framework/formats:landmark_cc_proto",
|
||||
"//mediapipe/framework/formats:rect_cc_proto",
|
||||
"//mediapipe/framework/port:ret_check",
|
||||
"//mediapipe/framework/port:status",
|
||||
],
|
||||
alwayslink = 1,
|
||||
)
|
||||
|
||||
mediapipe_proto_library(
|
||||
name = "landmarks_smoothing_calculator_proto",
|
||||
srcs = ["landmarks_smoothing_calculator.proto"],
|
||||
@@ -894,6 +908,31 @@ cc_library(
|
||||
alwayslink = 1,
|
||||
)
|
||||
|
||||
mediapipe_proto_library(
|
||||
name = "visibility_copy_calculator_proto",
|
||||
srcs = ["visibility_copy_calculator.proto"],
|
||||
visibility = ["//visibility:public"],
|
||||
deps = [
|
||||
"//mediapipe/framework:calculator_options_proto",
|
||||
"//mediapipe/framework:calculator_proto",
|
||||
],
|
||||
)
|
||||
|
||||
cc_library(
|
||||
name = "visibility_copy_calculator",
|
||||
srcs = ["visibility_copy_calculator.cc"],
|
||||
visibility = ["//visibility:public"],
|
||||
deps = [
|
||||
":visibility_copy_calculator_cc_proto",
|
||||
"//mediapipe/framework:calculator_framework",
|
||||
"//mediapipe/framework:timestamp",
|
||||
"//mediapipe/framework/formats:landmark_cc_proto",
|
||||
"//mediapipe/framework/port:ret_check",
|
||||
"@com_google_absl//absl/algorithm:container",
|
||||
],
|
||||
alwayslink = 1,
|
||||
)
|
||||
|
||||
cc_library(
|
||||
name = "landmarks_to_floats_calculator",
|
||||
srcs = ["landmarks_to_floats_calculator.cc"],
|
||||
|
||||
@@ -272,6 +272,15 @@ absl::Status AnnotationOverlayCalculator::Open(CalculatorContext* cc) {
|
||||
}
|
||||
|
||||
absl::Status AnnotationOverlayCalculator::Process(CalculatorContext* cc) {
|
||||
if (cc->Inputs().HasTag(kGpuBufferTag) &&
|
||||
cc->Inputs().Tag(kGpuBufferTag).IsEmpty()) {
|
||||
return absl::OkStatus();
|
||||
}
|
||||
if (cc->Inputs().HasTag(kImageFrameTag) &&
|
||||
cc->Inputs().Tag(kImageFrameTag).IsEmpty()) {
|
||||
return absl::OkStatus();
|
||||
}
|
||||
|
||||
// Initialize render target, drawn with OpenCV.
|
||||
std::unique_ptr<cv::Mat> image_mat;
|
||||
ImageFormat::Format target_format;
|
||||
|
||||
@@ -203,6 +203,9 @@ absl::Status DetectionsToRectsCalculator::Process(CalculatorContext* cc) {
|
||||
cc->Inputs().Tag(kDetectionsTag).IsEmpty()) {
|
||||
return absl::OkStatus();
|
||||
}
|
||||
if (rotate_ && !HasTagValue(cc, kImageSizeTag)) {
|
||||
return absl::OkStatus();
|
||||
}
|
||||
|
||||
std::vector<Detection> detections;
|
||||
if (cc->Inputs().HasTag(kDetectionTag)) {
|
||||
|
||||
@@ -32,11 +32,15 @@ typedef FilterCollectionCalculator<std::vector<::mediapipe::NormalizedRect>>
|
||||
FilterNormalizedRectCollectionCalculator;
|
||||
REGISTER_CALCULATOR(FilterNormalizedRectCollectionCalculator);
|
||||
|
||||
typedef FilterCollectionCalculator<
|
||||
std::vector<::mediapipe::NormalizedLandmarkList>>
|
||||
typedef FilterCollectionCalculator<std::vector<::mediapipe::LandmarkList>>
|
||||
FilterLandmarkListCollectionCalculator;
|
||||
REGISTER_CALCULATOR(FilterLandmarkListCollectionCalculator);
|
||||
|
||||
typedef FilterCollectionCalculator<
|
||||
std::vector<::mediapipe::NormalizedLandmarkList>>
|
||||
FilterNormalizedLandmarkListCollectionCalculator;
|
||||
REGISTER_CALCULATOR(FilterNormalizedLandmarkListCollectionCalculator);
|
||||
|
||||
typedef FilterCollectionCalculator<std::vector<::mediapipe::ClassificationList>>
|
||||
FilterClassificationListCollectionCalculator;
|
||||
REGISTER_CALCULATOR(FilterClassificationListCollectionCalculator);
|
||||
|
||||
@@ -130,8 +130,8 @@ absl::Status RectTransformationCalculator::Process(CalculatorContext* cc) {
|
||||
}
|
||||
cc->Outputs().Index(0).Add(output_rects.release(), cc->InputTimestamp());
|
||||
}
|
||||
if (cc->Inputs().HasTag(kNormRectTag) &&
|
||||
!cc->Inputs().Tag(kNormRectTag).IsEmpty()) {
|
||||
if (HasTagValue(cc->Inputs(), kNormRectTag) &&
|
||||
HasTagValue(cc->Inputs(), kImageSizeTag)) {
|
||||
auto rect = cc->Inputs().Tag(kNormRectTag).Get<NormalizedRect>();
|
||||
const auto& image_size =
|
||||
cc->Inputs().Tag(kImageSizeTag).Get<std::pair<int, int>>();
|
||||
@@ -139,8 +139,8 @@ absl::Status RectTransformationCalculator::Process(CalculatorContext* cc) {
|
||||
cc->Outputs().Index(0).AddPacket(
|
||||
MakePacket<NormalizedRect>(rect).At(cc->InputTimestamp()));
|
||||
}
|
||||
if (cc->Inputs().HasTag(kNormRectsTag) &&
|
||||
!cc->Inputs().Tag(kNormRectsTag).IsEmpty()) {
|
||||
if (HasTagValue(cc->Inputs(), kNormRectsTag) &&
|
||||
HasTagValue(cc->Inputs(), kImageSizeTag)) {
|
||||
auto rects =
|
||||
cc->Inputs().Tag(kNormRectsTag).Get<std::vector<NormalizedRect>>();
|
||||
const auto& image_size =
|
||||
|
||||
@@ -549,7 +549,7 @@ absl::Status MotionAnalysisCalculator::Process(CalculatorContext* cc) {
|
||||
timestamp_buffer_.push_back(timestamp);
|
||||
++frame_idx_;
|
||||
|
||||
VLOG_EVERY_N(0, 100) << "Analyzed frame " << frame_idx_;
|
||||
VLOG_EVERY_N(1, 100) << "Analyzed frame " << frame_idx_;
|
||||
|
||||
// Buffer input frames only if visualization is requested.
|
||||
if (visualize_output_ || video_output_) {
|
||||
|
||||
+1
-1
@@ -37,7 +37,7 @@ android_binary(
|
||||
srcs = glob(["*.java"]),
|
||||
assets = [
|
||||
"//mediapipe/graphs/face_detection:face_detection_mobile_cpu.binarypb",
|
||||
"//mediapipe/modules/face_detection:face_detection_front.tflite",
|
||||
"//mediapipe/modules/face_detection:face_detection_short_range.tflite",
|
||||
],
|
||||
assets_dir = "",
|
||||
manifest = "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:AndroidManifest.xml",
|
||||
|
||||
+60
@@ -0,0 +1,60 @@
|
||||
# Copyright 2019 The MediaPipe Authors.
|
||||
#
|
||||
# Licensed under the Apache License, Version 2.0 (the "License");
|
||||
# you may not use this file except in compliance with the License.
|
||||
# You may obtain a copy of the License at
|
||||
#
|
||||
# http://www.apache.org/licenses/LICENSE-2.0
|
||||
#
|
||||
# Unless required by applicable law or agreed to in writing, software
|
||||
# distributed under the License is distributed on an "AS IS" BASIS,
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
|
||||
licenses(["notice"])
|
||||
|
||||
package(default_visibility = ["//visibility:private"])
|
||||
|
||||
cc_binary(
|
||||
name = "libmediapipe_jni.so",
|
||||
linkshared = 1,
|
||||
linkstatic = 1,
|
||||
deps = [
|
||||
"//mediapipe/graphs/face_detection:face_detection_full_range_mobile_gpu_deps",
|
||||
"//mediapipe/java/com/google/mediapipe/framework/jni:mediapipe_framework_jni",
|
||||
],
|
||||
)
|
||||
|
||||
cc_library(
|
||||
name = "mediapipe_jni_lib",
|
||||
srcs = [":libmediapipe_jni.so"],
|
||||
alwayslink = 1,
|
||||
)
|
||||
|
||||
android_binary(
|
||||
name = "facedetectionfullrangegpu",
|
||||
srcs = glob(["*.java"]),
|
||||
assets = [
|
||||
"//mediapipe/graphs/face_detection:face_detection_full_range_mobile_gpu.binarypb",
|
||||
"//mediapipe/modules/face_detection:face_detection_full_range_sparse.tflite",
|
||||
],
|
||||
assets_dir = "",
|
||||
manifest = "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:AndroidManifest.xml",
|
||||
manifest_values = {
|
||||
"applicationId": "com.google.mediapipe.apps.facedetectionfullrangegpu",
|
||||
"appName": "Face Detection Full-range (GPU)",
|
||||
"mainActivity": "com.google.mediapipe.apps.basic.MainActivity",
|
||||
"cameraFacingFront": "False",
|
||||
"binaryGraphName": "face_detection_full_range_mobile_gpu.binarypb",
|
||||
"inputVideoStreamName": "input_video",
|
||||
"outputVideoStreamName": "output_video",
|
||||
"flipFramesVertically": "True",
|
||||
"converterNumBuffers": "2",
|
||||
},
|
||||
multidex = "native",
|
||||
deps = [
|
||||
":mediapipe_jni_lib",
|
||||
"//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:basic_lib",
|
||||
],
|
||||
)
|
||||
+1
-1
@@ -37,7 +37,7 @@ android_binary(
|
||||
srcs = glob(["*.java"]),
|
||||
assets = [
|
||||
"//mediapipe/graphs/face_detection:face_detection_mobile_gpu.binarypb",
|
||||
"//mediapipe/modules/face_detection:face_detection_front.tflite",
|
||||
"//mediapipe/modules/face_detection:face_detection_short_range.tflite",
|
||||
],
|
||||
assets_dir = "",
|
||||
manifest = "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:AndroidManifest.xml",
|
||||
|
||||
@@ -42,7 +42,7 @@ android_binary(
|
||||
"//mediapipe/graphs/face_effect/data:glasses.binarypb",
|
||||
"//mediapipe/graphs/face_effect/data:glasses.pngblob",
|
||||
"//mediapipe/graphs/face_effect:face_effect_gpu.binarypb",
|
||||
"//mediapipe/modules/face_detection:face_detection_front.tflite",
|
||||
"//mediapipe/modules/face_detection:face_detection_short_range.tflite",
|
||||
"//mediapipe/modules/face_geometry/data:geometry_pipeline_metadata_detection.binarypb",
|
||||
"//mediapipe/modules/face_geometry/data:geometry_pipeline_metadata_landmarks.binarypb",
|
||||
"//mediapipe/modules/face_landmark:face_landmark.tflite",
|
||||
|
||||
@@ -38,7 +38,7 @@ android_binary(
|
||||
assets = [
|
||||
"//mediapipe/graphs/face_mesh:face_mesh_mobile_gpu.binarypb",
|
||||
"//mediapipe/modules/face_landmark:face_landmark.tflite",
|
||||
"//mediapipe/modules/face_detection:face_detection_front.tflite",
|
||||
"//mediapipe/modules/face_detection:face_detection_short_range.tflite",
|
||||
],
|
||||
assets_dir = "",
|
||||
manifest = "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:AndroidManifest.xml",
|
||||
|
||||
+1
-1
@@ -37,7 +37,7 @@ android_binary(
|
||||
srcs = glob(["*.java"]),
|
||||
assets = [
|
||||
"//mediapipe/graphs/holistic_tracking:holistic_tracking_gpu.binarypb",
|
||||
"//mediapipe/modules/face_detection:face_detection_front.tflite",
|
||||
"//mediapipe/modules/face_detection:face_detection_short_range.tflite",
|
||||
"//mediapipe/modules/face_landmark:face_landmark.tflite",
|
||||
"//mediapipe/modules/hand_landmark:hand_landmark.tflite",
|
||||
"//mediapipe/modules/hand_landmark:handedness.txt",
|
||||
|
||||
@@ -39,7 +39,7 @@ android_binary(
|
||||
"//mediapipe/graphs/iris_tracking:iris_tracking_gpu.binarypb",
|
||||
"//mediapipe/modules/face_landmark:face_landmark.tflite",
|
||||
"//mediapipe/modules/iris_landmark:iris_landmark.tflite",
|
||||
"//mediapipe/modules/face_detection:face_detection_front.tflite",
|
||||
"//mediapipe/modules/face_detection:face_detection_short_range.tflite",
|
||||
],
|
||||
assets_dir = "",
|
||||
manifest = "//mediapipe/examples/android/src/java/com/google/mediapipe/apps/basic:AndroidManifest.xml",
|
||||
|
||||
@@ -37,6 +37,13 @@ constexpr char kFirstCropRect[] = "FIRST_CROP_RECT";
|
||||
// (configured through option us_to_first_rect). If provided, a non-zero integer
|
||||
// will allow the animated zoom to be used when the first detections arrive.
|
||||
constexpr char kAnimateZoom[] = "ANIMATE_ZOOM";
|
||||
// Can be used to control the maximum zoom; note that it is re-evaluated only
|
||||
// upon change of input resolution. A value of 100 disables zooming and is the
|
||||
// smallest allowed value. A value of 200 allows zooming such that a pixel of
|
||||
// the input may cover up to four times its original area. Note that
|
||||
// max_zoom_value_deg from options is always respected; MAX_ZOOM_PCT can only be
|
||||
// used to limit zooming further.
|
||||
constexpr char kMaxZoomFactorPercent[] = "MAX_ZOOM_FACTOR_PCT";
|
||||
// Field-of-view (degrees) of the camera's x-axis (width).
|
||||
// TODO: Parameterize FOV based on camera specs.
|
||||
constexpr float kFieldOfView = 60;
|
||||
@@ -75,11 +82,16 @@ class ContentZoomingCalculator : public CalculatorBase {
|
||||
int frame_height);
|
||||
// Saves state to a state-cache, if provided.
|
||||
absl::Status SaveState(mediapipe::CalculatorContext* cc) const;
|
||||
// Returns the factor for maximum zoom based on options and the
|
||||
// kMaxZoomFactorPercent input (if present).
|
||||
double GetMaxZoomFactor(mediapipe::CalculatorContext* cc) const;
|
||||
// Initializes the calculator for the given frame size, creating path solvers
|
||||
// and resetting history like last measured values.
|
||||
absl::Status InitializeState(int frame_width, int frame_height);
|
||||
absl::Status InitializeState(mediapipe::CalculatorContext* cc,
|
||||
int frame_width, int frame_height);
|
||||
// Adjusts state to work with an updated frame size.
|
||||
absl::Status UpdateForResolutionChange(int frame_width, int frame_height);
|
||||
absl::Status UpdateForResolutionChange(mediapipe::CalculatorContext* cc,
|
||||
int frame_width, int frame_height);
|
||||
// Returns true if we are animating to the first rect.
|
||||
bool IsAnimatingToFirstRect(const Timestamp& timestamp) const;
|
||||
// Builds the output rectangle when animating to the first rect.
|
||||
@@ -136,6 +148,9 @@ absl::Status ContentZoomingCalculator::GetContract(
|
||||
return mediapipe::UnknownErrorBuilder(MEDIAPIPE_LOC)
|
||||
<< "Input VIDEO or VIDEO_SIZE must be provided.";
|
||||
}
|
||||
if (cc->Inputs().HasTag(kMaxZoomFactorPercent)) {
|
||||
cc->Inputs().Tag(kMaxZoomFactorPercent).Set<int>();
|
||||
}
|
||||
if (cc->Inputs().HasTag(kSalientRegions)) {
|
||||
cc->Inputs().Tag(kSalientRegions).Set<DetectionSet>();
|
||||
}
|
||||
@@ -330,7 +345,7 @@ absl::Status ContentZoomingCalculator::MaybeLoadState(
|
||||
? cc->InputSidePackets().Tag(kStateCache).Get<StateCacheType*>()
|
||||
: nullptr;
|
||||
if (!state_cache || !state_cache->has_value()) {
|
||||
return InitializeState(frame_width, frame_height);
|
||||
return InitializeState(cc, frame_width, frame_height);
|
||||
}
|
||||
|
||||
const ContentZoomingCalculatorState& state = state_cache->value();
|
||||
@@ -350,7 +365,7 @@ absl::Status ContentZoomingCalculator::MaybeLoadState(
|
||||
last_measured_y_offset_ = state.last_measured_y_offset;
|
||||
MP_RETURN_IF_ERROR(UpdateAspectAndMax());
|
||||
|
||||
return UpdateForResolutionChange(frame_width, frame_height);
|
||||
return UpdateForResolutionChange(cc, frame_width, frame_height);
|
||||
}
|
||||
|
||||
absl::Status ContentZoomingCalculator::SaveState(
|
||||
@@ -379,8 +394,20 @@ absl::Status ContentZoomingCalculator::SaveState(
|
||||
return absl::OkStatus();
|
||||
}
|
||||
|
||||
absl::Status ContentZoomingCalculator::InitializeState(int frame_width,
|
||||
int frame_height) {
|
||||
double ContentZoomingCalculator::GetMaxZoomFactor(
|
||||
mediapipe::CalculatorContext* cc) const {
|
||||
double max_zoom_value =
|
||||
options_.max_zoom_value_deg() / static_cast<double>(kFieldOfView);
|
||||
if (cc->Inputs().HasTag(kMaxZoomFactorPercent)) {
|
||||
const double factor = std::max(
|
||||
1.0, cc->Inputs().Tag(kMaxZoomFactorPercent).Get<int>() / 100.0);
|
||||
max_zoom_value = std::max(max_zoom_value, 1.0 / factor);
|
||||
}
|
||||
return max_zoom_value;
|
||||
}
|
||||
|
||||
absl::Status ContentZoomingCalculator::InitializeState(
|
||||
mediapipe::CalculatorContext* cc, int frame_width, int frame_height) {
|
||||
frame_width_ = frame_width;
|
||||
frame_height_ = frame_height;
|
||||
path_solver_pan_ = std::make_unique<KinematicPathSolver>(
|
||||
@@ -390,8 +417,7 @@ absl::Status ContentZoomingCalculator::InitializeState(int frame_width,
|
||||
options_.kinematic_options_tilt(), 0, frame_height_,
|
||||
static_cast<float>(frame_height_) / kFieldOfView);
|
||||
MP_RETURN_IF_ERROR(UpdateAspectAndMax());
|
||||
int min_zoom_size = frame_height_ * (options_.max_zoom_value_deg() /
|
||||
static_cast<double>(kFieldOfView));
|
||||
int min_zoom_size = frame_height_ * GetMaxZoomFactor(cc);
|
||||
path_solver_zoom_ = std::make_unique<KinematicPathSolver>(
|
||||
options_.kinematic_options_zoom(), min_zoom_size,
|
||||
max_frame_value_ * frame_height_,
|
||||
@@ -405,7 +431,7 @@ absl::Status ContentZoomingCalculator::InitializeState(int frame_width,
|
||||
}
|
||||
|
||||
absl::Status ContentZoomingCalculator::UpdateForResolutionChange(
|
||||
int frame_width, int frame_height) {
|
||||
mediapipe::CalculatorContext* cc, int frame_width, int frame_height) {
|
||||
// Update state for change in input resolution.
|
||||
if (frame_width_ != frame_width || frame_height_ != frame_height) {
|
||||
double width_scale = frame_width / static_cast<double>(frame_width_);
|
||||
@@ -419,8 +445,7 @@ absl::Status ContentZoomingCalculator::UpdateForResolutionChange(
|
||||
MP_RETURN_IF_ERROR(path_solver_pan_->UpdateMinMaxLocation(0, frame_width_));
|
||||
MP_RETURN_IF_ERROR(
|
||||
path_solver_tilt_->UpdateMinMaxLocation(0, frame_height_));
|
||||
int min_zoom_size = frame_height_ * (options_.max_zoom_value_deg() /
|
||||
static_cast<double>(kFieldOfView));
|
||||
int min_zoom_size = frame_height_ * GetMaxZoomFactor(cc);
|
||||
MP_RETURN_IF_ERROR(path_solver_zoom_->UpdateMinMaxLocation(
|
||||
min_zoom_size, max_frame_value_ * frame_height_));
|
||||
MP_RETURN_IF_ERROR(path_solver_zoom_->UpdatePixelsPerDegree(
|
||||
@@ -493,7 +518,8 @@ absl::Status ContentZoomingCalculator::Process(
|
||||
MP_RETURN_IF_ERROR(MaybeLoadState(cc, frame_width, frame_height));
|
||||
initialized_ = !options_.is_stateless();
|
||||
} else {
|
||||
MP_RETURN_IF_ERROR(UpdateForResolutionChange(frame_width, frame_height));
|
||||
MP_RETURN_IF_ERROR(
|
||||
UpdateForResolutionChange(cc, frame_width, frame_height));
|
||||
}
|
||||
|
||||
bool only_required_found = false;
|
||||
@@ -634,14 +660,18 @@ absl::Status ContentZoomingCalculator::Process(
|
||||
// Prevent box from extending beyond the image after camera smoothing.
|
||||
if (path_offset_y - ceil(path_height / 2.0) < 0) {
|
||||
path_offset_y = ceil(path_height / 2.0);
|
||||
MP_RETURN_IF_ERROR(path_solver_tilt_->SetState(path_offset_y));
|
||||
} else if (path_offset_y + ceil(path_height / 2.0) > frame_height_) {
|
||||
path_offset_y = frame_height_ - ceil(path_height / 2.0);
|
||||
MP_RETURN_IF_ERROR(path_solver_tilt_->SetState(path_offset_y));
|
||||
}
|
||||
|
||||
if (path_offset_x - ceil(path_width / 2.0) < 0) {
|
||||
path_offset_x = ceil(path_width / 2.0);
|
||||
MP_RETURN_IF_ERROR(path_solver_pan_->SetState(path_offset_x));
|
||||
} else if (path_offset_x + ceil(path_width / 2.0) > frame_width_) {
|
||||
path_offset_x = frame_width_ - ceil(path_width / 2.0);
|
||||
MP_RETURN_IF_ERROR(path_solver_pan_->SetState(path_offset_x));
|
||||
}
|
||||
|
||||
// Convert to top/bottom borders to remove.
|
||||
|
||||
@@ -150,6 +150,29 @@ const char kConfigE[] = R"(
|
||||
}
|
||||
)";
|
||||
|
||||
const char kConfigF[] = R"(
|
||||
calculator: "ContentZoomingCalculator"
|
||||
input_stream: "VIDEO_SIZE:size"
|
||||
input_stream: "DETECTIONS:detections"
|
||||
input_stream: "MAX_ZOOM_FACTOR_PCT:max_zoom_factor_pct"
|
||||
output_stream: "CROP_RECT:rect"
|
||||
output_stream: "FIRST_CROP_RECT:first_rect"
|
||||
options: {
|
||||
[mediapipe.autoflip.ContentZoomingCalculatorOptions.ext]: {
|
||||
max_zoom_value_deg: 0
|
||||
kinematic_options_zoom {
|
||||
min_motion_to_reframe: 1.2
|
||||
}
|
||||
kinematic_options_tilt {
|
||||
min_motion_to_reframe: 1.2
|
||||
}
|
||||
kinematic_options_pan {
|
||||
min_motion_to_reframe: 1.2
|
||||
}
|
||||
}
|
||||
}
|
||||
)";
|
||||
|
||||
void CheckBorder(const StaticFeatures& static_features, int width, int height,
|
||||
int top_border, int bottom_border) {
|
||||
ASSERT_EQ(2, static_features.border().size());
|
||||
@@ -170,6 +193,7 @@ void CheckBorder(const StaticFeatures& static_features, int width, int height,
|
||||
|
||||
struct AddDetectionFlags {
|
||||
std::optional<bool> animated_zoom;
|
||||
std::optional<int> max_zoom_factor_percent;
|
||||
};
|
||||
|
||||
void AddDetectionFrameSize(const cv::Rect_<float>& position, const int64 time,
|
||||
@@ -211,6 +235,14 @@ void AddDetectionFrameSize(const cv::Rect_<float>& position, const int64 time,
|
||||
mediapipe::MakePacket<bool>(flags.animated_zoom.value())
|
||||
.At(Timestamp(time)));
|
||||
}
|
||||
|
||||
if (flags.max_zoom_factor_percent.has_value()) {
|
||||
runner->MutableInputs()
|
||||
->Tag("MAX_ZOOM_FACTOR_PCT")
|
||||
.packets.push_back(
|
||||
mediapipe::MakePacket<int>(flags.max_zoom_factor_percent.value())
|
||||
.At(Timestamp(time)));
|
||||
}
|
||||
}
|
||||
|
||||
void AddDetection(const cv::Rect_<float>& position, const int64 time,
|
||||
@@ -259,6 +291,7 @@ TEST(ContentZoomingCalculatorTest, ZoomTest) {
|
||||
CheckBorder(static_features, 1000, 1000, 495, 395);
|
||||
}
|
||||
|
||||
#if 0
|
||||
TEST(ContentZoomingCalculatorTest, ZoomTestFullPTZ) {
|
||||
auto runner = ::absl::make_unique<CalculatorRunner>(
|
||||
ParseTextProtoOrDie<CalculatorGraphConfig::Node>(kConfigD));
|
||||
@@ -694,8 +727,8 @@ TEST(ContentZoomingCalculatorTest, ResolutionChangeZoomingWithCache) {
|
||||
auto runner = ::absl::make_unique<CalculatorRunner>(config);
|
||||
runner->MutableSidePackets()->Tag("STATE_CACHE") = MakePacket<
|
||||
mediapipe::autoflip::ContentZoomingCalculatorStateCacheType*>(&cache);
|
||||
AddDetectionFrameSize(cv::Rect_<float>(.4, .4, .2, .2), 1000000, 1000, 1000,
|
||||
runner.get());
|
||||
AddDetectionFrameSize(cv::Rect_<float>(.4, .4, .2, .2), 1000000, 1000,
|
||||
1000, runner.get());
|
||||
AddDetectionFrameSize(cv::Rect_<float>(.4, .4, .2, .2), 2000000, 500, 500,
|
||||
runner.get());
|
||||
MP_ASSERT_OK(runner->Run());
|
||||
@@ -719,6 +752,36 @@ TEST(ContentZoomingCalculatorTest, MaxZoomValue) {
|
||||
CheckCropRect(500, 500, 916, 916, 0,
|
||||
runner->Outputs().Tag("CROP_RECT").packets);
|
||||
}
|
||||
#endif
|
||||
|
||||
TEST(ContentZoomingCalculatorTest, MaxZoomValueOverride) {
|
||||
auto config = ParseTextProtoOrDie<CalculatorGraphConfig::Node>(kConfigF);
|
||||
auto* options = config.mutable_options()->MutableExtension(
|
||||
ContentZoomingCalculatorOptions::ext);
|
||||
options->set_max_zoom_value_deg(30);
|
||||
auto runner = ::absl::make_unique<CalculatorRunner>(config);
|
||||
AddDetectionFrameSize(cv::Rect_<float>(.4, .4, .2, .2), 0, 640, 480,
|
||||
runner.get(), {.max_zoom_factor_percent = 133});
|
||||
// Change resolution and allow more zoom, and give time to use the new limit
|
||||
AddDetectionFrameSize(cv::Rect_<float>(.4, .4, .2, .2), 1000000, 1280, 720,
|
||||
runner.get(), {.max_zoom_factor_percent = 166});
|
||||
AddDetectionFrameSize(cv::Rect_<float>(.4, .4, .2, .2), 2000000, 1280, 720,
|
||||
runner.get(), {.max_zoom_factor_percent = 166});
|
||||
// Switch back to a smaller resolution with a more limited zoom
|
||||
AddDetectionFrameSize(cv::Rect_<float>(.4, .4, .2, .2), 3000000, 640, 480,
|
||||
runner.get(), {.max_zoom_factor_percent = 133});
|
||||
MP_ASSERT_OK(runner->Run());
|
||||
// Max. 133% zoomed in means min. (100/133) ~ 75% of height left: ~360
|
||||
// Max. 166% zoomed in means min. (100/166) ~ 60% of height left: ~430
|
||||
CheckCropRect(320, 240, 480, 360, 0,
|
||||
runner->Outputs().Tag("CROP_RECT").packets);
|
||||
CheckCropRect(640, 360, 769, 433, 2,
|
||||
runner->Outputs().Tag("CROP_RECT").packets);
|
||||
CheckCropRect(320, 240, 480, 360, 3,
|
||||
runner->Outputs().Tag("CROP_RECT").packets);
|
||||
}
|
||||
|
||||
#if 0
|
||||
TEST(ContentZoomingCalculatorTest, MaxZoomOutValue) {
|
||||
auto config = ParseTextProtoOrDie<CalculatorGraphConfig::Node>(kConfigD);
|
||||
auto* options = config.mutable_options()->MutableExtension(
|
||||
@@ -906,6 +969,7 @@ TEST(ContentZoomingCalculatorTest, ProvidesConstantFirstRect) {
|
||||
EXPECT_EQ(first_rect.height(), rect.height());
|
||||
}
|
||||
}
|
||||
#endif
|
||||
|
||||
} // namespace
|
||||
} // namespace autoflip
|
||||
|
||||
@@ -68,5 +68,10 @@ message FaceBoxAdjusterCalculatorOptions {
|
||||
|
||||
// The max amount of time to use an old eye distance when the face look angle
|
||||
// is unstable.
|
||||
optional int32 max_facesize_history_us = 9 [default = 8000000];
|
||||
optional int32 max_facesize_history_us = 9 [default = 300000000];
|
||||
|
||||
// Scale factor of face width to shift based on pan look angle.
|
||||
optional float pan_position_shift_scale = 15 [default = 0.5];
|
||||
// Scale factor of face height to shift based on tilt look angle.
|
||||
optional float tilt_position_shift_scale = 16 [default = 0.5];
|
||||
}
|
||||
|
||||
@@ -154,10 +154,18 @@ absl::Status KinematicPathSolver::AddObservation(int position,
|
||||
|
||||
// Time and position updates.
|
||||
double delta_t = (time_us - current_time_) / 1000000.0;
|
||||
// Time since last state/prediction update, smoothed by
|
||||
// mean_period_update_rate.
|
||||
if (mean_delta_t_ < 0) {
|
||||
mean_delta_t_ = delta_t;
|
||||
} else {
|
||||
mean_delta_t_ = mean_delta_t_ * (1 - options_.mean_period_update_rate()) +
|
||||
delta_t * options_.mean_period_update_rate();
|
||||
}
|
||||
|
||||
// Observed velocity and then weighted update of this velocity.
|
||||
double observed_velocity = delta_degs / delta_t;
|
||||
double update_rate = std::min(delta_t / options_.update_rate_seconds(),
|
||||
double update_rate = std::min(mean_delta_t_ / options_.update_rate_seconds(),
|
||||
options_.max_update_rate());
|
||||
double updated_velocity = current_velocity_deg_per_s_ * (1 - update_rate) +
|
||||
observed_velocity * update_rate;
|
||||
@@ -174,16 +182,6 @@ absl::Status KinematicPathSolver::UpdatePrediction(const int64 time_us) {
|
||||
RET_CHECK(current_time_ < time_us)
|
||||
<< "Prediction time added before a prior observation or prediction.";
|
||||
|
||||
// Time since last state/prediction update, smoothed by
|
||||
// mean_period_update_rate.
|
||||
double delta_t = (time_us - current_time_) / 1000000.0;
|
||||
if (mean_delta_t_ < 0) {
|
||||
mean_delta_t_ = delta_t;
|
||||
} else {
|
||||
mean_delta_t_ = mean_delta_t_ * (1 - options_.mean_period_update_rate()) +
|
||||
delta_t * options_.mean_period_update_rate();
|
||||
}
|
||||
|
||||
// Position update limited by min/max.
|
||||
double update_position_px =
|
||||
current_position_px_ +
|
||||
@@ -211,6 +209,12 @@ absl::Status KinematicPathSolver::GetState(int* position) {
|
||||
return absl::OkStatus();
|
||||
}
|
||||
|
||||
absl::Status KinematicPathSolver::SetState(const int position) {
|
||||
RET_CHECK(initialized_) << "SetState called before first observation added.";
|
||||
current_position_px_ = position;
|
||||
return absl::OkStatus();
|
||||
}
|
||||
|
||||
absl::Status KinematicPathSolver::GetTargetPosition(int* target_position) {
|
||||
RET_CHECK(initialized_)
|
||||
<< "GetTargetPosition called before first observation added.";
|
||||
|
||||
@@ -48,6 +48,8 @@ class KinematicPathSolver {
|
||||
absl::Status UpdatePrediction(const int64 time_us);
|
||||
// Get the state at a time.
|
||||
absl::Status GetState(int* position);
|
||||
// Overwrite the current state value.
|
||||
absl::Status SetState(const int position);
|
||||
// Update PixelPerDegree value.
|
||||
absl::Status UpdatePixelsPerDegree(const float pixels_per_degree);
|
||||
// Provide the current target position of the reframe action.
|
||||
|
||||
@@ -337,6 +337,61 @@ TEST(KinematicPathSolverTest, PassDegPerPxChange) {
|
||||
EXPECT_EQ(state, 516);
|
||||
}
|
||||
|
||||
TEST(KinematicPathSolverTest, NoTimestampSmoothing) {
|
||||
KinematicOptions options;
|
||||
options.set_min_motion_to_reframe(1.0);
|
||||
options.set_update_rate(1.0);
|
||||
options.set_max_velocity(6);
|
||||
options.set_mean_period_update_rate(1.0);
|
||||
KinematicPathSolver solver(options, 0, 1000, 1000.0 / kWidthFieldOfView);
|
||||
int state;
|
||||
MP_ASSERT_OK(solver.AddObservation(500, 0));
|
||||
MP_ASSERT_OK(solver.AddObservation(1000, 1000000));
|
||||
MP_ASSERT_OK(solver.GetState(&state));
|
||||
EXPECT_EQ(state, 600);
|
||||
MP_ASSERT_OK(solver.AddObservation(1000, 2200000));
|
||||
MP_ASSERT_OK(solver.GetState(&state));
|
||||
EXPECT_EQ(state, 720);
|
||||
}
|
||||
|
||||
TEST(KinematicPathSolverTest, TimestampSmoothing) {
|
||||
KinematicOptions options;
|
||||
options.set_min_motion_to_reframe(1.0);
|
||||
options.set_update_rate(1.0);
|
||||
options.set_max_velocity(6);
|
||||
options.set_mean_period_update_rate(0.05);
|
||||
KinematicPathSolver solver(options, 0, 1000, 1000.0 / kWidthFieldOfView);
|
||||
int state;
|
||||
MP_ASSERT_OK(solver.AddObservation(500, 0));
|
||||
MP_ASSERT_OK(solver.AddObservation(1000, 1000000));
|
||||
MP_ASSERT_OK(solver.GetState(&state));
|
||||
EXPECT_EQ(state, 600);
|
||||
MP_ASSERT_OK(solver.AddObservation(1000, 2200000));
|
||||
MP_ASSERT_OK(solver.GetState(&state));
|
||||
EXPECT_EQ(state, 701);
|
||||
}
|
||||
|
||||
TEST(KinematicPathSolverTest, PassSetPosition) {
|
||||
KinematicOptions options;
|
||||
// Set min motion to 2deg
|
||||
options.set_min_motion_to_reframe(1.0);
|
||||
options.set_update_rate_seconds(.0000001);
|
||||
options.set_max_update_rate(1.0);
|
||||
options.set_max_velocity(18);
|
||||
// Set degrees / pixel to 8.3
|
||||
KinematicPathSolver solver(options, 0, 500, 500.0 / kWidthFieldOfView);
|
||||
int state;
|
||||
MP_ASSERT_OK(solver.AddObservation(400, kMicroSecInSec * 0));
|
||||
// Move target by 10px / 8.3 = 1.2deg
|
||||
MP_ASSERT_OK(solver.AddObservation(410, kMicroSecInSec * 1));
|
||||
MP_ASSERT_OK(solver.GetState(&state));
|
||||
// Expect cam to move.
|
||||
EXPECT_EQ(state, 410);
|
||||
MP_ASSERT_OK(solver.SetState(400));
|
||||
MP_ASSERT_OK(solver.GetState(&state));
|
||||
EXPECT_EQ(state, 400);
|
||||
}
|
||||
|
||||
} // namespace
|
||||
} // namespace autoflip
|
||||
} // namespace mediapipe
|
||||
|
||||
@@ -38,7 +38,7 @@ node {
|
||||
output_stream: "TENSORS:detection_tensors"
|
||||
options: {
|
||||
[mediapipe.TfLiteInferenceCalculatorOptions.ext] {
|
||||
model_path: "mediapipe/modules/face_detection/face_detection_back.tflite"
|
||||
model_path: "mediapipe/modules/face_detection/face_detection_full_range_sparse.tflite"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
@@ -16,6 +16,14 @@ licenses(["notice"])
|
||||
|
||||
package(default_visibility = ["//mediapipe/examples:__subpackages__"])
|
||||
|
||||
cc_binary(
|
||||
name = "face_detection_full_range_cpu",
|
||||
deps = [
|
||||
"//mediapipe/examples/desktop:demo_run_graph_main",
|
||||
"//mediapipe/graphs/face_detection:face_detection_full_range_desktop_live_deps",
|
||||
],
|
||||
)
|
||||
|
||||
cc_binary(
|
||||
name = "face_detection_cpu",
|
||||
deps = [
|
||||
|
||||
@@ -55,7 +55,7 @@ objc_library(
|
||||
name = "FaceDetectionCpuAppLibrary",
|
||||
data = [
|
||||
"//mediapipe/graphs/face_detection:face_detection_mobile_cpu.binarypb",
|
||||
"//mediapipe/modules/face_detection:face_detection_front.tflite",
|
||||
"//mediapipe/modules/face_detection:face_detection_short_range.tflite",
|
||||
],
|
||||
deps = [
|
||||
"//mediapipe/examples/ios/common:CommonMediaPipeAppLibrary",
|
||||
|
||||
@@ -55,7 +55,7 @@ objc_library(
|
||||
name = "FaceDetectionGpuAppLibrary",
|
||||
data = [
|
||||
"//mediapipe/graphs/face_detection:face_detection_mobile_gpu.binarypb",
|
||||
"//mediapipe/modules/face_detection:face_detection_front.tflite",
|
||||
"//mediapipe/modules/face_detection:face_detection_short_range.tflite",
|
||||
],
|
||||
deps = [
|
||||
"//mediapipe/examples/ios/common:CommonMediaPipeAppLibrary",
|
||||
|
||||
@@ -66,7 +66,7 @@ objc_library(
|
||||
"//mediapipe/graphs/face_effect/data:facepaint.pngblob",
|
||||
"//mediapipe/graphs/face_effect/data:glasses.binarypb",
|
||||
"//mediapipe/graphs/face_effect/data:glasses.pngblob",
|
||||
"//mediapipe/modules/face_detection:face_detection_front.tflite",
|
||||
"//mediapipe/modules/face_detection:face_detection_short_range.tflite",
|
||||
"//mediapipe/modules/face_geometry/data:geometry_pipeline_metadata.binarypb",
|
||||
"//mediapipe/modules/face_geometry/data:geometry_pipeline_metadata_detection.binarypb",
|
||||
"//mediapipe/modules/face_geometry/data:geometry_pipeline_metadata_landmarks.binarypb",
|
||||
@@ -109,7 +109,7 @@ objc_library(
|
||||
"//mediapipe/graphs/face_effect/data:facepaint.pngblob",
|
||||
"//mediapipe/graphs/face_effect/data:glasses.binarypb",
|
||||
"//mediapipe/graphs/face_effect/data:glasses.pngblob",
|
||||
"//mediapipe/modules/face_detection:face_detection_front.tflite",
|
||||
"//mediapipe/modules/face_detection:face_detection_short_range.tflite",
|
||||
"//mediapipe/modules/face_geometry/data:geometry_pipeline_metadata.binarypb",
|
||||
"//mediapipe/modules/face_landmark:face_landmark.tflite",
|
||||
],
|
||||
|
||||
@@ -62,7 +62,7 @@ objc_library(
|
||||
copts = ["-std=c++17"],
|
||||
data = [
|
||||
"//mediapipe/graphs/face_mesh:face_mesh_mobile_gpu.binarypb",
|
||||
"//mediapipe/modules/face_detection:face_detection_front.tflite",
|
||||
"//mediapipe/modules/face_detection:face_detection_short_range.tflite",
|
||||
"//mediapipe/modules/face_landmark:face_landmark.tflite",
|
||||
],
|
||||
deps = [
|
||||
|
||||
@@ -55,7 +55,7 @@ objc_library(
|
||||
name = "HolisticTrackingGpuAppLibrary",
|
||||
data = [
|
||||
"//mediapipe/graphs/holistic_tracking:holistic_tracking_gpu.binarypb",
|
||||
"//mediapipe/modules/face_detection:face_detection_front.tflite",
|
||||
"//mediapipe/modules/face_detection:face_detection_short_range.tflite",
|
||||
"//mediapipe/modules/face_landmark:face_landmark.tflite",
|
||||
"//mediapipe/modules/hand_landmark:hand_landmark.tflite",
|
||||
"//mediapipe/modules/hand_landmark:handedness.txt",
|
||||
|
||||
@@ -62,7 +62,7 @@ objc_library(
|
||||
copts = ["-std=c++17"],
|
||||
data = [
|
||||
"//mediapipe/graphs/iris_tracking:iris_tracking_gpu.binarypb",
|
||||
"//mediapipe/modules/face_detection:face_detection_front.tflite",
|
||||
"//mediapipe/modules/face_detection:face_detection_short_range.tflite",
|
||||
"//mediapipe/modules/face_landmark:face_landmark.tflite",
|
||||
"//mediapipe/modules/iris_landmark:iris_landmark.tflite",
|
||||
],
|
||||
|
||||
@@ -953,6 +953,9 @@ cc_library(
|
||||
}) + select({
|
||||
"//conditions:default": [],
|
||||
"//mediapipe/gpu:disable_gpu": ["MEDIAPIPE_DISABLE_GPU=1"],
|
||||
}) + select({
|
||||
"//conditions:default": [],
|
||||
"//mediapipe/framework/port:disable_opencv": ["MEDIAPIPE_DISABLE_OPENCV=1"],
|
||||
}) + select({
|
||||
"//conditions:default": [],
|
||||
"//mediapipe/framework:disable_rtti_and_exceptions": [
|
||||
|
||||
@@ -17,6 +17,14 @@ namespace mediapipe {
|
||||
namespace api2 {
|
||||
namespace builder {
|
||||
|
||||
// Workaround for static_assert(false). Example:
|
||||
// dependent_false<T>::value returns false.
|
||||
// For more information, see:
|
||||
// https://en.cppreference.com/w/cpp/language/if#Constexpr_If
|
||||
// TODO: migrate to a common utility when available.
|
||||
template <class T>
|
||||
struct dependent_false : std::false_type {};
|
||||
|
||||
template <typename T>
|
||||
T& GetWithAutoGrow(std::vector<std::unique_ptr<T>>* vecp, int index) {
|
||||
auto& vec = *vecp;
|
||||
@@ -209,6 +217,21 @@ class NodeBase {
|
||||
return SideDestination<true>(&in_sides_[tag]);
|
||||
}
|
||||
|
||||
template <typename B, typename T, bool kIsOptional, bool kIsMultiple>
|
||||
auto operator[](const PortCommon<B, T, kIsOptional, kIsMultiple>& port) {
|
||||
if constexpr (std::is_same_v<B, OutputBase>) {
|
||||
return Source<kIsMultiple, T>(&out_streams_[port.Tag()]);
|
||||
} else if constexpr (std::is_same_v<B, InputBase>) {
|
||||
return Destination<kIsMultiple, T>(&in_streams_[port.Tag()]);
|
||||
} else if constexpr (std::is_same_v<B, SideOutputBase>) {
|
||||
return SideSource<kIsMultiple, T>(&out_sides_[port.Tag()]);
|
||||
} else if constexpr (std::is_same_v<B, SideInputBase>) {
|
||||
return SideDestination<kIsMultiple, T>(&in_sides_[port.Tag()]);
|
||||
} else {
|
||||
static_assert(dependent_false<B>::value, "Type not supported.");
|
||||
}
|
||||
}
|
||||
|
||||
// Convenience methods for accessing purely index-based ports.
|
||||
Source<false> Out(int index) { return Out("")[index]; }
|
||||
|
||||
@@ -429,6 +452,24 @@ class Graph {
|
||||
return Dst(&graph_boundary_.in_sides_[graph_output.Tag()]);
|
||||
}
|
||||
|
||||
template <typename B, typename T, bool kIsOptional, bool kIsMultiple>
|
||||
auto operator[](const PortCommon<B, T, kIsOptional, kIsMultiple>& port) {
|
||||
if constexpr (std::is_same_v<B, OutputBase>) {
|
||||
return Destination<kIsMultiple, T>(
|
||||
&graph_boundary_.in_streams_[port.Tag()]);
|
||||
} else if constexpr (std::is_same_v<B, InputBase>) {
|
||||
return Source<kIsMultiple, T>(&graph_boundary_.out_streams_[port.Tag()]);
|
||||
} else if constexpr (std::is_same_v<B, SideOutputBase>) {
|
||||
return SideDestination<kIsMultiple, T>(
|
||||
&graph_boundary_.in_sides_[port.Tag()]);
|
||||
} else if constexpr (std::is_same_v<B, SideInputBase>) {
|
||||
return SideSource<kIsMultiple, T>(
|
||||
&graph_boundary_.out_sides_[port.Tag()]);
|
||||
} else {
|
||||
static_assert(dependent_false<B>::value, "Type not supported.");
|
||||
}
|
||||
}
|
||||
|
||||
// Returns the graph config. This can be used to instantiate and run the
|
||||
// graph.
|
||||
CalculatorGraphConfig GetConfig() {
|
||||
|
||||
@@ -138,6 +138,35 @@ TEST(BuilderTest, TypedMultiple) {
|
||||
EXPECT_THAT(graph.GetConfig(), EqualsProto(expected));
|
||||
}
|
||||
|
||||
TEST(BuilderTest, TypedByPorts) {
|
||||
builder::Graph graph;
|
||||
auto& foo = graph.AddNode<Foo>();
|
||||
auto& adder = graph.AddNode<FloatAdder>();
|
||||
|
||||
graph[FooBar1::kIn].SetName("base") >> foo[Foo::kBase];
|
||||
foo[Foo::kOut] >> adder[FloatAdder::kIn][0];
|
||||
foo[Foo::kOut] >> adder[FloatAdder::kIn][1];
|
||||
adder[FloatAdder::kOut].SetName("out") >> graph[FooBar1::kOut];
|
||||
|
||||
CalculatorGraphConfig expected =
|
||||
mediapipe::ParseTextProtoOrDie<CalculatorGraphConfig>(R"pb(
|
||||
input_stream: "IN:base"
|
||||
output_stream: "OUT:out"
|
||||
node {
|
||||
calculator: "Foo"
|
||||
input_stream: "BASE:base"
|
||||
output_stream: "OUT:__stream_0"
|
||||
}
|
||||
node {
|
||||
calculator: "FloatAdder"
|
||||
input_stream: "IN:0:__stream_0"
|
||||
input_stream: "IN:1:__stream_0"
|
||||
output_stream: "OUT:out"
|
||||
}
|
||||
)pb");
|
||||
EXPECT_THAT(graph.GetConfig(), EqualsProto(expected));
|
||||
}
|
||||
|
||||
TEST(BuilderTest, PacketGenerator) {
|
||||
builder::Graph graph;
|
||||
auto& generator = graph.AddPacketGenerator("FloatGenerator");
|
||||
|
||||
@@ -262,7 +262,7 @@ TEST(MathUtil, IntRound) {
|
||||
|
||||
// A double-precision number has a 53-bit mantissa (52 fraction bits),
|
||||
// so the following value can be represented exactly.
|
||||
int64 value64 = GG_ULONGLONG(0x1234567890abcd00);
|
||||
int64 value64 = static_cast<int64_t>(0x1234567890abcd00);
|
||||
EXPECT_EQ(mediapipe::MathUtil::Round<int64>(static_cast<double>(value64)),
|
||||
value64);
|
||||
}
|
||||
@@ -369,7 +369,7 @@ class SafeCastTester {
|
||||
if (sizeof(FloatIn) >= 64) {
|
||||
// A double-precision number has a 53-bit mantissa (52 fraction bits),
|
||||
// so the following value can be represented exactly by a double.
|
||||
int64 value64 = GG_ULONGLONG(0x1234567890abcd00);
|
||||
int64 value64 = static_cast<int64_t>(0x1234567890abcd00);
|
||||
const IntOut expected =
|
||||
(sizeof(IntOut) >= 64) ? static_cast<IntOut>(value64) : imax;
|
||||
EXPECT_EQ(
|
||||
@@ -565,7 +565,7 @@ TEST(MathUtil, SafeCast) {
|
||||
-12345);
|
||||
EXPECT_EQ(mediapipe::MathUtil::SafeCast<int>(1E47), 2147483647);
|
||||
EXPECT_EQ(mediapipe::MathUtil::SafeCast<int>(-1E47),
|
||||
GG_LONGLONG(-2147483648));
|
||||
static_cast<int64_t>(-2147483648));
|
||||
}
|
||||
|
||||
template <class FloatIn, class IntOut>
|
||||
@@ -682,7 +682,7 @@ class SafeRoundTester {
|
||||
if (sizeof(FloatIn) >= 64) {
|
||||
// A double-precision number has a 53-bit mantissa (52 fraction bits),
|
||||
// so the following value can be represented exactly by a double.
|
||||
int64 value64 = GG_ULONGLONG(0x1234567890abcd00);
|
||||
int64 value64 = static_cast<int64_t>(0x1234567890abcd00);
|
||||
const IntOut expected =
|
||||
(sizeof(IntOut) >= 64) ? static_cast<IntOut>(value64) : imax;
|
||||
EXPECT_EQ(
|
||||
@@ -873,7 +873,7 @@ TEST(MathUtil, SafeRound) {
|
||||
-12345);
|
||||
EXPECT_EQ(mediapipe::MathUtil::SafeRound<int>(1E47), 2147483647);
|
||||
EXPECT_EQ(mediapipe::MathUtil::SafeRound<int>(-1E47),
|
||||
GG_LONGLONG(-2147483648));
|
||||
static_cast<int64_t>(-2147483648));
|
||||
}
|
||||
|
||||
} // namespace
|
||||
|
||||
@@ -186,6 +186,7 @@ cc_library(
|
||||
"//conditions:default": [
|
||||
"//mediapipe/framework/port:opencv_imgproc",
|
||||
],
|
||||
"//mediapipe/framework/port:disable_opencv": [],
|
||||
}) + select({
|
||||
"//conditions:default": [
|
||||
],
|
||||
|
||||
@@ -76,10 +76,7 @@ bool Image::ConvertToGpu() const {
|
||||
gpu_buffer_ = mediapipe::GpuBuffer(std::move(buffer));
|
||||
#else
|
||||
// GlCalculatorHelperImpl::MakeGlTextureBuffer (CreateSourceTexture)
|
||||
auto buffer = mediapipe::GlTextureBuffer::Create(
|
||||
image_frame_->Width(), image_frame_->Height(),
|
||||
mediapipe::GpuBufferFormatForImageFormat(image_frame_->Format()),
|
||||
image_frame_->PixelData());
|
||||
auto buffer = mediapipe::GlTextureBuffer::Create(*image_frame_);
|
||||
glBindTexture(GL_TEXTURE_2D, buffer->name());
|
||||
// See GlCalculatorHelperImpl::SetStandardTextureParams
|
||||
glTexParameteri(buffer->target(), GL_TEXTURE_MIN_FILTER, GL_LINEAR);
|
||||
|
||||
@@ -32,7 +32,12 @@
|
||||
|
||||
// clang-format off
|
||||
#if !defined(LOCATION_OPENCV)
|
||||
# define LOCATION_OPENCV 1
|
||||
# if !MEDIAPIPE_DISABLE_OPENCV && \
|
||||
(!defined(MEDIAPIPE_MOBILE) || defined(MEDIAPIPE_ANDROID_OPENCV))
|
||||
# define LOCATION_OPENCV 1
|
||||
# else
|
||||
# define LOCATION_OPENCV 0
|
||||
# endif
|
||||
#endif
|
||||
|
||||
#if LOCATION_OPENCV
|
||||
|
||||
@@ -8,6 +8,7 @@ def mediapipe_cc_test(
|
||||
data = [],
|
||||
deps = [],
|
||||
size = None,
|
||||
tags = [],
|
||||
timeout = None,
|
||||
additional_deps = DEFAULT_ADDITIONAL_TEST_DEPS,
|
||||
**kwargs):
|
||||
|
||||
@@ -158,12 +158,12 @@ cc_library(
|
||||
hdrs = [
|
||||
"gmock.h",
|
||||
"gtest.h",
|
||||
"gtest-spi.h",
|
||||
"status_matchers.h",
|
||||
],
|
||||
visibility = ["//visibility:public"],
|
||||
deps = [
|
||||
":status_matchers",
|
||||
"//mediapipe/framework:port",
|
||||
"@com_google_googletest//:gtest",
|
||||
],
|
||||
)
|
||||
@@ -174,12 +174,12 @@ cc_library(
|
||||
hdrs = [
|
||||
"gmock.h",
|
||||
"gtest.h",
|
||||
"gtest-spi.h",
|
||||
"status_matchers.h",
|
||||
],
|
||||
visibility = ["//visibility:public"],
|
||||
deps = [
|
||||
":status_matchers",
|
||||
"//mediapipe/framework:port",
|
||||
"//mediapipe/framework/deps:status_matchers",
|
||||
"@com_google_googletest//:gtest_main",
|
||||
],
|
||||
@@ -217,6 +217,16 @@ cc_library(
|
||||
deps = ["//mediapipe/framework/deps:numbers"],
|
||||
)
|
||||
|
||||
# Disabling opencv when defining MEDIAPIPE_DISABLE_OPENCV to 1 in the bazel command.
|
||||
# Note that this only applies to a select few calculators/framework components currently.
|
||||
config_setting(
|
||||
name = "disable_opencv",
|
||||
define_values = {
|
||||
"MEDIAPIPE_DISABLE_OPENCV": "1",
|
||||
},
|
||||
visibility = ["//visibility:public"],
|
||||
)
|
||||
|
||||
cc_library(
|
||||
name = "opencv_core",
|
||||
hdrs = ["opencv_core_inc.h"],
|
||||
|
||||
@@ -0,0 +1,20 @@
|
||||
// Copyright 2021 The MediaPipe Authors.
|
||||
//
|
||||
// Licensed under the Apache License, Version 2.0 (the "License");
|
||||
// you may not use this file except in compliance with the License.
|
||||
// You may obtain a copy of the License at
|
||||
//
|
||||
// http://www.apache.org/licenses/LICENSE-2.0
|
||||
//
|
||||
// Unless required by applicable law or agreed to in writing, software
|
||||
// distributed under the License is distributed on an "AS IS" BASIS,
|
||||
// WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
#ifndef MEDIAPIPE_PORT_GTEST_SPI_H_
|
||||
#define MEDIAPIPE_PORT_GTEST_SPI_H_
|
||||
|
||||
#include "gtest/gtest-spi.h"
|
||||
|
||||
#endif // MEDIAPIPE_PORT_GTEST_SPI_H_
|
||||
@@ -641,14 +641,20 @@ cc_library(
|
||||
"//mediapipe/framework:calculator_cc_proto",
|
||||
"//mediapipe/framework/deps:file_path",
|
||||
"//mediapipe/framework/deps:no_destructor",
|
||||
"//mediapipe/framework/formats:image_format_cc_proto",
|
||||
"//mediapipe/framework/formats:image_frame",
|
||||
"//mediapipe/framework/port:advanced_proto",
|
||||
"//mediapipe/framework/port:file_helpers",
|
||||
"//mediapipe/framework/port:gtest",
|
||||
"//mediapipe/framework/port:logging",
|
||||
"//mediapipe/framework/port:ret_check",
|
||||
"//mediapipe/framework/port:status",
|
||||
"@com_google_absl//absl/container:flat_hash_set",
|
||||
"@com_google_absl//absl/memory",
|
||||
"@com_google_absl//absl/status",
|
||||
"@com_google_absl//absl/status:statusor",
|
||||
"@com_google_absl//absl/strings",
|
||||
"@stblib//:stb_image",
|
||||
"@stblib//:stb_image_write",
|
||||
],
|
||||
)
|
||||
|
||||
|
||||
@@ -18,18 +18,27 @@
|
||||
#include <unistd.h>
|
||||
|
||||
#include <memory>
|
||||
#include <string>
|
||||
|
||||
#include "absl/container/flat_hash_set.h"
|
||||
#include "absl/memory/memory.h"
|
||||
#include "absl/status/status.h"
|
||||
#include "absl/strings/match.h"
|
||||
#include "absl/strings/str_cat.h"
|
||||
#include "absl/strings/str_join.h"
|
||||
#include "absl/strings/substitute.h"
|
||||
#include "mediapipe/framework/calculator.pb.h"
|
||||
#include "mediapipe/framework/deps/file_path.h"
|
||||
#include "mediapipe/framework/deps/no_destructor.h"
|
||||
#include "mediapipe/framework/formats/image_format.pb.h"
|
||||
#include "mediapipe/framework/port/advanced_proto_inc.h"
|
||||
#include "mediapipe/framework/port/file_helpers.h"
|
||||
#include "mediapipe/framework/port/logging.h"
|
||||
#include "mediapipe/framework/port/proto_ns.h"
|
||||
#include "mediapipe/framework/port/ret_check.h"
|
||||
#include "mediapipe/framework/port/status_macros.h"
|
||||
#include "stb_image.h"
|
||||
#include "stb_image_write.h"
|
||||
|
||||
namespace mediapipe {
|
||||
|
||||
@@ -43,15 +52,14 @@ bool EqualWithTolerance(const T value1, const T value2, const T max_diff) {
|
||||
}
|
||||
|
||||
template <typename T>
|
||||
bool CompareDiff(const ImageFrame& image1, const ImageFrame& image2,
|
||||
const T max_color_diff, const T max_alpha_diff,
|
||||
const float max_avg_diff, std::string* error_message) {
|
||||
absl::Status CompareDiff(const ImageFrame& image1, const ImageFrame& image2,
|
||||
const T max_color_diff, const T max_alpha_diff,
|
||||
const float max_avg_diff,
|
||||
std::unique_ptr<ImageFrame>& diff_image) {
|
||||
// Verify image byte depth matches expected byte depth.
|
||||
CHECK_EQ(sizeof(T), image1.ByteDepth());
|
||||
CHECK_EQ(sizeof(T), image2.ByteDepth());
|
||||
|
||||
const bool return_error = error_message != nullptr;
|
||||
|
||||
const int width = image1.Width();
|
||||
const int height = image1.Height();
|
||||
const int channels1 = image1.NumberOfChannels();
|
||||
@@ -68,57 +76,64 @@ bool CompareDiff(const ImageFrame& image1, const ImageFrame& image2,
|
||||
const int width_padding2 =
|
||||
image2.WidthStep() / image2.ByteDepth() - width * channels2;
|
||||
|
||||
diff_image = std::make_unique<ImageFrame>(image1.Format(), width, height);
|
||||
T* pixel_diff = reinterpret_cast<T*>(diff_image->MutablePixelData());
|
||||
const int width_padding_diff =
|
||||
diff_image->WidthStep() / diff_image->ByteDepth() - width * channels1;
|
||||
|
||||
float avg_diff = 0;
|
||||
uint diff_count = 0;
|
||||
uint total_count = 0;
|
||||
int different_color_components = 0;
|
||||
float max_color_diff_found = 0;
|
||||
int different_alpha_components = 0;
|
||||
float max_alpha_diff_found = 0;
|
||||
for (int row = 0; row < height; ++row) {
|
||||
for (int col = 0; col < width; ++col) {
|
||||
for (int channel = 0; channel < num_channels; ++channel) {
|
||||
// Check local difference.
|
||||
const T max_diff = channel < 3 ? max_color_diff : max_alpha_diff;
|
||||
const T value1 = pixel1[channel];
|
||||
const T value2 = pixel2[channel];
|
||||
if (!EqualWithTolerance<T>(value1, value2, max_diff)) {
|
||||
// We cast uint8 to int using this type (and leave other values as-is)
|
||||
// to avoid printing as a single char.
|
||||
using TypeToPrint =
|
||||
typename std::conditional<std::is_same<T, uint8>::value, int,
|
||||
T>::type;
|
||||
std::string error = absl::Substitute(
|
||||
"images differ: row = $0 col = $1 channel = $2 : pixel1 = $3, "
|
||||
"pixel2 = $4",
|
||||
row, col, channel, static_cast<TypeToPrint>(value1),
|
||||
static_cast<TypeToPrint>(value2));
|
||||
if (return_error) {
|
||||
*error_message = error;
|
||||
} else {
|
||||
LOG(ERROR) << error;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
// Check global average difference.
|
||||
const float diff =
|
||||
std::abs(static_cast<float>(value1) - static_cast<float>(value2));
|
||||
avg_diff += (diff - avg_diff) / ++diff_count;
|
||||
if (channel < 3) {
|
||||
different_color_components += diff > max_color_diff;
|
||||
max_color_diff_found = std::max(max_color_diff_found, diff);
|
||||
pixel_diff[channel] = diff;
|
||||
} else {
|
||||
different_alpha_components += diff > max_alpha_diff;
|
||||
max_alpha_diff_found = std::max(max_alpha_diff_found, diff);
|
||||
pixel_diff[channel] = 255; // opaque to see color difference
|
||||
}
|
||||
// Check global average difference.
|
||||
avg_diff += (diff - avg_diff) / ++total_count;
|
||||
}
|
||||
pixel1 += channels1;
|
||||
pixel2 += channels2;
|
||||
pixel_diff += channels1;
|
||||
}
|
||||
pixel1 += width_padding1;
|
||||
pixel2 += width_padding2;
|
||||
pixel_diff += width_padding_diff;
|
||||
}
|
||||
|
||||
if (avg_diff > max_avg_diff) {
|
||||
std::string error =
|
||||
absl::Substitute("images differ: avg pixel error = $0", avg_diff);
|
||||
if (return_error) {
|
||||
*error_message = error;
|
||||
} else {
|
||||
LOG(ERROR) << error;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
std::vector<std::string> errors;
|
||||
if (different_color_components)
|
||||
errors.push_back(absl::Substitute(
|
||||
"$0 color components differences above limit of $1, max found was $2",
|
||||
different_color_components, max_color_diff, max_color_diff_found));
|
||||
if (different_alpha_components)
|
||||
errors.push_back(absl::Substitute(
|
||||
"$0 alpha components differences above limit of $1, max found was $2",
|
||||
different_alpha_components, max_alpha_diff, max_alpha_diff_found));
|
||||
if (avg_diff > max_avg_diff)
|
||||
errors.push_back(
|
||||
absl::Substitute("the average component difference is $0 (limit: $1)",
|
||||
avg_diff, max_avg_diff));
|
||||
|
||||
return true;
|
||||
if (!errors.empty())
|
||||
return absl::InternalError(
|
||||
absl::StrCat("images differ: ", absl::StrJoin(errors, "; ")));
|
||||
return absl::OkStatus();
|
||||
}
|
||||
|
||||
#if defined(__linux__)
|
||||
@@ -134,77 +149,32 @@ std::string GetBinaryDirectory() {
|
||||
|
||||
} // namespace
|
||||
|
||||
bool CompareImageFrames(const ImageFrame& image1, const ImageFrame& image2,
|
||||
const float max_color_diff, const float max_alpha_diff,
|
||||
const float max_avg_diff, std::string* error_message) {
|
||||
const bool return_error = error_message != nullptr;
|
||||
|
||||
auto IsSupportedImageFormatComparison = [](const ImageFrame& image1,
|
||||
const ImageFrame& image2) {
|
||||
// Pairs of non-equal image formats that can be compared against each other.
|
||||
static const mediapipe::NoDestructor<absl::flat_hash_set<
|
||||
std::pair<ImageFormat::Format, ImageFormat::Format>>>
|
||||
kCompatibleImageFormats({
|
||||
{ImageFormat::SRGB, ImageFormat::SRGBA},
|
||||
{ImageFormat::SRGB48, ImageFormat::SRGBA64},
|
||||
});
|
||||
|
||||
auto* compatible_image_formats = kCompatibleImageFormats.get();
|
||||
|
||||
return image1.Format() == image2.Format() ||
|
||||
compatible_image_formats->contains(
|
||||
{image1.Format(), image2.Format()}) ||
|
||||
compatible_image_formats->contains(
|
||||
{image2.Format(), image1.Format()});
|
||||
absl::Status CompareImageFrames(const ImageFrame& image1,
|
||||
const ImageFrame& image2,
|
||||
const float max_color_diff,
|
||||
const float max_alpha_diff,
|
||||
const float max_avg_diff,
|
||||
std::unique_ptr<ImageFrame>& diff_image) {
|
||||
auto IsSupportedImageFormatComparison = [](ImageFormat::Format one,
|
||||
ImageFormat::Format two) {
|
||||
auto both = std::minmax(one, two);
|
||||
return one == two ||
|
||||
both == std::minmax(ImageFormat::SRGB, ImageFormat::SRGBA) ||
|
||||
both == std::minmax(ImageFormat::SRGB48, ImageFormat::SRGBA64);
|
||||
};
|
||||
|
||||
if (!IsSupportedImageFormatComparison(image1, image2)) {
|
||||
std::string error = absl::Substitute(
|
||||
"unsupported image format comparison; image1 = $0, image2 = $1",
|
||||
image1.Format(), image2.Format());
|
||||
if (return_error) {
|
||||
*error_message = error;
|
||||
} else {
|
||||
LOG(ERROR) << error;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
RET_CHECK(IsSupportedImageFormatComparison(image1.Format(), image2.Format()))
|
||||
<< "unsupported image format comparison; image1 = " << image1.Format()
|
||||
<< ", image2 = " << image2.Format();
|
||||
|
||||
if (image1.Width() != image2.Width()) {
|
||||
std::string error =
|
||||
absl::Substitute("image width mismatch: image1 = $0, image2 = $1",
|
||||
image1.Width(), image2.Width());
|
||||
if (return_error) {
|
||||
*error_message = error;
|
||||
} else {
|
||||
LOG(ERROR) << error;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
// Cannot use RET_CHECK_EQ because pair is not printable.
|
||||
RET_CHECK(std::make_pair(image1.Width(), image1.Height()) ==
|
||||
std::make_pair(image2.Width(), image2.Height()))
|
||||
<< "image size mismatch: " << image1.Width() << "x" << image1.Height()
|
||||
<< " != " << image2.Width() << "x" << image2.Height();
|
||||
|
||||
if (image1.Height() != image2.Height()) {
|
||||
std::string error =
|
||||
absl::Substitute("image height mismatch: image1 = $0, image2 = $1",
|
||||
image1.Height(), image2.Height());
|
||||
if (return_error) {
|
||||
*error_message = error;
|
||||
} else {
|
||||
LOG(ERROR) << error;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
if (image1.ByteDepth() != image2.ByteDepth()) {
|
||||
std::string error =
|
||||
absl::Substitute("image byte depth mismatch: image1 = $0, image2 = $1",
|
||||
image1.ByteDepth(), image2.ByteDepth());
|
||||
if (return_error) {
|
||||
*error_message = error;
|
||||
} else {
|
||||
LOG(ERROR) << error;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
RET_CHECK_EQ(image1.ByteDepth(), image2.ByteDepth())
|
||||
<< "image byte depth mismatch";
|
||||
|
||||
switch (image1.Format()) {
|
||||
case ImageFormat::GRAY8:
|
||||
@@ -212,45 +182,87 @@ bool CompareImageFrames(const ImageFrame& image1, const ImageFrame& image2,
|
||||
case ImageFormat::SRGBA:
|
||||
case ImageFormat::LAB8:
|
||||
return CompareDiff<uint8>(image1, image2, max_color_diff, max_alpha_diff,
|
||||
max_avg_diff, error_message);
|
||||
max_avg_diff, diff_image);
|
||||
case ImageFormat::GRAY16:
|
||||
case ImageFormat::SRGB48:
|
||||
case ImageFormat::SRGBA64:
|
||||
return CompareDiff<uint16>(image1, image2, max_color_diff, max_alpha_diff,
|
||||
max_avg_diff, error_message);
|
||||
max_avg_diff, diff_image);
|
||||
case ImageFormat::VEC32F1:
|
||||
case ImageFormat::VEC32F2:
|
||||
return CompareDiff<float>(image1, image2, max_color_diff, max_alpha_diff,
|
||||
max_avg_diff, error_message);
|
||||
max_avg_diff, diff_image);
|
||||
default:
|
||||
LOG(FATAL) << ImageFrame::InvalidFormatString(image1.Format());
|
||||
}
|
||||
}
|
||||
|
||||
bool CompareImageFrames(const ImageFrame& image1, const ImageFrame& image2,
|
||||
const float max_color_diff, const float max_alpha_diff,
|
||||
const float max_avg_diff, std::string* error_message) {
|
||||
std::unique_ptr<ImageFrame> diff_image;
|
||||
auto status = CompareImageFrames(image1, image2, max_color_diff,
|
||||
max_alpha_diff, max_avg_diff, diff_image);
|
||||
if (status.ok()) return true;
|
||||
if (error_message) *error_message = std::string(status.message());
|
||||
return false;
|
||||
}
|
||||
|
||||
std::string GetTestRootDir() {
|
||||
#if defined(__ANDROID__)
|
||||
char path[1024];
|
||||
char* ptr = getcwd(path, sizeof(path));
|
||||
CHECK_EQ(ptr, path);
|
||||
return path;
|
||||
#else
|
||||
return ::mediapipe::file::JoinPath(std::getenv("TEST_SRCDIR"), "mediapipe");
|
||||
#endif // defined(__ANDROID__)
|
||||
return file::JoinPath(std::getenv("TEST_SRCDIR"), "mediapipe");
|
||||
}
|
||||
|
||||
std::string GetTestOutputsDir() {
|
||||
const char* output_dir = getenv("TEST_UNDECLARED_OUTPUTS_DIR");
|
||||
if (!output_dir) {
|
||||
output_dir = "/tmp";
|
||||
}
|
||||
return output_dir;
|
||||
}
|
||||
|
||||
std::string GetTestDataDir(const std::string& package_base_path) {
|
||||
#if defined(__ANDROID__)
|
||||
std::string data_dir = GetTestRootDir();
|
||||
std::string binary_dir = GetBinaryDirectory();
|
||||
// In Mobile Harness, the cwd is "/" and the run dir is "/data/local/tmp".
|
||||
if (data_dir == "/" && absl::StartsWith(binary_dir, "/data")) {
|
||||
data_dir = binary_dir;
|
||||
return file::JoinPath(GetTestRootDir(), package_base_path, "testdata/");
|
||||
}
|
||||
|
||||
std::string GetTestFilePath(absl::string_view relative_path) {
|
||||
return file::JoinPath(GetTestRootDir(), relative_path);
|
||||
}
|
||||
|
||||
absl::StatusOr<std::unique_ptr<ImageFrame>> LoadTestImage(
|
||||
absl::string_view path, ImageFormat::Format format) {
|
||||
std::string encoded;
|
||||
MP_RETURN_IF_ERROR(mediapipe::file::GetContents(path, &encoded));
|
||||
|
||||
// stbi_load determines the output pixel format based on the desired channels.
|
||||
// 0 means "use whatever's in the file".
|
||||
int desired_channels = format == ImageFormat::UNKNOWN ? 0
|
||||
: format == ImageFormat::SRGBA ? 4
|
||||
: format == ImageFormat::SRGB ? 3
|
||||
: format == ImageFormat::GRAY8 ? 1
|
||||
: -1;
|
||||
RET_CHECK(desired_channels >= 0)
|
||||
<< "unsupported output format requested: " << format;
|
||||
|
||||
int width, height, channels_in_file;
|
||||
auto data = stbi_load_from_memory(reinterpret_cast<stbi_uc*>(encoded.data()),
|
||||
encoded.size(), &width, &height,
|
||||
&channels_in_file, desired_channels);
|
||||
RET_CHECK(data) << "failed to decode image data from: " << path;
|
||||
|
||||
// If we didn't specify a desired format, it will be determined by what the
|
||||
// file contains.
|
||||
int output_channels = desired_channels ? desired_channels : channels_in_file;
|
||||
if (format == ImageFormat::UNKNOWN) {
|
||||
format = output_channels == 4 ? ImageFormat::SRGBA
|
||||
: output_channels == 3 ? ImageFormat::SRGB
|
||||
: output_channels == 1 ? ImageFormat::GRAY8
|
||||
: ImageFormat::UNKNOWN;
|
||||
RET_CHECK(format != ImageFormat::UNKNOWN)
|
||||
<< "unsupported number of channels: " << output_channels;
|
||||
}
|
||||
return ::mediapipe::file::JoinPath(data_dir, package_base_path, "testdata/");
|
||||
#else
|
||||
return ::mediapipe::file::JoinPath(GetTestRootDir(), package_base_path,
|
||||
"testdata/");
|
||||
#endif // defined(__APPLE__)
|
||||
|
||||
return absl::make_unique<ImageFrame>(
|
||||
format, width, height, width * output_channels, data, stbi_image_free);
|
||||
}
|
||||
|
||||
std::unique_ptr<ImageFrame> LoadTestPng(const std::string& path,
|
||||
|
||||
@@ -15,6 +15,7 @@
|
||||
#ifndef MEDIAPIPE_FRAMEWORK_TEST_UTIL_H_
|
||||
#define MEDIAPIPE_FRAMEWORK_TEST_UTIL_H_
|
||||
|
||||
#include "absl/status/statusor.h"
|
||||
#include "mediapipe/framework/calculator.pb.h"
|
||||
#include "mediapipe/framework/formats/image_frame.h"
|
||||
|
||||
@@ -35,14 +36,29 @@ using mediapipe::CalculatorGraphConfig;
|
||||
// Note: Although max_color_diff and max_alpha_diff are floats, all uint8/uint16
|
||||
// values are exactly representable. (2^24 + 1 is the first non-representable
|
||||
// positive integral value.)
|
||||
absl::Status CompareImageFrames(const ImageFrame& image1,
|
||||
const ImageFrame& image2,
|
||||
const float max_color_diff,
|
||||
const float max_alpha_diff,
|
||||
const float max_avg_diff,
|
||||
std::unique_ptr<ImageFrame>& diff_image);
|
||||
|
||||
bool CompareImageFrames(const ImageFrame& image1, const ImageFrame& image2,
|
||||
const float max_color_diff, const float max_alpha_diff,
|
||||
const float max_avg_diff = 1.0,
|
||||
std::string* error_message = nullptr);
|
||||
|
||||
// Returns the absolute path to the directory that contains test source code.
|
||||
// Returns the absolute path to the directory that contains test source code
|
||||
// (TEST_SRCDIR).
|
||||
std::string GetTestRootDir();
|
||||
|
||||
// Returns the absolute path to a directory where tests can write outputs to
|
||||
// be sent to bazel (TEST_UNDECLARED_OUTPUTS_DIR or a fallback).
|
||||
std::string GetTestOutputsDir();
|
||||
|
||||
// Returns the absolute path to a file within TEST_SRCDIR.
|
||||
std::string GetTestFilePath(absl::string_view relative_path);
|
||||
|
||||
// Returns the absolute path to the contents of the package's "testdata"
|
||||
// directory.
|
||||
// This handles the different paths where test data ends up when using
|
||||
@@ -52,6 +68,10 @@ std::string GetTestDataDir(const std::string& package_base_path);
|
||||
// Loads a binary graph from path. Returns true iff successful.
|
||||
bool LoadTestGraph(CalculatorGraphConfig* proto, const std::string& path);
|
||||
|
||||
// Loads an image from path.
|
||||
absl::StatusOr<std::unique_ptr<ImageFrame>> LoadTestImage(
|
||||
absl::string_view path, ImageFormat::Format format = ImageFormat::SRGBA);
|
||||
|
||||
// Loads a PNG image from path using the given ImageFormat. Returns nullptr in
|
||||
// case of failure.
|
||||
std::unique_ptr<ImageFrame> LoadTestPng(
|
||||
|
||||
+5
-3
@@ -585,6 +585,7 @@ cc_library(
|
||||
"//mediapipe:apple": [
|
||||
":gl_calculator_helper_ios",
|
||||
"//mediapipe/objc:util",
|
||||
"//mediapipe/objc:CFHolder",
|
||||
],
|
||||
}),
|
||||
)
|
||||
@@ -714,11 +715,12 @@ cc_library(
|
||||
deps = [
|
||||
":gl_calculator_helper",
|
||||
"//mediapipe/framework:calculator_framework",
|
||||
"//mediapipe/framework:timestamp",
|
||||
"//mediapipe/framework/formats:image_frame",
|
||||
"//mediapipe/framework/port:ret_check",
|
||||
"//mediapipe/framework/port:status",
|
||||
],
|
||||
] + select({
|
||||
"//conditions:default": [],
|
||||
"//mediapipe:apple": ["//mediapipe/objc:util"],
|
||||
}),
|
||||
alwayslink = 1,
|
||||
)
|
||||
|
||||
|
||||
@@ -12,6 +12,9 @@
|
||||
// See the License for the specific language governing permissions and
|
||||
// limitations under the License.
|
||||
|
||||
#include <memory>
|
||||
|
||||
#include "mediapipe/framework/formats/image_frame.h"
|
||||
#include "mediapipe/gpu/gl_calculator_helper_impl.h"
|
||||
#include "mediapipe/gpu/gpu_buffer_format.h"
|
||||
#include "mediapipe/gpu/gpu_shared_data_internal.h"
|
||||
@@ -176,10 +179,8 @@ GlTexture GlCalculatorHelperImpl::MapGlTextureBuffer(
|
||||
GlTextureBufferSharedPtr GlCalculatorHelperImpl::MakeGlTextureBuffer(
|
||||
const ImageFrame& image_frame) {
|
||||
CHECK(gl_context_->IsCurrent());
|
||||
auto buffer = GlTextureBuffer::Create(
|
||||
image_frame.Width(), image_frame.Height(),
|
||||
GpuBufferFormatForImageFormat(image_frame.Format()),
|
||||
image_frame.PixelData());
|
||||
|
||||
auto buffer = GlTextureBuffer::Create(image_frame);
|
||||
|
||||
if (buffer->format_ != GpuBufferFormat::kUnknown) {
|
||||
glBindTexture(GL_TEXTURE_2D, buffer->name_);
|
||||
|
||||
@@ -75,26 +75,30 @@ absl::Status GlContext::CreateContextInternal(
|
||||
// TODO: Ensure this works with all options (in particular,
|
||||
// multithreading options, like the special-case combination of USE_PTHREADS
|
||||
// and OFFSCREEN_FRAMEBUFFER)
|
||||
EM_ASM(let init_once = true; if (init_once) {
|
||||
const cachedFindCanvasEventTarget = findCanvasEventTarget;
|
||||
// clang-format off
|
||||
EM_ASM(
|
||||
let init_once = true;
|
||||
if (init_once) {
|
||||
const cachedFindCanvasEventTarget = findCanvasEventTarget;
|
||||
|
||||
if (typeof cachedFindCanvasEventTarget != = 'function') {
|
||||
if (typeof console != = 'undefined') {
|
||||
console.error(
|
||||
'Expected Emscripten global function ' +
|
||||
'"findCanvasEventTarget" not found. WebGL context creation ' +
|
||||
'may fail.');
|
||||
if (typeof cachedFindCanvasEventTarget !== 'function') {
|
||||
if (typeof console !== 'undefined') {
|
||||
console.error('Expected Emscripten global function '
|
||||
+ '"findCanvasEventTarget" not found. WebGL context creation '
|
||||
+ 'may fail.');
|
||||
}
|
||||
return;
|
||||
}
|
||||
return;
|
||||
}
|
||||
|
||||
findCanvasEventTarget = function(target) {
|
||||
if (Module && Module.canvas) {
|
||||
return Module.canvas;
|
||||
} else if (Module && Module.canvasCssSelector) {
|
||||
return cachedFindCanvasEventTarget(Module.canvasCssSelector);
|
||||
} else {
|
||||
if (typeof console != = 'undefined') {
|
||||
findCanvasEventTarget = function(target) {
|
||||
if (target == 0) {
|
||||
if (Module && Module.canvas) {
|
||||
return Module.canvas;
|
||||
} else if (Module && Module.canvasCssSelector) {
|
||||
return cachedFindCanvasEventTarget(Module.canvasCssSelector);
|
||||
}
|
||||
}
|
||||
if (typeof console !== 'undefined') {
|
||||
console.warn('Module properties canvas and canvasCssSelector not ' +
|
||||
'found during WebGL context creation.');
|
||||
}
|
||||
@@ -102,15 +106,14 @@ absl::Status GlContext::CreateContextInternal(
|
||||
// cases it will not succeed, just in case the user does want to fall-
|
||||
// back.
|
||||
return cachedFindCanvasEventTarget(target);
|
||||
}
|
||||
}; // NOLINT: Necessary semicolon.
|
||||
init_once = false;
|
||||
});
|
||||
}; // NOLINT: Necessary semicolon.
|
||||
init_once = false;
|
||||
}
|
||||
);
|
||||
// clang-format on
|
||||
|
||||
// Note: below id parameter is only actually used if both Module.canvas and
|
||||
// Module.canvasCssSelector are undefined.
|
||||
EMSCRIPTEN_WEBGL_CONTEXT_HANDLE context_handle =
|
||||
emscripten_webgl_create_context(0 /* id */, &attrs);
|
||||
emscripten_webgl_create_context(nullptr, &attrs);
|
||||
|
||||
// Check for failure
|
||||
if (context_handle <= 0) {
|
||||
|
||||
@@ -32,15 +32,56 @@ std::unique_ptr<GlTextureBuffer> GlTextureBuffer::Wrap(
|
||||
|
||||
std::unique_ptr<GlTextureBuffer> GlTextureBuffer::Create(int width, int height,
|
||||
GpuBufferFormat format,
|
||||
const void* data) {
|
||||
const void* data,
|
||||
int alignment) {
|
||||
auto buf = absl::make_unique<GlTextureBuffer>(GL_TEXTURE_2D, 0, width, height,
|
||||
format, nullptr);
|
||||
if (!buf->CreateInternal(data)) {
|
||||
if (!buf->CreateInternal(data, alignment)) {
|
||||
return nullptr;
|
||||
}
|
||||
return buf;
|
||||
}
|
||||
|
||||
static inline int AlignedToPowerOf2(int value, int alignment) {
|
||||
// alignment must be a power of 2
|
||||
return ((value - 1) | (alignment - 1)) + 1;
|
||||
}
|
||||
|
||||
std::unique_ptr<GlTextureBuffer> GlTextureBuffer::Create(
|
||||
const ImageFrame& image_frame) {
|
||||
int base_ws = image_frame.Width() * image_frame.NumberOfChannels() *
|
||||
image_frame.ByteDepth();
|
||||
int actual_ws = image_frame.WidthStep();
|
||||
int alignment = 0;
|
||||
std::unique_ptr<ImageFrame> temp;
|
||||
const uint8* data = image_frame.PixelData();
|
||||
|
||||
// Let's see if the pixel data is tightly aligned to one of the alignments
|
||||
// supported by OpenGL, preferring 4 if possible since it's the default.
|
||||
if (actual_ws == AlignedToPowerOf2(base_ws, 4))
|
||||
alignment = 4;
|
||||
else if (actual_ws == AlignedToPowerOf2(base_ws, 1))
|
||||
alignment = 1;
|
||||
else if (actual_ws == AlignedToPowerOf2(base_ws, 2))
|
||||
alignment = 2;
|
||||
else if (actual_ws == AlignedToPowerOf2(base_ws, 8))
|
||||
alignment = 8;
|
||||
|
||||
// If no GL-compatible alignment was found, we copy the data to a temporary
|
||||
// buffer, aligned to 4. We do this using another ImageFrame purely for
|
||||
// convenience.
|
||||
if (!alignment) {
|
||||
temp = std::make_unique<ImageFrame>();
|
||||
temp->CopyFrom(image_frame, 4);
|
||||
data = temp->PixelData();
|
||||
alignment = 4;
|
||||
}
|
||||
|
||||
return Create(image_frame.Width(), image_frame.Height(),
|
||||
GpuBufferFormatForImageFormat(image_frame.Format()), data,
|
||||
alignment);
|
||||
}
|
||||
|
||||
GlTextureBuffer::GlTextureBuffer(GLenum target, GLuint name, int width,
|
||||
int height, GpuBufferFormat format,
|
||||
DeletionCallback deletion_callback,
|
||||
@@ -53,7 +94,7 @@ GlTextureBuffer::GlTextureBuffer(GLenum target, GLuint name, int width,
|
||||
deletion_callback_(deletion_callback),
|
||||
producer_context_(producer_context) {}
|
||||
|
||||
bool GlTextureBuffer::CreateInternal(const void* data) {
|
||||
bool GlTextureBuffer::CreateInternal(const void* data, int alignment) {
|
||||
auto context = GlContext::GetCurrent();
|
||||
if (!context) return false;
|
||||
|
||||
@@ -66,8 +107,11 @@ bool GlTextureBuffer::CreateInternal(const void* data) {
|
||||
GlTextureInfo info =
|
||||
GlTextureInfoForGpuBufferFormat(format_, 0, context->GetGlVersion());
|
||||
|
||||
if (alignment != 4 && data) glPixelStorei(GL_UNPACK_ALIGNMENT, alignment);
|
||||
|
||||
// See b/70294573 for details about this.
|
||||
if (info.gl_internal_format == GL_RGBA16F &&
|
||||
context->GetGlVersion() != GlVersion::kGLES2 &&
|
||||
SymbolAvailable(&glTexStorage2D)) {
|
||||
CHECK(data == nullptr) << "unimplemented";
|
||||
glTexStorage2D(target_, 1, info.gl_internal_format, width_, height_);
|
||||
@@ -76,6 +120,8 @@ bool GlTextureBuffer::CreateInternal(const void* data) {
|
||||
height_, 0 /* border */, info.gl_format, info.gl_type, data);
|
||||
}
|
||||
|
||||
if (alignment != 4 && data) glPixelStorei(GL_UNPACK_ALIGNMENT, 4);
|
||||
|
||||
glBindTexture(target_, 0);
|
||||
|
||||
// Use the deletion callback to delete the texture on the context
|
||||
|
||||
@@ -21,6 +21,7 @@
|
||||
#include <atomic>
|
||||
|
||||
#include "absl/memory/memory.h"
|
||||
#include "mediapipe/framework/formats/image_frame.h"
|
||||
#include "mediapipe/gpu/gl_base.h"
|
||||
#include "mediapipe/gpu/gl_context.h"
|
||||
#include "mediapipe/gpu/gpu_buffer_format.h"
|
||||
@@ -60,7 +61,11 @@ class GlTextureBuffer {
|
||||
// provided later via glTexSubImage2D.
|
||||
static std::unique_ptr<GlTextureBuffer> Create(int width, int height,
|
||||
GpuBufferFormat format,
|
||||
const void* data = nullptr);
|
||||
const void* data = nullptr,
|
||||
int alignment = 4);
|
||||
|
||||
// Create a texture with a copy of the data in image_frame.
|
||||
static std::unique_ptr<GlTextureBuffer> Create(const ImageFrame& image_frame);
|
||||
|
||||
// Wraps an existing texture, but does not take ownership of it.
|
||||
// deletion_callback is invoked when the GlTextureBuffer is released, so
|
||||
@@ -127,7 +132,7 @@ class GlTextureBuffer {
|
||||
// If data is provided, it is uploaded to the texture; otherwise, it can be
|
||||
// provided later via glTexSubImage2D.
|
||||
// Returns true on success.
|
||||
bool CreateInternal(const void* data = nullptr);
|
||||
bool CreateInternal(const void* data, int alignment = 4);
|
||||
|
||||
friend class GlCalculatorHelperImpl;
|
||||
|
||||
|
||||
@@ -164,7 +164,9 @@ const GlTextureInfo& GlTextureInfoForGpuBufferFormat(GpuBufferFormat format,
|
||||
}
|
||||
|
||||
auto iter = format_info->find(format);
|
||||
CHECK(iter != format_info->end()) << "unsupported format";
|
||||
CHECK(iter != format_info->end())
|
||||
<< "unsupported format: "
|
||||
<< static_cast<std::underlying_type_t<decltype(format)>>(format);
|
||||
const auto& planes = iter->second;
|
||||
#ifndef __APPLE__
|
||||
CHECK_EQ(planes.size(), 1)
|
||||
|
||||
@@ -51,8 +51,6 @@ namespace mediapipe {
|
||||
constexpr int kMaxShaderInfoLength = 1024;
|
||||
|
||||
GLint GlhCompileShader(GLenum target, const GLchar* source, GLuint* shader) {
|
||||
GLint status;
|
||||
|
||||
*shader = glCreateShader(target);
|
||||
if (*shader == 0) {
|
||||
return GL_FALSE;
|
||||
@@ -62,6 +60,11 @@ GLint GlhCompileShader(GLenum target, const GLchar* source, GLuint* shader) {
|
||||
|
||||
GL_DEBUG_LOG(Shader, *shader, "compile");
|
||||
|
||||
#if UNSAFE_EMSCRIPTEN_SKIP_GL_ERROR_HANDLING
|
||||
return GL_TRUE;
|
||||
#else
|
||||
GLint status;
|
||||
|
||||
glGetShaderiv(*shader, GL_COMPILE_STATUS, &status);
|
||||
LOG_IF(ERROR, status == GL_FALSE) << "Failed to compile shader:\n" << source;
|
||||
|
||||
@@ -72,19 +75,24 @@ GLint GlhCompileShader(GLenum target, const GLchar* source, GLuint* shader) {
|
||||
LOG(ERROR) << "Error message: " << std::string(cmessage, length);
|
||||
}
|
||||
return status;
|
||||
#endif // UNSAFE_EMSCRIPTEN_SKIP_GL_ERROR_HANDLING
|
||||
}
|
||||
|
||||
GLint GlhLinkProgram(GLuint program) {
|
||||
GLint status;
|
||||
|
||||
glLinkProgram(program);
|
||||
|
||||
#if UNSAFE_EMSCRIPTEN_SKIP_GL_ERROR_HANDLING
|
||||
return GL_TRUE;
|
||||
#else
|
||||
GLint status;
|
||||
|
||||
GL_DEBUG_LOG(Program, program, "link");
|
||||
|
||||
glGetProgramiv(program, GL_LINK_STATUS, &status);
|
||||
LOG_IF(ERROR, status == GL_FALSE) << "Failed to link program " << program;
|
||||
|
||||
return status;
|
||||
#endif // UNSAFE_EMSCRIPTEN_SKIP_GL_ERROR_HANDLING
|
||||
}
|
||||
|
||||
GLint GlhValidateProgram(GLuint program) {
|
||||
|
||||
@@ -11,6 +11,10 @@
|
||||
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
||||
# See the License for the specific language governing permissions and
|
||||
# limitations under the License.
|
||||
load(
|
||||
"//mediapipe/framework/tool:mediapipe_graph.bzl",
|
||||
"mediapipe_binary_graph",
|
||||
)
|
||||
|
||||
licenses(["notice"])
|
||||
|
||||
@@ -24,8 +28,8 @@ cc_library(
|
||||
"//mediapipe/calculators/util:detections_to_render_data_calculator",
|
||||
"//mediapipe/gpu:gpu_buffer_to_image_frame_calculator",
|
||||
"//mediapipe/gpu:image_frame_to_gpu_buffer_calculator",
|
||||
"//mediapipe/modules/face_detection:face_detection_front_cpu",
|
||||
"//mediapipe/modules/face_detection:face_detection_front_gpu",
|
||||
"//mediapipe/modules/face_detection:face_detection_short_range_cpu",
|
||||
"//mediapipe/modules/face_detection:face_detection_short_range_gpu",
|
||||
],
|
||||
)
|
||||
|
||||
@@ -35,7 +39,7 @@ cc_library(
|
||||
"//mediapipe/calculators/core:flow_limiter_calculator",
|
||||
"//mediapipe/calculators/util:annotation_overlay_calculator",
|
||||
"//mediapipe/calculators/util:detections_to_render_data_calculator",
|
||||
"//mediapipe/modules/face_detection:face_detection_front_cpu",
|
||||
"//mediapipe/modules/face_detection:face_detection_short_range_cpu",
|
||||
],
|
||||
)
|
||||
|
||||
@@ -45,15 +49,10 @@ cc_library(
|
||||
"//mediapipe/calculators/core:flow_limiter_calculator",
|
||||
"//mediapipe/calculators/util:annotation_overlay_calculator",
|
||||
"//mediapipe/calculators/util:detections_to_render_data_calculator",
|
||||
"//mediapipe/modules/face_detection:face_detection_front_gpu",
|
||||
"//mediapipe/modules/face_detection:face_detection_short_range_gpu",
|
||||
],
|
||||
)
|
||||
|
||||
load(
|
||||
"//mediapipe/framework/tool:mediapipe_graph.bzl",
|
||||
"mediapipe_binary_graph",
|
||||
)
|
||||
|
||||
mediapipe_binary_graph(
|
||||
name = "face_detection_mobile_cpu_binary_graph",
|
||||
graph = "face_detection_mobile_cpu.pbtxt",
|
||||
@@ -67,3 +66,30 @@ mediapipe_binary_graph(
|
||||
output_name = "face_detection_mobile_gpu.binarypb",
|
||||
deps = [":mobile_calculators"],
|
||||
)
|
||||
|
||||
cc_library(
|
||||
name = "face_detection_full_range_mobile_gpu_deps",
|
||||
deps = [
|
||||
"//mediapipe/calculators/core:flow_limiter_calculator",
|
||||
"//mediapipe/calculators/util:annotation_overlay_calculator",
|
||||
"//mediapipe/calculators/util:detections_to_render_data_calculator",
|
||||
"//mediapipe/modules/face_detection:face_detection_full_range_gpu",
|
||||
],
|
||||
)
|
||||
|
||||
mediapipe_binary_graph(
|
||||
name = "face_detection_full_range_mobile_gpu_binary_graph",
|
||||
graph = "face_detection_full_range_mobile_gpu.pbtxt",
|
||||
output_name = "face_detection_full_range_mobile_gpu.binarypb",
|
||||
deps = [":face_detection_full_range_mobile_gpu_deps"],
|
||||
)
|
||||
|
||||
cc_library(
|
||||
name = "face_detection_full_range_desktop_live_deps",
|
||||
deps = [
|
||||
"//mediapipe/calculators/core:flow_limiter_calculator",
|
||||
"//mediapipe/calculators/util:annotation_overlay_calculator",
|
||||
"//mediapipe/calculators/util:detections_to_render_data_calculator",
|
||||
"//mediapipe/modules/face_detection:face_detection_full_range_cpu",
|
||||
],
|
||||
)
|
||||
|
||||
@@ -1,169 +0,0 @@
|
||||
# MediaPipe graph that performs face detection with TensorFlow Lite on CPU.
|
||||
# Used in the examples in
|
||||
# mediapipe/examples/desktop/face_detection:face_detection_cpu.
|
||||
|
||||
# Images on GPU coming into and out of the graph.
|
||||
input_stream: "input_video"
|
||||
output_stream: "output_video"
|
||||
|
||||
# Throttles the images flowing downstream for flow control. It passes through
|
||||
# the very first incoming image unaltered, and waits for
|
||||
# TfLiteTensorsToDetectionsCalculator downstream in the graph to finish
|
||||
# generating the corresponding detections before it passes through another
|
||||
# image. All images that come in while waiting are dropped, limiting the number
|
||||
# of in-flight images between this calculator and
|
||||
# TfLiteTensorsToDetectionsCalculator to 1. This prevents the nodes in between
|
||||
# from queuing up incoming images and data excessively, which leads to increased
|
||||
# latency and memory usage, unwanted in real-time mobile applications. It also
|
||||
# eliminates unnecessarily computation, e.g., a transformed image produced by
|
||||
# ImageTransformationCalculator may get dropped downstream if the subsequent
|
||||
# TfLiteConverterCalculator or TfLiteInferenceCalculator is still busy
|
||||
# processing previous inputs.
|
||||
node {
|
||||
calculator: "FlowLimiterCalculator"
|
||||
input_stream: "input_video"
|
||||
input_stream: "FINISHED:detections"
|
||||
input_stream_info: {
|
||||
tag_index: "FINISHED"
|
||||
back_edge: true
|
||||
}
|
||||
output_stream: "throttled_input_video"
|
||||
}
|
||||
|
||||
# Transforms the input image on CPU to a 128x128 image. To scale the input
|
||||
# image, the scale_mode option is set to FIT to preserve the aspect ratio,
|
||||
# resulting in potential letterboxing in the transformed image.
|
||||
node: {
|
||||
calculator: "ImageTransformationCalculator"
|
||||
input_stream: "IMAGE:throttled_input_video"
|
||||
output_stream: "IMAGE:transformed_input_video_cpu"
|
||||
output_stream: "LETTERBOX_PADDING:letterbox_padding"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.ImageTransformationCalculatorOptions] {
|
||||
output_width: 192
|
||||
output_height: 192
|
||||
scale_mode: FIT
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Converts the transformed input image on CPU into an image tensor stored as a
|
||||
# TfLiteTensor.
|
||||
node {
|
||||
calculator: "TfLiteConverterCalculator"
|
||||
input_stream: "IMAGE:transformed_input_video_cpu"
|
||||
output_stream: "TENSORS:image_tensor"
|
||||
}
|
||||
|
||||
# Runs a TensorFlow Lite model on CPU that takes an image tensor and outputs a
|
||||
# vector of tensors representing, for instance, detection boxes/keypoints and
|
||||
# scores.
|
||||
node {
|
||||
calculator: "TfLiteInferenceCalculator"
|
||||
input_stream: "TENSORS:image_tensor"
|
||||
output_stream: "TENSORS:detection_tensors"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
|
||||
model_path: "mediapipe/modules/face_detection/face_detection_back.tflite"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Generates a single side packet containing a vector of SSD anchors based on
|
||||
# the specification in the options.
|
||||
node {
|
||||
calculator: "SsdAnchorsCalculator"
|
||||
output_side_packet: "anchors"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.SsdAnchorsCalculatorOptions] {
|
||||
num_layers: 1
|
||||
min_scale: 0.1484375
|
||||
max_scale: 0.75
|
||||
input_size_height: 192
|
||||
input_size_width: 192
|
||||
anchor_offset_x: 0.5
|
||||
anchor_offset_y: 0.5
|
||||
strides: 4
|
||||
aspect_ratios: 1.0
|
||||
fixed_anchor_size: true
|
||||
interpolated_scale_aspect_ratio: 0.0
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Decodes the detection tensors generated by the TensorFlow Lite model, based on
|
||||
# the SSD anchors and the specification in the options, into a vector of
|
||||
# detections. Each detection describes a detected object.
|
||||
node {
|
||||
calculator: "TfLiteTensorsToDetectionsCalculator"
|
||||
input_stream: "TENSORS:detection_tensors"
|
||||
input_side_packet: "ANCHORS:anchors"
|
||||
output_stream: "DETECTIONS:detections"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.TfLiteTensorsToDetectionsCalculatorOptions] {
|
||||
num_classes: 1
|
||||
num_boxes: 2304
|
||||
num_coords: 16
|
||||
box_coord_offset: 0
|
||||
keypoint_coord_offset: 4
|
||||
num_keypoints: 6
|
||||
num_values_per_keypoint: 2
|
||||
sigmoid_score: true
|
||||
score_clipping_thresh: 100.0
|
||||
reverse_output_order: true
|
||||
x_scale: 192.0
|
||||
y_scale: 192.0
|
||||
h_scale: 192.0
|
||||
w_scale: 192.0
|
||||
min_score_thresh: 0.6
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Performs non-max suppression to remove excessive detections.
|
||||
node {
|
||||
calculator: "NonMaxSuppressionCalculator"
|
||||
input_stream: "detections"
|
||||
output_stream: "filtered_detections"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.NonMaxSuppressionCalculatorOptions] {
|
||||
min_suppression_threshold: 0.3
|
||||
overlap_type: INTERSECTION_OVER_UNION
|
||||
algorithm: WEIGHTED
|
||||
return_empty_detections: true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Adjusts detection locations (already normalized to [0.f, 1.f]) on the
|
||||
# letterboxed image (after image transformation with the FIT scale mode) to the
|
||||
# corresponding locations on the same image with the letterbox removed (the
|
||||
# input image to the graph before image transformation).
|
||||
node {
|
||||
calculator: "DetectionLetterboxRemovalCalculator"
|
||||
input_stream: "DETECTIONS:filtered_detections"
|
||||
input_stream: "LETTERBOX_PADDING:letterbox_padding"
|
||||
output_stream: "DETECTIONS:output_detections"
|
||||
}
|
||||
|
||||
# Converts the detections to drawing primitives for annotation overlay.
|
||||
node {
|
||||
calculator: "DetectionsToRenderDataCalculator"
|
||||
input_stream: "DETECTIONS:output_detections"
|
||||
output_stream: "RENDER_DATA:render_data"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.DetectionsToRenderDataCalculatorOptions] {
|
||||
thickness: 4.0
|
||||
color { r: 255 g: 0 b: 0 }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Draws annotations and overlays them on top of the input images.
|
||||
node {
|
||||
calculator: "AnnotationOverlayCalculator"
|
||||
input_stream: "IMAGE:throttled_input_video"
|
||||
input_stream: "render_data"
|
||||
output_stream: "IMAGE:output_video"
|
||||
}
|
||||
|
||||
@@ -1,169 +0,0 @@
|
||||
# MediaPipe graph that performs face detection with TensorFlow Lite on GPU.
|
||||
# Used in the examples in
|
||||
# mediapipie/examples/android/src/java/com/mediapipe/apps/facedetectiongpu and
|
||||
# mediapipie/examples/ios/facedetectiongpu.
|
||||
|
||||
# Images on GPU coming into and out of the graph.
|
||||
input_stream: "input_video"
|
||||
output_stream: "output_video"
|
||||
|
||||
# Throttles the images flowing downstream for flow control. It passes through
|
||||
# the very first incoming image unaltered, and waits for
|
||||
# TfLiteTensorsToDetectionsCalculator downstream in the graph to finish
|
||||
# generating the corresponding detections before it passes through another
|
||||
# image. All images that come in while waiting are dropped, limiting the number
|
||||
# of in-flight images between this calculator and
|
||||
# TfLiteTensorsToDetectionsCalculator to 1. This prevents the nodes in between
|
||||
# from queuing up incoming images and data excessively, which leads to increased
|
||||
# latency and memory usage, unwanted in real-time mobile applications. It also
|
||||
# eliminates unnecessarily computation, e.g., a transformed image produced by
|
||||
# ImageTransformationCalculator may get dropped downstream if the subsequent
|
||||
# TfLiteConverterCalculator or TfLiteInferenceCalculator is still busy
|
||||
# processing previous inputs.
|
||||
node {
|
||||
calculator: "FlowLimiterCalculator"
|
||||
input_stream: "input_video"
|
||||
input_stream: "FINISHED:detections"
|
||||
input_stream_info: {
|
||||
tag_index: "FINISHED"
|
||||
back_edge: true
|
||||
}
|
||||
output_stream: "throttled_input_video"
|
||||
}
|
||||
|
||||
# Transforms the input image on GPU to a 128x128 image. To scale the input
|
||||
# image, the scale_mode option is set to FIT to preserve the aspect ratio,
|
||||
# resulting in potential letterboxing in the transformed image.
|
||||
node: {
|
||||
calculator: "ImageTransformationCalculator"
|
||||
input_stream: "IMAGE_GPU:throttled_input_video"
|
||||
output_stream: "IMAGE_GPU:transformed_input_video"
|
||||
output_stream: "LETTERBOX_PADDING:letterbox_padding"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.ImageTransformationCalculatorOptions] {
|
||||
output_width: 192
|
||||
output_height: 192
|
||||
scale_mode: FIT
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Converts the transformed input image on GPU into an image tensor stored as a
|
||||
# TfLiteTensor.
|
||||
node {
|
||||
calculator: "TfLiteConverterCalculator"
|
||||
input_stream: "IMAGE_GPU:transformed_input_video"
|
||||
output_stream: "TENSORS_GPU:image_tensor"
|
||||
}
|
||||
|
||||
# Runs a TensorFlow Lite model on GPU that takes an image tensor and outputs a
|
||||
# vector of tensors representing, for instance, detection boxes/keypoints and
|
||||
# scores.
|
||||
node {
|
||||
calculator: "TfLiteInferenceCalculator"
|
||||
input_stream: "TENSORS_GPU:image_tensor"
|
||||
output_stream: "TENSORS_GPU:detection_tensors"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.TfLiteInferenceCalculatorOptions] {
|
||||
model_path: "mediapipe/modules/face_detection/face_detection_back.tflite"
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Generates a single side packet containing a vector of SSD anchors based on
|
||||
# the specification in the options.
|
||||
node {
|
||||
calculator: "SsdAnchorsCalculator"
|
||||
output_side_packet: "anchors"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.SsdAnchorsCalculatorOptions] {
|
||||
num_layers: 1
|
||||
min_scale: 0.1484375
|
||||
max_scale: 0.75
|
||||
input_size_height: 192
|
||||
input_size_width: 192
|
||||
anchor_offset_x: 0.5
|
||||
anchor_offset_y: 0.5
|
||||
strides: 4
|
||||
aspect_ratios: 1.0
|
||||
fixed_anchor_size: true
|
||||
interpolated_scale_aspect_ratio: 0.0
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Decodes the detection tensors generated by the TensorFlow Lite model, based on
|
||||
# the SSD anchors and the specification in the options, into a vector of
|
||||
# detections. Each detection describes a detected object.
|
||||
node {
|
||||
calculator: "TfLiteTensorsToDetectionsCalculator"
|
||||
input_stream: "TENSORS_GPU:detection_tensors"
|
||||
input_side_packet: "ANCHORS:anchors"
|
||||
output_stream: "DETECTIONS:detections"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.TfLiteTensorsToDetectionsCalculatorOptions] {
|
||||
num_classes: 1
|
||||
num_boxes: 2304
|
||||
num_coords: 16
|
||||
box_coord_offset: 0
|
||||
keypoint_coord_offset: 4
|
||||
num_keypoints: 6
|
||||
num_values_per_keypoint: 2
|
||||
sigmoid_score: true
|
||||
score_clipping_thresh: 100.0
|
||||
reverse_output_order: true
|
||||
x_scale: 192.0
|
||||
y_scale: 192.0
|
||||
h_scale: 192.0
|
||||
w_scale: 192.0
|
||||
min_score_thresh: 0.6
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Performs non-max suppression to remove excessive detections.
|
||||
node {
|
||||
calculator: "NonMaxSuppressionCalculator"
|
||||
input_stream: "detections"
|
||||
output_stream: "filtered_detections"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.NonMaxSuppressionCalculatorOptions] {
|
||||
min_suppression_threshold: 0.3
|
||||
overlap_type: INTERSECTION_OVER_UNION
|
||||
algorithm: WEIGHTED
|
||||
return_empty_detections: true
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Adjusts detection locations (already normalized to [0.f, 1.f]) on the
|
||||
# letterboxed image (after image transformation with the FIT scale mode) to the
|
||||
# corresponding locations on the same image with the letterbox removed (the
|
||||
# input image to the graph before image transformation).
|
||||
node {
|
||||
calculator: "DetectionLetterboxRemovalCalculator"
|
||||
input_stream: "DETECTIONS:filtered_detections"
|
||||
input_stream: "LETTERBOX_PADDING:letterbox_padding"
|
||||
output_stream: "DETECTIONS:output_detections"
|
||||
}
|
||||
|
||||
# Converts the detections to drawing primitives for annotation overlay.
|
||||
node {
|
||||
calculator: "DetectionsToRenderDataCalculator"
|
||||
input_stream: "DETECTIONS:output_detections"
|
||||
output_stream: "RENDER_DATA:render_data"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.DetectionsToRenderDataCalculatorOptions] {
|
||||
thickness: 4.0
|
||||
color { r: 255 g: 0 b: 0 }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Draws annotations and overlays them on top of the input images.
|
||||
node {
|
||||
calculator: "AnnotationOverlayCalculator"
|
||||
input_stream: "IMAGE_GPU:throttled_input_video"
|
||||
input_stream: "render_data"
|
||||
output_stream: "IMAGE_GPU:output_video"
|
||||
}
|
||||
@@ -31,7 +31,7 @@ node {
|
||||
|
||||
# Subgraph that detects faces.
|
||||
node {
|
||||
calculator: "FaceDetectionFrontCpu"
|
||||
calculator: "FaceDetectionShortRangeCpu"
|
||||
input_stream: "IMAGE:throttled_input_video"
|
||||
output_stream: "DETECTIONS:face_detections"
|
||||
}
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
# MediaPipe graph that performs face detection with TensorFlow Lite on CPU.
|
||||
# Used in the examples in
|
||||
# mediapipe/examples/desktop/face_detection:face_detection_cpu.
|
||||
|
||||
# Images on GPU coming into and out of the graph.
|
||||
input_stream: "input_video"
|
||||
output_stream: "output_video"
|
||||
|
||||
# Throttles the images flowing downstream for flow control. It passes through
|
||||
# the very first incoming image unaltered, and waits for
|
||||
# TfLiteTensorsToDetectionsCalculator downstream in the graph to finish
|
||||
# generating the corresponding detections before it passes through another
|
||||
# image. All images that come in while waiting are dropped, limiting the number
|
||||
# of in-flight images between this calculator and
|
||||
# TfLiteTensorsToDetectionsCalculator to 1. This prevents the nodes in between
|
||||
# from queuing up incoming images and data excessively, which leads to increased
|
||||
# latency and memory usage, unwanted in real-time mobile applications. It also
|
||||
# eliminates unnecessarily computation, e.g., a transformed image produced by
|
||||
# ImageTransformationCalculator may get dropped downstream if the subsequent
|
||||
# TfLiteConverterCalculator or TfLiteInferenceCalculator is still busy
|
||||
# processing previous inputs.
|
||||
node {
|
||||
calculator: "FlowLimiterCalculator"
|
||||
input_stream: "input_video"
|
||||
input_stream: "FINISHED:detections"
|
||||
input_stream_info: {
|
||||
tag_index: "FINISHED"
|
||||
back_edge: true
|
||||
}
|
||||
output_stream: "throttled_input_video"
|
||||
}
|
||||
|
||||
# Detects faces.
|
||||
node {
|
||||
calculator: "FaceDetectionFullRangeCpu"
|
||||
input_stream: "IMAGE:throttled_input_video"
|
||||
output_stream: "DETECTIONS:detections"
|
||||
}
|
||||
|
||||
# Converts the detections to drawing primitives for annotation overlay.
|
||||
node {
|
||||
calculator: "DetectionsToRenderDataCalculator"
|
||||
input_stream: "DETECTIONS:detections"
|
||||
output_stream: "RENDER_DATA:render_data"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.DetectionsToRenderDataCalculatorOptions] {
|
||||
thickness: 4.0
|
||||
color { r: 255 g: 0 b: 0 }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Draws annotations and overlays them on top of the input images.
|
||||
node {
|
||||
calculator: "AnnotationOverlayCalculator"
|
||||
input_stream: "IMAGE:throttled_input_video"
|
||||
input_stream: "render_data"
|
||||
output_stream: "IMAGE:output_video"
|
||||
}
|
||||
|
||||
@@ -0,0 +1,60 @@
|
||||
# MediaPipe graph that performs face detection with TensorFlow Lite on GPU.
|
||||
# Used in the examples in
|
||||
# mediapipie/examples/android/src/java/com/mediapipe/apps/facedetectiongpu and
|
||||
# mediapipie/examples/ios/facedetectiongpu.
|
||||
|
||||
# Images on GPU coming into and out of the graph.
|
||||
input_stream: "input_video"
|
||||
output_stream: "output_video"
|
||||
|
||||
# Throttles the images flowing downstream for flow control. It passes through
|
||||
# the very first incoming image unaltered, and waits for
|
||||
# TfLiteTensorsToDetectionsCalculator downstream in the graph to finish
|
||||
# generating the corresponding detections before it passes through another
|
||||
# image. All images that come in while waiting are dropped, limiting the number
|
||||
# of in-flight images between this calculator and
|
||||
# TfLiteTensorsToDetectionsCalculator to 1. This prevents the nodes in between
|
||||
# from queuing up incoming images and data excessively, which leads to increased
|
||||
# latency and memory usage, unwanted in real-time mobile applications. It also
|
||||
# eliminates unnecessarily computation, e.g., a transformed image produced by
|
||||
# ImageTransformationCalculator may get dropped downstream if the subsequent
|
||||
# TfLiteConverterCalculator or TfLiteInferenceCalculator is still busy
|
||||
# processing previous inputs.
|
||||
node {
|
||||
calculator: "FlowLimiterCalculator"
|
||||
input_stream: "input_video"
|
||||
input_stream: "FINISHED:output_video"
|
||||
input_stream_info: {
|
||||
tag_index: "FINISHED"
|
||||
back_edge: true
|
||||
}
|
||||
output_stream: "throttled_input_video"
|
||||
}
|
||||
|
||||
# Detects faces.
|
||||
node {
|
||||
calculator: "FaceDetectionFullRangeGpu"
|
||||
input_stream: "IMAGE:throttled_input_video"
|
||||
output_stream: "DETECTIONS:detections"
|
||||
}
|
||||
|
||||
# Converts the detections to drawing primitives for annotation overlay.
|
||||
node {
|
||||
calculator: "DetectionsToRenderDataCalculator"
|
||||
input_stream: "DETECTIONS:detections"
|
||||
output_stream: "RENDER_DATA:render_data"
|
||||
node_options: {
|
||||
[type.googleapis.com/mediapipe.DetectionsToRenderDataCalculatorOptions] {
|
||||
thickness: 4.0
|
||||
color { r: 255 g: 0 b: 0 }
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
# Draws annotations and overlays them on top of the input images.
|
||||
node {
|
||||
calculator: "AnnotationOverlayCalculator"
|
||||
input_stream: "IMAGE_GPU:throttled_input_video"
|
||||
input_stream: "render_data"
|
||||
output_stream: "IMAGE_GPU:output_video"
|
||||
}
|
||||
@@ -41,7 +41,7 @@ node: {
|
||||
|
||||
# Subgraph that detects faces.
|
||||
node {
|
||||
calculator: "FaceDetectionFrontCpu"
|
||||
calculator: "FaceDetectionShortRangeCpu"
|
||||
input_stream: "IMAGE:input_video_cpu"
|
||||
output_stream: "DETECTIONS:face_detections"
|
||||
}
|
||||
|
||||
@@ -31,7 +31,7 @@ node {
|
||||
|
||||
# Subgraph that detects faces.
|
||||
node {
|
||||
calculator: "FaceDetectionFrontGpu"
|
||||
calculator: "FaceDetectionShortRangeGpu"
|
||||
input_stream: "IMAGE:throttled_input_video"
|
||||
output_stream: "DETECTIONS:face_detections"
|
||||
}
|
||||
|
||||
@@ -39,7 +39,7 @@ mediapipe_simple_subgraph(
|
||||
"//mediapipe/calculators/core:concatenate_detection_vector_calculator",
|
||||
"//mediapipe/calculators/core:split_vector_calculator",
|
||||
"//mediapipe/calculators/image:image_properties_calculator",
|
||||
"//mediapipe/modules/face_detection:face_detection_front_gpu",
|
||||
"//mediapipe/modules/face_detection:face_detection_short_range_gpu",
|
||||
"//mediapipe/modules/face_geometry:face_geometry_from_detection",
|
||||
],
|
||||
)
|
||||
|
||||
+1
-1
@@ -24,7 +24,7 @@ output_stream: "MULTI_FACE_GEOMETRY:multi_face_geometry"
|
||||
# Subgraph that detects faces and corresponding landmarks using the face
|
||||
# detection pipeline.
|
||||
node {
|
||||
calculator: "FaceDetectionFrontGpu"
|
||||
calculator: "FaceDetectionShortRangeGpu"
|
||||
input_stream: "IMAGE:input_image"
|
||||
output_stream: "DETECTIONS:multi_face_detection"
|
||||
}
|
||||
|
||||
Some files were not shown because too many files have changed in this diff Show More
Reference in New Issue
Block a user