talksik/mediapipe
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Project import generated by Copybara.

Browse Source 
PiperOrigin-RevId: 267274408
This commit is contained in:
MediaPipe Team
2019-09-04 19:00:29 -07:00
committed by jqtang
parent 731d2b9536
commit af67642055
80 changed files with 3181 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
mediapipe/docs/cycles.md
+4
View File
@@ -61,7 +61,11 @@ ordering and ignores packet timestamps, which will eliminate this inconvenience.
By default, MediaPipe calls the `Close()` method of a non-source calculator when
all of its input streams are done. In the example graph, we want to stop the
adder node as soon as the integer source is done. This is accomplished by
<<<<<<< HEAD
configuring the adder node with an alternative input stream hander,
=======
configuring the adder node with an alternative input stream handler,
>>>>>>> Project import generated by Copybara.
`EarlyCloseInputStreamHandler`.
## Relevant Source Code
mediapipe/docs/examples.md
+8
View File
@@ -85,6 +85,14 @@ and model details are described in the
[Hello World for C++](./hello_world_desktop.md) shows how to run a simple graph
using the MediaPipe C++ APIs.
<<<<<<< HEAD
=======
### Feature Extration for YouTube-8M Challenge
[Feature Extration for YouTube-8M Challenge](./youtube_8m.md) shows how to use
MediaPipe to prepare training data for the YouTube-8M Challenge.
>>>>>>> Project import generated by Copybara.
### Preparing Data Sets with MediaSequence
[Preparing Data Sets with MediaSequence](./media_sequence.md) shows how to use
mediapipe/docs/framework_concepts.md
+8
View File
@@ -17,7 +17,11 @@ packets and produces zero or more output streams and/or side packets.
### CalculatorBase
A calculator is created by defining a new sub-class of the
<<<<<<< HEAD
[`CalculatorBase`](http://github.com/google/mediapipe/mediapipe/framework/calculator_base.cc)
=======
[`CalculatorBase`](https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_base.cc)
>>>>>>> Project import generated by Copybara.
class, implementing a number of methods, and registering the new sub-class with
Mediapipe. At a minimum, a new calculator must implement the below four methods
@@ -31,7 +35,11 @@ Mediapipe. At a minimum, a new calculator must implement the below four methods
* After all calls to `Process()` finish or when all input streams close, the framework calls `Close()`. This function is always called if `Open()` was called and succeeded and even if the graph run terminated because of an error. No inputs are available via any input streams during `Close()`, but it still has access to input side packets and therefore may write outputs. After `Close()` returns, the calculator should be considered a dead node. The calculator object is destroyed as soon as the graph finishes running.
The following are code snippets from
<<<<<<< HEAD
[CalculatorBase.h](http://github.com/google/mediapipe/mediapipe/framework/calculator_base.h).
=======
[CalculatorBase.h](https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator_base.h).
>>>>>>> Project import generated by Copybara.
```c++
class CalculatorBase {
mediapipe/docs/hello_world_android.md
+4
View File
@@ -643,7 +643,11 @@ Initialize the asset manager in `onCreate(Bundle)` before initializing
`eglManager`:
```
<<<<<<< HEAD
// Initilize asset manager so that MediaPipe native libraries can access the app assets, e.g.,
=======
// Initialize asset manager so that MediaPipe native libraries can access the app assets, e.g.,
>>>>>>> Project import generated by Copybara.
// binary graphs.
AndroidAssetUtil.initializeNativeAssetManager(this);
```
mediapipe/docs/hello_world_ios.md
+4
View File
@@ -6,7 +6,11 @@ This codelab uses MediaPipe on an iOS device.
### What you will learn
<<<<<<< HEAD
How to develop an Android application that uses MediaPipe and run a MediaPipe
=======
How to develop an iOS application that uses MediaPipe and run a MediaPipe
>>>>>>> Project import generated by Copybara.
graph on iOS.
### What you will build
mediapipe/docs/help.md
+40
View File
@@ -1,9 +1,18 @@
<<<<<<< HEAD
## Getting help
- [Technical questions](#technical-questions)
- [Bugs and Feature requests](#bugs-and-feature-requests)
Below are the various ways to get help
=======
## Getting Help
- [Technical questions](#technical-questions)
- [Bugs and feature requests](#bugs-and-feature-requests)
Below are the various ways to get help:
>>>>>>> Project import generated by Copybara.
### Technical questions
@@ -11,9 +20,40 @@ For help with technical or algorithmic questions, visit
[Stack Overflow](https://stackoverflow.com/questions/tagged/mediapipe) to find
answers and support from the MediaPipe community.
<<<<<<< HEAD
### Bugs and Feature requests
To report bugs or make feature requests,
[file an issue on GitHub](https://github.com/google/mediapipe/mediapipe/issues).
Please choose the appropriate repository for the project from the
[MediaPipe repo](https://github.com/google/mediapipe/mediapipe)
=======
### Bugs and feature requests
To report bugs or make feature requests,
[file an issue on GitHub](https://github.com/google/mediapipe/issues).
If you open a GitHub issue, here is our policy:
1. It must be a bug, a feature request, or a significant problem with documentation (for small doc fixes please send a PR instead).
2. The form below must be filled out.
**Here's why we have that policy**: MediaPipe developers respond to issues. We want to focus on work that benefits the whole community, e.g., fixing bugs and adding features. Support only helps individuals. GitHub also notifies thousands of people when issues are filed. We want them to see you communicating an interesting problem, rather than being redirected to Stack Overflow.
------------------------
### System information
- **Have I written custom code**:
- **OS Platform and Distribution (e.g., Linux Ubuntu 16.04)**:
- **Mobile device (e.g. iPhone 8, Pixel 2, Samsung Galaxy) if the issue happens on mobile device**:
- **Bazel version**:
- **Android Studio, NDK, SDK versions (if issue is related to building in mobile dev enviroment)**:
- **Xcode & Tulsi version (if issue is related to building in mobile dev enviroment)**:
- **Exact steps to reproduce**:
### Describe the problem
Describe the problem clearly here. Be sure to convey here why it's a bug in MediaPipe or a feature request.
### Source code / logs
Include any logs or source code that would be helpful to diagnose the problem. If including tracebacks, please include the full traceback. Large logs and files should be attached instead of being pasted into the issue as text.
>>>>>>> Project import generated by Copybara.
mediapipe/docs/install.md
+170
View File
@@ -49,10 +49,17 @@ To build and run iOS apps:
[documentation](https://docs.bazel.build/versions/master/install-ubuntu.html)
to install any version of Bazel manually.
<<<<<<< HEAD
3. Install OpenCV.
Option 1. Use package manager tool to install the pre-compiled OpenCV
libraries.
=======
3. Install OpenCV and FFmpeg.
Option 1. Use package manager tool to install the pre-compiled OpenCV
libraries. FFmpeg will be installed via libopencv-video-dev.
>>>>>>> Project import generated by Copybara.
Note: Debian 9 and Ubuntu 16.04 provide OpenCV 2.4.9. You may want to take
option 2 or 3 to install OpenCV 3 or above.
@@ -83,6 +90,7 @@ To build and run iOS apps:
)
cc_library(
<<<<<<< HEAD
name = "opencv",
srcs = glob(
[
@@ -101,6 +109,24 @@ To build and run iOS apps:
visibility = ["//visibility:public"],
)
=======
name = "opencv",
srcs = glob(
[
"lib/libopencv_core.so",
"lib/libopencv_highgui.so",
"lib/libopencv_imgcodecs.so",
"lib/libopencv_imgproc.so",
"lib/libopencv_video.so",
"lib/libopencv_videoio.so",
],
),
hdrs = glob(["include/opencv4/**/*.h*"]),
includes = ["include/opencv4/"],
linkstatic = 1,
visibility = ["//visibility:public"],
)
>>>>>>> Project import generated by Copybara.
```
4. Run the [Hello World desktop example](./hello_world_desktop.md).
@@ -168,6 +194,7 @@ To build and run iOS apps:
)
cc_library(
<<<<<<< HEAD
name = "opencv",
srcs = glob(
[
@@ -186,6 +213,24 @@ To build and run iOS apps:
visibility = ["//visibility:public"],
)
=======
name = "opencv",
srcs = glob(
[
"lib/libopencv_core.so",
"lib/libopencv_highgui.so",
"lib/libopencv_imgcodecs.so",
"lib/libopencv_imgproc.so",
"lib/libopencv_video.so",
"lib/libopencv_videoio.so",
],
),
hdrs = glob(["include/opencv4/**/*.h*"]),
includes = ["include/opencv4/"],
linkstatic = 1,
visibility = ["//visibility:public"],
)
>>>>>>> Project import generated by Copybara.
```
4. Run the [Hello World desktop example](./hello_world_desktop.md).
@@ -239,10 +284,17 @@ To build and run iOS apps:
[documentation](https://docs.bazel.build/versions/master/install-ubuntu.html)
to install any version of Bazel manually.
<<<<<<< HEAD
4. Install OpenCV.
Option 1. Use HomeBrew package manager tool to install the pre-compiled
OpenCV libraries.
=======
4. Install OpenCV and FFmpeg.
Option 1. Use HomeBrew package manager tool to install the pre-compiled
OpenCV libraries. FFmpeg will be installed via OpenCV.
>>>>>>> Project import generated by Copybara.
```bash
$ brew install opencv
@@ -254,6 +306,7 @@ To build and run iOS apps:
$ port install opencv
```
<<<<<<< HEAD
Note: when using MacPorts, please edit the [`WORKSAPCE`] and
[`opencv_linux.BUILD`] files like the following:
@@ -281,6 +334,60 @@ To build and run iOS apps:
linkstatic = 1,
visibility = ["//visibility:public"],
)
=======
Note: when using MacPorts, please edit the [`WORKSAPCE`],
[`opencv_macos.BUILD`], and [`ffmpeg_macos.BUILD`] files like the following:
```bash
new_local_repository(
name = "macos_opencv",
build_file = "@//third_party:opencv_macos.BUILD",
path = "/opt",
)
new_local_repository(
name = "macos_ffmpeg",
build_file = "@//third_party:ffmpeg_macos.BUILD",
path = "/opt",
)
cc_library(
name = "opencv",
srcs = glob(
[
"local/lib/libopencv_core.dylib",
"local/lib/libopencv_highgui.dylib",
"local/lib/libopencv_imgcodecs.dylib",
"local/lib/libopencv_imgproc.dylib",
"local/lib/libopencv_video.dylib",
"local/lib/libopencv_videoio.dylib",
],
),
hdrs = glob(["local/include/opencv2/**/*.h*"]),
includes = ["local/include/"],
linkstatic = 1,
visibility = ["//visibility:public"],
)
cc_library(
name = "libffmpeg",
srcs = glob(
[
"local/lib/libav*.dylib",
],
),
hdrs = glob(["local/include/libav*/*.h"]),
includes = ["local/include/"],
linkopts = [
"-lavcodec",
"-lavformat",
"-lavutil",
],
linkstatic = 1,
visibility = ["//visibility:public"],
)
>>>>>>> Project import generated by Copybara.
```
5. Run the [Hello World desktop example](./hello_world_desktop.md).
@@ -350,10 +457,17 @@ To build and run iOS apps:
username@DESKTOP-TMVLBJ1:~$ cd mediapipe
```
<<<<<<< HEAD
7. Install OpenCV.
Option 1. Use package manager tool to install the pre-compiled OpenCV
libraries.
=======
7. Install OpenCV and FFmpeg.
Option 1. Use package manager tool to install the pre-compiled OpenCV
libraries. FFmpeg will be installed via libopencv-video-dev.
>>>>>>> Project import generated by Copybara.
```bash
username@DESKTOP-TMVLBJ1:~/mediapipe$ sudo apt-get install libopencv-core-dev libopencv-highgui-dev \
@@ -381,6 +495,7 @@ To build and run iOS apps:
)
cc_library(
<<<<<<< HEAD
name = "opencv",
srcs = glob(
[
@@ -399,6 +514,24 @@ To build and run iOS apps:
visibility = ["//visibility:public"],
)
=======
name = "opencv",
srcs = glob(
[
"lib/libopencv_core.so",
"lib/libopencv_highgui.so",
"lib/libopencv_imgcodecs.so",
"lib/libopencv_imgproc.so",
"lib/libopencv_video.so",
"lib/libopencv_videoio.so",
],
),
hdrs = glob(["include/opencv4/**/*.h*"]),
includes = ["include/opencv4/"],
linkstatic = 1,
visibility = ["//visibility:public"],
)
>>>>>>> Project import generated by Copybara.
```
8. Run the [Hello World desktop example](./hello_world_desktop.md).
@@ -428,7 +561,11 @@ To build and run iOS apps:
This will use a Docker image that will isolate mediapipe's installation from the rest of the system.
1. [Install Docker](https://docs.docker.com/install/#supported-platforms) on
<<<<<<< HEAD
your host sytem.
=======
your host system.
>>>>>>> Project import generated by Copybara.
2. Build a docker image with tag "mediapipe".
@@ -473,7 +610,35 @@ This will use a Docker image that will isolate mediapipe's installation from the
# Hello World!
```
<<<<<<< HEAD
<!-- 4. Uncomment the last line of the Dockerfile
=======
4. Build Mediapipe [Android demos](./examples.md).
```bash
$ docker run -it --name mediapipe mediapipe:latest
root@bca08b91ff63:/mediapipe# bash ./setup_android_sdk_and_ndk
# Should print:
# Android NDK is now installed. Consider setting $ANDROID_NDK_HOME environment variable to be /root/Android/Sdk/ndk-bundle/android-ndk-r18b
# Set android_ndk_repository and android_sdk_repository in WORKSPACE
# Done
root@bca08b91ff63:/mediapipe# bazel build -c opt --config=android_arm64 mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu:objectdetectiongpu
# Should print:
# Target //mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu:objectdetectiongpu up-to-date:
# bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu/objectdetectiongpu_deploy.jar
# bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu/objectdetectiongpu_unsigned.apk
# bazel-bin/mediapipe/examples/android/src/java/com/google/mediapipe/apps/objectdetectiongpu/objectdetectiongpu.apk
# INFO: Elapsed time: 144.462s, Critical Path: 79.47s
# INFO: 1958 processes: 1 local, 1863 processwrapper-sandbox, 94 worker.
# INFO: Build completed successfully, 2028 total actions
```
<!-- 5. Uncomment the last line of the Dockerfile
>>>>>>> Project import generated by Copybara.
```bash
RUN bazel build -c opt --define MEDIAPIPE_DISABLE_GPU=1 mediapipe/examples/desktop/demo:object_detection_tensorflow_demo
@@ -563,5 +728,10 @@ The steps below use Android Studio to build and install a MediaPipe example app.
[`WORKSAPCE`]: https://github.com/google/mediapipe/tree/master/WORKSPACE
[`opencv_linux.BUILD`]: https://github.com/google/mediapipe/tree/master/third_party/opencv_linux.BUILD
<<<<<<< HEAD
=======
[`opencv_macos.BUILD`]: https://github.com/google/mediapipe/tree/master/third_party/opencv_macos.BUILD
[`ffmpeg_macos.BUILD`]:https://github.com/google/mediapipe/tree/master/third_party/ffmpeg_macos.BUILD
>>>>>>> Project import generated by Copybara.
[`setup_opencv.sh`]: https://github.com/google/mediapipe/tree/master/setup_opencv.sh
[`setup_android_sdk_and_ndk.sh`]: https://github.com/google/mediapipe/tree/master/setup_android_sdk_and_ndk.sh
mediapipe/docs/media_sequence.md
+8
View File
@@ -1,6 +1,10 @@
## Preparing Data Sets with MediaSequence
<<<<<<< HEAD
MediaPipe is useful and general framework for media processing that can
=======
MediaPipe is a useful and general framework for media processing that can
>>>>>>> Project import generated by Copybara.
assist with research, development, and deployment of ML models. This example
focuses on development by demonstrating how to prepare video data for training
a TensorFlow model.
@@ -18,7 +22,11 @@ process new data sets, in the documentation of
### Preparing an example data set
<<<<<<< HEAD
1. Checkout mediapipe repository
=======
1. Checkout the mediapipe repository
>>>>>>> Project import generated by Copybara.
```bash
git clone https://github.com/google/mediapipe.git
mediapipe/docs/mediapipe_ios_setup.md
+16
View File
@@ -78,7 +78,13 @@ Debug to Release. Note that this is set independently for each target.
## Building an iOS app from the command line
<<<<<<< HEAD
1. Build one of the example apps for iOS. We will be using the
=======
1. Modify the `bundle_id` field of the app's ios_application rule to use your own identifier, e.g. for [Face Detection GPU App example](./face_detection_mobile_gpu.md), you need to modify the line 26 of the [BUILD file](https://github.com/google/mediapipe/blob/master/mediapipe/examples/ios/facedetectiongpu/BUILD).
2. Build one of the example apps for iOS. We will be using the
>>>>>>> Project import generated by Copybara.
[Face Detection GPU App example](./face_detection_mobile_gpu.md)
```bash
@@ -88,6 +94,7 @@ Debug to Release. Note that this is set independently for each target.
You may see a permission request from `codesign` in order to sign the app.
<<<<<<< HEAD
2. In Xcode, open the `Devices and Simulators` window (command-shift-2).
3. Make sure your device is connected. You will see a list of installed apps.
@@ -95,3 +102,12 @@ Debug to Release. Note that this is set independently for each target.
Bazel.
4. You can now run the app on your device.
=======
3. In Xcode, open the `Devices and Simulators` window (command-shift-2).
4. Make sure your device is connected. You will see a list of installed apps.
Press the "+" button under the list, and select the `.ipa` file built by
Bazel.
5. You can now run the app on your device.
>>>>>>> Project import generated by Copybara.
mediapipe/docs/troubleshooting.md
+4
View File
@@ -41,7 +41,11 @@ When the library for a calculator is linked into an application binary, the
calculator is automatically registered by name through the
[`REGISTER_CALCULATOR`] macro using the [`registration.h`] library. Note that
[`REGISTER_CALCULATOR`] can register a calculator with a namespace prefix,
<<<<<<< HEAD
identical to its C++ namespace. In this case, the calcultor graph must also use
=======
identical to its C++ namespace. In this case, the calculator graph must also use
>>>>>>> Project import generated by Copybara.
the same namespace prefix.
## Out Of Memory error
mediapipe/docs/youtube_8m.md
+83
View File
@@ -0,0 +1,83 @@
## Extracting Video Features for YouTube-8M Challenge
MediaPipe is a useful and general framework for media processing that can assist
with research, development, and deployment of ML models. This example focuses on
model development by demonstrating how to prepare training data for the
YouTube-8M Challenge.
[Youtube-8M Challenge](https://www.kaggle.com/c/youtube8m-2019) is an annual
video classification challenge hosted by Google. Over the last two years, the
first two challenges have collectively drawn 1000+ teams from 60+ countries to
further advance large-scale video understanding research. In addition to the
feature extraction Python code released in the
[google/youtube-8m](https://github.com/google/youtube-8m/tree/master/feature_extractor)
repo, we release a MediaPipe based feature extraction pipeline that can extract
both video and audio features from a local video. The MediaPipe based pipeline
utilizes two machine learning models,
[Inception v3](https://github.com/tensorflow/models/tree/master/research/inception)
and
[VGGish](https://github.com/tensorflow/models/tree/master/research/audioset/vggish),
to extract features from video and audio respectively.
To visualize the
[graph](https://github.com/google/mediapipe/tree/master/mediapipe/graphs/youtube8m/feature_extraction.pbtxt),
copy the text specification of the graph and paste it into
[MediaPipe Visualizer](https://viz.mediapipe.dev/). The feature extraction
pipeline is highly customizable. You are welcome to add new calculators or use
your own machine learning models to extract more advanced features from the
videos.
### Steps to run the YouTube-8M feature extraction graph
1. Checkout the mediapipe repository
```bash
git clone https://github.com/google/mediapipe.git
cd mediapipe
```
2. Download the PCA and model data
```bash
mkdir /tmp/mediapipe
cd /tmp/mediapipe
curl -O http://data.yt8m.org/pca_matrix_data/inception3_mean_matrix_data.pb
curl -O http://data.yt8m.org/pca_matrix_data/inception3_projection_matrix_data.pb
curl -O http://data.yt8m.org/pca_matrix_data/vggish_mean_matrix_data.pb
curl -O http://data.yt8m.org/pca_matrix_data/vggish_projection_matrix_data.pb
curl -O http://download.tensorflow.org/models/image/imagenet/inception-2015-12-05.tgz
tar -xvf /tmp/mediapipe/inception-2015-12-05.tgz
```
3. Get the VGGish frozen graph
Note: To run step 3 and step 4, you must have Python 2.7 or 3.5+ installed
with the TensorFlow 1.14+ package installed.
```bash
# cd to the root directory of the MediaPipe repo
cd -
python -m mediapipe.examples.desktop.youtube8m.generate_vggish_frozen_graph
```
4. Generate a MediaSequence metadata from the input video
Note: the output file is /tmp/mediapipe/metadata.tfrecord
```bash
python -m mediapipe.examples.desktop.youtube8m.generate_input_sequence_example \
--path_to_input_video=/absolute/path/to/the/local/video/file
```
5. Run the MediaPipe binary to extract the features
```bash
bazel build -c opt \
--define MEDIAPIPE_DISABLE_GPU=1 --define no_aws_support=true \
mediapipe/examples/desktop/youtube8m:extract_yt8m_features
./bazel-bin/mediapipe/examples/desktop/youtube8m/extract_yt8m_features
--calculator_graph_config_file=mediapipe/graphs/youtube8m/feature_extraction.pbtxt \
--input_side_packets=input_sequence_example=/tmp/mediapipe/metadata.tfrecord \
--output_side_packets=output_sequence_example=/tmp/mediapipe/output.tfrecord
```