talksik/mediapipe
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

Project import generated by Copybara.

Browse Source 
GitOrigin-RevId: f72a0f86c2c2acdb1920973c718a9e26ed3ec4b6
This commit is contained in:
MediaPipe Team
2020-06-08 12:08:33 -04:00
committed by chuoling
parent 5d028d923b
commit cd2b69d58c
315 changed files with 8162 additions and 7502 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs/tools/performance_benchmarking.md
+18
View File
@@ -0,0 +1,18 @@
---
layout: default
title: Performance Benchmarking
parent: Tools
nav_order: 3
---
# Performance Benchmarking
{: .no_toc }
1. TOC
{:toc}
---
*Coming soon.*
Future mediapipe releases will include tools for visualizing and analysing the
latency histograms and timed events captured for performance benchmarking.
docs/tools/tools.md
+13
View File
@@ -0,0 +1,13 @@
---
layout: default
title: Tools
nav_order: 4
has_children: true
---
# Tools
{: .no_toc }
1. TOC
{:toc}
---
docs/tools/tracing_and_profiling.md
+303
View File
@@ -0,0 +1,303 @@
---
layout: default
title: Tracing and Profiling
parent: Tools
nav_order: 2
---
# Tracing and Profiling
{: .no_toc }
1. TOC
{:toc}
---
The MediaPipe framework includes a built-in tracer and profiler. The tracer
records various timing events related to packet processing, including the start
and end time of each Calculator::Process call. The tracer writes trace log files
in binary protobuf format. The profiler further accumulates for each running
calculator a histogram of latencies for Process calls. Tracing and profiling is
available on Linux, Android, or iOS.
## Enabling tracing and profiling
To enable tracing/profiling of a mediapipe graph, the `CalculatorGraphConfig` (in
[calculator.proto](https://github.com/google/mediapipe/tree/master/mediapipe/framework/calculator.proto))
representing the graph must have a `profiler_config` message at its root. Here
is a simple setup that turns on a few extra options:
```
profiler_config {
enable_profiler: true
trace_enabled: true
trace_log_count: 5
}
```
* `enable_profiler` is required to emit any logging at all.
* `trace_enabled` gives us packet level information needed for offline
profiling.
* `trace_log_count` is a convenience that allows us to, by default, to chop up
our log into five separate files which are filled up in a round robin
fashion (after the fifth file is recorded, the first file is used again).
The trace log files are named `trace_0.log` through `trace_k.log`.
See [Profiler configuration](#profiler_configuration) for other settings
available in the profiler config. Note that most of the other settings are
considered advanced, and in general should not be needed.
## Collecting the Logs
MediaPipe will emit data into a pre-specified directory:
* On the desktop, this will be the `/tmp` directory.
* On Android, this will be the `/sdcard` directory.
* On iOS, this can be reached through XCode. Select "Window/Devices and
Simulators" and select the "Devices" tab.
![Windows Select Devices](../images/visualizer/ios_window_devices.png)
You can open the Download Container. Logs will be located in `application
container/.xcappdata/AppData/Documents/`
![Windows Download Container](../images/visualizer/ios_download_container.png)
Log files are written to `\<trace_log_path index\>.binarypb` where, by default,
`\<trace_log_path\>` is equal to `mediapipe_trace_` (the entire path and file
prefix can be overwritten by setting `trace_log_path` within the
`profiler_config` message). The index will, by default, alternate between 0 and
1, unless you've overridden the trace_log_count as we did, above.
By default, each file records five seconds of events. (Advanced: Specifically,
we record ten intervals of half a second each. This can be overridden by adding
`trace_log_interval_usec` and `trace_log_interval_count` to `profiler_config`).
### Tracing on Linux
1. Follow the instructions stated above in `Enable tracing`
2. Build and run your MediaPipe graph. The running graph writes trace events as
stated above in `Collect the logs`
### Tracing on Android
* Ensure that the Android app has write permissions to external storage.
* Include the line below in your `AndroidManifest.xml` file.
```xml
<uses-permission android:name="android.permission.WRITE_EXTERNAL_STORAGE" />
```
* Grant the permission either upon first app launch, or by going into
`Settings -> Apps & notifications -> $YOUR_APP -> Permissions` and
enable `Storage`.
* Add the following protobuf message into the existing calculator-graph-config
protobuf, such as the existing `.pbtxt` file. Follow the instructions stated
above in `Enable tracing`
* Connect your Android device and run `adb devices`.
```bash
adb devices
# should print:
# List of devices attached
# 805KPWQ1876505 device
```
* Use `bazel build` to compile the Android app and use `adb install` to get it
installed on your Android device.
* Open the installed Android app. The running MediaPipe graph appends trace
events to a trace log files at:
```bash
/sdcard/mediapipe_trace_0.binarypb
/sdcard/mediapipe_trace_1.binarypb
```
After every 5 sec, writing shifts to a successive trace log file, such that
the most recent 5 sec of events are preserved. You can check whether the
trace files have been written to the device using adb shell.
```bash
adb shell "ls -la /sdcard/"
```
On android, MediaPipe selects the external storage directory `/sdcard` for
trace logs. This directory can be overridden using the setting
`trace_log_path`, like:
```bash
profiler_config {
trace_enabled: true
trace_log_path: "/sdcard/profiles"
}
```
* Download the trace files from the device.
```bash
# from your terminal
adb pull /sdcard/mediapipe_trace_0.binarypb
# if successful you should see something like
# /sdcard/mediapipe_trace_0.binarypb: 1 file pulled. 0.1 MB/s (6766 bytes in 0.045s)
```
## Analyzing the Logs
Trace logs can be analyzed from within the visualizer.
1. Navigate to
[viz.mediapipe.dev](https://viz.mediapipe.dev)
2. Click on the "Upload" button in the upper right.
![Click on Upload](../images/visualizer/viz_click_upload.png)
3. Click on "Upload trace file".
![Click on Upload](../images/visualizer/viz_click_upload_trace_file.png)
A sample trace file has been generated for you:
[sample_trace_binary.pb](../data/visualizer/sample_trace.binarypb)
4. A file selection popup will appear. Select the `.binarypb` that holds your
trace information.
5. A chart view will appear. All of your calculators will appear along the left
with profiling information listed along the top.
![Click on Upload](../images/visualizer/viz_chart_view.png)
Click on a header to alternately sort that column in ascending or descending
order. You can also scroll horizontally and vertically within the control to
see more columns and more calculators.
### Explanation of columns:
name
: The name of the calculator.
fps
: The number of frames that this calculator can generate each second, on
average. `1 / (input_latency_mean + time_mean`) (Units are 1 / second).
frequency
: The rate that this calculator was asked to process packets per second.
(Computed by `# of calls total / (last_call_time - first_call_time))`.
(Units are `1 / second`)
counter
: Number of times process() was called on the calculator. It is the `sum of
dropped + completed`.
dropped
: Number of times the calculator was called but did not produce an output.
completed
: Number of times that this calculator was asked to process inputs after which
it generated outputs.
processing_rate
: `1E+6 / time_mean`. The number of times per second this calculator could run
process, on average. (Units are `1 / second`).
thread_count
: The number of threads that made use of each calculator.
time_mean
: Average time spent within a calculator (in microseconds).
time_stddev
: Standard deviation of time_mean (in microseconds).
time_total
: Total time spent within a calculator (in microseconds).
time_percent
: Percent of total time spent within a calculator.
input_latency_mean
: Average latency between earliest input packet used by a iteration of the
calculator and when the calculator actually begins processing (in
microseconds).
input_latency_stddev
: Standard deviation of input_latency_mean (in microseconds).
input_latency_total
: Total accumulated input_latency (in microseconds).
## Profiler configuration
Many of the following settings are advanced and not recommended for general
usage. Consult [Enabling tracing and profiling](#enabling-tracing-and-profiling)
for a friendlier introduction.
histogram_interval_size_usec :Specifies the size of the runtimes histogram
intervals (in microseconds) to generate the histogram of the Process() time. The
last interval extends to +inf. If not specified, the interval is 1000000 usec =
1 sec.
num_histogram_intervals :Specifies the number of intervals to generate the
histogram of the `Process()` runtime. If not specified, one interval is used.
enable_profiler
: If true, the profiler starts profiling when graph is initialized.
enable_stream_latency
: If true, the profiler also profiles the stream latency and input-output
latency. No-op if enable_profiler is false.
use_packet_timestamp_for_added_packet
: If true, the profiler uses packet timestamp (as production time and source
production time) for packets added by calling
`CalculatorGraph::AddPacketToInputStream()`. If false, uses the profiler's
clock.
trace_log_capacity
: The maximum number of trace events buffered in memory. The default value
buffers up to 20000 events.
trace_event_types_disabled
: Trace event types that are not logged.
trace_log_path
: The output directory and base-name prefix for trace log files. Log files are
written to: StrCat(trace_log_path, index, "`.binarypb`")
trace_log_count
: The number of trace log files retained. The trace log files are named
"`trace_0.log`" through "`trace_k.log`". The default value specifies 2
output files retained.
trace_log_interval_usec
: The interval in microseconds between trace log output. The default value
specifies trace log output once every 0.5 sec.
trace_log_margin_usec
: The interval in microseconds between TimeNow and the highest times included
in trace log output. This margin allows time for events to be appended to
the TraceBuffer.
trace_log_duration_events
: False specifies an event for each calculator invocation. True specifies a
separate event for each start and finish time.
trace_log_interval_count
: The number of trace log intervals per file. The total log duration is:
`trace_log_interval_usec * trace_log_file_count * trace_log_interval_count`.
The default value specifies 10 intervals per file.
trace_log_disabled
: An option to turn ON/OFF writing trace files to disk. Saving trace files to
disk is enabled by default.
trace_enabled
: If true, tracer timing events are recorded and reported.
docs/tools/visualizer.md
+106
View File
@@ -0,0 +1,106 @@
---
layout: default
title: Visualizer
parent: Tools
nav_order: 1
---
# Visualizer
{: .no_toc }
1. TOC
{:toc}
---
To help users understand the structure of their calculator graphs and to
understand the overall behavior of their machine learning inference pipelines,
we have built the [MediaPipe Visualizer](https://viz.mediapipe.dev/)
that is available online.
* A graph view allows users to see a connected calculator graph as expressed
through a graph configuration that is pasted into the graph editor or
uploaded. The user can visualize and troubleshoot a graph they have created.
![Startup screen](../images/startup_screen.png)
## Working within the Editor
Getting Started:
The graph can be modified by adding and editing code in the Editor view.
![Editor UI](../images/editor_view.png)
* Pressing the "New" button in the upper right corner will clear any existing
code in the Editor window.
![New Button](../images/upload_button.png)
* Pressing the "Upload" button will prompt the user to select a local PBTXT
file, which will everwrite the current code within the editor.
* Alternatively, code can be pasted directly into the editor window.
* Errors and informational messages will appear in the Feedback window.
![Error Msg](../images/console_error.png)
## Understanding the Graph
The visualizer graph shows the connections between calculator nodes.
* Streams exit from the bottom of the calculator producing the stream and
enter the top of any calculator receiving the stream. (Notice the use of the
key, "input_stream" and "output_stream").
![Stream UI](../images/stream_ui.png)
![Stream_code](../images/stream_code.png)
* Sidepackets work the same, except that they exit a node on the right and
enter on the left. (Notice the use of the key, "input_side_packet" and
"output_side_packet").
![Sidepacket UI](../images/side_packet.png)
![Sidepacket_code](../images/side_packet_code.png)
* There are special nodes that represent inputs and outputs to the graph and
can supply either side packets or streams.
![Special nodes](../images/special_nodes.png)
![Special nodes](../images/special_nodes_code.png)
## Visualizing Subgraphs
The MediaPipe visualizer can display multiple graphs in separate tabs. If a
graph has a `type` field in the top level of the graph's text proto definition,
and that value of graph `type` is used as a calculator name in another graph, it
is considered a subgraph by the visualizer and colored appropriately where it is
used. Clicking on a subgraph will navigate to the corresponding tab which holds
the subgraph's definition.
For instance, there are two graphs involved in
[MediaPipe Hand](../solutions/hand.md): the main graph
([source pbtxt file](https://github.com/google/mediapipe/blob/master/mediapipe/graphs/hand_tracking/hand_detection_mobile.pbtxt))
and its associated subgraph
([source pbtxt file](https://github.com/google/mediapipe/blob/master/mediapipe/graphs/hand_tracking/subgraphs/hand_detection_gpu.pbtxt)).
To visualize them:
* In the MediaPipe visualizer, click on the upload graph button and select the
2 pbtxt files to visualize (main graph and its associated subgraph).
![Upload graph button](../images/upload_button.png)
![Choose the 2 files](../images/upload_2pbtxt.png)
* There will be 2 additional tabs. The main graph tab is
`hand_detection_mobile.pbtxt`.
![hand_detection_mobile_gpu.pbtxt](../images/maingraph_visualizer.png)
* Clicking on the `HandDetection` node in purple redirects the view to the
`hand_detection_gpu.pbtxt` tab.
![Hand detection subgraph](../images/click_subgraph_handdetection.png)