Converting Tensorflow graphs into TensorRT networks

Story

A few weeks ago we needed to convert one of our own Tensorflow graphs into a TensorRT network. As many of you probably know, there are a few options to accomplish this, like the Tensorflow to UFF and UFF to TensorRT parser or the Tensorflow to ONNX and ONNX to TensorRT parser. When trying the first approach the following error message was one of many we encountered: UffParser: Validator error: slice_9-26_9-26: Unsupported operation Slice. Some of the problems are circumventable but in the end we had to abandon the UFF to TensorRT parser, since it is full of bugs and closed source. The ONNX way seemed more promising since its intermediate format was visualisable and changeable. Unfortunately the packages provided by Anaconda and PyPI were flawed and fixing the C++ source code felt like a lot of work. Especially since the python API of TensorRT to construct networks looked clean and had all operations we needed.

(more…)