PaddlePaddle/Anakin

PaddlePaddle/Anakin

Description: High performance Cross-platform Inference-engine, you could run Anakin on x86-cpu,arm, nv-gpu, amd-gpu,bitmain and cambricon devices.

Language: C++

License: Apache-2.0

Stars: 536

Forks: 135

Open issues: 68

Created: 2018-05-18T05:32:25Z

Pushed: 2022-09-23T22:22:29Z

Default branch: master

Fork: no

Archived: yes

README:

Anakin2.0

 

Welcome to the Anakin GitHub.

Anakin is a cross-platform, high-performance inference engine, which is originally developed by Baidu engineers and is a large-scale application of industrial products.

Please refer to our release announcement to track the latest feature of Anakin.

Features

• Flexibility

Anakin is a cross-platform, high-performance inference engine, supports a wide range of neural network architectures and different hardware platforms. It is easy to run Anakin on GPU / x86 / ARM platform.

Anakin has integrated with NVIDIA TensorRT and open source this part of integrated API to provide services, developers can call the API directly or modify it as needed, which will be more flexible for development requirements.

• High performance

In order to give full play to the performance of hardware, we optimized the forward prediction at different levels.

• Automatic graph fusion. The goal of all performance optimizations under a

given algorithm is to make the ALU as busy as possible. Operator fusion can effectively reduce memory access and keep the ALU busy.

• Memory reuse. Forward prediction is a one-way calculation. We reuse

the memory between the input and output of different operators, thus reducing the overall memory overhead.

• Assembly level optimization. Saber is a underlying DNN library for Anakin, which

is deeply optimized at assembly level.

NV GPU Benchmark

Machine And Enviornment

> CPU: Intel(R) Xeon(R) CPU 5117 @ 2.0GHz > GPU: Tesla P4 > cuda: CUDA8 > cuDNN: v7

• Time：warmup 10，running 1000 times to get average time
• Latency (ms) and Memory(MB) of different batch

> The counterpart of `Anakin` is the acknowledged high performance inference engine `NVIDIA TensorRT 5` , The models which TensorRT 5 doesn't support we use the custom plugins to support.

VGG16

| Batch_Size | RT latency FP32(ms) | Anakin2 Latency FP32 (ms) |RT Memory (MB) | Anakin2 Memory (MB) | |------------|---------------------|---------------------------|---------------|---------------------| | 1 | 8.52532 | 8.2387 |1090.89 | 702 | | 2 | 14.1209 | 13.8772 |1056.02 | 768.76 | | 4 | 24.4529 | 24.3391 |1002.17 | 840.54 | | 8 | 46.7956 | 46.3309 |1098.98 | 935.61 |

Resnet50

| Batch_Size | RT latency FP32(ms) | Anakin2 Latency FP32 (ms) | RT Latency INT8 (ms) | Anakin2 Latency INT8 (ms) | RT Memory FP32(MB) | Anakin2 Memory FP32(MB) | |------------|---------------------|---------------------------|----------------------|---------------------------|--------------------|-------------------------| | 1 | 4.6447 | 3.0863 | 1.78892 | 1.61537 | 1134.88 | 311.25 | | 2 | 6.69187 | 5.13995 | 2.71136 | 2.70022 | 1108.86 | 382 | | 4 | 11.1943 | 9.20513 | 4.16771 | 4.77145 | 885.96 | 406.86 | | 8 | 19.8769 | 17.1976 | 6.2798 | 8.68197 | 813.84 | 532.61 |

Resnet101

| Batch_Size | RT latency (ms) | Anakin2 Latency (ms) | RT Latency INT8 (ms) | Anakin2 Latency INT8 (ms) | RT Memory (MB) | Anakin2 Memory (MB) | |------------|-----------------|----------------------|----------------------|---------------------------|----------------|---------------------| | 1 | 9.98695 | 5.44947 | 2.81031 | 2.74399 | 1159.16 | 500.5 | | 2 | 17.3489 | 8.85699 | 4.8641 | 4.69473 | 1158.73 | 492 | | 4 | 20.6198 | 16.8214 | 7.11608 | 8.45324 | 1021.68 | 541.08 | | 8 | 31.9653 | 33.5015 | 11.2403 | 15.4336 | 914.49 | 611.54 |

X86 CPU Benchmark

Machine And Enviornment

> CPU: Intel(R) Xeon(R) CPU E5-2650 v4 @ 2.20GHz with HT, for FP32 test > CPU: Intel(R) Xeon(R) Gold 6271 CPU @ 2.60GHz with HT, for INT8 test > System: CentOS 6.3 with GCC 4.8.2, for benchmark between Anakin and Intel Caffe

• All test enable 8 thread parallel
• Time：warmup 10，running 200 times to get average time

> The counterpart of `Anakin` is `Intel Cafe`(1.1.6) with mklml.

| Net_Name | Batch_Size | Anakin2 Latency(2650v4) fp32 (ms) | caffe Latency(2650v4) fp32 (ms) | Anakin2 Latency int8(6271) (ms) | |-------------|----|-------------------------------------|-----------------------------------|---------------------------------| | resnet50 | 1 | 20.6201 | 24.1369 | 3.20866 | | resnet50 | 2 | 39.2286 | 43.1096 | 5.44311 | | resnet50 | 4 | 77.1392 | 81.8814 | 9.93424 | | resnet50 | 8 | 152.941 | 158.321 | 19.5618 | | vgg16 | 1 | 55.6132 | 70.532 | 15.3181 | | vgg16 | 2 | 96.5034 | 131.451 | 22.5082 | | vgg16 | 4 | 180.479 | 247.926 | 37.2974 | | vgg16 | 8 | 346.619 | 485.44 | 67.6682 | | mobilenetv1 | 1 | 3.98104 | 5.42775 | 0.926546 | | mobilenetv1 | 2 | 7.27079 | 9.16058 | 1.35007 | | mobilenetv1 | 4 | 14.4029 | 16.2505 | 2.37271 | | mobilenetv1 | 8 | 29.1651 | 29.8381 | 3.75992 | | vgg16_ssd | 1 | 125.948 | 143.412 | | | vgg16_ssd | 2 | 247.242 | 266.22 | | | vgg16_ssd | 4 | 488.377 | 510.978 | | | vgg16_ssd | 8 | 972.762 | 995.407 | | | mobilenetv2 | 1 | 3.78504 | 23.0066 | | | mobilenetv2 | 2 | 7.24622 | 65.9301 | | | mobilenetv2 | 4 | 13.7638 | 85.3893 | | | mobilenetv2 | 8 | 28.4093 | 131.669 |

ARM CPU Benchmark

Machine And Enviornment

> CPU: Kirin 980 > CPU: Snapdragon 652 > CPU: Snapdragon 855 > CPU: RK3399

• Compile circumstance: Android ndk cross compile，gcc 4.9，enable neon
• Time：warmup 10，running 10 times to get average time
• Note: 1、shufflenetv2 int8 model add swish operator

> The counterpart of `Anakin` is `ncnn`(20190320). This benchmark we test ARMv7 ARMv8 splitly

ARMv8 TEST

• ABI： arm64-v8a
• Latency (ms) of one batch

| Kirin 980 | Anakin fp32 | | | Anakin int8 | | | NCNN fp32 | | | NCNN int8 | | | |---------------|-------------|----------|----------|-------------|----------|----------|-----------|----------|----------|-----------|----------|----------| | | 1 thread | 2 thread | 4 thread | 1 thread | 2 thread | 4 thread | 1 thread | 2 thread | 4 thread | 1 thread…