PointPillars

PointPillars/README.md

@@ -0,0 +1,172 @@
+# 基于PointPillars的3D点云目标检测模型
+
+## 1 介绍
+
+  基于PointPillars的3D点云目标检测模型在晟腾芯片上进行目标检测，可对3D点云数据进行目标检测，并把检测结果输出。
+
+  项目主要流程为：首先获取KITTI数据集中四维点数据，之后将其转化为pillars数据，送入PFE模型进行推理，获取pillar feature数据，并经由Pillar Scatter转化为二维图像信息，然后将其送入RPN网络进行推理，获取预测候选框信息数据,最后结合anchor进行解码、NMS等操作获取最终预测框信息，并结合第三方库Mayavi进行3-D点云目标检测结果的可视化。
+
+### 1.1 支持的产品
+
+昇腾310(推理),昇腾200dk
+
+### 1.2 支持的版本
+
+本样例配套的CANN版本为[5.0.5](https://www.hiascend.com/software/cann/commercial)。支持的SDK版本为[2.0.4](https://www.hiascend.com/software/Mindx-sdk)。
+
+### 1.3 软件方案介绍
+
+表1.1 系统方案各子系统功能描述：
+
+| 序号 | 子系统         | 功能描述                                         |
+| ---- | -------------- | :----------------------------------------------- |
+| 1    | 数据输入       | 获取bin数据，从中提取4-D Point数据               |
+| 2    | pillar数据提取 | 用于将4-D Point数据转化为8-D Pillar数据          |
+| 3    | PFE模型推理    | 对Pillar数据进行推理，获取64-D特征数据           |
+| 4    | Scatter处理    | 处理PFE推理结果，将其转化为64通道的伪图像        |
+| 5    | RPN模型推理    | 对64通道伪图像数据进行推理，获取检测结果数据信息 |
+| 6    | 候选框处理     | 从候选框中获取最终预测框数据                     |
+
+### 1.4 代码目录结构与说明
+
+本工程名称为PointPillars，工程目录如下图所示：
+
+```
+PointPillars/
+├── image
+│   └── operations.jpg
+├── README.md
+├── pipeline
+│   ├── pfe.pipeline
+│   └── rpn.pipeline
+├── requirments
+│   └── requiements.txt
+├── src
+│   ├── eval.py
+│   ├── get_preds.py
+│   ├── infer.py
+│   └── point_to_pillars.py
+├── eval.sh
+├── infer.sh
+└── view.sh
+```
+
+### 1.5 技术实现流程图
+
+![](./image/operations.jpg)
+
+## 2 环境依赖
+
+| 软件名称            | 版本          |
+| ------------------- | ------------- |
+| MindX SDK           | 2.0.4         |
+| ubuntu              | 18.04.5 LTS   |
+| Ascend-CANN-toolkit | 5.0.5alpha001 |
+| python              | 3.7.10        |
+
+环境搭建可参考[200dk开发板环境搭建](https://gitee.com/ascend/docs-openmind/blob/master/guide/mindx/ascend_community_projects/tutorials/200dk%E5%BC%80%E5%8F%91%E6%9D%BF%E7%8E%AF%E5%A2%83%E6%90%AD%E5%BB%BA.md)
+
+在编译运行项目前，需要设置环境变量：
+1、根据自己安装的ascend-toolkit下的set_env.sh设置环境变量。
+
+```
+. ${SDK-path}/set_env.sh
+. ${ascend-toolkit-path}/set_env.sh
+```
+
+2、通过命令把全局日志级别设置为error级别
+
+```
+export ASCEND_GLOBAL_LOG_LEVEL=3
+```
+
+环境变量介绍
+
+```
+SDK-path: SDK mxVision 安装路径  
+ascend-toolkit-path: CANN 安装路径  
+```
+
+## 3 软件依赖
+
+推理中涉及到第三方软件依赖详见PointPillars/requirments/requirments.txt。
+
+其中，PyQt5、traits以及VTK为mayavi安装所需依赖，在安装时，可从[镜像网站](https://www.lfd.uci.edu/~gohlke/pythonlibs/)预先下载各whl包，之后***依次***安装，完成后可成功安装mayavi。
+
+注：安装软件及依赖后，本项目可完全在Atlas 200DK上运行，若使用者不便在开发板上安装，则可在本地完成配置后，将数据预处理、精度验证以及结果可视化的部分在本地进行。
+
+## 4 模型准备
+
+本项目中适用的模型是[Pointillars](https://arxiv.org/abs/1812.05784)模型，该模型源码可从[gitee仓](https://gitee.com/shy718/nutonomy_pointpillars?_from=gitee_search)中下载。
+下载后，按照[流程](https://gitee.com/shy718/nutonomy_pointpillars/blob/master/README.md)下载配置KITTI数据集，搭建训练环境进行模型的训练，并将tckpt模型转化为onnx模型，并将onnx模型存储在 `PointPillars/models/model_onnx/`目录下。
+之后使用ATC工具，将onnx模型转化为om模型，转化指令如下：
+
+```
+atc  --input_shape="pillar_x:1,1,12000,100;pillar_y:1,1,12000,100;pillar_z:1,1,12000,100;pillar_i:1,1,12000,100;num_points_per_pillar:1,12000;x_sub_shaped:1,1,12000,100;y_sub_shaped:1,1,12000,100;mask:1,1,12000,100" --input_fp16_nodes="pillar_x;pillar_y;pillar_z;pillar_i;num_points_per_pillar;x_sub_shaped;y_sub_shaped;mask" --check_report=/home/bronyale/modelzoo/pfe/Ascend310/network_analysis.report --input_format=NCHW --output="/home/bronyale/modelzoo/pfe/Ascend310/pfe" --soc_version=Ascend310 --framework=5 --model="PointPillars/model/model_onnx/pfe.onnx" 
+```
+
+```
+atc  --input_shape="input.1:1,64,496,432" --input_fp16_nodes="input.1" --check_report=/home/bronyale/modelzoo/rpn/Ascend310/network_analysis.report --input_format=NCHW --output="/home/bronyale/modelzoo/rpn/Ascend310/rpn" --soc_version=Ascend310 --framework=5 --model="PointPillars/model/model_onnx/rpn.onnx 
+```
+
+注：该操作步骤适用于手动训练模型并进行模型转换，除此之外，可通过接下来 `编译与运行`中 `步骤3`中的操作，直接下载训练后的onnx模型与转换后的om模型。
+
+## 5 编译与运行
+
+**步骤1** 按照第2小结**环境依赖**中的步骤设置环境变量。
+
+**步骤2** 按照第4小节 **模型获取** 中的步骤获取模型文件，把模型放置在 `./models/model_om/` 目录下。
+
+**步骤3** 执行模型推理。
+首先将推理所用的输入数据从[数据下载地址](https://mindx.sdk.obs.cn-north-4.myhuaweicloud.com/ascend_community_projects/pointpillar/data.zip)中下载，将推理所需模型从[模型下载地址](https://mindx.sdk.obs.cn-north-4.myhuaweicloud.com/ascend_community_projects/pointpillar/models.zip)中下载，将精度验证所需的标杆推理结果从[标杆下载地址](https://mindx.sdk.obs.cn-north-4.myhuaweicloud.com/ascend_community_projects/pointpillar/benchmark.rar)中下载，并将下载的目录置于PointPillars目录中，完成后目录结构如下所示：
+
+```
+PointPillars/
+├── benchmark
+│   └── test
+├── data
+│   └── test
+├── image
+├── models
+│   ├── model_om
+│   └── model_onnx
+├── README.md
+├── pipeline
+├── requirments
+├── src
+├── eval.sh
+├── infer.sh
+└── view.sh
+```
+
+之后在 `PointPillars` 目录下执行命令：
+
+```
+bash infer.sh
+```
+
+之后候选框信息将保存在 `PointPillars/data/test/`中。
+
+**步骤4**处理候选框信息及可视化。在 `PointPillars`目录下执行：
+
+```
+bash view.sh
+```
+
+之后检测结果将显示到控制台，检测结果的可视化将通过mayavi显示。
+
+注：由于不同操作系统的文件路径表示方法不同，使用者需根据自身情况修改view.sh以及get_preds.py中的文件路径。
+
+## 6 精度测试
+
+进入 `PointPillars` 目录下执行命令：
+
+```
+bash eval.sh
+```
+
+执行后om模型相比于原模型推理结果的精度损失将显示到控制台。  
+
+## 7 适用场景
+
+PointPillars网络模型用于3-D点云数据的目标检测，其在检测过程中衡了检测速度与精度，因此适用于自动驾驶领域，用于实时检测目前场景中需要进行避障的目标物体。

PointPillars/eval.sh

@@ -0,0 +1,20 @@
+#!/bin/bash
+# Copyright(C) 2022. Huawei Technologies Co.,Ltd. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+set -e
+
+cd src/
+python eval.py evaluate --file_dir="../result/test/" --benchmark_dir="../benchmark/test/"
+
+exit 0

PointPillars/infer.sh

@@ -0,0 +1,33 @@
+#!/bin/bash
+# Copyright(C) 2022. Huawei Technologies Co.,Ltd. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+set -e
+
+out_path="result/"
+if [ -d "$out_path" ]; then
+    rm -rf "$out_path"
+else
+    echo "file $out_path is not exist."
+fi
+
+mkdir -p "$out_path"
+cd result
+mkdir test
+cd ..
+
+cd src/
+python point_to_pillars.py generate --file_dir="../data/test/"
+python infer.py infer --file_dir="../data/test/"
+
+exit 0

PointPillars/pipeline/pfe.pipeline

@@ -0,0 +1,77 @@
+{
+    "pfe": {
+        "stream_config": {
+            "deviceId": "0"
+        },
+        "appsrc0": {
+            "props": {
+                "blocksize": "1200000"
+            },
+           "factory": "appsrc",
+           "next": "mxpi_tensorinfer0:0"
+        },
+        "appsrc1": {
+            "props": {
+                "blocksize": "1200000"
+            },
+           "factory": "appsrc",
+           "next": "mxpi_tensorinfer0:1"
+        },
+        "appsrc2": {
+            "props": {
+                "blocksize": "1200000"
+            },
+           "factory": "appsrc",
+           "next": "mxpi_tensorinfer0:2"
+        },
+        "appsrc3": {
+            "props": {
+                "blocksize": "1200000"
+            },
+           "factory": "appsrc",
+           "next": "mxpi_tensorinfer0:3"
+        },
+        "appsrc4": {
+            "props": {
+                "blocksize": "12000"
+            },
+           "factory": "appsrc",
+           "next": "mxpi_tensorinfer0:4"
+        },
+        "appsrc5": {
+            "props": {
+                "blocksize": "1200000"
+            },
+           "factory": "appsrc",
+           "next": "mxpi_tensorinfer0:5"
+        },
+        "appsrc6": {
+            "props": {
+                "blocksize": "1200000"
+            },
+           "factory": "appsrc",
+           "next": "mxpi_tensorinfer0:6"
+        },
+        "appsrc7": {
+            "props": {
+                "blocksize": "1200000"
+            },
+           "factory": "appsrc",
+           "next": "mxpi_tensorinfer0:7"
+        },
+        "mxpi_tensorinfer0": {
+            "props": {
+                "dataSource":"appsrc0,appsrc1,appsrc2,appsrc3,appsrc4,appsrc5,appsrc6,appsrc7",
+                "modelPath": "../models/model_om/pfe.om"
+            },
+            "factory": "mxpi_tensorinfer",
+            "next": "appsink0"
+        },
+        "appsink0": {
+            "props": {
+                "blocksize": "10000000000"
+            },
+            "factory": "appsink"
+        }
+    }
+}

PointPillars/pipeline/rpn.pipeline

@@ -0,0 +1,28 @@
+{
+    "rpn": {
+        "stream_config": {
+            "deviceId": "0"
+        },
+        "appsrc0": {
+            "props": {
+                "blocksize": "13713408"
+            },
+           "factory": "appsrc",
+           "next": "mxpi_tensorinfer0"
+        },
+        "mxpi_tensorinfer0": {
+            "props": {
+                "dataSource":"appsrc0",
+                "modelPath": "../models/model_om/rpn.om"
+            },
+            "factory": "mxpi_tensorinfer",
+            "next": "appsink0"
+        },
+        "appsink0": {
+            "props": {
+                "blocksize": "10000000000"
+            },
+            "factory": "appsink"
+        }
+    }
+}

PointPillars/requirments/requiements.txt

@@ -0,0 +1,7 @@
+fire
+numpy
+torch
+PyQt5
+traits
+VTK
+mayavi

PointPillars/src/eval.py

@@ -0,0 +1,38 @@
+"""
+# Copyright(C) 2022. Huawei Technologies Co.,Ltd. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+import numpy as np
+import fire
+
+
+def evaluate(file_dir="../result/test/", benchmark_dir="../benchmark/test/"):
+    om = np.fromfile(f"{file_dir}/result.bin", dtype=np.float32).reshape(-1, 7)
+    benchmark = np.fromfile(f"{benchmark_dir}/result.bin", dtype=np.float32).reshape(-1, 7)
+    cnt = om.shape
+    error = 0
+    for i in range(cnt[0]):
+        miss = 0
+        benchmark_sum = 0
+        for j in range(0, 3):
+            miss += abs(benchmark[i][j] - om[i][j])
+            benchmark_sum += abs(benchmark[i][j])
+
+        error = max(error, miss / benchmark_sum)
+   
+    print('the error of the model is :', error)
+
+
+if __name__ == '__main__':
+    fire.Fire()

PointPillars/src/get_preds.py

@@ -0,0 +1,319 @@
+"""
+# Copyright(C) 2022. Huawei Technologies Co.,Ltd. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+import os
+from math import cos, sin
+from mayavi import mlab
+import numpy as np
+import torch
+import fire
+
+
+def box_decode(box_encodings, anchors, encode_angle_to_vector=False, smooth_dim=False):
+    # need to convert box_encodings to z-bottom format
+    xa, ya, za, wa, la, ha, ra = np.split(anchors, 7, axis=-1)
+    if encode_angle_to_vector:
+        xt, yt, zt, wt, lt, ht, rtx, rty = np.split(box_encodings, 8, axis=-1)
+    else:
+        xt, yt, zt, wt, lt, ht, rt = np.split(box_encodings, 7, axis=-1)
+    za = za + ha / 2
+    diagonal = np.sqrt(la**2 + wa**2)
+    xg = xt * diagonal + xa
+    yg = yt * diagonal + ya
+
+    zg = zt * ha + za
+    if smooth_dim:
+        lg = (lt + 1) * la
+        wg = (wt + 1) * wa
+        hg = (ht + 1) * ha
+    else:
+        lg = np.exp(lt) * la
+        wg = np.exp(wt) * wa
+        hg = np.exp(ht) * ha
+    if encode_angle_to_vector:
+        rax = np.cos(ra)
+        ray = np.sin(ra)
+        rgx = rtx + rax
+        rgy = rty + ray
+        rg = np.arctan2(rgy, rgx)
+    else:
+        rg = rt + ra
+    zg = zg - hg / 2
+    return torch.Tensor(np.concatenate([xg, yg, zg, wg, lg, hg, rg], axis=-1))
+
+
+def nms_op_kernel(dets, thresh=0.01, eps=0.0):
+    x1 = dets[:, 0]
+    y1 = dets[:, 1]
+    x2 = dets[:, 2]
+    y2 = dets[:, 3]
+    scores = dets[:, 4].numpy()
+    areas = (x2 - x1 + eps) * (y2 - y1 + eps)
+    nms_order = scores.argsort()[::-1].astype(np.int32)
+    ndets = dets.shape[0]
+    suppressed = np.zeros((ndets), dtype=np.int32)
+    index_to_keep = []
+    for _i in range(ndets):
+        i = nms_order[_i] 
+        if suppressed[
+                i] == 1: 
+            continue
+        index_to_keep.append(i)
+        for _j in range(_i + 1, ndets):
+            j = nms_order[_j]
+            if suppressed[j] == 1:
+                continue
+            w = max(min(x2[i], x2[j]) - max(x1[i], x1[j]) + eps, 0.0)
+            h = max(min(y2[i], y2[j]) - max(y1[i], y1[j]) + eps, 0.0)
+            inter = w * h
+            ovr = inter / (areas[i] + areas[j] - inter)
+            if ovr >= thresh:
+                suppressed[j] = 1
+    return index_to_keep
+
+
+def nms_op(boxes, scores, pre_maxsize=None):
+    nms_order = scores.sort(0, descending=True)[1]
+
+    if pre_maxsize is not None:
+        nms_order = nms_order[:pre_maxsize]
+    boxes = boxes[nms_order].contiguous()
+    index_to_keep = nms_op_kernel(boxes)
+    return index_to_keep
+
+
+def limit_period(val, offset=0.5, period=np.pi):
+    limited_val = val - np.floor(val / period + offset) * period
+    return limited_val
+
+
+def generate_anchors(feature_size,
+                     anchor_range,
+                     sizes,
+                     rotations,
+                     dtype):
+    anchor_range = np.array(anchor_range, dtype)
+    z_centers = np.linspace(
+        (anchor_range[2] + anchor_range[5]) / 2, anchor_range[5], feature_size[0], dtype=dtype)
+    y_centers = np.linspace(
+        anchor_range[1], anchor_range[4], feature_size[1], dtype=dtype)
+    x_centers = np.linspace(
+        anchor_range[0], anchor_range[3], feature_size[2], dtype=dtype)
+    sizes = np.reshape(np.array(sizes, dtype=dtype), [-1, 3])
+    rotations = np.array(rotations, dtype=dtype)
+    rets = np.meshgrid(
+        x_centers, y_centers, z_centers, rotations, indexing='ij')
+    tile_shape = [1] * 5
+    tile_shape[-2] = int(sizes.shape[0])
+    length = len(rets)
+    for i in range(length):
+        rets[i] = np.tile(rets[i][..., np.newaxis, :], tile_shape)
+        rets[i] = rets[i][..., np.newaxis]
+    sizes = np.reshape(sizes, [1, 1, 1, -1, 1, 3])
+    tile_size_shape = list(rets[0].shape)
+    tile_size_shape[3] = 1
+    sizes = np.tile(sizes, tile_size_shape)
+    rets.insert(3, sizes)
+    ret = np.concatenate(rets, axis=-1)
+    return np.transpose(ret, [2, 1, 0, 3, 4, 5])
+
+
+def get_predict_result(bbox_cls_pred, bbox_pred, bbox_dir_cls_pred, anchors):
+    bbox_cls_pred = bbox_cls_pred.reshape(-1, 1)
+    bbox_pred = bbox_pred.reshape(-1, 7)
+    bbox_dir_cls_pred = bbox_dir_cls_pred.reshape(-1, 2)
+    anchors = anchors.reshape(-1, 7)
+    bbox_cls_pred = torch.sigmoid(bbox_cls_pred)
+    bbox_dir_cls_pred = torch.max(bbox_dir_cls_pred, dim=1)[1]
+
+    inds = bbox_cls_pred.max(1)[0].topk(100)[1]
+    bbox_cls_pred = bbox_cls_pred[inds]
+    bbox_pred = bbox_pred[inds]
+    bbox_dir_cls_pred = bbox_dir_cls_pred[inds]
+    anchors = anchors[inds]
+
+    bbox_pred = box_decode(bbox_pred, anchors)
+
+    bbox_2d_xy = bbox_pred[:, [0, 1]]
+    bbox_2d_wl = bbox_pred[:, [3, 4]]
+    bbox_pred2d = torch.cat([bbox_2d_xy - bbox_2d_wl / 2,
+                             bbox_2d_xy + bbox_2d_wl / 2,
+                             bbox_cls_pred], dim=-1)
+    ret_bboxes, ret_labels, ret_scores = [], [], []
+    for i in range(1):
+        cur_bbox_cls_pred = bbox_cls_pred[:, i]
+        score_inds = cur_bbox_cls_pred > 0.1
+        if score_inds.sum() == 0:
+            continue
+
+        cur_bbox_cls_pred = cur_bbox_cls_pred[score_inds]
+        cur_bbox_pred2d = bbox_pred2d[score_inds]
+        cur_bbox_pred = bbox_pred[score_inds]
+        cur_bbox_dir_cls_pred = bbox_dir_cls_pred[score_inds]
+        
+        keep_inds = nms_op(boxes=cur_bbox_pred2d, 
+                                scores=cur_bbox_cls_pred, 
+                                pre_maxsize=None)
+
+        cur_bbox_cls_pred = cur_bbox_cls_pred[keep_inds]
+        cur_bbox_pred = cur_bbox_pred[keep_inds]
+        cur_bbox_dir_cls_pred = cur_bbox_dir_cls_pred[keep_inds]
+        cur_bbox_pred[:, -1] = limit_period(cur_bbox_pred[:, -1].detach().cpu(), 1, np.pi).to(cur_bbox_pred)
+        cur_bbox_pred[:, -1] += (1 - cur_bbox_dir_cls_pred) * np.pi
+
+        ret_bboxes.append(cur_bbox_pred)
+        ret_labels.append(torch.zeros_like(cur_bbox_pred[:, 0], dtype=torch.long) + i)
+        ret_scores.append(cur_bbox_cls_pred)
+
+    if len(ret_bboxes) == 0:
+        return [], [], []
+    ret_bboxes = torch.cat(ret_bboxes, 0)
+    ret_labels = torch.cat(ret_labels, 0)
+    ret_scores = torch.cat(ret_scores, 0)
+    cnt = 0
+    for i in range(50):
+        cnt += 1
+        if ret_scores[i] < 0.6:
+            cnt -= 1
+            break
+    final_inds = ret_scores.topk(cnt)[1]
+    ret_bboxes = ret_bboxes[final_inds]
+    ret_labels = ret_labels[final_inds]
+    ret_scores = ret_scores[final_inds]
+    result = {
+        'lidar_bboxes': ret_bboxes.detach().cpu().numpy(),
+        'labels': ret_labels.detach().cpu().numpy(),
+        'scores': ret_scores.detach().cpu().numpy()
+    }
+    return result
+
+
+def get_box_points(box_list):
+    box_points_list = []
+    length = len(box_list)
+    for cnt in range(length):
+        box = box_list[cnt]
+        box_points = np.zeros((8, 3))
+        x = float(box[0])
+        y = float(box[1])
+        z = float(box[2])
+        width = float(box[3])
+        long = float(box[4])
+        depth = float(box[5])
+        theta = np.pi / 2 - box[6]
+        box_points[0] = [x + long / 2 * cos(theta) + width / 2 * sin(theta),
+                         y + long / 2 * sin(theta) - width / 2 * cos(theta),
+                         z + depth / 2]
+        box_points[3] = [x + long / 2 * cos(theta) + width / 2 * sin(theta),
+                         y + long / 2 * sin(theta) - width / 2 * cos(theta), 
+                         z - depth / 2]
+
+        box_points[1] = [x + long / 2 * cos(theta) - width / 2 * sin(theta), 
+                         y + width / 2 * cos(theta) + long / 2 * sin(theta),
+                         z + depth / 2]
+        box_points[2] = [x + long / 2 * cos(theta) - width / 2 * sin(theta),
+                         y + width / 2 * cos(theta) + long / 2 * sin(theta),
+                         z - depth / 2]
+
+        box_points[5] = [2 * x - (x + long / 2 * cos(theta) + width / 2 * sin(theta)),
+                         2 * y - (y + long / 2 * sin(theta) - width / 2 * cos(theta)), 
+                         z + depth / 2]
+        box_points[6] = [2 * x - (x + long / 2 * cos(theta) + width / 2 * sin(theta)),
+                         2 * y - (y + long / 2 * sin(theta) - width / 2 * cos(theta)), 
+                         z - depth / 2]
+
+        box_points[4] = [2 * x - (x + long / 2 * cos(theta) - width / 2 * sin(theta)),
+                         2 * y - (y + width / 2 * cos(theta) + long / 2 * sin(theta)),
+                         z + depth / 2]
+        box_points[7] = [2 * x - (x + long / 2 * cos(theta) - width / 2 * sin(theta)),
+                         2 * y - (y + width / 2 * cos(theta) + long / 2 * sin(theta)),
+                         z - depth / 2]
+
+        box_points_list.append(box_points)
+
+    return np.array(box_points_list)
+
+
+def draw_box(box_point_list):
+    length = len(box_point_list)
+    for cnt in range(length):
+        for k in range(0, 4):
+            box_point = box_point_list[cnt]
+            i, j = k, (k + 1) % 4
+            mlab.plot3d([box_point[i, 0], box_point[j, 0]],
+                        [box_point[i, 1], box_point[j, 1]],
+                        [box_point[i, 2], box_point[j, 2]])
+            i, j = k + 4, (k + 3) % 4 + 4
+            mlab.plot3d([box_point[i, 0], box_point[j, 0]],
+                        [box_point[i, 1], box_point[j, 1]],
+                        [box_point[i, 2], box_point[j, 2]])
+            i , j = k, k + 4
+            mlab.plot3d([box_point[i, 0], box_point[j, 0]],
+                        [box_point[i, 1], box_point[j, 1]],
+                        [box_point[i, 2], box_point[j, 2]])
+
+
+def point_show(points, box):
+    x = points[:, 0]
+    y = points[:, 1]
+    z = points[:, 2]
+    fig = mlab.figure(bgcolor=(0, 0, 0), size=(640, 360))
+    mlab.points3d(x, y, z,
+                        z,  # Values used for Color
+                        mode="point",
+                        colormap='spectral',
+                        figure=fig,
+                        )
+    box_point = get_box_points(box)
+    draw_box(box_point)
+    
+    mlab.show()
+
+
+def get_result(file_dir='../result/test/'):
+    anchors = torch.as_tensor(generate_anchors(feature_size=[1, 248, 216],
+                                               anchor_range = [0, -39.68, -3, 69.12, 39.68, 1],
+                                               sizes=[1.6, 3.9, 1.56],
+                                               rotations=[0, np.pi / 2],
+                                               dtype=np.float32).reshape((248, 216, 1, 2, 7)))
+    bbox_cls_pred = torch.as_tensor(np.fromfile(f"{file_dir}/cls.bin", dtype=np.float32)
+                         .reshape((1, 248, 216, 2)))
+    bbox_pred = torch.as_tensor(np.fromfile(f"{file_dir}/box.bin", dtype=np.float32)
+                     .reshape((1, 248, 216, 14)))
+    bbox_dir_cls_pred = torch.as_tensor(np.fromfile(f"{file_dir}/dir.bin", dtype=np.float32)
+                             .reshape((1, 248, 216, 4)))
+    result = get_predict_result(bbox_cls_pred, bbox_pred, bbox_dir_cls_pred, anchors)
+    return result
+
+
+def viewer(file_dir='../result/test/'):
+    if os.path.exists(file_dir):
+        file = file_dir + "point.bin"
+        if os.path.exists(file):
+            points = np.fromfile(file, dtype=np.float32).reshape([-1, 4])
+            result = get_result(file_dir)
+            print(result)
+            box = result['lidar_bboxes']
+            boxes = np.array(box)
+            boxes.tofile(f"{file_dir}/result.bin")
+            point_show(points, box)
+        else:
+            print(f"file : {file} does not exist")
+    else:
+        print(f"path : {file_dir} does not exist")
+
+
+if __name__ == '__main__':
+    fire.Fire()

PointPillars/src/infer.py

@@ -0,0 +1,422 @@
+"""
+# Copyright(C) 2022. Huawei Technologies Co.,Ltd. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+import math
+import datetime
+import fire
+import numpy as np
+import MxpiDataType_pb2 as MxpiDataType
+from StreamManagerApi import StreamManagerApi, StringVector, MxDataInput, InProtobufVector, MxProtobufIn
+
+
+def get_pseudo_image(pillar_feature, coors):
+    pseudo_image = np.zeros((1, 64, 496, 432))
+    for i in range(0, 12000):
+        x = math.ceil(coors[i, 0])
+        y = math.ceil(coors[i, 1])
+        for j in range(0, 64):
+            pseudo_image[0, j, y, x] = pillar_feature[0, j, i, 0]
+    return pseudo_image
+
+
+def infer(file_dir = "../data/test/"):
+    stream_manager_api = StreamManagerApi()
+    ret = stream_manager_api.InitManager()
+    if ret != 0:
+        print("Failed to init Stream manager, ret=%s" % str(ret))
+        exit()
+    path = b"../pipeline/pfe.pipeline"
+    ret = stream_manager_api.CreateMultipleStreamsFromFile(path)
+    if ret != 0:
+        print("Failed to create Stream, ret=%s" % str(ret))
+        exit()
+    stream_name = b'pfe'
+    
+    stream_manager_api_rpn = StreamManagerApi()
+    ret = stream_manager_api_rpn.InitManager()
+    if ret != 0:
+        print("Failed to init Stream manager, ret=%s" % str(ret))
+        exit()
+    path_rpn = b"../pipeline/rpn.pipeline"
+    ret = stream_manager_api_rpn.CreateMultipleStreamsFromFile(path_rpn)
+    if ret != 0:
+        print("Failed to create Stream, ret=%s" % str(ret))
+        exit()
+    stream_name_rpn = b'rpn'
+
+    # Get the pillar_x 
+    pillar_x = np.fromfile(f"{file_dir}/pillar_x.bin", dtype=np.float16)
+    pillar_x = pillar_x.astype(np.float16).reshape((1, 12000, 100))
+    pillar_x_tensor = pillar_x[None]
+    print("---------------PILLAR_X INFO--------------")
+    print(pillar_x_tensor.size)
+    print(pillar_x_tensor.shape)
+    pillar_x_panckage_list = MxpiDataType.MxpiTensorPackageList()
+    pillar_x_panckage = pillar_x_panckage_list.tensorPackageVec.add()
+    pillar_x_vec = pillar_x_panckage.tensorVec.add()
+
+    pillar_x_byte = pillar_x_tensor.tobytes()
+    pillar_x_input = MxDataInput()
+    pillar_x_input.data = pillar_x_byte
+
+    pillar_x_vec.deviceId = 0
+    pillar_x_vec.memType = 0
+    for i in pillar_x_tensor.shape:
+        pillar_x_vec.tensorShape.append(i)
+    pillar_x_vec.dataStr = pillar_x_input.data
+    pillar_x_vec.tensorDataSize = len(pillar_x_byte)
+
+    plugin_id_x = 0
+    key = "appsrc{}".format(plugin_id_x).encode('utf-8')
+    buffer_vec_x = InProtobufVector()
+    xbuf = MxProtobufIn()
+    xbuf.key = key
+    xbuf.type = b'MxTools.MxpiTensorPackageList'
+    xbuf.protobuf = pillar_x_panckage_list.SerializeToString()
+    buffer_vec_x.push_back(xbuf)
+
+    # Get the pillar_y
+    pillar_y = np.fromfile(f"{file_dir}/pillar_y.bin", dtype=np.float16)
+    pillar_y = pillar_y.astype(np.float16).reshape((1, 12000, 100))
+    pillar_y_tensor = pillar_y[None]
+    print("---------------PILLAR_Y INFO--------------")
+    print(pillar_y_tensor.size)
+    print(pillar_y_tensor.shape)
+    pillar_y_panckage_list = MxpiDataType.MxpiTensorPackageList()
+    pillar_y_panckage = pillar_y_panckage_list.tensorPackageVec.add()
+    pillar_y_vec = pillar_y_panckage.tensorVec.add()
+
+    pillar_y_byte = pillar_y_tensor.tobytes()
+    pillar_y_input = MxDataInput()
+    pillar_y_input.data = pillar_y_byte
+
+    pillar_y_vec.deviceId = 0
+    pillar_y_vec.memType = 0
+    for i in pillar_y_tensor.shape:
+        pillar_y_vec.tensorShape.append(i)
+    pillar_y_vec.dataStr = pillar_y_input.data
+    pillar_y_vec.tensorDataSize = len(pillar_y_byte)
+
+    plugin_id_y = 1
+    key = "appsrc{}".format(plugin_id_y).encode('utf-8')
+    buffer_vec_y = InProtobufVector()
+    ybuf = MxProtobufIn()
+    ybuf.key = key
+    ybuf.type = b'MxTools.MxpiTensorPackageList'
+    ybuf.protobuf = pillar_y_panckage_list.SerializeToString()
+    buffer_vec_y.push_back(ybuf)
+
+    # Get the pillar_z
+    pillar_z = np.fromfile(f"{file_dir}/pillar_z.bin", dtype=np.float16)
+    pillar_z = pillar_z.astype(np.float16).reshape((1, 12000, 100))
+    pillar_z_tensor = pillar_z[None]
+    print("---------------PILLAR_Z INFO--------------")
+    print(pillar_z_tensor.size)
+    print(pillar_z_tensor.shape)
+    pillar_z_panckage_list = MxpiDataType.MxpiTensorPackageList()
+    pillar_z_panckage = pillar_z_panckage_list.tensorPackageVec.add()
+    pillar_z_vec = pillar_z_panckage.tensorVec.add()
+
+    pillar_z_byte = pillar_z_tensor.tobytes()
+    pillar_z_input = MxDataInput()
+    pillar_z_input.data = pillar_z_byte
+
+    pillar_z_vec.deviceId = 0
+    pillar_z_vec.memType = 0
+    for i in pillar_z_tensor.shape:
+        pillar_z_vec.tensorShape.append(i)
+    pillar_z_vec.dataStr = pillar_z_input.data
+    pillar_z_vec.tensorDataSize = len(pillar_z_byte)
+
+    plugin_id_z = 2
+    key = "appsrc{}".format(plugin_id_z).encode('utf-8')
+    buffer_vec_z = InProtobufVector()
+    zbuf = MxProtobufIn()
+    zbuf.key = key
+    zbuf.type = b'MxTools.MxpiTensorPackageList'
+    zbuf.protobuf = pillar_z_panckage_list.SerializeToString()
+    buffer_vec_z.push_back(zbuf)
+
+    # Get the pillar_i
+    pillar_i = np.fromfile(f"{file_dir}/pillar_i.bin", dtype=np.float16)
+    pillar_i = pillar_i.astype(np.float16).reshape((1, 12000, 100))
+    pillar_i_tensor = pillar_i[None]
+    print("---------------PILLAR_I INFO--------------")
+    print(pillar_i_tensor.size)
+    print(pillar_i_tensor.shape)
+    pillar_i_panckage_list = MxpiDataType.MxpiTensorPackageList()
+    pillar_i_panckage = pillar_i_panckage_list.tensorPackageVec.add()
+    pillar_i_vec = pillar_i_panckage.tensorVec.add()
+
+    pillar_i_byte = pillar_i_tensor.tobytes()
+    pillar_i_input = MxDataInput()
+    pillar_i_input.data = pillar_i_byte
+
+    pillar_i_vec.deviceId = 0
+    pillar_i_vec.memType = 0
+    for i in pillar_i_tensor.shape:
+        pillar_i_vec.tensorShape.append(i)
+    pillar_i_vec.dataStr = pillar_i_input.data
+    pillar_i_vec.tensorDataSize = len(pillar_i_byte)
+
+    plugin_id_i = 3
+    key = "appsrc{}".format(plugin_id_i).encode('utf-8')
+    buffer_vec_i = InProtobufVector()
+    ibuf = MxProtobufIn()
+    ibuf.key = key
+    ibuf.type = b'MxTools.MxpiTensorPackageList'
+    ibuf.protobuf = pillar_i_panckage_list.SerializeToString()
+    buffer_vec_i.push_back(ibuf)
+
+    # Get the num_points_per_pillar
+    num_points_per_pillar = np.fromfile(f"{file_dir}/num_points_per_pillar.bin", dtype=np.float16)
+    num_points_per_pillar = num_points_per_pillar.astype(np.float16).reshape((12000,))
+    num_points_per_pillar_tensor = num_points_per_pillar[None]
+    print("---------------NUM INFO--------------")
+    print(num_points_per_pillar_tensor.size)
+    print(num_points_per_pillar_tensor.shape)
+    num_points_per_pillar_panckage_list = MxpiDataType.MxpiTensorPackageList()
+    num_points_per_pillar_panckage = num_points_per_pillar_panckage_list.tensorPackageVec.add()
+    num_points_per_pillar_vec = num_points_per_pillar_panckage.tensorVec.add()
+
+    num_points_per_pillar_byte = num_points_per_pillar_tensor.tobytes()
+    num_points_per_pillar_input = MxDataInput()
+    num_points_per_pillar_input.data = num_points_per_pillar_byte
+
+    num_points_per_pillar_vec.deviceId = 0
+    num_points_per_pillar_vec.memType = 0
+    for i in num_points_per_pillar_tensor.shape:
+        num_points_per_pillar_vec.tensorShape.append(i)
+    num_points_per_pillar_vec.dataStr = num_points_per_pillar_input.data
+    num_points_per_pillar_vec.tensorDataSize = len(num_points_per_pillar_byte)
+
+    plugin_id_num = 4
+    key = "appsrc{}".format(plugin_id_num).encode('utf-8')
+    buffer_vec_num = InProtobufVector()
+    numbuf = MxProtobufIn()
+    numbuf.key = key
+    numbuf.type = b'MxTools.MxpiTensorPackageList'
+    numbuf.protobuf = num_points_per_pillar_panckage_list.SerializeToString()
+    buffer_vec_num.push_back(numbuf)
+
+    # Get the x_sub
+    x_sub = np.fromfile(f"{file_dir}/x_sub_shaped.bin", dtype=np.float16)
+    x_sub = x_sub.astype(np.float16).reshape((1, 12000, 100))
+    x_sub_tensor = x_sub[None]
+    print("---------------X_SUB INFO--------------")
+    print(x_sub_tensor.size)
+    print(x_sub_tensor.shape)
+    x_sub_panckage_list = MxpiDataType.MxpiTensorPackageList()
+    x_sub_panckage = x_sub_panckage_list.tensorPackageVec.add()
+    x_sub_vec = x_sub_panckage.tensorVec.add()
+
+    x_sub_byte = x_sub_tensor.tobytes()
+    x_sub_input = MxDataInput()
+    x_sub_input.data = x_sub_byte
+
+    x_sub_vec.deviceId = 0
+    x_sub_vec.memType = 0
+    for i in x_sub_tensor.shape:
+        x_sub_vec.tensorShape.append(i)
+    x_sub_vec.dataStr = x_sub_input.data
+    x_sub_vec.tensorDataSize = len(x_sub_byte)
+
+    plugin_id_x_sub = 5
+    key = "appsrc{}".format(plugin_id_x_sub).encode('utf-8')
+    buffer_vec_x_sub = InProtobufVector()
+    x_sub_buf = MxProtobufIn()
+    x_sub_buf.key = key
+    x_sub_buf.type = b'MxTools.MxpiTensorPackageList'
+    x_sub_buf.protobuf = x_sub_panckage_list.SerializeToString()
+    buffer_vec_x_sub.push_back(x_sub_buf)
+
+    # Get the y_sub
+    y_sub = np.fromfile(f"{file_dir}/y_sub_shaped.bin", dtype=np.float16)
+    y_sub = y_sub.astype(np.float16).reshape((1, 12000, 100))
+    y_sub_tensor = y_sub[None]
+    print("---------------Y_SUB INFO--------------")
+    print(y_sub_tensor.size)
+    print(y_sub_tensor.shape)
+    y_sub_panckage_list = MxpiDataType.MxpiTensorPackageList()
+    y_sub_panckage = y_sub_panckage_list.tensorPackageVec.add()
+    y_sub_vec = y_sub_panckage.tensorVec.add()
+
+    y_sub_byte = y_sub_tensor.tobytes()
+    y_sub_input = MxDataInput()
+    y_sub_input.data = y_sub_byte
+
+    y_sub_vec.deviceId = 0
+    y_sub_vec.memType = 0
+    for i in y_sub_tensor.shape:
+        y_sub_vec.tensorShape.append(i)
+    y_sub_vec.dataStr = y_sub_input.data
+    y_sub_vec.tensorDataSize = len(y_sub_byte)
+
+    plugin_id_y_sub = 6
+    key = "appsrc{}".format(plugin_id_y_sub).encode('utf-8')
+    buffer_vec_y_sub = InProtobufVector()
+    y_sub_buf = MxProtobufIn()
+    y_sub_buf.key = key
+    y_sub_buf.type = b'MxTools.MxpiTensorPackageList'
+    y_sub_buf.protobuf = y_sub_panckage_list.SerializeToString()
+    buffer_vec_y_sub.push_back(y_sub_buf)
+
+    # Get the mask
+    mask = np.fromfile(f"{file_dir}/mask.bin", dtype=np.float16)
+    mask = mask.astype(np.float16).reshape((1, 12000, 100))
+    mask_tensor = mask[None]
+    print("---------------MASK INFO--------------")
+    print(mask_tensor.size)
+    print(mask_tensor.shape)
+    mask_panckage_list = MxpiDataType.MxpiTensorPackageList()
+    mask_panckage = mask_panckage_list.tensorPackageVec.add()
+    mask_vec = mask_panckage.tensorVec.add()
+
+    mask_byte = mask_tensor.tobytes()
+    mask_input = MxDataInput()
+    mask_input.data = mask_byte
+
+    mask_vec.deviceId = 0
+    mask_vec.memType = 0
+    for i in mask_tensor.shape:
+        mask_vec.tensorShape.append(i)
+    mask_vec.dataStr = mask_input.data
+    mask_vec.tensorDataSize = len(mask_byte)
+
+    plugin_id_mask = 7
+    key = "appsrc{}".format(plugin_id_mask).encode('utf-8')
+    buffer_vec_mask = InProtobufVector()
+    mask_buf = MxProtobufIn()
+    mask_buf.key = key
+    mask_buf.type = b'MxTools.MxpiTensorPackageList'
+    mask_buf.protobuf = mask_panckage_list.SerializeToString()
+    buffer_vec_mask.push_back(mask_buf)
+
+    # Send data to the stream
+    unique_id_x = stream_manager_api.SendProtobuf(stream_name, plugin_id_x, buffer_vec_x)
+    unique_id_y = stream_manager_api.SendProtobuf(stream_name, plugin_id_y, buffer_vec_y)
+    unique_id_z = stream_manager_api.SendProtobuf(stream_name, plugin_id_z, buffer_vec_z)
+    unique_id_i = stream_manager_api.SendProtobuf(stream_name, plugin_id_i, buffer_vec_i)
+    unique_id_num = stream_manager_api.SendProtobuf(stream_name, plugin_id_num, buffer_vec_num)
+    unique_id_x_sub = stream_manager_api.SendProtobuf(stream_name, plugin_id_x_sub, buffer_vec_x_sub)
+    unique_id_y_sub = stream_manager_api.SendProtobuf(stream_name, plugin_id_y_sub, buffer_vec_y_sub)
+    unique_id_mask = stream_manager_api.SendProtobuf(stream_name, plugin_id_mask, buffer_vec_mask)
+    begin_time = datetime.datetime.now()
+    if unique_id_x < 0 or unique_id_y < 0 or unique_id_z < 0 or unique_id_i < 0 \
+       or unique_id_num < 0 or unique_id_x_sub < 0 or unique_id_y_sub < 0 or unique_id_mask < 0:
+        print("Failed to send data to stream.")
+        exit()
+
+    key_vec = StringVector()
+    key_vec.push_back(b'mxpi_tensorinfer0')
+    # get inference result
+    get_result = stream_manager_api.GetResult(stream_name, b'appsink0', key_vec)
+    spend_time = (datetime.datetime.now() - begin_time).total_seconds()
+    if get_result.errorCode != 0:
+        print("ERROR")
+        exit()
+    print("-----------Result---------------")
+    print(get_result)
+
+    infer_result = get_result.metadataVec[0]
+
+    result = MxpiDataType.MxpiTensorPackageList()
+    result.ParseFromString(infer_result.serializedMetadata)
+    result.tensorPackageVec[0].tensorVec[0].dataStr
+    result_np = np.frombuffer(result.tensorPackageVec[0].tensorVec[0].dataStr, dtype = np.float32)
+    result_np.tofile(f"{file_dir}/feature.bin")
+
+    # Pillar Scatter
+    pillar_feature = np.fromfile(f"{file_dir}/feature.bin", dtype=np.float32)
+    pillar_feature = pillar_feature.astype(np.float16).reshape((1, 64, 12000, 1))
+    print(pillar_feature.shape)
+    coors = np.load(f"{file_dir}/coor.npy")
+    print(coors.shape) 
+    pseudo_image = get_pseudo_image(pillar_feature, coors).astype(np.float16)
+    print(pseudo_image.shape)
+    pseudo_image.tofile(f"{file_dir}/pseudo_image.bin")
+
+    # Get the pseudo image
+    pseudo_image = np.fromfile(f"{file_dir}/pseudo_image.bin", dtype=np.float16)
+    pseudo_image = pseudo_image.astype(np.float32).reshape((64, 496, 432))
+    pseudo_image_tensor = pseudo_image[None]
+    print("---------------PSEUDO IMAGE INFO--------------")
+    print(pseudo_image_tensor.size)
+    print(pseudo_image_tensor.shape)
+    pseudo_image_panckage_list = MxpiDataType.MxpiTensorPackageList()
+    pseudo_image_panckage = pseudo_image_panckage_list.tensorPackageVec.add()
+    pseudo_image_vec = pseudo_image_panckage.tensorVec.add()
+
+    pseudo_image_byte = pseudo_image_tensor.tobytes()
+    pseudo_image_input = MxDataInput()
+    pseudo_image_input.data = pseudo_image_byte
+
+    pseudo_image_vec.deviceId = 0
+    pseudo_image_vec.memType = 0
+    for i in pseudo_image_tensor.shape:
+        pseudo_image_vec.tensorShape.append(i)
+    pseudo_image_vec.dataStr = pseudo_image_input.data
+    pseudo_image_vec.tensorDataSize = len(pseudo_image_byte)
+
+    plugin_id_pseudo_image = 0
+    key = "appsrc{}".format(plugin_id_pseudo_image).encode('utf-8')
+    buffer_vec_pseudo_image = InProtobufVector()
+    pseudo_image_buf = MxProtobufIn()
+    pseudo_image_buf.key = key
+    pseudo_image_buf.type = b'MxTools.MxpiTensorPackageList'
+    pseudo_image_buf.protobuf = pseudo_image_panckage_list.SerializeToString()
+    buffer_vec_pseudo_image.push_back(pseudo_image_buf)
+
+    # Send data to the stream
+    unique_id_pseudo_image = stream_manager_api_rpn.\
+        SendProtobuf(stream_name_rpn, plugin_id_pseudo_image, buffer_vec_pseudo_image)
+    begin_time = datetime.datetime.now()
+    if unique_id_pseudo_image < 0:
+        print("Failed to send data to stream.")
+        exit()
+
+
+    key_vec = StringVector()
+    key_vec.push_back(b'mxpi_tensorinfer0')
+    # get inference result
+    get_result = stream_manager_api_rpn.GetResult(stream_name_rpn, b'appsink0', key_vec)
+    spend_time += (datetime.datetime.now() - begin_time).total_seconds()
+    if get_result.errorCode != 0:
+        print("ERROR")
+        exit()
+    print("-----------Result---------------")
+    infer_result = get_result.metadataVec[0]
+    result = MxpiDataType.MxpiTensorPackageList()
+    result.ParseFromString(infer_result.serializedMetadata)
+    result_box = result.tensorPackageVec[0].tensorVec[0].dataStr
+    result_cls = result.tensorPackageVec[0].tensorVec[1].dataStr
+    result_dir = result.tensorPackageVec[0].tensorVec[2].dataStr
+    result_shape0 = result.tensorPackageVec[0].tensorVec[0].tensorShape
+    result_shape1 = result.tensorPackageVec[0].tensorVec[1].tensorShape
+    result_shape2 = result.tensorPackageVec[0].tensorVec[2].tensorShape
+    print(result_shape0)
+    print(result_shape1)
+    print(result_shape2)
+    result_box_np = np.frombuffer(result_box, dtype = np.float32)
+    result_cls_np = np.frombuffer(result_cls, dtype = np.float32)
+    result_dir_np = np.frombuffer(result_dir, dtype = np.float32)
+    result_dir = "../result/test/"
+    result_box_np.tofile(f"{result_dir}/box.bin")
+    result_cls_np.tofile(f"{result_dir}/cls.bin")
+    result_dir_np.tofile(f"{result_dir}/dir.bin")
+    print("The total time consumed for model inference is : ", spend_time, "s")
+
+if __name__ == '__main__':
+    fire.Fire()

PointPillars/src/point_to_pillars.py

@@ -0,0 +1,197 @@
+"""
+# Copyright(C) 2022. Huawei Technologies Co.,Ltd. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+"""
+import os
+import struct
+import numpy as np
+import torch
+import fire
+
+
+def read_lidar_info(file_path):
+    size = os.path.getsize(file_path)
+    point_num = int(size / 16)
+    assert point_num * 16 == size
+
+    lidar_pt_list = np.zeros((point_num, 4))
+    with open(file_path, 'rb') as f:
+        for i in range(point_num * 4):
+            data = f.read(4)
+            val = struct.unpack('f', data)
+            row = int(i / 4)
+            col = i % 4
+            lidar_pt_list[row][col] = val[0]
+    return lidar_pt_list
+
+
+def points_to_voxel_kernel(points,
+                           voxel_size,
+                           coors_range,
+                           num_points_per_voxel,
+                           coor_to_voxelidx,
+                           voxels,
+                           coors,
+                           max_points=100,
+                           max_voxels=12000):
+    point_cnt = points.shape[0]
+    ndim = 3
+    grid_size = (coors_range[3:] - coors_range[:3]) / voxel_size
+    grid_size = np.round(grid_size, 0, grid_size).astype(np.int32)
+
+    coor = np.zeros(shape=(3, ), dtype=np.int32)
+    voxel_num = 0
+    failed = False
+    for i in range(point_cnt):
+        failed = False
+        for j in range(ndim):
+            c = np.floor((points[i, j] - coors_range[j]) / voxel_size[j])
+            if c < 0 or c >= grid_size[j]:
+                failed = True
+                break
+            coor[j] = c
+        if failed:
+            continue
+        voxelidx = coor_to_voxelidx[coor[0], coor[1], coor[2]]
+        if voxelidx == -1:
+            voxelidx = voxel_num
+            if voxel_num >= max_voxels:
+                break
+            voxel_num += 1
+            coor_to_voxelidx[coor[0], coor[1], coor[2]] = voxelidx
+            coors[voxelidx] = coor
+        num = num_points_per_voxel[voxelidx]
+        if num < max_points:
+            voxels[voxelidx, num] = points[i]
+            num_points_per_voxel[voxelidx] += 1
+    return voxel_num
+
+
+def points_to_voxel(points,
+                    voxel_size,   # (0.16, 0.16, 4.0)
+                    coors_range,  # (0.0, -39.68, -3.0, 69.12, 39.68, 1.0)
+                    max_points=100,
+                    max_voxels=12000):
+    if not isinstance(voxel_size, np.ndarray):
+        voxel_size = np.array(voxel_size, dtype=points.dtype)
+    if not isinstance(coors_range, np.ndarray):
+        coors_range = np.array(coors_range, dtype=points.dtype)
+    voxelmap_shape = (coors_range[3:] - coors_range[:3]) / voxel_size
+    voxelmap_shape = tuple(np.round(voxelmap_shape).astype(np.int32).tolist())
+    num_points_per_voxel = np.zeros(shape=(max_voxels, ), dtype=np.int32)
+    coor_to_voxelidx = -np.ones(shape=voxelmap_shape, dtype=np.int32)
+    voxels = np.zeros(
+        shape=(max_voxels, max_points, points.shape[-1]), dtype=points.dtype)
+    coors = np.zeros(shape=(max_voxels, 3), dtype=np.int32)
+    voxel_num = points_to_voxel_kernel(
+        points, voxel_size, coors_range, num_points_per_voxel,
+        coor_to_voxelidx, voxels, coors, max_points, max_voxels)
+
+    coors = coors[:voxel_num]
+    voxels = voxels[:voxel_num]
+    num_points_per_voxel = num_points_per_voxel[:voxel_num]
+    return voxels, coors, num_points_per_voxel
+
+
+def get_sub_shaped(coors):
+    x_sub = coors[:, 0] * 0.16 + 0.08
+    y_sub = coors[:, 1] * 0.16 - 39.6
+    x_sub_shaped = np.zeros((12000, 100))
+    y_sub_shaped = np.zeros((12000, 100))
+    for i in range(0, 100):
+        x_sub_shaped[:12000, i] = x_sub
+        y_sub_shaped[:12000, i] = y_sub
+    x_sub_shaped = torch.as_tensor(x_sub_shaped).unsqueeze(0).unsqueeze(0).numpy()
+    y_sub_shaped = torch.as_tensor(y_sub_shaped).unsqueeze(0).unsqueeze(0).numpy()
+    return x_sub_shaped, y_sub_shaped
+
+
+def pillar_expand(voxel):
+    pillar = np.zeros((12000, 100))
+    pillar_len = voxel.shape[0]
+    for i in range(0, 100):
+        pillar[:pillar_len, i] = voxel[:, i]
+    return pillar
+
+
+def cnt_expand(num_points_per_vexols):
+    cnt = np.zeros((12000))
+    cnt_len = num_points_per_vexols.shape[0]
+    cnt[:cnt_len] = num_points_per_vexols
+    return cnt
+
+
+def coors_expand(coor):
+    coors = np.zeros((12000, 3))
+    coors_len = coor.shape[0]
+    coors[:coors_len, :] = coor[:, :]
+    return coors
+
+
+def get_mask(actual_num_numpy, max_num, axis=0):
+    actual_num = torch.as_tensor(actual_num_numpy)
+    actual_num = torch.unsqueeze(actual_num, axis+1)
+    max_num_shape = [1] * len(actual_num.shape)
+    max_num_shape[axis+1] = -1
+    max_num = torch.arange(max_num, dtype=torch.int, device=actual_num.device).view(max_num_shape)
+    paddings_indicator = actual_num.int() > max_num
+    paddings_indicator = paddings_indicator.permute(0, 2, 1)
+    paddings_indicator = paddings_indicator.unsqueeze(1)
+    return paddings_indicator
+
+
+def generate(file_dir="../data/test/"):
+    point = read_lidar_info(f"{file_dir}/point.bin")
+    voxel_size = [0.16, 0.16, 4.0]
+    coors_range = [0.0, -39.68, -3.0, 69.12, 39.68, 1.0]
+    voxels, coor, num_points_per_vexols = points_to_voxel(point, voxel_size, coors_range)
+    coors = coors_expand(coor)
+    print(point.shape)
+    print(voxels.shape)
+    print(coors.shape)
+    print(num_points_per_vexols.shape)
+    print(voxels)
+    pillar_x = torch.as_tensor(pillar_expand(voxels[:, :, 0])).unsqueeze(0).unsqueeze(0).numpy().astype(np.float16)
+    pillar_y = torch.as_tensor(pillar_expand(voxels[:, :, 1])).unsqueeze(0).unsqueeze(0).numpy().astype(np.float16)
+    pillar_z = torch.as_tensor(pillar_expand(voxels[:, :, 2])).unsqueeze(0).unsqueeze(0).numpy().astype(np.float16)
+    pillar_i = torch.as_tensor(pillar_expand(voxels[:, :, 3])).unsqueeze(0).unsqueeze(0).numpy().astype(np.float16)
+    x_sub_shaped, y_sub_shaped = get_sub_shaped(coors)
+    x_sub_shaped = x_sub_shaped.astype(np.float16)
+    y_sub_shaped = y_sub_shaped.astype(np.float16)
+    num_points_per_pillar = torch.as_tensor(cnt_expand(num_points_per_vexols)).unsqueeze(0).numpy().astype(np.float16)
+    num_points_for_pillar = torch.as_tensor(pillar_x).size()[3]
+    mask = get_mask(num_points_per_pillar, num_points_for_pillar, axis=0).numpy().astype(np.float16)
+    print(pillar_x.shape)
+    print(pillar_y.shape)
+    print(pillar_z.shape)
+    print(pillar_i.shape)
+    print(x_sub_shaped.shape)
+    print(y_sub_shaped.shape)
+    print(num_points_per_pillar.shape)
+    print(mask.shape)
+
+    pillar_x.tofile(f"{file_dir}/pillar_x.bin")
+    pillar_y.tofile(f"{file_dir}/pillar_y.bin")
+    pillar_z.tofile(f"{file_dir}/pillar_z.bin")
+    pillar_i.tofile(f"{file_dir}/pillar_i.bin")
+    x_sub_shaped.tofile(f"{file_dir}/x_sub_shaped.bin")
+    y_sub_shaped.tofile(f"{file_dir}/y_sub_shaped.bin")
+    num_points_per_pillar.tofile(f"{file_dir}/num_points_per_pillar.bin")
+    mask.tofile(f"{file_dir}/mask.bin")
+
+    np.save(f"{file_dir}/coor.npy", coors)
+
+
+if __name__ == '__main__':
+    fire.Fire()

PointPillars/view.sh

@@ -0,0 +1,21 @@
+#!/bin/bash
+# Copyright(C) 2022. Huawei Technologies Co.,Ltd. All rights reserved.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+set -e
+
+cp data/test/point.bin result/test/
+cd src/
+python get_preds.py viewer --file_dir="../result/test/"
+
+exit 0