diff --git a/PointPillars/README.md b/PointPillars/README.md new file mode 100644 index 0000000..1218d4f --- /dev/null +++ b/PointPillars/README.md @@ -0,0 +1,172 @@ +# 基于PointPillars的3D点云目标检测模型 + +## 1 介绍 + + 基于PointPillars的3D点云目标检测模型在晟腾芯片上进行目标检测，可对3D点云数据进行目标检测，并把检测结果输出。 + + 项目主要流程为：首先获取KITTI数据集中四维点数据，之后将其转化为pillars数据，送入PFE模型进行推理，获取pillar feature数据，并经由Pillar Scatter转化为二维图像信息，然后将其送入RPN网络进行推理，获取预测候选框信息数据,最后结合anchor进行解码、NMS等操作获取最终预测框信息，并结合第三方库Mayavi进行3-D点云目标检测结果的可视化。 + +### 1.1 支持的产品 + +昇腾310(推理),昇腾200dk + +### 1.2 支持的版本 + +本样例配套的CANN版本为[5.0.5](https://www.hiascend.com/software/cann/commercial)。支持的SDK版本为[2.0.4](https://www.hiascend.com/software/Mindx-sdk)。 + +### 1.3 软件方案介绍 + +表1.1 系统方案各子系统功能描述： + +| 序号 | 子系统 | 功能描述 | +| ---- | -------------- | :----------------------------------------------- | +| 1 | 数据输入 | 获取bin数据，从中提取4-D Point数据 | +| 2 | pillar数据提取 | 用于将4-D Point数据转化为8-D Pillar数据 | +| 3 | PFE模型推理 | 对Pillar数据进行推理，获取64-D特征数据 | +| 4 | Scatter处理 | 处理PFE推理结果，将其转化为64通道的伪图像 | +| 5 | RPN模型推理 | 对64通道伪图像数据进行推理，获取检测结果数据信息 | +| 6 | 候选框处理 | 从候选框中获取最终预测框数据 | + +### 1.4 代码目录结构与说明 + +本工程名称为PointPillars，工程目录如下图所示： + +``` +PointPillars/ +├── image +│   └── operations.jpg +├── README.md +├── pipeline +│ ├── pfe.pipeline +│ └── rpn.pipeline +├── requirments +│   └── requiements.txt +├── src +│ ├── eval.py +│ ├── get_preds.py +│ ├── infer.py +│ └── point_to_pillars.py +├── eval.sh +├── infer.sh +└── view.sh +``` + +### 1.5 技术实现流程图 + +![](./image/operations.jpg) + +## 2 环境依赖 + +| 软件名称 | 版本 | +| ------------------- | ------------- | +| MindX SDK | 2.0.4 | +| ubuntu | 18.04.5 LTS | +| Ascend-CANN-toolkit | 5.0.5alpha001 | +| python | 3.7.10 | + +环境搭建可参考[200dk开发板环境搭建](https://gitee.com/ascend/docs-openmind/blob/master/guide/mindx/ascend_community_projects/tutorials/200dk%E5%BC%80%E5%8F%91%E6%9D%BF%E7%8E%AF%E5%A2%83%E6%90%AD%E5%BB%BA.md) + +在编译运行项目前，需要设置环境变量： +1、根据自己安装的ascend-toolkit下的set_env.sh设置环境变量。 + +``` +. ${SDK-path}/set_env.sh +. ${ascend-toolkit-path}/set_env.sh +``` + +2、通过命令把全局日志级别设置为error级别 + +``` +export ASCEND_GLOBAL_LOG_LEVEL=3 +``` + +环境变量介绍 + +``` +SDK-path: SDK mxVision 安装路径 +ascend-toolkit-path: CANN 安装路径 +``` + +## 3 软件依赖 + +推理中涉及到第三方软件依赖详见PointPillars/requirments/requirments.txt。 + +其中，PyQt5、traits以及VTK为mayavi安装所需依赖，在安装时，可从[镜像网站](https://www.lfd.uci.edu/~gohlke/pythonlibs/)预先下载各whl包，之后***依次***安装，完成后可成功安装mayavi。 + +注：安装软件及依赖后，本项目可完全在Atlas 200DK上运行，若使用者不便在开发板上安装，则可在本地完成配置后，将数据预处理、精度验证以及结果可视化的部分在本地进行。 + +## 4 模型准备 + +本项目中适用的模型是[Pointillars](https://arxiv.org/abs/1812.05784)模型，该模型源码可从[gitee仓](https://gitee.com/shy718/nutonomy_pointpillars?_from=gitee_search)中下载。 +下载后，按照[流程](https://gitee.com/shy718/nutonomy_pointpillars/blob/master/README.md)下载配置KITTI数据集，搭建训练环境进行模型的训练，并将tckpt模型转化为onnx模型，并将onnx模型存储在 `PointPillars/models/model_onnx/`目录下。 +之后使用ATC工具，将onnx模型转化为om模型，转化指令如下： + +``` +atc --input_shape="pillar_x:1,1,12000,100;pillar_y:1,1,12000,100;pillar_z:1,1,12000,100;pillar_i:1,1,12000,100;num_points_per_pillar:1,12000;x_sub_shaped:1,1,12000,100;y_sub_shaped:1,1,12000,100;mask:1,1,12000,100" --input_fp16_nodes="pillar_x;pillar_y;pillar_z;pillar_i;num_points_per_pillar;x_sub_shaped;y_sub_shaped;mask" --check_report=/home/bronyale/modelzoo/pfe/Ascend310/network_analysis.report --input_format=NCHW --output="/home/bronyale/modelzoo/pfe/Ascend310/pfe" --soc_version=Ascend310 --framework=5 --model="PointPillars/model/model_onnx/pfe.onnx" +``` + +``` +atc --input_shape="input.1:1,64,496,432" --input_fp16_nodes="input.1" --check_report=/home/bronyale/modelzoo/rpn/Ascend310/network_analysis.report --input_format=NCHW --output="/home/bronyale/modelzoo/rpn/Ascend310/rpn" --soc_version=Ascend310 --framework=5 --model="PointPillars/model/model_onnx/rpn.onnx +``` + +注：该操作步骤适用于手动训练模型并进行模型转换，除此之外，可通过接下来 `编译与运行`中 `步骤3`中的操作，直接下载训练后的onnx模型与转换后的om模型。 + +## 5 编译与运行 + +**步骤1** 按照第2小结**环境依赖**中的步骤设置环境变量。 + +**步骤2** 按照第4小节 **模型获取** 中的步骤获取模型文件，把模型放置在 `./models/model_om/` 目录下。 + +**步骤3** 执行模型推理。 +首先将推理所用的输入数据从[数据下载地址](https://mindx.sdk.obs.cn-north-4.myhuaweicloud.com/ascend_community_projects/pointpillar/data.zip)中下载，将推理所需模型从[模型下载地址](https://mindx.sdk.obs.cn-north-4.myhuaweicloud.com/ascend_community_projects/pointpillar/models.zip)中下载，将精度验证所需的标杆推理结果从[标杆下载地址](https://mindx.sdk.obs.cn-north-4.myhuaweicloud.com/ascend_community_projects/pointpillar/benchmark.rar)中下载，并将下载的目录置于PointPillars目录中，完成后目录结构如下所示： + +``` +PointPillars/ +├── benchmark +│ └── test +├── data +│ └── test +├── image +├── models +│ ├── model_om +│ └── model_onnx +├── README.md +├── pipeline +├── requirments +├── src +├── eval.sh +├── infer.sh +└── view.sh +``` + +之后在 `PointPillars` 目录下执行命令： + +``` +bash infer.sh +``` + +之后候选框信息将保存在 `PointPillars/data/test/`中。 + +**步骤4**处理候选框信息及可视化。在 `PointPillars`目录下执行： + +``` +bash view.sh +``` + +之后检测结果将显示到控制台，检测结果的可视化将通过mayavi显示。 + +注：由于不同操作系统的文件路径表示方法不同，使用者需根据自身情况修改view.sh以及get_preds.py中的文件路径。 + +## 6 精度测试 + +进入 `PointPillars` 目录下执行命令： + +``` +bash eval.sh +``` + +执行后om模型相比于原模型推理结果的精度损失将显示到控制台。 + +## 7 适用场景 + +PointPillars网络模型用于3-D点云数据的目标检测，其在检测过程中衡了检测速度与精度，因此适用于自动驾驶领域，用于实时检测目前场景中需要进行避障的目标物体。 diff --git a/PointPillars/eval.sh b/PointPillars/eval.sh new file mode 100644 index 0000000..dfde828 --- /dev/null +++ b/PointPillars/eval.sh @@ -0,0 +1,20 @@ +#!/bin/bash +# Copyright(C) 2022. Huawei Technologies Co.,Ltd. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +set -e + +cd src/ +python eval.py evaluate --file_dir="../result/test/" --benchmark_dir="../benchmark/test/" + +exit 0 diff --git a/PointPillars/image/operations.jpg b/PointPillars/image/operations.jpg new file mode 100644 index 0000000..d55ca51 Binary files /dev/null and b/PointPillars/image/operations.jpg differ diff --git a/PointPillars/infer.sh b/PointPillars/infer.sh new file mode 100644 index 0000000..f6d95e5 --- /dev/null +++ b/PointPillars/infer.sh @@ -0,0 +1,33 @@ +#!/bin/bash +# Copyright(C) 2022. Huawei Technologies Co.,Ltd. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +set -e + +out_path="result/" +if [ -d "$out_path" ]; then + rm -rf "$out_path" +else + echo "file $out_path is not exist." +fi + +mkdir -p "$out_path" +cd result +mkdir test +cd .. + +cd src/ +python point_to_pillars.py generate --file_dir="../data/test/" +python infer.py infer --file_dir="../data/test/" + +exit 0 diff --git a/PointPillars/pipeline/pfe.pipeline b/PointPillars/pipeline/pfe.pipeline new file mode 100644 index 0000000..3b174ec --- /dev/null +++ b/PointPillars/pipeline/pfe.pipeline @@ -0,0 +1,77 @@ +{ + "pfe": { + "stream_config": { + "deviceId": "0" + }, + "appsrc0": { + "props": { + "blocksize": "1200000" + }, + "factory": "appsrc", + "next": "mxpi_tensorinfer0:0" + }, + "appsrc1": { + "props": { + "blocksize": "1200000" + }, + "factory": "appsrc", + "next": "mxpi_tensorinfer0:1" + }, + "appsrc2": { + "props": { + "blocksize": "1200000" + }, + "factory": "appsrc", + "next": "mxpi_tensorinfer0:2" + }, + "appsrc3": { + "props": { + "blocksize": "1200000" + }, + "factory": "appsrc", + "next": "mxpi_tensorinfer0:3" + }, + "appsrc4": { + "props": { + "blocksize": "12000" + }, + "factory": "appsrc", + "next": "mxpi_tensorinfer0:4" + }, + "appsrc5": { + "props": { + "blocksize": "1200000" + }, + "factory": "appsrc", + "next": "mxpi_tensorinfer0:5" + }, + "appsrc6": { + "props": { + "blocksize": "1200000" + }, + "factory": "appsrc", + "next": "mxpi_tensorinfer0:6" + }, + "appsrc7": { + "props": { + "blocksize": "1200000" + }, + "factory": "appsrc", + "next": "mxpi_tensorinfer0:7" + }, + "mxpi_tensorinfer0": { + "props": { + "dataSource":"appsrc0,appsrc1,appsrc2,appsrc3,appsrc4,appsrc5,appsrc6,appsrc7", + "modelPath": "../models/model_om/pfe.om" + }, + "factory": "mxpi_tensorinfer", + "next": "appsink0" + }, + "appsink0": { + "props": { + "blocksize": "10000000000" + }, + "factory": "appsink" + } + } +} diff --git a/PointPillars/pipeline/rpn.pipeline b/PointPillars/pipeline/rpn.pipeline new file mode 100644 index 0000000..74c28a5 --- /dev/null +++ b/PointPillars/pipeline/rpn.pipeline @@ -0,0 +1,28 @@ +{ + "rpn": { + "stream_config": { + "deviceId": "0" + }, + "appsrc0": { + "props": { + "blocksize": "13713408" + }, + "factory": "appsrc", + "next": "mxpi_tensorinfer0" + }, + "mxpi_tensorinfer0": { + "props": { + "dataSource":"appsrc0", + "modelPath": "../models/model_om/rpn.om" + }, + "factory": "mxpi_tensorinfer", + "next": "appsink0" + }, + "appsink0": { + "props": { + "blocksize": "10000000000" + }, + "factory": "appsink" + } + } +} diff --git a/PointPillars/requirments/requiements.txt b/PointPillars/requirments/requiements.txt new file mode 100644 index 0000000..8ffb919 --- /dev/null +++ b/PointPillars/requirments/requiements.txt @@ -0,0 +1,7 @@ +fire +numpy +torch +PyQt5 +traits +VTK +mayavi diff --git a/PointPillars/src/eval.py b/PointPillars/src/eval.py new file mode 100644 index 0000000..1671980 --- /dev/null +++ b/PointPillars/src/eval.py @@ -0,0 +1,38 @@ +""" +# Copyright(C) 2022. Huawei Technologies Co.,Ltd. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +import numpy as np +import fire + + +def evaluate(file_dir="../result/test/", benchmark_dir="../benchmark/test/"): + om = np.fromfile(f"{file_dir}/result.bin", dtype=np.float32).reshape(-1, 7) + benchmark = np.fromfile(f"{benchmark_dir}/result.bin", dtype=np.float32).reshape(-1, 7) + cnt = om.shape + error = 0 + for i in range(cnt[0]): + miss = 0 + benchmark_sum = 0 + for j in range(0, 3): + miss += abs(benchmark[i][j] - om[i][j]) + benchmark_sum += abs(benchmark[i][j]) + + error = max(error, miss / benchmark_sum) + + print('the error of the model is :', error) + + +if __name__ == '__main__': + fire.Fire() diff --git a/PointPillars/src/get_preds.py b/PointPillars/src/get_preds.py new file mode 100644 index 0000000..8fd52c5 --- /dev/null +++ b/PointPillars/src/get_preds.py @@ -0,0 +1,319 @@ +""" +# Copyright(C) 2022. Huawei Technologies Co.,Ltd. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +import os +from math import cos, sin +from mayavi import mlab +import numpy as np +import torch +import fire + + +def box_decode(box_encodings, anchors, encode_angle_to_vector=False, smooth_dim=False): + # need to convert box_encodings to z-bottom format + xa, ya, za, wa, la, ha, ra = np.split(anchors, 7, axis=-1) + if encode_angle_to_vector: + xt, yt, zt, wt, lt, ht, rtx, rty = np.split(box_encodings, 8, axis=-1) + else: + xt, yt, zt, wt, lt, ht, rt = np.split(box_encodings, 7, axis=-1) + za = za + ha / 2 + diagonal = np.sqrt(la**2 + wa**2) + xg = xt * diagonal + xa + yg = yt * diagonal + ya + + zg = zt * ha + za + if smooth_dim: + lg = (lt + 1) * la + wg = (wt + 1) * wa + hg = (ht + 1) * ha + else: + lg = np.exp(lt) * la + wg = np.exp(wt) * wa + hg = np.exp(ht) * ha + if encode_angle_to_vector: + rax = np.cos(ra) + ray = np.sin(ra) + rgx = rtx + rax + rgy = rty + ray + rg = np.arctan2(rgy, rgx) + else: + rg = rt + ra + zg = zg - hg / 2 + return torch.Tensor(np.concatenate([xg, yg, zg, wg, lg, hg, rg], axis=-1)) + + +def nms_op_kernel(dets, thresh=0.01, eps=0.0): + x1 = dets[:, 0] + y1 = dets[:, 1] + x2 = dets[:, 2] + y2 = dets[:, 3] + scores = dets[:, 4].numpy() + areas = (x2 - x1 + eps) * (y2 - y1 + eps) + nms_order = scores.argsort()[::-1].astype(np.int32) + ndets = dets.shape[0] + suppressed = np.zeros((ndets), dtype=np.int32) + index_to_keep = [] + for _i in range(ndets): + i = nms_order[_i] + if suppressed[ + i] == 1: + continue + index_to_keep.append(i) + for _j in range(_i + 1, ndets): + j = nms_order[_j] + if suppressed[j] == 1: + continue + w = max(min(x2[i], x2[j]) - max(x1[i], x1[j]) + eps, 0.0) + h = max(min(y2[i], y2[j]) - max(y1[i], y1[j]) + eps, 0.0) + inter = w * h + ovr = inter / (areas[i] + areas[j] - inter) + if ovr >= thresh: + suppressed[j] = 1 + return index_to_keep + + +def nms_op(boxes, scores, pre_maxsize=None): + nms_order = scores.sort(0, descending=True)[1] + + if pre_maxsize is not None: + nms_order = nms_order[:pre_maxsize] + boxes = boxes[nms_order].contiguous() + index_to_keep = nms_op_kernel(boxes) + return index_to_keep + + +def limit_period(val, offset=0.5, period=np.pi): + limited_val = val - np.floor(val / period + offset) * period + return limited_val + + +def generate_anchors(feature_size, + anchor_range, + sizes, + rotations, + dtype): + anchor_range = np.array(anchor_range, dtype) + z_centers = np.linspace( + (anchor_range[2] + anchor_range[5]) / 2, anchor_range[5], feature_size[0], dtype=dtype) + y_centers = np.linspace( + anchor_range[1], anchor_range[4], feature_size[1], dtype=dtype) + x_centers = np.linspace( + anchor_range[0], anchor_range[3], feature_size[2], dtype=dtype) + sizes = np.reshape(np.array(sizes, dtype=dtype), [-1, 3]) + rotations = np.array(rotations, dtype=dtype) + rets = np.meshgrid( + x_centers, y_centers, z_centers, rotations, indexing='ij') + tile_shape = [1] * 5 + tile_shape[-2] = int(sizes.shape[0]) + length = len(rets) + for i in range(length): + rets[i] = np.tile(rets[i][..., np.newaxis, :], tile_shape) + rets[i] = rets[i][..., np.newaxis] + sizes = np.reshape(sizes, [1, 1, 1, -1, 1, 3]) + tile_size_shape = list(rets[0].shape) + tile_size_shape[3] = 1 + sizes = np.tile(sizes, tile_size_shape) + rets.insert(3, sizes) + ret = np.concatenate(rets, axis=-1) + return np.transpose(ret, [2, 1, 0, 3, 4, 5]) + + +def get_predict_result(bbox_cls_pred, bbox_pred, bbox_dir_cls_pred, anchors): + bbox_cls_pred = bbox_cls_pred.reshape(-1, 1) + bbox_pred = bbox_pred.reshape(-1, 7) + bbox_dir_cls_pred = bbox_dir_cls_pred.reshape(-1, 2) + anchors = anchors.reshape(-1, 7) + bbox_cls_pred = torch.sigmoid(bbox_cls_pred) + bbox_dir_cls_pred = torch.max(bbox_dir_cls_pred, dim=1)[1] + + inds = bbox_cls_pred.max(1)[0].topk(100)[1] + bbox_cls_pred = bbox_cls_pred[inds] + bbox_pred = bbox_pred[inds] + bbox_dir_cls_pred = bbox_dir_cls_pred[inds] + anchors = anchors[inds] + + bbox_pred = box_decode(bbox_pred, anchors) + + bbox_2d_xy = bbox_pred[:, [0, 1]] + bbox_2d_wl = bbox_pred[:, [3, 4]] + bbox_pred2d = torch.cat([bbox_2d_xy - bbox_2d_wl / 2, + bbox_2d_xy + bbox_2d_wl / 2, + bbox_cls_pred], dim=-1) + ret_bboxes, ret_labels, ret_scores = [], [], [] + for i in range(1): + cur_bbox_cls_pred = bbox_cls_pred[:, i] + score_inds = cur_bbox_cls_pred > 0.1 + if score_inds.sum() == 0: + continue + + cur_bbox_cls_pred = cur_bbox_cls_pred[score_inds] + cur_bbox_pred2d = bbox_pred2d[score_inds] + cur_bbox_pred = bbox_pred[score_inds] + cur_bbox_dir_cls_pred = bbox_dir_cls_pred[score_inds] + + keep_inds = nms_op(boxes=cur_bbox_pred2d, + scores=cur_bbox_cls_pred, + pre_maxsize=None) + + cur_bbox_cls_pred = cur_bbox_cls_pred[keep_inds] + cur_bbox_pred = cur_bbox_pred[keep_inds] + cur_bbox_dir_cls_pred = cur_bbox_dir_cls_pred[keep_inds] + cur_bbox_pred[:, -1] = limit_period(cur_bbox_pred[:, -1].detach().cpu(), 1, np.pi).to(cur_bbox_pred) + cur_bbox_pred[:, -1] += (1 - cur_bbox_dir_cls_pred) * np.pi + + ret_bboxes.append(cur_bbox_pred) + ret_labels.append(torch.zeros_like(cur_bbox_pred[:, 0], dtype=torch.long) + i) + ret_scores.append(cur_bbox_cls_pred) + + if len(ret_bboxes) == 0: + return [], [], [] + ret_bboxes = torch.cat(ret_bboxes, 0) + ret_labels = torch.cat(ret_labels, 0) + ret_scores = torch.cat(ret_scores, 0) + cnt = 0 + for i in range(50): + cnt += 1 + if ret_scores[i] < 0.6: + cnt -= 1 + break + final_inds = ret_scores.topk(cnt)[1] + ret_bboxes = ret_bboxes[final_inds] + ret_labels = ret_labels[final_inds] + ret_scores = ret_scores[final_inds] + result = { + 'lidar_bboxes': ret_bboxes.detach().cpu().numpy(), + 'labels': ret_labels.detach().cpu().numpy(), + 'scores': ret_scores.detach().cpu().numpy() + } + return result + + +def get_box_points(box_list): + box_points_list = [] + length = len(box_list) + for cnt in range(length): + box = box_list[cnt] + box_points = np.zeros((8, 3)) + x = float(box[0]) + y = float(box[1]) + z = float(box[2]) + width = float(box[3]) + long = float(box[4]) + depth = float(box[5]) + theta = np.pi / 2 - box[6] + box_points[0] = [x + long / 2 * cos(theta) + width / 2 * sin(theta), + y + long / 2 * sin(theta) - width / 2 * cos(theta), + z + depth / 2] + box_points[3] = [x + long / 2 * cos(theta) + width / 2 * sin(theta), + y + long / 2 * sin(theta) - width / 2 * cos(theta), + z - depth / 2] + + box_points[1] = [x + long / 2 * cos(theta) - width / 2 * sin(theta), + y + width / 2 * cos(theta) + long / 2 * sin(theta), + z + depth / 2] + box_points[2] = [x + long / 2 * cos(theta) - width / 2 * sin(theta), + y + width / 2 * cos(theta) + long / 2 * sin(theta), + z - depth / 2] + + box_points[5] = [2 * x - (x + long / 2 * cos(theta) + width / 2 * sin(theta)), + 2 * y - (y + long / 2 * sin(theta) - width / 2 * cos(theta)), + z + depth / 2] + box_points[6] = [2 * x - (x + long / 2 * cos(theta) + width / 2 * sin(theta)), + 2 * y - (y + long / 2 * sin(theta) - width / 2 * cos(theta)), + z - depth / 2] + + box_points[4] = [2 * x - (x + long / 2 * cos(theta) - width / 2 * sin(theta)), + 2 * y - (y + width / 2 * cos(theta) + long / 2 * sin(theta)), + z + depth / 2] + box_points[7] = [2 * x - (x + long / 2 * cos(theta) - width / 2 * sin(theta)), + 2 * y - (y + width / 2 * cos(theta) + long / 2 * sin(theta)), + z - depth / 2] + + box_points_list.append(box_points) + + return np.array(box_points_list) + + +def draw_box(box_point_list): + length = len(box_point_list) + for cnt in range(length): + for k in range(0, 4): + box_point = box_point_list[cnt] + i, j = k, (k + 1) % 4 + mlab.plot3d([box_point[i, 0], box_point[j, 0]], + [box_point[i, 1], box_point[j, 1]], + [box_point[i, 2], box_point[j, 2]]) + i, j = k + 4, (k + 3) % 4 + 4 + mlab.plot3d([box_point[i, 0], box_point[j, 0]], + [box_point[i, 1], box_point[j, 1]], + [box_point[i, 2], box_point[j, 2]]) + i , j = k, k + 4 + mlab.plot3d([box_point[i, 0], box_point[j, 0]], + [box_point[i, 1], box_point[j, 1]], + [box_point[i, 2], box_point[j, 2]]) + + +def point_show(points, box): + x = points[:, 0] + y = points[:, 1] + z = points[:, 2] + fig = mlab.figure(bgcolor=(0, 0, 0), size=(640, 360)) + mlab.points3d(x, y, z, + z, # Values used for Color + mode="point", + colormap='spectral', + figure=fig, + ) + box_point = get_box_points(box) + draw_box(box_point) + + mlab.show() + + +def get_result(file_dir='../result/test/'): + anchors = torch.as_tensor(generate_anchors(feature_size=[1, 248, 216], + anchor_range = [0, -39.68, -3, 69.12, 39.68, 1], + sizes=[1.6, 3.9, 1.56], + rotations=[0, np.pi / 2], + dtype=np.float32).reshape((248, 216, 1, 2, 7))) + bbox_cls_pred = torch.as_tensor(np.fromfile(f"{file_dir}/cls.bin", dtype=np.float32) + .reshape((1, 248, 216, 2))) + bbox_pred = torch.as_tensor(np.fromfile(f"{file_dir}/box.bin", dtype=np.float32) + .reshape((1, 248, 216, 14))) + bbox_dir_cls_pred = torch.as_tensor(np.fromfile(f"{file_dir}/dir.bin", dtype=np.float32) + .reshape((1, 248, 216, 4))) + result = get_predict_result(bbox_cls_pred, bbox_pred, bbox_dir_cls_pred, anchors) + return result + + +def viewer(file_dir='../result/test/'): + if os.path.exists(file_dir): + file = file_dir + "point.bin" + if os.path.exists(file): + points = np.fromfile(file, dtype=np.float32).reshape([-1, 4]) + result = get_result(file_dir) + print(result) + box = result['lidar_bboxes'] + boxes = np.array(box) + boxes.tofile(f"{file_dir}/result.bin") + point_show(points, box) + else: + print(f"file : {file} does not exist") + else: + print(f"path : {file_dir} does not exist") + + +if __name__ == '__main__': + fire.Fire() diff --git a/PointPillars/src/infer.py b/PointPillars/src/infer.py new file mode 100644 index 0000000..85a8281 --- /dev/null +++ b/PointPillars/src/infer.py @@ -0,0 +1,422 @@ +""" +# Copyright(C) 2022. Huawei Technologies Co.,Ltd. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +import math +import datetime +import fire +import numpy as np +import MxpiDataType_pb2 as MxpiDataType +from StreamManagerApi import StreamManagerApi, StringVector, MxDataInput, InProtobufVector, MxProtobufIn + + +def get_pseudo_image(pillar_feature, coors): + pseudo_image = np.zeros((1, 64, 496, 432)) + for i in range(0, 12000): + x = math.ceil(coors[i, 0]) + y = math.ceil(coors[i, 1]) + for j in range(0, 64): + pseudo_image[0, j, y, x] = pillar_feature[0, j, i, 0] + return pseudo_image + + +def infer(file_dir = "../data/test/"): + stream_manager_api = StreamManagerApi() + ret = stream_manager_api.InitManager() + if ret != 0: + print("Failed to init Stream manager, ret=%s" % str(ret)) + exit() + path = b"../pipeline/pfe.pipeline" + ret = stream_manager_api.CreateMultipleStreamsFromFile(path) + if ret != 0: + print("Failed to create Stream, ret=%s" % str(ret)) + exit() + stream_name = b'pfe' + + stream_manager_api_rpn = StreamManagerApi() + ret = stream_manager_api_rpn.InitManager() + if ret != 0: + print("Failed to init Stream manager, ret=%s" % str(ret)) + exit() + path_rpn = b"../pipeline/rpn.pipeline" + ret = stream_manager_api_rpn.CreateMultipleStreamsFromFile(path_rpn) + if ret != 0: + print("Failed to create Stream, ret=%s" % str(ret)) + exit() + stream_name_rpn = b'rpn' + + # Get the pillar_x + pillar_x = np.fromfile(f"{file_dir}/pillar_x.bin", dtype=np.float16) + pillar_x = pillar_x.astype(np.float16).reshape((1, 12000, 100)) + pillar_x_tensor = pillar_x[None] + print("---------------PILLAR_X INFO--------------") + print(pillar_x_tensor.size) + print(pillar_x_tensor.shape) + pillar_x_panckage_list = MxpiDataType.MxpiTensorPackageList() + pillar_x_panckage = pillar_x_panckage_list.tensorPackageVec.add() + pillar_x_vec = pillar_x_panckage.tensorVec.add() + + pillar_x_byte = pillar_x_tensor.tobytes() + pillar_x_input = MxDataInput() + pillar_x_input.data = pillar_x_byte + + pillar_x_vec.deviceId = 0 + pillar_x_vec.memType = 0 + for i in pillar_x_tensor.shape: + pillar_x_vec.tensorShape.append(i) + pillar_x_vec.dataStr = pillar_x_input.data + pillar_x_vec.tensorDataSize = len(pillar_x_byte) + + plugin_id_x = 0 + key = "appsrc{}".format(plugin_id_x).encode('utf-8') + buffer_vec_x = InProtobufVector() + xbuf = MxProtobufIn() + xbuf.key = key + xbuf.type = b'MxTools.MxpiTensorPackageList' + xbuf.protobuf = pillar_x_panckage_list.SerializeToString() + buffer_vec_x.push_back(xbuf) + + # Get the pillar_y + pillar_y = np.fromfile(f"{file_dir}/pillar_y.bin", dtype=np.float16) + pillar_y = pillar_y.astype(np.float16).reshape((1, 12000, 100)) + pillar_y_tensor = pillar_y[None] + print("---------------PILLAR_Y INFO--------------") + print(pillar_y_tensor.size) + print(pillar_y_tensor.shape) + pillar_y_panckage_list = MxpiDataType.MxpiTensorPackageList() + pillar_y_panckage = pillar_y_panckage_list.tensorPackageVec.add() + pillar_y_vec = pillar_y_panckage.tensorVec.add() + + pillar_y_byte = pillar_y_tensor.tobytes() + pillar_y_input = MxDataInput() + pillar_y_input.data = pillar_y_byte + + pillar_y_vec.deviceId = 0 + pillar_y_vec.memType = 0 + for i in pillar_y_tensor.shape: + pillar_y_vec.tensorShape.append(i) + pillar_y_vec.dataStr = pillar_y_input.data + pillar_y_vec.tensorDataSize = len(pillar_y_byte) + + plugin_id_y = 1 + key = "appsrc{}".format(plugin_id_y).encode('utf-8') + buffer_vec_y = InProtobufVector() + ybuf = MxProtobufIn() + ybuf.key = key + ybuf.type = b'MxTools.MxpiTensorPackageList' + ybuf.protobuf = pillar_y_panckage_list.SerializeToString() + buffer_vec_y.push_back(ybuf) + + # Get the pillar_z + pillar_z = np.fromfile(f"{file_dir}/pillar_z.bin", dtype=np.float16) + pillar_z = pillar_z.astype(np.float16).reshape((1, 12000, 100)) + pillar_z_tensor = pillar_z[None] + print("---------------PILLAR_Z INFO--------------") + print(pillar_z_tensor.size) + print(pillar_z_tensor.shape) + pillar_z_panckage_list = MxpiDataType.MxpiTensorPackageList() + pillar_z_panckage = pillar_z_panckage_list.tensorPackageVec.add() + pillar_z_vec = pillar_z_panckage.tensorVec.add() + + pillar_z_byte = pillar_z_tensor.tobytes() + pillar_z_input = MxDataInput() + pillar_z_input.data = pillar_z_byte + + pillar_z_vec.deviceId = 0 + pillar_z_vec.memType = 0 + for i in pillar_z_tensor.shape: + pillar_z_vec.tensorShape.append(i) + pillar_z_vec.dataStr = pillar_z_input.data + pillar_z_vec.tensorDataSize = len(pillar_z_byte) + + plugin_id_z = 2 + key = "appsrc{}".format(plugin_id_z).encode('utf-8') + buffer_vec_z = InProtobufVector() + zbuf = MxProtobufIn() + zbuf.key = key + zbuf.type = b'MxTools.MxpiTensorPackageList' + zbuf.protobuf = pillar_z_panckage_list.SerializeToString() + buffer_vec_z.push_back(zbuf) + + # Get the pillar_i + pillar_i = np.fromfile(f"{file_dir}/pillar_i.bin", dtype=np.float16) + pillar_i = pillar_i.astype(np.float16).reshape((1, 12000, 100)) + pillar_i_tensor = pillar_i[None] + print("---------------PILLAR_I INFO--------------") + print(pillar_i_tensor.size) + print(pillar_i_tensor.shape) + pillar_i_panckage_list = MxpiDataType.MxpiTensorPackageList() + pillar_i_panckage = pillar_i_panckage_list.tensorPackageVec.add() + pillar_i_vec = pillar_i_panckage.tensorVec.add() + + pillar_i_byte = pillar_i_tensor.tobytes() + pillar_i_input = MxDataInput() + pillar_i_input.data = pillar_i_byte + + pillar_i_vec.deviceId = 0 + pillar_i_vec.memType = 0 + for i in pillar_i_tensor.shape: + pillar_i_vec.tensorShape.append(i) + pillar_i_vec.dataStr = pillar_i_input.data + pillar_i_vec.tensorDataSize = len(pillar_i_byte) + + plugin_id_i = 3 + key = "appsrc{}".format(plugin_id_i).encode('utf-8') + buffer_vec_i = InProtobufVector() + ibuf = MxProtobufIn() + ibuf.key = key + ibuf.type = b'MxTools.MxpiTensorPackageList' + ibuf.protobuf = pillar_i_panckage_list.SerializeToString() + buffer_vec_i.push_back(ibuf) + + # Get the num_points_per_pillar + num_points_per_pillar = np.fromfile(f"{file_dir}/num_points_per_pillar.bin", dtype=np.float16) + num_points_per_pillar = num_points_per_pillar.astype(np.float16).reshape((12000,)) + num_points_per_pillar_tensor = num_points_per_pillar[None] + print("---------------NUM INFO--------------") + print(num_points_per_pillar_tensor.size) + print(num_points_per_pillar_tensor.shape) + num_points_per_pillar_panckage_list = MxpiDataType.MxpiTensorPackageList() + num_points_per_pillar_panckage = num_points_per_pillar_panckage_list.tensorPackageVec.add() + num_points_per_pillar_vec = num_points_per_pillar_panckage.tensorVec.add() + + num_points_per_pillar_byte = num_points_per_pillar_tensor.tobytes() + num_points_per_pillar_input = MxDataInput() + num_points_per_pillar_input.data = num_points_per_pillar_byte + + num_points_per_pillar_vec.deviceId = 0 + num_points_per_pillar_vec.memType = 0 + for i in num_points_per_pillar_tensor.shape: + num_points_per_pillar_vec.tensorShape.append(i) + num_points_per_pillar_vec.dataStr = num_points_per_pillar_input.data + num_points_per_pillar_vec.tensorDataSize = len(num_points_per_pillar_byte) + + plugin_id_num = 4 + key = "appsrc{}".format(plugin_id_num).encode('utf-8') + buffer_vec_num = InProtobufVector() + numbuf = MxProtobufIn() + numbuf.key = key + numbuf.type = b'MxTools.MxpiTensorPackageList' + numbuf.protobuf = num_points_per_pillar_panckage_list.SerializeToString() + buffer_vec_num.push_back(numbuf) + + # Get the x_sub + x_sub = np.fromfile(f"{file_dir}/x_sub_shaped.bin", dtype=np.float16) + x_sub = x_sub.astype(np.float16).reshape((1, 12000, 100)) + x_sub_tensor = x_sub[None] + print("---------------X_SUB INFO--------------") + print(x_sub_tensor.size) + print(x_sub_tensor.shape) + x_sub_panckage_list = MxpiDataType.MxpiTensorPackageList() + x_sub_panckage = x_sub_panckage_list.tensorPackageVec.add() + x_sub_vec = x_sub_panckage.tensorVec.add() + + x_sub_byte = x_sub_tensor.tobytes() + x_sub_input = MxDataInput() + x_sub_input.data = x_sub_byte + + x_sub_vec.deviceId = 0 + x_sub_vec.memType = 0 + for i in x_sub_tensor.shape: + x_sub_vec.tensorShape.append(i) + x_sub_vec.dataStr = x_sub_input.data + x_sub_vec.tensorDataSize = len(x_sub_byte) + + plugin_id_x_sub = 5 + key = "appsrc{}".format(plugin_id_x_sub).encode('utf-8') + buffer_vec_x_sub = InProtobufVector() + x_sub_buf = MxProtobufIn() + x_sub_buf.key = key + x_sub_buf.type = b'MxTools.MxpiTensorPackageList' + x_sub_buf.protobuf = x_sub_panckage_list.SerializeToString() + buffer_vec_x_sub.push_back(x_sub_buf) + + # Get the y_sub + y_sub = np.fromfile(f"{file_dir}/y_sub_shaped.bin", dtype=np.float16) + y_sub = y_sub.astype(np.float16).reshape((1, 12000, 100)) + y_sub_tensor = y_sub[None] + print("---------------Y_SUB INFO--------------") + print(y_sub_tensor.size) + print(y_sub_tensor.shape) + y_sub_panckage_list = MxpiDataType.MxpiTensorPackageList() + y_sub_panckage = y_sub_panckage_list.tensorPackageVec.add() + y_sub_vec = y_sub_panckage.tensorVec.add() + + y_sub_byte = y_sub_tensor.tobytes() + y_sub_input = MxDataInput() + y_sub_input.data = y_sub_byte + + y_sub_vec.deviceId = 0 + y_sub_vec.memType = 0 + for i in y_sub_tensor.shape: + y_sub_vec.tensorShape.append(i) + y_sub_vec.dataStr = y_sub_input.data + y_sub_vec.tensorDataSize = len(y_sub_byte) + + plugin_id_y_sub = 6 + key = "appsrc{}".format(plugin_id_y_sub).encode('utf-8') + buffer_vec_y_sub = InProtobufVector() + y_sub_buf = MxProtobufIn() + y_sub_buf.key = key + y_sub_buf.type = b'MxTools.MxpiTensorPackageList' + y_sub_buf.protobuf = y_sub_panckage_list.SerializeToString() + buffer_vec_y_sub.push_back(y_sub_buf) + + # Get the mask + mask = np.fromfile(f"{file_dir}/mask.bin", dtype=np.float16) + mask = mask.astype(np.float16).reshape((1, 12000, 100)) + mask_tensor = mask[None] + print("---------------MASK INFO--------------") + print(mask_tensor.size) + print(mask_tensor.shape) + mask_panckage_list = MxpiDataType.MxpiTensorPackageList() + mask_panckage = mask_panckage_list.tensorPackageVec.add() + mask_vec = mask_panckage.tensorVec.add() + + mask_byte = mask_tensor.tobytes() + mask_input = MxDataInput() + mask_input.data = mask_byte + + mask_vec.deviceId = 0 + mask_vec.memType = 0 + for i in mask_tensor.shape: + mask_vec.tensorShape.append(i) + mask_vec.dataStr = mask_input.data + mask_vec.tensorDataSize = len(mask_byte) + + plugin_id_mask = 7 + key = "appsrc{}".format(plugin_id_mask).encode('utf-8') + buffer_vec_mask = InProtobufVector() + mask_buf = MxProtobufIn() + mask_buf.key = key + mask_buf.type = b'MxTools.MxpiTensorPackageList' + mask_buf.protobuf = mask_panckage_list.SerializeToString() + buffer_vec_mask.push_back(mask_buf) + + # Send data to the stream + unique_id_x = stream_manager_api.SendProtobuf(stream_name, plugin_id_x, buffer_vec_x) + unique_id_y = stream_manager_api.SendProtobuf(stream_name, plugin_id_y, buffer_vec_y) + unique_id_z = stream_manager_api.SendProtobuf(stream_name, plugin_id_z, buffer_vec_z) + unique_id_i = stream_manager_api.SendProtobuf(stream_name, plugin_id_i, buffer_vec_i) + unique_id_num = stream_manager_api.SendProtobuf(stream_name, plugin_id_num, buffer_vec_num) + unique_id_x_sub = stream_manager_api.SendProtobuf(stream_name, plugin_id_x_sub, buffer_vec_x_sub) + unique_id_y_sub = stream_manager_api.SendProtobuf(stream_name, plugin_id_y_sub, buffer_vec_y_sub) + unique_id_mask = stream_manager_api.SendProtobuf(stream_name, plugin_id_mask, buffer_vec_mask) + begin_time = datetime.datetime.now() + if unique_id_x < 0 or unique_id_y < 0 or unique_id_z < 0 or unique_id_i < 0 \ + or unique_id_num < 0 or unique_id_x_sub < 0 or unique_id_y_sub < 0 or unique_id_mask < 0: + print("Failed to send data to stream.") + exit() + + key_vec = StringVector() + key_vec.push_back(b'mxpi_tensorinfer0') + # get inference result + get_result = stream_manager_api.GetResult(stream_name, b'appsink0', key_vec) + spend_time = (datetime.datetime.now() - begin_time).total_seconds() + if get_result.errorCode != 0: + print("ERROR") + exit() + print("-----------Result---------------") + print(get_result) + + infer_result = get_result.metadataVec[0] + + result = MxpiDataType.MxpiTensorPackageList() + result.ParseFromString(infer_result.serializedMetadata) + result.tensorPackageVec[0].tensorVec[0].dataStr + result_np = np.frombuffer(result.tensorPackageVec[0].tensorVec[0].dataStr, dtype = np.float32) + result_np.tofile(f"{file_dir}/feature.bin") + + # Pillar Scatter + pillar_feature = np.fromfile(f"{file_dir}/feature.bin", dtype=np.float32) + pillar_feature = pillar_feature.astype(np.float16).reshape((1, 64, 12000, 1)) + print(pillar_feature.shape) + coors = np.load(f"{file_dir}/coor.npy") + print(coors.shape) + pseudo_image = get_pseudo_image(pillar_feature, coors).astype(np.float16) + print(pseudo_image.shape) + pseudo_image.tofile(f"{file_dir}/pseudo_image.bin") + + # Get the pseudo image + pseudo_image = np.fromfile(f"{file_dir}/pseudo_image.bin", dtype=np.float16) + pseudo_image = pseudo_image.astype(np.float32).reshape((64, 496, 432)) + pseudo_image_tensor = pseudo_image[None] + print("---------------PSEUDO IMAGE INFO--------------") + print(pseudo_image_tensor.size) + print(pseudo_image_tensor.shape) + pseudo_image_panckage_list = MxpiDataType.MxpiTensorPackageList() + pseudo_image_panckage = pseudo_image_panckage_list.tensorPackageVec.add() + pseudo_image_vec = pseudo_image_panckage.tensorVec.add() + + pseudo_image_byte = pseudo_image_tensor.tobytes() + pseudo_image_input = MxDataInput() + pseudo_image_input.data = pseudo_image_byte + + pseudo_image_vec.deviceId = 0 + pseudo_image_vec.memType = 0 + for i in pseudo_image_tensor.shape: + pseudo_image_vec.tensorShape.append(i) + pseudo_image_vec.dataStr = pseudo_image_input.data + pseudo_image_vec.tensorDataSize = len(pseudo_image_byte) + + plugin_id_pseudo_image = 0 + key = "appsrc{}".format(plugin_id_pseudo_image).encode('utf-8') + buffer_vec_pseudo_image = InProtobufVector() + pseudo_image_buf = MxProtobufIn() + pseudo_image_buf.key = key + pseudo_image_buf.type = b'MxTools.MxpiTensorPackageList' + pseudo_image_buf.protobuf = pseudo_image_panckage_list.SerializeToString() + buffer_vec_pseudo_image.push_back(pseudo_image_buf) + + # Send data to the stream + unique_id_pseudo_image = stream_manager_api_rpn.\ + SendProtobuf(stream_name_rpn, plugin_id_pseudo_image, buffer_vec_pseudo_image) + begin_time = datetime.datetime.now() + if unique_id_pseudo_image < 0: + print("Failed to send data to stream.") + exit() + + + key_vec = StringVector() + key_vec.push_back(b'mxpi_tensorinfer0') + # get inference result + get_result = stream_manager_api_rpn.GetResult(stream_name_rpn, b'appsink0', key_vec) + spend_time += (datetime.datetime.now() - begin_time).total_seconds() + if get_result.errorCode != 0: + print("ERROR") + exit() + print("-----------Result---------------") + infer_result = get_result.metadataVec[0] + result = MxpiDataType.MxpiTensorPackageList() + result.ParseFromString(infer_result.serializedMetadata) + result_box = result.tensorPackageVec[0].tensorVec[0].dataStr + result_cls = result.tensorPackageVec[0].tensorVec[1].dataStr + result_dir = result.tensorPackageVec[0].tensorVec[2].dataStr + result_shape0 = result.tensorPackageVec[0].tensorVec[0].tensorShape + result_shape1 = result.tensorPackageVec[0].tensorVec[1].tensorShape + result_shape2 = result.tensorPackageVec[0].tensorVec[2].tensorShape + print(result_shape0) + print(result_shape1) + print(result_shape2) + result_box_np = np.frombuffer(result_box, dtype = np.float32) + result_cls_np = np.frombuffer(result_cls, dtype = np.float32) + result_dir_np = np.frombuffer(result_dir, dtype = np.float32) + result_dir = "../result/test/" + result_box_np.tofile(f"{result_dir}/box.bin") + result_cls_np.tofile(f"{result_dir}/cls.bin") + result_dir_np.tofile(f"{result_dir}/dir.bin") + print("The total time consumed for model inference is : ", spend_time, "s") + +if __name__ == '__main__': + fire.Fire() diff --git a/PointPillars/src/point_to_pillars.py b/PointPillars/src/point_to_pillars.py new file mode 100644 index 0000000..9056da1 --- /dev/null +++ b/PointPillars/src/point_to_pillars.py @@ -0,0 +1,197 @@ +""" +# Copyright(C) 2022. Huawei Technologies Co.,Ltd. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +""" +import os +import struct +import numpy as np +import torch +import fire + + +def read_lidar_info(file_path): + size = os.path.getsize(file_path) + point_num = int(size / 16) + assert point_num * 16 == size + + lidar_pt_list = np.zeros((point_num, 4)) + with open(file_path, 'rb') as f: + for i in range(point_num * 4): + data = f.read(4) + val = struct.unpack('f', data) + row = int(i / 4) + col = i % 4 + lidar_pt_list[row][col] = val[0] + return lidar_pt_list + + +def points_to_voxel_kernel(points, + voxel_size, + coors_range, + num_points_per_voxel, + coor_to_voxelidx, + voxels, + coors, + max_points=100, + max_voxels=12000): + point_cnt = points.shape[0] + ndim = 3 + grid_size = (coors_range[3:] - coors_range[:3]) / voxel_size + grid_size = np.round(grid_size, 0, grid_size).astype(np.int32) + + coor = np.zeros(shape=(3, ), dtype=np.int32) + voxel_num = 0 + failed = False + for i in range(point_cnt): + failed = False + for j in range(ndim): + c = np.floor((points[i, j] - coors_range[j]) / voxel_size[j]) + if c < 0 or c >= grid_size[j]: + failed = True + break + coor[j] = c + if failed: + continue + voxelidx = coor_to_voxelidx[coor[0], coor[1], coor[2]] + if voxelidx == -1: + voxelidx = voxel_num + if voxel_num >= max_voxels: + break + voxel_num += 1 + coor_to_voxelidx[coor[0], coor[1], coor[2]] = voxelidx + coors[voxelidx] = coor + num = num_points_per_voxel[voxelidx] + if num < max_points: + voxels[voxelidx, num] = points[i] + num_points_per_voxel[voxelidx] += 1 + return voxel_num + + +def points_to_voxel(points, + voxel_size, # (0.16, 0.16, 4.0) + coors_range, # (0.0, -39.68, -3.0, 69.12, 39.68, 1.0) + max_points=100, + max_voxels=12000): + if not isinstance(voxel_size, np.ndarray): + voxel_size = np.array(voxel_size, dtype=points.dtype) + if not isinstance(coors_range, np.ndarray): + coors_range = np.array(coors_range, dtype=points.dtype) + voxelmap_shape = (coors_range[3:] - coors_range[:3]) / voxel_size + voxelmap_shape = tuple(np.round(voxelmap_shape).astype(np.int32).tolist()) + num_points_per_voxel = np.zeros(shape=(max_voxels, ), dtype=np.int32) + coor_to_voxelidx = -np.ones(shape=voxelmap_shape, dtype=np.int32) + voxels = np.zeros( + shape=(max_voxels, max_points, points.shape[-1]), dtype=points.dtype) + coors = np.zeros(shape=(max_voxels, 3), dtype=np.int32) + voxel_num = points_to_voxel_kernel( + points, voxel_size, coors_range, num_points_per_voxel, + coor_to_voxelidx, voxels, coors, max_points, max_voxels) + + coors = coors[:voxel_num] + voxels = voxels[:voxel_num] + num_points_per_voxel = num_points_per_voxel[:voxel_num] + return voxels, coors, num_points_per_voxel + + +def get_sub_shaped(coors): + x_sub = coors[:, 0] * 0.16 + 0.08 + y_sub = coors[:, 1] * 0.16 - 39.6 + x_sub_shaped = np.zeros((12000, 100)) + y_sub_shaped = np.zeros((12000, 100)) + for i in range(0, 100): + x_sub_shaped[:12000, i] = x_sub + y_sub_shaped[:12000, i] = y_sub + x_sub_shaped = torch.as_tensor(x_sub_shaped).unsqueeze(0).unsqueeze(0).numpy() + y_sub_shaped = torch.as_tensor(y_sub_shaped).unsqueeze(0).unsqueeze(0).numpy() + return x_sub_shaped, y_sub_shaped + + +def pillar_expand(voxel): + pillar = np.zeros((12000, 100)) + pillar_len = voxel.shape[0] + for i in range(0, 100): + pillar[:pillar_len, i] = voxel[:, i] + return pillar + + +def cnt_expand(num_points_per_vexols): + cnt = np.zeros((12000)) + cnt_len = num_points_per_vexols.shape[0] + cnt[:cnt_len] = num_points_per_vexols + return cnt + + +def coors_expand(coor): + coors = np.zeros((12000, 3)) + coors_len = coor.shape[0] + coors[:coors_len, :] = coor[:, :] + return coors + + +def get_mask(actual_num_numpy, max_num, axis=0): + actual_num = torch.as_tensor(actual_num_numpy) + actual_num = torch.unsqueeze(actual_num, axis+1) + max_num_shape = [1] * len(actual_num.shape) + max_num_shape[axis+1] = -1 + max_num = torch.arange(max_num, dtype=torch.int, device=actual_num.device).view(max_num_shape) + paddings_indicator = actual_num.int() > max_num + paddings_indicator = paddings_indicator.permute(0, 2, 1) + paddings_indicator = paddings_indicator.unsqueeze(1) + return paddings_indicator + + +def generate(file_dir="../data/test/"): + point = read_lidar_info(f"{file_dir}/point.bin") + voxel_size = [0.16, 0.16, 4.0] + coors_range = [0.0, -39.68, -3.0, 69.12, 39.68, 1.0] + voxels, coor, num_points_per_vexols = points_to_voxel(point, voxel_size, coors_range) + coors = coors_expand(coor) + print(point.shape) + print(voxels.shape) + print(coors.shape) + print(num_points_per_vexols.shape) + print(voxels) + pillar_x = torch.as_tensor(pillar_expand(voxels[:, :, 0])).unsqueeze(0).unsqueeze(0).numpy().astype(np.float16) + pillar_y = torch.as_tensor(pillar_expand(voxels[:, :, 1])).unsqueeze(0).unsqueeze(0).numpy().astype(np.float16) + pillar_z = torch.as_tensor(pillar_expand(voxels[:, :, 2])).unsqueeze(0).unsqueeze(0).numpy().astype(np.float16) + pillar_i = torch.as_tensor(pillar_expand(voxels[:, :, 3])).unsqueeze(0).unsqueeze(0).numpy().astype(np.float16) + x_sub_shaped, y_sub_shaped = get_sub_shaped(coors) + x_sub_shaped = x_sub_shaped.astype(np.float16) + y_sub_shaped = y_sub_shaped.astype(np.float16) + num_points_per_pillar = torch.as_tensor(cnt_expand(num_points_per_vexols)).unsqueeze(0).numpy().astype(np.float16) + num_points_for_pillar = torch.as_tensor(pillar_x).size()[3] + mask = get_mask(num_points_per_pillar, num_points_for_pillar, axis=0).numpy().astype(np.float16) + print(pillar_x.shape) + print(pillar_y.shape) + print(pillar_z.shape) + print(pillar_i.shape) + print(x_sub_shaped.shape) + print(y_sub_shaped.shape) + print(num_points_per_pillar.shape) + print(mask.shape) + + pillar_x.tofile(f"{file_dir}/pillar_x.bin") + pillar_y.tofile(f"{file_dir}/pillar_y.bin") + pillar_z.tofile(f"{file_dir}/pillar_z.bin") + pillar_i.tofile(f"{file_dir}/pillar_i.bin") + x_sub_shaped.tofile(f"{file_dir}/x_sub_shaped.bin") + y_sub_shaped.tofile(f"{file_dir}/y_sub_shaped.bin") + num_points_per_pillar.tofile(f"{file_dir}/num_points_per_pillar.bin") + mask.tofile(f"{file_dir}/mask.bin") + + np.save(f"{file_dir}/coor.npy", coors) + + +if __name__ == '__main__': + fire.Fire() diff --git a/PointPillars/view.sh b/PointPillars/view.sh new file mode 100644 index 0000000..13d3196 --- /dev/null +++ b/PointPillars/view.sh @@ -0,0 +1,21 @@ +#!/bin/bash +# Copyright(C) 2022. Huawei Technologies Co.,Ltd. All rights reserved. +# +# Licensed under the Apache License, Version 2.0 (the "License"); +# you may not use this file except in compliance with the License. +# You may obtain a copy of the License at +# +# http://www.apache.org/licenses/LICENSE-2.0 +# +# Unless required by applicable law or agreed to in writing, software +# distributed under the License is distributed on an "AS IS" BASIS, +# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. +# See the License for the specific language governing permissions and +# limitations under the License. +set -e + +cp data/test/point.bin result/test/ +cd src/ +python get_preds.py viewer --file_dir="../result/test/" + +exit 0