3D重建评估指标对比表
每个指标的具体代码位于文章末尾
| 指标 | 计算方法 | 数值范围 | 评估重点 | 优缺点 | 适用场景 |
|---|---|---|---|---|---|
| Chamfer Distance (C1) | 从预测网格到真实网格的平均距离 | [0, +∞) | 几何形状准确性 | 优点：直观、计算高效 缺点：对噪声敏感 | 整体形状评估 |
| Chamfer Distance (C2) | 从真实网格到预测网格的平均距离 | [0, +∞) | 几何形状完整性 | 优点：检测缺失部分 缺点：可能被异常值影响 | 完整性评估 |
| Normal Consistency | 对应点法向量点积的平均值 | [0, 1] | 表面细节质量 | 优点：反映表面光滑度 缺点：不关注几何形状 | 表面质量评估 |
| F-Score | 基于距离阈值的精确率/召回率调和平均 | [0, 100] | 高精度区域占比 | 优点：关注高精度区域 缺点：依赖阈值选择 | 精度评估 |
| Bounding Box IoU | 边界框交集体积/并集体积 | [0, 1] | 整体形状重叠度 | 优点：计算简单快速 缺点：忽略细节差异 | 粗略形状评估 |
详细指标说明
1. Chamfer Distance (C1 & C2)
# C1: 预测→真实
c1 = np.mean(dist_pred_to_gt) * 1000# C2: 真实→预测
c2 = np.mean(dist_gt_to_pred) * 1000
特点对比：
- C1：检测预测网格中多余的部分
- C2：检测预测网格中缺失的部分
- 理想情况：C1 ≈ C2，且都接近0
2. Normal Consistency
normal_consistency = np.mean(normal_pred_to_gt) + np.mean(normal_gt_to_pred)
评估维度：
- 表面光滑度：法向量变化是否平滑
- 细节保持：能否保持原始表面的细节特征
- 方向一致性：表面朝向是否一致
3. F-Score
tau = 1e-2 # 1cm阈值
prec_tau = (dist_pred_to_gt <= tau).mean() * 100
recall_tau = (dist_gt_to_pred <= tau).mean() * 100
fscore = (2 * prec_tau * recall_tau) / (prec_tau + recall_tau)
评估重点：
- 高精度区域：关注距离小于1cm的区域
- 平衡性：同时考虑精确率和召回率
- 实用性：反映实际应用中的可用性
4. Bounding Box IoU
iou = inter_vol / (vol1 + vol2 - inter_vol)
评估范围：
- 整体形状：不考虑内部细节
- 空间位置：反映整体定位准确性
- 尺度一致性：检测尺寸是否合理
指标组合使用建议
| 评估目标 | 推荐指标组合 | 原因 |
|---|---|---|
| 整体质量 | C1 + C2 + F-Score | 全面评估几何准确性 |
| 表面质量 | Normal Consistency | 专注表面细节 |
| 快速筛选 | Bounding Box IoU | 计算快速，适合大规模筛选 |
| 高精度应用 | F-Score | 关注高精度区域 |
| 研究对比 | 全部指标 | 提供全面的评估维度 |
实际应用中的选择
- 服装重建：重点关注C1、C2和Normal Consistency
- 快速原型：使用Bounding Box IoU进行初步筛选
- 生产应用：重点关注F-Score确保高精度
- 学术研究：使用全部指标进行综合评估
这些指标各有侧重，组合使用能够全面评估3D重建的质量。
import os
import torch
import scipy as sp
import numpy as np
import argparse
import trimeshfrom tqdm import tqdm def compute_iou_bbox(mesh, gt_mesh):mesh_bounds = mesh.boundsgt_mesh_bounds = gt_mesh.boundsxx1 = np.max([mesh_bounds[0, 0], gt_mesh_bounds[0, 0]])yy1 = np.max([mesh_bounds[0, 1], gt_mesh_bounds[0, 1]])zz1 = np.max([mesh_bounds[0, 2], gt_mesh_bounds[0, 2]])xx2 = np.min([mesh_bounds[1, 0], gt_mesh_bounds[1, 0]])yy2 = np.min([mesh_bounds[1, 1], gt_mesh_bounds[1, 1]])zz2 = np.min([mesh_bounds[1, 2], gt_mesh_bounds[1, 2]])vol1 = (mesh_bounds[1, 0] - mesh_bounds[0, 0]) * (mesh_bounds[1, 1] - mesh_bounds[0, 1]) * (mesh_bounds[1, 2] -mesh_bounds[0, 2])vol2 = (gt_mesh_bounds[1, 0] - gt_mesh_bounds[0, 0]) * (gt_mesh_bounds[1, 1] - gt_mesh_bounds[0, 1]) * (gt_mesh_bounds[1, 2] -gt_mesh_bounds[0, 2])inter_vol = np.max([0, xx2 - xx1]) * np.max([0, yy2 - yy1]) * np.max([0, zz2 - zz1])iou = inter_vol / (vol1 + vol2 - inter_vol + 1e-11)return ioudef calculate_iou(gt, prediction):intersection = torch.logical_and(gt, prediction)union = torch.logical_or(gt, prediction)return torch.sum(intersection) / torch.sum(union)def compute_surface_metrics(mesh_pred, mesh_gt):"""Compute surface metrics (chamfer distance and f-score) for one example.Args:mesh: trimesh.Trimesh, the mesh to evaluate.Returns:chamfer: float, chamfer distance.fscore: float, f-score."""# Chamfereval_points = 100000point_gt, idx_gt = mesh_gt.sample(eval_points, return_index=True)normal_gt = mesh_gt.face_normals[idx_gt]point_gt = point_gt.astype(np.float32)point_pred, idx_pred = mesh_pred.sample(eval_points, return_index=True)normal_pred = mesh_pred.face_normals[idx_pred]point_pred = point_pred.astype(np.float32)dist_pred_to_gt, normal_pred_to_gt = distance_field_helper(point_pred, point_gt, normal_pred, normal_gt)dist_gt_to_pred, normal_gt_to_pred = distance_field_helper(point_gt, point_pred, normal_gt, normal_pred)# TODO: subdivide by 2 following OccNet # https://github.com/autonomousvision/occupancy_networks/blob/406f79468fb8b57b3e76816aaa73b1915c53ad22/im2mesh/eval.py#L136chamfer_l1 = np.mean(dist_pred_to_gt) + np.mean(dist_gt_to_pred)c1 = np.mean(dist_pred_to_gt)c2 = np.mean(dist_gt_to_pred)normal_consistency = np.mean(normal_pred_to_gt) + np.mean(normal_gt_to_pred)# Fscoretau = 1e-2eps = 1e-6#dist_pred_to_gt = (dist_pred_to_gt**2)#dist_gt_to_pred = (dist_gt_to_pred**2)prec_tau = (dist_pred_to_gt <= tau).astype(np.float32).mean() * 100.recall_tau = (dist_gt_to_pred <= tau).astype(np.float32).mean() * 100.fscore = (2 * prec_tau * recall_tau) / max(prec_tau + recall_tau, eps)# Following the tradition to scale chamfer distance up by 10.return c1 * 1000., c2 * 1000., normal_consistency / 2., fscoredef distance_field_helper(source, target, normals_src=None, normals_tgt=None):target_kdtree = sp.spatial.cKDTree(target)distances, idx = target_kdtree.query(source, n_jobs=-1)if normals_src is not None and normals_tgt is not None:normals_src = \normals_src / np.linalg.norm(normals_src, axis=-1, keepdims=True)normals_tgt = \normals_tgt / np.linalg.norm(normals_tgt, axis=-1, keepdims=True)normals_dot_product = (normals_tgt[idx] * normals_src).sum(axis=-1)# Handle normals that point into wrong direction gracefully# (mostly due to mehtod not caring about this in generation)normals_dot_product = np.abs(normals_dot_product)else:normals_dot_product = np.array([np.nan] * source.shape[0], dtype=np.float32)return distances, normals_dot_productdef main(args):input_subfolder = [x for x in sorted(os.listdir(args.input_path)) if os.path.isdir(os.path.join(args.input_path, x))]gt_subfolder = [x for x in sorted(os.listdir(args.gt_path)) if os.path.isdir(os.path.join(args.gt_path, x))]eval_name = args.input_path.split('/')[-1]mean_c1_list = []mean_c2_list = []mean_fscore_list = []mean_normal_consistency_list = []iou_list = []for pred, gt in tqdm(zip(input_subfolder, gt_subfolder)):pred_path = [x for x in sorted(os.listdir(os.path.join(args.input_path, pred))) ifx.endswith('shoes.obj') and not x.startswith('init')and not x.startswith('.')]if len(pred_path) == 0:continuemesh_pred = trimesh.load(os.path.join(args.input_path, pred, pred_path[0]))gt_path = [x for x in sorted(os.listdir(os.path.join(args.gt_path, gt, 'clothing'))) if x.endswith('shoe.obj')and not x.startswith('.')][0]mesh_gt = trimesh.load(os.path.join(args.gt_path, gt, 'clothing', gt_path))pred_2_scan, scan_2_pred, normal_consistency, fscore = compute_surface_metrics(mesh_pred, mesh_gt)iou = compute_iou_bbox(mesh_pred, mesh_gt)#print('Chamfer: {:.3f}, {:.3f}, Normal Consistency: {:.3f}, Fscore: {:.3f}, IOU: {:.3f}'.format(pred_2_scan, scan_2_pred, normal_consistency, fscore, iou))#print((pred_2_scan + scan_2_pred) / 2.0)iou_list.append(iou)mean_c1_list.append(pred_2_scan)mean_c2_list.append(scan_2_pred)mean_fscore_list.append(fscore)mean_normal_consistency_list.append(normal_consistency)mean_c1 = np.mean(mean_c1_list)mean_c2 = np.mean(mean_c2_list)mean_fscore = np.mean(mean_fscore_list)mean_normal_consistency = np.mean(mean_normal_consistency_list)mean_iou = np.mean(iou_list)std_c1 = np.std(mean_c1_list)std_c2 = np.std(mean_c2_list)std_fscore = np.std(mean_fscore_list)std_normal_consistency = np.std(mean_normal_consistency_list)std_iou = np.std(iou_list)print('Mean Chamfer: {:.3f} ({:.3f}), {:.3f} ({:.3f}), Normal Consistency: {:.3f} ({:.3f}), Fscore: {:.3f} ({:.3f})'.format(mean_c1, std_c1, mean_c2, std_c2, mean_normal_consistency, std_normal_consistency, mean_fscore, std_fscore))print('{:.3f} ({:.3f}),{:.3f} ({:.3f}),{:.3f} ({:.3f}),{:.3f} ({:.3f}),{:.3f} ({:.3f})'.format(mean_c1, std_c1, mean_c2, std_c2, mean_normal_consistency, std_normal_consistency, mean_fscore, std_fscore, mean_iou, std_iou))print('{:.6f}, {:.6f}, {:.6f}, {:.6f}, {:.6f}'.format(mean_c1, mean_c2, mean_normal_consistency, mean_fscore, mean_iou))output_txt = eval_name + '.txt'out = np.stack([mean_c1_list, mean_c2_list, mean_normal_consistency_list, mean_fscore_list], axis=1)np.savetxt(output_txt, out, fmt='%.6f', delimiter=' ')
if __name__ == '__main__':parser = argparse.ArgumentParser()#parser.add_argument('-o', '--output_dir', required=True, help='Where to store the processed images and other data.')parser.add_argument('-i', '--input_path', required=True ,type=str)parser.add_argument('-g', '--gt_path', required=True ,type=str)main(parser.parse_args())
)
