此页面由 AI 自动翻译。查看英文原版
最后更新

本页目录

  • 如何放置
  • 输入和输出
  • 用法
  • 功能示例
  • 参考
Claude CodeCodexCursor
Luxonis 智能体工具包为你常用的编程智能体加载 Luxonis 上下文与技能,更快地基于 OAK 开发应用。
安装工具包

SpatialDetectionNetwork

最后更新
Supported on:RVC2RVC4
空间检测节点类似于 DetectionNetwork 和 SpatialLocationCalculator 的组合。

如何放置

Python

Python
1modelDescription = dai.NNModelDescription("yolov6-nano")
2pipeline = dai.Pipeline()
3spatialDetectionNetwork = pipeline.create(dai.node.SpatialDetectionNetwork).build(camRgb, stereo, modelDescription)

C++

C++
1dai::NNModelDescription modelDescription("yolov6-nano");
2dai::Pipeline pipeline;
3auto spatialDetectionNetwork = pipeline.create<dai::node::SpatialDetectionNetwork>()->build(camRgb, stereo, modelDescription);

输入和输出

配置空间检测

SpatialDetectionNetwork 节点的管道在以下示意图中描述:
空间检测节点本质上只是检测网络(DetectionNetwork)和 SpatialLocationCalculator 的抽象。它通过将每个检测到的对象的边界框链接到空间位置计算器来工作。过程如下:
检测网络负责检测输入帧中的对象。它输出一个检测到的对象列表,每个对象由一个边界框、标签和置信度分数表示。
深度图与输入帧对齐。这是必要的,因为 DetectionNetwork 在输入帧上操作,而 SpatialLocationCalculator 在深度图上操作。
来自网络的边界框被发送到 SpatialLocationCalculator 并根据 BoundingBoxScaleFactor 进行缩放。这样做是为了确保它包含整个对象。然后,边界框与深度一起用于计算对象的空间坐标。
  • X 和 Y 坐标取自边界框中心。它们根据与帧中心的偏移量和该点的深度计算。
  • 对于深度 (Z),考虑缩放边界框 (ROI) 内的每个像素。这为我们提供了一组深度值,然后将其平均以获得最终深度值。

平均方法

  • 平均值/均值:使用 ROI 的平均值进行计算。
  • 最小值:使用 ROI 内的最小值进行计算。
  • 最大值:使用 ROI 内的最大值进行计算。
  • 众数:使用 ROI 内最频繁的值进行计算。
  • 中位数:使用 ROI 内的中位数进行计算。
默认方法是中位数。

常见错误

大多数错误源于不正确的边界框重叠。缩放后的边界框可能包含背景的一部分,这会使深度计算产生偏差。
  • 细长物体(如杆子)的空间位置可能不准确,因为边界框中只有一小部分实际位于检测到的物体上。在这种情况下,如果可能,最好使用较小的 BoundingBoxScaleFactor。
  • 带孔的物体 - 环、圈等。为了获得正确的深度,边界框应包含整个物体。而不是使用中位数深度,请使用 MIN 深度方法将背景排除在计算之外。或者,可以设置深度阈值以在静态环境中忽略背景。

用法

Python

Python
1with dai.Pipeline() as p:
2    camRgb = p.create(dai.node.Camera).build(dai.CameraBoardSocket.CAM_A)
3    monoLeft = p.create(dai.node.Camera).build(dai.CameraBoardSocket.CAM_B)
4    monoRight = p.create(dai.node.Camera).build(dai.CameraBoardSocket.CAM_C)
5    stereo = p.create(dai.node.StereoDepth)
6    spatialDetectionNetwork = p.create(dai.node.SpatialDetectionNetwork).build(camRgb, stereo, modelDescription, fps=FPS)
7
8    spatialDetectionNetwork.input.setBlocking(False)
9    spatialDetectionNetwork.setBoundingBoxScaleFactor(0.5)
10    spatialDetectionNetwork.setDepthLowerThreshold(100)
11    spatialDetectionNetwork.setDepthUpperThreshold(5000)
12
13    labelMap = spatialDetectionNetwork.getClasses()

C++

C++
1dai::Pipeline pipeline;
2auto camRgb = pipeline.create<dai::node::Camera>()->build(dai::CameraBoardSocket::CAM_A);
3auto monoLeft = pipeline.create<dai::node::Camera>()->build(dai::CameraBoardSocket::CAM_B);
4auto monoRight = pipeline.create<dai::node::Camera>()->build(dai::CameraBoardSocket::CAM_C);
5auto stereo = pipeline.create<dai::node::StereoDepth>();
6auto spatialDetectionNetwork = pipeline.create<dai::node::SpatialDetectionNetwork>()->build(camRgb, stereo, modelDescription, FPS);
7spatialDetectionNetwork->input.setBlocking(false);
8spatialDetectionNetwork->setBoundingBoxScaleFactor(0.5);
9spatialDetectionNetwork->setDepthLowerThreshold(100);
10spatialDetectionNetwork->setDepthUpperThreshold(5000);
11auto labelMap = spatialDetectionNetwork->getClasses();

功能示例

空间坐标系

OAK 摄像头对所有空间坐标都使用 RDF(右-下-前)坐标系。
帧的中心在 X、Y 坐标上是 0,0。如果向下移动,Y 会增加;如果向右移动,X 会增加。

参考

class

dai::node::SpatialDetectionNetwork

#include SpatialDetectionNetwork.hpp
using
using
Model = NeuralNetwork::Model
variable
Properties & properties
variable
Subnode< NeuralNetwork > neuralNetwork
variable
Subnode< DetectionParser > detectionParser
variable
Subnode< SpatialLocationCalculator > spatialLocationCalculator
variable
std::unique_ptr< Subnode< ImageAlign > > depthAlign
variable
Input & input
Input message with data to be inferred upon Default queue is blocking with size 5
variable
Output & outNetwork
Outputs unparsed inference results.
variable
Output & passthrough
Passthrough message on which the inference was performed.Suitable for when input queue is set to non-blocking behavior.
variable
Input & inputDepth
Input message with depth data used to retrieve spatial information about detected object Default queue is non-blocking with size 4
variable
Output & out
Outputs ImgDetections message that carries parsed detection results.
variable
Output & passthroughDepth
Passthrough message for depth frame on which the spatial location calculation was performed. Suitable for when input queue is set to non-blocking behavior.
explicit function
SpatialDetectionNetwork(const std::shared_ptr< Device > & device)
explicit function
SpatialDetectionNetwork(std::unique_ptr< Properties > props)
function
SpatialDetectionNetwork(std::unique_ptr< Properties > props, bool confMode)
function
SpatialDetectionNetwork(const std::shared_ptr< Device > & device, std::unique_ptr< Properties > props, bool confMode)
function
std::shared_ptr< SpatialDetectionNetwork > build(const std::shared_ptr< Camera > & inputRgb, const DepthSource & depthSource, const Model & model, std::optional< float > fps, std::optional< dai::ImgResizeMode > resizeMode)
Build SpatialDetectionNetwork node with specified depth source. Connect Camera and depth source outputs to this node's inputs and configure the inference model.
Parameters
Returns
Shared pointer to SpatialDetectionNetwork node
function
std::shared_ptr< SpatialDetectionNetwork > build(const std::shared_ptr< Camera > & inputRgb, const DepthSource & depthSource, const Model & model, const ImgFrameCapability & capability)
Build SpatialDetectionNetwork node with specified depth source. Connect Camera and depth source outputs to this node's inputs and configure the inference model.
Parameters
Returns
Shared pointer to SpatialDetectionNetwork node
function
void setNNArchive(const NNArchive & nnArchive)
Set NNArchive for this Node. If the archive's type is SUPERBLOB, use default number of shaves.
Parameters
function
void setFromModelZoo(NNModelDescription description, bool useCached)
Download model from zoo and set it for this Node.
Parameters
  • description: Model description to download
  • useCached: Use cached model if available
function
void setNNArchive(const NNArchive & nnArchive, int numShaves)
Set NNArchive for this Node, throws if the archive's type is not SUPERBLOB.
Parameters
  • nnArchive: NNArchive to set
  • numShaves: Number of shaves to use
function
void setBlobPath(const std::filesystem::path & path)
Backwards compatibility interface Load network blob into assets and use once pipeline is started.
Parameters
  • Error: if file doesn't exist or isn't a valid network blob.
Parameters
  • path: Path to network blob
function
void setBlob(const OpenVINO::Blob & blob)
Load network blob into assets and use once pipeline is started.
Parameters
  • blob: Network blob
function
void setBlob(const std::filesystem::path & path)
Same functionality as the setBlobPath(). Load network blob into assets and use once pipeline is started.
Parameters
  • Error: if file doesn't exist or isn't a valid network blob.
Parameters
  • path: Path to network blob
function
void setModelPath(const std::filesystem::path & modelPath)
Load a network model into assets. DLC and other custom model files are loaded lazily and must remain available and unchanged until the pipeline has been built.
Parameters
  • modelPath: Path to the model file.
function
void setNumPoolFrames(int numFrames)
Specifies how many frames will be available in the pool
Parameters
  • numFrames: How many frames will pool have
function
void setNumInferenceThreads(int numThreads)
How many threads should the node use to run the network.
Parameters
  • numThreads: Number of threads to dedicate to this node
function
void setNumNCEPerInferenceThread(int numNCEPerThread)
How many Neural Compute Engines should a single thread use for inference
Parameters
  • numNCEPerThread: Number of NCE per thread
function
void setNumShavesPerInferenceThread(int numShavesPerThread)
How many Shaves should a single thread use for inference
Parameters
  • numShavesPerThread: Number of shaves per thread
function
void setBackend(const std::string & backend)
Specifies backend to use
Parameters
  • backend: String specifying backend to use
function
void setBackendProperties(const std::map< std::string, std::string > & properties)
Set backend properties
Parameters
  • backendProperties: backend properties map
function
int getNumInferenceThreads()
How many inference threads will be used to run the network
Returns
Number of threads, 0, 1 or 2. Zero means AUTO
function
void setConfidenceThreshold(float thresh)
Specifies confidence threshold at which to filter the rest of the detections.
Parameters
  • thresh: Detection confidence must be greater than specified threshold to be added to the list
function
float getConfidenceThreshold()
Retrieves threshold at which to filter the rest of the detections.
Returns
Detection confidence
function
void setBoundingBoxScaleFactor(float scaleFactor)
Custom interface Specifies scale factor for detected bounding boxes.
Parameters
  • scaleFactor: Scale factor must be in the interval (0,1].
function
void setDepthLowerThreshold(uint32_t lowerThreshold)
Specifies lower threshold in depth units (millimeter by default) for depth values which will used to calculate spatial data
Parameters
  • lowerThreshold: LowerThreshold must be in the interval [0,upperThreshold] and less than upperThreshold.
function
void setDepthUpperThreshold(uint32_t upperThreshold)
Specifies upper threshold in depth units (millimeter by default) for depth values which will used to calculate spatial data
Parameters
  • upperThreshold: UpperThreshold must be in the interval (lowerThreshold,65535].
function
void setSpatialCalculationAlgorithm(dai::SpatialLocationCalculatorAlgorithm calculationAlgorithm)
Specifies spatial location calculator algorithm: Average/Min/Max
Parameters
  • calculationAlgorithm: Calculation algorithm.
function
void setSpatialCalculationStepSize(int stepSize)
Specifies spatial location calculator step size for depth calculation. Step size 1 means that every pixel is taken into calculation, size 2 means every second etc.
Parameters
  • stepSize: Step size.
function
std::optional< std::vector< std::string > > getClasses()
Get classes labels.
function
void buildInternal()
Function called from within the create function to build the node. This function is useful for initialization, setting up inputs and outputs = stuff that cannot be perform in the constuctor.

需要帮助?

请访问讨论论坛以获取技术支持或您可能有的任何其他问题。