# 帧同步

有两种方法可以同步来自不同传感器（帧、IMU数据包、ToF等）的消息：

 * [硬件同步](#Hardware%20syncing)：多传感器亚毫秒精度，硬件触发
 * [软件同步](#Software%20syncing)：基于时间戳/序列号

#### Software syncing

软件同步有两种方法：

 * [序列号同步](#Sequence%20number%20syncing) —— 适用于设置为相同FPS的流，可实现亚毫秒级精度
 * [时间戳同步](#Timestamp%20syncing) —— 适用于不同FPS的流，可与板载传感器（如IMU）或连接到主机的其他传感器（如USB ToF传感器）同步

### 序列号同步

如果需要同步来自同一个OAK的多个消息，例如：

 * 来自 [ColorCamera](https://docs.luxonis.com/software/depthai-components/nodes/color_camera.md#colorcamera) 或
   [MonoCamera](https://docs.luxonis.com/software/depthai-components/nodes/mono_camera.md#monocamera) 的相机帧（彩色、左、右帧）
 * 由相机帧生成的消息（NN结果、视差/深度、边缘检测、tracklets、编码帧、跟踪特征等）

我们可以使用序列号同步，[参考这里的demo](https://github.com/luxonis/oak-examples/tree/master/gen2-syncing#message-syncing)。
来自ColorCamera/MonoCamera的每一帧都会被分配一个序列号，该序列号随后会被复制到从该帧生成的消息中。

对于序列号同步，所有相机的FPS必须相同。在主机或脚本节点内部，可以像这样获取消息的序列号：

```py
# 从队列获取消息
message = queue.get()
# message 可以是 ImgFrame, NNData, Tracklets, ImgDetections, TrackedFeatures...
seqNum = message.getSequenceNum()
```

通过固件同步，我们监测漂移并对齐所有摄像头（左、右、彩色）的捕获时间戳，这些时间戳是在MIPI帧起始（SoF）事件时获取的。左/右全局快门相机由同一时钟驱动，通过I2C广播写入启动，因此在自由运行时，即使没有硬件同步，也不会随时间发生漂移。

RGB卷帘快门在时钟/帧时间上有微小差异，因此当检测到微小漂移时，我们通过调整下一帧的帧时间（行数）来补偿。

如果传感器设置为相同的FPS（默认为30），上述两种方法已集成到depthai中并默认启用，这使我们能够在所有帧以及这些帧生成的消息之间实现亚毫秒级的延迟。

```bash
[Seq 325] RGB 时间戳: 0:02:33.549449
[Seq 325] 视差时间戳: 0:02:33.549402
-----------
[Seq 326] RGB 时间戳: 0:02:33.582756
[Seq 326] 视差时间戳: 0:02:33.582715
-----------
[Seq 327] RGB 时间戳: 0:02:33.616075
[Seq 327] 视差时间戳: 0:02:33.616031
```

视差和彩色帧的时间戳表明我们实现了远低于亚毫秒级的精度。

### 时间戳同步

与序列号同步不同，时间戳同步可以同步：

 * 不同FPS的流
 * IMU结果与其他消息
 * 与连接到计算机的其他设备的消息，因为时间戳是[与主机时钟同步的](https://docs.luxonis.com/software-v3/depthai/depthai-components/device.md)

DepthAI 2.24 引入了
Sync节点，可用于同步不同流或不同传感器（如IMU和彩色帧）的消息。详细信息请参见[Sync节点](https://docs.luxonis.com/software/depthai-components/nodes/sync.md)。Sync节点目前不支持多设备同步，因此如果需要同步来自多个设备的消息，应使用手动方法。

可以查看[这个demo](https://github.com/luxonis/oak-examples/tree/master/gen2-syncing#imu--rgb--depth-timestamp-syncing)，它使用时间戳来同步IMU、彩色和视差帧，所有这些流以不同的FPS生成消息。

在多个流具有不同FPS的情况下，有两种同步方法：

 1. 从较快流中移除一些消息，以与较慢流的同步FPS保持一致
 2. 复制较慢流中的一些消息，以与最快流的同步FPS保持一致

时间戳在帧的MIPI帧起始（SoF）事件时被分配，[更多细节请见此处](https://docs.luxonis.com/hardware/platform/deploy/frame-sync.md)。

```py
# 从队列获取消息
message = queue.get()
timestamp = message.getTimestamp() # 与主机时钟同步的时间戳

# 如果消息是 ImgFrame，可以选择帧曝光的开始/中间/结束
# 也可以使用 .START 或 .END
imgFrame.getTimestamp(dai.CameraExposureOffset.MIDDLE)
```

#### Hardware syncing

Allows precise synchronization (< 10µs) across multiple camera sensors and potentially with other hardware, e.g. flash LED,
external IMU, or other cameras.

## FSYNC signal

FSYNC/FSIN (frame sync) signal is a pulse that is driven high at the start of each frame capture. Its length is not proportional
to the exposure time. It can be either input or output. It operates in 1.8V logic.

On stereo cameras (OAK-D*), we want stereo camera pair (monochrome cameras) to be perfectly in sync, so one camera sensor (eg.
left) has FSYNC set to INPUT, while the other camera sensor (eg. right) has FSYNC set to OUTPUT. In such configuration the right
camera drives left camera.

> **FSYNC signal output**
> At the moment, only OV9282/OV9782 can output FSYNC signal, while IMX378/477/577/etc should also have the capability, but isn't
yet supported (so these can not drive FSYNC signal, only be driven by it). AR0234 has input-only FSYNC trigger.

### Synchronizing frames externally

If we would like to drive cameras with an outside signal, we would need to set FSIN as INPUT for camera sensors.

All [Series 2 OAK PoE models](https://docs.luxonis.com/hardware.md) have an M8 I/O connector which exposes FSIN signal (and also
STROBE). We have developed [FSYNC Y-Adapter](https://docs.luxonis.com/hardware/products/FSYNC%2520Y-Adapter.md) that allows you to
sync (daisy-chain) multiple OAK cameras together.

```py
# Example: we have 3 cameras on ports A,B, and C
cam_A.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)
cam_B.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)
cam_C.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)
```

You can also control FSIN line via GPIO from within
[Script](https://docs.luxonis.com/software-v3/depthai/depthai-components/nodes/script.md), see example
[here](https://gist.github.com/Erol444/a9189a8215371ff9f4cf4472960e1d66).

### Sensor FSYNC support

As noted above the paragraph, only some sensors support FSYNC syncing. There are 2 types of FSYNC syncing:

 * Continuous streaming with external syncing, configured with CameraControl.setFrameSyncMode(). In this mode, the FSIN signal is
   expected to arrive at a continuous rate matching the configured sensor FPS, and trigger can't arrive at arbitrary times as that
   would disrupt internal sensor operations (leading to bad frames, etc). It can only correct for very small amounts of drift over
   time.
 * Snapshot mode with external syncing, configured with CameraControl.setExternalTrigger(). In this mode, trigger can arrive to
   the sensor at any time, and the sensor will take the photo/snapshot.

| Sensor | Shutter | Support |
| --- | --- | --- |
| [OV9282](https://docs.luxonis.com/hardware/sensors/OV9282.md), [OV9782](https://docs.luxonis.com/hardware/sensors/OV9782.md) |
Global | FSYNC input/output, both continuous mode and snapshot (arbitrary external trigger) supported |
| [OV7251](https://docs.luxonis.com/hardware/sensors/OV7251.md) | Global | Should have the same hardware support as OV9*82, but
not implemented in FW as of now |
| [AR0234](https://docs.luxonis.com/hardware/sensors/AR0234.md) | Global | FSYNC input, both continuous and snapshot mode
supported |
| [IMX378](https://docs.luxonis.com/hardware/sensors/IMX378.md), [IMX477](https://docs.luxonis.com/hardware/sensors/IMX477.md),
[IMX577](https://docs.luxonis.com/hardware/sensors/IMX577.md), IMX380 | Rolling | FSYNC input, only continuous mode supported for
rolling shutter sensors. Hardware also supports FSYNC output, but not implemented in FW yet |
| [IMX582](https://docs.luxonis.com/hardware/sensors/IMX582.md) | Rolling | Similar to IMX378, but not yet tested |
| IMX296 (RPi GS Camera) | Global | Arbitrary external trigger supported on XTR/XTRIG pin. Pulse length determines exposure time
(sensor feature). Global Arbitrary external trigger supported on XTR/XTRIG pin. Pulse length determines exposure time (sensor
feature). |

### External FSYNC Example

#### Older devices

Here's an example of how to use external FSYNC signal to trigger camera sensors. You can use any Series 2 OAK-D PoE model to
trigger the FSYNC. We used M8 breakout board to expose the GND/FSYNC lines.

In this example ([script here](https://gist.github.com/Erol444/0138af63378dc8de5b3f7d80db1ea1a5)), sensors were set to Snapshot
mode, as we were triggering the signal with a switch button. Only stereo cameras (2x OV9282) were triggered by the button, as
IMX378 color camera does not support snapshot mode. If we were to use OV9782 color camera, it could be triggered by the button as
well.

#### Newer devices

Devices that use the new M8 connector (like the [OAK-D ToF](https://docs.luxonis.com/hardware/products/OAK-D%2520ToF.md)) will
expect a 5V trigger signal.

By default, pin 2 on the M8 connector (FSYNC) is pulled high internally. When the pin is pulled high, it will stream frames
continuously (when camera is either in INPUT or OUTPUT mode). To trigger frames externally, you will need to pull the pin low (eg.
to pin 8 - GND) to stop the frame streaming, then pull it high again to start streaming.

This way you can send 5V pulses to the FSYNC pin to trigger the frame capture.

> **FSYNC Triggering**
> Only global shutter (OV9282, OV9782, AR0234...) cameras support FSYNC triggering in photo/snapshot mode. Rolling shutter cameras
(IMX378, IMX477, IMX577, etc) don't support it.

### Strobe signal

STROBE signal is an output from the image sensor, and is active (high) during the exposure of the image sensor. It would be used
to eg. drive an external LED lighting for illumination - so lighting would only be active during exposure times, instead
constantly on, which would decrease power consumption and heating of the lighting.

We have used STROBE signal on Pro version of OAK cameras (which have on-board illumination IR LED and IR laser dot projector) to
drive the laser/LED.

### Strobe demo

Cameras with M8 connector allow you to connect external lighting to the STROBE signal, as demonstrated in the video below ([blog
post here](https://discuss.luxonis.com/blog/5491-external-strobe-illumination-with-oak-camera)):

### Frame capture graphs

Frame timestamp is assigned to the frame at the MIPI SoF (start of frame) event, when the sensor starts streaming the frame (MIPI
readout).

For global shutter sensors, this follows immediately after the exposure for the whole frame was finished, so we can say the
timestamp assigned is aligned with end of exposure-window (within a margin of few microseconds). Here's an example graph of the
global shutter sensor timings, which demonstrates when timestamp is assigned to the frame:

#### Global shutter sensor timings

For rolling shutter, the example graph looks a bit different. MIPI SoF follows after the first row of the image was fully exposed
and it's being streamed, but the following rows are still exposing or may have not started exposing yet (depending on exposure
time).

#### Rolling shutter sensor timings

Below there's an example graph of rolling shutter sensor (IMX378) at 1080p and 30fps (33.3ms frame time). MIPI readout time varies
between sensors/resolutions, but for IMX378 it's 16.54ms at 1080P, 23.58ms at 4K, and 33.04ms at 12MP.

### OAK-FFC hardware syncing

On [OAK-FFC-4P](https://docs.luxonis.com/hardware/products/OAK-FFC%25204P.md), we have 4 camera ports; A (rgb), B (left), C
(right), and D (cam_d). A & D are 4-lane MIPI, and B & C are 2-lane MIPI. Each pair (A&D and B&C) share an I2C bus, and the B&C
bus is configured for HW syncing left+right cameras by default.

For A&D ports, you need to explicitly enable hardware syncing:

```py
cam_A.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.OUTPUT)
cam_D.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT)
```

### Arducam FFC camera syncing

Arducam FFC cameras have a 22-pin connector, which don't have lines for FSIN/STROBE. As seen below, to connect Arducam FFC camera
to our OAK-FFC baseboard you need to use 26-to-22 pin converter connector which only exposes STROBE/FSIN lines via test pads. To
sync these cameras, you could either solder a wire from test pad to the camera module's FSIN header pin, or connect all FSIN
header pins together, as done [here](https://discuss.luxonis.com/d/934-ffc-4p-hardware-synchronization/3).

### Connecting FSIN/STROBE

As mentioned, all [Series 2 OAK PoE models](https://docs.luxonis.com/hardware.md) have an M8 I/O connector with FSYNC/STROBE
signal. But if you won't be using these, you will likely need to solder a wire to the PCB on your device. Most PCB designs are
open-source (on [oak-hardware](https://github.com/luxonis/oak-hardware) repository), so you can easily check where FSIN/STROBE
signals are on the PCB.

## OAK-FFC-4P FSIN

As shown on image above, on [OAK-FFC-4P](https://docs.luxonis.com/hardware/products/OAK-FFC%25204P.md) you can enable connection
of FSIN_4LANE and FSIN_2LANE with the MXIO6. The script below will sync together all 4 cameras that are connected to the
OAK-FFC-4P.

```python
# CAM_A will drive FSIN signal for all other cameras:
cam_A.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT) # 4LANE
cam_B.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.OUTPUT) # 2LANE
cam_C.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT) # 2LANE
cam_D.initialControl.setFrameSyncMode(dai.CameraControl.FrameSyncMode.INPUT) # 4LANE

# AND importantly to tie the FSIN signals of A+D and B+C pairs, by setting a GPIO:
# OAK-FFC-4P requires driving MXIO6 high (FSIN_MODE_SELECT) to connect together
# the A+D FSIN group (4-lane pair) with the B+C group (2-lane pair)
config = dai.Device.Config()
config.board.gpio[6] = dai.BoardConfig.GPIO(dai.BoardConfig.GPIO.OUTPUT,
                                            dai.BoardConfig.GPIO.Level.HIGH)

with dai.Device(config) as device:
    device.startPipeline(pipeline)
```

Additional info can be found in [this forum discussion](https://discuss.luxonis.com/d/934-ffc-4p-hardware-synchronization/3).

> **CAM_B and CAM_C**
> **CAM_B and CAM_C**
> (2-lane MIPI ports) share the same I2C bus, which means if both of them have the same sensor (FFC module) connected to them,
they will be
> **in sync regardless**
> of any setting (as I2C command "start exporute" will arrive at the same time). This mechanism is used accross OAK-D cameras to
sync stereo camera pair.

## Series 2 USB OAKs

FSIN lines on DM9098 board ([OAK-D S2](https://docs.luxonis.com/hardware/products/OAK-D%2520S2.md), [OAK-D
W](https://docs.luxonis.com/hardware/products/OAK-D%2520W.md), [OAK-D
Pro](https://docs.luxonis.com/hardware/products/OAK-D%2520Pro.md), [OAK-D Pro
W](https://docs.luxonis.com/hardware/products/OAK-D%2520Pro%2520W.md)):

## USB OAK-1* FSIN

FSIN test pad on NG9093 board ([OAK-1](https://docs.luxonis.com/hardware/products/OAK-1.md), [OAK-1
W](https://docs.luxonis.com/hardware/products/OAK-1%2520W.md), [OAK-1
Lite](https://docs.luxonis.com/hardware/products/OAK-1%2520Lite.md), [OAK-1 Lite
W](https://docs.luxonis.com/hardware/products/OAK-1%2520Lite%2520W.md), [OAK-1
Max](https://docs.luxonis.com/hardware/products/OAK-1%2520Max.md)):

## OAK-D-Lite FSIN

Note that stereo camera pair and color cameras aren't connected together.