Our sensor platform contains a pair of stereo cameras and two 77GHz FMCW radar antenna arrays. The sensors are well-calibrated and synchronized.
Our sensor unit:
Camera (x2): FLIR BFS-U3-16S2C-CS
Radar (x2): TI AWR1843 + DCA1000
Parking lot (PL)
Campus road (CR)
City street (CS)
The annotations provided by CRUW dataset include the following:
Camera: object classes, bboxes, masks, etc.
Radar: object classes, centers, etc.
The detailed format description can be found here.
Note: The annotation format for ROD2021 Challenge is different, which can be found at our evaluation server hosted by CodaLab.
Radar Frequency Data Annotation Methods
Camera-Only (CO) Annotation
Problems to address:
Accurately localize the 3D positions of the objects in videos captured by a camera mounted on an autonomous vehicle.
Adaptively estimate ground plane of each frame for more robust object 3D localization.
Monocular depth estimation or other 3D sensors to obtain depth information.
Object depth histogram analysis or 3D point cloud clustering for object depth initialization.
Adaptive ground plane estimation taking advantage of sparse and dense ground features.
Tracklet smoothing using the results from multi-object tracking.
Detailed information: http://yizhouwang.net/blog/2019/07/15/object-3d-localization/
Reference for CO annotation method:
Camera-Radar Fusion (CRF) Annotation
An intuitive way of improving the above camera-only annotation is by taking advantage of radar, which has a plausible capability of range estimation without any systematic bias.
Heuristic Fusion Algorithm:
Fuse the location results from camera and radar by the distances between each pair. During the fusion process, we trust the range from radar, and trust the azimuth from camera. The brief pipeline of this algorithm can be concluded as follows:
Calculate the fused locations for each pair of the camera-radar locations.
Remove nearby redundant radar locations according to their distances.
First, find and keep the best matching pair for each radar detection. Then, find and keep the best matching pair for each camera detection.
Now, the above matching becomes a one-to-one mapping. Collect all the mappings as the final CRF annotations.
Probabilistic Fusion Algorithm:
Align camera and radar coordinates with sensor calibration results.
Generate two probability maps for camera and radar locations separately.
Fuse two probability maps by element-wise product.
The fused annotations are derived from the fused probability maps by peak detection.
The paper that introduces the above method about CRF annotation is accepted by WACV 2021:
For the ground truth needed for evaluation purposes, we human-annotate the testing sequences with different scenarios. The annotations are operated on the RF images by labeling the object classes and locations according to the corresponding RGB and RF images.