Evaluation Methodology

Dataset

The total number of images in our dataset is 72,043 images spanning 80 different sub-datasets. All images have a resolution of 352 x 288 with a top-down view.  The data is split into a train set, a validation set, and a test set (see below).

• The ratio for the train, validation, and test sets are 70%, 10%, and 20%, respectively.
  • 49,999 images for the train set (52 sub-datasets)
  • 7,193 images for the validation set (8 sub-datasets)
  • 14,851 images for the test set (20 sub-datasets) 

The data is split in a way that preserves the difficulties of the scenarios between the train, validation, and test set, to make sure we have a validation and a test set that are as challenging as the training set. To create a dataset that matches our real-world application, the following ground-truth bounding box annotations are excluded from the evaluation procedure

Ground truths that have an area less than 2% of the area of the image and at the same time are at the boundary of the image. Ground truths that have an area between 1% and 2% of the area of the image and are in the middle of the image but are highly occluded, e.g. only an arm or a leg of a person can be seen. Ground truths that have an area less than 1% of the area of the image.

The model is trained on the train set only and has never seen any image in the validation and test sets.

Evaluation

The model is evaluated by the following metrics:

F1-score, which is the harmonic mean of precision and recall, the higher the better. Ssd_resnet50_fpn (or retinanet_50) is a much deeper model with a very large model size, and cannot be deployed on edge devices. Our model is very lightweight (< 5 MB), and is compatible with edge devices while achieving competitive results.

Benchmarks

Updated May 18, 2023 20:39