New technology helps re-identify people in videos

The software reduces the time taken for police to review and process vital surveillance footage, and has even been used to help solve a murder.

What’s the problem?

Current CCTV surveillance is a repetitive and time-consuming manual task that is often reliant on a person spotting a momentary incident occurring on several different monitors at the same time.

It is critical when a target disappears from one view that they can be re-identified in another view at a different location among a crowd of people. This can be particularly difficult to do when tracking people across a large area of public space covering by multiple cameras that don’t overlap. To solve this problem, the same person needs to be tracked through different camera views located at different physical sites over different times.

The close monitoring of CCTV footage in this way is not always accurate or effective, as people frequently don’t know what to look for. As well as this, even if the precise image frames of the incident are finally identified, the image data may not be of a very good quality either for recognition or as evidence. As a result, most CCTV recordings are never used, or occasionally retrieved only after an incident has occurred.

How can Re-ID help?

Re-ID provides a unique solution to the market with a complete forensic video solution integrating hardware, software and video.

To help solve a key problem of re-identifying people as they move across locations, the software provides a set of features extracted from each person’s image. An optimal distance measure is then provided, which helps to correctly match individuals. Furthermore, the re-identification methods are both video and single-image based, which enables video-based matching faster than real time on large surveillance videos.

In addition to this, research found there may be times when a machine or computer system can’t be trusted as a fully automated ‘black box’, and needs human intervention and verification. Professor Gong’s work has uniquely enabled human-in-the-loop – a form of artificial intelligence using both human and machine intelligence, to help machines make quick and accurate decision based on human feedback. This is crucial for commercial applications, as it makes the artificial intelligence systems more transparent and trustworthy.

Another issue found in the real world, people added to a watch list might only have a few images used to identify them, and in some cases only one. Existing re-identification methods are not really suitable to address this challenge as they are designed for a close-world scenario, where the gallery (the image used) and probe sets (footage from the video) are assumed to contain exactly the same people. Under the open-world person re-identification settings, a large amount of people will be captured in footage along with those on a watch list. Their images will subsequently also appear in the gallery set, as some of them will look visually similar to targets on the watch list. Professor Gong’s research has developed beyond the closed-set benchmark matching, to enable person re-identification over distributed and disjointed domains of open-world search.

Finally, subtle appearance discrepancies can occur between different fine-grained attribute classes. For example, “Woollen-Coat” can appear very similar to “Cotton-Coat”, whilst “Moustache” can be visually difficult to differentiate. To discriminate subtle classes from multi-labelled images at large scale, Professor Gong has developed an imbalanced deep learning method to explore large quantity of class imbalanced training data for all labels to extend beyond standard learning algorithms.

Solving cases

Re-ID has been used by police forces internationally, including for a case in America that had gathered footage from more than 30 cameras running 24/7 over five months. It would have taken a single person 15 years to review the video footage on their own, or a small team approximately two years. Using the Re-ID software, all the video footage was reviewed in only four days, gathering vital evidence, and solving the case.

Likewise in Australia the Re-ID software was able to process 21,000 hours of surveillance video in only 30 hours and helped solve a murder.

Supporting clinical decision making

Professor Gong is a pioneer of computer vision and machine learning for visual surveillance, and a world authority on Person Re-Identification. His Computer Vision research group are internationally renowned for their work on unusual behaviour recognition, person re-identification, multi-camera tracking, video search and categorisation, and face analysis in video and images. In 2007 VSL spun out from this group and has since been developing their revolutionary Re-ID software system.

Re-ID was officially launched on 6 October 2016 at the Royal Institution in London, and in 2017 won the Aerospace Defence Security Innovation Award for a ‘revolutionary solution to reviewing CCTV footage’.

Last year, Professor Gong was awarded the Institution of Engineering and Technology (IET) 2020 Achievement Medal in Vision Engineering for “outstanding achievement and superior performance in contributing to public safety”.