WP1: Infrastructure and datasets

This work package aims at the initial establishment and maintenance of a development framework for the remaining work packages.

T1.1: Infrastructure update and maintenance (M1-M28)

Arrangement and configuration of the available equipment, and the acquisition of complementary equipment, for establishing the necessary infrastructure to meet the objectives of the project (storage, communication, and computing infrastructures).

T1.2: Design and generation of datasets (M5-M32)

Support to other tasks for generating test data and defining evaluation methodologies. It includes the selection of appropriate datasets (sequences and associated ground-truth) and their generation, if required, as well as the research, selection, and proposal of evaluation metrics and benchmarks.

Milestones:

• M1.1 Acquisition of required hardware for WP2 and WP3 (T1.1, M6)
• M1.2 Generation of datasets for WP2 (T1.2, M12)
• M1.3 Update of required hardware for WP2 (T1.1, M20)
• M1.2 Generation of datasets for case studies in T3.4 (T1.2, M24)

Deliverables:

• D1.1 System infrastructure (M12 and updates every 12 months if required)
• D1.2 Evaluation frameworks: datasets, metrics, and benchmarks (M12 and updates every 12 months if required)

WP.2: Dealing with low availability of annotated data

To obtain research contributions to the current state-of-the-art in absence of large human-annotated datasets. Such contributions will be performed on public datasets. If required, small scenarios will be recorded or generated within WP1.

T2.1: Real data in absence of annotations (M1-M32)

To quantify the impact of new learning-based methods in absence of large human-annotated real datasets. In this direction, we propose: (1) searching for the learning capabilities of different unsupervised and self-supervised learning alternatives in order to reduce the gap between supervised vs unsupervised learning capabilities; (2) studying the use of multiple complementary self-supervised pretext tasks of different nature, and assess their contribution to the knowledge acquired by the trained model; (3) evaluating the impact of data in self-supervised learning regimes, analyzing the effect of the size, diversity and noise of the data in the training of self-supervised learning models, and quantifying their impact in the knowledge encoded by the trained model; (4) exploring situations in which information becomes incrementally available over time with three different continual lifelong learning alternatives: incremental domain learning, incremental class learning and incremental task learning.

T2.2: Creation and use of synthetic data (M1-M32)

To explore the creation and use of synthetic datasets to complement the training process. Jointly with the existing real datasets, the synthetic data will provide diversity in the training process helping to obtain more robust visual models. In this direction, we propose: (1) the automatic generation of complexity-variable scenarios, associated synthetic data and ground-truth for detection, tracking and semantic segmentation tasks; (2) exploring supervised and weakly supervised domain adaptation approaches for video object tracking and semantic segmentation tasks by analyzing adversarial strategies; (3) to develop new unsupervised domain adaptation approaches for video object tracking and semantic segmentation tasks based on distance metrics, mutual information and self-training strategies.

T2.3: Model profiling (M5-M32)

To explore methodologies for the visualization of DL models, focusing on aiding human interpretation. Utilize model profiling tools to scrutinize the knowledge encoded in self-supervised trained modes in order to: (1) detect and mitigate possible biases; (2) improve the self-supervised learning process considering the characteristics of the target visual task and target data distribution; (3) provide qualitative and quantitative explanations of the model predictions by measuring the attribution of the model predictions on the target data.

Milestones:

• M2.1 First version of non-supervised and self-supervised algorithms (T2.1, M16)
• M2.2 First version of continual lifelong learning algorithms (T2.1, M16)
• M2.3 First version of creation and use of synthetic datasets (T2.2, M16)
• M2.4 First version of model profiling (T2.3, M16)
• M2.5 Final version of non-supervised and self-supervised algorithms (T2.1, M32)
• M2.6 Final version of continual lifelong learning algorithms (T2.1, M32)
• M2.7 Final version of creation and use of synthetic datasets (T2.2, M32)
• M2.8 Final version of model profiling (T2.3, M32)

Deliverables:

• D2 Technologies for low availability of annotated data (M16 and M32)

WP.3: Management, dissemination and uses cases

To coordinate the project activities, follow-up progress, the dissemination, define the final uses cases (to be defined with the help of the Observing Partners) and develop technology transfer initiatives. This workpackage makes use of the results from WP1 and WP2.

T3.1 Management (M1-M36)

This task has the following activities: monthly follow up of Project progress and achievements, workplan milestones and outcomes deadlines control, workplan updates, corrective actions, and administrative issues.

T3.2 Communication (M1-M36)

This task coordinates the communication of the project advances to Observing Partners and general public. A web site will include a description of the project, status, advances, and announcements, as well as the biannual HVD Newsletters.

T3.3 Dissemination (M1-M36)

This task coordinates the compilation and internal publication of intermediate results, in order to have a clear plan for dissemination based on a commitment for bi-monthly reports’ updates that will constitute seeds for publications, as well as the analysis of target journals and conferences for the different works in progress. It will also coordinate the dissemination of results via Research Reports, GitHub software repositories and generated datasets, to be made available at the project’s web page, via links to open repositories. This task also handles the Data Management plan.

T3.4 Use cases and technology transfer (M12-M36)

Using WP2 outcomes, use cases will be defined with the help and recommendations of the Observing Partners (as described in section 4.3). Information meetings will be held with the Observing Partners and up to two Workshops, open to other companies and interested parties, will be held in the form of Industry Days. This task also considers, during the last year, the implementation of the software required for each use case (Minimum Viable Product prototype) as a tool for boosting the technology transfer initiatives with our Observing Partners and other stakeholders.

Milestones:

• M3.1 Web page (T3.2, M1)
• M3.2 Initial results and issues for the use cases (T3.4, M22)

Deliverables:

• D3.1 Data management plan (M9)
• D3.2 Results report (M9 and updates every 9 months)

Last update 01/09/2022