2024
2023
2022
2021

Multi-Object Navigation (MultiON) Challenge

Held in conjunction with the CVPR 2021 Embodied AI Workshop

Overview

Multi-Object Navigation (MultiON) challenge is hosted at Embodied AI workshop, CVPR 2021. This challenge is built upon the task introduced in MultiON: Benchmarking Semantic Map Memory using Multi-Object Navigation (NeurIPS 2020). MultiON is a long-horizon generalization of the object-navigation task i.e. where the agent has to navigate to multiple goals.

Task

In MultiON, an agent is tasked with navigating to a sequence of objects. These objects are flexibly inserted into a realistic 3D environment. The task is based on the Habitat platform and Matterport3D scenes. For easy identification, the target objects are randomly sampled from a set of 8 cylinders with identical shapes but different colors.
In each episode, the agent is initialized at a random starting position and orientation in an unseen environment and provided a sequence of target objects randomly sampled (without replacement) from the set of 8 objects. The agent must navigate to each object in the sequence (in the given order) and enact the FOUND action to indicate discovery.
The agent has access to an RGB-D camera and a noiseless GPS+Compass sensor. GPS+Compass sensor provides the agent's current location and orientation information relative to the start of the episode.

Evaluation Details

The episode terminates when an agent discovers all objects in the sequence of the current episode or when it calls an incorrect FOUND action. A FOUND action is incorrect if it is called when the agent is not within a 1.5m Euclidean distance from its current target object. Note that this does not require the agent to be viewing the object at the time of calling FOUND. After the episode terminates, the agent is evaluated using the Progress and PPL metrics that are defined below.
Progress: The proportion of objects correctly found in the episode.
PPL: Progress weighted by path length. PPL quantifies the efficiency of agent's trajectory with respect to the optimal trajectory.

Submission Guidelines

Participants must make submission through our EvalAI page. There are three phases in the challenge.

Phase 1: Minival

This phase evaluates MultiON agents on the val set of the MultiON dataset. The results from this phase are not visible on the leaderboard. This phase is meant to be used for sanity checking the results of remote evaluation against your local evaluations.

Phase 2: Test-Standard

This results of this phase will be used to prepare the public leaderboard. We suggest using this phase for reporting results in papers and for comparing with other methods. Each team is allowed a maximum of 1 submission per day for this phase.

Phase 3: Test-Challenge

Only submissions made in this phase are considered as entries to the MultiON Challenge since this will be used to decide the winners. Each team is allowed a total of 3 submissions to this phase until the end of this phase.

For detailed submission instruction, please refer this.

Challenge Updates

Any updates related to the challenge will be posted here. You can also join our Google Group email list to receive updates: click here to join or send an email to multion-challenge-2021+subscribe@googlegroups.com.

Terms and Conditions

The Habitat-Sim is released under MIT license. The Matterport3D dataset is released under Matterport3D Terms of use. If you use Habitat-Sim or the MultiON dataset in a paper, please consider citing the following publications: 
@inproceedings{habitat19iccv,
    title     =     {Habitat: {A} {P}latform for {E}mbodied {AI} {R}esearch},
    author    =     {Manolis Savva and Abhishek Kadian and Oleksandr Maksymets and Yili Zhao and Erik Wijmans and Bhavana Jain and Julian Straub and Jia Liu and Vladlen Koltun and Jitendra Malik and Devi Parikh and Dhruv Batra},
    booktitle =     {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    year      =     {2019}
    }
@inproceedings{wani2020multion,
    title       =   {Multi-ON: Benchmarking Semantic Map Memory using Multi-Object Navigation},
    author      =   {Saim Wani and Shivansh Patel and Unnat Jain and Angel X. Chang and Manolis Savva},
    booktitle   =   {Neural Information Processing Systems (NeurIPS)},
    year        =   {2020},
    }

Dates

Challenge starts February 17, 2021
Leaderboard opens March 1, 2021
Challenge submission deadline May 31, 2021 (AoE)
Winner announcement at Embodied AI Workshop, CVPR 2021 June, 2021

Challenge Results

Challenge workshop talk:
Challenge Leaderboard:
Rank Team Progress PPL Success SPL
1 Lyon 67 44 55 35
2 SGoLAM 64 38 52 32
3 VIMP 57 36 41 26
1st place: Team Lyon, Pierre Marza, Laetitia Matignon, Olivier Simonin, Christian Wolf, Learning mapping and spatial reasoning with auxiliary tasks [Paper].
2nd place: Team SGoLAM, Junho Kim, Eun Sun Lee, Mingi Lee, Dongsu Zhang, Young Min Kim, Simultaneous goal localization and mapping.
3rd place: Team VIMP, Tommaso Campari, Luciano Serafini, Lamberto Ballan, A memory driven approach for the multi-object navigation task.

Organizers