Multi-Object Navigation (MultiON) Challenge

Held in conjunction with the Embodied AI Workshop at CVPR 2022

CYLINDER Track

REAL-OBJECTS Track

Goals for CYLINDER Track (left):

Goals for REAL-OBJECTS Track (right):

Overview

Multi-Object Navigation (MultiON) challenge is hosted at Embodied AI workshop, CVPR 2022. This challenge is built upon the task introduced in MultiON: Benchmarking Semantic Map Memory using Multi-Object Navigation (NeurIPS 2020). MultiON is a long-horizon generalization of the object-navigation task i.e. where the agent has to navigate to multiple goals.

Dates

Challenge starts (Dataset and starter code available and EvalAI opens up for Minival Phase submissions)	February 21, 2022
Leaderboard opens (Test Standard Phase and Test Challenge Phase submissions)	March 1, 2022
Challenge submission deadline	June 7, 2022 (AoE)
Winner announcement at Embodied AI Workshop, CVPR 2022	June, 2022

Task

In MultiON, an agent is tasked with navigating to a sequence of objects. These objects are flexibly inserted into a realistic 3D environment. The task is based on the Habitat platform and Habitat-Matterport 3D (HM3D) scenes.

This year the challenge has two separate tracks.

CYLINDER Track
Each episode contains 5 target objects randomly sampled from a set of 8 cylinders with identical shapes but different colors.
REAL-OBJECTS Track
Each episode contains 5 target objects randomly sampled from a set of 8 realistic looking 3D objects.

Additionally, there are some other changes from last year.

Both the tracks contain five target objects (5ON) in contrast to three (3ON) from last year's challenge.
The episodes contain some distractor (non-target) objects randomly scattered around the environment, to increase the difficulty of the task.

In summary, in each episode, the agent is initialized at a random starting position and orientation in an unseen environment and provided a sequence of 5 target objects randomly sampled (without replacement) from the set of 8 objects. The agent must navigate to each target object in the sequence (in the given order), avoiding distractor objects and enact the FOUND action to indicate discovery. The agent has access to an RGB-D camera and a noiseless GPS+Compass sensor. GPS+Compass sensor provides the agent's current location and orientation information relative to the start of the episode.

Evaluation Details

The episode terminates when an agent discovers all objects in the sequence of the current episode or when it calls an incorrect FOUND action. A FOUND action is incorrect if it is called when the agent is not within a 1.5m Euclidean distance from its current target object. Note that this does not require the agent to be viewing the object at the time of calling FOUND. After the episode terminates, the agent is evaluated using the Progress and PPL metrics that are defined below.
Progress: The proportion of objects correctly found in the episode.
PPL: Progress weighted by path length. PPL quantifies the efficiency of agent's trajectory with respect to the optimal trajectory.

Submission Guidelines

Participants must make submission through our EvalAI page. There are three phases in the challenge.

Phase 1: CYLINDER Track Minival Phase

This phase evaluates MultiON agents on the val set of the CYLINDER Track MultiON dataset. This phase is meant to be used for sanity checking the results of remote evaluation against your local evaluations.

Phase 2: CYLINDER Track Test Standard Phase

This results of this phase will be used to prepare the public leaderboard for the CYLINDER Track. We suggest using this phase for reporting results in papers and for comparing with other methods. Each team is allowed a maximum of 3 submissions per day for this phase.

Phase 3: CYLINDER Track Test Challenge Phase

Only submissions made in this phase are considered as entries to the CYLINDER Track MultiON Challenge since this will be used to decide the winners. Each team is allowed a total of 3 submissions to this phase until the end of this phase.

Phase 4: REAL-OBJECTS Track Minival Phase

This phase evaluates MultiON agents on the val set of the REAL-OBJECTS Track MultiON dataset. This phase is meant to be used for sanity checking the results of remote evaluation against your local evaluations.

Phase 5: REAL-OBJECTS Track Test Standard Phase

This results of this phase will be used to prepare the public leaderboard for the REAL-OBJECTS Track. We suggest using this phase for reporting results in papers and for comparing with other methods. Each team is allowed a maximum of 3 submissions per day for this phase.

Phase 6: REAL-OBJECTS Track Test Challenge Phase

Only submissions made in this phase are considered as entries to the REAL-OBJECTS Track MultiON Challenge since this will be used to decide the winners. Each team is allowed a total of 3 submissions to this phase until the end of this phase.

For detailed submission instruction, please refer this.

Challenge Updates

Any updates related to the challenge will be posted here. You can also join our Google Group email list to receive updates: click here to join or send an email to multion-challenge-2022+subscribe@googlegroups.com.

Terms and Conditions

The Habitat-Sim is released under MIT license. To use HM3D dataset, please refer here. If you use Habitat-Sim or the MultiON dataset in a paper, please consider citing the following publications:

@inproceedings{habitat19iccv,
    title     =     {Habitat: {A} {P}latform for {E}mbodied {AI} {R}esearch},
    author    =     {Manolis Savva and Abhishek Kadian and Oleksandr Maksymets and Yili Zhao and Erik Wijmans and Bhavana Jain and Julian Straub and Jia Liu and Vladlen Koltun and Jitendra Malik and Devi Parikh and Dhruv Batra},
    booktitle =     {Proceedings of the IEEE/CVF International Conference on Computer Vision (ICCV)},
    year      =     {2019}
    }

@inproceedings{wani2020multion,
    title       =   {Multi-ON: Benchmarking Semantic Map Memory using Multi-Object Navigation},
    author      =   {Saim Wani and Shivansh Patel and Unnat Jain and Angel X. Chang and Manolis Savva},
    booktitle   =   {Neural Information Processing Systems (NeurIPS)},
    year        =   {2020},
    }

Organizers

Sonia Raychaudhuri
SFU

Tommaso Campari
SFU, U of Padova, FBK

Unnat Jain
Facebook AI Research

Angel X. Chang
SFU

Manolis Savva
SFU