{"id":1859,"title":"DialFRED Challenge","short_description":"Challenge to create dialogue-enabled agents for embodied instruction following","description":"<p>\r\nWelcome to the DialFRED challenge for Embodied AI.\r\nDialFRED adds <b>Dial</b>og interactions to the Al<b>fred</b> benchmark,\r\nby allowing the Embodied agents to actively ask questions to gain\r\nmore information to complete tasks in the AI2Thor environment. The dataset for this challenge was described in our <a href=\"https://arxiv.org/abs/2202.13330\">paper</a>.\r\n</p>\r\n\r\n<p>\r\n<img src=\"https://raw.githubusercontent.com/xfgao/DialFRED/main/misc/dialfred_title.png\" alt=\"\" width=\"600\">\r\n</p>\r\n\r\n<p>\r\n<img src=\"https://raw.githubusercontent.com/xfgao/DialFRED/main/misc/dialfred_table.png\" alt=\"\" width=\"600\">\r\n</p>\r\n\r\n<p>\r\nIn the training and validation dataset for this challenge,\r\nwe provide human-annotated data with <b>53K</b> task-relevant questions and answers,\r\nas well as an <b>oracle</b> to answer the respective questions from the embodied agent.\r\nDialFRED challenge invites researchers to submit the results of their models on a <b>new test-set</b>.\r\nFor the training, validation and test data, please refer to our <a href=\"https://github.com/xfgao/DialFRED\">code repository</a>.\r\nFor further details refer to the <a href=\"https://arxiv.org/abs/2202.13330\">DialFRED paper</a>.\r\n</p>\r\n\r\n<link rel=\"stylesheet\"\r\n        href=\"https://cdnjs.cloudflare.com/ajax/libs/highlight.js/10.0.3/styles/default.min.css\">\r\n<script src=\"https://cdnjs.cloudflare.com/ajax/libs/highlight.js/10.0.3/highlight.min.js\"></script>\r\n<script>hljs.initHighlightingOnLoad();</script>\r\n\r\n<pre><code class=\"python\"><tt>\r\n    @article{gao2022dialfred,<br>\r\n        &nbsp;&nbsp;&nbsp;&nbsp;title={Dialfred: Dialogue-enabled agents for embodied instruction following},<br>\r\n        &nbsp;&nbsp;&nbsp;&nbsp;author={Gao, Xiaofeng and Gao, Qiaozi and Gong, Ran and Lin, Kaixiang <br>\r\n            and Thattai, Govind and Sukhatme, Gaurav S},<br>\r\n        &nbsp;&nbsp;&nbsp;&nbsp;journal={arXiv preprint arXiv:2202.13330},<br>\r\n        &nbsp;&nbsp;&nbsp;&nbsp;year={2022}<br>\r\n      }\r\n</tt></code></pre>\r\n\r\n\r\n<!-- <p>\r\nWelcome to the DialFRED challenge.\r\nBased on the Alfred benchmark, this challenge allows for the agent to actively ask questions to gain more information.\r\nIn the training and validation set, we provide human-annotated data with 53K task-relevant questions and answers as well as an oracle to answer the questions.\r\nIn this challenge, we invite researchers to submit the results of their models on a new test set.\r\nFor the training, validation and test data, please refer to our <a href=\"https://github.com/xfgao/DialFRED\">code repository</a>.\r\nFor further details refer to the <a href=\"https://arxiv.org/abs/1912.01734\">DialFRED</a> paper.\r\n</p> -->\r\n\r\n\r\n\r\n<!-- Language-guided Embodied AI benchmarks requiring an agent to navigate an environment and manipulate objects\r\ntypically allow one-way communication: the human user gives anatural language command to the agent, and the\r\nagent can only follow the command passively.\r\n\r\nWe present DialFRED, a dialogue enabled embodied instruction following benchmark based on the\r\nALFRED benchmark.\r\n\r\nDialFRED allows an agent to actively ask\r\nquestions to the human user; the additional information in the\r\nuser's response is used by the agent to better complete its task.\r\n\r\nWe release a human-annotated dataset with 53K task-relevant\r\nquestions and answers and an oracle to answer questions.\r\n\r\nTo tackle DialFRED, we propose a questioner-performer framework\r\nwherein the questioner is pre-trained with the human-annotated\r\ndata and fine-tuned with reinforcement learning.\r\n\r\nExperimental results show that asking the right questions leads to signifi-\r\ncantly improved task performance.\r\n\r\nWe make DialFRED publicly available and encourage researchers to propose and evaluate\r\ntheir solutions to building dialog-enabled embodied agents -->","terms_and_conditions":"<ul>\r\n    <li>All submission must attempt to solve the DialFRED task.</li>\r\n    <li>Share who you are: you must provide a team name and affiliation.</li>\r\n    <li>Share how you solved it: if possible, share information about how the task was solved. Link an academic paper or code repository if public.</li>\r\n    <li>Only submit your own work.</li>\r\n</ul>","submission_guidelines":"<div><!--block--><br>The submission must consist of a zip file containing a set of json files, one for each trial. The names of the json files must be [trial_id - 4 digits with leading zeros].json. Please make sure that all trial jsons are in the root of the zip file. Do not include any folder structure.&nbsp;<br><br></div><div><!--block--><br>The ZIP file structure should look like this:<br><br></div><div><!--block--><span data-trix-cursor-target=\"left\" data-trix-serialize=\"false\"></span><figure contenteditable=\"false\" data-trix-attachment=\"{&quot;contentType&quot;:&quot;image&quot;,&quot;height&quot;:207,&quot;url&quot;:&quot;https://raw.githubusercontent.com/xfgao/DialFRED/main/misc/submission_format.png&quot;,&quot;width&quot;:286}\" data-trix-content-type=\"image\" data-trix-id=\"9547\" class=\"attachment attachment--preview\"><img src=\"https://raw.githubusercontent.com/xfgao/DialFRED/main/misc/submission_format.png\" data-trix-mutable=\"true\" width=\"286\" height=\"207\" data-trix-store-key=\"imageElement/9547/https://raw.githubusercontent.com/xfgao/DialFRED/main/misc/submission_format.png/286/207\"><figcaption class=\"attachment__caption\" data-trix-placeholder=\"Add a caption…\"></figcaption></figure><span data-trix-cursor-target=\"right\" data-trix-serialize=\"false\"></span></div><div><!--block--><br>&nbsp;Each trial file is a list of environment state metadata. The environment should be instantiated from <a href=\"https://github.com/xfgao/DialFRED/blob/main/alfred/env/thor_env.py#L21\">Alfred</a>. You should set up the trial with data from the input files we provide.&nbsp;<br><br></div><pre><!--block--> env = ThorEnv(x_display=0)\n traj_data = json.load(open(\"path/to/input/9999.json\", \"r\"))\n \n scene_num = traj_data['scene']['scene_num']\n object_poses = traj_data['scene']['object_poses']\n dirty_and_empty = traj_data['scene']['dirty_and_empty']\n object_toggles = traj_data['scene']['object_toggles']\n \n scene_name = 'FloorPlan%d' % scene_num\n env.reset(scene_name)\n env.restore_scene(object_poses, object_toggles, dirty_and_empty)\n \n # initialize to start position\n env.step(dict(traj_data['scene']['init_action']))</pre><div><!--block--><br></div><div><!--block--><br>&nbsp;Your model will execute a series of actions, after <a href=\"https://github.com/xfgao/DialFRED/blob/main/alfred/env/thor_env.py#L141\">executing each action</a> you will save the state metadata member of the output. A rough pseudocode example is provided below.&nbsp;</div><div><!--block--><br></div><pre><!--block--> meta_data = []\n while still_doing_actions:\n     act = get_next_action()\n     event = env.step(act)\n     m_data = event.metadata\n     m_data[\"pose_discrete\"] = event.pose_discrete\n     meta_data.append(m_data)\n json.dump(meta_data, open(\"path/to/output/9999.json\", \"w\"), sort_keys=True, indent=4)</pre><div><!--block--><br>An example tiral submission file can be found <a href=\"https://github.com/xfgao/DialFRED/blob/main/misc/9999.json\">here</a>.&nbsp;</div><div><!--block--><br>The development phase is for internal use only.</div><div><!--block--><br><strong>Please submit to the CVPR 2023 EAI Workshop phase to participate in the competition.<br></strong><br></div>","evaluation_details":"<p>Submissions will be scored on the average success rate on the subgoals over all trials.</p>","image":"https://media.eval.ai/media/logos/a73b1004-cb8e-4330-a04d-f37489819678.jpg","start_date":"2022-10-01T00:00:00Z","end_date":"2023-06-30T07:00:59Z","creator":{"id":2026,"team_name":"DialFRED_host","created_by":"xiaofenggao","team_url":""},"domain":null,"domain_name":null,"challenge_usage_type":"paid","list_tags":[],"has_prize":false,"has_sponsors":false,"published":true,"submission_time_limit":86400,"is_registration_open":false,"enable_forum":true,"anonymous_leaderboard":false,"manual_participant_approval":false,"is_active":false,"leaderboard_description":"Leader board for the challenge.","allowed_email_domains":[],"blocked_email_domains":[],"banned_email_ids":[],"require_complete_profile":false,"max_team_members":null,"approved_by_admin":true,"is_approval_requested":false,"forum_url":null,"is_docker_based":false,"is_static_dataset_code_upload":false,"slug":"dialfred-challenge-1859","max_docker_image_size":42949672960,"cli_version":null,"remote_evaluation":false,"allow_resuming_submissions":false,"allow_host_cancel_submissions":false,"allow_cancel_running_submissions":false,"allow_participants_resubmissions":false,"workers":null,"created_at":"2022-09-08T00:52:00.042487Z","queue":"random-number-generator-challenge-1859-production-9fe45990-c73e-459f-ba89-71f9ac","worker_cpu_cores":1024,"worker_memory":2048,"cpu_only_jobs":false,"job_cpu_cores":"2000m","job_memory":"8Gi","uses_ec2_worker":false,"ec2_storage":8,"ephemeral_storage":21,"evaluation_module_error":null,"worker_image_url":null,"worker_python_version":"3.9","worker_instance_type":"g4dn.xlarge","sqs_retention_period":345600,"use_fifo_sqs":false,"github_repository":"xfgao/EvalAI_DialFRED","github_branch":"","is_frozen":true,"is_submission_paused":false}