EvalAI Submission Guide#

This document provides a step-by-step guide on how to submit your code to the EvalAI platform for the PhyRC challenge on an Ubuntu computer.

Download and Install Docker#

Visit the Docker Download Page.
Select the appropriate installer for Ubuntu and follow the provided installation instructions.

Set Up Your Docker Environment#

We provide an example of how the code should be structured and how to set up the Dockerfile in the template folder.

A sample Dockerfile can be found here. Use this as a reference to set up your Docker environment.

The Dockerfile runs run_python.shell to execute the code. If you are submitting for only one track, modify run_python.shell to include only the Python script for that track.

Ensure you have the codebase installed on your local machine by following the instructions in the README.md file.

Build and Package Your Docker Environment#

Run the packaging script:
```
# This command line is executed outside of Docker, in the cloned environment.

sudo bash ./save_docker.shell
```
The save_docker.shell script will build the Docker image, run the container, and generate a ZIP file.

Like run_docker.shell, save_docker.shell starts a Docker container that runs in the background by default. To save memory, the script removes the running Docker container after packaging. You can comment out the relevant statements to disable this behavior if preferred.

*[OPTIONAL]* Alternatively, you can manually package your Docker image:

# Find your Docker image ID
# If Docker was installed using sudo, please remember to use `sudo docker`.
docker images

# Use the image ID to package your Docker image:
docker save <image_id> -o your_docker_name.tar

# Create a directory to unpack the tar file
mkdir unpacked_docker

# Unpack the tar file into the directory
tar -xf your_docker_name.tar -C unpacked_docker

# Compress the unpacked files into a zip file
zip -r your_docker_name.zip unpacked_docker

# Clean up the unpacked directory and the original tar file
rm -rf unpacked_docker
rm your_docker_name.tar

After packaging, submit the compressed .zip file.

Note: You can also upload a .tar file directly. However, we recommend packaging in .zip format. We accept and evaluate both .tar and .zip files, but other file types will not be accepted and will receive a score of zero even if uploaded successfully.

Uploading to EvalAI#

Visit the EvalAI Challenge Page and click on the ‘Participate’ button. Sign up or log in to your EvalAI account.
Select Phase 1.
Choose the upload method depending on the file size:

You should upload the Docker zip file generated by save_docker.shell. The default file name is your_docker_name.zip and it is located in the /RCareWorld/template/docker-template folder.
- Use CLI for file size > 400MB
- Upload directly if file size < 400MB
If using the CLI, upload with:
```
pip install "evalai"

# conda create -n evalai python=3.8
# conda activate evalai
# pip install "evalai"
```
If you encounter issues with pip install evalai, you can create a new Python 3.8 conda environment that’s just for submitting to EvalAI.

First, visit the participate page to log in, then visit the submission page and follow the instructions to submit your code.

Each person’s token and submission command will differ, so copy the command from the webpage to submit.
If uploading directly, select the .zip or .tar file and upload:
```
# Choose the file
# Click on the 'Upload File' button
```
Your submission should appear on the leaderboard after processing, which may take anywhere from 10 minutes to several hours depending on the file size and the number of submissions in the queue.

Run Your Docker Image#

When you deploy your own Docker environment, you might encounter issues. The run_docker.shell script can be used to check if Docker is running correctly, and it will also, by default, enter the Docker environment at the end.

Navigate to the docker-template folder:
```
cd RCareWorld/template/docker-template
```
Run the Docker script:
```
# This command line is executed outside of Docker, in the cloned environment.

sudo bash ./run_docker.shell
```
The run_docker.shell script will build the Docker image and run the container.

After the script completes, it will automatically enter Docker’s interactive execute environment. By default, the Docker container will continue running in detached mode, allowing you to interact and run your code. If you don’t need it, you can manually stop it by typing exit in the Docker shell.

To prevent the Docker from entering the interactive mode, you can remove the last line docker exec -it $CONTAINER_ID /bin/bash from the run_docker.shell script.

By removing the sleep infinity line from the run_python.shell script, the container will automatically stop after the script finishes executing.

Write and Test Code Inside the Docker#

You can also actively enter Docker to perform testing.

Please note, if you are installing Docker as the root user, make sure to prepend sudo to your Docker commands, otherwise you may encounter permission errors.

Check the Docker container’s ID (run this before entering the Docker environment):
```
docker ps
```
This command displays information about the running Docker containers. Containers built with run_docker.shell or save_docker.shell will run in the background by default.
Access your Docker container’s shell:
```
docker exec -it <container_id> bash
```
Navigate to the project directory:
```
cd /app/RCareWorld/template
```
Run your script (e.g., test_bathing.py):
```
# This command line is executed inside of Docker

python test_bathing.py
```
Note: If running code in the Docker container, ensure you have set graphics=False in RCareWorld() before copying your code.

EvalAI Submission Guide