How to Submit?

This page contains the information about the submission procedure. First, you need to run the benchmark .

The IO500 list is released during ISC and SC each year. Submissions to the upcoming list can be made all year. However, to be included in the next submission, we must receive the submission before the deadline listed above, unless you have previously contacted the committee and approved an extension for exceptional circumstances.

Production and Research Lists

Starting with ISC'23, we implemented a separation of the IO500 list into Production and Research lists. This better reflects the important distinction between storage systems that run in production environments and those that may use more experimental hardware and software configurations. At ISC'24, we continue to populate these two lists and users will be able to submit to either of the two lists (and their 10 client-node counterparts). Please see the requirements for each list and the IO500 submission rules page.

Reproducibility Questionnaire

Every submission will now receive a reproducibility score based upon the provided system details and the reproducibility questionnaire. This score will inform the community on the amount of details provided in the submission and the obtainability of the storage system. Further, this score will be used to evaluate if a submission is eligible for the Production list.

Submission Instructions

There are two options to submit to the IO500, we strongly prefer the online form:

  1. Use our online form. Users can create accounts and then update and manage all of their submissions through that account.
    1. Click on "[Register]" to create a new account
    2. You will receive an email to activate your account (also check your spam folder)
    3. Once your account is activated you can [Login]
    4. Go to the "[New Submission]" menu
    5. Complete the submission form with all required information
      • You will be able to incrementally complete the submission (i.e., your progress will be saved)
      • You are able to edit your submission until you submit it
      • Once you double-check your submission data, click on "[Submit]" to send it to the IO500 committee for review
    As part of this new tool, we have improved the submission fields that describe the hardware and software of the system under test. For reproducibility and analysis reasons, we now made the easily obtainable fields mandatory - data from storage servers are for users often difficult to obtain, therefore, most remain optional.

    As a new system, there may be quirks, please reach out on Slack or the mailing list if you see any issues.

  2. If you experience problems with the online form, please reach out on Slack to see if this can be resolved quickly. Alternately, send an email with the following as attachments:
    • The (potentially) adapted io500.sh
    • The output directory of the benchmark (variable io500_result_dir in io500.sh).
    • If possible, please mention which system is covered of the CDCL or provide system information such that we can add the system to the CDCL!

    We will reply to you, to confirm reception and to ask any question that may arise.

Handling of the Submitted Data

Until the next release of the list, the submission committee will handle all submitted data confidentially. That means that we will not disclose any submitted data to individuals/companies, or institutions.

Privacy

We will publish all data submitted, so by submitting the information you give us the right to publish the uploaded data.

Submitter Name

Submissions will be visible immediately to the members of the IO500 Steering Committee. If there is sensitivity about early visibility to your results being seen by any of the committee, please feel free to email results privately to a subset of the committee (i.e. do not use the official submission tools if you have privacy concerns).

Starting with the SC'18 list, submissions include the name of the submitter (or team) to give them the credit they deserve to execute the benchmark; this can be opted out.

Annonymity

With the online form, the submitter is able to individually opt-out the submission the name of the submitter/team (this will be then an anonymous submission).

Previous Calls for Submissions