outlier

Chapter 1: Introduction

  • Overview of Project

    • Introduction to the project by Laurel and Son.

    • Objectives: Understanding the project's goals and how to produce high-quality tasks.

    • Importance of context: Knowing how the customer's use of data affects task completion.

  • Role of Attemptors

    • Each attemptor works on the same task with the same prompt.

    • Example task: Finding the closest prime number to a given math operation.

    • Attemptors must provide a concise final answer and a step-by-step hint.

Chapter 2: A Correct Answer

  • Contribution Overview

    • Example answers from four contributors (a, b, c, and d).

      • Contributor a: 241 and 239

      • Contributor b: 233

      • Contributor c: Similar to contributor a but phrased differently.

      • Contributor d: Incomplete answer (just 241).

  • Hint Contribution

    • Hints are simplified but crucial for guiding the model without revealing the final answer.

    • Contributor b provided an incorrect answer, requiring scrutiny.

Chapter 3: The Review Process

  • Role of Reviewers

    • Reviewers must analyze submissions from different contributors.

    • Aim to standardize identical answers and fix incorrect contributions:

      • Ensure diversity in hints while maintaining a consistent final answer.

    • Importance of diversity in submitted hints for effective model training.

    • Customer's future use of consolidated tasks for model training.

Chapter 4: Quality Task Creation

  • Guidelines for Attemptors

    • Provide the correct final answer in its simplest form.

    • Quality hints are essential; should be detailed yet not obvious, avoiding spoilers.

    • Contributors are encouraged to think creatively to enhance diversity in hints.

Chapter 5: Reviewer Expectations

  • Reviewer's Responsibilities

    • Ensure accuracy across all tasks, including handling batches as a whole.

    • Fix tasks that are incorrect and encourage diversity in hints.

    • Handle situations where no correct answers exist and manage time accordingly.

Chapter 6: Effective Feedback

  • Importance of Constructive Feedback

    • Reviewers should not only point out mistakes but also explain them.

    • Emphasize proper grammar and clarity in hints to ensure model training effectiveness.

    • Reviewers should guide models step-by-step without revealing solutions directly.

Chapter 7: Requirements for Reviewers and Attemptors

  • Responsibilities of Attemptors

    • Attemptors should aim for high-quality submissions to minimize reviewer workload.

    • Identify and label impossible or nonexistent prompts correctly; mark them as unsolvable.

    • Provide guidance on how to approach unsolvable problems.

Chapter 8: Conclusion

  • Final Remarks

    • Stress accurate mathematical expression to avoid quality issues.

    • Encourage selectivity in choosing tasks to maintain overall quality in submissions.

    • The importance of clear, non-spoiling explanations in hints.

    • Offer to address questions in the discussion channel and encourage engagement.