Requirements and Test Coverage Realignment

Two major changes are being introduced, affecting both the requirements and test coverage sections.

The data used for agents was producing false negatives due to tests relying on specific configurations not mentioned in the requirements.
Example Problem Statement: Ineffective multiple slicing on search objects lacks user notifications.
Requirement Example: The system should ensure users are aware of ineffective multiple slicing operations and should communicate this limitation as temporary, pending future enhancements. It must also maintain compatibility with existing search object functionality.

A basic PR involves implementing a warning when a user tries to perform multiple slices.

The initial test checks for a deprecation warning with a specific message:
$DeprecationWarning$
$message = 'slicing multiple times' + …$

The model could implement the warning as a FutureWarning or UserWarning, which are valid but will fail the test because the test specifically looks for a DeprecationWarning.
Even if the agent picks DeprecationWarning, a slightly different warning message (e.g., "You are unable to slice multiple times") will cause the test to fail because it does not match the expected string.
The tests are too specific and are causing false negatives.

The requirements must now take into account test expectations.
When writing requirements, it's essential to examine what the tests are expecting and ensure the requirements provide adequate information for the golden patch to pass the unit tests.
If a warning message is implemented in the PR, it needs to be described in the requirements, including specific configurations expected by the tests.
For the example given, the requirements need to specify that a DeprecationWarning is expected, and the message must start with "slicing multiple times."
The documentation should explicitly describe all behaviors required to pass the test patch, including any specific functions invoked by the test.
Every test behavior must be clearly addressed in either the problem statement or the requirements.
The problem statement and requirements should complement each other; if the problem statement is vague, the requirements should be very specific, and vice versa.

A new specificity rubric is introduced to rate the problem statement and requirements combined.
Goal: Aim for a specificity rating of 5, where it is well-specified, and it is clear what is required for a successful solution.
The rating considers both the problem statement and the requirements.
Specificity is the key to pass the automated tests.

If a test imports a specific constant variable, the requirements must name that variable.
If there are multiple tests, explain each one if they have distinct configurations; otherwise, describe the different scenarios that should be tested.
Over-specificity is generally acceptable.
Mention any files or functions being targeted in the test patch in either the requirements or the rationale.
Only modified/created tests are considered, not audit tests.

A problem statement might have a specificity of 3, while the requirements have a specificity of 5. This is acceptable, as the goal is for the combination to achieve a specificity of 5.
The focus is on whether the combined context provides everything needed to satisfy the unit tests.
Still describe the behavior of the solution, not just the unit tests.
The detailed requirements must ensure the tests are accurately passed.

Mention YAML and Markdown files in the requirements only if their behavior or data is used in the golden patch or test behavior.
If a YAML file contains mock user data used in the test suites, it should be included in the requirements because it is a key configuration for the test to pass.

Providing specific configurations needed to pass the test is not considered leaking the solution.
Avoid step-by-step instructions on how to implement the golden patch.
Specific configurations that cannot be inferred should be included in the requirements to avoid false negatives.

There is no strict rule that you need to name the functions in the requirments, unless they are being imported into test files.
It is important to name them when they are being imported into test files because the test suite needs to identify the newly implemented component.
A lack of a function name is not enough to make something underspecific unless it's a new function or a new class that wasn't created earlier.

Emphasis on defining broad and narrow test types and how this affects test coverage rating.

Example Golden Patch: Create a new public interface, invalidate_sessions, which is going to invalidate the current sessions.
A broad test lacks specificity and does not specifically test the changes made by the golden patch.
These lead to false positives.
Improving broad unit tests is not within the scope of this project.

Example

To fix the narrow test include specific deprecation warnings. The message it writes out should start with "slicing multiple times”.
If tests are too generic, they will pass, even if the solution does not fix the bug (false positive).
The new specifications in requirements aim to properly pass tests rather than cause false negatives because requirements didn't speak about configuration.