Notes on Scoring and Ranking Methods

SCORING AND RANKING

INTRODUCTION TO SCORING AND RANKING

  • Scoring assigns numerical values to entities based on specific criteria.

  • Ranking arranges entities according to their scores.

  • Common applications include:

    • Search engines

    • AI models

    • Performance evaluations

SIMPLE NUMERICAL SCORING

  • Involves assigning direct numerical values based on predefined criteria.

  • The formula for scoring is expressed as:

    Score = Σ (criterion value)

  • The final score is obtained by summing up individual criterion values.

  • Example:

    • ML Model: Accuracy = 80, Precision = 75, Recall = 70

    • Total Score = 80 + 75 + 70 = 225

  • This method is useful for basic evaluations but does not weight important factors accordingly.

WEIGHTED SCORING METHOD

  • This method assigns different weights to each criterion based on its importance.

  • The scoring formula is as follows:

    Score = Σ (Weight_i * Criterion Value_i)

  • Ensures more weight is placed on significant parameters, leading to more accurate evaluations.

  • Example:

    • Accuracy (50%) = 80, Precision (30%) = 75, Recall (20%) = 70

    • Final Score = (0.580) + (0.375) + (0.2*70) = 76.5

  • This method effectively highlights important metrics but requires subjective decisions regarding weight assignments.

BORDA COUNT METHOD

  • A ranking method that assigns points based on preference positions.

  • The score is calculated as:
    Score = Σ (Rank Weight * Votes)

  • Higher-ranked choices receive more points in a preference-based system.

  • Example: Three AI models evaluated by five judges:

    • Model A: Ranked 1st, 2nd, 3rd, 2nd, 1st → Points: 5 + 4 + 3 + 4 + 5 = 21

    • Model B: Ranked 2nd, 3rd, 1st, 1st, 3rd → Points: 4 + 3 + 5 + 5 + 3 = 20

    • Model C: Ranked 3rd, 1st, 2nd, 3rd, 2nd → Points: 3 + 5 + 4 + 3 + 4 = 19

  • Model A wins with 21 points, making this method useful in voting-based AI ranking models.

PAGERANK ALGORITHM

  • Google's algorithm for ranking web pages, focusing on the importance of sites based on incoming links.

  • Example:

    • If Page A is linked to by Page B and Page C, which have different importances, Page A's rank is affected by the quality and number of these links.

ELO RANKING SYSTEM

  • This system dynamically adjusts rankings based on match results.

  • The formula used is:
    •R(A) = R_old(A) + k * (SA - EA)

  • Where:

    • k is a fixed parameter for maximum score adjustment per match.

    • A smaller k leads to more static rankings.

    • A larger k results in rapid swings in rankings.

    • S_A = Scoring result for player A (1 = Win, 0 = Loss).

    • E_A = Expected result:

    • For players of equal skill level, E_A = 0.5.

    • For a higher skill level, E_A > 0.5 (closer to 1).

ELO RANKING EXAMPLE

  • Consider a Chess AI Model rated 2000 against a Human Player rated 1800:

    • If the AI Model wins:
      R_{AI} = 2000 + 32 * (1 - 0.76) = 2008

    • If the AI loses:
      R_{AI} = 2000 + 32 * (0 - 0.76) = 1976