Managing Duplicate Scoring Formulas

A duplicate scoring formula sets the duplicate score for a pair of articles in Duplicates and Conflicts jobs. A formula weights four components (title, article dimensions, topics, and question-answer pairs) and sets the thresholds used to flag two articles as duplicates or identical. The default formula applies to new jobs unless a job uses another formula.

Formulas List

The Duplicate Scoring tab lists all formulas in your partition. The table includes the following columns:

  • Formula: The formula name. The built-in formula displays the System tag, and the current default formula displays the Default tag.

  • Title, Article Dimensions, Topics, and Q&A: The weight assigned to each component.

  • Duplicate Threshold: The score at or above which a pair is flagged as a duplicate.

  • Identical At: The score assigned when two articles match exactly.

  • An Options (...) menu with the following actions:

    • Open: View or edit the formula.

    • Delete: Remove the formula. You cannot delete the system formula.

Scoring Components

Each formula scores article pairs using the following components. The weights assigned to the four components must always total 100%.

  • Title: Compares the titles of the two articles. Dates or version numbers that are incidental to the subject (for example, "Maintenance Guide (April 2018)") are ignored during comparison, while dates or versions that are part of the subject itself (for example, "Annual Report 2024") are kept and compared.

  • Article Dimensions: Compares the article dimensions assigned to each article. The more dimension values two articles share, the higher their article dimensions similarity.

  • Topics: Compares structured details extracted from each article, such as product, module, region, policy, intent, training, and audience. The more of these details two articles share, the higher their topic similarity.

  • Q&A: Compares the questions and answers extracted from each article. Articles with the same or similar questions and matching answers are treated as duplicate content. Articles with the same or similar questions but different answers are flagged as a conflict.

Threshold Fields

  • Duplicate threshold: The similarity score, as a percentage, at or above which two articles are considered duplicates of each other.

  • Identical score: The similarity score assigned when the content of two articles matches exactly.

System Default Formula

Every partition includes a System default formula, identified by the System tag. It applies the following weights: Title 20%, Article Dimensions 10%, Topics 10%, and Q&A 60%, with no minimum duplicate threshold and an identical score of 100%. You cannot delete the system default formula, but you can create custom formulas alongside it. If no custom formula is set as the default, the System default formula is used.

Creating a Formula

To create a duplicate scoring formula:
  1. On the Duplicate Scoring tab, select + Create Formula, or select + (Create) > Duplicate Scoring Formula.
  2. In the Add Formula dialog, enter a Formula name.
  3. Under Scoring Weights, enter a weight for Title, Article Dimensions, Topics, and Q&A so that the four values total 100%. The total shown below the fields updates as you type. Select Use default weights to fill in the System default weights.
  4. Enter a Duplicate threshold and an Identical score.
  5. Optionally, expand Advanced and enter Additional Rules to give the AI extra instructions for how this formula scores articles. This field has a 2,000-character limit.
  6. Select Use as the default scoring formula if you want new Duplicates and Conflicts jobs to use this formula. Only one formula can be the default at a time. Selecting this option for a formula clears it for the formula that was previously the default.
  7. Select Save.

To edit a custom formula, select Open from its Options (...) menu.

Applying a Formula to a Job

The default formula applies to new Duplicates and Conflicts jobs. To rescore the findings of an existing job with a different formula, use the Scoring formula control on the job's Duplicates & Conflicts findings list. You can preview the new scores before you apply the formula, and the job keeps a scoring history of each formula used. For more information, see Changing the Scoring Formula for a Job.

Access to the Duplicate Scoring tab is subject to your assigned Knowledge Base permissions.

Related Topics