Skip to content

Difficulty-weighted scoring, field by field

Two settings, one published formula

StudyDrome Exam Manager can weight each question by its difficulty. The weight ranges from 0.4 for the easiest item to 1.0 for the hardest, and the weighted result is scored out of 1000. A candidate then clears two thresholds instead of one. The model is off until you enable it.

Book a pilot

What is difficulty-weighted scoring?

A written paper usually treats every point the same. Ten points for a hard item, ten for an easy one.

Difficulty-weighted scoring changes that. Each item contributes in proportion to the difficulty its author set. A hard question is worth more of the paper than an easy one.

The model is off until you enable it. Enable it and two things change. Points are assigned a weight, and the pass rule receives a second test.

The difficulty here is the author's own 0-10 label, chosen while writing the item. The eleven question types cover that scale and its four named bands. The measured p-value is an entirely different number, and it is determined afterward in item analysis.

How is each question weighted?

Each item carries a weight based on its difficulty. The formula is published, and two settings drive it.

weight = Weight Minimum + (1 − Weight Minimum) × (difficulty ÷ 10) ^ Gamma

Weight Minimum is the floor. At 0.4, the easiest item is still worth 40% as much as the hardest. Raise it, and the paper flattens. Lower it and easy items count for less.

Gamma is the curve. At 1.0 the weight climbs in a straight line. Above 1.0, the hardest items pull further ahead of the rest.

At the defaults, the formula reads: weight = 0.4 + 0.6 × (difficulty ÷ 10).

Difficulty

Weight

0

0.40

2

0.52

5

0.70

8

0.88

10

1.00

An item marked 0 takes the floor. An item marked 10 takes the full weight. Everything between follows the curve you set.

What is the weighted score out of?

The weighted result is a share of the paper, then a score out of 1000.

weighted share = total of (earned × weight) ÷ total of (available × weight)

That share is multiplied by 1000 and rounded. A share of 0.581 becomes 581.

Here is a three-question paper, worth 10 points each, at difficulties 2, 5, and 8. Their weighted maximums are 5.2, 7.0 and 8.8, so the paper is worth 21.0 weighted points.

Candidate

Answers correctly

Weighted points

Weighted score

Raw mark

A

The 2 and the 5

12.2

581

20 / 30

B

The 5 and the 8

15.8

752

20 / 30

Both scored 20 out of 30. The weighted score separates them because it asks where the marks came from rather than how many there were.

How does a candidate pass?

With weighting enabled, a pass has more than one condition. Every one of them must hold.

Condition

Default

The raw percentage clears the nominal threshold

0.60

The weighted score clears the weighted threshold

600 out of 1000

Every domain minimum is met

None set

Assessment people call this a conjunctive standard. Each hurdle counts on its own, and a strong total cannot carry a weak part.

The two thresholds do different jobs. The nominal one is the pass mark everybody already knows. The weighted one asks which questions the marks came from. Candidate A above clears the first and misses the second.

How do you require a minimum in one subject?

A domain is a tag with a minimum attached. Name the tag, set a minimum percentage, and that slice of the paper becomes its own hurdle. Miss it, and the candidate fails, no matter what the total says. The wizard calls these Must-Pass Domains, and one exam can carry several.

Tags do the addressing, so a domain is any dimension you already classify by: a subject, a body system, a professional standard. Tagging and clinical skills cover how tags are built.

One rule is worth knowing before you rely on it. A domain is judged only when the paper holds questions carrying that tag, and those questions carry points. A domain that finds nothing is skipped. It is never counted as a failure.

That is deliberate. Blueprints change, and a form assembled from a different bank can leave a domain matching nothing. A candidate should not fail over questions that were never on their paper.

Where do you set all of this?

Scoring policy is step seven of the exam wizard, under the heading Scoring Policy. Six fields carry the whole model.

Field

What it does

Default

Enable difficulty-weighted scoring

Turns the model on for this exam

Off

Weight Minimum

The floor weight, for the easiest item

0.4

Gamma

The curve from easy to hard

1.0

Nominal Pass Threshold

The raw share of marks needed

0.60

Weighted Pass Threshold

The weighted score needed, out of 1000

600

Must-Pass Domains

A tag and a minimum percentage, repeated as needed

None

Save the exam as a template and the scoring policy travels with it. It is one of the ten configuration sections a template carries, so a department sets the model once and applies it to every paper.

What freezes at publish covers which settings stay editable once an exam is live.

Notes for a reviewer

A scoring model is only defensible if somebody outside the vendor can check it. These are the properties that make that possible.

  • The weight formula is published so that any weighted score can be rebuilt by hand from the paper.

  • Two settings control the whole curve, and both appear in the wizard with their defaults on screen.

  • The model is off unless an exam turns it on. An exam without it scores exactly as it always did.

  • Difficulty is the author's label. The measured p-value is a separate figure, reported in item analysis.

  • A domain minimum that matches no question on the paper is skipped and never counted as a failure.

  • The scoring policy is saved in a test template, so a department applies one model to every paper.

Frequently asked questions

Does difficulty weighting change who passes?

It can, and that is the point. Two candidates on the same three-question paper both score 20 out of 30. The one who answered the harder questions reaches 752 out of 1000 weighted; the one who answered the easier ones reaches 581. At the default weighted threshold of 600, one clears it, and one does not.

What do the two pass thresholds mean?

The nominal threshold is the familiar pass mark, taken on the raw share of marks, and it defaults to 0.60. The weighted threshold is taken on the weighted score out of 1000, and it defaults to 600. A candidate must clear both, along with every domain minimum set by the exam.

How is a question's weight worked out?

Weight equals the Weight Minimum, plus one minus that minimum, multiplied by the difficulty divided by ten and raised to Gamma. At the defaults of 0.4 and 1.0, that gives 0.4 for an item marked 0, 0.70 for one marked 5, and 1.00 for one marked 10. Both settings are yours to change.

What happens if a domain minimum matches no questions on the paper?

It is skipped rather than failed. A domain is judged only when the paper actually contains questions with that tag, and those questions carry points. Blueprints change, and forms are assembled from different banks, so a candidate is never failed over a domain that had no questions on the paper they sat.

Can a weighted score be checked by hand at an appeal?

Yes. The formula is published, both settings are recorded on the exam, and every question carries its own difficulty and points. Take the paper, apply the weights, and the number reconciles. That is why the model is a transparent formula with two settings you choose rather than a statistical model you would have to trust.

Book a pilot

Send us a past paper with its difficulty set. We will score it both ways in a sandbox and show you which candidates the weighting moves.

Book a pilot

Related