Class: Vangrail::Evidence

Inherits:
Struct
  • Object
show all
Defined in:
lib/vangrail/evidence.rb

Overview

One rail's measured operating point, read as evidence rather than as a verdict.

Every rail in this gem answers a yes-or-no question, and the engine combines those answers by taking the first "yes". That is what the published defences do too, and it throws away almost everything the rails know. It cannot say how much a hit is worth, cannot add up three near misses, and cannot tell an operator what a block actually means about the text.

What a hit is worth is a ratio, and it is measurable: how much likelier this rail is to fire on an attack than on ordinary documentation. That number, the likelihood ratio, is all a rail needs to contribute to a shared judgement, and it is exactly what the corpora in this repository already measure. Every entry here comes from running a rail over the same attack and benign sets as every other rail, which is what makes the numbers comparable in the first place: a detection rate measured on one paper's corpus and a false-positive rate measured on another's cannot be combined at all.

Rates are smoothed with the Jeffreys prior, (hits + 1/2) / (n + 1), for a reason that is not decoration. A rail that caught 60 of 60 has an unsmoothed detection rate of exactly 1, an unsmoothed likelihood ratio of infinity, and would single-handedly decide every judgement it appears in on the strength of a sixty-item corpus. Smoothing keeps the evidence finite and proportional to how much was actually measured.

Instance Attribute Summary collapse

Instance Method Summary collapse

Instance Attribute Details

#attacks ⇒ Object

Returns the value of attribute attacks

Returns:

  • (Object) —

    the current value of attacks



30
31
32
 
# File 'lib/vangrail/evidence.rb', line 30

def attacks
  @attacks
end

#attacks_caught ⇒ Object

Returns the value of attribute attacks_caught

Returns:

  • (Object) —

    the current value of attacks_caught



30
31
32
 
# File 'lib/vangrail/evidence.rb', line 30

def attacks_caught
  @attacks_caught
end

#benign ⇒ Object

Returns the value of attribute benign

Returns:

  • (Object) —

    the current value of benign



30
31
32
 
# File 'lib/vangrail/evidence.rb', line 30

def benign
  @benign
end

#benign_flagged ⇒ Object

Returns the value of attribute benign_flagged

Returns:

  • (Object) —

    the current value of benign_flagged



30
31
32
 
# File 'lib/vangrail/evidence.rb', line 30

def benign_flagged
  @benign_flagged
end

#group ⇒ Object

Returns the value of attribute group

Returns:

  • (Object) —

    the current value of group



30
31
32
 
# File 'lib/vangrail/evidence.rb', line 30

def group
  @group
end

#rail ⇒ Object

Returns the value of attribute rail

Returns:

  • (Object) —

    the current value of rail



30
31
32
 
# File 'lib/vangrail/evidence.rb', line 30

def rail
  @rail
end

Instance Method Details

#bits(fired, confidence: nil) ⇒ Object

Evidence in bits, positive towards attack. Bits rather than nats because an operator has to read these: one bit is a doubling of the odds, and "this rail is worth four bits" is a sentence somebody can act on.

With a confidence, the bits are what the corpus can defend at that level rather than what it measured. A table built from a few hundred texts should be read this way; the point estimate is what it would say if the corpus were the world.



82
83
84
85
86
87
88
 
# File 'lib/vangrail/evidence.rb', line 82

def bits(fired, confidence: nil)
  return Math.log2(fired ? ratio_fired : ratio_silent) if confidence.nil?

  detection = detection_bound(confidence)
  false_alarm = false_alarm_bound(confidence)
  Math.log2(fired ? detection / false_alarm : (1 - detection) / (1 - false_alarm))
end

#capability(prior:) ⇒ Object

How much of the question this rail actually answers, at a given base rate.

Detection and false-alarm rates describe a rail; they do not describe what it is worth in a deployment, because they say nothing about how often the thing being detected happens. The intrusion-detection literature settled this with an information-theoretic measure: the fraction of the uncertainty about "is this an attack" that the rail's verdict removes.

Measured on the shipped table, the base rate costs every rail roughly two fifths of its capability between a balanced corpus and one attack in ten thousand: paraphrase falls from 0.52 to 0.28, and every other rail sits under 0.1 at both. many_shot manages 0.001, which is the honest reading of a rail that caught six of 270 because the corpus is mostly not its attack.

Ranking rails by this rather than by detection rate is the point. It is the only number here that changes when the deployment does.



106
107
108
109
110
 
# File 'lib/vangrail/evidence.rb', line 106

def capability(prior:)
  return 0.0 unless measured?

  mutual_information(prior) / entropy(prior)
end

#detection ⇒ Object

Probability the rail fires given the text is an attack.



33
34
35
 
# File 'lib/vangrail/evidence.rb', line 33

def detection
  (attacks_caught + 0.5) / (attacks + 1.0)
end

#detection_bound(confidence) ⇒ Object

The rates a corpus this size can actually defend, rather than the ones it happens to have produced.

A rail that fired on none of 48 benign texts has a point estimate of one in a hundred and a 95% upper bound of one in twenty-six. The difference is two bits of evidence that nobody measured, and reporting the point estimate spends them.

Pessimistic on both sides at once: detection at the low end of its posterior and false alarms at the high end. That single operating point is conservative for a hit and for silence alike, because both ratios move the same way under it.



66
67
68
 
# File 'lib/vangrail/evidence.rb', line 66

def detection_bound(confidence)
  Beta.quantile(1 - confidence, attacks_caught + 0.5, attacks - attacks_caught + 0.5)
end

#false_alarm ⇒ Object

Probability it fires given the text is ordinary.



38
39
40
 
# File 'lib/vangrail/evidence.rb', line 38

def false_alarm
  (benign_flagged + 0.5) / (benign + 1.0)
end

#false_alarm_bound(confidence) ⇒ Object 



70
71
72
 
# File 'lib/vangrail/evidence.rb', line 70

def false_alarm_bound(confidence)
  Beta.quantile(confidence, benign_flagged + 0.5, benign - benign_flagged + 0.5)
end

#measured? ⇒ Boolean

A rail that never fired on either corpus has measured nothing, whatever its detection rate looks like after smoothing.

Returns:

  • (Boolean) 


123
124
125
 
# File 'lib/vangrail/evidence.rb', line 123

def measured?
  attacks.positive? && benign.positive?
end

#ratio_fired ⇒ Object

How much likelier a hit is on an attack than on ordinary text.



43
44
45
 
# File 'lib/vangrail/evidence.rb', line 43

def ratio_fired
  detection / false_alarm
end

#ratio_silent ⇒ Object

And how much likelier silence is on ordinary text than on an attack. This is the half that OR-combination cannot express at all: a sensitive rail staying quiet is evidence too, and it points the other way.



50
51
52
 
# File 'lib/vangrail/evidence.rb', line 50

def ratio_silent
  (1 - detection) / (1 - false_alarm)
end

#to_bits_h(prior:, confidence: nil) ⇒ Object 



112
113
114
115
116
117
118
119
 
# File 'lib/vangrail/evidence.rb', line 112

def to_bits_h(prior:, confidence: nil)
  {
    'rail' => rail,
    'bits_if_fired' => bits(true, confidence: confidence).round(2),
    'bits_if_silent' => bits(false, confidence: confidence).round(2),
    'capability' => capability(prior: prior).round(4),
  }
end

#to_h ⇒ Object 



127
128
129
130
131
132
133
134
135
 
# File 'lib/vangrail/evidence.rb', line 127

def to_h
  {
    'rail' => rail, 'group' => group,
    'attacks_caught' => attacks_caught, 'attacks' => attacks,
    'benign_flagged' => benign_flagged, 'benign' => benign,
    'detection' => detection.round(4), 'false_alarm' => false_alarm.round(4),
    'bits_if_fired' => bits(true).round(2), 'bits_if_silent' => bits(false).round(2)
  }
end