Class: Vangrail::Rails::Bayes

Inherits:
Vangrail::Rail show all
Defined in:
lib/vangrail/rails/bayes.rb

Overview

A naive Bayes classifier over word n-grams, which is the oldest working text classifier there is and the only rail here that says how sure it is.

Every other rail answers yes or no, so it hands the evidence arithmetic exactly one bit of information however certain it was. This one computes a log-likelihood ratio directly, which is the quantity that arithmetic actually wants: a clause scoring twelve bits and a clause scoring three both "fire", and they are not the same observation. A rail that puts bits in its result's raw is read that way by Engine#assess, and this is the first rail to do it.

Scored clause by clause, taking the worst. The same dilution problem the containment rail hit applies here with force: an attack document is ordinary documentation with one injected sentence in it, and a bag of features over the whole page is mostly evidence about the handbook.

The shipped weights are a demonstration, and the honest number says so. Cross-validated over five folds on 48 attack clauses and 56 benign ones, at a threshold no held-out benign document reached, it catches 15 of 48 attacks: worse than the lexicon rails, which catch three quarters. The reason is the corpus rather than the method. Forty-eight training clauses written to be varied share almost no vocabulary with the held-out ones, and the junk-mail filters this borrows from were fitted on millions of examples.

So it is off by default, and what it is for is the retraining path: a deployment with its own traffic runs script/train_bayes.rb against its own corpus and gets a rail fitted to the attacks it actually receives, with a cross-validated number attached rather than a promise.

Instance Attribute Summary collapse

Instance Method Summary collapse

Constructor Details

#initialize(weights: BayesData::WEIGHTS, threshold: BayesData::THRESHOLD, calibration: BayesData::CALIBRATION, name: 'bayes', sides: %i[input context])) ⇒ Bayes

Returns a new instance of Bayes.



39
40
41
42
43
44
45
 
# File 'lib/vangrail/rails/bayes.rb', line 39

def initialize(weights: BayesData::WEIGHTS, threshold: BayesData::THRESHOLD,
               calibration: BayesData::CALIBRATION, name: 'bayes', sides: %i[input context])
  super(name: name, sides: sides)
  @weights = weights
  @threshold = threshold
  @calibration = calibration
end

Instance Attribute Details

#calibration ⇒ Object (readonly)

Returns the value of attribute calibration.



47
48
49
 
# File 'lib/vangrail/rails/bayes.rb', line 47

def calibration
  @calibration
end

#threshold ⇒ Object (readonly)

Returns the value of attribute threshold.



47
48
49
 
# File 'lib/vangrail/rails/bayes.rb', line 47

def threshold
  @threshold
end

#weights ⇒ Object (readonly)

Returns the value of attribute weights.



47
48
49
 
# File 'lib/vangrail/rails/bayes.rb', line 47

def weights
  @weights
end

Instance Method Details

#bits(text) ⇒ Object

What that score is actually worth, from the calibration fitted on held-out folds and read at the 95% bound. This is the number that goes into a posterior, and it is bounded by what 48 attack clauses can demonstrate rather than by how loudly the classifier scored.



96
97
98
99
100
 
# File 'lib/vangrail/rails/bayes.rb', line 96

def bits(text)
  score = score_for(text)
  band = calibration.reverse.detect { |floor, _| score > floor }
  band ? band.last : calibration.first.last
end

#cache_key(text, _context) ⇒ Object 



57
58
59
 
# File 'lib/vangrail/rails/bayes.rb', line 57

def cache_key(text, _context)
  "#{threshold}\n#{text}"
end

#call(text, _context) ⇒ Object 



61
62
63
64
65
66
67
68
69
70
71
72
73
74
75
 
# File 'lib/vangrail/rails/bayes.rb', line 61

def call(text, _context)
  score = score_for(text)
  evidence = bits(text)
  payload = { 'bits' => evidence, 'score' => score }
  # The middle calibration band has both classes in it (22 attacks and
  # 8 ordinary pages between 0 and the threshold). A score there is
  # not a decision. Above the threshold no held-out benign document
  # landed, and at or below 0 no held-out attack did.
  return unchecked('score sits in a band the calibration cannot separate', raw: payload) if overlap?(score)
  return pass(raw: payload) if score <= threshold

  block(categories: ['bayes'], raw: payload,
        reason: format('scores %<score>+.1f, worth %<bits>+.1f bits of evidence', score: score,
                                                                                  bits: evidence))
end

#offline? ⇒ Boolean

Returns:

  • (Boolean) 


49
50
51
 
# File 'lib/vangrail/rails/bayes.rb', line 49

def offline?
  true
end

#overlap?(score) ⇒ Boolean

Returns:

  • (Boolean) 


77
78
79
 
# File 'lib/vangrail/rails/bayes.rb', line 77

def overlap?(score)
  score.positive? && score <= threshold
end

#quantifies? ⇒ Boolean

Returns:

  • (Boolean) 


53
54
55
 
# File 'lib/vangrail/rails/bayes.rb', line 53

def quantifies?
  true
end

#score_for(text) ⇒ Object

The worst clause's raw naive Bayes score. Not a likelihood ratio and not to be added to one: the features are counted as independent and are not, so this number is confidently wrong about its own size. It decides the block, because a threshold only needs an ordering.



85
86
87
88
89
90
 
# File 'lib/vangrail/rails/bayes.rb', line 85

def score_for(text)
  clauses = NLP.clauses(text)
  return clause_score(text.to_s) if clauses.empty?

  clauses.map { |clause| clause_score(clause) }.max
end