Linting English: rules before machine learning

Last year I wrote that a requirement is a test nobody runs, and that I wanted the tool to complain at the moment a bad sentence is written. This is what happened when we built that.

The task: given a requirement in English, say whether it is well formed, and if not, say why. The obvious 2015 answer is to train a classifier. We did not. I was tempted, and I am glad we resisted. I think the reasons apply outside requirements.

Why not a classifier

There was no training set. A classifier needs thousands of requirements labelled good or bad. Nobody had labelled them, the people qualified to do it are expensive, and they do not agree with each other on the borderline cases.

"Probably bad" is not a finding. An engineer who is told a requirement scored 0.71 cannot fix it. An engineer who is told that the word "appropriate" on column 34 makes the requirement unverifiable can.

The rules already existed. Requirements writing guides are lists of rules. Encoding a rule that a domain expert already believes is cheaper than learning it again from data, and nobody argues with the result.

So the design is a linter. Each rule is small, has an identifier, points at a span of text and explains itself.

Three tiers of rule

Word lists. Vague terms (adequate, sufficient, user-friendly), escape clauses (if possible, as appropriate), open lists (etc., and so on). Trivial to implement, and they catch a surprising share of real problems.

Part-of-speech patterns. A requirement needs a modal verb, and exactly one. "Shall" appearing twice usually means two requirements. A passive with no agent ("the data shall be validated") hides who is responsible. A tagger gets these right nearly always, because requirements are written in a narrow, repetitive style.

Structure from a parse. With a dependency parse you can ask better questions: what is the subject of the main verb, is there a condition, does the condition come with a measurable value and a unit? "Within 2 seconds" passes. "In a timely manner" does not.

The third tier is where it stops being a word search. It is also where the errors come from, because a parser trained on newspaper text is not always right about a sentence full of part numbers and acronyms.

Precision matters more than recall

This was the main lesson, and I did not see it coming. A linter that is wrong one time in five gets switched off, and then it catches nothing.

So every rule had to earn its place on real requirements. If a rule fired wrongly too often, we narrowed it or dropped it, even when the idea behind it was good. We ended up with fewer rules than we started with, each one boring and almost always right.

Two decisions helped:

• Severity. A missing modal verb is an error. A long sentence is a warning. Mixing the two teaches people to ignore both.

• Suppression with a reason. Sometimes the engineer is right and the rule is wrong. They can silence it for one requirement, but they have to write why. The reasons became our best source of rule fixes.

Where statistics did help

Not for the verdict, but for finding candidates. Requirements that say nearly the same thing in different words are a real problem (two teams, one need, two implementations). Comparing requirements as weighted bags of words and listing the closest pairs gave reviewers a short list to look at. A person still decides. The machine only says where to look.

That split is the part I would reuse: rules for anything that must be explained, statistics for ranking what a human will review anyway.

What I would tell myself a year ago

Start with the dumbest rule that produces a true finding, and ship it. The word list took a day and was useful that week. The parser took months and added less than I expected, although the things it added could not have been done any other way.

And the learning approach is not wrong, it is early. If the tool keeps every finding, every suppression and every reason, then in a few years there will be a labelled dataset, made by experts, as a side effect of normal work. That would be the time to train something. I may be wrong about how long it takes. I would be surprised to be wrong about the order.