Measuring Effectiveness of Capture Solutions - Key Metrics

Many of our clients are trying to take advantage of advanced recognition techniques for data extraction in the capture of paper documents, images and faxes. Recent advances with artificial intelligence, machine learning, computer vision, etc. have dramatically improved the ability to automate capture – across a much broader spectrum of document types and including both machine and handprint.

Having numerous advanced recognition solutions on the market to choose from is great, but measuring effectiveness can be tricky since there are varying definitions of success. Before we dive into the details of what to measure, it’s important to understand the recognition process as a whole.

The Primary Steps in Recognition

There are three primary steps in the process where automation can be applied:

  • Classification, which is identifying what pages, what documents, and what document packages the system is getting. It may also include separating different document types from each other, eliminating extraneous pages and documents, and pagination, putting the pages in desired order. 
  • Extraction, which is identifying the data that’s on the pages, and possibly enriching it. Extraction is typically determining the string of characters that are in each of the relevant fields. Note that this makes field accuracy more important than simple character accuracy. If a system gets 10 characters wrong but they are in the same field, that’s just a wrong field. But if the 10 characters are in 10 different fields, that’s 10 fields wrong, requiring a lot more error detection and correction effort – and not a lot of cost savings.
  • Validation, which is checking for the completeness and correctness of the information and document sets. Validation checks that classification and extraction were performed correctly and that the documents are in good order. This can be a simple or terribly complex step that is very application-dependent. In any case, we are applying business rules to the data that has been extracted to ensure it’s appropriate for passing to the downstream systems.

Defining the Criteria to Measure Success

To help our clients determine if the advanced capabilities introduced into the market recently will be effective, it is critical to define specifically the criteria to measure success.

For classification, we might have 20 different document types: 10 one-page documents, five ten-page documents, and five documents requiring three sub-documents, each with two pages.

  Doc Type # of Pages # of Sub Docs
Sample #1 A 1 0
Sample #2 B 1 0
Sample #11 K 5 0
Sample #16 Z 6 3
Etc.      

In this case, accuracy would be measured for each metric – Doc Type, # of Pages, and # of Sub Docs. If we ran a proof of concept with 100 samples, we typically calculate four different accuracy metrics one for each of the above, and then an overall classification metric, which would be the combined metrics (for example: 98% doc type accuracy x 96% page identification accuracy, 92% sub-document identification = 86% overall.

For extraction, the process is similar. We track character, field, and document level accuracy.

  Doc Type # of Fields # of Characters
Sample #1  A 10  100 
Sample #2 70 
Sample #11 20  250 
Etc.      

For validation, the process always focuses on Field and Document level accuracy.

  Doc Type # of Fields
Sample #1  A 10 
Sample #2
Sample #11 20 
Etc.    

Again, with measurement, it’s important to determine if we are measuring accuracy at the document, fields, or character level. Across the population of 100 samples, we may get 99% accuracy at the character level, but if there is one wrong character in each field, the field-level accuracy would be 0%.

Here are some additional points to consider:

  • What kind of accuracy you’ll get is heavily dependent on the specific documents and document types you’re running through the system. Be sure to get a representative set of samples for testing.
  • Accuracy is not the only metric critical to estimating success. The other metric is read rate, the amount of data produced from the software (versus accuracy, the subset of that produced data that is correct). If you can only do classification for 75% of a relevant population of documents, and you get 90% accuracy on those documents, then your effective accuracy is not 75% but 67.5%.
  • Be careful to define what is an error. Is it a wrong character, capitalized or not, wrong punctuation, wrong format of the account number

Conclusion

This post is a primer to understand capture metrics, which is particularly helpful when comparing capture vendors. Given the complexity and critical nature of a capture assessment, many of our clients come to us for assistance in benchmarking capture providers or conducting proofs-of-concept. We’re happy to help if you need us.

How to Select Enterprise Software

Rich Medina
James Watson
I’m President and co-founder of Doculabs, serving as executive sponsor on consulting engagements for financial services clients.