{"id":3579,"date":"2020-11-04T15:01:30","date_gmt":"2020-11-04T23:01:30","guid":{"rendered":"https:\/\/tolstoy.ai\/?p=3579"},"modified":"2020-11-04T15:01:30","modified_gmt":"2020-11-04T23:01:30","slug":"bits-of-paper-how-digitized-is-your-organization","status":"publish","type":"post","link":"https:\/\/staging.tolstoy-ai.com\/?p=3579","title":{"rendered":"Parsing Scanned Images to Text: How Digitized is Your Organization?"},"content":{"rendered":"
\n
\n
\n

The world is digitizing its records. Remote work during the pandemic has only accelerated this trend, with workplaces realizing that the ability to physically access records is not a given. As we move into a new era of hybrid and remote work, the drive to convert hard-copy records to online versions \u2014 where staff can easily access, search, and analyze the data \u2014 is picking up steam.<\/p>\n

Paper to pixels<\/h1>\n

Many workplaces, especially in heavily\u00a0<\/span>regulated industries \u2014 government, energy\/utilities, accounting, law \u2014 keep their records for years. Currently, much of that is in paper form, with documents piling up over decades in cabinets, rooms, entire floors. Companies have to keep these copies long-term, whether mandated by law or out of pure necessity. But not only are hard-copy forms liable to be lost, damaged, or even forgotten \u2014 it\u2019s \u201chellish\u201d to retrieve them, as described by one U.S. EPA official who deals with public FOIA requests. People have to walk through rooms, digging through boxes and cabinets, physically reading forms to find what they want.<\/p>\n

\n

Data entry takes up 75% of the time and resources of a digitization project<\/p>\n<\/blockquote>\n

Digitization not a trivial task, though. Imaging the records is itself a daunting task when faced with millions of records. But that pales in comparison to transcribing the images. Data entry takes up 75% of the time and resources of a digitization project, which often runs into decades and several millions of dollars. But it\u2019s the key step that will actually unlock the data.<\/p>\n

\n
\n
\n
\n
\n
<\/div>\n
\"Image
Historically, humans are needed for data entry from unstructured images \u2014 but that is changing. Tolstoy has tagged this image without human intervention.<\/figcaption><\/figure>\n<\/div>\n<\/div>\n<\/div>\n<\/div>
<\/figcaption><\/figure>\n

<\/h1>\n

The power and limits of OCR<\/h1>\n

Optical character recognition, or OCR, is the main answer to these data entry challenges. It\u2019s the technology that recognizes and extracts text from raw images. Google famously digitized and OCR\u2019d millions of books from libraries around the world in the 2000\u2019s. Their technology is still used today, embedded in many OCR services.<\/p>\n

However, there\u2019s an important catch. Google\u2019s OCR, and that of many other providers out there, is built to read the text of a book \u2014 single column, top-down printed text. Introduce columns, fields, boxes, images, handwriting, and it struggles to comprehend it without further work. Yet complex, unstructured images are the bulk of the text data in the world \u2014 the iceberg\u2019s vast submerged heft.<\/p>\n

\n

Complex, unstructured images are the bulk of the text data in the world \u2014 the iceberg\u2019s vast submerged heft.<\/p>\n<\/blockquote>\n

And it\u2019s a frontier of interesting questions. How do you successfully transcribe century-old records with faded text? How do you search records for fields that variably occur anywhere in the image? What if a record included cursive handwriting \u2014 in another language?<\/p>\n

\"\"
Examples of digitization texts<\/figcaption><\/figure>\n

<\/h1>\n

AI-assisted digitization<\/h1>\n

These are questions Tolstoy has plumbed in projects for libraries, newspapers, and the private sector. We\u2019ve written about them\u00a0here<\/a>,\u00a0here<\/a>, and\u00a0here<\/a>. What we\u2019ve learned is that with the right AI models and training, it\u2019s not only possible \u2014 it\u2019s the best path forward. AI reduces the time and resources organizations need for digitization projects by orders of magnitude.<\/p>\n

For example, a prominent university needed 3\u20134 staff to spend several months processing a historical document set. It involved indexing thousands of pages of legal court transcripts. Normally, someone reads through the transcripts \u2014 human dialogue \u2014 to find and tag document references. After they got in touch with us, we developed and refined the right AI model to parse the 150,000+ pages of legal transcripts in an afternoon.<\/p>\n

In another project, a venerable newspaper was struggling to transcribe their historical broadsheets in time for their 130th Anniversary edition. Normally, someone would take months manually transcribing these papers. None of the OCR software the newspaper tried worked well, or at all. We wrote a custom OCR parser and returned all transcribed images within a week.<\/p>\n

The future is digital<\/h1>\n

Our goal is to reduce both the cost and time barriers to entry for organizations to truly digitize their archives. With true digitization, an organization has a much more complete and powerful view of their recorded projects, assets, and clients. They are able to instantly access, share, and compare records; analyze specific aspects of the contents; track their holdings; find what they seek.<\/p>\n

What is the state of digitization at your organization? And what thorny questions do you have about your text records that you\u2019ll like us to tackle next?<\/p>\n<\/div>\n<\/div>\n<\/section>\n","protected":false},"excerpt":{"rendered":"

The world is digitizing its records. Remote work during the pandemic has only accelerated this trend, with workplaces realizing that the ability to physically access records is not a given..<\/p>\n","protected":false},"author":1,"featured_media":3554,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[8,12,14,15,16],"tags":[],"class_list":["post-3579","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-entity-tagging","category-legal-documents","category-machine-vision","category-natural-language-processing","category-ocr"],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"https:\/\/staging.tolstoy-ai.com\/wp-content\/uploads\/2020\/11\/geometric-1732847_1920.jpg","_links":{"self":[{"href":"https:\/\/staging.tolstoy-ai.com\/index.php?rest_route=\/wp\/v2\/posts\/3579","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/staging.tolstoy-ai.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/staging.tolstoy-ai.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/staging.tolstoy-ai.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/staging.tolstoy-ai.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3579"}],"version-history":[{"count":0,"href":"https:\/\/staging.tolstoy-ai.com\/index.php?rest_route=\/wp\/v2\/posts\/3579\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/staging.tolstoy-ai.com\/index.php?rest_route=\/wp\/v2\/media\/3554"}],"wp:attachment":[{"href":"https:\/\/staging.tolstoy-ai.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3579"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/staging.tolstoy-ai.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3579"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/staging.tolstoy-ai.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3579"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}