\t\t\t\tCoronavirus has cut a large and uneven swathe through the world\u2019s businesses. While some fields \u2014 hospitality, transportation, restaurants \u2014 have been crushed, other fields \u2014 grocery delivery, e-commerce \u2014 have been crushed in a very different way: with too much demand from sheltering consumers. This is the story of how Tolstoy\u00a0helped an online garden retailer handle a massive inflow of customer emails as they waged their way through the pandemic.<\/p>\n
While some businesses have seen their demand dry up overnight, other businesses have experienced record inflow. One of the activities consumers stuck at home have turned to is gardening. Garden retailers both in the US and Europe have seen orders skyrocket hundreds, or thousands, in percent growth.<\/p>\n
One of these garden retailers approached Tolstoy with a dilemma. The coronavirus spigot had sent them several times more orders than they had capacity to fulfill. Correspondingly, they had received over 50,000 customer emails since the start of March \u2014 most of them angry emails inquiring on order status.<\/p>\n
The company usually routed customer inquires through Freshdesk, an email platform, that helped categorize inbound mail. But now they were receiving correspondence from many sources, including direct email, which had no categorization applied. They were confronted with the prospect of reading through more than 50,000 emails \u2014 with thousands added per week \u2014 just to triage high priority emails, such as cancellation requests.<\/p>\n
They estimated that if they hired 4\u20135 staff, it would take about a month to read the emails \u2014 during which tens of thousands more emails would arrive.<\/p>\n
Tolstoy took a look at the company\u2019s email data. Email text data can often be messy \u2014 everything from the sender fields to legal disclaimers are included, so we cleaned it up with some preprocessing. This meant stripping out all forwarded emails and auto-email generated text, after which we normalized the text by removing punctuation and capitalization.<\/p>\n
Next, we considered the training set of about 3,000 categorized emails. These emails were classified through Freshdesk, which meant that customers picked the category. While most of them were useful categories, customers often picked wrong categories \u2014 for example, some selected \u201cCancel\u201d when they actually wanted to inquire about a new product, or \u201cDelivery query\u201d when they meant to cancel. We manually corrected and labelled a set of more than 5,000 emails to assert \u201cground truth\u201d.<\/p>\n
We also balanced the limitations of the training data by hard-coding certain heuristics into the predictions.<\/p>\n
After testing different models, we settled on a logistic regression, which is based on the frequency of words found in the email text. To help the model focus on the more important features, we only included words that surpassed a fixed number of appearances across the dataset and incorporated elastic net regularization.<\/p>\n
Ultimately, we configured the model to read and categorize emails with ~98% accuracy, and 97\u201399% precision\/recall (a measure of true\/false positives captured), based on a set of 1,300 manually corrected emails. This was far more accurate than even the existing categorized emails via Freshdesk.<\/p>\n
The garden retailer sent over their 50,000+ batch of emails on a Friday. We tuned the model on the following Monday, and returned the categorized results back on Tuesday.<\/p>\n
This enabled the retailer to immediately start processing and responding to all their customer emails, which they had not been able to do for months. Finally, customers could get their answers and orders fulfilled or refunded.<\/p>\n
From Tolstoy to the gardeners of the world, Here\u2019s to hoping you get your gardening in during shelter-in-place!<\/p>\n","protected":false},"excerpt":{"rendered":"
Coronavirus has cut a large and uneven swathe through the world\u2019s businesses. While some fields \u2014 hospitality, transportation, restaurants \u2014 have been crushed, other fields \u2014 grocery delivery, e-commerce \u2014.<\/p>\n","protected":false},"author":1,"featured_media":3193,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_jetpack_newsletter_access":"","_jetpack_dont_email_post_to_subs":false,"_jetpack_newsletter_tier_id":0,"_jetpack_memberships_contains_paywalled_content":false,"_jetpack_feature_clip_id":0,"_jetpack_memberships_contains_paid_content":false,"footnotes":"","jetpack_post_was_ever_published":false},"categories":[6,7,15],"tags":[],"class_list":["post-3192","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-classification","category-emails","category-natural-language-processing"],"jetpack_sharing_enabled":true,"jetpack_featured_media_url":"https:\/\/staging.tolstoy-ai.com\/wp-content\/uploads\/2020\/06\/volodymyr-hryshchenko-V5vqWC9gyEU-unsplash-scaled.jpg","_links":{"self":[{"href":"https:\/\/staging.tolstoy-ai.com\/index.php?rest_route=\/wp\/v2\/posts\/3192","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/staging.tolstoy-ai.com\/index.php?rest_route=\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/staging.tolstoy-ai.com\/index.php?rest_route=\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/staging.tolstoy-ai.com\/index.php?rest_route=\/wp\/v2\/users\/1"}],"replies":[{"embeddable":true,"href":"https:\/\/staging.tolstoy-ai.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcomments&post=3192"}],"version-history":[{"count":0,"href":"https:\/\/staging.tolstoy-ai.com\/index.php?rest_route=\/wp\/v2\/posts\/3192\/revisions"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/staging.tolstoy-ai.com\/index.php?rest_route=\/wp\/v2\/media\/3193"}],"wp:attachment":[{"href":"https:\/\/staging.tolstoy-ai.com\/index.php?rest_route=%2Fwp%2Fv2%2Fmedia&parent=3192"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/staging.tolstoy-ai.com\/index.php?rest_route=%2Fwp%2Fv2%2Fcategories&post=3192"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/staging.tolstoy-ai.com\/index.php?rest_route=%2Fwp%2Fv2%2Ftags&post=3192"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}