Empower Your Best People With Desktop Automation

May I introduce Matilda? 🙋🏾‍♀️

She visualizes numbers as shapes (♾) and colors (🟠), and can do differential calculus in her head. The lady is genuinely a genius! 🪔🧞‍♀️

Matilda works for a financial company. Her job is to copy & paste data from email invoice attachments to the company’s SPA system (Stupendously Potent Accountinator)! She might be slightly overqualified for this task, but she is currently the only person in the department who can complete it. 🤷‍♀️

Matilda logs in to the desktop computer, opens the email application, downloads and opens the invoice attachments, opens the financial desktop application, and copy-pastes the invoice details. Click. Type. Clickety. Typety. Save. Done. Rinse and repeat!

Matilda completes this process tens of times a day and only makes a tiny mistake now and then. Sometimes she needs to ask for a corrected invoice from the sender if she spots something weird about the numbers!

Matilda's task is:

• Repeatable: Same thing over and over again. 🔁
• Slow: Many systems are involved. 🐌
• Error-prone: Sometimes, there are typos in the “incoives” (👀).
• Mentally exhausting in the long run. 🤯
• Impossible to automate due to desktop applications?! 🙀

Matilda's company has faced challenging times lately. Her number wizardry would come super useful in helping the business if she had time! If only there were a way to transfer her current job to someone else. Or something else. 🤔

googling intensifies

Aha! An article titled "Desktop automation" by Robocorp, how convenient and utterly surprising! Let's see:

Desktop automation allows your robot to accomplish tasks acting like a human operator, directly controlling a desktop interface. This includes operations like opening and closing applications, simulating mouse movements and clicks, triggering keyboard keys and shortcuts, taking screenshots.

Yes! That's exactly what her company needs. A robot to free Matilda to help us overcoming the business challenges!

There seems to exist many strategies for tackling desktop automation:

Image template matching based automation: Take screenshots of the desktop application and instruct your robot to find and interact with the elements it finds! Robocorp provides tools for capturing the screenshots with either Robocorp Lab or Visual Studio Code with the Robocorp extensions. The image-based automation is cross-platform, too (Windows, Linux, macOS)! Interesting! 🖼

Textual locator-based desktop automation: Instead of images, target the UI elements by their accessibility identifiers. This sounds wonderful and would be an excellent fit for our Windows-based desktop workflow! 💯

Optical Character Recognition (OCR): Find UI elements by text. Wow! Think of the possibilities! There's out-of-the-box solutions and integrations to many external document data extraction service providers! 🔎

Keyboard and mouse-based automation: Shortcuts, typing, all very familiar to power users whizzing through application screens without even glancing at their mouse! Robocorp's solution supports both the keyboard and the mouse in a cross-platform way! ⌨️ + 🐁

Oh no, coffee break approaches! I will continue reading this excellent overview of desktop automation possibilities after downing a cup of liquid goodness! ☕️

Fast-forward a few weeks. Matilda's company built a robot to take over her copy & paste duties. Matilda solved critical business challenges for her company, grew old, traveled the world, enjoyed life, and lived happily ever after! Some say the robot is still copying & pasting the invoices without a single complaint!