When we created crul, we wanted to give RPA developers the ability to query any data across the modern enterprise technology stack. Unfortunately, data isn’t organized and stored in one easy-to-access place. Instead, data is distributed across a loosely coupled system of ever-changing technologies. 

Think of crul as kind of like a "data accordion." At its core, crul is a map/reduce/expand engine with a query language that empowers users to transform web pages and API requests into shapeable data sets. Crul makes it easy to access, crawl, shape, organize, enrich,  and clean data from just about anywhere and deliver it to just about anywhere. 

The name crul comes from a mashup of the word "crawl" (as in web crawling) and "curl," one of our favorite tools for interacting with the web. crul was built from our desire to transform the web (open, SaaS, API, and dark) into a dynamically explorable data set in real-time.

Let’s look at a crul query language example. We can start with a core api command to make an API request, then pipe the API request's results to a normalize operation (breaking out each result into a separate row), and finally, a find "keyword" command, which will search for "keyword" in the API request results. Here's what that query and results might look like in crul:

api get https://pokeapi.co/api/v2/pokemon

A few months ago, through our friends at Harpoon Ventures, we had the opportunity to share our company vision and product demo with the Founder and CEO of Robocorp, Antti Karjalainen. Subsequently, we began working with Tommi Holmgren, VP of Solutions at Robocorp, on a cool use case and our first Robocorp Portal bot.

Use Case: Let’s go shopping for a great deal.

It was recently the holiday season, and if you’re anything like Nic and I, you may be spending more time hacking than shopping. So in our first use case, we decided to explore extracting data from a Shopify website to get prices on some products.

Here are the Shopify use case requirements:

Iterate through each product listing page

Extract the relevant product information and shape / transform the data into a digestible format

The primary way to use crul is through authoring and running queries. These queries are composed of one or more stages, "piped" together using the double pipe || syntax. If you are familiar with the Unix/Linux shell, crul language will seem rather familiar.

open https://www.tentree.ca/collections/mens-shorts --html --hashtml

|| filter "(attributes.class == 'justify-between product-attr-container mt-2 relative')"

|| filter "(_html.nodeName == 'A') or (_html.attributes.class == 'flex' or _html.attributes.class == 'text-discount-price')"

|| excludes _html.innerHTML "line-through"

|| table _html.innerText outerHTMLHash _sequence

|| rename _group.0._html.innerText product

|| sort _group.0._sequence --order "ascending"

Let's break down this query - stage by stage - to extract a table of products, prices, and timestamps from a web page - using the crul query language.

This stage opens the provided URL, renders the page, and transforms it into a tabular structure that includes the HTML and hashes of the HTML for future grouping.

This stage filters the page data to only include elements matching the filter expression.

This stage will add a _sequence column to each row containing the row number.

This stage processes the element HTML into a row for each of its children. Note that the element hash is shared by each of the expanded children, allowing us to group them back together.

This stage excludes any row containing the value “line-through” in the _html.innerHTML. For example, the data set includes both the regular price and the sale price, so this stage will remove the regular price entry.

This includes only the desired columns in the data set.

This stage groups page elements by the parent hash calculated per element in the first stage of this query.

This stage renames the _group.0._html.innerText column to product.

This stage renames the _group.1._html.innerText column to price.

This sorts the list according to the previously added sequence number in ascending order to preserve the order of products as they appear on the page.

Adds a timestamp column with the current ISO timestamp to each row.

Blog

Robocorp offers its clients the ability to build, scale, deploy, and maintain their robot workforce.

All posts by Carl Yestrau and Nic Stone, Co-Founders Crul, Inc.

How crul + Gen2 RPA gets data that transforms your business

products

solutions

developers

digital workers

resources

company