In our recent showcases with Action Server, including CurlGPT and Database examples, we've uncovered a way to leverage LLMs beyond their everyday use as chat interfaces

In our recent showcases with Action Server, including CurlGPT and Database example, we've uncovered a way to leverage Large Language Models (LLMs) beyond their everyday use as chat interfaces.

LLMs have been popularized as advanced chatbots, interpreting and responding to user inputs in conversational formats. However, the real breakthrough comes when these models go beyond superficial conversations. They can understand and interact with data in ways that were previously complex and inaccessible to many. This advanced interaction is facilitated by the LLM's ability to learn from a data source's schema. In this context, a "schema" refers to any representation of the data's structure—be it a formal, detailed blueprint of a database or merely an example snippet of the actual data.

Domain-specific languages (DSLs), such as SQL for database queries or regex for pattern matching, have always been powerful tools for specific tasks. Our approach simplifies this by having LLMs learn the data schema directly and use DSL-driven actions to execute tasks while interacting through a user-friendly chat interface.

A Streamlined Process for Data Interaction

The process we've used unfolds in four straightforward steps:

An LLM establishes a connection with a data source.

It then receives a description of the data's schema.

The LLM uses this schema to learn about the data in context.

Finally, users interact with an LLM-based chatbot. This chatbot uses the learned schema and DSL to perform tasks, significantly simplifying the data interaction.

Users can leverage the LLM's understanding to carry out complex tasks through simple dialogues without needing to master DSLs or understand the data schemas.

Looking Toward a Future of Accessible Data Interaction

The implications of this development seem profound. It enhances accessibility, allowing anyone to interact with data systems without the prerequisite of programming knowledge or familiarity with data structure in question. It also introduces the concept of generic actions, accepting input in DSL form, that LLMs can employ to interface with any type of data source. This approach means we might no longer need custom APIs or functions for specific tasks. Instead, we can use more universal methods to work with data from many domains.

This LLM-driven data interaction opens up new possibilities for engaging with technology. Patterns like this that tap into the potential of LLM learning could democratize access to complex data systems and redefine our interaction with the digital world.

Beyond Chatbots: Empowering LLMs to Learn and Interact with Data

This post dives into the application of generative AI agents, taking a close look at the challenges and solutions encountered in my project, llmstatemachine, which explores ways to navigate and interact within dynamic digital environments

“An agent is anything that can be viewed as perceiving its environment through sensors and acting upon that environment through actuators.” - Stuart Russell and Peter Norvig, Artificial Intelligence: A Modern Approach, 2nd ed. (Pearson Education, Inc., 2003), p. 32.

Building on this definition, in this post, we'll dive into the application of generative AI agents, closely examining the challenges and solutions encountered in my project, llmstatemachine, which explores ways to navigate and interact within dynamic digital environments.

Having spent 18 years in the software industry, my relationship with AI has evolved from viewing it as a fascinating theoretical concept to recognizing its potential for real-world applications. This journey led me to explore the intersection of AI and problem-solving.

Beginning of last year 2022 I recall being first exposed to GitHub Copilot. With a group of friends, we did an example of a test case where the data was about Tony Stark, the fictional character in the Marvel Universe whose alter ego is Iron Man. GitHub Copilot suggested another case with data related to Bruce Banner and his alter ego The Hulk. - I was amazed.

Despite the rapid advancements in AI development, the agent's ability to follow a plan remains challenging, resulting in high error rates in GenAI-based agents and highlighting AI's complexity in maintaining focus and purpose.

One potential approach to improve the task completion rate would be to restrict agents' actions and force them to perform a set of steps to reach their goals. This led me to finite state machines that can represent a workflow with limited allowed actions that need to be accomplished. I began to explore this concept with my llmstatemachine project.

I focused on cases where an agent may not observe the whole environment. Examples of these partially observable environments are all around.

The first agent problem I focused on was HTML element locator discovery. This is a specific problem for web browser automation. When searching for dynamically changing information from a web page, locating the elements containing the target information can be difficult.

In this context, the environment that an agent is working on is the HTML DOM. This is an element tree containing the target information. The goal is to find a method, a CSS selector to be specific, to select the tree branches with the target information. Then you can re-use the same method later to discover the new changed information from the exact same positions. Building a robust strategy that matches all targets and works over a long time can be challenging.

I made functionalities for an agent to observe some representation of the HTML DOM. I’ll call a function that makes an observation a view.

a partial representation of the environment.

In my explorations for the HTML DOM, I implemented 3 views: focus, select, and validate.

Focus view gives a standard size response in the whole HTML DOM tokens. It takes a parameter text that it will try to highlight from the HTML DOM, opening the relevant element structures.

Select view takes a CSS selector as an argument and returns the standard size response of a paged list of first matching elements in a simplified format from the HTML DOM.

Validate works internally like Select. It takes a CSS selector and tests it. But instead of returning the result to the agent, it checks if the result elements contain all the example texts. Based on that it either gives feedback to the agent about failure or success.

The experimental agent tests were promising, although sometimes the context window was fully consumed or went into a loop if the focus action failed to retrieve a relevant view.

Mostly the agent came up with a good CSS selector to find a set of elements from the HTML DOM. One noteworthy thing in this problem is that there is not only one working solution but multiple that satisfy the requirements.

A complete example of agent code can be found here.

A high-level overview of the state machine:

    builder = WorkflowAgentBuilder()
    builder.add_system_message(
        (
            "You are a helpful HTML css selector finding assistant.\n"
            "Assignment: Create CSS Selectors Based on Text Content.\n"
            "Your task is to develop CSS selectors that can target HTML elements containing specific text contents. "
            "You are provided with a list of example texts. Use these examples to create selectors that can identify "
            "elements containing these texts in a given HTML structure.\n\n"
            "Instructions:\n"
            f"- Use the provided list of examples: {examples_str}.\n"
            "Your goal is to create selectors that are both precise and efficient, tailored to the specific"
            " content and structure of the HTML elements."
        )
    )
    builder.add_state_and_transitions("INIT", {focus, select})
    builder.add_state_and_transitions("SELECTED_NON_EMPTY", {focus, select, validate})
    builder.add_state_and_transitions("VALIDATED", {focus, select, validate, result})
    builder.add_end_state("DONE")
    workflow_agent = builder.build()

Coming up with the general design of a state machine based LLM agent

Most popular Large Language Models (LLMs) of today are focused on chat interfaces, where the LLM plays one part of the discussion and a user or a function plays the other part in the chat messages. llmstatemachine focuses on using the chat history as the agent's memory.

In llmstatemachine a state can enable and disable functions that the LLM can call. The state defines the transitions that the agent can take. The transition functions are responsible for changing the state.

Under the hood, this is accomplished by OpenAI function calling and enforcing the function call to happen. The current API allows only one specific function to be called. I made it so that it will call an ActionSelector function with arguments that define the actual transition function and parameters to be called. Function calling API can be considered a simple way to enforce the model to return a schema following JSON formatted output (this is stricter than the JSON mode that does not enforce a schema).

The ActionSelector also takes a parameter called thinking. This parameter is there to allow the LLM to reason in tokens before making the actual decision on what transition to take. It will also be displayed in the message history, giving the agent access to the reasoning it has already made later on.

To create descriptions for the JSON specification of a function, I made a convenient method that fulfills those descriptions directly with the help of an LLM. This worked very well and could result in its own project.

The maze game agent was made without giving access for the agent to a complete map of the 2D maze it was exploring. The success rate for mazes with the agent was low in my tests. It ended up going into loops or the same paths again and again.

thinking: Given that I just moved from the right, and since generally the objective is to move towards the upper left corner of the maze, my next move should be upwards to get closer to the goal.
action: move_up
..
Stopped: at a cross-section
From current location you may move: DOWN, LEFT, RIGHT
#########E#
#   # #   #
# ### # ###
#         #
### # #####
#   # #   #
##### ### #
# #    X  #
# ##### ###
#         #
#########S#

I learned that when the same LLM (GPT-4-turbo) is asked to create an agent or an algorithm to solve a 2D maze, it can easily create that. This to me is a really interesting observation. I would like to explore more patterns of LLMs as systems builders instead of directly being operational agents in them.

The complete maze example agent code can be found here. The states are dynamically generated based on available next directions.

maze_game_agent_builder = (
    WorkflowAgentBuilder()
    .add_system_message(
        f"You are a player in a 2 dimensional {maze_height*2}x{maze_width*2} maze. "
        + "Find your way through the maze."
    )
    .add_end_state("DONE")
)
def match_directions_to_callables(direction_combination) -> set[Callable]:
    callables = set()
    direction_to_function = {
        "UP": move_up,
        "DOWN": move_down,
        "LEFT": move_left,
        "RIGHT": move_right,
    }
    for direction in direction_combination:
        if direction in direction_to_function:
            callables.add(direction_to_function[direction])
    return callables
for state in player.all_direction_combinations():
    state_str = ":".join(sorted(state))
    maze_game_agent_builder.add_state_and_transitions(
        state_str, match_directions_to_callables(state)
    )
maze_game_agent_builder.add_state_and_transitions("INIT", {start})
memory_game_agent = maze_game_agent_builder.build()

In summary, GPT-4-turbo, as the current forefront of publicly accessible generative AI, shows promise as a core engine for agent-based systems within a state machine framework. However, it encounters challenges with complex, high-level planning and execution.

What's fascinating is its ability to craft algorithmic agents capable of solving puzzles it can't directly tackle. Imagine a scenario where GPT-4-turbo designs an agent that easily navigates a labyrinth, a task it finds too hard. This isn't just about an AI model's knowledge of algorithms but hints at a new frontier where generative AI can be harnessed to build deterministic agents, complementing its inherently non-deterministic nature. Such a future, where AI not only solves problems but creates solvers, is not just promising — it's a glimpse into the potential of AI in our world.

Blog

Robocorp offers its clients the ability to build, scale, deploy, and maintain their robot workforce.

All posts by Mikko Korpela, Principal Engineer, Robocorp

Beyond Chatbots: Empowering LLMs to Learn and Interact with Data

Exploring AI Agents: A Journey with 'llmstatemachine'

products

solutions

developers

digital workers

resources

company