Text-State-to-Action: A Lightweight Alternative to Vision-Language-Action Models for Robotics

Xingyi Zhao

The recent progress of Vision-Language-Action models has pushed robotics toward a new paradigm: directly mapping visual observations and human instructions into robot actions. In this framework, a model receives images, language commands, and sometimes robot states, then outputs actions that can be executed by a robot. This direction is powerful and promising, but it also comes with a heavy cost: large-scale multimodal training, expensive robot demonstrations, embodiment-specific action spaces, and limited interpretability.

An alternative route is worth exploring: instead of training an end-to-end Vision-Language-Action model, we can convert visual perception into structured textual world states, and then use a pure-text large language model to perform reasoning and action planning.

I call this idea Text-State-to-Action.

Difference between Vision-Language-Action (VLA) and Text-State-to-Action (T2A). In a Vision-Language-Action system, the model is asked to learn a direct multimodal mapping from perception to action. It must look at the world, understand the instruction, infer the relevant state, and produce executable behavior in one learned system. In Text-State-to-Action, perception is handled by external modules that convert the environment into a structured textual state. The language model can then focus on what it is good at: reasoning over state, decomposing goals, coordinating agents, and choosing the next action.

Aspect	Vision-Language-Action	Text-State-to-Action
Input	Raw image or video with language	Textual world state with language
Model type	Multimodal policy model	Pure-text reasoning model
Output	Robot actions or action tokens	Structured action streams or skill calls
Training data	Vision-language-action trajectories	Text-state-action pairs
Training cost	High, because it depends on multimodal demonstrations	Lower, because state-action supervision can be more lightweight
Interpretability	Often limited by end-to-end representations	Higher, because intermediate states and plans are readable
Debugging	Harder to localize errors	Easier to inspect perception, state, reasoning, and action separately
Embodiment dependence	Stronger, since action spaces are often tied to a specific robot	Weaker, since textual states and high-level skills can transfer more naturally
Safety control	Harder to inspect before execution	Easier to constrain through explicit plans and action checks
Best use case	End-to-end visuomotor control	Goal-level planning, multi-agent coordination, and skill composition

Why Text-State-to-Action Is Attractive

1. Lower Training Cost

Training Vision-Language-Action models usually requires expensive multimodal data. A typical training example looks like this:

image / video + instruction + robot action

This kind of data is costly because it often depends on real robots, human demonstrations, teleoperation, or high-fidelity simulation. Each data point may require careful setup, physical execution, and embodiment-specific annotation.

Text-State-to-Action can use a lighter form of supervision:

textual world state + instruction + action sequence

This data is much cheaper to generate. It can be created from simulators, games, virtual environments, manually designed tasks, or even synthetic data generation. Instead of collecting every action through a physical robot, we can describe the world state in text and train the model to reason about what should happen next.

2. Better Use of Existing Pure-Text LLMs

Another advantage is that we do not need to start from scratch. There are already many strong pure-text large language models, including LLaMA, Qwen, Mistral, and others. These models already have useful abilities for action planning:

instruction following
reasoning
tool use
planning
code generation
structured output
multi-step decomposition

Instead of training a large VLA model to learn perception, reasoning, and control all at once, we can adapt existing text LLMs through post-training. The goal is not to teach the model how pixels map to motor torques. The goal is to teach the model how to operate over a textual state representation:

how to read a textual world state
how to ground instructions to objects
how to select valid skills
how to generate executable action streams
how to replan after failure
how to obey safety constraints

This makes the learning problem narrower, more interpretable, and much more efficient.

3. More Interpretable and Debuggable

A major challenge with end-to-end VLA systems is that failures can be hard to diagnose. If a VLA-controlled robot fails to pick up a cup, the reason may come from many different places:

wrong visual understanding
wrong object grounding
wrong action prediction
wrong grasp pose
low-level control failure
distribution shift

The difficulty is that these components are often entangled inside one large model. A failure at the output does not always tell us whether the model misunderstood the scene, misread the instruction, selected the wrong skill, or failed at execution.

In Text-State-to-Action, the system can be designed more modularly. We can inspect the textual world state, check whether the instruction is grounded correctly, examine the planned action sequence, and validate each skill call before execution. This makes errors easier to localize, and it also gives developers more places to add safety checks, constraints, or recovery policies.

4. More Controllable and Safer

Because the LLM outputs structured actions, we can add explicit constraints before execution. Instead of directly trusting a low-level action prediction, the system can first check whether the proposed action is valid, safe, and consistent with the current world state.

This creates a natural control layer between reasoning and execution. For example, the system can reject impossible actions, prevent unsafe tool use, enforce task rules, or require replanning when the current plan violates a constraint. In robotics, this kind of intermediate validation is especially valuable because mistakes can have physical consequences.

5. More Embodiment-Agnostic

VLA models are often tied to specific robot embodiments and action spaces. A robot arm, a mobile robot, and a humanoid robot may all need very different low-level commands.

For example, one robot arm may use:

Δx, Δy, Δz, Δroll, Δpitch, Δyaw, gripper

A mobile robot may use:

linear_velocity, angular_velocity

A humanoid robot may have an even larger and more complex action space. Text-State-to-Action separates high-level reasoning from low-level control. The LLM can output general skills:

move_to(cup_1)
grasp(cup_1)
place(cup_1, table_1)

Different robots can implement these skills in different ways. As a result, the high-level planner can be reused across platforms, while the low-level controller remains robot-specific. This makes the system more flexible and adaptable to different embodiments with minimal changes. (Train Once, Run Anywhere)

6. Unified Interface for Multisensory Robots

Text-State-to-Action can also naturally integrate information from many different sensing modules, not only cameras.

In real robotic systems, visual perception is only one part of the sensory input. A robot may also receive information from depth sensors, tactile sensors, force sensors, pressure sensors, temperature sensors, velocity sensors, radio-frequency sensors, LiDAR, microphones, IMUs, and other external sensing modules. These sensors produce highly heterogeneous data with different formats, frequencies, resolutions, and physical meanings.

For end-to-end VLA models, this creates a major scalability challenge. If we want the model to directly consume every sensor modality, we need to align vision, language, action, and many additional sensor streams inside one unified training framework. This can quickly become expensive and difficult, because every new sensor introduces a new modality that must be represented, synchronized, and trained together with language and action.

Text-State-to-Action offers a simpler and more scalable alternative. External sensing modules do not need to be fused into the LLM as raw multimodal inputs. Instead, each sensor can act as a perception module that extracts decision-relevant information and converts it into a structured textual state. After conversion, all sensor information is exposed through the same interface: structured text. The LLM does not need to learn separate raw representations for vision, touch, temperature, pressure, radio frequency, or motion. It only needs to learn how to reason over textualized world states. Different sensors can be integrated as separate perception modules, while their outputs are unified into the same textual state representation.

As a result, post-training becomes much cheaper and more scalable. Instead of training a large multimodal model to align every possible sensor modality with language and action, we can post-train a pure-text LLM on structured state-action data. New sensors can be added by designing new textual state fields, without retraining the entire model from scratch.

This makes Text-State-to-Action especially suitable for real-world robotics, where robots often rely on many heterogeneous sensors beyond cameras. The framework treats external sensing systems as providers of structured state information, allowing the LLM planner to reason over a unified textual representation of the world.