Natalie Y PRO
NatalieY
AI & ML interests
AI agents, edge AI, on-device LLM, embedded systems, autonomous agents
Recent Activity
published an article 4 days ago
Grounding an agent's answers about its own environment: a small but useful pattern posted an update 9 days ago
Shipped a full-duplex agent architecture for Aiden: real-time voice model handles the live conversation, a separate stronger model handles device-control tasks in the background, coordinated async through a task queue instead of one shared loop.
Grounded in full-duplex speech research (Moshi's parallel-stream model, dual-tower dialogue modeling), not just voice bolted onto an existing agent pipeline.
A few design choices worth sharing: task "completed" and "successful" are tracked as separate signals so the agent never reports a stalled task as done. Backend results get a 500ms aggregation window before reaching the foreground, so closely-timed results don't trigger competing responses. Device tasks run strictly one at a time: ambiguous screen/input ownership wasn't worth the complexity of parallelizing.
Dev-board stage, not validated across every phone/board/audio setup yet. Full writeup: https://aidenai.io/blog/when-voice-meets-the-physical-world-inside-aidens-full-duplex-agent-architecture/
Curious how others here have handled foreground/background coordination when combining real-time voice with a separate task-execution model.Organizations
posted an update 4 days ago
posted an update 9 days ago
Post
65
Shipped a full-duplex agent architecture for Aiden: real-time voice model handles the live conversation, a separate stronger model handles device-control tasks in the background, coordinated async through a task queue instead of one shared loop.
Grounded in full-duplex speech research (Moshi's parallel-stream model, dual-tower dialogue modeling), not just voice bolted onto an existing agent pipeline.
A few design choices worth sharing: task "completed" and "successful" are tracked as separate signals so the agent never reports a stalled task as done. Backend results get a 500ms aggregation window before reaching the foreground, so closely-timed results don't trigger competing responses. Device tasks run strictly one at a time: ambiguous screen/input ownership wasn't worth the complexity of parallelizing.
Dev-board stage, not validated across every phone/board/audio setup yet. Full writeup: https://aidenai.io/blog/when-voice-meets-the-physical-world-inside-aidens-full-duplex-agent-architecture/
Curious how others here have handled foreground/background coordination when combining real-time voice with a separate task-execution model.
Grounded in full-duplex speech research (Moshi's parallel-stream model, dual-tower dialogue modeling), not just voice bolted onto an existing agent pipeline.
A few design choices worth sharing: task "completed" and "successful" are tracked as separate signals so the agent never reports a stalled task as done. Backend results get a 500ms aggregation window before reaching the foreground, so closely-timed results don't trigger competing responses. Device tasks run strictly one at a time: ambiguous screen/input ownership wasn't worth the complexity of parallelizing.
Dev-board stage, not validated across every phone/board/audio setup yet. Full writeup: https://aidenai.io/blog/when-voice-meets-the-physical-world-inside-aidens-full-duplex-agent-architecture/
Curious how others here have handled foreground/background coordination when combining real-time voice with a separate task-execution model.
posted an update 12 days ago
Post
102
Sharing a small but instructive fix from Aiden's firmware. Before PR #586, giving the agent a remote skill URL didn't reliably route to an installation action: it could just as easily be read as a general content-retrieval request, with the agent reaching for a web scraper, shelling out to curl, creating files manually, and patching them in step by step. Every individual tool call worked. The composition was still the wrong shape for a request with one clear, singular intent.
The fix gives skill installation its own dedicated action. A supported HTTP(S), GitHub tree, or GitHub blob Skill URL now routes to skill_manage install, which stages SKILL.md and companion files, validates the staged content against supported requirements, and only then publishes atomically, never exposing partial files mid-install.
https://huggingface.co/blog/NatalieY/aiden-can-now-install-ai-agent-skills-directly
The fix gives skill installation its own dedicated action. A supported HTTP(S), GitHub tree, or GitHub blob Skill URL now routes to skill_manage install, which stages SKILL.md and companion files, validates the staged content against supported requirements, and only then publishes atomically, never exposing partial files mid-install.
https://huggingface.co/blog/NatalieY/aiden-can-now-install-ai-agent-skills-directly
reacted to their post with 🔥 23 days ago
Post
1139
👋 Quick update on what shipped in Aiden's firmware this week, it is mostly developer-experience stuff, but genuinely useful if you're building against the agent:
Native Anthropic support: pick Claude models right in the config page now, streaming + tool calls included
No hardware? No problem: new Docker sandbox lets you run Config Web + Agent Web without a dev board, connects to MobileGym/ADB
Two HDMI bridge chips supported (RK628D + TC358743), auto-detected, RK628D gets you up to 1080p60
Python packages now persist across sessions instead of disappearing every restart
Provider switching got smarter: remembers your last model, less manual config-syncing between LLM/TTS/STT
None of these are new agent behaviors, just a lot less friction if you're actually building. Full technical breakdown here if you want the details: https://huggingface.co/blog/NatalieY/aiden-updates
Has anyone tried the Docker sandbox path yet?
AidenAgent
Native Anthropic support: pick Claude models right in the config page now, streaming + tool calls included
No hardware? No problem: new Docker sandbox lets you run Config Web + Agent Web without a dev board, connects to MobileGym/ADB
Two HDMI bridge chips supported (RK628D + TC358743), auto-detected, RK628D gets you up to 1080p60
Python packages now persist across sessions instead of disappearing every restart
Provider switching got smarter: remembers your last model, less manual config-syncing between LLM/TTS/STT
None of these are new agent behaviors, just a lot less friction if you're actually building. Full technical breakdown here if you want the details: https://huggingface.co/blog/NatalieY/aiden-updates
Has anyone tried the Docker sandbox path yet?
- 2 replies
posted an update 25 days ago
Post
1139
👋 Quick update on what shipped in Aiden's firmware this week, it is mostly developer-experience stuff, but genuinely useful if you're building against the agent:
Native Anthropic support: pick Claude models right in the config page now, streaming + tool calls included
No hardware? No problem: new Docker sandbox lets you run Config Web + Agent Web without a dev board, connects to MobileGym/ADB
Two HDMI bridge chips supported (RK628D + TC358743), auto-detected, RK628D gets you up to 1080p60
Python packages now persist across sessions instead of disappearing every restart
Provider switching got smarter: remembers your last model, less manual config-syncing between LLM/TTS/STT
None of these are new agent behaviors, just a lot less friction if you're actually building. Full technical breakdown here if you want the details: https://huggingface.co/blog/NatalieY/aiden-updates
Has anyone tried the Docker sandbox path yet?
AidenAgent
Native Anthropic support: pick Claude models right in the config page now, streaming + tool calls included
No hardware? No problem: new Docker sandbox lets you run Config Web + Agent Web without a dev board, connects to MobileGym/ADB
Two HDMI bridge chips supported (RK628D + TC358743), auto-detected, RK628D gets you up to 1080p60
Python packages now persist across sessions instead of disappearing every restart
Provider switching got smarter: remembers your last model, less manual config-syncing between LLM/TTS/STT
None of these are new agent behaviors, just a lot less friction if you're actually building. Full technical breakdown here if you want the details: https://huggingface.co/blog/NatalieY/aiden-updates
Has anyone tried the Docker sandbox path yet?
- 2 replies
reacted to their post with 🔥 about 2 months ago
Post
2052
Spent a while chasing a genuinely strange iOS bug while building Aiden (a physical agent that drives phones over USB HID): modifier-key shortcuts like Cmd+V would silently fail while plain keystrokes worked fine every time.
Turned out iOS was routing the command to the wrong process (SpringBoard, not the actual foreground app) whenever a keyboard and mouse were both present at the same time as AssistiveTouch. Confirmed it wasn't specific to our hardware, reproduced it on a completely unrelated gaming keyboard and a Bluetooth keyboard too.
Full writeup with the actual experiment table and log output: https://huggingface.co/blog/NatalieY/debugging-aiden
Curious if anyone here has hit this same failure mode building on iOS accessibility APIs.
Repo: https://github.com/AidenAI-IO/aiden-firmware
Turned out iOS was routing the command to the wrong process (SpringBoard, not the actual foreground app) whenever a keyboard and mouse were both present at the same time as AssistiveTouch. Confirmed it wasn't specific to our hardware, reproduced it on a completely unrelated gaming keyboard and a Bluetooth keyboard too.
Full writeup with the actual experiment table and log output: https://huggingface.co/blog/NatalieY/debugging-aiden
Curious if anyone here has hit this same failure mode building on iOS accessibility APIs.
Repo: https://github.com/AidenAI-IO/aiden-firmware
posted an update about 2 months ago
Post
2052
Spent a while chasing a genuinely strange iOS bug while building Aiden (a physical agent that drives phones over USB HID): modifier-key shortcuts like Cmd+V would silently fail while plain keystrokes worked fine every time.
Turned out iOS was routing the command to the wrong process (SpringBoard, not the actual foreground app) whenever a keyboard and mouse were both present at the same time as AssistiveTouch. Confirmed it wasn't specific to our hardware, reproduced it on a completely unrelated gaming keyboard and a Bluetooth keyboard too.
Full writeup with the actual experiment table and log output: https://huggingface.co/blog/NatalieY/debugging-aiden
Curious if anyone here has hit this same failure mode building on iOS accessibility APIs.
Repo: https://github.com/AidenAI-IO/aiden-firmware
Turned out iOS was routing the command to the wrong process (SpringBoard, not the actual foreground app) whenever a keyboard and mouse were both present at the same time as AssistiveTouch. Confirmed it wasn't specific to our hardware, reproduced it on a completely unrelated gaming keyboard and a Bluetooth keyboard too.
Full writeup with the actual experiment table and log output: https://huggingface.co/blog/NatalieY/debugging-aiden
Curious if anyone here has hit this same failure mode building on iOS accessibility APIs.
Repo: https://github.com/AidenAI-IO/aiden-firmware
reacted to their post with 👀🔥 2 months ago
Post
2460
Aiden: a physical AI agent that controls phones over USB HID
Most GUI agent work assumes the agent lives inside the device or
drives it through a debugging interface. We went the other way.
Aiden is a small board that sits outside the host. It captures the
screen over HDMI-to-CSI, runs the agent loop on-device, and sends
actions back as a standard USB HID device — the host sees a keyboard
and a mouse, nothing else. No app install, no root, no ADB, no cloud.
Runtime is Go. Frame capture, full-duplex audio with VAD, the agent
loop, and HID output all run as independent goroutines. There's no
backend — nothing leaves the device, which is the only defensible
design when the input is a live feed of someone's phone screen.
Open questions we haven't solved:
· Action verification — inferring success from a re-read of the
screen breaks when loading states lie
· Prompt injection — an agent that reads screens reads whatever an
attacker puts on them
· iOS pointer control requires AssistiveTouch
Repo, including the HID gadget config and capture pipeline:
github.com/AidenAI-IO/aiden-hardware-demo
Wrote up how this differs from cloud-based computer use agents here:
https://aidenai.io/blog/mobile-ai-agent-vs-computer-use-agent-whats-the-difference/
Note: current hardware is a dev board, not a finished product.
Most GUI agent work assumes the agent lives inside the device or
drives it through a debugging interface. We went the other way.
Aiden is a small board that sits outside the host. It captures the
screen over HDMI-to-CSI, runs the agent loop on-device, and sends
actions back as a standard USB HID device — the host sees a keyboard
and a mouse, nothing else. No app install, no root, no ADB, no cloud.
Runtime is Go. Frame capture, full-duplex audio with VAD, the agent
loop, and HID output all run as independent goroutines. There's no
backend — nothing leaves the device, which is the only defensible
design when the input is a live feed of someone's phone screen.
Open questions we haven't solved:
· Action verification — inferring success from a re-read of the
screen breaks when loading states lie
· Prompt injection — an agent that reads screens reads whatever an
attacker puts on them
· iOS pointer control requires AssistiveTouch
Repo, including the HID gadget config and capture pipeline:
github.com/AidenAI-IO/aiden-hardware-demo
Wrote up how this differs from cloud-based computer use agents here:
https://aidenai.io/blog/mobile-ai-agent-vs-computer-use-agent-whats-the-difference/
Note: current hardware is a dev board, not a finished product.
- 3 replies
posted an update 2 months ago
Post
2460
Aiden: a physical AI agent that controls phones over USB HID
Most GUI agent work assumes the agent lives inside the device or
drives it through a debugging interface. We went the other way.
Aiden is a small board that sits outside the host. It captures the
screen over HDMI-to-CSI, runs the agent loop on-device, and sends
actions back as a standard USB HID device — the host sees a keyboard
and a mouse, nothing else. No app install, no root, no ADB, no cloud.
Runtime is Go. Frame capture, full-duplex audio with VAD, the agent
loop, and HID output all run as independent goroutines. There's no
backend — nothing leaves the device, which is the only defensible
design when the input is a live feed of someone's phone screen.
Open questions we haven't solved:
· Action verification — inferring success from a re-read of the
screen breaks when loading states lie
· Prompt injection — an agent that reads screens reads whatever an
attacker puts on them
· iOS pointer control requires AssistiveTouch
Repo, including the HID gadget config and capture pipeline:
github.com/AidenAI-IO/aiden-hardware-demo
Wrote up how this differs from cloud-based computer use agents here:
https://aidenai.io/blog/mobile-ai-agent-vs-computer-use-agent-whats-the-difference/
Note: current hardware is a dev board, not a finished product.
Most GUI agent work assumes the agent lives inside the device or
drives it through a debugging interface. We went the other way.
Aiden is a small board that sits outside the host. It captures the
screen over HDMI-to-CSI, runs the agent loop on-device, and sends
actions back as a standard USB HID device — the host sees a keyboard
and a mouse, nothing else. No app install, no root, no ADB, no cloud.
Runtime is Go. Frame capture, full-duplex audio with VAD, the agent
loop, and HID output all run as independent goroutines. There's no
backend — nothing leaves the device, which is the only defensible
design when the input is a live feed of someone's phone screen.
Open questions we haven't solved:
· Action verification — inferring success from a re-read of the
screen breaks when loading states lie
· Prompt injection — an agent that reads screens reads whatever an
attacker puts on them
· iOS pointer control requires AssistiveTouch
Repo, including the HID gadget config and capture pipeline:
github.com/AidenAI-IO/aiden-hardware-demo
Wrote up how this differs from cloud-based computer use agents here:
https://aidenai.io/blog/mobile-ai-agent-vs-computer-use-agent-whats-the-difference/
Note: current hardware is a dev board, not a finished product.
- 3 replies
posted an update 3 months ago
Post
175
This is a short office demo showing how Aiden works in practice.
Aiden is a physical mobile AI agent device that plugs into any phone or computer via USB. It sees the screen, hears your voice, and operates the device for you — no app install required.
In this video Aiden is receiving a voice command and completing a multi-step task
Built for the AI-Native Era. Works on the phone you already have.
AidenAgent
aidenai.io
Aiden is a physical mobile AI agent device that plugs into any phone or computer via USB. It sees the screen, hears your voice, and operates the device for you — no app install required.
In this video Aiden is receiving a voice command and completing a multi-step task
Built for the AI-Native Era. Works on the phone you already have.
aidenai.io
reacted to their post with 👍 3 months ago
Post
89
Launching Aiden Mobile Agent Prototype. Looks incredible. Thoughts?
https://github.com/AidenAI-IO/aiden-hardware-demo
https://github.com/AidenAI-IO/aiden-hardware-demo
posted an update 3 months ago
Post
89
Launching Aiden Mobile Agent Prototype. Looks incredible. Thoughts?
https://github.com/AidenAI-IO/aiden-hardware-demo
https://github.com/AidenAI-IO/aiden-hardware-demo
replied to their post 3 months ago
thanks for the feedback! let me know what kind of use cases you personally would like to have on such a device, we're trying to explore as many as possible
posted an update 3 months ago
Post
126
Introducing AidenAgent
We built a physical AI agent device. It plugs into any phone or computer via USB. It watches the screen through HDMI. It listens, speaks, and controls the connected device autonomously. No app install, no API, no permissions required.
The host just sees a keyboard and a mouse, the intelligence is in the device.
Full builder story + technical architecture at huggingface.co/blog/NatalieY/aiden-hardware-ai-agent-device
We built a physical AI agent device. It plugs into any phone or computer via USB. It watches the screen through HDMI. It listens, speaks, and controls the connected device autonomously. No app install, no API, no permissions required.
The host just sees a keyboard and a mouse, the intelligence is in the device.
Full builder story + technical architecture at huggingface.co/blog/NatalieY/aiden-hardware-ai-agent-device
- 2 replies
reacted to Jiaqi-hkust's post with ❤️ 3 months ago
Post
3521
Happy to introduce Response-G1 #ACL2026 — a proactive agent for streaming video understanding.
📄 Paper: http://arxiv.org/abs/2605.07575
📷 Code: http://github.com/kadmkbl/Response-G1
We are happy to have a further discussion!!!
#ACL2026 #AI #Multimodal #VideoUnderstanding #OpenSource #LLM
📄 Paper: http://arxiv.org/abs/2605.07575
📷 Code: http://github.com/kadmkbl/Response-G1
We are happy to have a further discussion!!!
#ACL2026 #AI #Multimodal #VideoUnderstanding #OpenSource #LLM
- 1 reply