AR App Lets You Control Robots Using Your Smartphone

Methods for programming robots vary from connecting to a computer via USB or Wi-Fi, to using an expensive “teach pendant” if you’re working in an industrial setting. In either case, you generally need to be able to envision what the robot will be doing in your mind so you can adjust patterns appropriately.

As seen on IEEE Spectrum, another technique is being developed by Jared Alan Frank at NYU. He’s using the cameras built into normal smartphones or tablets to display information on robots in an augmented reality view. This means that operators who might not be familiar with the nuances of robotics or control software, can now command their robotic minions using a device that they already keep in their pockets.

To accomplish this, Frank devised an app that enables a user to simply point their mobile gadget at a robot and then tell it where to go by drawing a path on their screen, or even tap on an object that they’d like the robot to retrieve and then drag where you’d like it to bring it. Frank’s custom app communicates with the robots over Wi-Fi using a Raspberry Pi.

Using Xcode, Apple’s software development platform, Frank built an app that can detect robots and objects in the environment and create a virtual grid — along with a coordinate system — to keep track of those objects on the screen. The user can then manipulate the objects on the device and watch as the robots carry out the desired actions in the real world.

Interested in reading more? You can find the entire article on this incredible project here.