Overview

History

The project began as an attempt to control a Nintendo Switch programmatically to automate certain games using scripts.

One application is in the game Splatoon, where players can create custom banners. Using these scripts, players can draw banners with pixel-perfect precision.

There's an existing project that reverse-engineered the controller and mapped its commands.

How it works

Essentially, we forward commands to an Arduino board using a UART device by sending them through a serial port. The Arduino board emulates a switch controller and translates the commands into controller inputs.

The same communication also works backwards. The Arduino board can send feedbacks back to the computer.

This bidirectional communication allows you to:

1. Control Nintendo Switch using keyboard and mouse by mapping the inputs to the controller.
2. Play Nintendo Switch on your phone by streaming the video to your phone and sending the controller inputs back to the computer and Nintendo Switch.
3. Automate games by capturing the Nintendo Switch screen and sending controller inputs to it.

Streaming Interaction

After playing around with the control methods and hardware. I decided to build a project that allows users from streaming platforms to directly interact with my local Nintendo Switch using chat. You may have heard of Twitch Plays Pokémon. This project is similar, but with a Nintendo Switch.

The project consists of:

1. Capturing, sanitizing, and mapping the chat commands to controller inputs.
2. Making custom macros and commands to speed up the interaction.
3. Using image recognition to capture real-time game states (e.g., inventory, player location) and display them outside the game window.
4. Detect game state and block commands that would break the progress entirely.
5. Effectively display how the commands work in real-time.

To achieve this, I made a backend, HTML pages, and video assets to collect and cycle through all the information in OBS (streaming software).