Connecting a Desktop Flash Application to a Bluetooth-Enabled Mobile Phone

Motivation

An interesting interaction paradigm is interfacing with software applications on projection/big-screen displays in common spaces. If the design goals are to encourage shared or group interaction as well as to constrain physical movement as little as possible, the traditional keyboard and mouse may not be the best choices as input devices. Mobile devices offer a variety of benefits in this scenario, including high user familiarity and penetration (at least in the developed world), rich data input capabilities, and of course affordance of high physical mobility.

For a class project, we designed a video-based social headline system called BreakíníNews, which supported creation and consumption of videos in a corporate break room to promote social interaction among co-workers in remote offices. We built the Flash-based application interface to run on a 40-inch LCD screen, and used the system described on this page to connect the application to mobile phones.


The project was a success, and a big reason why was that the use of a mobile phone as a remote control for the application was very compelling. We thus thought it might be useful to show others how to set up the system, so that they may incorporate it into their own projects.

Introduction

This page will provide you with all the software and instructions needed to get a Macromedia Flash application (.swf) that is running on your desktop to communicate with a mobile phone through Bluetooth. The software provided is quite basic, but should provide a sufficient illustration of how things work so that you may go off and create a more sophisticated system. Here, we will step through setting up the "pipe" that will allow the phone to navigate and click buttons on a simple Flash app with the phone's D-pad, as well as enter text with the phone's soft keys, prompted by the app itself.

At the top of this page is an illustration of the pipe we will be building to bridge the Flash app ("app") to the Nokia S60 phone ("phone"). The phone is running a Python script that listens for key presses on the phone. Once this occurs, the script encodes the key press and sends the data off to a serial port on the PC, through the Bluetooth connection. A piece of software called serproxy.exe is listening for data on the port, and once it receives it, forwards on to a specified TCP port. The Flash application then reads from that TCP port using the XMLSocket class, and passes the information on to listeners in the application that update state depending on the action from the phone.

Ingredients

To complete the steps below, you will need the following hardware and software components:

Setup

  1. Make sure Bluetooth is enabled on both your PC and mobile phone, and that the two devices are paired together (i.e. they show up in each other's respective known device menus).
  2. On your PC, find out what COM port is accepting incoming serial data. I did this on my PC by going to the Control Panel and clicking on "Bluetooth Local COM" and finding the COM port corresponding to "Local-COM Server [SerialPort...". If you donít see such a port, you may have to create one.
  3. Remembering that port number, open up serproxy.cfg with a text editor like WordPad from the directory where you unzipped the serproxy package.
  4. Set the comm_ports variable to the port number from above. You can remove the other port numbers. Change the variable comm_baud to 115200.
  5. Change net_port1=5331 to be net_portn=5331, where n is the port number. You can remove the other port settings. Save the file and close it.
  6. Make sure the latest version of Python is installed on your mobile device. If you donít have it, you can download it from the link above.
  7. Open up navigation.py in a text editor and enter your PCís Bluetooth address at the top of the file, where indicated. You can get your address by hovering over the Bluetooth icon in your systray or something similar. Mine is entered by default.
  8. Send navigation.py to your mobile device from your PC (this is a good way to make sure step 1 is done) and install the file as a python script (not a lib module).
  9. Go to Python on your mobile and run the script you just installed. Once you do, there should be a white screen and nothing else happening.
  10. Execute serproxy.exe. A shell should appear.
  11. Run the movie FlashToBluetoothApp.fla. When you do, the following should appear in the serproxy shell:

    Serproxy - (C)1999 Stefano Busti, (C)2005 David A. Mellis - Waiting for clients
    Server thread launched
    server(7) - thread started

  12. Once the flash app is launched, click allow on the security settings and now use your phoneís D-pad arrow keys to navigate, with enter corresponding to the middle (push).
Note: When restarting the system, the order in which the components are run matters; You should always run the Python script on the phone first, then start serproxy.exe, then run the Flash application.

Possible Extensions/Improvements

As stated earlier, this is a quick toolkit to get you up and running with Flash-to-Bluetooth. There are definitely things that can be improved upon, but that we just didnít get the time to pursue. Here are a few:

Resources/Links

Contact

This system was put together by Neil Patel and Dean Eckles. Feel free to contact them with any questions and comments.