Connecting a Desktop Flash Application to a Bluetooth-Enabled Mobile Phone
Motivation
An interesting interaction paradigm is interfacing with software applications on projection/big-screen displays in common spaces.
If the design goals are to encourage shared or group interaction as well as to constrain physical movement as little as possible,
the traditional keyboard and mouse may not be the best choices as input devices. Mobile devices offer a variety of benefits in this
scenario, including high user familiarity and penetration (at least in the developed world), rich data input capabilities, and of
course affordance of high physical mobility.
For a class project, we designed a video-based social headline system called Break’n’News, which supported creation and consumption
of videos in a corporate break room to promote social interaction among co-workers in remote offices. We built the Flash-based
application interface to run on a 40-inch LCD screen, and used the system described on this page to connect the application to mobile phones.
The project was a success, and a big reason why was that the use of a mobile phone as a remote control for the application was very compelling.
We thus thought it might be useful to show others how to set up the system, so that they may incorporate it into their own projects.
Introduction
This page will provide you with all the software and instructions needed to get a Macromedia Flash
application (.swf) that is running
on your desktop to communicate with a mobile phone through Bluetooth. The software provided is quite basic, but should provide
a sufficient illustration of how things work so that you may go off and create a more sophisticated system. Here, we will step through
setting up the "pipe" that will allow the phone to navigate and click buttons on a simple Flash app with the phone's D-pad, as well
as enter text with the phone's soft keys, prompted by the app itself.
At the top of this page is an illustration of the pipe we will be building to bridge the Flash app ("app") to the Nokia S60 phone ("phone").
The phone is running a Python script that listens for key presses on the phone. Once this occurs, the script encodes the key press and
sends the data off to a serial port on the PC, through the Bluetooth connection. A piece of software called serproxy.exe is listening for data
on the port, and once it receives it, forwards on to a specified TCP port. The Flash application then reads from that TCP port using
the XMLSocket class, and passes the information on to listeners in the application that update state depending on the action from the phone.
Ingredients
To complete the steps below, you will need the following hardware and software components:
Setup
-
Make sure Bluetooth is enabled on both your PC and mobile phone,
and that the two devices are paired together (i.e. they show up in each other's respective known device menus).
-
On your PC, find out what COM port is accepting incoming serial data. I did this on my PC by going to the Control Panel and clicking on
"Bluetooth Local COM" and finding the COM port corresponding to "Local-COM Server [SerialPort...". If you don’t see such a port,
you may have to create one.
-
Remembering that port number, open up serproxy.cfg with a text editor like WordPad from the directory where you unzipped the serproxy package.
-
Set the comm_ports variable to the port number from above. You can remove the other port numbers. Change the variable comm_baud to 115200.
-
Change net_port1=5331 to be net_portn=5331, where n is the port number. You can remove the other port settings. Save the file and close it.
-
Make sure the latest version of Python is installed on your mobile device. If you don’t have it, you can download it from the link above.
-
Open up navigation.py in a text editor and enter your PC’s Bluetooth address at the top of the file, where indicated.
You can get your address by hovering over the Bluetooth icon in your systray or something similar. Mine is entered by default.
-
Send navigation.py to your mobile device from your PC (this is a good way to make sure step 1 is done)
and install the file as a python script (not a lib module).
-
Go to Python on your mobile and run the script you just installed. Once you do, there should be a white
screen and nothing else happening.
-
Execute serproxy.exe. A shell should appear.
-
Run the movie FlashToBluetoothApp.fla. When you do, the following should appear in the serproxy shell:
Serproxy - (C)1999 Stefano Busti, (C)2005 David A. Mellis - Waiting for clients
Server thread launched
server(7) - thread started
-
Once the flash app is launched, click allow on the security settings and now use
your phone’s D-pad arrow keys to navigate, with enter corresponding to the middle (push).
Note: When restarting the system, the order in which the components are run matters; You should always run the Python script on the phone first,
then start serproxy.exe, then run the Flash application.
Possible Extensions/Improvements
As stated earlier, this is a quick toolkit to get you up and running with Flash-to-Bluetooth. There are definitely things that
can be improved upon, but that we just didn’t get the time to pursue. Here are a few:
-
Currently the PC’s Bluetooth address must be entered manually into the Python script. This is inflexible. Auto-detection of the nearby host would be preferred.
-
Multiple phones attempting to connect at the same time doesn’t currently work; the Phyton script cannot open a socket on the second phone that
attempts to connect. Ideally, any number of devices can simultaneously communicate with the application.
-
A great bit of functionality to have in this system would be presence detection; not only to allow the application to identify the user through
Bluetooth address, but to also determine how close users are to the display. This capability would allow the application to do things
like have an idle/ambient state that would change as soon as a user stepped into a defined zone, or delegate control to the application
based on proximity. One possible way to approximate physical distance could be to compare Bluetooth signal strength.
Resources/Links
Contact
This system was put together by Neil Patel and Dean Eckles. Feel free to contact them with any questions and comments.
