Debugging Problems with the Linux sunxi rtl8188eu Wireless Driver

New: Updates are at the bottom!

I have a problem with rtl8188eu wireless driver.  I develop the OS for the MICILE tablet system and sometimes the wireless modules fail to initialize on boot or reboot.  Needless to say, this makes for unhappy customers.  This posting is a preliminary analysis of the problem with a possible path to a solution

What do I currently know of the problem?

• Seems to happen when a lot of wifi networks are present
• When it happens, the entire USB bus that the module is on fails to appear when lsusb is called
• The rtl8188eu kernel module can not be unloaded, so only one chance to debug per reboot

WIFI Failed to Initialize!

So how can I solve such a problem?

Some items that need to be done before a solution can be found are:

1.  Replicate the problem reliably
2.  Need to sniff the USB power lines and the data packets
3.  Fix the rtl8188eu driver so it can be unloaded

To solve #1, I plan on finding every old wireless router I can and hook them all up in the same room to replicate an apartment with wifi on all sides.  I think 9 routers should do it, as that should form a grid of nine networks representing an apartment surrounded by wifi routers on all sides.  In addition, some kind of automated reboot loop will have to be written to power cycle the tablet until the problem occurs.  The problem only occurs 5 to to percent of the time, so some automated way must be found to make the problem happen.

Crowded Apartment WIFI Configuration

To solve #2, the tablet will be opened up, wires will be soldered to the onboard usb wifi module and a logic analyzer will monitor the usb bus and an oscilloscope will monitor the voltage rail.  Since the usb voltage is software controlled, it could be a bug in the way the power is brought up from a standby state.

USB Debug Points

To solve #3, I don't really care about the functionality of the driver because once the module comes up it works just fine.  I just need enough functionality in the driver to bring up the usb bus and the module and then fix the driver enough to be able to unload cleanly.  To that end, everything in between those two steps will be deleted from the module.

Wish me luck that the answer comes easily!

Day 2 Update!

Wires have been soldered to the USB pins.

USB Debug Wires

And interesting information has been found!  It was as I suspected!  When rebooting the tablet, the USB bus that the WIFI adapter is attached to does not power cycle correctly.

I instrumented the kernel with printk statements that clearly show the power to the USB bus should be turned off for several seconds during the boot sequence, but that is clearly not happening as shown by the logic analyzer hooked up to the USB power line.  On a cold boot however, it takes several seconds for the USB power line to come up, so that gives hope that it is in fact controlled by the AXP power controller and can be toggled.  My thinking is that if the power is turned off the USB correctly during a reboot sequence then the USB WIFI module will reset and come back properly every time.  Without the power cycle, the USB WIFI module can be left in a bad state between reboots and then fail to enumerate itself on the USB bus on a reboot.

From looking at the reference design schematics, it looks like the USB WIFI power should be hooked up to the LDO3 of the AXP209 power controller.  Let's hope the factory didn't deviate from the reference schematic.

USB WIFI Power Schematic

After some careful probing with a multimeter I can confirm that the power rail of the USB WIFI module is indeed connected to pin 41 (LDO3) of the AXP209 power controller chip.

I think we are almost there!  Now we just need to figure out how to turn on and off the LDO3 output of the AXP209 from the kernel.  Most likely at this point it will turn out to be a wrong setting in the script.bin configuration file for the tablet and need no code changes to the kernel or kernel drivers.  The datasheet for the AXP209 is in chinese, so this should be fun trying to figure out the proper I2C commands to turn the LDO3 output on and off.  The worst part is I just desoldered the very fine wires to the 0603 resistors used for spying on the I2C bus.  I'm not going to push my luck and try to reattach them so I will see if we can live without seeing what is happening on the bus.

More to come later!

MAME on MICILE (Part 1)

Overview

This is the first part of a multi part series on running the MAME (Multiple Arcade Machine Emulator) on the MICILE development system.

• Part 1 will show how to play a simple MAME game (this article you are reading)
• Part 2 will show how to write a front end to pick different MAME games (coming soon)
• Part 3 will show how to construct a little arcade cabinet to hold the tablet in viewable position so you can play all day long. (coming soon)

This not intended to be an exhaustive guide, there is lots of information about AdvanceMAME and Linux on the internet, so if you have more in depth questions, there is a wealth of information on the internet on how to fully use a MAME emulator on a Linux system.

Parts Needed

• MICILE development system

  The MICILE development system consists of hardware and software that allows a linux based operating system to run on Allwinner A13 tablets. It also allows the easy development of python applications that run on the tablet. www.micile.com Note: You must be running a MICILE Operating System build of at least Beta1 RC3 E6 for these instructions to work. You must be running a MICILE Operating System build of at least "Beta1 RC1 E1". This first batch of pre-release MICILE development system tablets shipped with "Beta1 RC1". See the MICILE web site for how to check your operating system version and how to upgrade to the latest version of the operating system.

• MAME (Multiple Arcade Machine Emulator

  MAME is an open source program which allows the emulation of arcade video games on a variety of hardware. It is so versatile that it has been successfully ported to run on everything from regular PC computers, smart phones, and even on a digital camera. MAME needs the ROMS for games in order to play them. A ROM is a dump of the software for the arcade game. http://advancemame.sourceforge.net/ You will also need ROM dumps from video game circuit boards that you own.  After purchasing a video game circuit board, you can get ROM dumps from places like www.romnation.net for games that you own the circuit boards of.

• Cheap chinese game pad off of eBay
  

  These are cheap generic gamepads sold on ebay that use the DragonRise chipset. I like cheap things. I got mine from this vendor on ebay. www.ebay.com

Assemble the hardware

• Turn off the MICILE tablet if it is on by holding down the power button for 10 seconds and then releasing
• Plug the gamepad into the tablet using the tablet's micro USB to USB Host adapter cable.
  
• Turn on the tablet by holding the power button down for 5 seconds and then releasing. If needed, configure the MICILE tablet's wifi.
• Wait for the tablet to finish booting. When the tablet has completed booting it will show its IP address on the screen.
• Activate analog mode on the gamepad by pressing the ANALOG button on the joystick. The analog LED will be lit when the gamepad is in analog mode.
  

Installing MAME

• MAME comes preinstalled on the MICILE tablet. I bet you thought this step would be a lot harder, didn't you?

Adding ROMs to MAME

• When the MICILE tablet boots, it will show you its IP address, as shown in the last picture of the Assemble the Hardware section. We will use that IP address to access the MICILE tablet. The MICILE tablet shares a folder on the network which the MAME ROMs for games should be placed in.
• To access the share folder, start by clicking on the Windows Icon on your taskbar.
  
• That will bring up the start menu. Type in two backslashes followed by the IP address of the tablet. The backslash key is usually located above the space bar on the keyboard and is shared with the question mark.
  
• A network window should appear showing the directories that the MICILE tablet is sharing on the network.
  
• Double click on the folder mame_roms. A login window should appear the first time. Login using the the username of root and a password of password
  
• Now drag and drop in your MAME ROMS. In my case, since I own an original Street Fighter II Circuit Board from a vintage standup arcade game of Street Fighter II and I feel like playing Street Figher II, I will be installing the ROM for Street Fighter II in this example.
• Here is a shot of my vintage Street Fighter II circuit board. It is amazing how much they were able to do with such little computing power 20 years ago. Look at the size of those chips!
  

Launch MAME

• Since we don't have a front end for MAME yet, we will need to launch MAME from a shell terminal. This will require a PC with a web browser running chrome. Type in the IP address of the MICILE tablet into the chrome browser and the MICILE web based IDE (Integrated Development Environment) will launch.
  
• Click on the button labeled "Open Terminal". If this is the first time you have ever launched the shell, it will take up to 10 minutes for the tablet to generate its ssh encryption keys. Other times it may take up to a minute for the shell. Be patient, or use your favorite ssh client if you know how to use ssh already.
  
• Eventually, the shell will ask you for a MICILE login otherwise known as a username. The default username is root. The default password for root user is password. When you type in the password nothing will appear on the screen as you are typing. This is expected. Just go ahead and type the password and press enter when the shell asks for the password.
  
• Following my example earlier of adding the Street Fighter II ROM named sf2, we will launch Street Fighter 2 by typing the following command into the shell:

  run_advmame sf2

  
  The screen show below is out of date, please use the correct updated command of run_advmame instead of advmame!
• A few seconds later, the load screen of the game will appear. Read the load screen and follow it's instructions.
  
• Once past the load screens and the game is running, use the SELECT button on the gamepad to add a coin. Use the START button to start the game after adding the coin.
  
• So this is the game in action
  

What's Next?

• It's not too cool to have to lug around a laptop every time we want to play an arcade game. In Part 2 we will write a front end for MAME that runs on the MICILE tablet everytime the tablet boots and will allow us to choose a game using the touchscreen so the system becomes self contained. I've also got some cheapy wireless gamepads I picked up off ebay that I want to try.